How Call AI works

01

Overview

One call loop drives both the browser demo and the phone. A CallSession knows nothing about WebSockets, Twilio, or audio formats — it streams audio to Gemini Live, dispatches tool calls to the mock loans-and-deposits data, and runs the escalate/end-call lifecycle. Everything specific to how the audio arrives sits behind a small Transport interface, implemented once for the browser and once for Twilio.

02

Request flow

What actually happens between a caller speaking and the agent replying:

1. 1

   Caller speaks

   Raw mic audio in the browser, or 8 kHz μ-law over the phone network via Twilio Media Streams.

2. 2

   The transport normalizes it

   The browser audio is already 16 kHz PCM16 (the Web Audio API resamples the mic for free). Twilio's audio is decoded from μ-law and resampled 8 → 16 kHz. Either way, CallSession receives the same 16 kHz PCM16 stream.

3. 3

   Gemini Live streams back

   Continuously, as the conversation runs: speech audio, live transcripts, or a tool call.

4. 4

   On a tool call

   CallSession looks up the handler by name and runs it with the call's CallContext — the same object verify_customer set earlier. The handler never receives a customer ID from the model; it reads the verified one from the context.

5. 5

   The result goes back

   Gemini continues speaking using whatever the handler returned — never a number it made up itself.

6. 6

   If the caller talks over it

   Gemini reports interrupted; the transport clears whatever audio was already queued for playback.

7. 7

   Ending the call

   escalate_to_agent or end_call sets state on the context. CallSession waits for the goodbye audio to actually finish playing, then tells the transport to close — never mid-sentence.

03

The transport abstraction

Adding Twilio meant writing one new file — protocol framing, μ-law conversion, waiting for Twilio's mark echo before hanging up — and changing nothing in the agent, the tools, or the escalate/end-call state machine. Both transports satisfy the same five-method shape:

class Transport(Protocol):
    async def receive_audio(self) -> bytes | None: ...
    async def send_audio(self, pcm24k: bytes) -> None: ...
    async def clear_playback(self) -> None: ...
    async def send_event(self, event: dict) -> None: ...
    async def finish(self) -> None: ...

This paid off once already. The rule that a call must only end after its goodbye audio finishes — not on the network event that triggers it — lives entirely in CallSession. Fixed once, both the browser and the phone got it for free. The split was tested by swapping the transport under a fake WebSocket and confirming the browser call flows were unaffected, before a line of Twilio code existed.

04

Audio pipeline

Gemini Live takes 16 kHz mono PCM16 in and returns 24 kHz PCM16 out — two different rates, in two different directions. Phone networks add a third: 8 kHz μ-law, a compressed 1-byte-per-sample format designed for voice-grade telephony, not for a neural model.

The browser sidesteps most of this — asking for a 16 kHz AudioContext makes the browser resample the mic for you, and the returned audio is played at the 24 kHz rate it already arrives in. The Twilio transport does the real conversion: μ-law decode, then 8 → 16 kHz for Gemini; 24 → 8 kHz then μ-law encode for the caller.

The resampler is stateful on purpose. Phone audio arrives as 20 ms frames, and resampling each frame in isolation — treating it as a standalone signal — produces audible clicks at every frame boundary. Carrying the filter state between calls to audioop.ratecv removes them.

05

Security & verification

Authorization lives in code, not in the prompt. A prompt can be talked around; a function can't. The verified customer ID is stored on a per-call CallContext that only verify_customer can set. Data tools take no customer ID from the model — they read it from the context — and the data layer refuses to return any record that belongs to someone else. Asking for another customer's loan, claiming to be a spouse, or saying "ignore your instructions" all end at code that has nothing to return.

The model never decides when the call ends. Escalation and hang-up set state on the server; the server waits for the goodbye audio, tells the transport, and the transport lets it finish playing before closing.

It can't claim actions it didn't take. Early testing surfaced a real failure: the model announced "connecting you to a human agent" when no tool existed yet to do that. The prompt now forbids describing any action before its tool has actually returned success, and escalation is honestly a callback, not a live transfer.

06

Tools

Five functions, each backed by mock loans-and-deposits data, described to Gemini as callable tools.

07

Latency

Time from the end of the caller's speech to the first byte of audio back, measured against the same speech clip, two runs each from a development machine — indicative, not a benchmark suite:

gemini-3.8-live is what the agent runs on — chosen by measurement, not by release date.

08

Stack