Real-time voice: barge-in, handoffs, and three languages on one brain

Phone calls are the hardest realtime surface I've shipped on: sub-second latency budgets, no undo, callers who interrupt, switch topics, and switch languages, sometimes in one sentence. My production fleet runs English, Mandarin, and Spanish on the same agent brain with different phonetic front-ends, and every hard-won lesson generalizes to realtime agents anywhere.

Barge-in is table stakes and harder than it sounds. Callers talk over the agent constantly; the pipeline must detect it fast, stop speaking, and, the subtle part, reconcile state: did the caller hear the appointment time before interrupting, or does confirmation need to restart? Treat everything the agent says as unacknowledged until evidence says otherwise. Optimistic assumptions about what the caller heard become double-bookings.

Mid-call tool calls fight the latency budget. Checking a calendar or saving a screening answer takes real seconds; dead air kills trust. The choreography is conversational cover, acknowledge, narrate briefly, keep the turn alive, while the tool call runs. And multilingual isn't translation: politeness norms, number formats, and interruption patterns differ per language. Same brain, per-language conversation design.

The most important engineering decision is when the agent should stop being the agent. Handoff logic runs on explicit triggers, topic classes that must escalate, frustration signals, repeated misunderstanding, and a good handoff delivers the human a summary and a caller who isn't angry yet. Voice agents earn trust by knowing their limits out loud.