The first voice agent I shipped was accurate and completely unusable. You asked it a question, waited a beat too long, assumed it had died, and started talking over it. The model was not slow. The pipeline was.
A voice turn is four hops — speech to text, retrieval, the LLM, text to speech — and the naive version runs them one after another. Every hop waits for the one before it to finish, so the user pays the sum of all four. Humans start feeling the lag at about 800ms. Sequential hops blow past that easily.
Three changes did most of the work.
Start transcribing before the user stops talking. Streaming ASR gives you partial transcripts mid-sentence. Kick off retrieval on the partial as soon as the intent is clear, rather than waiting for the endpoint signal.
Stream the LLM into the TTS, sentence by sentence. This is the big one. Don't wait for the full completion — cut the token stream at sentence boundaries and hand each piece to the synthesizer while the model is still writing the rest.
let buffer = "";
for await (const chunk of llm.stream(prompt)) {
buffer += chunk;
// Flush on sentence boundaries so TTS starts on sentence one
// while the model is still generating sentence three.
const match = buffer.match(/^(.*?[.!?])\s+(.*)$/s);
if (match) {
const [, sentence, rest] = match;
void tts.speak(sentence);
buffer = rest;
}
}
if (buffer.trim()) void tts.speak(buffer);Time-to-first-audio now depends on the first sentence, not the last one. That single change took a 2.3s felt latency down to roughly 600ms — the rest of the response generates while the user is already listening.
Make barge-in cheap. Users interrupt. If interrupting means tearing down and rebuilding the whole pipeline, the recovery costs more than the original delay. Keep the sockets warm and cancel only the synthesis queue.
None of this makes the model smarter. It just stops the pipeline from hiding how fast the model already was.