Speech-to-speech vs cascaded voice agents

A cascaded voice agent is three models in a row: speech to text, a language model, then text to speech. A speech-to-speech model is one model that takes audio in and emits audio out. The first design is what almost every voice agent in production runs on today. The second is better at being a conversation, and the reasons have less to do with speed than most people assume.

Here is the short version. A cascade throws away everything in a voice that is not a word, pays for a turn-end decision it has to guess from outside the model, and cannot listen and speak at the same time. A speech-to-speech model keeps the audio, learns the turn-end from the audio, and can do both at once.

Here is the other short version, because we would rather you trust the rest of this page. A cascade is still better at reasoning, at tool calling, at handing you a transcript, and at running on a GPU you can actually buy. Both halves are below, with the numbers.

What is a speech-to-speech model?#

It is one network that reads audio and writes audio, trained end to end on conversations rather than assembled from three parts that were each trained on something else.

Mechanically it looks a lot like a text language model with a different alphabet. A neural audio codec compresses the waveform into discrete tokens, usually with residual vector quantisation, so one short frame of speech becomes a small stack of integers instead of thousands of samples. A transformer predicts the next frame of those tokens. A decoder turns the predicted tokens back into sound. Frames are short, on the order of 80 ms, which is why the model can start speaking before it has decided how the sentence ends.

Two design details matter more than the rest.

The first is the second token stream. The strongest models do not predict audio alone, they predict time-aligned text next to it. Kyutai's Moshi calls this Inner Monologue and reports that it significantly improves linguistic quality, which makes intuitive sense: text is where the model's grammar and world knowledge already live. The side effect is the thing your logging pipeline wanted anyway, a transcript, produced by the model rather than by a fourth vendor. The code and weights are public if you want to read a real implementation.

The second is how many streams the model listens to. A half-duplex model takes turns like a walkie-talkie. A full-duplex model models its own speech and the user's speech as parallel streams, so silence, overlap and interruption are inputs rather than exceptions. Moshi reports a theoretical latency of 160 ms and 200 ms in practice, which is inside the gap humans actually leave between turns.

What a cascade actually costs you#

Every arrow between two models in a cascade is a place where the next stage cannot start until the previous stage has produced something parseable, and that constraint costs you three separate things.

The first cost is time, and it is not only inference time. Speech to text has to commit to a word before it can ship it. The language model needs enough tokens to form a clause. Text to speech needs a clause to choose prosody, because you cannot pick the melody of a sentence you have not finished reading. A good streaming implementation overlaps most of this, which is why the honest comparison is against a streaming cascade and not against a naive one.

The second cost is information. The only thing that crosses from stage one to stage two is a string. Everything else the microphone captured is gone at that boundary.

The third cost is a decision. Somebody has to tell the pipeline that the caller has stopped talking. That somebody is a heuristic sitting outside all three models, working from silence duration or from a classifier over a partial transcript, and it is wrong in both directions: too eager and it cuts people off mid-thought, too patient and the agent feels slow on every single turn.

Why does speech-to-speech feel faster?#

Because it removes the turn-end guess and the handoffs, not because any single model in it is quicker.

Published figures for the two architectures land roughly here. These are ranges from the public literature and vendor write-ups rather than our own harness, so treat them as the shape of the problem rather than as a measurement.

Two notes on that table, one in each direction.

In the cascade's favour: the stages overlap, so the real total is closer to the slowest stage than to the sum, and the delta between a tuned cascade and an S2S model is smaller than the raw arithmetic suggests. If your cascade is sitting at two seconds, the architecture is not your problem yet.

Against it: the turn-end line is the one that does not overlap with anything. Fixed-silence endpointing has traditionally waited about 500 ms of quiet to declare a turn over, and semantic endpointing gets that down to roughly 150 to 250 ms, but both are still a decision made from outside the model on incomplete evidence. Studies of human turn-taking across ten languages put the usual gap between speakers near 200 ms. A pipeline that spends its whole human-sized budget deciding whether you are finished has nothing left for thinking.

An end-to-end model can also start working before the turn ends, which a cascade structurally cannot. "Can Speech LLMs Think while Listening?" is the clearest result here: reasoning in text space raised accuracy on spoken reasoning tasks by 2.4x on average, and letting the model begin that reasoning before the user finished the question, guided by an entropy-based question-completeness signal, produced a 70 percent reduction in latency with no loss in accuracy.

One more thing worth saying plainly, since we build text to speech for a living: the TTS leg is usually not where a slow cascade is slow. We measure 68.7 ms to first audio at p50 for Gepard with 20.5 ms of leading silence, and the full methodology is published along with the parts where we do not look good. If you are chasing 300 ms of turn delay, the 70 ms leg is not the one to optimise. Our WebSocket latency write-up has the same conclusion from the other direction.

What survives the text bottleneck#

Nothing survives except the words, and the words are a small fraction of what a caller actually sent you.

Run through what a microphone captures and a transcript does not. Emotional state: angry, panicked, bored, relieved. Certainty: the difference between someone reading a code off a card and someone guessing at it. Emphasis, which changes meaning outright, because "I said nine nine six" with the stress on the last word is a correction and the same string with flat stress is not. Rate and volume, which is how you hear that a caller is walking into a hospital or driving. Hesitation and filled pauses, which is how a human knows to wait. Laughter, crying, sighing. Accent and age cues, which drive how you should speak back. Overlapping speech from a second person in the room. Background sound, which tells you whether the caller can hear you at all.

That is not a style complaint, it is the substance of the call. A frustrated caller and a calm caller sending the identical sentence need different responses, and a cascade cannot tell them apart because by the time the language model sees the turn, both are the same string.

The research here is more nuanced than the marketing, and it cuts both ways. "Heard but Not Heeded", from September 2026, traced speaking style through four open audio models and found that all of them encode style strongly in the upper third of the audio encoder's layers, but that the information is consistently degraded before it reaches the output. Some models end up content-driven, where predictions depend mainly on the words, and some end up acoustic-driven. The authors call it a gap between what models encode and what they use. So feeding a model raw audio is necessary for keeping paralinguistics, and it is not automatically sufficient. Ask a vendor for evidence that their model acts on tone, not just that it receives it.

The output side has the same asymmetry and it is simpler to state. A text-to-speech model is handed a sentence and no conversation. It cannot know that the caller is upset, that this is the third time they have repeated an order number, or that the right move is to slow down. The best it can do is a consistent, well-chosen neutral read, which is exactly what we build for and exactly where the ceiling is. We wrote up what that ceiling is made of in how text to speech works. A model that generates speech conditioned on the audio of the whole conversation does not have that ceiling, because the emotional context never left the model.

Why does barge-in break most voice agents?#

Because being interrupted is not an edge case on a phone call, it is how people steer a conversation, and an agent that handles it badly reads as stupid no matter how good its answers are.

Start with how often it happens. Language models are verbose by default, and callers interrupt to correct them, to skip ahead, to say "no, the other one", to answer before the question finishes. If an interruption is expensive, every wrong turn is paid for twice: once while the agent says the wrong thing in full, and once while it recovers. This is where containment rate goes to die and where "let me talk to a human" comes from. People do not rate an agent on its best answer, they rate it on what happens when they cut in.

Now the five places it actually breaks.

The microphone is full of the agent's own voice. While the agent speaks, its output is in the input. The usual fix in a cascade is to gate the microphone while playing audio, or to lean on acoustic echo cancellation. Gating means the agent is deaf for exactly as long as it is talking, which is why so many deployed agents cannot be interrupted at all, and echo cancellation is one more component to tune per device, per codec and per network. A model trained on two parallel streams was trained on this overlap rather than protected from it.

Energy cannot tell an interruption from a noise. Voice activity detection sees loudness. It cannot separate "mhm" from "no, stop", a cough from a correction, a second person in the room or a television from the caller. Tune the threshold to be responsive and the agent halts mid-sentence at every throat clear; tune it to be safe and it talks over people. No threshold fixes both, because the difference is semantic and prosodic rather than energetic, and those are precisely the features the transcript threw away.

Nothing records what the caller actually heard. This is the bug nobody budgets for. The agent is reading a twelve-word booking reference and is cut off after the seventh word. The conversation state has to carry the seven. A cascade knows what it handed to the text-to-speech vendor, not what left the caller's speaker, so it believes it read the whole thing and later asks the caller to confirm a number they never heard. An end-to-end model generated that audio itself, frame by frame, so the point where generation stopped is the point where the caller stopped hearing.

Stopping has its own latency budget, and nobody writes it down. If the agent keeps talking for half a second after the caller starts, both sides are speaking at once, both back off, and you get the awkward double pause that makes people hang up. Time to stop belongs in the same budget as time to first audio, and in a cascade it spans the detector, the orchestration layer and whatever audio is already buffered in the playout path.

There is no resume policy. After an interruption, should the agent repeat the sentence, continue from where it was, drop it, or answer the new thing first? A cascade has to be told, usually with a rule that is wrong in half of the cases. A model trained on real dialogue has seen all four and picks by context.

To be fair to the cascade: with good echo cancellation and a clean stop protocol, barge-in can be made to work acceptably, and plenty of production agents do. The two parts that stay hard are knowing what was said during the overlap and knowing where the agent's own speech was cut, and both of those are architectural rather than a matter of tuning.

On telephony it is harder again. There is one mixed audio channel rather than a clean near end and far end, the band is 8 kHz, jitter moves the playout point around, and the carrier may be doing echo control of its own. If you are building for the phone network specifically, running the call path yourself at least puts those knobs where you can reach them.

Turn-taking is more than a stop button#

Barge-in is the loud half of turn-taking. The quiet half is everything a listener does while not taking the turn, and that is where agents stop sounding like software.

Backchannels are the one almost nobody implements. During a long explanation a human listener says "mhm", "right", "yeah" without claiming the turn. A cascade structurally cannot: to produce any audio it first has to decide the user has finished, which is the opposite of what a backchannel means. A model with two streams can emit one short acknowledgement while still listening, and the effect on how a caller experiences the agent is out of all proportion to how small the change looks on a diagram.

Graceful overlap is the last piece. Two people talking at once is normal human behaviour, not a fault condition, and recovering from it means remembering both streams rather than discarding one. Set that against the 200 ms gap humans actually leave between turns, quoted earlier: a pipeline that needs longer than that just to decide a turn ended has no room left to handle two voices.

This is why full duplex is an architectural property rather than a feature flag, and it is worth being blunt about the vocabulary: when a vendor says their cascade supports barge-in, ask whether the model receives the audio it was interrupted with, or whether there is simply a stop button on the player.

What else gets simpler#

One model means one deployment, and the operational savings are larger than they look on a diagram.

In a cascade, your p99 is the worst p99 among three vendors, and any one of them can hand you a 429 on a live call. You have three rate limits, three billing relationships, three places where a version bump changes behaviour, and three sets of credentials rotating on different schedules. Debugging a regression means working out which leg moved, which is a genuinely annoying class of incident because the symptom is always the same: the agent got slow.

An end-to-end model collapses that to one version pin, one GPU pool and one place to look. If you run it yourself, it is also one thing to keep warm, which matters more than it sounds: cold start on three services is three cold starts. We went through the arithmetic of owning the serving layer in self-hosted versus an API, and most of it transfers.

Where speech-to-speech still loses#

It loses on reasoning, auditability, control and hardware, and these are exactly the four things an enterprise buyer asks about first.

Reasoning is the best-documented gap. Speech LLMs reason measurably worse on audio input than the same backbone does on the equivalent text, even when the transcription is correct. "Closing the Modality Reasoning Gap" frames it as representational drift across layers plus behaviour deviation on long chains, and proposes alignment training to narrow it on MMSU and OBQA. Narrow, not close.

Then there is the uncomfortable result. "The Cascade Equivalence Hypothesis", from February 2026, used matched-backbone testing and mechanistic probing to argue that in most deployed use cases current speech LLMs behave like expensive cascades: the literal transcript emerges from the model's hidden states, text representations are causally necessary, and under noise the picture gets worse, with clean-condition advantages reversing by up to 7.6 percent at 0 dB. If you are buying S2S for robustness in a noisy car or a loud shop floor, test that specifically. It may be the exact condition where your cascade was better.

Auditability is a procurement problem rather than a research one. If the model does not emit text, you have no verbatim record without adding a speech-to-text pass purely for logs, and redacting a card number from an audio stream is harder than redacting it from a string.

Control is the quiet one. In a cascade the voice is a configuration field: you change one line and the agent sounds different, in a different language, with a different brand. In an S2S model the voice is part of the model, so switching it is a training or prompting problem and sometimes not possible at all. The same applies to domain vocabulary, where a cascade lets you bias the recogniser toward your product names and part numbers.

Hardware and licensing is where it gets concrete.

The last two rows are the gate most teams hit first. The strongest open full-duplex model as of October 2026 is NVIDIA's NemotronLabs VoiceChat, released 3 August 2026 at 11B parameters in a hybrid Mamba and Transformer stack, scoring 38.8 percent on Big Bench Audio and second among open full-duplex systems on VoiceBench. It also will not run on less than 80 GB of GPU memory and ships under a research-use licence. On the hosted side the picture is better: OpenAI's gpt-realtime-2 landed on 7 May 2026 with parallel tool calling over WebRTC, WebSocket and SIP, with gpt-realtime-2.1 and a mini tier on 6 July 2026. If you need open weights and commercial terms on a card you can buy, that combination is still thin. We have written before about why that combination is the one worth holding out for.

So which one should you ship?#

Ship the cascade if the call has to be auditable, and ship speech-to-speech if the call has to feel like a conversation. Most teams know within one sentence which of those they are.

Pick a cascade when you are in a regulated industry and a verbatim transcript of both sides is a requirement rather than a nice-to-have, when the agent does real multi-step work with tools and APIs, when a specific brand voice or a long list of languages is part of the product, or when you are on telephony and the audio path is already constrained by carriers and codecs. That last case is more common than the discourse suggests, and it is why we wrote up running a voice agent on jambonz rather than on a closed platform.

Pick speech-to-speech when turns are short and conversational, when the caller's emotional state changes what the right answer is, when people interrupt constantly, or when perceived latency is the product rather than a metric. Companions, tutoring, drive-thru ordering, roadside assistance and front-line qualification all live here.

The interesting answer is the hybrid, and it is what the better teams are quietly already running. One version puts an S2S model at the front to own listening, turn-taking and backchannels while a text model does the reasoning and the tool calls behind it. Another keeps the cascade and adds a paralinguistic side channel, so the language model receives an emotion and intent label alongside the transcript instead of the transcript alone. Both are uglier than a diagram with one box, and both are shipping today.

What decides this is not which architecture is more modern. It is which one you can run on hardware you can get, with a record your compliance team will accept, at a delay a human will forgive. Today that answer is genuinely different for a drive-thru and for a bank. It is not going to stay different forever, and the number to watch is the GPU floor, because that is the line that has moved fastest in every other part of this stack.