Do you need a speech-to-speech model?

Most posts comparing voice agent architectures are written to sell you one. This one is written so you can stop thinking about it after ten minutes and go back to building.

The short answer: a speech-to-speech model is worth it when the conversation depends on how something is said, not only on what is said. Everything below is how to tell which one you have, using your own calls rather than a vendor's demo.

If you want the mechanics of why the two architectures differ, that is a separate piece on speech-to-speech versus cascaded voice agents. This one is only about whether you need it.

What is the one question that decides it?#

Ask whether the information you need is in the words or in the voice, because that single distinction predicts the right architecture better than anything else on a feature matrix.

A transcript carries the words perfectly well. If your agent's job is to find out which order the caller is asking about, look it up and say when it ships, then every piece of information that matters survived the conversion to text, and a streaming cascade of speech to text, a language model and text to speech will do that job cheaper, with a transcript your compliance team already accepts.

Now take the same sentence, "my order still has not arrived", said by someone who is mildly curious and by someone who has called three times this week and is done being polite. Identical transcript. Different correct response, different escalation, different outcome for the account. A cascade cannot tell them apart, because by the time the language model sees the turn, both callers are the same string. This is measurable rather than rhetorical: a 2026 analysis of four open audio models traced speaking style through the stack and found it strongly encoded in the upper layers of the audio encoder and consistently degraded before it reached the output.

That is the whole decision, and the rest of this post is how to measure it.

Five questions, and what each answer means#

Run these five against your own product rather than against the general case, because the general case is exactly what makes this choice look harder than it is.

Two yes answers in the first two rows and you are in speech-to-speech territory regardless of what the rest say. Two no answers there and the question is settled the other way, no matter how modern the alternative sounds.

The last two rows are constraints rather than preferences. A verbatim transcript requirement does not rule speech-to-speech out, but it does rule out any model that emits audio tokens alone, because then you are running a separate speech-to-text pass purely for the logs and you have quietly put half the cascade back into your stack.

Who it is actually for#

Seven profiles come up again and again, and what they share is that the caller's voice carries information the transcript does not.

Drive-thru and counter ordering. Background noise is constant, people change their mind inside a single sentence, and the queue behind them turns every half second into money. Interruption is the normal mode of the conversation rather than an exception, and nobody needs a transcript of a burger order.

Roadside assistance, emergency and incident intake. How the caller sounds tells you the priority before the words do. Panic, breathlessness, a motorway in the background, a smoke alarm. "My car stopped" from someone calmly parked on a driveway and from someone on a hard shoulder in traffic are the same five characters and completely different tickets.

Companions, coaching and language tutoring. Here prosody is not a nice-to-have, it is the product. A tutor has to hear the hesitation to know the learner is unsure rather than wrong. A companion that reads every line in the same even tone has failed at the only job it was hired for.

Front-line sales qualification and outbound calling. People decide whether they are talking to a robot in the first couple of seconds, and they interrupt constantly to cut to the point. An agent that cannot be interrupted gracefully gets hung up on, and a hang-up is the most expensive outcome in the funnel.

In-car and hands-busy assistants. The user cannot look at a screen, speaks in fragments, and is sitting inside continuous road noise. Half-duplex turn-taking feels broken when your hands are on a wheel and you want to correct the destination mid-sentence.

Accessibility and elderly care. Slow speech, long pauses in the middle of a thought, repetition, and a lot of "sorry, what?". Fixed-silence endpointing cuts exactly these users off, and this is the population least likely to give your agent a second try after it talks over them once.

Games, toys and interactive entertainment. Expressiveness and response time are the entire experience, there is no compliance requirement, and there is no transcript anyone will ever read.

Who it is not for#

An honest list of the cases where picking speech-to-speech costs you money and buys you nothing, which is most of the enterprise voice market today.

Identity and payment verification, insurance claims intake, anything where the recording and its transcript are the legal record. The audio here is a formality around a data-collection task, the caller is reading numbers off a card, and tone carries no information you are allowed to act on anyway.

Agents that are really workflow engines. Look up the booking, check availability, move it, confirm, send the email. The hard part is the tool calls and the state machine, both of which live in the text layer where they are a solved problem, and voice is a rendering step at either end.

White-label and multi-tenant products, where each customer wants their own brand voice and their own language list. In a cascade that is a configuration field you change per tenant. In a speech-to-speech model the voice is part of the model, so it becomes a training problem or a hard no.

Anything dominated by the agent reading things out. Confirmations, disclosures, legal scripts, IVR prompts, appointment reminders. The job is a clean, consistent, neutral read at a sane cost per character, which is what text to speech is for, and we have written up how to pick one and what the market actually charges.

How do you decide in one afternoon?#

Pull twenty real call recordings and count four things. This beats every demo you will be shown, because a demo is a scripted conversation in a quiet room with a cooperative human.

Interruptions per call. Count every time the caller speaks while the agent is still speaking. Above roughly one and a half per call, turn-taking is a real cost centre and speech-to-speech is solving a problem you have.

Turns where tone changed the answer. Listen for turns where a transcript alone would have led you to the wrong response. If that is more than about one turn in ten, the text bottleneck is costing you outcomes rather than just elegance.

Your current delay from end of speech to first syllable. If it is above 1.2 seconds, do not change architecture yet. You have cheaper wins in your existing pipeline, usually in endpointing and in how early you start generating, and you should take those first so that you are comparing a tuned cascade against the alternative rather than a neglected one.

Whether compliance treats a verbatim transcript as mandatory. Ask before you prototype, not after. The answer decides which models are even eligible.

Four numbers, one afternoon, and they come from your callers instead of from a vendor's landing page.

What does it cost you to get this wrong?#

Less than the decision feels, because the expensive parts of a voice product are the same under both architectures.

The telephony path, the business logic, the integrations with your CRM and booking system, the evaluation harness, the call recordings and the prompt library all survive a change of audio architecture. What you rewrite is the orchestration layer and the turn-taking logic. That is real work, but it is weeks rather than a rebuild, and it is a much smaller number than the cost of shipping nothing for a quarter while you deliberate.

The asymmetry worth knowing is the hardware floor. Running a cascade, you can self-host the text-to-speech leg on a single consumer card: our own open model is 555M parameters under Apache 2.0 and runs on one retail RTX 4090, and the benchmark numbers are public. Self-hosting a current open speech-to-speech model means an 80 GB datacentre GPU and, for the strongest one available today, a research-use licence. There is also a sobering result worth reading before you commit a quarter to this: The Cascade Equivalence Hypothesis argues that in most deployed use cases current speech models behave like expensive cascades, and that under noise their clean-condition advantage can reverse by up to 7.6 percent at 0 dB. If your reason for self-hosting is data residency or cost, check that gap before you commit, and our write-up on open weights covers why that combination matters.

What most teams actually ship#

The honest answer in late 2026 is a hybrid, and it is worth knowing that before you frame this as a binary.

One shape puts a speech-to-speech model at the front to own listening, turn-taking, interruption and backchannels, while a text model behind it does the reasoning and the tool calls. You get the conversational feel where it is perceived and the mature text tooling where the work happens. Kyutai's Moshi is the open reference implementation if you want to read how the two parallel streams are actually modelled.

The other keeps the cascade and adds a paralinguistic side channel, so the language model receives an emotion and intent label alongside the transcript rather than the transcript alone. It is less elegant and far cheaper, and for a lot of products it closes most of the gap.

Both are uglier than a diagram with one box. Both are in production right now.

So what should you do#

Count the interruptions, count the turns where tone mattered, and let those two numbers decide. If they are low, build the cascade, ship it, and spend the saved months on the parts of the product your customers will actually notice.

If they are high, the thing to check next is not which model is best on a leaderboard, it is whether you can run one at all within your licence, your latency budget and your GPU budget. That is where this decision is currently made, and it is a more boring constraint than the architecture argument suggests.

And if you are on the phone network specifically, own the call path first. We wrote about running a voice agent on jambonz rather than on a closed platform, and that choice is worth making before the model choice, because it is the one that keeps both options open.