How text to speech works, and how to choose

Choosing a text-to-speech engine looks like a comparison-shopping problem and is actually a measurement problem. Every vendor publishes numbers. The numbers describe different things, and several of them are easy to make look good without making the product better.

This is a working explanation of how these systems are built and which of the published numbers to believe, aimed at someone about to pick one.

How does modern text to speech actually work?#

Two stages: a language model that predicts audio tokens, and a neural codec that turns those tokens into sound. Nearly every current system, ours included, has this shape.

The text is normalised first, which is duller and more important than it sounds: $1,204.50 has to become words, and how a system does that decides whether it says "twelve oh four" or "one thousand two hundred and four".

Then a transformer reads the text and predicts discrete audio tokens. These are not samples. They are indices into a learned codebook, produced by a neural codec at something like 21.5 frames per second across a few dozen channels. In our case that is NVIDIA's NanoCodec running at 22.05 kHz and 1.89 kbps, which is a startling amount of compression for speech that still sounds natural. The quantisation scheme underneath it is finite scalar quantisation, which replaced the older vector-quantised codebooks in most current systems.

Finally the codec decodes those tokens back into a waveform.

The reason this architecture won is that it turns speech synthesis into next-token prediction, which is the one problem the field has poured a decade of engineering into. Everything that made language models fast applies directly.

How we got here#

Worth thirty seconds of history, because it explains why the older options still exist.

Concatenative systems recorded a voice actor reading for dozens of hours, chopped the result into units, and glued them back together. When it worked it sounded exactly like the actor, because it was the actor. When it failed at a join, it produced the seam everyone recognises from old satnavs.

Parametric systems modelled the vocal tract and synthesised from parameters. Smooth, flexible, and unmistakably robotic.

Neural systems learned the mapping end to end. First with mel-spectrograms and a separate vocoder, now increasingly with a single model predicting codec tokens directly.

Streaming is the part that matters for conversation#

A non-streaming model processes the whole input and returns a file. A streaming model emits audio while it is still generating.

This is the difference between a one-second pause and a fifty-millisecond one, and for a voice agent it is the whole ballgame. It also changes the architecture: the model has to be able to start before it has seen the end of the sentence, which constrains what you can put in the decode loop. We wrote up what that constraint costs and buys separately.

Which metrics do vendors quote, and which lie?#

Five numbers show up on most model cards. Each is real, and each has a failure mode.

TTFA, time to first audio. How long between sending text and receiving the first audio. The number that decides whether a voice agent feels alive.

How it lies: it usually measures the first byte, not the first audible sample. A model whose stream opens with a moment of silence wins the metric while the caller hears nothing. Ask whether the figure is time to first byte or time to first sound, and note that the honest answer is often the less flattering one.

RTF, real-time factor. Synthesis time divided by audio duration. Below 1.0 is faster than real time.

How it lies: it rewards slow speech and padding. A model that speaks more slowly produces more audio for the same compute and posts a better RTF. Useful for capacity planning, close to useless for comparing vendors.

WER, word error rate. Transcribe the generated audio, compare against the input text.

How it lies: it is almost always measured on prose. Prose is redundant, so a listener recovers from one wrong word. Identifiers are not: in an account number every digit is load-bearing. A model with excellent prose WER can still be unusable for a product that reads back order numbers, and no published benchmark will warn you.

SIM, speaker similarity. How close a cloned voice is to its reference.

How it lies: it says nothing about whether the result is pleasant to listen to. High SIM with poor naturalness is a good impression of someone's voice on a bad phone line.

MOS and its predicted variants. Perceived quality, from human raters or a model trained to imitate them.

How it lies: the predicted variants disagree with each other, sometimes substantially, and none of them measure whether the words were right. UTMOS and NISQA can rank the same set of models differently. Treat a single MOS number as one opinion.

Which engine fits which job?#

The requirements genuinely differ, and a model that is wrong for one job can be the right answer for another.

Notice how little overlap there is. A vendor optimised for dubbing will look mediocre on a voice-agent benchmark and vice versa, and neither result means the model is bad.

Eight questions to ask before you commit#

Work through these against your own product. Most take minutes and each has burned somebody.

1. Is the latency figure time to first byte or first audible sound? They can differ by hundreds of milliseconds.
2. Does time to first audio change with input length? If it does, budget for your longest utterance on every turn.
3. How does interruption work? Barge-in is what separates an agent from an IVR. Ask whether there is a cancel message or whether you have to tear down the connection.
4. What is the word error rate on text that looks like yours? Especially digits, currency and dates. Test it yourself, because nobody publishes it.
5. Which languages, and how good is each one? Coverage lists tend to be generous. Listen to the third-priority language before you plan around it.
6. What licence covers the weights, not the repository? These are frequently different, and the difference decides whether you can self-host.
7. What does it cost at ten times your current volume? Subscription tiers change the effective rate substantially. We put ten vendors on one axis if you want a starting point.
8. Can you leave? If your prompts, voices and client code are portable, a bad choice costs a sprint. If they are not, it costs a quarter.

Do you actually need a custom voice?#

Usually not, and this is where the most money gets wasted.

Stock voice. Covers most products. Free, instant, and consistent.

Zero-shot cloning. A few seconds of reference audio, no training. On a system where the speaker is captured once up front, cloning adds nothing to the per-word cost. Reach for this when a specific person's voice is the actual requirement.

Fine-tuning. Hours to days of work and a curated dataset. Justified for a new language, a domain vocabulary the base model mangles, or a house style. Not justified for "we want our own voice", which cloning already solves. There is a longer guide on where the line falls.

The rule of thumb: try them in that order, and only move down when the option above genuinely fails on your own audio.