Coval TTS benchmark: Gepard vs 25 APIs

Coval runs a public text-to-speech leaderboard, the harness behind it is open source, and on 25 August 2026 we pointed it at our own production API to see what Gepard 1.0 actually measures next to the 25 APIs on its public board.

Here is what came out, over 30 samples with zero errors: 68.7 ms to first audio at p50, 85.8 ms at p90, 104.8 ms worst observed, 20.5 ms of silence at the head of the audio, 5.23% word error rate.

Those numbers were measured by us, from us-west-1, against an endpoint in California. The published board is measured by Coval, from us-east1, against everyone else's endpoints. One asymmetry to hold on to while you read: our figures have the network round trip normalised out, 2.8 ms on our path, and the published rows do not, so each of them still carries whatever path Coval's runner had. That flatters us at both ends. It is a small effect next to the gaps in the table, but you should know it is there.

One thing belongs up front rather than buried in a methodology note: production serves all of this from a single RTX 4090. Not a rack, not a datacentre accelerator, a card you can buy retail for a couple of thousand dollars. That is the part of the result you can actually copy, because the weights are Apache 2.0 and the hardware is on a shelf.

What does Gepard measure against the rest of the board?#

Lower than everything published, on a shorter network path. Both halves of that sentence matter, so here are both of them in one table rather than a ranking. Fourteen of the twenty-six rows are shown, the fast end of the board plus the names most people are actually choosing between. The board figures were read from Coval's public endpoint on 25 August 2026 and the window is a rolling seven days, so they drift by a millisecond or two from one day to the next.

The honest reading of the latency columns: our row has its network hop normalised out, 2.8 ms on our path, and theirs still carry whatever the path from us-east1 to each vendor costs. Take the difference between our 68.7 and Cartesia's 272 as real but not as a clean 4x. Take the two right-hand columns as directly comparable, because neither of them can be moved by where you sit.

Worth saying that the unknown runs in both directions. Not one vendor on that board publishes which region served each request, and several of them are California companies, so some of those grey rows are being measured over a short hop and some over a long one. Nobody can tell you which from the published data, us included.

The p90 column is the one worth staring at. Deepgram goes from 288 to 503 between p50 and p90. OpenAI goes from 778 to 2854. Gepard goes from 68.7 to 85.8, and the worst single sample out of 30 was 104.8 ms. A tail that stays inside about 105 ms is the difference between a voice agent that always feels immediate and one that feels immediate most of the time.

Which numbers here are location-proof?#

Leading silence and word error rate. Both are read out of the audio file after the fact, so they are the same wherever you run the harness from.

That is not a claim, it is something we checked. Running the harness from a third location against Cartesia and Gradium reproduced their published silence figures within 5 ms: Cartesia came back at 127 against the 131 on the board, Gradium at 164 against 167. Same instrument, same answer, from somewhere else entirely.

So these two ranks are ours to claim outright:

• Leading silence 20.5 ms, fifth of 26. Ahead of Cartesia at 131, Deepgram at 148, Google at 344 and Murf at 458. Behind Palabra at 1 ms, two of the three Fish Audio rows at 7 ms and Speechify Simba 3.2 at 20 ms. Murf spends close to half a second of dead air before it says a word.
• Word error rate 5.23%, eighteenth of 26. Mid-table. Better than Cartesia at 5.89% and Deepgram at 5.48%, behind Soniox v1 at 3.72% and Gradium at 4.57%. More on what that number actually contains further down.

What does Coval count as time to first audio?#

The arrival of the first audio plus any silence at the head of it. Almost every vendor latency page counts the first byte and stops. The difference is not academic, and it caught us out.

An intermediate build of ours looked like a triumph by every transport measure. Bytes arrived faster than they ever had, 52.9 ms to the first chunk. But the first 289 ms of those bytes were silence, so the perceived time to first audio rose to 382 ms and all 30 samples came in worse than the build before it. Word error rate was unchanged at 5.29%, which confirmed the speech itself was intact and the audio was simply padded.

That build would have improved the headline number on our own latency benchmark while making the product 5.6x worse for a caller. Only a metric that opens the audio catches it. We verified the finding with a separate RMS scan rather than trusting the harness's own silence detector, and the two agreed within 2 ms.

If you are evaluating vendors, this is the single most useful thing to copy: measure to first audible energy, not to first byte.

What happens if you measure from a different region?#

We move the box. That is the whole answer, and it is the one thing on this page a vendor on the grey rows cannot say back to you.

Start with what the 68.7 ms is made of, because there is less in it than you would think. 48.6 ms is server-side cost, from request sent to first byte back, and 20.5 ms is silence at the head of the audio. Those two add up to the whole number. The network is not a third slice, it is already normalised out, and on our path it was only 2.8 ms because the harness host and production sat in the same region.

So the California endpoint is not what makes the number small. Run the same harness from the other coast against that same box and you add a real cross-country hop, on the order of the 49 ms round trip we measured on that path, but you are then timing the distance between two computers rather than the model. The published rows have exactly that problem baked in and unlabelled: Coval records a single coarse region per model, us for every row in this table, and nothing finer.

So here is the offer instead of the argument. Name the region. We will stand a box up in it and you run Coval's harness yourself, from wherever you like, against an endpoint that is next to you rather than next to us. The compute and the silence come along unchanged, because they are the model. For us that is a deployment, not a roadmap item, and if you would rather not wait on us at all, the weights are Apache 2.0 and the box can be yours.

What would a bigger GPU do to this?#

It would take a bite out of exactly half the number, and the more interesting half would not move. Splitting 68.7 into 48.6 of compute and 20.5 of silence is not just bookkeeping, it tells you what money can buy.

Compute is what hardware buys. We have run the same build on a datacentre-class Blackwell part and the compute segment drops by roughly 18 ms, which would put the median near 53 ms. Leading silence does not move at all, because that is the model deciding when to start talking, and no amount of GPU fixes a model that clears its throat first.

Production stays on the 4090 on purpose. The point of an open-weights model is not that we can afford good hardware, it is that you do not need any. A card that costs a couple of thousand dollars at retail is already at the top of this table, and the ceiling above it is real but it is not where the argument is.

How many models on the leaderboard ship their weights?#

One, and it is the one at the top of the board. The fastest published row on the public TTS leaderboard is fluxions/vui at 101.7 ms, and Vui is a 300M model whose code and checkpoint are both on Hugging Face under Apache 2.0. Not a funded API vendor with a private checkpoint. A small open model, beating every commercial row on the board it is measured against.

Coval's own registry will tell you otherwise, and it is worth knowing why before you cite it. The registry has a licensing field that defaults to proprietary, and only one TTS entry sets it explicitly: Alibaba's Qwen3-TTS served through Baseten, which sits at early access and is hidden from the public site. Vui simply has no line, so it inherits the default and shows up as proprietary. The field is a label someone has to remember to set, not a fact about the model.

Fish Audio is the other case that deserves care. They do publish weights: s1-mini under CC BY-NC-SA, s2-pro under the Fish Audio Research License. Both bar commercial use without a separate written agreement, and neither is what is on the board, which is the hosted s1 and s2.1-pro. Alibaba is the same shape, an open Qwen3-TTS checkpoint on one side and a closed hosted row on the other. Published weights and usable weights are not the same claim, which is a distinction we have had to make about our own older models too.

So the honest count is two Apache 2.0 rows out of 26, Vui and Gepard, and 24 closed APIs. That is the part of this exercise that survives every argument about network paths. You can dispute a latency figure measured from a different region. You cannot dispute a licence. The model you benchmark through our API is the same model you can run on your own hardware. Most of the rest of that table is a rental: if the vendor changes the voice, raises the price or sunsets the model, you migrate.

On the two metrics a measurement location cannot move, the two open rows land close. Leading silence is 20.5 ms for us and 21 ms for Vui, which is a tie. Word error rate is 5.23% against 6.20%, which is a real gap but not a rout. We would rather be one of two than the only one: a leaderboard where the open entries are competitive is a better argument for open weights than any single result of ours.

The hardware is not a moat either. Everything at the top of this article came off one consumer GPU, so "run it yourself" is a Tuesday afternoon rather than a procurement cycle.

We have written separately about what open weights actually buy you and about why Gepard runs on vLLM, which is what makes self-hosting a normal deployment rather than a research project.

What is really inside a TTS word error rate?#

Ours came in at 5.23%, with 18 of 30 prompts transcribing perfectly. Before reading much into that, or into anyone else's, look at what the scorer does with a clock time.

source     ...yesterday around 11:30 PM.
whisper-1  ...yesterday around 11.30 p.m.     -> 35.7% WER
gpt-4o     ...yesterday around 11:30 PM.      ->  0.0% WER

Identical audio, scored twice, and one of those numbers is a spelling convention. The benchmark compares a Whisper transcript against the source text after normalisation, and the normaliser does not reconcile clock-time formats. Re-scoring all 30 prompts through both transcribers put whisper-1 at 4.57% and gpt-4o-transcribe at 2.41%. Roughly half the measured error is the instrument.

We still think whisper-1 is the right scorer to publish, because every number on the leaderboard uses it and switching would make ours incomparable with the field. A two-point improvement that is purely orthographic is not a real win. The more useful role for a second transcriber is as a disagreement detector: where two independent recognisers agree a word is wrong, go listen to it.

How do you reproduce this?#

The harness is at github.com/coval-ai/benchmarks and the dataset ships inside the wheel, so no cloud storage or database is needed if you bypass the orchestrator.

git clone https://github.com/coval-ai/benchmarks
# add runner/src/coval_bench/providers/tts/nineninesix.py
export NINENINESIX_API_KEY=...
uv run python -m coval_bench run --benchmark TTS --dataset tts-v1 \
  --only nineninesix/gepard-1.0

One thing worth knowing before you read your own numbers: our API gateway adds 0.0 ms to the fast path, measured p50 against a direct connection to the model server, so what the harness times is the model rather than our plumbing.

And one thing the harness does not count, for us or for anyone else on the board: the clock starts once the WebSocket is open. Every streaming provider is measured that way, so the comparison between rows is fair, but a caller who opens a fresh connection for every turn pays roughly 150 ms more than any number on this page. Hold the socket open. That is true of every vendor here, and it will cost you more latency than the choice between the top few rows.

The provider is the official cartesia SDK pointed at a different host, because our API speaks the Cartesia wire protocol. Two things differ. Request audio at Gepard's native 22050 Hz, since asking for 8000 or 16000 makes the server resample. And send no language field, because language here is a property of the voice rather than of the request.

The public aggregates endpoint is worth knowing about on its own:

GET https://benchmarks-api-6wxgp27p2a-ue.a.run.app/v1/results/aggregates
    ?benchmark=TTS&window=7d&dataset=tts-v1

benchmark has to be uppercase. That single call returns every competitor's p50, p90, p95, leading silence and word error rate, which is a better starting point for a vendor shortlist than any vendor's own page, ours included.

What we would not claim from this run#

Our latency numbers are ours, measured from our region. They are the right basis for tracking our own drift over time and for comparing our builds against each other, which is exactly what caught the padded-silence build above. They are not a position on Coval's board, and we are not going to dress them up as one. The way to settle that is for Coval to run it, which is the next thing on our list.

Our row is n=30. The published rows pool up to about 3,350 samples each, and the newest few far less than that. Thirty samples resolve a large effect cleanly, zero slow requests where four were expected, but they only bound a rare failure rate below roughly eleven percent. Rare is not the same as fixed, and we are not going to pretend thirty samples settle a tail. More samples is the other reason we want Coval to run this rather than us.

And the two right-hand columns of that table are the ones to argue with. Leading silence and word error rate are read out of the audio file, so they are the same numbers no matter who runs the harness or from where. If you want to check one claim on this page, check one of those.

If you want to try the API the run was measured against, it is at nineninesix.ai and the weights are on Hugging Face.