Why Gepard runs on vLLM, and what it buys

Most text-to-speech models are designed for quality first and served however they can be afterwards. Gepard was designed the other way round.

The rule, set before the architecture, was this: the model must run on a standard vLLM engine without modifying the source of its compute kernels. Everything else in the design follows from that one sentence, including the things we gave up.

Why does the serving engine get to constrain the model?#

Because for a voice agent, the serving engine is where the cost and most of the latency live.

An interactive voice product imposes two limits at once: time to first audio has to be low enough that a human does not notice the gap, and cost per concurrent conversation has to be low enough that the product has margin. Both of those are properties of the serving path, not of the model file.

vLLM already solved the hard parts. Continuous batching keeps the GPU fed when requests arrive and finish at different times. PagedAttention stops KV-cache fragmentation from capping how many sequences fit in memory. These took the community years to get right, and a custom TTS server has to either reimplement them or accept a much worse cost curve under concurrency.

So the choice was: build a model that vLLM can serve, or build a better model and then rebuild vLLM around it. We took the first one.

What does that rule actually forbid?#

Anything custom inside the autoregressive decode loop.

The decode loop is a standard full-attention transformer and nothing else: 14 blocks, hidden dimension 1024, 8 attention heads, about 500M parameters of backbone. There is no text generation head at all. It decodes audio tokens and predicts the end of speech, and that is the whole job.

The things a modern speech decoder would normally reach for are exactly the things that break this:

• Layer-wise depth transformers over codec codes. A second, smaller transformer that walks the codebooks inside each frame. Common, effective, and a custom operation in the middle of the loop.
• Cross-attention to an audio interface at intermediate layers. Same problem.
• Two-pass classifier-free guidance. Doubles the compute per frame and changes the shape of the loop.

Each of those would have forced a forked engine. So each of them was moved.

Voice cloning became a Q-Former prefix computed once, up front, from the reference clip. The speaker is captured before generation starts, so cloning adds nothing to the per-token cost. That is a genuine architectural win rather than a workaround: on most systems cloning is a per-request tax, and here it is not.

Classifier-free guidance was distilled into the weights. The two-pass version is a known quality gain and a known 2x cost. Using DPO on self-generated preference pairs, the behaviour of two-pass CFG was baked into single-pass weights, so the model produces the better output in one pass. Two-pass remains available as a quality dial, but production runs single-pass.

Text augmentation moved into preprocessing, which is where it always belonged.

The audio frame is sampled in one clean pass. All 32 orthogonal FSQ channels come out in a single step, with no depth transformer walking them sequentially.

What does it buy in throughput?#

Measured end-to-end over the SSE streaming protocol on a server-class GPU:

The number that matters is the third one. A single stream at 15x real time is a nice model. 204x aggregate on one GPU is a business model. It is the difference between serving a demo and serving a contact centre, and it comes almost entirely from continuous batching doing its job because nothing in the loop got in its way.

The recommended range is 64 to 128 rather than 256 because throughput is not the only axis. Pushing to 256 maximises aggregate speedup and lengthens the queue each individual request waits in. If you are running batch narration, use 256. If you are running live calls, stay in the 64 to 128 band and keep time to first audio at 50 ms.

Tuning the streaming configuration was worth roughly a twofold reduction in single-stream time to first frame against baseline settings, with peak aggregate speedup landing at 203.9x. Four configurations were swept: default, low-latency, throughput-oriented and balanced.

A word on real-time factor#

Our RTF numbers are good and we do not lean on them.

RTF is synthesis time divided by audio duration. It rewards a model that produces more audio for the same compute, which means it rewards speaking slowly and padding. A vendor can improve their RTF by making the voice more leisurely, and the metric will applaud.

We report it because the report would be incomplete without it, and we treat time to first audio and aggregate throughput under concurrency as the numbers that actually decide anything. The full latency benchmark makes the same point with four vendors' data.

What did the constraint cost us?#

Speaker similarity, mostly.

On Seed-TTS-eval, Gepard scores 0.585 on SIM against 0.867 for the best model in the comparison, and 0.036 on WER against 0.015. Some of that gap is the streaming-first design and some of it is the decode-loop rule: the techniques that would close it are the ones that need custom operations per frame.

In exchange the same evaluation puts Gepard first on every perceived-quality axis, and the serving numbers above are not achievable any other way. Whether that is a good trade depends entirely on what you are building, and the model guide lays out both columns.

There is also a genuinely interesting failure mode the constraint surfaced: the short register. Utterances of one or two words behave differently from ordinary speech in autoregressive decoders, badly enough that the technical report treats it as a fundamental issue in the class rather than a bug in one model, along with a way to measure it and a way to eliminate it. Anyone building a voice agent that says "Yes." and "Got it." a hundred times an hour should care about this more than about MOS.

Running it yourself#

The weights are Apache 2.0 and the serving path is public.

# vLLM-native serving
git clone https://github.com/nineninesix-ai/gepard-vllm

If you would rather not run a GPU, the hosted API serves this exact model at api.nineninesix.ai, speaks the Cartesia wire protocol so existing client code works with a base-URL change, and costs $5 per million characters. The self-hosted versus API arithmetic tells you which one is cheaper at your volume, and the crossover is higher than most people expect.

The technical report is Gepard: A GEnerative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue, by Denis Pavlov, Ulanbek Abdurazakov and Nursultan Bakashov, published alongside the weights on Hugging Face.