How to fine-tune Gepard (and when not to)

Most teams that ask how to fine-tune a speech model do not need to fine-tune a speech model. They need a voice that sounds like a particular person, and there is a much cheaper way to get one.

So this guide starts with when not to, then covers how to do it properly when the cheaper path genuinely fails.

Three ways to get a custom voice#

Try them in this order. Only move down when the option above fails on your own audio.

Zero-shot cloning. Five to fifteen seconds of reference audio, no training at all. In Gepard the speaker is captured once, up front, as a Q-Former prefix computed before generation starts, which is why cloning adds nothing to the per-token cost. Minutes of work, no GPU, no dataset.

LoRA fine-tune. About an hour of clean transcribed audio, a single GPU, a few hours end to end. The backbone stays frozen and you train adapters.

Full fine-tune. A curated dataset in the tens of hours, multiple GPUs, days. Reserved for a new language or a genuine domain shift.

The honest rule: if the requirement is "make it sound like this person", cloning solves it. If the requirement is "make it handle Kyrgyz" or "stop mangling our product names", cloning will not, and you are in fine-tuning territory.

When is fine-tuning actually justified?#

Four cases, in rough order of how often they come up.

A domain vocabulary the base model mangles. Drug names, part numbers, street names in one city, an industry's acronyms. The base model has never seen them and pronounces them by analogy, badly.

A language the model does not cover. Gepard ships English, Spanish (Mexico), Portuguese (Brazil) and Dutch. Anything else is a fine-tune, and a substantial one.

A register the base model handles poorly. The clearest example in our own work is the short register: one- and two-word utterances behave differently from ordinary speech in autoregressive decoders. Our technical report treats this as a fundamental issue in the model class rather than a bug, and the shipped training pipeline includes a short-phrase adaptation stage specifically for it. If your agent says "Yes." and "Got it." a hundred times an hour, this matters more than MOS.

A house style. Consistent pacing, a particular energy level, a way of handling questions. Cloning copies a voice; it does not reliably copy a delivery style across arbitrary text.

LoRA or a full fine-tune?#

LoRA, in almost every case, and the evidence has converged on this.

Published comparisons find LoRA reaching near-parity with full-parameter fine-tuning on voice similarity and naturalness, at a fraction of the memory and training time. One line of work reports that tuning around 15% of backbone parameters gets you comparable results to full fine-tuning, where earlier attempts at 0.3% to 5% left quality on the table. The LoRP-TTS work on low-rank personalised speech covers the speaker-adaptation case in more detail, and Unsloth's TTS fine-tuning guide is a practical starting point for the general workflow. Full-parameter fine-tuning on speech models is also frequently unstable, because dataset quality is uneven and the model has more room to forget.

That last point is the one that decides it. A full fine-tune on an hour of one speaker will make the model excellent at that speaker and measurably worse at everything else. LoRA leaves the base intact.

How much data, and what kind?#

About one hour of clean single-speaker audio is the working figure for a LoRA voice adaptation. But duration is the least important of the three variables.

Transcript accuracy beats quantity. Thirty minutes with exact transcripts beats three hours with approximate ones. Errors in the text teach the model the wrong mapping, and it learns them faithfully.

Recording consistency beats variety. One microphone, one room, one session. Mixed sources teach the model to reproduce the mix, including the room.

Coverage of what you will actually synthesise. If the product reads out order numbers, the dataset needs order numbers. A dataset of read prose produces a model good at read prose.

Running the training#

The pipeline is public in gepard-train, with a Hydra config tree under conf/. Four stages:

1. Dataset build. Encodes raw audio into NanoCodec tokens. The model trains on discrete audio codes, not waveforms, so this is a required preprocessing step rather than an optimisation.

make dataset

The reference corpus on Hugging Face, nineninesix/emolia_filtered_nano_codec_21_dataset, is already tokenised. Inspect it before preparing your own so you know the shape you are aiming at.

2. Pretrain. Base model training on the full corpus, four GPUs with FSDP. You almost certainly do not need this stage.

make train

3. LoRA fine-tune. Short-phrase adaptation with a frozen backbone and trained adapters, single GPU. This is the stage most people want.

make finetune
make merge CHECKPOINT=checkpoints/checkpoint-N

4. DPO. Offline classifier-free-guidance distillation. Self-generated preference pairs bake the quality of two-pass CFG into single-pass weights, which is how the shipped model gets two-pass quality at one-pass cost.

make dpo-sample-sharded
make dpo-score
make dpo-pairs
make dpo-train

Publishing the result:

make upload REPO=you/your-model CHECKPOINT=<merged-dir>

Because the weights are Apache 2.0, a model you fine-tune is yours to ship, sell or keep private, with no revenue ceiling and no obligation to publish it.

How do you know it worked?#

Not by listening to three clips and feeling good about them.

Hold out real text. Sentences from your actual product that were never in training. Read-prose evaluation on a voice agent model tells you almost nothing.

Measure word error rate, on your text. Transcribe the output with a separate ASR model and compare. Pay particular attention to digits, which break in ways prose does not and which no public benchmark measures.

Check what you broke. Run the base evaluation set through the fine-tuned model. LoRA makes forgetting unlikely, not impossible, and a model that got better at your voice and worse at numbers is a bad trade you want to catch before shipping.

Listen at the boundaries. Very short utterances, very long ones, unusual punctuation, all-caps, numbers next to words. Failures cluster at the edges.

Compare against cloning. Run the same text through zero-shot cloning with a reference clip. If the fine-tune is not clearly better, you spent a week for nothing, and knowing that is worth the ten minutes.

Serving what you trained#

A merged checkpoint runs on the same vLLM path as the base model, so the throughput numbers carry over and the serving setup does not change.

Whether it is worth running your own GPU to serve it is a separate arithmetic problem, and the answer depends more on utilisation than on volume.