KaniTTS v0.3

Overview

KaniTTS pretrain 0.3 uses a two-stage pipeline combining a large language model with an efficient audio codec for exceptional speed and audio quality. The architecture generates compressed token representations through a backbone LLM, then rapidly synthesizes waveforms via neural audio codec, achieving extremely low latency.

Key Specifications:

Model Size: 400M parameters
Sample Rate: 22kHz
Language: English, Chinese, Korean, Spanish, German, Japanese, German
License: Apache 2.0

Performance

On NovitaAI RTX 5090 using vLLM:

RTF: ~0.2 (5 times faster than realtime)
Memory: 16GB GPU VRAM used
Source Code: https://github.com/nineninesix-ai/kanitts-vllm

Datasets

Use Cases

Conversational AI: Real-time speech for chatbots and virtual assistants
Edge/Server Deployment: Resource-efficient inference on affordable hardware
Accessibility: Screen readers and language learning applications
Research: Fine-tuning for specific voices, accents, or emotions

Limitations

Performance degrades with inputs exceeding 15 seconds (need to use sliding window chunking)
Limited expressivity without fine-tuning for specific emotions
May inherit biases from training data in prosody or pronunciation
Optimized primarily for English; other languages may require additional training

Optimization Tips

Multilingual Performance: Continually pretrain on target language datasets and fine-tune NanoCodec
Batch Processing: Use batches of 8-16 for high-throughput scenarios
Hardware: Optimized for NVIDIA Blackwell architecture GPUs

Resources

Models:

Pretrained Model: https://huggingface.co/nineninesix/kani-tts-500m-0.3-pt
English: https://huggingface.co/nineninesix/kani-tts-400m-en
Chinese: https://huggingface.co/nineninesix/kani-tts-400m-zh
Korean: https://huggingface.co/nineninesix/kani-tts-400m-ko
German: https://huggingface.co/nineninesix/kani-tts-400m-de
Spanish: https://huggingface.co/nineninesix/kani-tts-400m-es
Arabic: https://huggingface.co/nineninesix/kani-tts-400m-ar
Japanese: https://huggingface.co/nineninesix/kani-tts-370m-expo2025-osaka-ja

Examples:

Space: https://huggingface.co/spaces/nineninesix/KaniTTS
OpenAI compatible API: https://github.com/nineninesix-ai/kanitts-vllm
Finetuning code pipeline: https://github.com/nineninesix-ai/KaniTTS-Finetune-pipeline
Dataset preparation pipeline: https://github.com/nineninesix-ai/nano-codec-dataset-pipeline
Example Dataset: https://huggingface.co/datasets/nineninesix/expresso-conversational-en-nano-codec-dataset
ComfyUI node: https://github.com/wildminder/ComfyUI-KaniTTS by WildAi
NextJS basic app: https://github.com/nineninesix-ai/open-audio. It uses the OpenAI npm package to connect to the OpenAI-compatible server API provided by kanitts-vllm.
GitHub Repository: https://github.com/nineninesix-ai/kani-tts

Acknowledgments

Built on top of LiquidAI LFM2 350M as the backbone and Nvidia NanoCodec for audio processing.

Citation

@inproceedings{emilialarge,
  author={He, Haorui and Shang, Zengqiang and Wang, Chaoren and Li, Xuyuan and Gu, Yicheng and Hua, Hua and Liu, Liwei and Yang, Chen and Li, Jiaqi and Shi, Peiyang and Wang, Yuancheng and Chen, Kai and Zhang, Pengyuan and Wu, Zhizheng},
  title={Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation},
  booktitle={arXiv:2501.15907},
  year={2025}
}

@article{emonet_voice_2025,
  author={Schuhmann, Christoph and Kaczmarczyk, Robert and Rabby, Gollam and Friedrich, Felix and Kraus, Maurice and Nadi, Kourosh and Nguyen, Huu and Kersting, Kristian and Auer, Sören},
  title={EmoNet-Voice: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detection},
  journal={arXiv preprint arXiv:2506.09827},
  year={2025}
}

@dataset{masrispeech_full,
  author       = {Yahya Muhammad Alnwsany},
  title        = {MasriSpeech-Full: Large-Scale Egyptian Arabic Speech Corpus},
  year         = {2025},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/collections/NightPrince/masrispeech-dataset-68594e59e46fd12c723f1544}
}