hanzo-koe

Koe is a 1.6B parameter text-to-speech model that directly generates realistic multi-speaker dialogue from text transcripts.

Hanzo Koe - Text-to-Dialogue Model

Category: Hanzo Ecosystem Related Skills: hanzo/hanzo-mugen.md, hanzo/hanzo-engine.md, hanzo/hanzo-ml.md

Overview

Koe is a 1.6B parameter text-to-speech model that directly generates realistic multi-speaker dialogue from text transcripts. It supports two-speaker conversations with speaker tags ([S1], [S2]), non-verbal sounds (laughter, coughs, sighs), and voice cloning from audio prompts.

The architecture uses an encoder-decoder transformer with RoPE positional embeddings. The encoder processes text tokens, and the decoder autoregressively generates 9-channel audio codes that are decoded by the Descript Audio Codec (DAC) into 44.1kHz waveforms.

Key Capabilities

Multi-speaker dialogue: Generate conversations between two speakers from tagged transcripts
Non-verbal generation: Produce laughter, coughs, throat clearing, sighs, gasps, etc.
Voice cloning: Condition output on reference audio (5-10 seconds) for speaker consistency
Real-time generation: Up to 2.2x realtime factor with torch.compile on RTX 4090

When to use

Generating dialogue audio from scripts with speaker annotations
Voice cloning and style transfer from reference audio
Adding non-verbal vocalizations to synthesized speech
Multi-speaker TTS without per-speaker fine-tuning

Hard requirements

Python 3.10+ with PyTorch 2.6
GPU: CUDA or MPS (Apple Silicon with torch.compile disabled)
VRAM: ~10GB (float16/bfloat16), ~13GB (float32)
HF_TOKEN: Required for downloading model config from HuggingFace Hub

Quick reference

| Item | Value | |------|-------| | Repo | github.com/hanzoai/koe | | Branch | main | | Language | Python | | Parameters | 1.6B | | Package | nari-tts | | Sample Rate | 44.1kHz | | Audio Channels | 9 (DAC codebooks) | | Install | pip install git+https://github.com/hanzoai/koe.git | | Run UI | uv run app.py | | License | Apache 2.0 |

Architecture

Text Input: "[S1] Hello. [S2] Hi there. (laughs)"
 |
 v
┌─────────────────────────────┐
│ Text Tokenizer │
│ src_vocab_size: 128 │
│ text_length: padded to 128 │
└──────────┬──────────────────┘
 |
 v
┌─────────────────────────────┐
│ Encoder (Transformer) │
│ n_layer layers │
│ n_head attention heads │
│ head_dim per head │
│ RoPE positional encoding │
│ RMSNorm normalization │
│ DenseGeneral (tensordot) │
└──────────┬──────────────────┘
 |
 v
┌─────────────────────────────┐
│ Decoder (Transformer) │
│ Cross-attention to encoder │
│ 9-channel audio code output │
│ tgt_vocab_size: 1028 │
│ Special tokens: │
│ EOS=1024, PAD=1025, │
│ BOS=1026 │
│ Delay pattern: [0,8..15] │
└──────────┬──────────────────┘
 |
 v
┌─────────────────────────────┐
│ Descript Audio Codec (DAC) │
│ Decode 9 codebook streams │
│ Output: 44.1kHz waveform │
└─────────────────────────────┘

Audio Delay Pattern

The decoder uses a delay pattern across 9 channels: [0, 8, 9, 10, 11, 12, 13, 14, 15]. This staggers codebook generation to improve coherence across the multi-stream output. Token positions are shifted via pre-computed delay/revert indices.

Inference Sampling

Token generation uses configurable sampling:

Temperature: Controls randomness (0.0 = greedy argmax)
Top-p: Nucleus sampling with cumulative probability threshold
Top-k: Limits candidate tokens to top-k logits
EOS suppression: EOS token masked unless it has the highest logit

One-file quickstart

from dia.model import Dia

model = Dia.from_pretrained("nari-labs/Dia-1.6B", compute_dtype="float16")

text = "[S1] Koe is a text to dialogue model. [S2] You get full control over scripts and voices. [S1] Wow. Amazing. (laughs) [S2] Try it now."

output = model.generate(text, use_torch_compile=True, verbose=True)
model.save_audio("dialogue.mp3", output)

Voice Cloning

from dia.model import Dia

model = Dia.from_pretrained("nari-labs/Dia-1.6B", compute_dtype="float16")

# Provide transcript of reference audio before generation text
text = "[S1] This is the reference speaker talking. [S2] And this is another voice. [S1] Now generate new content in my voice."

output = model.generate(
 text,
 audio_prompt="reference_audio.mp3", # 5-10 seconds
 use_torch_compile=True,
)
model.save_audio("cloned.mp3", output)

Apple Silicon (MPS)

from dia.model import Dia

model = Dia.from_pretrained("nari-labs/Dia-1.6B", compute_dtype="float16")
# torch.compile not supported on MPS
output = model.generate(text, use_torch_compile=False, verbose=True)

Non-Verbal Tags

Recognized tags (place in transcript where desired):

(laughs), (clears throat), (sighs), (gasps), (coughs), (singing), (sings), (mumbles), (beep), (groans), (sniffs), (claps), (screams), (inhales), (exhales), (applause), (burps), (humming), (sneezes), (chuckle), (whistles)

Performance

| Precision | Realtime Factor (compile) | Realtime Factor (no compile) | VRAM | |-----------|---------------------------|------------------------------|------| | bfloat16 | 2.1x | 1.5x | ~10GB | | float16 | 2.2x | 1.3x | ~10GB | | float32 | 1.0x | 0.9x | ~13GB |

Benchmarked on RTX 4090.

Dependencies

descript-audio-codec>=1.0.0
gradio>=5.25.2
huggingface-hub>=0.30.2
numpy>=2.2.4
pydantic>=2.11.3
safetensors>=0.5.3
soundfile>=0.13.1
torch==2.6.0
torchaudio==2.6.0
triton==3.2.0 (Linux)

Project Structure

koe/
├── dia/
│ ├── __init__.py
│ ├── audio.py # Delay pattern indices, audio processing
│ ├── config.py # DiaConfig (Pydantic): data, encoder, decoder, model
│ ├── layers.py # DiaModel: DenseGeneral, transformer blocks, attention
│ ├── model.py # Dia: main model class, generate(), save_audio()
│ ├── state.py # Inference state: KV cache, decoder/encoder state
│ └── static/images/ # Banner assets
├── docker/
│ ├── Dockerfile.cpu
│ └── Dockerfile.gpu
├── example/
│ ├── benchmark.py
│ ├── simple.py
│ ├── simple-mac.py
│ ├── simple_batch.py
│ ├── voice_clone.py
│ └── voice_clone_batch.py
├── app.py # Gradio web UI
├── cli.py # CLI interface
├── pyproject.toml
└── uv.lock

Generation Guidelines

Input text should correspond to 5-20 seconds of audio for best quality
Always begin with [S1] and alternate between [S1] and [S2]
Use non-verbal tags sparingly; overuse causes artifacts
For voice cloning: provide 5-10 second reference audio with matching transcript
End text with the second-to-last speaker tag to improve trailing audio quality
1 second of audio is approximately 86 tokens

Related Skills

hanzo/hanzo-mugen.md - Audio generation (music, sound effects)
hanzo/hanzo-engine.md - Model serving infrastructure
hanzo/hanzo-ml.md - Rust ML framework

Last Updated: 2026-03-13 Category: Hanzo Ecosystem Related: tts, dialogue, speech-synthesis, voice-cloning, audio Prerequisites: Python, PyTorch, audio processing concepts