11. Neural audio codecs and speech tokens

What you will learn

• What a neural audio codec is: an encoder, a quantizer and a decoder, trained together.
• How to read a codec's frame rate and codes per frame, and turn them into bits per second, model steps per second and the smallest piece of audio a TTS model can emit.
• The difference between acoustic and semantic tokens, and which codecs can encode and decode in a stream.
• What I measured for Mimi, EnCodec and DAC on a CPU: bitrate, reconstruction distance and speed.

Before you start

• Chapters: 1 (bitrates of raw audio), 2 (mel spectrograms), 10 (the TTS pipeline).
• Tools: the ssh-tts-cpu environment from code/tts/README.md.

Encoder, quantizer, decoder

Text language models predict tokens from a vocabulary. To let a language model write audio the same way, audio must first become a sequence of tokens, and tokens must turn back into audio. A neural audio codec does both.

1. The encoder is a stack of strided convolutions that shortens the waveform into a sequence of vectors, one per codec frame. SoundStream's encoder at 24 kHz uses strides (2, 4, 5, 8), so "one embedding is computed every M = 2 · 4 · 5 · 8 = 320 input samples", 75 frames per second (Zeghidour et al., arXiv:2107.03312, Sec. III).
2. The quantizer replaces each vector by the index of the nearest entry in a learned codebook: now each frame is a number, a token.
3. The decoder mirrors the encoder and turns the sequence of codebook vectors back into samples.

One codebook of 1,024 entries carries 10 bits per frame, not enough for good audio. Residual vector quantization (RVQ) stacks quantizers: the second codebook encodes what the first got wrong, the third what the first two got wrong, and so on. SoundStream at 6 kbit/s spends "r = 6000/75 = 80 bits allocated to each frame" as eight codebooks of 1,024 entries (Sec. III). Each frame is then a column of 8 numbers, and dropping the last codebooks lowers the bitrate and the quality without retraining.

The other common design keeps one codebook and makes it large. Finite scalar quantization (FSQ) rounds each of a few dimensions to a small set of levels; CosyVoice 2's tokenizer has 6,561 entries, all of them used (arXiv:2412.10117, Table 4). X-codec2, used by Llasa, has one codebook of 65,536 entries at 50 Hz (arXiv:2502.04128, p. 5).

Reading a codec: three numbers

Three numbers describe a codec for streaming purposes: frames per second, codes per frame, and bits per code (log₂ of the codebook size). From them follow the bitrate and how many steps a language model must take per second of audio.

Codec	Frame rate	Codes per frame	Bits per code	Bitrate	Tokens per second, flattened	Causal
EnCodec 24 kHz at 6 kbit/s	75 Hz	8	10	6,000 bit/s	600	yes
EnCodec 24 kHz at 24 kbit/s	75 Hz	32	10	24,000 bit/s	2,400	yes
DAC (Descript Audio Codec) 44.1 kHz	86.13 Hz	9	10	7,752 bit/s	775	no
DAC 24 kHz (the one measured below)	75 Hz	32	10	24,000 bit/s	2,400	no
SNAC 24 kHz (used by Orpheus, a TTS model that writes SNAC codes)	11.72 + 23.44 + 46.88 Hz	1 + 2 + 4 per 85.3 ms	12	984 bit/s	82	no
Mimi, 8 codebooks (Moshi)	12.5 Hz	8	11	1,100 bit/s	100	yes
Mimi, 32 codebooks (Kyutai TTS)	12.5 Hz	32	11	4,400 bit/s	400	yes
Qwen3-TTS 12 Hz tokenizer	12.5 Hz	16	12 (first), 11 (others)	2,212.5 bit/s	200	yes
CosyVoice 2 and 3 tokens	25 Hz	1	12.68	317 bit/s	25	— (decoded by flow matching)
X-codec2 (Llasa)	50 Hz	1	16	800 bit/s	50	no
XTTS VQ-VAE	21.53 Hz	1	10	215 bit/s	21.5	— (decoded by HiFi-GAN)
raw 24 kHz 16-bit PCM, for reference	—	—	—	384,000 bit/s	—	—

The rows come from each codec's code or paper: EnCodec model.py (frame_rate, bins=1024, 32 codebooks at 24 kbit/s; arXiv:2210.13438); DAC dac.py (encoder_rates=[2, 4, 8, 8], n_codebooks=9; arXiv:2306.06546); SNAC's snac_24khz configuration (vq_strides [4,2,1], codebook_size 4096); Mimi from the Moshi paper ("Q = 8 quantizers, each with a codebook size of NA = 2048. At 12.5Hz, this represents a bitrate of 1.1kbps", arXiv:2410.00037, p. 11) and moshi/models/loaders.py; the Qwen3-TTS tokenizer's config.json (encode_downsample_rate 1920, 16 quantizers, arXiv:2601.15621, Sec. 2.2); CosyVoice 2's cosyvoice2.yaml (speech_token_size: 6561, token_frame_rate: 25); XTTS ("8192 codes at a 21.53 Hz frame rate ... keeping only the first 1024 most frequent codes", arXiv:2406.04904, p. 2). Bitrate is frames per second times codes per frame times bits per code.

Steps per second and the real-time budget

A language model that writes every code as its own token ("flattened") must take, for Mimi with 32 codebooks, 400 steps per second of audio: 2.5 ms per step to keep up with real time. That is too fast for a large model. So the designs of chapter 12 split the work: a large model takes one step per frame, and a small "depth" model writes the codes inside each frame. Then the large model's budget is the frame duration: 13.3 ms at 75 Hz, 40 ms at 25 Hz, 80 ms at 12.5 Hz. This is why recent speech language models moved to low frame rates. XTTS's authors credit their speed over VALL-E to exactly this, "tokens at a 21.53 Hz frame rate as compared with 75Hz" (arXiv:2406.04904, p. 4).

The smallest emittable unit

The other side of a low frame rate is a large smallest unit. A model cannot emit less than one frame of audio. For EnCodec at 24 kHz that is 13 ms: "the model can output 320 samples (13 ms) as soon as the first 320 samples (13 ms) are received" (arXiv:2210.13438, p. 4). For Mimi it is 80 ms: "given a first audio frame of 80ms, Mimi outputs a first latent timestep, which can be decoded to 80ms of output audio" (arXiv:2410.00037, p. 11). The Moshi paper notes that one extra frame of delay between codebooks "significantly improves the quality", at 80 ms more acoustic delay (p. 26).

Causal and non-causal codecs

A codec streams only if its encoder and decoder are causal: each output depends only on the present and the past. SoundStream says "all convolutions are causal" (Sec. III). EnCodec's 24 kHz model is causal; its 48 kHz model is not and processes audio in 1 s windows (causal=False, segment=1. in model.py). Mimi's convolutions and Transformers are all causal, so both its encoder and its decoder stream, with a Transformer context of 10 s ("context": 250 at the encoder's 25 Hz internal rate; the paper says both 10 s and 20 s, and the code settles it).

DAC and SNAC pad their convolutions symmetrically: every output sample depends on audio after it. They can still be used in a stream, but only by decoding overlapping windows and waiting for future frames. Orpheus, which writes SNAC codes, shows the cost in its code: it decodes a sliding window of 28 tokens (4 frames) and keeps only samples 2,048 to 4,096, the second frame (orpheus_tts/decoder.py, lines 62 and 106-108). Each emitted frame therefore waits for two more frames, 170.7 ms of right context, and the very first frame's audio is never emitted at all. Chapter 13 measures what happens at chunk boundaries when a non-causal decoder is cut.

Acoustic and semantic tokens

A codec trained only to reconstruct audio learns acoustic tokens: they describe the sound well, including the speaker, the room and the noise, but not necessarily what was said. Language models trained on such tokens tended to skip words and get them wrong; X-Codec's authors injected semantic features before the RVQ to fix this (arXiv:2408.17175).

Semantic tokens describe the content. CosyVoice 2's tokens are "supervised semantic tokens": an FSQ layer inserted into the encoder of a speech recognizer, SenseVoice-Large, so that the tokens must keep what the recognizer needs (arXiv:2412.10117, pp. 3-4). Its tokenizer reads a 128-bin Whisper log-mel of 16 kHz audio (cosyvoice/cli/frontend.py, lines 96-98, whisper.log_mel_spectrogram(speech, n_mels=128)). CosyVoice 3 keeps 25 Hz and moves the FSQ layer into a larger, multi-task encoder (arXiv:2505.17589, Sec. 2.1). Semantic tokens cannot be decoded to audio by themselves; CosyVoice turns them into mel frames with flow matching, at 2 mel frames per token (50 Hz mel, 25 Hz tokens), and a vocoder does the rest.

Mimi does both in one codec. Its first codebook is trained to match features of WavLM, a self-supervised speech model, and the other seven are an ordinary RVQ: "we distill semantic information into a plain VQ and apply an RVQ with 7 levels" (arXiv:2410.00037, p. 12). WavLM itself is not causal; it is used only as a training target, so Mimi stays causal. The Qwen3-TTS tokenizer builds on the same idea, with 16 codebooks of 2,048 entries each (one semantic, 15 acoustic; the speech_tokenizer/config.json of the model repository), and an encoder that "emits semantic and acoustic tokens at 12.5 Hz without look-ahead" (arXiv:2601.15621, Sec. 2.2). Its encoder class subclasses Hugging Face's Mimi implementation.

Measured: three codecs on a CPU

I encoded and decoded two English (LibriTTS-R) and two Mandarin (AISHELL-1, resampled from 16 to 24 kHz) clips with Mimi, EnCodec 24 kHz and DAC 24 kHz at three settings each, on four threads of an Intel i5-12600K (code/tts/ch11_codecs/codec_roundtrip.py; results in code/tts/results/ch11_codecs.json). The distance is the mean absolute difference between the original's and the reconstruction's log-mel spectrograms (80 bins), in dB: zero means identical spectra. It is a distance, not a listening test.

Codec	Setting	Frame rate	Bitrate	Log-mel distance	Encode RTF	Decode RTF
Mimi	8 codebooks	12.5 Hz	1.1 kbit/s	6.21 dB	0.031	0.043
Mimi	16 codebooks	12.5 Hz	2.2 kbit/s	5.79 dB	0.028	0.033
Mimi	32 codebooks	12.5 Hz	4.4 kbit/s	5.31 dB	0.030	0.034
EnCodec 24 kHz	1.5 kbit/s (2 codebooks)	75 Hz	1.5 kbit/s	6.58 dB	0.022	0.032
EnCodec 24 kHz	6 kbit/s (8 codebooks)	75 Hz	6.0 kbit/s	5.20 dB	0.021	0.023
EnCodec 24 kHz	24 kbit/s (32 codebooks)	75 Hz	24.0 kbit/s	4.45 dB	0.023	0.023
DAC 24 kHz	8 codebooks	75 Hz	6.0 kbit/s	3.32 dB	0.160	0.376
DAC 24 kHz	16 codebooks	75 Hz	12.0 kbit/s	2.57 dB	0.161	0.337
DAC 24 kHz	32 codebooks	75 Hz	24.0 kbit/s	1.91 dB	0.159	0.337

Three things to read from it. Within each codec, more codebooks reduce the distance: the residual idea at work. For Mimi and DAC each doubling of codebooks, and so of bitrate, buys a roughly constant 0.4 to 0.8 dB, so the return per bit falls. At the same 6 kbit/s, DAC's reconstruction is much closer to the original spectrum than EnCodec's (3.32 against 5.20 dB), and costs about seventeen times more CPU to decode (RTF 0.38 against 0.02). Mimi reaches EnCodec's 1.5 kbit/s distance at 1.1 kbit/s while running six times fewer frames per second, which is what a language model on top of it cares about. All three decode faster than real time on four CPU threads.

Common misconceptions

"Fewer tokens per second is always better." Fewer steps per second for the large model, yes; but the smallest emittable unit grows with the frame duration, 80 ms for Mimi against 13 ms for EnCodec.

"Any codec can stream." Only causal ones can stream without waiting. DAC and SNAC need future frames, which a streaming decoder must wait for (Orpheus waits 170.7 ms).

"Bitrate tells you quality." Codecs trained for different goals sit at different places: at the same bitrate of 6 kbit/s, DAC and EnCodec differ by almost 2 dB of log-mel distance in my measurement, and log-mel distance is itself not what a listener hears.

Try it

CPU track

1. Run python ch11_codecs/codec_roundtrip.py --threads 4 from code/tts/ in the ssh-tts-cpu environment. It writes the reconstructions under $BOOK_DATA/runs/ch11_codecs/. Listen to Mimi at 8 codebooks and EnCodec at 1.5 kbit/s on the same clip.
2. Count frames yourself: a 3.0 s clip at 24 kHz is 72,000 samples. Predict how many frames Mimi and EnCodec produce, then check the script's frames_minus_duration_over_hop field: Mimi and EnCodec pad the last partial frame to a whole one, while DAC drops it (−0.30 and −0.15 on the Mandarin clips).
3. Compute, for each codec in the first table, the time per step a large model has if it predicts one frame per step, and compare it with the 28 ms that the CPU track's Zipformer needed per 320 ms chunk in chapter 3.

GPU track

The GPU track's Qwen3-TTS uses its own 12.5 Hz tokenizer with 16 codebooks (its configuration is recorded in ch11_codecs.json under front_end_frame_rates). From the table, compute its bitrate and the duration of the 4-frame packets its paper streams in (chapter 12).

English and Mandarin

The Mandarin clips in the measurement were recorded at 16 kHz, so they contain nothing above 8 kHz. Compare their log-mel distances with the English clips' in the raw output ($BOOK_DATA/runs/ch11_codecs/rows.jsonl). Every codec and setting was further off on the Mandarin clips: EnCodec at 6 kbit/s, for example, 3.65 and 3.98 dB on the English clips against 6.48 and 6.69 dB on the Mandarin ones. Part of it is the empty band: a clip recorded at 16 kHz has almost nothing above 8 kHz, so any energy a codec puts there is a large difference in dB. Splitting one Mandarin clip (BAC009S0764W0121, EnCodec 6 kbit/s) by band gives about 17 dB in the 10 mel bins above 8 kHz against 5 dB below; but below 8 kHz it was still further off than an English clip (4.9 against 3.5 dB). So the distance is not a fair cross-language comparison, and the table's four-clip average mixes the two languages.

Key terms

• neural audio codec: an encoder, quantizer and decoder trained together to turn audio into tokens and back.
• codec frame: one step of the codec's encoder output; frame rate in Hz.
• codebook: the learned table a quantizer picks entries from; bits per code = log₂ of its size.
• residual vector quantization (RVQ): stacked codebooks, each coding the previous ones' error.
• finite scalar quantization (FSQ): rounding a few dimensions to fixed levels; one large implicit codebook.
• acoustic tokens / semantic tokens: tokens that describe the sound, or the content.
• causal codec: one whose outputs depend only on present and past input, so it streams without waiting.
• flattened tokens: every code of every frame written as a separate language-model token.

Sources

• Zeghidour et al., "SoundStream: An End-to-End Neural Audio Codec", arXiv:2107.03312.
• Défossez et al., "High Fidelity Neural Audio Compression" (EnCodec), arXiv:2210.13438; facebookresearch/encodec (commit 0e2d0ae), encodec/model.py.
• Kumar et al., "High-Fidelity Audio Compression with Improved RVQGAN" (DAC), arXiv:2306.06546; descriptinc/descript-audio-codec (commit c7cfc5d).
• hubertsiuzdak/snac (commit 8f79a71) and the hubertsiuzdak/snac_24khz configuration (revision d73ad17).
• Défossez et al., "Moshi: a speech-text foundation model for real-time dialogue", arXiv:2410.00037; kyutai-labs/moshi (commit e6a55d2), moshi/moshi/models/loaders.py.
• Hu et al., "Qwen3-TTS Technical Report", arXiv:2601.15621; Qwen/Qwen3-TTS-12Hz-1.7B-Base tokenizer configuration (revision fd4b254).
• Du et al., "CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models", arXiv:2412.10117; Du et al., "CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training", arXiv:2505.17589.
• Ye et al., "Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis", arXiv:2502.04128.
• Casanova et al., "XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model", arXiv:2406.04904.
• Ye et al., "Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model" (X-Codec), arXiv:2408.17175.
• canopyai/Orpheus-TTS (commit e64661f), orpheus_tts_pypi/orpheus_tts/decoder.py.