Skip to content

Appendix E. Fine-tuning a streaming model on a small set

This book does not train models from scratch. It does show how to adapt one: you have a few hours of audio from your domain (product names, a dialect, a noisy room) and a pretrained streaming model that gets them wrong. This appendix lists what a fine-tune must keep, so that the result still streams, and where the official scripts put each setting. The pinned versions are icefall at commit 9c96676 (the one chapters 2 to 9 cite) and NeMo v3.0.0 (commit fd6a877, the one chapters 5, 6 and 15 cite); the two NeMo files named below are the same at commit 8eb458b, which chapters 2 and 3 cite.

Try biasing first

If the errors are a list of words (names, product codes), contextual biasing (chapter 6) fixes many of them with no training: sherpa-onnx's hotwords with modified beam search, or NeMo's GPU phrase boosting. Fine-tune when the errors are in how things sound (accent, channel, noise) or when the list is too long to bias.

What a streaming fine-tune must keep

A fine-tune changes the weights. It must not change the conditions the weights were trained for, or the model stops working as a streaming model even if its offline accuracy improves.

  1. The streaming structure. The model must keep training with the same chunked attention masks and causal convolutions it was trained with (chapter 5), and with the same set of chunk sizes, so that it still works at every setting it supported.
  2. The front end. The same features, scale and normalization (chapter 2). A streaming model trained without normalization must be fine-tuned without normalization.
  3. The vocabulary. Changing the tokenizer changes the output layer and loses what it learned; keep it unless the new domain needs characters the old one cannot write.
  4. The streaming evaluation. Measure the fine-tuned model in streaming mode, with the harness of chapter 9, on your domain and on the original test set: a fine-tune that helps one and hurts the other by more is not an improvement.

icefall Zipformer (the model behind the CPU track)

The CPU track's English model (sherpa-onnx-streaming-zipformer-en-2023-06-26) is an icefall streaming Zipformer from the zipformer recipe, exported to ONNX; the bilingual model of chapter 15 is an older Zipformer from the pruned_transducer_stateless7_streaming recipe and needs that recipe's own scripts. Fine-tune with icefall on a GPU, then export again. The recipe's fine-tuning script is egs/librispeech/ASR/zipformer/finetune.py, and its usage text shows the core flags:

./zipformer/finetune.py \
  --world-size 4 \
  --num-epochs 30 \
  --start-epoch 1 \
  --use-fp16 1 \
  --do-finetune 1 \
  --finetune-ckpt path/to/ckpt \
  --base-lr 0.0045 \
  --use-mux 0 \
  --exp-dir zipformer/exp_finetune \
  --max-duration 1000

That command, as written, fine-tunes a non-streaming model. The streaming switches have their own flags, and their defaults are off (finetune.py, lines 274-295):

  • --causal defaults to False ("If True, use causal version of model");
  • --chunk-size defaults to "16,32,64,-1", "Chunk sizes (at 50Hz frame rate) will be chosen randomly from this list during training", and its help adds "Must be just -1 if --causal=False";
  • --left-context-frames defaults to "64,128,256,-1".

For a streaming checkpoint, pass --causal 1 and the chunk sizes and left contexts the checkpoint was trained with, and the model architecture flags that match it. Two other options matter for a small set:

  • --use-mux 1 mixes "5% of the new data with 95% of the original data to fine-tune", which helps keep accuracy on the original domain (it needs the original training data);
  • --init-modules initializes only the named modules from the checkpoint.

Then export with zipformer/export-onnx-streaming.py and the chunk size you will serve (chapter 15: the chunk is fixed in the ONNX file) and load the result in sherpa-onnx as before. Its features are lhotse's 80-bin fbank with icefall's options (chapter 2); compute the new data's features the same way.

NeMo FastConformer (the GPU track)

NeMo's fine-tuning entry point is examples/asr/speech_to_text_finetune.py with the configuration examples/asr/conf/asr_finetune/speech_to_text_finetune.yaml. The script describes itself as being for "when users want to fine-tune an existing model without changing its core architecture but may change the tokenizer", initialized from init_from_nemo_model or init_from_pretrained_model; the data are JSON-lines manifests given as model.train_ds.manifest_filepath and model.validation_ds.manifest_filepath, and model.tokenizer.update_tokenizer (default false) keeps the original vocabulary.

Because the script keeps the architecture, it keeps the checkpoint's encoder configuration. For the GPU track's FastConformer, that configuration, read from the released .nemo file, is the streaming one:

normalize: NA
causal_downsampling: true
att_context_size:
- - 70
  - 13
- - 70
  - 6
- - 70
  - 1
- - 70
  - 0
att_context_style: chunked_limited
conv_context_size: causal

The list of four att_context_size settings is what makes the model work at all four latencies (chapter 15: "each model is trained for a specific latency and supporting multiple latencies need training of multiple models", arXiv:2312.17279, Sec. 4.4, unless trained with several). Read the .nemo file's model_config.yaml before you start, as code/ch02_features/model_frontends.py does for the front end, and check that the fine-tuning run's logged configuration shows the same list. If you override the encoder configuration, you are no longer fine-tuning the streaming model you measured.

How much data, and how to tell it worked

There is no fixed amount. What you can control is the measurement:

  • hold out part of your domain data as a test set before you start;
  • measure the pretrained model on it and on the book's original test sets with the chapter 9 harness, in streaming mode, at the chunk setting you will serve;
  • fine-tune, then measure the same sets the same way, with the same normalizer;
  • compare the first-token and final latencies too: a fine-tune can change when the model emits (chapter 6), not only what.

Mandarin

Both tracks have Mandarin models whose vocabularies already cover common characters. For a domain with rare characters or many English words inside Mandarin speech, check first whether the tokenizer can write them; a word the tokenizer cannot produce cannot be learned by fine-tuning with the vocabulary kept. Score Mandarin with the zh-full normalizer of chapter 9 before and after, so that numerals and full-width characters do not hide or fake an improvement.