1. Audio as numbers

What you will learn

• What a digital audio signal is on disk and in memory (samples, sample rate, bit depth, channels, bytes), and how to convert between seconds, samples, bytes and bits per second without a calculator, using the book's running example.
• Why speech recognizers run at 16 kHz and many speech synthesizers at 24 kHz, where conversions between them happen, and the handful of byte-level bugs (wrong scale, wrong data type, wrong sample rate; byte order just before them) that make a speech system "work, but badly".
• The road one utterance travels from microphone to text and from text to loudspeaker, and what real speech looks like on that road: voiced and unvoiced sounds, and the pitch period and fundamental frequency measured from the samples.
• What changes between Mandarin and English at the bottom of the stack (tones, characters without spaces, CER and WER), and the audio formats you will meet: 8 kHz telephone audio, 16 kHz recognition input, 24 kHz synthesis output, lossy codecs such as Opus.

Before you start

• Chapters: none. This is the first chapter. No signal-processing background is assumed; the pitch measurement uses nothing beyond sums of products.
• Tools: Python 3.11 with numpy, soundfile and torchaudio (see Try it). No GPU. The English and Mandarin recordings are LibriSpeech test-clean and AISHELL-1 test files (see code/README.md for where the code looks for them).

The road from microphone to text and back

This book follows one utterance. Someone says "so we harried the coast of Norway" into a microphone. The recording used later in this chapter is exactly that sentence: 2.165 s, 34,640 samples at 16 kHz (code/ch01_audio/results/speech_pitch.json). The picture below is the road that these samples travel, in both directions. Every chapter of the book covers one stretch of it.

Speech recognition (ASR), top row. The microphone and its analog-to-digital converter produce samples: 16,000 numbers per second for the recognizers in this book. This chapter covers that step and the bugs that can happen there. A model does not read samples one by one. Chapter 2 cuts them into overlapping frames and turns each frame into a vector of features, one vector every 10 ms. The recording above becomes about 217 of them (n_hops in the same result file). Chapter 3 gives names to the parts of delay. Chapter 4 shows how a model turns a sequence of feature vectors into text, with three kinds of output layer. Chapter 5 shows how its encoder can run on audio that is still arriving. Chapter 6 covers the partial results that appear while someone is still talking, and chapter 7 the decision that they have stopped. Chapter 8 treats Whisper and language-model recognizers. Chapter 9 measures all of it.

Speech synthesis, also called text-to-speech (TTS), bottom row. Text goes in. Chapter 10 describes the whole pipeline. An acoustic model decides which sounds the text needs, how long each lasts and at what pitch (chapter 12 covers the models that can do this while the text is still coming). Its output is not yet audio. It is a compact description of audio: mel frames (one short list of numbers every 10 ms or so, saying how strong each band of frequencies is, as in chapter 2), or the discrete codes of a neural codec (chapter 11). A vocoder or codec decoder turns that description back into samples, 24,000 per second for the models in this book (chapter 13 covers what goes wrong at the boundary between two chunks). The converter and the loudspeaker finish the job. Chapter 14 measures synthesis.

Around the road. Chapters 15 and 16 serve the recognizer and the synthesizer to many users at once. Chapter 17 moves the samples between a client and a server, usually as Opus. Chapter 18 joins both rows with a language model in the middle, which is a voice agent, and adds up its delay.

Three facts about this road will save you time later.

1. Every arrow carries an array whose size you can compute. 34,640 numbers enter the recognizer's front end. About 217 vectors of 80 numbers leave it (chapter 2). When a number in a log does not match this arithmetic, something upstream is wrong.
2. The rate changes at every step. 16,000 samples per second become 100 feature vectors per second, and then a handful of text tokens per second (a token is a word or a piece of a word that the model writes). Streaming is the question of how much of the earlier, faster stream a step must see before it can emit anything.
3. The two rows share only their ends. Recognition ends in text, synthesis starts from text, and nothing else crosses. That is why the book can teach them one after the other, and why a voice agent (chapter 18) can put any language model between them.

A signal is a list of numbers

A microphone turns air pressure into a voltage. An analog-to-digital converter measures that voltage at regular intervals and stores each measurement as a number, called a sample. A digital audio signal is nothing more than that list of numbers plus two facts about it: how often it was measured and how each number is stored.

The sample rate is the number of samples per second, in hertz. Speech recognition models are commonly trained at 16 kHz, 16,000 samples per second. The Hugging Face Audio Course explains why: "The audible frequencies in human speech are below 8 kHz and therefore sampling speech at 16 kHz is sufficient" (HF Audio Course, Unit 1). That is a statement about what recognition needs, not a claim that nothing audible lies above 8 kHz. A sampled signal can represent frequencies up to half its sample rate, "known as the Nyquist limit and is exactly half the sampling rate" (same page); above it, frequencies fold back as false lower ones (aliasing). At 8 kHz, the old telephone rate, the limit is 4 kHz, and "speech sampled at 8 kHz will sound muffled".

So five seconds of speech at 16 kHz is 5 × 16,000 = 80,000 samples (the course gives the same number).

Bit depth and scale

Each sample is stored with a fixed number of bits. With 16 bits a sample can take 65,536 different values; with 24 bits, 16,777,216 (HF Audio Course, lines 62-65). The speech corpora in this book store 16-bit samples (LibriSpeech's FLAC files, AISHELL-1's WAV files), and 16 bits is enough: "the quantization noise of 16-bit audio is already small enough to be inaudible" (same place). Classic telephony uses 8 bits per sample after logarithmic scaling (G.711, below).

Integer samples are signed: a 16-bit sample runs from −32,768 to 32,767. Models work with floating-point numbers, and the common convention maps them into the range −1 to 1 by dividing by 32,768. Whisper's loader does exactly that: np.frombuffer(out, np.int16).flatten().astype(np.float32) / 32768.0 (whisper/audio.py, lines 45-62). So −32,768 becomes exactly −1.0 and +32,767 becomes 0.99997: the range is off by one step at the top, which never matters but explains why you will not see an exact +1.0.

Loudness is usually given in decibels relative to the largest possible value, dBFS: every −6 dB halves the amplitude (20 log₁₀ 0.5 = −6.02), and one 16-bit step is 20 log₁₀(1/32,768) = −90.3 dBFS.

Channels and frames

A stereo recording has two channels, stored interleaved: left sample, right sample, left, right. In the WAV file format a "frame" means one sample for every channel; Python's wave module computes nframes = len(data) // (sampwidth * nchannels) (CPython Lib/wave.py, line 576). This word will mean something else in chapter 2 (a window of audio for feature extraction) and in chapter 11 (a codec frame). Speech recognition almost always uses one channel, so everything in this book is mono unless it says otherwise.

Counting in the running example

Throughout the book one example recurs: 16 kHz mono audio, cut into 160 ms chunks. A streaming recognizer receives one chunk at a time. It is worth knowing these numbers by heart:

Quantity	Value
1 s at 16 kHz, 16-bit mono	16,000 samples = 32,000 bytes = 256 kbit/s
one 160 ms chunk	2,560 samples = 5,120 bytes as int16, 10,240 bytes as float32
one 10 ms step (chapter 2's hop)	160 samples = 320 bytes as int16
5 s WAV file, 16 kHz int16 mono	80,000 samples = 160,000 data bytes + 44 header bytes

The general rule is bytes per second = sample rate × channels × bytes per sample. A few more rates you will meet:

Format	Bit rate
8 kHz, 16-bit mono	128 kbit/s
24 kHz, 16-bit mono (a common TTS output)	384 kbit/s
48 kHz, 16-bit mono (WebRTC's internal rate)	768 kbit/s
44.1 kHz, 16-bit stereo (CD audio)	1,411.2 kbit/s
G.711 telephony, 8 kHz, 8 bits per sample	64 kbit/s
Opus, 20 ms frames, fullband speech	28 to 40 kbit/s recommended (RFC 6716 §2.1.1)

The last two lines are compressed formats: G.711 stores "eight bits per sample, after logarithmic scaling" (RFC 3551 §4.5.14), and Opus, covered in chapter 17, compresses 48 kHz speech (768 kbit/s as raw 16-bit mono) by a factor of 19 to 27: 768 divided by 40 and by 28.

WAV files and byte order

A WAV file is a short header followed by the samples. Python's wave writer builds the canonical header in one call (PCM, pulse-code modulation, is the plain format in which every sample is stored as one number; WAVE_FORMAT_PCM is the header's code for it): struct.pack('<L4s4sLHHLLHH4s', 36 + self._datalength, b'WAVE', b'fmt ', 16, WAVE_FORMAT_PCM, self._nchannels, self._framerate, self._nchannels * self._framerate * self._sampwidth, self._nchannels * self._sampwidth, self._sampwidth * 8, b'data') (Lib/wave.py, lines 620-636). Read it as a recipe: a 12-byte "RIFF ... WAVE" preamble, a 24-byte format chunk (channels, sample rate, bytes per second, bytes per frame, bits per sample), and an 8-byte "data" header, 44 bytes in all, then the samples. The < means little-endian: the least significant byte comes first.

The same 16-bit samples travel the other way round over RTP, the protocol behind voice calls: L16 audio (RTP's name for plain 16-bit PCM) is "transmitted in network byte order (most significant byte first)" (RFC 3551 §4.5.11). Byte order is therefore not a theoretical worry. The sample 1000 is the bytes e8 03 little-endian; read big-endian, they are −6,141. A whole file read with the wrong byte order turns into loud noise.

Three silent bugs

Audio bugs are rarely crashes. The model receives a sequence of numbers of the right length and returns text that is worse than it should be. Three bugs account for most of it.

Wrong scale. Some feature extractors expect samples in the int16 range, others in −1 to 1. sherpa-onnx says so in a comment: "In k2, audio samples are in range [-1..+1], in kaldi the range was [-32k..+32k]" (sherpa-onnx features.h, lines 40-53), and WeNet multiplies by 2¹⁵ before calling a Kaldi-compatible extractor (waveform = waveform * (1 << 15), wenet/dataset/processor.py, lines 253-255). Feed one where the other is expected and every value is off by a factor of 32,768. Because the features of chapter 2 are logarithms of power, that becomes a constant offset of ln(32,768²) = 20.79 on every feature value. Nothing raises an error. In the test of chapter 2, a NeMo streaming model given features with this offset returned empty text for all 50 test utterances: the bug looks like a model that hears nothing, not like a crash.

Wrong data type. A float32 sample takes four bytes, an int16 two. The float 0.5 is the bytes 00 00 00 3f; read as int16 it becomes two samples, 0 and 16,128. The clip doubles in length and turns into a buzz. The reverse mistake, int16 bytes read as float32, halves the length and produces huge values and non-numbers (the measurement in Try it shows it).

Wrong sample rate. A 24 kHz recording played or recognized as 16 kHz plays at two-thirds of its speed and pitch; the recognizer hears a different, deeper voice speaking slowly, and its accuracy drops without any error message.

The defence is the same for all three: check the numbers. Print the sample rate, the data type, the minimum and maximum, and the duration of every file at the boundary of your system, and listen to one.

Resampling

When two parts of a system disagree on the sample rate, one of them must resample. "Resampling is the process of making the sampling rates match" (HF Audio Course, line 46). It is not just dropping or repeating samples: to go down from 48 to 16 kHz, a resampler first removes everything above the new 8 kHz limit with a low-pass filter, or those frequencies would alias, and then keeps every third sample. torchaudio's default resampler is a windowed sinc filter with a Hann window, 6 zero crossings and a cut-off at 0.99 of the lower Nyquist frequency (lowpass_filter_width: int = 6, rolloff: float = 0.99, torchaudio functional.py, lines 1435-1442), which for 48 to 16 kHz puts the cut-off at 7,920 Hz. Ratios matter for cost: 48 to 16 kHz is a clean 3:1, 44.1 to 16 kHz is 160:441.

In a voice system, conversions are everywhere. The book's ASR models all expect 16 kHz (Whisper's SAMPLE_RATE = 16000, audio.py, line 13; NeMo's streaming configuration sample_rate: 16000, fastconformer_transducer_bpe_streaming.yaml at 8eb458b). Its TTS models and codecs produce 24 kHz: Kokoro's model card writes its output with sf.write(f'{i}.wav', audio, 24000) (hexgrad/Kokoro-82M README at f3ff357, line 67); CosyVoice 2's configuration has sample_rate: 24000 (cosyvoice2.yaml at 074ca6d, line 8); Mimi's code has SAMPLE_RATE = 24000 (moshi loaders.py at e6a55d2, line 28). WebRTC transport runs at 48 kHz (chapter 17). A round trip through a voice agent therefore converts at least twice: 48 to 16 kHz on the way in, 24 to 48 kHz on the way out. Sometimes the conversion happens inside the codec: "any Opus decoder operating at 8 kHz can decode an FB Opus frame" (RFC 6716 §2.1.3), so a server can ask Opus for 16 kHz directly. (FB is Opus's fullband mode, the widest of its five audio bandwidths; they are listed at the end of this chapter.)

Libraries also resample for you, sometimes silently. sherpa-onnx: "If it is different from the sampling rate of the input waveform, we will do resampling inside" (features.h, lines 16-18). Convenient, but it hides the rate you are actually feeding. Know the rate at every boundary.

What speech looks like

So far a recording was a list of numbers. This section looks at what those numbers do for real speech. Two recordings serve for this chapter (later chapters use others; chapter 2, for example, reads LibriSpeech 1089-134686-0006 and AISHELL-1 BAC009S0764W0227): an English one, LibriSpeech test-clean utterance 5142-33396-0016 ("so we harried the coast of Norway", 2.165 s, 34,640 samples), and a Mandarin one, AISHELL-1 test utterance BAC009S0904W0260 (特别是企业自主创新方面, 2.657 s, 42,512 samples). Both are 16 kHz. Both come from the book's test subsets (code/data/manifests/).

The whole sentence (top row). Seen from a distance, speech is bursts of energy with dips between them. The dips are pauses and quiet sounds. Nothing in this view tells you what was said. Zoom in on the shaded voiced stretch.

One voiced stretch, 100 ms (middle row). The curve is no longer random. A shape repeats. In the English stretch it repeats 25.08 times in the 100 ms, and in the Mandarin stretch 13.07 times (periods_in_window).

A few periods, 30 ms (bottom row). At this scale you can count the repeats by eye. The dashed lines are one repeat apart. For the English stretch that distance is 3.99 ms, for the Mandarin stretch 7.65 ms. This distance is the pitch period, the time after which the waveform repeats. The fundamental frequency, written f0, is its reciprocal: f0 = 1/T, "the lowest frequency of a periodic waveform" (Wikipedia, Fundamental frequency, revision 1361397526). The English stretch has T = 3.99 ms, so f0 = 250.8 Hz. The Mandarin stretch has T = 7.65 ms, so f0 = 130.7 Hz. What a listener hears as how high the voice sounds is the pitch; f0 is the number we measure for it. Chapter 2 reads a vowel of a different English recording and finds 93.4 Hz: these are different speakers, and f0 is a property of the speaker as much as of the sound.

Voiced and unvoiced. The repeating stretches come from the vocal folds. In phonetics, voicing is "the articulatory process in which the vocal folds vibrate", and speech sounds are either voiced or voiceless (unvoiced) (Wikipedia, Voice (phonetics), revision 1373877429). The page's own example is the English pair [s] and [z]: the two differ by voicing, and a finger on the throat feels a vibration during [z] but not during [s]. Unvoiced fricatives such as [s] look like noise on the page: the waveform does not repeat. Other unvoiced sounds look different, and this chapter measures only the noise-like kind.

Two stretches picked by rule. The two shaded 100 ms stretches were picked by the script, not by listening and not from the transcript, so they are not aligned to any sound or word. The voiced stretch is the loudest 100 ms in which every 10 ms step has a clear repeat (r of at least 0.7, defined below) and a period within 15% of the stretch's median. The noise-like stretch is the loudest 100 ms in which no step repeats (r below 0.5), the signal is not silent (no more than 40 dB below the loudest frame) and it crosses zero at least 4,000 times per second (the thresholds are under method in the result file). A stretch picked this way is steady and voiced by construction. What the measurements below show is how differently the two kinds of stretch behave.

Three numbers separate them.

• r, the normalized autocorrelation. Shift the stretch by a lag, a number of samples, and compare it with the unshifted copy. For each lag, r is a number between −1 and 1: 1 if the shifted copy is identical, near 0 if the two are unrelated. The table gives r at the best lag from 2.5 to 12.5 ms.
• Zero crossings per second. How many times per second the signal changes sign. Slow repeating waves cross zero rarely; noise crosses zero often.
• Share of power above 4 kHz. The power of a stretch is the mean of its squared samples. Every frequency carries part of it, and this column is the part carried by frequencies above 4 kHz. Splitting power by frequency is the subject of chapter 2; here read it as "how much of the sound sits in the high frequencies".

100 ms stretch	r at its best lag	zero crossings per second	share of power above 4 kHz
English voiced	0.97	1,400	1%
English noise-like	0.28	8,030	95%
Mandarin voiced	0.84	1,050	0.05%
Mandarin noise-like	0.32	6,770	47%

(Same file, keys voiced_stretch and noise_stretch.) The voiced stretches repeat, cross zero rarely and keep nearly all their power below 4 kHz. The noise-like stretches do not repeat (r is 0.29 and 0.39 of the voiced r, from 0.281 / 0.966 and 0.324 / 0.839), cross zero often and keep much of their power above 4 kHz. These shares are facts about these two stretches only; chapter 2 measures an English [s] over its own range.

How f0 was measured. The script code/ch01_audio/speech_pitch.py cuts the recording into 40 ms frames every 10 ms. For each frame it computes r for every lag from 2.5 ms to 12.5 ms (400 to 80 Hz) and takes the shortest lag whose r is within 10% of the best one; this avoids reporting two periods instead of one. A frame counts as voiced when r is at least 0.6, when it is within 30 dB of the loudest frame, and when the lag is not at the edge of the range. It refines the lag between samples by a parabola through three neighboring values of r, which matters because one sample at 16 kHz is 62.5 µs and the English period, 63.79 samples, is not a whole number of samples. On a synthetic harmonic signal at 100, 125, 210 and 330 Hz the estimate is within 0.01 Hz (selfcheck in the result file).

What it says about the whole sentence. In the English recording 124 of 217 hops are voiced (57%), with a median f0 of 191.7 Hz and the 10th to 90th percentile from 139.9 to 252.9 Hz. In the Mandarin recording 107 of 266 hops are voiced (40%), median 132.9 Hz, 10th to 90th percentile 110.0 to 155.5 Hz. For comparison, a source on telephony gives "a fundamental frequency from 90 to 155 Hz" for the voiced speech of a typical adult male and "165 to 255 Hz" for a typical adult female (Wikipedia, Voice frequency, revision 1381487731). Both medians lie inside one of these bands, and f0 is not constant: it moves by tens of hertz within a sentence. That movement carries the melody of the sentence in English. In Mandarin it also carries words, as the next section shows.

Why this matters for the next chapters. A 25 ms window, the unit of chapter 2, holds 25 / 3.99 = 6.3 periods of the English voiced stretch and 25 / 7.65 = 3.3 periods of the Mandarin one. Each window of chapter 2 therefore contains several repeats of the voice's own period, and the spectrum computed from it can show them.

Mandarin and English: what changes at the bottom of the stack

The road of the first section is the same for both languages. Two things change at the bottom of the stack: what the sounds carry, and how the text is written.

Tones. Standard Chinese "is tonal. This means that in addition to consonants and vowels, the pitch contour of a syllable is used to distinguish words from each other" (Wikipedia, Standard Chinese phonology, revision 1383211084). There are four main tones and a fifth, the neutral tone, which has "no specific contour" and sits on weak syllables. On a scale where 1 is low and 5 is high, the main tones are a high level tone, a rise from mid to high, a low dipping tone, and a short sharp fall from high to the bottom of the speaker's range (Wikipedia, Tone (linguistics), revision 1379552375). The dipping tone is a low fall that is followed by a rise when the syllable is said alone or ends a sentence; in running speech the rise is usually missing (Standard Chinese phonology, same revision as above, table of tones and "Third tone"). The same page gives the standard example, one syllable with five readings, written in pinyin with a mark over the vowel for the tone: mā 妈 "mother", má 麻 "hemp", mǎ 马 "horse", mà 骂 "scold", and ma 吗, a question particle. Same consonant, same vowel, different f0 contour, different word. The f0 that the last section measured is not only the melody of the sentence here. It is part of the spelling.

The Mandarin recording has 11 characters and 11 syllables: tè bié shì qǐ yè zì zhǔ chuàng xīn fāng miàn (the standard pinyin of the sentence). Each syllable carries one tone. In 80% of the recording's voiced 10 ms steps f0 lies between 110.0 and 155.5 Hz (the 10th to 90th percentile). A tone is a contour relative to the speaker's own range, which is why the sources describe it on a five-level scale and not in hertz. The same source counts "only about 400 different syllables when tone is ignored, and only about 1300 when it is taken into account", and "a lot of homophones" (Standard Chinese phonology, same revision). So the pitch separates syllables, and context has to choose among characters that sound alike.

Characters, no spaces. Chinese is written in characters, and is among the scripts "consistently written without punctuation to separate words" (Wikipedia, Word divider, revision 1380602443). The Mandarin transcript in the book's manifest does contain spaces (特别 是 企业 自主 创新 方面): six words as the transcript segments them, over 11 characters (n_words_as_transcribed and n_characters_without_spaces in speech_pitch.json). Another segmenter could draw other boundaries and get another count. The English transcript has 7 words and 27 letters, and nothing needs to decide where a word ends.

Pinyin is "the most common romanization system for Standard Chinese", the name meaning "spelled sounds" (Wikipedia, Pinyin, revision 1383248903). It writes Mandarin pronunciation in Latin letters and tone marks. In this book it is a way of writing sounds in explanations and the intermediate form of a Mandarin synthesizer's text front end (chapter 10). A recognizer outputs characters.

CER and WER. The word error rate is (S + D + I) / N: substituted, deleted and inserted words over the N words of the reference (Wikipedia, Word error rate, revision 1281062858). In Mandarin N depends on the segmenter, so the usual metric counts characters instead, the character error rate (CER). The WeNet paper, for example, reports CER on AISHELL-1 (Yao et al., arXiv 2102.01547). One wrong word in the English recording is 1/7 = 14.3% WER. One wrong character in the Mandarin one is 1/11 = 9.1% CER. Chapter 9 measures both. From here on, every hands-on chapter gives its example in English (LibriSpeech test-clean) and in Mandarin (AISHELL-1 test).

The audio you will meet in practice.

• 8 kHz telephone audio. The usable voice band in telephony runs from "approximately 300 to 3400 Hz" (Voice frequency, revision 1381487731). G.711 is listed at a clock rate of 8,000 Hz in RFC 3551 (Table 4, §6) and codes each sample in 8 bits (§4.5.14). The Nyquist limit is then 4 kHz. In the two noise-like stretches above, 95% (English) and 47% (Mandarin) of the power lies above 4 kHz, and an 8 kHz recording keeps none of that. Resampling to 16 kHz fixes the sample rate; it does not bring the band back.
• 16 kHz recognition input and 24 kHz synthesis output, as in Resampling above.
• Lossy codecs, Opus above all. Opus is a "lossy audio coding format" (Wikipedia, Opus (audio format), revision 1379947013). It "scales from low bitrate narrowband speech at 6 kbit/s to very high quality stereo music at 510 kbit/s", with audio bandwidths from 4 kHz (8 kHz sample rate) to 20 kHz (48 kHz) (RFC 6716, abstract and §2, Table 1). Lossy means the decoded samples are not the samples that went in. Raw PCM (see WAV files above), as in a WAV file, keeps every number, at 256 kbit/s for 16 kHz mono; chapter 17 sends audio both ways and compares them.

Common misconceptions

"Higher sample rates make speech recognition better." Speech models are trained at 16 kHz; feeding them 48 kHz audio means a resampler throws the extra bandwidth away first, or, if nothing resamples, the model hears the wrong speed. Match the model's training rate.

"A Mandarin tone is a fixed pitch, like a note on a piano." A tone is a contour, and it is relative to the speaker. The English recording has a median f0 of 191.7 Hz and the Mandarin one 132.9 Hz (two different speakers), and a high level tone is "high" for each of them, not at the same number of hertz. Tone sources therefore describe tones on a five-level scale from low to high.

"If nothing raised an error, the audio input is fine." None of the bugs above raises an error. A wrong scale can give empty text, a slightly wrong sample rate can give plausible but worse text. Check the numbers at every boundary.

Try it

The code for this chapter is in code/ (see code/README.md, chapter 1). It needs only a CPU.

CPU track

Run python code/ch01_audio/audio_numbers.py (setup in code/README.md). It takes one LibriSpeech test-clean utterance (1089-134686-0006, 10.555 s) and does four things; my numbers below come from its output file, code/ch01_audio/results/audio_numbers.json.

1. Sizes. The utterance has 168,880 samples. As a 16-bit WAV file it is 337,804 bytes: 337,760 bytes of samples plus the 44-byte header. As float32 it is 675,600 bytes: 675,520 bytes of samples plus an 80-byte header, because a float WAV file carries an extra chunk. Check both against the formula before you look.
2. Chunks. Cut into 160 ms chunks it gives 66 chunks of 2,560 samples (5,120 bytes each), except the last, which has 2,480 samples (155 ms). A streaming client must send that short last chunk too.
3. Misreads. The same bytes read four wrong ways:

Read as	Samples	Duration	What you hear
int16 little-endian (correct)	168,880	10.555 s	the sentence; peak 0.58
int16 big-endian	168,880	10.555 s	loud broadband noise; peak 1.0
float32	84,440	5.278 s	garbage: 18% of values are not finite, and 43% of the rest lie outside −1 to 1
stereo (two channels)	84,440 per channel	5.278 s	every second sample: half length, chipmunk speed
8 kHz instead of 16 kHz	168,880	21.11 s	slow and an octave low

A check of the peak catches the big-endian and float32 rows; a check of the duration against the expected length catches the stereo and wrong-rate rows. Only listening catches everything, so do all three. 4. Resampling. A 1 s signal at 48 kHz holds a 1 kHz and a 10 kHz tone of equal strength. Going to 16 kHz by keeping every third sample folds 10 kHz to 16 − 10 = 6 kHz: half the energy of the result (−3.0 dB of the total) sits at 6 kHz, a tone that was never there. torchaudio's resampler leaves −45.1 dB there, because its low-pass filter removed the 10 kHz tone first. A round trip of the utterance through 48 kHz and back to 16 kHz returns it with a signal-to-error ratio of 40.8 dB.

Pitch of real speech. python code/ch01_audio/speech_pitch.py (the ssh-asr environment from code/README.md, CPU, a few seconds) reads one English and one Mandarin recording, measures the pitch period of each by autocorrelation, and writes code/ch01_audio/results/speech_pitch.json. Expect first the check on a synthetic signal, four lines in which the estimate matches the true f0 to within 0.01 Hz (for example synthetic 125.0 Hz -> 125.00 Hz r=1.000), then, for each recording, a line of the form en 5142-33396-0016 2.165 s voiced 57% median f0 191.7 Hz followed by a voiced_stretch: line and a noise_stretch: line. In the voiced_stretch: line of the English recording look for period_ms=3.987 and f0_hz=250.804 (period_samples=63.795 is not a whole number, on purpose), and for the Mandarin recording period_ms=7.652 and f0_hz=130.681. In both noise_stretch: lines r_peak is about 0.3 to 0.4 of the voiced one: 0.281 against 0.966 (0.29) and 0.324 against 0.839 (0.39). To redraw the three-scale figure run PYTHONPATH=code python figures/ch01/data/export_speech_scales.py and then tools/build-figures.sh -f figures/ch01/speech_scales.py. Then try another recording (--utt en librispeech-test-other-200 <utt_id> --out work/pitch_try.json, which leaves the committed result file alone). Before you read the output, guess from listening whether its f0 is above or below 191.7 Hz: a deeper voice must give a longer period.

GPU track

None for this chapter.

English and Mandarin

Repeat step 1 on one English and one Mandarin recording from the book's test sets: --wav <path>, with paths from code/data/manifests/librispeech-test-clean-200.tsv and aishell1-test-200.tsv (the manifests store paths relative to $BOOK_DATASETS; prefix them with it). The numbers do not depend on the language; the check is that your loader does not either. The pitch script is the opposite case: it already takes one recording of each language, and its f0 numbers do depend on the speaker. Compare the two medians in speech_pitch.json and say which of the two recordings you expect to sound deeper.

Key terms

• sample: one measurement of the signal.
• sample rate: samples per second, in Hz; speech models use 16 kHz.
• Nyquist limit: half the sample rate, the highest frequency a sampled signal can represent.
• aliasing: frequencies above the Nyquist limit folding back as false lower ones.
• bit depth: bits per sample; 16 bits gives 65,536 values.
• dBFS: decibels relative to the largest representable value.
• interleaved: channels stored sample by sample, left then right.
• little-endian / big-endian: least or most significant byte first.
• resampling: changing the sample rate, with filtering to prevent aliasing.
• voiced / unvoiced: a speech sound is voiced when the vocal folds vibrate while it is made; its waveform repeats. An unvoiced sound has no such vibration; an unvoiced fricative such as [s] looks like noise.
• pitch period: the shortest time after which a voiced waveform repeats, for example 3.99 ms in the English recording of this chapter.
• fundamental frequency (f0): the reciprocal of the pitch period, in Hz; for the 3.99 ms period above it is 250.8 Hz. Pitch is how high a listener hears the voice; f0 is the number measured for it.
• normalized autocorrelation: how well a stretch of signal matches itself shifted by a lag, from −1 to 1; 1 means an exact repeat.
• tone: in Mandarin, the pitch contour of a syllable, which distinguishes words; four main tones (high level, rising, dipping, falling) and a neutral one.
• neutral tone: the fifth Mandarin tone, with no specific contour, on weak syllables.
• pinyin: the standard romanization of Mandarin, in Latin letters with tone marks.
• word error rate (WER) / character error rate (CER): edits over reference words, or over reference characters; WER is the usual metric for English and CER for Mandarin (chapter 9).
• Opus: a lossy audio codec for speech and music, 6 to 510 kbit/s; chapter 17.
• lossy / lossless: a lossy codec returns samples that differ from the input; raw PCM keeps every number.

Sources

• Hugging Face Audio Course, Unit 1, "Introduction to audio data" (commit 56b6e83): https://github.com/huggingface/audio-transformers-course/tree/56b6e8334aef7e0efa0137574f068c968f3f6e1c
• openai/whisper (commit 86098128), whisper/audio.py: https://github.com/openai/whisper/tree/86098128c0b4f24f0e2aa2994de830614b474227
• CPython 3.13, Lib/wave.py: https://github.com/python/cpython/blob/v3.13.0/Lib/wave.py
• RFC 3551, RTP Profile for Audio and Video Conferences: https://datatracker.ietf.org/doc/html/rfc3551
• RFC 6716, Definition of the Opus Audio Codec: https://datatracker.ietf.org/doc/html/rfc6716
• sherpa-onnx (commit 935e5de), sherpa-onnx/csrc/features.h: https://github.com/k2-fsa/sherpa-onnx/tree/935e5ded3d949a8b557ac44ef4dad03a7d2038e7
• WeNet (commit be6ad97), wenet/dataset/processor.py: https://github.com/wenet-e2e/wenet/tree/be6ad97fdb512589df113042f7ff3a29d392fb38
• torchaudio (commit 9b19155), src/torchaudio/functional/functional.py: https://github.com/pytorch/audio/tree/9b19155e6fce04a83704eadc85d3dfe99e520c0c
• Wikipedia, "Voice (phonetics)", revision 1373877429: https://en.wikipedia.org/w/index.php?oldid=1373877429
• Wikipedia, "Fundamental frequency", revision 1361397526: https://en.wikipedia.org/w/index.php?oldid=1361397526
• Wikipedia, "Voice frequency", revision 1381487731 (the 300 to 3400 Hz telephone band; typical adult f0 ranges, citing Baken, Clinical Measurement of Speech and Voice, 2nd ed., p. 177): https://en.wikipedia.org/w/index.php?oldid=1381487731
• Wikipedia, "Standard Chinese phonology", revision 1383211084 (tones; syllable counts; homophones): https://en.wikipedia.org/w/index.php?oldid=1383211084
• Wikipedia, "Tone (linguistics)", revision 1379552375 (the five Mandarin tones and the ma set): https://en.wikipedia.org/w/index.php?oldid=1379552375
• Wikipedia, "Pinyin", revision 1383248903: https://en.wikipedia.org/w/index.php?oldid=1383248903
• Wikipedia, "Word divider", revision 1380602443: https://en.wikipedia.org/w/index.php?oldid=1380602443
• Wikipedia, "Word error rate", revision 1281062858: https://en.wikipedia.org/w/index.php?oldid=1281062858
• Wikipedia, "Opus (audio format)", revision 1379947013: https://en.wikipedia.org/w/index.php?oldid=1379947013
• RFC 6716 §2 Table 1 (audio bandwidths and sample rates), §2.1.1 (bitrates) and abstract; RFC 3551 §6 Table 4 (PCMU at 8,000 Hz): see the RFC links above.
• Yao et al., "WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit" (arXiv 2102.01547), CER on AISHELL-1: https://arxiv.org/abs/2102.01547
• LibriSpeech (OpenSLR 12, CC BY 4.0) and AISHELL-1 (OpenSLR 33, Apache-2.0), the corpora of the two recordings: https://www.openslr.org/12 and https://www.openslr.org/33