Listen
unlabeled.

The ladder changes modality here, and the constraint in speech is the same one language and vision had: labels. Training a recogniser needs transcribed audio, transcription is skilled manual work, and for most of the world's languages there is very little of it. Audio itself is abundant.

The change is to learn from raw audio with no transcripts at all, by masking part of the signal's latent representation and requiring the model to identify what was hidden. The obstacle is that audio is continuous, so there is no correct token to predict and a regression on the waveform rewards modelling noise. The fix is to quantise the targets — learn a codebook of discrete speech units, the same device as discrete latents on the generative ladder — and then pose the task as contrastive: pick the true quantised unit for the masked position out of a set of distractors. A discrete, checkable question replaces an ill-posed continuous one.

Stated in units, the result is the one that mattered: pre-train on unlabelled audio, then fine-tune on a very small amount of transcribed speech — on the order of tens of minutes — and get usable recognition. Labelled data stops being the gate, which is what made speech recognition practical for languages that will never have thousands of transcribed hours.

Supervised speech recognition needs transcribed audio — thousands of hours of it — and someone has to produce those transcripts by hand. That's why strong recognizers existed for English and a few other well-resourced languages and were poor or nonexistent for most of the world's ~7,000 languages. Meanwhile, unlabeled audio is everywhere and nearly free. The obvious question: can a model learn most of what it needs from raw audio alone, so only a tiny bit of labeled data is required at the end?

NLP had already answered "yes" with BERT-style masked pretraining. wav2vec 2.0 (Baevski, Zhou, Mohamed & Auli, 2020) brought that answer to speech. It pretrains on large amounts of unlabeled audio by masking chunks of the signal and learning to recover them through a contrastive task, then fine-tunes on a small labeled set. The headline result was startling: competitive recognition from as little as ten minutes of transcribed speech, and strong systems from an hour — roughly a hundredfold cut in the labeled data required, by the paper’s own comparison.

wav2vec 2.0 has three moving parts working on the raw waveform — no spectrograms, no hand-designed features:

The Transformer context network is what gives each masked position access to the surrounding audio, so it can infer what belongs in the gap from context — exactly the masked-prediction idea, but on continuous audio. The twist that makes speech work well is the quantizer: rather than asking the model to reproduce the raw continuous latent (which invites cheating on low-level acoustic texture), the target is a discrete quantized latent from a learned codebook. This nudges the model toward discovering a compact, phone-like inventory of speech units.

The pretraining objective is contrastive, like a lineup. For each masked step, the model's context vector c is shown a set of candidate quantized latents: the one true target q for that step, plus K distractors sampled from the quantized latents of other masked positions in the same utterance. The model must pick out the true one.

"Picking" means scoring: it computes the cosine similarity between c and each candidate, divides by a temperature κ, and turns those scores into probabilities with a softmax. Training maximizes the probability assigned to the true target — equivalently, minimizes its negative log, the InfoNCE loss. To succeed, the context vector for a masked frame must become more aligned with its own quantized content than with any distractor, which forces the representation to encode what's actually being said at that moment. It's the same contrastive principle behind CLIP (match the right pair against many wrong ones), applied within a single audio stream.

For a masked step with context c, true target q, and candidate set Q (the true target plus K distractors), the contrastive loss is:

sim(a, b) is cosine similarity; κ is a temperature. The numerator rewards similarity to the TRUE target; the denominator sums over the true target and all K distractors. This is a softmax classification over candidates, and the loss is the negative log-probability of the correct one.

Two everyday properties fall out. When the context is far more similar to its true target than to the distractors, the softmax puts nearly all mass on the truth and the loss approaches 0; when the true target is indistinguishable from the distractors, the loss approaches log(K+1) (chance). And solving the task simply means the true quantized target is the most similar candidate. The runnable version below implements cosine similarity, the InfoNCE loss over candidates, and this "did the true target win?" check.

wav2vec 2.0 pre-trains one network on unlabeled audio, then fine-tunes it on whatever transcribed audio exists. The numbers below are the paper's two configurations, BASE (about 95 million parameters) and LARGE (about 317 million).

1. Encode the waveform. Raw 16 kHz audio passes through seven convolutional blocks with 512 channels each. The output is one latent vector z every 20 ms, each covering 25 ms (400 samples) of audio.
2. Quantize the targets. From the unmasked latents, a quantizer with G = 2 codebooks of V = 320 entries picks one entry per codebook through a Gumbel-softmax; the two picks are concatenated and linearly transformed into the target q. That allows 320 × 320 = 102,400 distinct targets. The Gumbel temperature is annealed from 2 down to 0.5 for BASE and 0.1 for LARGE.
3. Mask spans. Each time step is chosen as a span start with probability 0.065, and the 10 steps from each start are masked. Spans overlap, so about 49% of steps end up masked, in spans averaging 14.7 steps (about 300 ms). Masked latents are replaced by one shared learned vector before they reach the Transformer; the quantizer still sees the originals.
4. Contextualize. A Transformer reads the masked sequence: 12 blocks of width 768 with 8 heads for BASE, 24 blocks of width 1,024 with 16 heads for LARGE. Position information comes from a convolution (kernel 128, 16 groups) whose output is added to the input, not from fixed position embeddings.
5. Score. At each masked step, the context vector c is compared by cosine similarity with its own target q and with K = 100 distractors drawn from other masked steps of the same utterance, at temperature κ = 0.1. A diversity penalty with weight 0.1 keeps the codebooks in use.
6. Pre-train. The unlabeled data is either LibriSpeech's 960 hours or LibriVox's 53,200 hours (LV-60k). BASE trained for 400K updates on 64 V100 GPUs in 1.6 days; LARGE trained on 128 V100 GPUs for 250K updates on LibriSpeech (2.3 days) or 600K updates on LibriVox (5.2 days).
7. Fine-tune. A randomly initialized linear layer maps each context vector to characters, and the model is trained with CTC loss. The convolutional encoder stays frozen; for the first 10,000 updates only the new layer trains, then the Transformer joins. Labeled subsets range from 10 minutes to the full 960 hours.
8. Decode. Transcripts come from beam search with a 4-gram or Transformer language model trained on the LibriSpeech text corpus.

Why continuous inputs but discrete targets

The Transformer reads continuous latents, which keep everything the encoder found, while it is scored against quantized ones. A continuous target would let the model win the lineup by matching speaker and background details of the utterance rather than what was said; a quantized input would throw information away before the context network could use it. The paper tests all four combinations, and the result is in the evidence below.

• Lm — the contrastive loss already shown on this page: the true target against K = 100 distractors at κ = 0.1
• G = 2, V = 320 — two codebooks of 320 entries each
• lg,v — the score the quantizer gives entry v of codebook g for one frame; n — Gumbel noise
• τ — the Gumbel temperature; the forward pass takes the hard argmax, and the backward pass uses these soft probabilities (straight-through)
• p̄g — codebook g's softmax without noise, averaged over every frame in the batch; H — its entropy

At the extremes: if every frame picks the same entry, each H(p̄_g) = 0 and L_d = 0, its worst value. If all 320 entries are used equally, H = ln 320 and L_d = −ln(320)/320 ≈ −0.018, its best. Because p̄ is averaged over the batch, the penalty does not push any single frame toward indecision; it rewards spreading choices across frames. As τ → 0 the probabilities become one-hot; a large τ makes them nearly uniform, smooth to train but vague as targets, which is why τ starts at 2 and is annealed down.

Models were pre-trained on 960 hours (LibriSpeech) or 53,200 hours (LibriVox) of audio without transcripts, fine-tuned on labeled subsets from 10 minutes to 960 hours, and compared with earlier self-training and semi-supervised systems by word error rate (WER) on LibriSpeech's clean and other test sets.

The ablation that justifies the design changes what is quantized. BASE, pre-trained on LibriSpeech and fine-tuned on 10 hours, averaged over three seeds on the dev sets:

Quantizing the input does the most damage, and the paper uses it to explain the weaker results of earlier work such as vq-wav2vec, which quantized the features its Transformer read. Continuous targets are the smaller loss: they let the model lean on speaker and background artifacts that make the task easier without helping recognition. The second row of the first table is the honest caveat on the headline: at 10 minutes, most of the word-level accuracy comes from the language model used in decoding.

It is a representation learner, not a recogniser: there is always a fine-tune, so deploying it means a labelled dataset per language and per domain, however small. The contrastive setup is also sensitive in the ways contrastive setups are — the masking span, the number and source of distractors, and the codebook's health all matter, and codebook collapse is the same hazard it is on the generative ladder.

Robustness is the deeper limitation and it is inherited from the data rather than the method: a model pre-trained on clean read speech transfers poorly to noisy, accented, spontaneous audio, because nothing in the objective asks it to.

How it held up: self-supervised pre-training became standard in speech, and the low-resource result is a genuine and lasting contribution — there are working recognisers for languages that would otherwise have none. But the paradigm was directly challenged by the next rung, which is why the two sit adjacent here. This rung's bet is that clean supervision is scarce so you should learn representations and fine-tune; the next one's is that weak supervision is abundant, and that using enough of it removes the fine-tune entirely. Both are right about something, and they are not compatible strategies.

wav2vec 2.0 established self-supervised pretraining as the default first stage for speech models, mirroring what masked pretraining did for text. Its ideas propagated fast: HuBERT swapped the contrastive objective for predicting cluster assignments (a more BERT-like target), data2vec unified the recipe across speech, vision, and text with a single self-distillation objective, and multilingual variants like XLS-R scaled the approach to dozens of languages at once — directly attacking the low-resource gap that motivated the work.

Zoom out and it's a tidy convergence point for representation learning. The mask-and-predict idea from BERT, the contrastive idea from earlier contrastive predictive coding (wav2vec 2.0 predates CLIP), and discrete learned codebooks all meet here in the audio domain — and the pretrained representations became a strong starting point that even large supervised systems like Whisper can be compared against or built alongside. wav2vec 2.0 is the paper that made "learn from the audio you already have, label almost none of it" the standard way to teach machines to listen.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Joining text to images, audio, video and finally to a robot arm. Starts with the contrastive objective the whole field standardised on.

1. 1CLIP2021
2. 2Flamingo2022
3. 3LLaVA2023
4. 4wav2vec 2.02020
5. 5Whisper2022
6. 6V-JEPA2024
7. 7RT-22023
8. 8Genie2024

All 110 papers, in 19 ladders →