Listen
unlabeled.

Supervised speech recognition needs transcribed audio — thousands of hours of it — and someone has to produce those transcripts by hand. That's why strong recognizers existed for English and a few other well-resourced languages and were poor or nonexistent for most of the world's ~7,000 languages. Meanwhile, unlabeled audio is everywhere and nearly free. The obvious question: can a model learn most of what it needs from raw audio alone, so only a tiny bit of labeled data is required at the end?

NLP had already answered "yes" with BERT-style masked pretraining. wav2vec 2.0 (Baevski, Zhou, Mohamed & Auli, 2020) brought that answer to speech. It pretrains on large amounts of unlabeled audio by masking chunks of the signal and learning to recover them through a contrastive task, then fine-tunes on a small labeled set. The headline result was startling: competitive recognition from as little as ten minutes of transcribed speech, and strong systems from an hour — an order-of-magnitude cut in the labeled data required.

wav2vec 2.0 has three moving parts working on the raw waveform — no spectrograms, no hand-designed features:

The Transformer context network is what gives each masked position access to the surrounding audio, so it can infer what belongs in the gap from context — exactly the masked-prediction idea, but on continuous audio. The twist that makes speech work well is the quantizer: rather than asking the model to reproduce the raw continuous latent (which invites cheating on low-level acoustic texture), the target is a discrete quantized latent from a learned codebook. This nudges the model toward discovering a compact, phone-like inventory of speech units.

The pretraining objective is contrastive, like a lineup. For each masked step, the model's context vector c is shown a set of candidate quantized latents: the one true target q for that step, plus K distractors sampled from the quantized latents of other masked positions in the same utterance. The model must pick out the true one.

"Picking" means scoring: it computes the cosine similarity between c and each candidate, divides by a temperature κ, and turns those scores into probabilities with a softmax. Training maximizes the probability assigned to the true target — equivalently, minimizes its negative log, the InfoNCE loss. To succeed, the context vector for a masked frame must become more aligned with its own quantized content than with any distractor, which forces the representation to encode what's actually being said at that moment. It's the same contrastive principle behind CLIP (match the right pair against many wrong ones), applied within a single audio stream.

For a masked step with context c, true target q, and candidate set Q (the true target plus K distractors), the contrastive loss is:

sim(a, b) is cosine similarity; κ is a temperature. The numerator rewards similarity to the TRUE target; the denominator sums over the true target and all K distractors. This is a softmax classification over candidates, and the loss is the negative log-probability of the correct one.

Two everyday properties fall out. When the context is far more similar to its true target than to the distractors, the softmax puts nearly all mass on the truth and the loss approaches 0; when the true target is indistinguishable from the distractors, the loss approaches log(K+1) (chance). And solving the task simply means the true quantized target is the most similar candidate. The runnable version below implements cosine similarity, the InfoNCE loss over candidates, and this "did the true target win?" check.

wav2vec 2.0 collapsed the labeled-data requirement for speech recognition:

There are engineering subtleties that matter in practice. The Gumbel-softmax quantizer needs a diversity penalty so the codebook doesn't collapse to a few entries; the contrastive task is sensitive to how many distractors you use and the temperature; and fine-tuning uses a CTC loss on top. But the headline stands: most of the learning can happen without labels, and that reshaped how speech models are built.

wav2vec 2.0 established self-supervised pretraining as the default first stage for speech models, mirroring what masked pretraining did for text. Its ideas propagated fast: HuBERT swapped the contrastive objective for predicting cluster assignments (a more BERT-like target), data2vec unified the recipe across speech, vision, and text with a single self-distillation objective, and multilingual variants like XLS-R scaled the approach to dozens of languages at once — directly attacking the low-resource gap that motivated the work.

Zoom out and it's a tidy convergence point for representation learning. The mask-and-predict idea from BERT, the contrastive idea from models like CLIP, and discrete learned codebooks all meet here in the audio domain — and the pretrained representations became a strong starting point that even large supervised systems like Whisper can be compared against or built alongside. wav2vec 2.0 is the paper that made "learn from the audio you already have, label almost none of it" the standard way to teach machines to listen.