Whisper,
explained.

This rung supersedes rung 4's strategy rather than its architecture, and the disagreement is about what is scarce. Rung 4 treats transcripts as the bottleneck and works around them with self-supervision. This rung observes that transcripts are not actually scarce if you accept imperfect ones: the web has an enormous quantity of audio paired with text of varying quality, and the change is to take 680,000 hours of it and train an ordinary supervised encoder-decoder transformer on the lot.

No self-supervision, no pre-train-then-fine-tune, nothing clever about the architecture. The one design idea is that the task is specified in the decoder's own output: special tokens say which language this is, whether to transcribe or translate, and whether to emit timestamps. So one model does several jobs, selected by tokens, and multilingual data trains one set of weights instead of many.

Stated in rung 4's units, the payoff is the absence of a step: no fine-tuning at all. The model is used zero-shot on datasets it has never seen, and — the finding that made the paper — it is more robust out of distribution than models fine-tuned on those datasets, on noisy, accented and spontaneous speech. The explanation is that a model fine-tuned on a clean benchmark has specialised to it, while one trained across a vast diverse mess never had the chance. Diversity of weak supervision beat quality of clean supervision, and that is a claim about data strategy far beyond speech.

Before Whisper, the best speech recognizers were trained on carefully curated, hand-transcribed audio — sometimes with clever self-supervised pre-training on top. They scored beautifully on their own benchmarks. But move them to real life — a noisy café, a strong accent, a podcast, a different microphone — and accuracy fell off a cliff. They had learned the narrow world of their training set, not speech in general.

The catch was data. Clean, human-transcribed audio is scarce and expensive, so datasets stayed small and homogeneous, and models stayed brittle. To fine-tune one for your audio, you needed labeled examples of your audio. Whisper's whole premise was to escape that trap by trading label quality for label quantity.

The internet is full of audio that already comes with transcripts — videos with captions, talks with subtitles, podcasts with show notes. Individually these labels are imperfect; collectively they're a firehose. Whisper's team scraped this weakly supervised data at massive scale, then filtered aggressively: they removed machine-generated transcripts (learning from another ASR system's mistakes would just launder them), used language detection to align audio and text, and dropped low-quality pairs.

What survived was about 680,000 hours of audio — orders of magnitude more than typical supervised sets — spanning many languages, accents, topics, and recording conditions. The wager: a model trained on this vast, noisy, diverse pile would generalize far better than one trained on a small, spotless one. Diversity as the source of robustness.

A neural network can't chew on a raw waveform easily, so Whisper first turns sound into a picture-like input: a log-mel spectrogram. Slice the audio into short overlapping windows, measure how much energy sits at each frequency in each window, and lay it out as a 2D grid — frequency up the side, time across the bottom. It's essentially a heatmap of the sound.

Audio is processed in 30-second chunks, each becoming one spectrogram. This fixed-size, image-like representation is exactly the sort of grid a Transformer handles well — and it's why the same attention machinery that reads text and image patches can read speech too.

The architecture is deliberately unremarkable — a standard encoder-decoder Transformer, the same family as seq2seq and T5. The encoder reads the spectrogram and builds a rich representation of the audio; the decoder generates the transcript token by token, attending to the encoder's output — exactly like translating one sequence into another.

That plainness is the point. Whisper's authors intentionally avoided architectural novelty to isolate their real variable: the data. The message is that with enough diverse, weakly-labeled audio, a bog-standard Transformer becomes a robust, general speech recognizer. The breakthrough is the training recipe, not the network.

Whisper doesn't just transcribe. It folds several speech tasks into one model using special tokens in the decoder's input — a trick borrowed from the text-to-text playbook. A little sequence of control tokens tells the same model what to do:

Swap <|transcribe|> for <|translate|> and it renders the speech as English text instead. Set the language token, or let the model detect the language itself. Timestamps are emitted by default unless the no-timestamps token is given. Multilingual transcription, speech-to-English translation, language ID, and timestamped output — all one model, selected by tokens.

The interface is just a template. The same audio, two different jobs, one token apart:

Task, language, and timestamps are all just tokens in the prompt — the model reads them and complies. The runnable version below builds and parses these sequences.

Weak supervision leaves fingerprints. Whisper sometimes hallucinates — inventing plausible words during silence or noise, since a generative decoder is always predicting some likely text. Its quality is uneven across languages, strongest for those (like English) with the most training audio and weaker for low-resource ones. And the 30-second window means long audio must be chunked, which can drop context at the seams.

It's also purely a recognizer: no speaker diarization ("who said what"), and being an autoregressive Transformer, real-time streaming isn't its native mode. These are real gaps — but they're the price of the design that made it so broadly robust in the first place.

Whisper made robust, multilingual speech recognition a commodity. By open-sourcing a model that "just works" on real-world audio, it removed a huge barrier for anyone building voice products, and it became the default speech-to-text engine across a swathe of the industry.

Its deeper lesson echoes the rest of modern AI: scale and diversity of data can beat careful curation. Rather than engineer a cleverer model or a cleaner dataset, Whisper scaled up messy, weakly-labeled data and let a plain Transformer generalize. It's the same bet that powers large language models, applied to sound — and it's why speech joined text and images as a domain the Transformer conquered.

Whisper is trained like a captioning model: a 30-second log-mel spectrogram goes in and a sequence of tokens comes out. The paper's contributions are the data pipeline, a token format that encodes the task, and a set of decoding rules for audio longer than one window.

1. Collect the data. 680,000 hours of audio paired with transcripts found on the internet: 117,000 hours of speech in 96 languages other than English with same-language text, 125,000 hours of non-English speech with English text, used as translation data, and the remaining roughly 438,000 hours of English speech with English text.
2. Filter it. Heuristics remove transcripts that look machine-generated, since training on another recognizer's output would teach its mistakes. An audio language detector checks the spoken language against the transcript's language: mismatched pairs are dropped, unless the transcript is English, in which case they become translation examples. Transcripts are fuzzily de-duplicated, and after a first model was trained, data sources with high error rates were inspected by hand and removed when they proved low quality.
3. Cut 30-second segments. Each segment is paired with the part of the transcript that falls inside it. Segments without speech are kept, at a reduced rate, to train voice-activity detection.
4. Featurize. Audio resampled to 16 kHz becomes an 80-channel log-mel spectrogram with 25 ms windows every 10 ms: 3,000 frames per segment, scaled to roughly −1 to 1.
5. Encode and decode. Two convolutions (filter width 3, GELU, the second with stride 2) halve the frames to 1,500, one per 20 ms, and sinusoidal positions and pre-activation Transformer blocks follow. The decoder has learned positions and ties its input and output token embeddings, using GPT-2's byte-level tokenizer, refit at the same size for the multilingual models.
6. Write targets in the task format. Optionally the previous segment's text behind its own special token, then <|startoftranscript|>, a language token (one of 99) or <|nospeech|>, then <|transcribe|> or <|translate|>, then either <|notimestamps|> or text interleaved with timestamp tokens quantized to 20 ms, and finally <|endoftranscript|>. The loss skips the previous-text context.
7. Train five sizes. Tiny (39M; 4 layers, width 384), base (74M; 6, 512), small (244M; 12, 768), medium (769M; 24, 1,024) and large (1.55B; 32, 1,280), with encoder and decoder the same size. AdamW with gradient-norm clipping, batches of 256 segments, a 2,048-update warmup then linear decay, for 2^²⁰ updates, two to three passes over the data. No augmentation or regularization; large-v2, released later, trained 2.5 times as many epochs with SpecAugment, stochastic depth and BPE dropout.

Decoding audio longer than 30 seconds

Long recordings are transcribed one 30-second window at a time, and the predicted timestamps decide where the next window starts: at the end of the last complete segment, so no word is cut in half. Around that loop the paper adds five rules. Decode with a beam of 5. If the result's average log-probability is below −1 or its gzip compression ratio is above 2.4, decode again at a temperature 0.2 higher, up to 1.0. Pass a window's text on as context only if it was decoded at a temperature below 0.5. Treat a window as silent when the no-speech probability is above 0.6 and the average log-probability is below −1. And force the first timestamp to fall within the first second, so the model cannot skip the opening words.

• X — the 80 × 3,000 log-mel spectrogram of one 30-second window
• prev — the previous window's text, given as context and excluded from the loss
• y — the target: special tokens, text and timestamps, where timestamps come in 20 ms steps, 1,501 values from 0.00 to 30.00
• n — the number of generated tokens, so the average log-probability is the decoder's confidence per token
• bytes(text) / bytes(gzip(text)) — how well the text compresses; repetitive text compresses far better than speech does
• T — the sampling temperature; T = 0 means beam search

At the extremes: a loop that repeats one phrase across the window compresses many times over, so its ratio passes 2.4 even when every token is confident, and the window is decoded again with more randomness. Ordinary speech passes both tests at T = 0. A high no-speech probability alone never deletes a window; the decoder must also be unsure of what it wrote, because the paper found the no-speech probability by itself an unreliable sign of silence.

Whisper was evaluated zero-shot, with no fine-tuning on any test set, across a dozen English speech-recognition sets (LibriSpeech, TED-LIUM 3, Common Voice, CHiME-6, AMI, Switchboard, CallHome and others), plus multilingual recognition, speech translation, language identification, added noise and long recordings. WER was computed after a text normalizer the authors built and released; they note it was developed alongside Whisper and so risks favoring its transcription style. The key comparison holds in-distribution accuracy fixed and asks how much each model degrades elsewhere, an effective-robustness measure.

At the same LibriSpeech score, Whisper made 55.2% fewer errors on average on the other sets. Even the 39M-parameter tiny model, at 6.7 WER on LibriSpeech test-clean, was roughly competitive out of distribution with the best supervised LibriSpeech model.

The scaling and transfer results are less flattering:

• English WER fell quickly from 3,000 to 13,000 training hours, slowed noticeably by 54,000, and the full 680,000 hours bought only about one more point.
• Across languages, log WER tracked log training hours closely (squared correlation 0.83); the authors estimate WER halves for every 16× more data, so low-resource languages lag badly.
• For small models, training on many tasks and languages hurt English results compared with English-only training; the advantage reversed only at the largest scales.
• The long-form heuristics lowered WER overall but not evenly across datasets.
• The paper names its stubborn failure modes: repeat loops, skipping the first or last words of a segment, and complete hallucination of text unrelated to the audio.

The paper is clear that it is not state of the art in-distribution: on a clean benchmark, a model fine-tuned on that benchmark wins. The claim is specifically about robustness and generality, which is the correct comparison for something intended to be used rather than scored, and it is honest about the trade.

The most consequential real-world failure is hallucinated transcripts, and it follows from the architecture: an autoregressive decoder trained on web text will, given silence, music or noise, sometimes generate fluent text that no one said — repeated loops, or phrases characteristic of its training data such as subtitle credits. This has been documented repeatedly since, including in settings where an invented sentence in a transcript is genuinely harmful, and it is a good example of a known, structural failure that aggregate accuracy metrics hide. Long audio also needs chunking with heuristics for stitching, which introduces its own errors at the seams.

How it held up: it became the default speech recogniser, open weights and widely deployed, and the weak-supervision-at-scale argument was influential well beyond audio. Read rungs 4 and 5 as a genuine unresolved disagreement rather than a progression: where audio is plentiful and transcripts are absent, rung 4's approach is the only one available; where a large diverse weakly-labelled corpus can be assembled, rung 5 says stop engineering and collect it.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Joining text to images, audio, video and finally to a robot arm. Starts with the contrastive objective the whole field standardised on.

1. 1CLIP2021
2. 2Flamingo2022
3. 3LLaVA2023
4. 4wav2vec 2.02020
5. 5Whisper2022
6. 6V-JEPA2024
7. 7RT-22023
8. 8Genie2024

All 110 papers, in 19 ladders →