Seq2Seq,
explained.

Rung 1, so the comparison is against the machine translation it displaced. A strong translation system in 2014 was an assembly of separately-trained components: a word-alignment model, an extracted phrase table, a reordering model and a separate language model, combined by a tuned scoring function. Each piece was individually sensible and the whole thing was engineered rather than learned.

The change is to train one network, end to end, on pairs of sentences. An encoder reads the source sequence one token at a time and its final hidden state is taken as a representation of the entire sentence; a decoder is initialised from that state and generates the target sequence one token at a time. No alignments, no phrase table, no explicit reordering — the model is handed the input and the desired output and nothing else.

The unit to carry up the ladder is the fixed-length vector, because it is both the idea and the flaw. Everything the decoder will ever know about the source has to fit in that one vector, whether the sentence is four words or forty. The paper's most telling empirical result is a trick that should not have helped as much as it did: reversing the order of the source sentence improved translation quality noticeably. Reversing changes no capacity and no parameters — it only shortens the distance between the first words of the source and the first words of the output. That the fix worked is evidence that distance, not capacity, was the binding constraint, which is exactly what rung 2 attacks.

Classic neural networks assume a fixed shape: a fixed number of inputs, a fixed number of outputs. That's fine for classifying a photo into one of 1,000 labels. It falls apart for language, where the input is a sentence of arbitrary length and the output is another sentence of a different, also-arbitrary length. Translation, summarization, dialogue — all are sequence-to-sequence, and none fit a fixed-size box.

Recurrent networks could consume a variable-length sequence, but the standard setups produced one output per input step — no good when the output length differs from the input and the alignment isn't one-to-one. The field needed a way to map any sequence to any other sequence, learned end to end from data. That's the gap this paper closed.

The solution splits the job across two recurrent networks. An encoder reads the entire input sequence and boils it down to a single fixed-length vector — a compressed "meaning" of the whole sentence, often called the context vector. A decoder then takes that vector and unrolls it into the output sequence, one token at a time.

It's like a translator who listens to a full sentence, forms a complete mental impression of it, and only then starts speaking the translation — rather than translating word by word as they hear it. The bridge between the two languages is that one internal impression: the context vector.

The encoder is an LSTM — a recurrent network with gates designed to carry information across many steps without the gradients vanishing. It ingests the input tokens one by one, updating an internal hidden state each step. After the last token, that final hidden state is treated as the summary of the entire input: everything the decoder will get to know about what was said.

LSTMs mattered here because plain RNNs forget too fast — by the end of a long sentence, the influence of the early words has washed out. The gated memory of the LSTM was what made encoding a whole sentence into one usable vector even feasible.

The decoder is a second LSTM, initialized with the encoder's context vector as its starting memory. It generates the output autoregressively: predict the first output token, then feed that prediction back in to help predict the second, and so on — each new word conditioned on the context vector plus everything generated so far.

Special tokens frame the process: a start token kicks off decoding, and the decoder keeps going until it emits an end-of-sequence token, which is how a variable-length output naturally terminates itself. This "generate one token, feed it back" loop is exactly how modern language models still produce text — seq2seq is where that pattern went mainstream.

Two practical choices moved the needle a lot. First, depth: stacking several LSTM layers gave the model enough capacity to handle real translation. Second, and famously, reversing the input. Feeding the source sentence backwards improved results markedly.

Why would reversing help? It shortens the distance between related words. With the input reversed, the first source words sit close in time to the first target words the decoder must produce, so the LSTM has shorter dependency chains to learn at the crucial start of the sentence. A tiny change in data ordering, a surprisingly large gain — a reminder that in deep learning, the plumbing often matters as much as the theory.

The architecture's elegance is also its flaw. Everything about the input must pass through one fixed-size context vector. The whole model factorizes the output probability as a product of per-token predictions, but every one of them is conditioned on the same frozen summary c:

A 5-word sentence and a 50-word sentence get the exact same number of bytes to be remembered by.

For a short sentence, fine. For a long, information-dense one, that single vector becomes a straw through which a river must flow — it simply can't retain every detail, and translation quality visibly degrades as sentences get longer.

This bottleneck was the clear, well-understood weakness — and naming it so precisely is part of the paper's legacy. It set up the very next breakthrough: what if the decoder didn't have to rely on one summary vector, but could look back at all the encoder's hidden states and focus on the relevant ones? That question is attention.

Beyond the bottleneck, seq2seq inherits the costs of recurrence. Because each step depends on the previous hidden state, the encoder and decoder must run sequentially — you can't process all tokens in parallel, which makes training on long sequences slow and hard to scale. And even with LSTMs, very long-range dependencies remain difficult.

Both limitations pointed the field forward. Attention fixed the bottleneck; a few years later the Transformer removed recurrence entirely, so a whole sequence could be processed in parallel — while keeping seq2seq's encoder-decoder skeleton. Seq2seq was a crucial rung on the ladder, not the top of it.

Seq2seq established the mental model that all of modern NLP runs on: cast a task as mapping an input sequence to an output sequence, learned end to end. Translation, summarization, question answering, dialogue, code generation — all are seq2seq problems, and this paper is where that framing was proven at scale on real machine translation.

Its exact ingredients live on. The encoder-decoder split survives in the Transformer and in T5. The autoregressive "generate a token, feed it back, stop at end-of-sequence" loop is exactly how GPT-style models write. And the bottleneck it exposed directly motivated attention — the mechanism that became the beating heart of the entire field.

The paper is candid that performance degrades on long sentences, which is the fixed-vector bottleneck showing up in the metric, and it reports the reversal trick as an empirical finding it cannot fully explain rather than dressing it up as a principle. Both admissions are unusually useful, because together they localise the problem precisely.

It also depends on beam search at decoding time to get its headline numbers, and on an ensemble of models — so the reported figure is not what a single greedy model does, and the paper says so.

How it held up: the encoder-decoder frame survived completely, and the deeper claim — that a general sequence model trained end to end beats a hand-assembled pipeline of specialists — is now so thoroughly accepted that it is hard to see as a claim at all. The fixed vector did not last a year: Bahdanau and colleagues let the decoder look back at every encoder state through a learned weighting, which removed the bottleneck and introduced attention as a component. Rung 2's contribution is to notice that once you have that component, the recurrence it was bolted onto is no longer doing anything essential.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

The architecture itself: what it replaced, how position is encoded once recurrence is gone, and what it turns out to have learned inside.

1. 1Seq2Seq2014
2. 2Attention Is All You Need2017
3. 3RoPE2021
4. 4Concept Circuits2026

All 110 papers, in 19 ladders →