Seq2Seq,
explained.

Rung 1, so the comparison is against the machine translation it displaced. A strong translation system in 2014 was an assembly of separately-trained components: a word-alignment model, an extracted phrase table, a reordering model and a separate language model, combined by a tuned scoring function. Each piece was individually sensible and the whole thing was engineered rather than learned.

The change is to train one network, end to end, on pairs of sentences. An encoder reads the source sequence one token at a time and its final hidden state is taken as a representation of the entire sentence; a decoder is initialised from that state and generates the target sequence one token at a time. No alignments, no phrase table, no explicit reordering — the model is handed the input and the desired output and nothing else.

The unit to carry up the ladder is the fixed-length vector, because it is both the idea and the flaw. Everything the decoder will ever know about the source has to fit in that one vector, whether the sentence is four words or forty. The paper's most telling empirical result is a trick that should not have helped as much as it did: reversing the order of the source sentence improved translation quality noticeably. Reversing changes no capacity and no parameters — it only shortens the distance between the first words of the source and the first words of the output. That the fix worked is evidence that distance, not capacity, was the binding constraint, which is exactly what rung 2 attacks.

Classic neural networks assume a fixed shape: a fixed number of inputs, a fixed number of outputs. That's fine for classifying a photo into one of 1,000 labels. It falls apart for language, where the input is a sentence of arbitrary length and the output is another sentence of a different, also-arbitrary length. Translation, summarization, dialogue — all are sequence-to-sequence, and none fit a fixed-size box.

Recurrent networks could consume a variable-length sequence, but the standard setups produced one output per input step — no good when the output length differs from the input and the alignment isn't one-to-one. The field needed a way to map any sequence to any other sequence, learned end to end from data. That's the gap this paper closed.

The solution splits the job across two recurrent networks. An encoder reads the entire input sequence and boils it down to a single fixed-length vector — a compressed "meaning" of the whole sentence, often called the context vector. A decoder then takes that vector and unrolls it into the output sequence, one token at a time.

It's like a translator who listens to a full sentence, forms a complete mental impression of it, and only then starts speaking the translation — rather than translating word by word as they hear it. The bridge between the two languages is that one internal impression: the context vector.

The encoder is an LSTM — a recurrent network with gates designed to carry information across many steps without the gradients vanishing. It ingests the input tokens one by one, updating an internal hidden state each step. After the last token, that final hidden state is treated as the summary of the entire input: everything the decoder will get to know about what was said.

LSTMs mattered here because plain RNNs forget too fast — by the end of a long sentence, the influence of the early words has washed out. The gated memory of the LSTM was what made encoding a whole sentence into one usable vector even feasible.

The decoder is a second LSTM, initialized with the encoder's context vector as its starting memory. It generates the output autoregressively: predict the first output token, then feed that prediction back in to help predict the second, and so on — each new word conditioned on the context vector plus everything generated so far.

Special tokens frame the process: a start token kicks off decoding, and the decoder keeps going until it emits an end-of-sequence token, which is how a variable-length output naturally terminates itself. This "generate one token, feed it back" loop is exactly how modern language models still produce text — seq2seq is where that pattern went mainstream.

Two practical choices moved the needle a lot. First, depth: stacking several LSTM layers gave the model enough capacity to handle real translation. Second, and famously, reversing the input. Feeding the source sentence backwards improved results markedly.

Why would reversing help? It shortens the distance between related words. With the input reversed, the first source words sit close in time to the first target words the decoder must produce, so the LSTM has shorter dependency chains to learn at the crucial start of the sentence. A tiny change in data ordering, a surprisingly large gain — a reminder that in deep learning, the plumbing often matters as much as the theory.

The architecture's elegance is also its flaw. Everything about the input must pass through one fixed-size context vector. The whole model factorizes the output probability as a product of per-token predictions, but every one of them is conditioned on the same frozen summary c:

A 5-word sentence and a 50-word sentence get the exact same number of bytes to be remembered by.

For a short sentence, fine. For a long, information-dense one, that single vector becomes a straw through which a river must flow — one vector looks like it can't retain every detail. Other encoder-decoder models of the same year (Cho et al., Bahdanau et al.) did lose quality as sentences got longer; this paper's own deep LSTM, helped by reversing the source sentence, held up well on long ones.

This bottleneck was the clear, well-understood weakness — and naming it so precisely is part of the paper's legacy. It set up the very next breakthrough: what if the decoder didn't have to rely on one summary vector, but could look back at all the encoder's hidden states and focus on the relevant ones? That question is attention.

Beyond the bottleneck, seq2seq inherits the costs of recurrence. Because each step depends on the previous hidden state, the encoder and decoder must run sequentially — you can't process all tokens in parallel, which makes training on long sequences slow and hard to scale. And even with LSTMs, very long-range dependencies remain difficult.

Both limitations pointed the field forward. Attention fixed the bottleneck; a few years later the Transformer removed recurrence entirely, so a whole sequence could be processed in parallel — while keeping seq2seq's encoder-decoder skeleton. Seq2seq was a crucial rung on the ladder, not the top of it.

Seq2seq established the mental model that all of modern NLP runs on: cast a task as mapping an input sequence to an output sequence, learned end to end. Translation, summarization, question answering, dialogue, code generation — all are seq2seq problems, and this paper is where that framing was proven at scale on real machine translation.

Its exact ingredients live on. The encoder-decoder split survives in the Transformer and in T5. The autoregressive "generate a token, feed it back, stop at end-of-sequence" loop is exactly how GPT-style models write. And the bottleneck it exposed directly motivated attention — the mechanism that became the beating heart of the entire field.

The system in Sutskever, Vinyals and Le's paper is two deep LSTMs and a search procedure, trained on WMT'14 English-to-French. Nothing about either language is built in except the two vocabularies.

1. Fix the vocabularies. The 160,000 most frequent English words form the input vocabulary and the 80,000 most frequent French words form the output vocabulary. Every other word is replaced by a single unknown token. Each word maps to a learned 1,000-dimensional embedding.
2. Reverse the source, never the target. The English sentence a, b, c is fed in as c, b, a, while the French side keeps its normal order. The first source word now arrives last, immediately before the decoder writes the first target word.
3. Encode with a deep LSTM. A 4-layer LSTM with 1,000 cells per layer reads the reversed sentence one word per step. Its state after the last word is the fixed-size representation v of the whole sentence.
4. Decode with a second, separate LSTM. A different 4-layer LSTM with its own weights starts from v and works as a French language model conditioned on it. At each step it produces a softmax over the 80,000 output words, and the word it emits becomes its next input. Generation ends when it emits the end-of-sentence token. The authors note that two LSTMs instead of one add parameters at almost no extra compute.
5. Train for likelihood. The objective is the average log probability of the correct French sentence given the English one. Training used a cleaned 12M-pair subset of the WMT'14 data (348M French words, 304M English words). Weights were initialised uniformly in ±0.08. The optimizer was plain SGD without momentum at a learning rate of 0.7, halved every half epoch after the fifth epoch, for 7.5 epochs in total, with minibatches of 128 sentences. The gradient norm was clipped at 5, and sentences of similar length were batched together, which roughly doubled training speed.
6. Spread it over eight GPUs. Each of the four LSTM layers ran on its own GPU and the other four GPUs split the 80,000-way softmax. That reached about 6,300 words per second, and training took about ten days. The full model has 384M parameters, 64M of them recurrent connections.
7. Search for the output. Decoding is a left-to-right beam search. It keeps the B most probable partial translations, extends each with every word in the vocabulary, keeps the B best again, and sets aside any hypothesis that has emitted end-of-sentence as finished. A beam of 1 already worked well and a beam of 2 gave most of the benefit. The headline numbers use 12.
8. Or rescore a phrase-based system. In a second mode the LSTMs score each of the 1,000 best candidates produced by the phrase-based baseline. The final ranking takes an even average of the LSTM log probability and the baseline's own score.

What reversal changes, precisely

Reversal does not change the average distance between a source word and the target word it corresponds to. What it changes is the minimum: the first few source words end up a step or two from the first few target words, where before every pair was a whole sentence apart. The authors' explanation is that these short dependencies give backpropagation an early foothold between the two sentences, which then helps on the rest. They offer that as a hypothesis, not a proof.

Long sentences were not the failure

The paper reports that its LSTM handled long sentences well: BLEU showed no drop for sentences under about 35 words and only a small drop on the very longest. The authors credit the reversal for this. The well-known complaint that a single fixed vector cannot carry a long sentence came from Cho et al. and Bahdanau et al. in 2014, whose models did lose quality as sentences grew.

• 𝒮 — the training set of (target T, source S) sentence pairs; J is the average log-likelihood the SGD steps climb.
• p(T | S) — the decoder's probability of the whole French sentence, a product of one softmax per word, each conditioned on v and the words already written.
• T̂ — the translation the system outputs. The exact argmax would search every possible sentence, so a beam of width B approximates it.
• g — the minibatch gradient, divided by the batch size of 128 before the norm is checked.
• 5 — the clipping threshold. Below it the gradient passes unchanged; above it the direction is kept and the length is cut to 5.

At the extremes: B = 1 is greedy decoding, which the paper found already close; as B grows the search approaches the true argmax, with diminishing returns after 2. A clipping threshold of infinity means no clipping, which leaves training exposed to the occasional exploding gradient that LSTMs still produce; a tiny threshold would make every step the same length and turn SGD into sign-following.

All results are on the WMT'14 English-to-French test set (newstest2014), scored with BLEU on tokenized output. The comparison point is a strong phrase-based statistical system, which the paper also used as a source of candidate translations for its rescoring mode.

The headline: a purely neural system, translating directly, beat a mature phrase-based system by 1.5 BLEU on a large task, even though every word outside its 80,000-word output vocabulary came out as an unknown token and was scored as wrong.

The ablations are the instructive part. Reversing the source cut test perplexity from 5.8 to 4.7 and was worth more than four BLEU points for a single model. Depth mattered too: each extra LSTM layer reduced perplexity by nearly 10%. Search mattered least: a beam of 1 on the ensemble already reached 33.00.

The paper also found that its model did not fall apart on long sentences, which is the opposite of what is often attributed to it. And a two-dimensional PCA projection of the sentence vectors showed them grouping by meaning, sensitive to word order and nearly unchanged between active and passive voice.

The negative result is the gap that remained. Direct translation stayed 2.2 BLEU short of the best WMT'14 entry, and the best neural figure, 36.5, still needed the phrase-based system to propose the candidates.

The paper is candid that its most useful trick — reversing the source sentence — is an empirical finding it cannot fully explain, rather than dressing it up as a principle. It also reports something that surprised its own authors: the LSTM did well on long sentences, helped by that reversal. The long-sentence weakness of squeezing a whole input into one fixed vector was measured in other work the same year (Cho et al. on RNN encoder-decoders, and the Bahdanau et al. attention paper that answered it), not in this one.

It also depends on beam search at decoding time to get its headline numbers, and on an ensemble of models — so the reported figure is not what a single greedy model does, and the paper says so.

How it held up: the encoder-decoder frame survived completely, and the deeper claim — that a general sequence model trained end to end beats a hand-assembled pipeline of specialists — is now so thoroughly accepted that it is hard to see as a claim at all. The fixed vector did not last a year: Bahdanau and colleagues let the decoder look back at every encoder state through a learned weighting, which removed the bottleneck and introduced attention as a component. Rung 2's contribution is to notice that once you have that component, the recurrence it was bolted onto is no longer doing anything essential.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

The architecture itself: what it replaced, how position is encoded once recurrence is gone, and what it turns out to have learned inside.

1. 1Seq2Seq2014
2. 2Attention Is All You Need2017
3. 3RoPE2021
4. 4Concept Circuits2026

All 110 papers, in 19 ladders →