Seq2Seq,
explained.

Classic neural networks assume a fixed shape: a fixed number of inputs, a fixed number of outputs. That's fine for classifying a photo into one of 1,000 labels. It falls apart for language, where the input is a sentence of arbitrary length and the output is another sentence of a different, also-arbitrary length. Translation, summarization, dialogue — all are sequence-to-sequence, and none fit a fixed-size box.

Recurrent networks could consume a variable-length sequence, but the standard setups produced one output per input step — no good when the output length differs from the input and the alignment isn't one-to-one. The field needed a way to map any sequence to any other sequence, learned end to end from data. That's the gap this paper closed.

The solution splits the job across two recurrent networks. An encoder reads the entire input sequence and boils it down to a single fixed-length vector — a compressed "meaning" of the whole sentence, often called the context vector. A decoder then takes that vector and unrolls it into the output sequence, one token at a time.

It's like a translator who listens to a full sentence, forms a complete mental impression of it, and only then starts speaking the translation — rather than translating word by word as they hear it. The bridge between the two languages is that one internal impression: the context vector.

The encoder is an LSTM — a recurrent network with gates designed to carry information across many steps without the gradients vanishing. It ingests the input tokens one by one, updating an internal hidden state each step. After the last token, that final hidden state is treated as the summary of the entire input: everything the decoder will get to know about what was said.

LSTMs mattered here because plain RNNs forget too fast — by the end of a long sentence, the influence of the early words has washed out. The gated memory of the LSTM was what made encoding a whole sentence into one usable vector even feasible.

The decoder is a second LSTM, initialized with the encoder's context vector as its starting memory. It generates the output autoregressively: predict the first output token, then feed that prediction back in to help predict the second, and so on — each new word conditioned on the context vector plus everything generated so far.

Special tokens frame the process: a start token kicks off decoding, and the decoder keeps going until it emits an end-of-sequence token, which is how a variable-length output naturally terminates itself. This "generate one token, feed it back" loop is exactly how modern language models still produce text — seq2seq is where that pattern went mainstream.

Two practical choices moved the needle a lot. First, depth: stacking several LSTM layers gave the model enough capacity to handle real translation. Second, and famously, reversing the input. Feeding the source sentence backwards improved results markedly.

Why would reversing help? It shortens the distance between related words. With the input reversed, the first source words sit close in time to the first target words the decoder must produce, so the LSTM has shorter dependency chains to learn at the crucial start of the sentence. A tiny change in data ordering, a surprisingly large gain — a reminder that in deep learning, the plumbing often matters as much as the theory.

The architecture's elegance is also its flaw. Everything about the input must pass through one fixed-size context vector. For a short sentence, fine. For a long, information-dense one, that single vector becomes a straw through which a river must flow — it simply can't retain every detail, and translation quality visibly degrades as sentences get longer.

This bottleneck was the clear, well-understood weakness — and naming it so precisely is part of the paper's legacy. It set up the very next breakthrough: what if the decoder didn't have to rely on one summary vector, but could look back at all the encoder's hidden states and focus on the relevant ones? That question is attention.

Beyond the bottleneck, seq2seq inherits the costs of recurrence. Because each step depends on the previous hidden state, the encoder and decoder must run sequentially — you can't process all tokens in parallel, which makes training on long sequences slow and hard to scale. And even with LSTMs, very long-range dependencies remain difficult.

Both limitations pointed the field forward. Attention fixed the bottleneck; a few years later the Transformer removed recurrence entirely, so a whole sequence could be processed in parallel — while keeping seq2seq's encoder-decoder skeleton. Seq2seq was a crucial rung on the ladder, not the top of it.

Seq2seq established the mental model that all of modern NLP runs on: cast a task as mapping an input sequence to an output sequence, learned end to end. Translation, summarization, question answering, dialogue, code generation — all are seq2seq problems, and this paper is where that framing was proven at scale on real machine translation.

Its exact ingredients live on. The encoder-decoder split survives in the Transformer and in T5. The autoregressive "generate a token, feed it back, stop at end-of-sequence" loop is exactly how GPT-style models write. And the bottleneck it exposed directly motivated attention — the mechanism that became the beating heart of the entire field.