Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path →
Encoder vs decoder: which way does attention flow?
Both are made of the same Transformer blocks. The whole difference is one design choice — whether a token is allowed to look at the tokens after it. That single rule splits the family into models that understand (BERT) and models that generate (GPT), and decides what each is good for.
The one distinction that decides everything
An encoder uses bidirectional attention: every token can attend to every other token, before and after it, so the model reads the whole input at once and builds a representation of it. A decoder uses causal attention: a token can only attend to tokens before it, so the model generates left to right, one token at a time, never peeking at the future it hasn't written yet. Encoders understand; decoders generate.
If the task is to understand a fixed input — classify it, tag it, embed it, retrieve against it — an encoder's bidirectional view is the right tool. If the task is to produce text — chat, code, summaries, agent actions — a decoder's left-to-right generation is what you need.
Head to head
| Dimension | Encoder (BERT-style) | Decoder (GPT-style) |
|---|---|---|
| Attention | Bidirectional — sees the whole input | Causal — sees only tokens before it |
| Training objective | Masked language modeling (fill in blanks) | Next-token prediction |
| Primary output | Representations / embeddings / labels | Generated text, token by token |
| Best at | Classification, NER, retrieval, reranking | Generation, chat, code, agents |
| Can it generate text? | Not naturally — it's built to understand | Yes — that's the whole point |
| In-context / few-shot learning | Weak | Strong — the basis of prompting |
| Typical examples | BERT, RoBERTa, DeBERTa, embedding models | GPT, Llama, Mistral, most chat LLMs |
| Inference cost pattern | One forward pass over the input | One forward pass per generated token |
When to use each
Reach for an encoder
- Text classification, sentiment, intent detection
- Named-entity recognition and token tagging
- Embeddings for semantic search and retrieval (the "R" in RAG)
- Reranking retrieved candidates by relevance
- Any task where you need a representation of a fixed input, not new text
Reach for a decoder
- Chat assistants and open-ended generation
- Code completion and code assistants
- Summarization, rewriting, translation via prompting
- Agents that plan and call tools
- Anything few-shot: give examples in the prompt, get behavior
Why decoders took the spotlight — but encoders never left
The headline models of the LLM era are all decoder-only, and for good reason: next-token prediction is a perfect fit for generation and chat, it unlocked in-context (few-shot) learning, and decoder-only stacks scaled cleanly to enormous sizes. So when people say "LLM" today, they usually mean a decoder. But that's a spotlight effect, not the whole stage. Encoders quietly run some of the highest-volume workloads in production: every embedding for semantic search, every reranker, and a huge share of text classification is an encoder, because those tasks want a representation of the input, not a stream of generated tokens — and a bidirectional read is simply better at building one.
Building something that reads and sorts existing text (search, classify, route, tag)? An encoder is usually smaller, faster and better. Building something that writes text or takes actions? A decoder. Many real systems use both: an encoder to retrieve, a decoder to answer.
The encoder-decoder middle ground
There's a third design: encoder-decoder (sequence-to-sequence) models like T5 and BART. The encoder reads the whole source with bidirectional attention, and a separate decoder generates the target conditioned on that encoding. This shines for tasks that transform one full sequence into another — translation and summarization are the textbook cases, where you want a complete understanding of the source before you start writing the output.
So why isn't everything encoder-decoder? Because modern decoder-only LLMs turned out to handle translation and summarization very well by just conditioning on the source in the prompt, without a dedicated encoder — and a single decoder-only stack is simpler to scale and serve than a two-part architecture. Encoder-decoder models are still excellent at what they do; they're just less common now because a big decoder often does the same job with one architecture instead of two.
Encoder-decoder buys a clean split between "understand the source" and "generate the target," which suits seq2seq tasks — at the cost of a more complex architecture that a large decoder-only model can often replace.
A worked scenario: a support-ticket classifier vs a support chatbot
You need to route incoming support tickets to the right team. That's an understanding problem: read the whole ticket, output one of a dozen labels. An encoder (a fine-tuned BERT-style model, or an embedding model plus a classifier) is the right call — it's small, fast, cheap to run at high volume, and its bidirectional read of the full ticket gives it every clue at once. Reaching for a giant generative LLM here is slower and more expensive for no accuracy gain.
Now you want a chatbot that actually answers the customer. That's a generation problem — and a decoder-only LLM is the tool, generating a fluent reply token by token, ideally grounded by retrieved docs. Notice the retrieval step in that sentence: the search that finds the right docs is almost certainly powered by an encoder embedding model. The mature system uses both — encoder to find, decoder to answer — which is exactly why "encoder vs decoder" is less a fight than a division of labor.
Read-and-sort → encoder. Write-and-act → decoder. Transform-one-sequence-into-another → encoder-decoder (or a capable decoder). Real products often chain an encoder and a decoder together.
The picture: the attention mask is the whole difference
n tokens the encoder computes n² pairs and the decoder n(n+1)/2: for 512 tokens, 262,144 against 131,328.Worked example: classifying 50 million tickets both ways
Take the page’s quiz case: 50 million support tickets, 12 categories, about 300 tokens per ticket. Compare a fine-tuned BERT-base encoder (110M parameters) with an illustrative 7B decoder prompted to name the category. Estimate compute with the standard rule of thumb: a forward pass costs about 2 × parameters FLOPs per token.
| Step | Encoder (BERT-base) | Decoder (7B, prompted) |
|---|---|---|
| Tokens read per ticket | 300 | 550: the ticket plus 250 of instructions and the 12 label names |
| Sequential passes | 1 | 1 prefill + 4 decode steps (a label of up to 3 tokens, then the end token) |
| Tokens processed | 300 | 554 |
| Compute per ticket (≈ 2 × params × tokens) | 66 GFLOP | 7,756 GFLOP |
| Compute for 50M tickets | 3.3 × 10¹⁸ FLOP | 3.9 × 10²⁰ FLOP, about 117× more |
| What comes out | A probability for each of the 12 labels | Text to parse, which can be a label that is not on the list |
Most of the 117× comes from model size (7B against 110M is about 64×). Almost all the rest comes from the prompt the decoder needs to know what the labels are. The 4 decode steps add little compute, but each one is a separate sequential pass, which adds latency per ticket. The last row matters as much as the compute. The encoder returns a score for every class, so you can set a threshold, route low-confidence tickets to a human and measure calibration. The decoder returns a string, so you need parsing and a rule for off-list answers. A decoder can still win this job, for example when there are no labelled examples to fine-tune on. Few-shot prompting works on day one, while fine-tuning an encoder needs labelled tickets first. A common path is to start with the prompted decoder, use it to label data, then train the encoder for the high-volume path.
Cost and trade-offs by architecture
For an input of n tokens and an output of m tokens. Time is the passes and attention work; Space is what must be held in memory.
| Case | Time | Space | Why |
|---|---|---|---|
| Encoder-only | One pass; attention over n² pairs | Activations for one pass; nothing kept afterwards | Fast, parallel, one-shot. BERT-style models cap n at 512 tokens. |
| Decoder-only | Prefill of n tokens, then m sequential steps | A KV cache that grows with n + m | The cache turns each step into one new token’s work. Latency grows with output length. |
| Encoder-decoder | One encoder pass over n, then m decoder steps that also attend to the encoder output | Encoder output plus the decoder’s KV cache | A clean source/target split for translation and summarisation, at the cost of two stacks to train and serve. |
| Bi-encoder retrieval | Documents embedded once, offline; one short pass per query | One vector per document in an index | Scales to millions of documents. Query and document never attend to each other. |
| Cross-encoder reranking | One pass per (query, document) pair | None beyond the pass | More accurate, because the pair attends jointly, but 100 candidates cost 100 passes. Use it on a shortlist only. |
The KV cache exists only on the decoder side, for a reason the mask explains. In a causal stack a token’s keys and values never change once computed, because later tokens cannot affect earlier ones, so they can be stored and reused at every step. In a bidirectional stack, adding a token would change every earlier token’s representation, so there is nothing safe to cache.
Common mistakes with encoders and decoders
- Using raw BERT vectors as sentence embeddings
A pretrained BERT was trained to fill in masked words, not to put similar sentences close together. Averaging its token vectors, or taking the
[CLS]vector, gives poor similarity scores. For search, use an embedding model trained contrastively on sentence pairs, which is what the Sentence-BERT line of work showed. - Silent truncation past the length limit
A BERT-style encoder stops at 512 tokens, and most pipelines cut the rest without an error. A long ticket whose key sentence is at the end gets classified on its opening pleasantries. Chunk long inputs and combine the chunk scores, or pick a long-context encoder, and log how often inputs exceed the limit.
- Free-text output for a closed label set
Prompting a decoder to “reply with the category” works until it replies with a synonym, a sentence or a category that does not exist. If you use a decoder for classification, constrain the output to the label set, or read the probabilities of the label tokens instead of parsing text.
- Mixing embedding models between index and query
Vectors from two different encoders live in different spaces, even when they have the same dimension. Upgrading the query encoder without re-embedding the corpus breaks retrieval silently: scores look normal and results get worse. Re-embed everything, or run both indexes side by side during the move.
- Thinking a causal model cannot see the whole prompt
Causal attention only stops a token from seeing tokens after it. The last token of the prompt sees every token before it, which is why decoder-based embedding models take the last token’s state. What a decoder lacks is the earlier tokens seeing the later ones, which is exactly what a bidirectional encoder adds.
Where this shows up in production and interviews
- A modern RAG stack uses all of it
A bi-encoder embeds the corpus and the query, a cross-encoder reranks the top candidates, and a decoder writes the answer from the winners. Each stage uses the architecture that suits its job: cheap retrieval over millions, accurate scoring over a shortlist, generation once.
- High-volume filters in the request path
Moderation, spam, intent routing and PII detection run on every message, often before a generative model sees it. These are classification jobs with tight latency budgets, which is encoder territory. The cost gap in the worked example is paid on every request.
- Speech and translation
Encoder-decoder models are still common wherever the input is a whole sequence in a different form from the output. OpenAI’s Whisper, for example, is an encoder-decoder Transformer: the encoder reads the audio and the decoder writes the transcript.
- Interviews
Expect these: “Why can’t BERT generate text?” (it was trained to fill blanks with both sides visible, and has no left-to-right objective). “Draw the causal mask” (the lower triangle above). “Why do only decoders have a KV cache?” (earlier keys and values never change under a causal mask). “Would you use GPT to classify a billion items?” (price it, as in the worked example).
Which would you pick?
Three situations. Pick the side you'd actually build — the explanation follows.
You must classify 50 million support tickets into a fixed set of categories, as fast and cheaply as possible.
This is read-and-sort: the output is a choice from a known set, not new text. An encoder produces a representation and a label in one pass, while a decoder generates the answer token by token — paying generation cost 50 million times to select from a list it already had.
You are building an assistant that writes replies and calls tools based on what it decides to do next.
Generating open-ended text and choosing actions at runtime is exactly the decoder case. An encoder with action heads can only pick from actions you enumerated in advance, and an encoder-decoder adds architectural complexity for a source-to-target split this task does not have.
Someone frames the choice as "encoders are outdated, decoders won."
The page's framing is read-and-sort versus write-and-act, and both jobs still exist: retrieval embeddings, classification and reranking are encoder work sitting inside products whose generation is decoder work. A capable decoder *can* do classification; that does not make it the sensible way to do it 50 million times.
Quick answers
What's the difference between an encoder and a decoder model?
An encoder reads the whole input at once with bidirectional attention and builds representations — ideal for classification, NER and embeddings (BERT). A decoder generates one token at a time with causal attention — ideal for generation, chat and code (GPT). Encoder-decoder models like T5 do both.
Is GPT an encoder or a decoder?
Decoder-only. GPT, Llama, Mistral and most chat LLMs use causal attention and next-token prediction. BERT and its relatives are encoder-only, trained with masked language modeling to build representations.
Why did decoder-only models take over?
Generation, few-shot learning and chat are natural fits for next-token prediction, and decoders scaled cleanly. But encoders still dominate retrieval, semantic search, reranking and classification, because those tasks want a representation of the input, not generated text.
What are encoder-decoder models used for?
Sequence-to-sequence tasks — translation and summarization are the classics. The encoder reads the whole source, the decoder generates the target. Modern decoder-only LLMs now handle many of these too, which is why pure encoder-decoder models are less common.
Encoder vs Decoder: a reader, or a writer (BERT vs GPT)
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The Prompting HandbookA friendly, hands-on field guide for everyday humans — learn the CRISP framework, spot bad prompts, practice with real recipes, play a drag-and-drop game, and test yourself with a quiz. No code required.Read →
- The Senior AI Engineer Interview Handbook60 questions across architecture, production incidents, agentic systems, RAG, evals, cost, safety, and leadership — what staff-level AI interviewers actually probe for.Read →
- The LLM Serving HandbookHow to serve large language models fast and cheap — prefill vs decode, the KV cache, continuous batching and PagedAttention (vLLM), quantization, speculative decoding, and the latency-vs-throughput tradeoffs that decide your inference bill.Read →
- The Transformers HandbookThe architecture behind every LLM, built up from scratch — tokens and embeddings, positional encoding, self-attention (query/key/value), multi-head attention, the transformer block (FFN, residuals, layer norm), and how a decoder-only model generates text autoregressively.Read →
- The Fine-Tuning HandbookWhen to fine-tune vs RAG vs prompting, full fine-tuning vs parameter-efficient methods, how LoRA and QLoRA make it cheap enough for a single GPU, why the data matters more than the method, catastrophic forgetting, and how to evaluate and decide.Read →
- The On-Device AI HandbookSmall models, everywhere: why 1–14B models got frontier-adjacent (distillation + textbook data), the napkin law of local inference (tokens/sec ≈ bandwidth ÷ bytes), quantization and the llama.cpp / GGUF / Ollama / MLX stack, the honest local-vs-cloud decision table, four hybrid patterns that ship, and the evals craft of choosing a small model that actually fits the device.Read →