Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

Handbooks  /  Encoder vs Decoder Models
AI~9 min readComparison
Head to Head

Encoder vs decoder: which way does attention flow?

EncodervsDecoder

Both are made of the same Transformer blocks. The whole difference is one design choice — whether a token is allowed to look at the tokens after it. That single rule splits the family into models that understand (BERT) and models that generate (GPT), and decides what each is good for.

01

The one distinction that decides everything

An encoder uses bidirectional attention: every token can attend to every other token, before and after it, so the model reads the whole input at once and builds a representation of it. A decoder uses causal attention: a token can only attend to tokens before it, so the model generates left to right, one token at a time, never peeking at the future it hasn't written yet. Encoders understand; decoders generate.

→ The rule

If the task is to understand a fixed input — classify it, tag it, embed it, retrieve against it — an encoder's bidirectional view is the right tool. If the task is to produce text — chat, code, summaries, agent actions — a decoder's left-to-right generation is what you need.

02

Head to head

DimensionEncoder (BERT-style)Decoder (GPT-style)
AttentionBidirectional — sees the whole inputCausal — sees only tokens before it
Training objectiveMasked language modeling (fill in blanks)Next-token prediction
Primary outputRepresentations / embeddings / labelsGenerated text, token by token
Best atClassification, NER, retrieval, rerankingGeneration, chat, code, agents
Can it generate text?Not naturally — it's built to understandYes — that's the whole point
In-context / few-shot learningWeakStrong — the basis of prompting
Typical examplesBERT, RoBERTa, DeBERTa, embedding modelsGPT, Llama, Mistral, most chat LLMs
Inference cost patternOne forward pass over the inputOne forward pass per generated token
03

When to use each

Reach for an encoder

  • Text classification, sentiment, intent detection
  • Named-entity recognition and token tagging
  • Embeddings for semantic search and retrieval (the "R" in RAG)
  • Reranking retrieved candidates by relevance
  • Any task where you need a representation of a fixed input, not new text

Reach for a decoder

  • Chat assistants and open-ended generation
  • Code completion and code assistants
  • Summarization, rewriting, translation via prompting
  • Agents that plan and call tools
  • Anything few-shot: give examples in the prompt, get behavior
04

Why decoders took the spotlight — but encoders never left

The headline models of the LLM era are all decoder-only, and for good reason: next-token prediction is a perfect fit for generation and chat, it unlocked in-context (few-shot) learning, and decoder-only stacks scaled cleanly to enormous sizes. So when people say "LLM" today, they usually mean a decoder. But that's a spotlight effect, not the whole stage. Encoders quietly run some of the highest-volume workloads in production: every embedding for semantic search, every reranker, and a huge share of text classification is an encoder, because those tasks want a representation of the input, not a stream of generated tokens — and a bidirectional read is simply better at building one.

→ The cheap default

Building something that reads and sorts existing text (search, classify, route, tag)? An encoder is usually smaller, faster and better. Building something that writes text or takes actions? A decoder. Many real systems use both: an encoder to retrieve, a decoder to answer.

05

The encoder-decoder middle ground

There's a third design: encoder-decoder (sequence-to-sequence) models like T5 and BART. The encoder reads the whole source with bidirectional attention, and a separate decoder generates the target conditioned on that encoding. This shines for tasks that transform one full sequence into another — translation and summarization are the textbook cases, where you want a complete understanding of the source before you start writing the output.

So why isn't everything encoder-decoder? Because modern decoder-only LLMs turned out to handle translation and summarization very well by just conditioning on the source in the prompt, without a dedicated encoder — and a single decoder-only stack is simpler to scale and serve than a two-part architecture. Encoder-decoder models are still excellent at what they do; they're just less common now because a big decoder often does the same job with one architecture instead of two.

→ The trade you're actually making

Encoder-decoder buys a clean split between "understand the source" and "generate the target," which suits seq2seq tasks — at the cost of a more complex architecture that a large decoder-only model can often replace.

06

A worked scenario: a support-ticket classifier vs a support chatbot

You need to route incoming support tickets to the right team. That's an understanding problem: read the whole ticket, output one of a dozen labels. An encoder (a fine-tuned BERT-style model, or an embedding model plus a classifier) is the right call — it's small, fast, cheap to run at high volume, and its bidirectional read of the full ticket gives it every clue at once. Reaching for a giant generative LLM here is slower and more expensive for no accuracy gain.

Now you want a chatbot that actually answers the customer. That's a generation problem — and a decoder-only LLM is the tool, generating a fluent reply token by token, ideally grounded by retrieved docs. Notice the retrieval step in that sentence: the search that finds the right docs is almost certainly powered by an encoder embedding model. The mature system uses both — encoder to find, decoder to answer — which is exactly why "encoder vs decoder" is less a fight than a division of labor.

→ The pattern generalizes

Read-and-sort → encoder. Write-and-act → decoder. Transform-one-sequence-into-another → encoder-decoder (or a capable decoder). Real products often chain an encoder and a decoder together.

The picture

The picture: the attention mask is the whole difference

encoder: bidirectional decoder: causal IlovethisfilmIlovethisfilm IlovethisfilmIlovethisfilm 16 of 16 pairs 10 of 16 pairs row = token being built, column = token it may look at
Same four tokens, same Transformer block, one different mask. In the encoder every token reads the whole sentence, so “love” already knows the sentence ends in “film”. In the decoder each row stops at the diagonal: “love” sees only “I” and itself. That is what lets a decoder generate left to right. When a token is generated, nothing to its right exists yet. Over n tokens the encoder computes n² pairs and the decoder n(n+1)/2: for 512 tokens, 262,144 against 131,328.
By the numbers

Worked example: classifying 50 million tickets both ways

Take the page’s quiz case: 50 million support tickets, 12 categories, about 300 tokens per ticket. Compare a fine-tuned BERT-base encoder (110M parameters) with an illustrative 7B decoder prompted to name the category. Estimate compute with the standard rule of thumb: a forward pass costs about 2 × parameters FLOPs per token.

StepEncoder (BERT-base)Decoder (7B, prompted)
Tokens read per ticket300550: the ticket plus 250 of instructions and the 12 label names
Sequential passes11 prefill + 4 decode steps (a label of up to 3 tokens, then the end token)
Tokens processed300554
Compute per ticket (≈ 2 × params × tokens)66 GFLOP7,756 GFLOP
Compute for 50M tickets3.3 × 10¹⁸ FLOP3.9 × 10²⁰ FLOP, about 117× more
What comes outA probability for each of the 12 labelsText to parse, which can be a label that is not on the list

Most of the 117× comes from model size (7B against 110M is about 64×). Almost all the rest comes from the prompt the decoder needs to know what the labels are. The 4 decode steps add little compute, but each one is a separate sequential pass, which adds latency per ticket. The last row matters as much as the compute. The encoder returns a score for every class, so you can set a threshold, route low-confidence tickets to a human and measure calibration. The decoder returns a string, so you need parsing and a rule for off-list answers. A decoder can still win this job, for example when there are no labelled examples to fine-tune on. Few-shot prompting works on day one, while fine-tuning an encoder needs labelled tickets first. A common path is to start with the prompted decoder, use it to label data, then train the encoder for the high-volume path.

Trade-offs

Cost and trade-offs by architecture

For an input of n tokens and an output of m tokens. Time is the passes and attention work; Space is what must be held in memory.

CaseTimeSpaceWhy
Encoder-onlyOne pass; attention over n² pairsActivations for one pass; nothing kept afterwardsFast, parallel, one-shot. BERT-style models cap n at 512 tokens.
Decoder-onlyPrefill of n tokens, then m sequential stepsA KV cache that grows with n + mThe cache turns each step into one new token’s work. Latency grows with output length.
Encoder-decoderOne encoder pass over n, then m decoder steps that also attend to the encoder outputEncoder output plus the decoder’s KV cacheA clean source/target split for translation and summarisation, at the cost of two stacks to train and serve.
Bi-encoder retrievalDocuments embedded once, offline; one short pass per queryOne vector per document in an indexScales to millions of documents. Query and document never attend to each other.
Cross-encoder rerankingOne pass per (query, document) pairNone beyond the passMore accurate, because the pair attends jointly, but 100 candidates cost 100 passes. Use it on a shortlist only.

The KV cache exists only on the decoder side, for a reason the mask explains. In a causal stack a token’s keys and values never change once computed, because later tokens cannot affect earlier ones, so they can be stored and reused at every step. In a bidirectional stack, adding a token would change every earlier token’s representation, so there is nothing safe to cache.

Watch for

Common mistakes with encoders and decoders

  • Using raw BERT vectors as sentence embeddings

    A pretrained BERT was trained to fill in masked words, not to put similar sentences close together. Averaging its token vectors, or taking the [CLS] vector, gives poor similarity scores. For search, use an embedding model trained contrastively on sentence pairs, which is what the Sentence-BERT line of work showed.

  • Silent truncation past the length limit

    A BERT-style encoder stops at 512 tokens, and most pipelines cut the rest without an error. A long ticket whose key sentence is at the end gets classified on its opening pleasantries. Chunk long inputs and combine the chunk scores, or pick a long-context encoder, and log how often inputs exceed the limit.

  • Free-text output for a closed label set

    Prompting a decoder to “reply with the category” works until it replies with a synonym, a sentence or a category that does not exist. If you use a decoder for classification, constrain the output to the label set, or read the probabilities of the label tokens instead of parsing text.

  • Mixing embedding models between index and query

    Vectors from two different encoders live in different spaces, even when they have the same dimension. Upgrading the query encoder without re-embedding the corpus breaks retrieval silently: scores look normal and results get worse. Re-embed everything, or run both indexes side by side during the move.

  • Thinking a causal model cannot see the whole prompt

    Causal attention only stops a token from seeing tokens after it. The last token of the prompt sees every token before it, which is why decoder-based embedding models take the last token’s state. What a decoder lacks is the earlier tokens seeing the later ones, which is exactly what a bidirectional encoder adds.

In the wild

Where this shows up in production and interviews

  • A modern RAG stack uses all of it

    A bi-encoder embeds the corpus and the query, a cross-encoder reranks the top candidates, and a decoder writes the answer from the winners. Each stage uses the architecture that suits its job: cheap retrieval over millions, accurate scoring over a shortlist, generation once.

  • High-volume filters in the request path

    Moderation, spam, intent routing and PII detection run on every message, often before a generative model sees it. These are classification jobs with tight latency budgets, which is encoder territory. The cost gap in the worked example is paid on every request.

  • Speech and translation

    Encoder-decoder models are still common wherever the input is a whole sequence in a different form from the output. OpenAI’s Whisper, for example, is an encoder-decoder Transformer: the encoder reads the audio and the decoder writes the transcript.

  • Interviews

    Expect these: “Why can’t BERT generate text?” (it was trained to fill blanks with both sides visible, and has no left-to-right objective). “Draw the causal mask” (the lower triangle above). “Why do only decoders have a KV cache?” (earlier keys and values never change under a causal mask). “Would you use GPT to classify a billion items?” (price it, as in the worked example).

Your call

Which would you pick?

Three situations. Pick the side you'd actually build — the explanation follows.

You must classify 50 million support tickets into a fixed set of categories, as fast and cheaply as possible.

You are building an assistant that writes replies and calls tools based on what it decides to do next.

Someone frames the choice as "encoders are outdated, decoders won."

Frequently asked

Quick answers

What's the difference between an encoder and a decoder model?

An encoder reads the whole input at once with bidirectional attention and builds representations — ideal for classification, NER and embeddings (BERT). A decoder generates one token at a time with causal attention — ideal for generation, chat and code (GPT). Encoder-decoder models like T5 do both.

Is GPT an encoder or a decoder?

Decoder-only. GPT, Llama, Mistral and most chat LLMs use causal attention and next-token prediction. BERT and its relatives are encoder-only, trained with masked language modeling to build representations.

Why did decoder-only models take over?

Generation, few-shot learning and chat are natural fits for next-token prediction, and decoders scaled cleanly. But encoders still dominate retrieval, semantic search, reranking and classification, because those tasks want a representation of the input, not generated text.

What are encoder-decoder models used for?

Sequence-to-sequence tasks — translation and summarization are the classics. The encoder reads the whole source, the decoder generates the target. Modern decoder-only LLMs now handle many of these too, which is why pure encoder-decoder models are less common.

▶  Watch it explained

Encoder vs Decoder: a reader, or a writer (BERT vs GPT)

Encoder vs Decoder Models · Vibe Engines · 2026
Finished this one? 0 / 209 Handbooks done
NextMoE vs Dense Models Next on the “How models actually work” ladder · rung 5 of 11

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Handbooks