Llama 2,
explained.

Rung 1, so the comparison is against the frontier practice of the time, which was to release a capable model as an API and a capability report. This rung publishes the weights, in three sizes, with a licence permitting commercial use, together with an unusually detailed account of how the chat version was aligned. The predecessor family had leaked rather than shipped and carried a research-only licence; the change here is that building on it was permitted.

The architecture is worth reading as a summary of what had settled by then, because every choice is a rung from another ladder: RMSNorm in place of LayerNorm, a gated feed-forward activation, rotary positions, and — in the largest model — grouped-query attention, which lets several query heads share one set of keys and values and so shrinks the cache that dominates serving memory. Pretraining runs to 2 trillion tokens, which is the number the next rung multiplies.

The post-training contribution is the part that is genuinely new here, and it is a design response to a problem rung 1 of the post-training ladder left. A single reward model has to score helpfulness and harmlessness together, which forces it to encode a trade-off at every point. This release trains two separate reward models and combines them, so the tension lives somewhere explicit and adjustable instead of being baked into one learned function.

By mid-2023, the most capable models — GPT-4, Claude, PaLM — were all accessed the same way: through a paid API, weights hidden. That's fine for using a model, but it walls off almost everything else. You can't inspect what it learned, run it offline or on private data, fine-tune it deeply, or study its failure modes. Researchers and builders were renting intelligence they could never own.

The first Llama had cracked the door — its weights leaked and the community erupted with fine-tunes — but it was research-only. Llama 2's move was deliberate: release a strong, chat-aligned family with a license that permits commercial use. Not a leak, a policy. The bet was that an open foundation would unleash more innovation than any single lab could produce alone.

Llama 2 is a family, not a single model. It ships in three sizes — 7B, 13B, and 70B parameters — each in two flavors: a raw pretrained base (a next-word predictor, like GPT-3) and a chat version (Llama 2-Chat) aligned to hold a conversation and follow instructions.

The sizes matter. A 7B model runs on a single consumer GPU; the 70B rivals much larger closed models on many tasks. That range is what made Llama 2 a platform: hobbyists ran the 7B on laptops, startups fine-tuned the 13B, and serious deployments reached for the 70B — all from the same released weights.

The base models are trained on roughly 2 trillion tokens of publicly available text — 40% more data than the first Llama — with a longer 4,096-token context window. The objective is the same humble one every GPT-style model uses: predict the next token. No magic, just a lot of high-quality data and compute, with deliberate up-weighting of the most factual sources.

This is the expensive, once-per-model stage that produces raw capability. Everything downstream — the chat tuning, the community's thousands of fine-tunes — starts from these pretrained weights. Meta's contribution here is less a new idea than a well-executed, openly-shared one: a clean, strong base others could build on.

Llama 2 is a standard Transformer decoder with three now-common refinements: RMSNorm for cheaper, stabler normalization, SwiGLU activations in the feed-forward layers, and RoPE (rotary positional embeddings) that encode position by rotating query/key vectors. None are new to Llama 2, but together they define the modern open-model recipe.

The 70B adds a genuinely important inference trick: grouped-query attention (GQA). Normally each attention head keeps its own keys and values, and during generation those pile up in a memory-hungry KV cache. GQA lets several query heads share one key/value head — shrinking the cache and the memory bandwidth per token, so the big model generates faster with almost no quality cost.

A pretrained base only predicts text; making it a good chat assistant takes alignment. Llama 2-Chat follows the RLHF playbook InstructGPT established, with a distinctive twist: it trains two separate reward models — one scoring helpfulness, one scoring safety — so the two goals can be balanced explicitly rather than blurred into a single score.

Why two models rather than one blended score? Because a single reward is a soft tradeoff — a big helpfulness gain can outweigh a safety loss, and an unsafe-but-impressive answer wins. Two models let safety be a hard floor instead of a bargaining chip:

Gate on safety first, then maximize helpfulness among what's left — so the unsafe answer never wins no matter how helpful it looks. The runnable version below shows the two picks diverge.

It also introduced Ghost Attention (GAtt), a training trick that helps the model keep obeying a system instruction ("always answer in French") across many turns instead of forgetting it after a couple of messages. Small, but it's the kind of multi-turn control real assistants need.

Llama 2-Chat outperformed essentially every other open chat model of its day, and on human evaluations of helpfulness it was competitive with closed models like ChatGPT (GPT-3.5) — while being fully downloadable. On safety benchmarks it produced fewer harmful responses than many peers, a direct payoff of the dedicated safety reward model.

The headline wasn't a single benchmark — it was the combination: near-frontier chat quality, credible safety, and open weights, all at once. That mix is what turned Llama 2 from "a good model" into "the base everyone builds on."

Llama 2 is not GPT-4. It hallucinates, its knowledge is frozen at training time, the 4K context is short by today's standards, and it is heavily English-centric — weaker in other languages. As a chat model it can be overcautious, refusing benign requests because the safety training erred toward caution.

And "open" deserves an asterisk. The license is permissive but not true open source: companies with over 700 million monthly active users need a special license, and you may not use Llama 2 to train competing models. It's best called open weights — you get the model, under terms — rather than open source in the strict sense.

Llama 2 lit the fuse on the open-model ecosystem. Because the weights were free and commercially usable, a Cambrian explosion followed: quantized versions that run on laptops, thousands of domain fine-tunes, tooling like llama.cpp and Ollama, and a research community that could finally poke at a real, strong model. Much of today's local and open-source AI traces directly to this release.

It also normalized the pattern of shipping a model family with a proper technical report — architecture, data, and a candid safety section — rather than a closed black box. Later open models (Mistral, Llama 3, and beyond) build on the template Llama 2 set: strong pretraining, RLHF alignment, and open weights as a strategy, not an accident.

The safety work's cost is reported rather than hidden: the aligned chat model over-refuses, declining benign requests that pattern-match to something disallowed, and the paper shows examples. Read next to the principles-based approach, it is a clean illustration of why refusing with a reason became the goal — a model trained on preferences alone learns that the safest output is the shortest one.

Two limits on the openness are worth stating precisely, because the word does a lot of unexamined work. The licence is not an open-source licence by the usual definition — it carries an acceptable-use policy and a clause restricting the very largest deployers — and the training data mixture is not disclosed, so the artefact is reproducible in architecture and not in provenance. "Open weights" is the accurate term and "open source" is not.

How it held up: it created the ecosystem this ladder describes. Nearly everything downstream — the quantisation toolchains, the local inference runtimes, the single-GPU fine-tuning wave, the thousands of derived models — dates from this release being usable, and the secondary effect was to establish that a published account of alignment methodology is something a lab can do without losing its position. Grouped-query attention became standard. The two-reward-model design was largely superseded by simpler preference methods, but the diagnosis it responded to did not go away.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

Five releases you can actually download, in the order each one answered the last. This is where "what does a frontier recipe look like" is documented in public.

1. 1Llama 22023
2. 2Llama 32024
3. 3Qwen32025
4. 4Kimi K22025
5. 5Kimi K32026

All 110 papers, in 19 ladders →