Llama 2,
explained.

By mid-2023, the most capable models — GPT-4, Claude, PaLM — were all accessed the same way: through a paid API, weights hidden. That's fine for using a model, but it walls off almost everything else. You can't inspect what it learned, run it offline or on private data, fine-tune it deeply, or study its failure modes. Researchers and builders were renting intelligence they could never own.

The first Llama had cracked the door — its weights leaked and the community erupted with fine-tunes — but it was research-only. Llama 2's move was deliberate: release a strong, chat-aligned family with a license that permits commercial use. Not a leak, a policy. The bet was that an open foundation would unleash more innovation than any single lab could produce alone.

Llama 2 is a family, not a single model. It ships in three sizes — 7B, 13B, and 70B parameters — each in two flavors: a raw pretrained base (a next-word predictor, like GPT-3) and a chat version (Llama 2-Chat) aligned to hold a conversation and follow instructions.

The sizes matter. A 7B model runs on a single consumer GPU; the 70B rivals much larger closed models on many tasks. That range is what made Llama 2 a platform: hobbyists ran the 7B on laptops, startups fine-tuned the 13B, and serious deployments reached for the 70B — all from the same released weights.

The base models are trained on roughly 2 trillion tokens of publicly available text — 40% more data than the first Llama — with a longer 4,096-token context window. The objective is the same humble one every GPT-style model uses: predict the next token. No magic, just a lot of high-quality data and compute, with deliberate up-weighting of the most factual sources.

This is the expensive, once-per-model stage that produces raw capability. Everything downstream — the chat tuning, the community's thousands of fine-tunes — starts from these pretrained weights. Meta's contribution here is less a new idea than a well-executed, openly-shared one: a clean, strong base others could build on.

Llama 2 is a standard Transformer decoder with three now-common refinements: RMSNorm for cheaper, stabler normalization, SwiGLU activations in the feed-forward layers, and RoPE (rotary positional embeddings) that encode position by rotating query/key vectors. None are new to Llama 2, but together they define the modern open-model recipe.

The 70B adds a genuinely important inference trick: grouped-query attention (GQA). Normally each attention head keeps its own keys and values, and during generation those pile up in a memory-hungry KV cache. GQA lets several query heads share one key/value head — shrinking the cache and the memory bandwidth per token, so the big model generates faster with almost no quality cost.

A pretrained base only predicts text; making it a good chat assistant takes alignment. Llama 2-Chat follows the RLHF playbook InstructGPT established, with a distinctive twist: it trains two separate reward models — one scoring helpfulness, one scoring safety — so the two goals can be balanced explicitly rather than blurred into a single score.

It also introduced Ghost Attention (GAtt), a training trick that helps the model keep obeying a system instruction ("always answer in French") across many turns instead of forgetting it after a couple of messages. Small, but it's the kind of multi-turn control real assistants need.

Llama 2-Chat outperformed essentially every other open chat model of its day, and on human evaluations of helpfulness it was competitive with closed models like ChatGPT (GPT-3.5) — while being fully downloadable. On safety benchmarks it produced fewer harmful responses than many peers, a direct payoff of the dedicated safety reward model.

The headline wasn't a single benchmark — it was the combination: near-frontier chat quality, credible safety, and open weights, all at once. That mix is what turned Llama 2 from "a good model" into "the base everyone builds on."

Llama 2 is not GPT-4. It hallucinates, its knowledge is frozen at training time, the 4K context is short by today's standards, and it is heavily English-centric — weaker in other languages. As a chat model it can be overcautious, refusing benign requests because the safety training erred toward caution.

And "open" deserves an asterisk. The license is permissive but not true open source: companies with over 700 million monthly active users need a special license, and you may not use Llama 2 to train competing models. It's best called open weights — you get the model, under terms — rather than open source in the strict sense.

Llama 2 lit the fuse on the open-model ecosystem. Because the weights were free and commercially usable, a Cambrian explosion followed: quantized versions that run on laptops, thousands of domain fine-tunes, tooling like llama.cpp and Ollama, and a research community that could finally poke at a real, strong model. Much of today's local and open-source AI traces directly to this release.

It also normalized the pattern of shipping a model family with a proper technical report — architecture, data, and a candid safety section — rather than a closed black box. Later open models (Mistral, Llama 3, and beyond) build on the template Llama 2 set: strong pretraining, RLHF alignment, and open weights as a strategy, not an accident.