The Llama 3 herd,
explained.

Llama 3 is a family — an 8B, a 70B, and a flagship 405B — all dense, decoder-only transformers. Architecturally it is almost boringly conventional: the same transformer block, rotary position embeddings, grouped-query attention to shrink the KV cache, a bigger vocabulary. No exotic mechanism, no mixture-of-experts in the main models. If you were grading on novelty, you'd shrug.

But the paper's real contribution is a method thesis: quality at a given size comes overwhelmingly from data — how much, and how clean — and from a willingness to spend training compute lavishly to make the deployed model small. Llama 3 trained on ~15T tokens with aggressive filtering, dedup, and quality classifiers, and it deliberately pushed the smaller models far past the "optimal" token count. To see why that isn't a mistake, we need the scaling law it's departing from.

The Chinchilla scaling laws answered a specific question: given a fixed training compute budget, how should you split it between model size and training tokens to get the best model? Their empirical answer was a clean rule of thumb — scale parameters and tokens together, roughly 20 tokens per parameter. Under that rule an 8B model "should" see about 160B tokens; anything past that is, by Chinchilla's accounting, a poor use of the training budget, since the same FLOPs would buy more quality spent on a larger model.

Llama 3's 8B saw ~15T tokens — call it ~94× the Chinchilla-optimal count. By the training-compute yardstick, that is deep into diminishing returns: those last trillions of tokens each nudge quality only a little. So why do it? Because Chinchilla optimizes the wrong bill.

Chinchilla counts only the training FLOPs. But a production model's compute has two parts, and they're paid on wildly different schedules. Training is a one-time cost. Inference is paid per query, forever, and a popular model answers billions of them. For a heavily-served model the inference total swamps the training total.

Both costs scale with parameters. Training a dense transformer costs about 6ND FLOPs (N params, D tokens); each token of inference costs about 2N FLOPs. Crucially, inference cost is proportional to N, the thing you deploy. So a smaller model is permanently cheaper to run — every query, forever. Over-training is the price you pay once, in training, to buy a smaller N at a target quality, and thereby cut the inference cost you pay always. This is the inference-optimal frontier, and it sits at a smaller model + more tokens than Chinchilla's training-optimal one.

Write the total lifetime compute as training plus inference over Q queries (each generating about t tokens):

First term: one-time training on D tokens. Second term: inference, paid per query, and it grows without bound as Q grows. Both are proportional to N — so shrinking the model helps both, but it helps the inference term Q times as often.

Now the key move. Over-training raises D (the training term) to buy a smaller N at fixed quality. That trade is worth it once the inference term dominates — i.e. once the model is served enough. Set the two terms equal to find the crossover query count:

Past Q_* queries, inference has cost more than the entire training run — so any parameter you can shave off (paid for with extra tokens D) is repaid many times over. With D≈15T and t≈10³, Q_*≈4.5×10¹⁰: beyond ~45 billion queries, inference rules the bill. A widely-deployed model blows past that quickly, which is exactly why over-training the small models is the frugal choice. The runnable version below computes the over-training ratio and shows inference overtaking training.

Llama 3 delivered open-weight models competitive with the best closed models of its moment, and its choices read as a lesson in what actually moves quality at scale:

The 405B flagship showed how far a straightforward dense recipe scales; the 8B and 70B showed that the practically important models are the small over-trained ones that most people actually run. Both halves make the same point: at scale, data and serving economics are the levers, not architectural novelty.

Llama 3 crystallized a shift in how the field thinks about scaling. Chinchilla was framed as "the" scaling law, but it answered a training-only question. Llama 3 (and its Llama 2 predecessor) made the inference-optimal view standard practice: because deployment cost, not training cost, dominates the lifetime of a popular model, the industry now routinely over-trains smaller models on trillions of tokens. The "right" model size shrank, and token counts exploded.

It also mattered as open weights. A frontier-class model whose weights anyone could download and fine-tune reset the ecosystem — it became the default base for countless downstream models, from multimodal systems to domain fine-tunes, precisely because a strong, cheap-to-serve base is the most useful thing you can hand the community. The paper's quiet lesson is that the frontier isn't only pushed by new mechanisms; sometimes it's pushed by taking a known recipe and being ruthless about data, scale, and the bill you'll actually pay.