The Llama 3 herd,
explained.

Rung 1 trains on 2 trillion tokens. This rung trains on 15 trillion, and the small models in the family are pushed furthest past where the compute-optimal allocation says to stop. That is not an accident or a bigger budget applied uniformly; it is a deliberate rejection of the objective that allocation optimises, and the reasoning is the most useful thing on this rung.

Compute-optimal scaling answers: given a fixed training budget, what model size and token count minimise loss? That is the right question if training is the cost you care about. For a model you intend to serve billions of times, it is the wrong question, because the lifetime compute is dominated by inference, and inference cost is set by the model's size — every query pays for every parameter, forever.

So the trade inverts. Spend more training compute than is optimal, on a smaller model, to get a model that is cheaper on every request it will ever answer. Stated in the units the scaling ladder uses: Chinchilla's ratio is roughly twenty training tokens per parameter, and an 8B model trained on 15 trillion tokens is at something closer to two thousand — two orders of magnitude past compute-optimal, on purpose. Training-optimal and inference-optimal are different objectives, and this rung is where the industry publicly picked the second one.

Llama 3 is a family — an 8B, a 70B, and a flagship 405B — all dense, decoder-only transformers. Architecturally it is almost boringly conventional: the same transformer block, rotary position embeddings, grouped-query attention to shrink the KV cache, a bigger vocabulary. No exotic mechanism, no mixture-of-experts in the main models. If you were grading on novelty, you'd shrug.

But the paper's real contribution is a method thesis: quality at a given size comes overwhelmingly from data — how much, and how clean — and from a willingness to spend training compute lavishly to make the deployed model small. Llama 3 trained on ~15T tokens with aggressive filtering, dedup, and quality classifiers, and it deliberately pushed the smaller models far past the "optimal" token count. To see why that isn't a mistake, we need the scaling law it's departing from.

The Chinchilla scaling laws answered a specific question: given a fixed training compute budget, how should you split it between model size and training tokens to get the best model? Their empirical answer was a clean rule of thumb — scale parameters and tokens together, roughly 20 tokens per parameter. Under that rule an 8B model "should" see about 160B tokens; anything past that is, by Chinchilla's accounting, a poor use of the training budget, since the same FLOPs would buy more quality spent on a larger model.

Llama 3's 8B saw ~15T tokens — call it ~94× the Chinchilla-optimal count. By the training-compute yardstick, that is deep into diminishing returns: those last trillions of tokens each nudge quality only a little. So why do it? Because Chinchilla optimizes the wrong bill.

Chinchilla counts only the training FLOPs. But a production model's compute has two parts, and they're paid on wildly different schedules. Training is a one-time cost. Inference is paid per query, forever, and a popular model answers billions of them. For a heavily-served model the inference total swamps the training total.

Both costs scale with parameters. Training a dense transformer costs about 6ND FLOPs (N params, D tokens); each token of inference costs about 2N FLOPs. Crucially, inference cost is proportional to N, the thing you deploy. So a smaller model is permanently cheaper to run — every query, forever. Over-training is the price you pay once, in training, to buy a smaller N at a target quality, and thereby cut the inference cost you pay always. This is the inference-optimal frontier, and it sits at a smaller model + more tokens than Chinchilla's training-optimal one.

Write the total lifetime compute as training plus inference over Q queries (each generating about t tokens):

First term: one-time training on D tokens. Second term: inference, paid per query, and it grows without bound as Q grows. Both are proportional to N — so shrinking the model helps both, but it helps the inference term Q times as often.

Now the key move. Over-training raises D (the training term) to buy a smaller N at fixed quality. That trade is worth it once the inference term dominates — i.e. once the model is served enough. Set the two terms equal to find the crossover query count:

Past Q_* queries, inference has cost more than the entire training run — so any parameter you can shave off (paid for with extra tokens D) is repaid many times over. With D≈15T and t≈10³, Q_*≈4.5×10¹⁰: beyond ~45 billion queries, inference rules the bill. A widely-deployed model blows past that quickly, which is exactly why over-training the small models is the frugal choice. The runnable version below computes the over-training ratio and shows inference overtaking training.

"The Llama 3 Herd of Models" comes from the Llama team at Meta, released in July 2024, and describes the 8B, 70B and 405B models released as Llama 3.1. All three are dense Transformers; the report chose that over mixture-of-experts to keep training stable at scale.

1. Build the data. About 15T multilingual tokens after PII and safety filtering, a custom HTML extractor, deduplication at the URL, document (MinHash) and line level, heuristic filters, and model-based quality classifiers, including a fastText classifier and RoBERTa-based classifiers trained on Llama 2's quality judgments. The final mix is roughly 50% general knowledge, 25% math and reasoning, 17% code and 8% multilingual text.
2. Size the flagship with a scaling law. IsoFLOP sweeps from 6×10^¹⁸ to 10^²² FLOPs, with models from 40M to 16B parameters, locate the compute-optimal token count at each budget. Extrapolated to the flagship's 3.8×10^²⁵ FLOPs, almost 50 times the largest Llama 2, the fit points to about a 402B model on 16.55T tokens. A second step maps compute to the normalized log-likelihood of correct benchmark answers, and that log-likelihood to accuracy, fitted with both the sweep models and Llama 2 models; it forecast the 405B model's ARC-Challenge score closely.
3. Pretrain the 405B model. AdamW at a peak rate of 8×10^⁻⁵, warmed up over 8,000 steps and decayed by cosine to 8×10^⁻⁷. The batch grows during training, from 4M tokens in 4,096-token sequences, to 8M tokens in 8,192-token sequences, to 16M. Documents packed into one sequence are masked so they cannot attend to each other, and the vocabulary has 128K tokens.
4. Extend the context in six stages. From 8K to 128K tokens over about 800B tokens, moving to the next stage only when short-context performance has fully recovered and needle-in-a-haystack retrieval is perfect at the current length.
5. Anneal. Over the final 40M tokens the learning rate falls linearly to zero while high-quality sources are upsampled, and checkpoints from the annealing phase are averaged (Polyak averaging) to give the final pretrained model.
6. Post-train in six rounds. Each round trains a reward model on human preference pairs, now with an edited-response tier above the chosen one and without Llama 2's margin term; samples 10 to 30 responses per prompt and keeps the reward model's best (rejection sampling); fine-tunes on those plus synthetic data; then runs DPO on fresh preference pairs. Models from runs with different data or hyperparameters are averaged at each stage.

Why DPO, and how it was modified

Meta chose DPO over PPO because it needed less compute at this scale and did better, especially on instruction-following benchmarks such as IFEval. Two changes made it stable. Formatting tokens, the header and end-of-turn tokens shared by chosen and rejected answers, are masked out of the loss, because their contradictory gradients caused tail repetition and abruptly ended answers. And a negative log-likelihood term on the chosen answer, weighted 0.2, keeps the model from lowering the probability of good answers while it pushes rejected ones down.

Running on 16K GPUs

The 405B run used up to 16K H100 GPUs with four kinds of parallelism, tensor, pipeline, context and fully sharded data parallelism, at 38 to 43% model FLOPs utilization. Over one 54-day period there were 466 job interruptions, 419 of them unexpected and about 78% of those traced to confirmed or suspected hardware faults, yet effective training time stayed above 90%.

For a prompt x with a chosen response y_w and a rejected one y_l, Llama 3's post-training minimizes the DPO loss plus a likelihood term on the chosen response:

• πθ, πref — the model being trained and the frozen fine-tuned model of the same round
• β = 0.1 — how far the policy may move from the reference to separate the pair
• α = 0.2 — the weight on the chosen answer's negative log-likelihood term
• masking — header and end-of-turn tokens, shared by y_w and y_l, are left out of both log-probabilities

The first term only cares about the gap between chosen and rejected, so it can fall while both answers become less likely, a known way for DPO to degrade. The α term holds up the chosen answer's likelihood, so the gap opens mostly by pushing the rejected answer down. With α = 0 this is plain DPO; as β grows, the model stays closer to π_ref and learns less from each pair. Masking matters because a token present in both answers is pushed up through y_w and down through y_l at once, and Meta traced tail repetition and abruptly ended answers to that conflict.

The report compares the herd with Llama 2 and with contemporary models, including GPT-4, GPT-4o and Claude 3.5 Sonnet for the 405B, on knowledge, code, math, reasoning, tool use, long-context and multilingual benchmarks, plus human preference evaluations.

Meta's reading is that the 405B model is close to the leading closed models of mid-2024 on most benchmarks, and that the 8B and 70B models lead their size classes, the payoff of training them far past their compute-optimal token counts.

Negative results and limits. Annealing on small amounts of high-quality code and math raised the 8B model's GSM8K score by 24.0 points and MATH by 6.4, but gave the 405B model negligible gains, which the authors read as the flagship not needing in-domain examples to perform well. PPO was dropped because it cost more compute and did worse on instruction following. Removing the reward-model margin was itself a negative result for Llama 2's design: it stopped helping once preference data was scaled. Human evaluations against GPT-4o and Claude 3.5 Sonnet were mixed rather than clear wins. And reliability was a real cost: 419 unexpected interruptions in 54 days, most of them hardware.

The returns are admitted to be diminishing — the loss curve keeps improving well past the compute-optimal point, which is the finding, but it improves slowly, so the strategy only makes sense when amortised over a very large number of queries. For anyone training a model they will serve occasionally, this is the wrong advice, and the conditional is the point rather than a caveat.

The data mixture is given only coarsely (roughly 50% general knowledge, 25% math and reasoning, 17% code, 8% multilingual, with no sources named), and 15 trillion tokens raises questions about the supply of text that the report does not settle — how much of that volume is high-quality, how much is repeated, and what the ceiling is. The data-quality argument on the scaling ladder is the other half of this conversation.

How it held up: over-training became standard practice for any model intended for deployment, and the framing — that a scaling law optimises an objective you must choose deliberately — is one of the most transferable ideas here. Read together, this rung and Chinchilla make the site's best case for why reading order matters: neither is wrong, they optimise different things, and knowing which is which is the entire skill.

Llama 3 crystallized a shift in how the field thinks about scaling. Chinchilla was framed as "the" scaling law, but it answered a training-only question. Llama 3 (and its Llama 2 predecessor) made the inference-optimal view standard practice: because deployment cost, not training cost, dominates the lifetime of a popular model, the industry now routinely over-trains smaller models on trillions of tokens. The "right" model size shrank, and token counts exploded.

It also mattered as open weights. A frontier-class model whose weights anyone could download and fine-tune reset the ecosystem — it became the default base for countless downstream models, from multimodal systems to domain fine-tunes, precisely because a strong, cheap-to-serve base is the most useful thing you can hand the community. The paper's quiet lesson is that the frontier isn't only pushed by new mechanisms; sometimes it's pushed by taking a known recipe and being ruthless about data, scale, and the bill you'll actually pay.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Five releases you can actually download, in the order each one answered the last. This is where "what does a frontier recipe look like" is documented in public.

1. 1Llama 22023
2. 2Llama 32024
3. 3Qwen32025
4. 4Kimi K22025
5. 5Kimi K32026

All 110 papers, in 19 ladders →