PaLM,
explained.

Rung 1, and the headline is a number — 540 billion dense parameters, every one of them active on every token, which makes it among the largest dense models anyone has described, though the dense lineage went on (Llama 3.1 405B is dense too) while the frontier moved toward sparse designs. But the size is the instrument rather than the result.

The finding is about the shape of how capability arrives. Across a range of tasks, performance stays near chance as models get bigger — flat, unpromising, no sign of progress — and then jumps sharply past some scale. If you were tracking those tasks you would have concluded the approach did not work, right up until it did.

The explanation worth carrying up the ladder is arithmetic rather than mystical, and it is what the page works through. Suppose a task requires k sequential steps, all of which must be right, and suppose the model's per-step reliability improves smoothly with scale. Task success is then roughly that per-step probability raised to the power k — and raising a smoothly-improving number below one to a high power produces a curve that stays near zero for a long time and then rises steeply. A smooth improvement in a component, compounded over a chain, looks like a discontinuity in the whole. Nothing has to change in kind for the behaviour to appear to.

PaLM is a 540-billion-parameter dense transformer, trained across thousands of TPU chips with Google's Pathways system (which the model is named after). It set state-of-the-art on a broad sweep of benchmarks. But its most quoted contribution isn't a number on a leaderboard — it's a shape of curve: the demonstration that certain abilities are emergent, near-absent below a scale and present above it.

With chain-of-thought prompting, PaLM solved multi-step arithmetic and commonsense problems that smaller models failed outright; it could even explain why a novel joke is funny. Plotted against scale, these tasks stay pinned at chance and then leap. That discontinuity provoked a lot of philosophizing about "phase transitions" in intelligence. The more grounded story is simpler and worth understanding, because it demystifies the whole phenomenon.

An emergent ability is one that's essentially at random-chance performance in smaller models and then rises, sometimes abruptly, once scale crosses a threshold. The defining feature is discontinuity: you scale and scale, the metric stays flat, and then it moves. Smooth quantities like loss keep improving predictably (that's scaling laws); emergent task metrics are the ones that look like a cliff.

PaLM made this vivid across many BIG-bench tasks. But "the model suddenly gained a new skill" is a strong claim — and mostly the wrong frame. What actually changed smoothly under the hood was the model's per-step competence; what looked discontinuous was a composed metric built on top of it. To see why, follow a multi-step task and count where it can go wrong.

A multi-step reasoning problem is an all-or-nothing chain: get any single step wrong and the final answer is wrong. So if a task needs k sequential steps and the model gets each one right with probability p, the whole task succeeds only when every step lands.

That single power is the whole trick. While per-step accuracy is modest, raising it to a high power crushes the task score toward zero — it looks like the model simply can't do the task. As per-step accuracy climbs past a point, the same power law snaps the task score upward. Nothing discontinuous happened to the model; a smooth curve got run through an exponent. Harder tasks (bigger k) need higher per-step accuracy before they clear the bar, so they should emerge later — that is a prediction of this toy model, not something the paper measured.

Let per-step accuracy p rise smoothly with scale, and let a task require k correct steps. Task accuracy is:

A tiny per-step failure rate compounds: at p=0.8, an 8-step task scores 0.8⁸ ≈ 0.17; at p=0.95 it's 0.95⁸ ≈ 0.66. A modest per-step gain nearly quadruples the task score.

The amplification grows with the chain length. Compare two per-step accuracies; the task-level advantage of the better model is the per-step ratio raised to k:

The same per-step edge (say 0.8→0.9) is a 1.8× task advantage at k=5 but a ~10× advantage at k=20. So the longer the reasoning chain, the more a smooth per-step improvement looks like an abrupt emergence — and the later (in scale) it kicks in. The runnable version below turns a smooth per-step curve into the sharp task jump.

PaLM is a dense decoder-only Transformer trained at three sizes on identical data. The largest has 540 billion parameters, every one of them used for every token.

1. Change six things in the Transformer block. SwiGLU in the feed-forward layer, which needs three weight matrices instead of two but improves quality at equal compute. A parallel block, in which attention and the feed-forward layer read the same layer-normed input and their outputs are added. Multi-query attention, in which every head has its own query but all heads share one key and one value projection. Rotary position embeddings (RoPE). Shared input and output embeddings. And no bias terms in any dense layer or layer norm, which the authors found made large models train more stably.
2. Build a lossless vocabulary. A 256,000-token SentencePiece vocabulary, generated from the training data, keeps whitespace exactly, which matters for code, falls back to UTF-8 bytes for characters it does not know, and splits every number into single digits.
3. Fix the shapes. PaLM 8B has 32 layers, 16 heads and a model width of 4,096; 62B has 64 layers, 32 heads and width 8,192; 540B has 118 layers, 48 heads and width 18,432. Head size is always 256, and the feed-forward width is four times the model width.
4. Train once over 780 billion tokens. The mixture is 50 percent multilingual social-media conversations, 27 percent filtered multilingual webpages, 13 percent English books, 5 percent GitHub code, 4 percent multilingual Wikipedia and 1 percent English news, covering more than 100 languages. Each model sees exactly one epoch, with proportions chosen so no source repeats. Sequences are 2,048 tokens, and there is no dropout.
5. Optimize for stability. Adafactor without factorization, which behaves like Adam with learning rates scaled to each parameter matrix; a learning rate of 10^⁻² for 10,000 steps, then decaying as 1/√k; β₂ = 1 − k^^−0.8, which the authors found more stable for rare tokens than a fixed 0.99; gradient clipping at 1.0; weight decay of lr^²; and an auxiliary z-loss. The batch grows from 512 sequences (1 million tokens) to 1,024 at step 50k and 2,048 (4 million tokens) at step 115k, and training ends at step 255k.
6. Spread it over two TPU pods with Pathways. 6,144 TPU v4 chips. Inside each 3,072-chip pod the model is split 12 ways by model parallelism and 256 ways by fully sharded data parallelism, with no pipeline stages. Across pods, a single Pathways client sends half of each batch to each pod; the pods exchange gradients over the data-center network and apply identical updates. The run reached 46.2 percent model FLOPs utilization, and 57.8 percent hardware FLOPs utilization, which also counts recomputed activations.
7. Handle loss spikes by hand. The 540B run spiked about 20 times despite gradient clipping, at irregular intervals and sometimes late in training. The fix was to restart from a checkpoint about 100 steps before the spike and skip roughly 200 to 500 batches. Replaying those same batches from a different, earlier checkpoint did not spike, so the authors concluded the cause was a particular combination of data and parameter state rather than bad data.

• LN — layer normalization. In the parallel form one normalized input feeds both sublayers, so their input matrix multiplications can be fused into one.
• W, V, W2 — the three feed-forward matrices SwiGLU needs; ⊙ is elementwise multiplication, so one branch gates the other.
• Z — the softmax normalizer over the vocabulary. The z-loss pulls log Z toward 0, that is Z toward 1.
• T — the tokens in the batch; the main term is the ordinary next-token loss, with no label smoothing.

The parallel form gives something up: within a layer the feed-forward block no longer sees attention's output. The ablation put the cost at a small quality drop at 8B and none at 62B, against roughly 15 percent faster training at large scale, so the authors extrapolated it to be quality-neutral at 540B. The z-loss weight, 10^⁻⁴, is tiny; the authors include the term because they found that keeping the normalizer near 1 made training more stable.

Results are few-shot prompting of the pretrained model unless noted, compared with published results for GPT-3, Gopher, Chinchilla and other single-checkpoint models.

PaLM roughly matched Codex 12B on code with about 50 times less Python in training: 2.7 billion tokens against 100 billion.

The scaling analysis is where the emergence claim comes from, and it is quantified. Across 150 BIG-bench tasks the authors compared the jump from 62B to 540B with a log-linear extrapolation of the jump from 8B to 62B. About 25 percent of tasks beat the extrapolation by more than 10 points, and 15 percent by more than 20. Logical sequence went from 13 to 25 to 87 percent against a projected 37, a discontinuity of +50; English proverbs went from 25 percent at 62B to 87 percent at 540B. Scale did not help everywhere: on navigate and mathematical induction, 540B barely beat 62B and both stayed far below the best humans.

The design ablations read as engineering trade-offs rather than wins: the parallel block cost a little quality at 8B, multi-query attention was neutral on quality and training speed, and the 540B run hit about 20 loss spikes that were handled by restarting from an earlier checkpoint and skipping data.

The emergence framing drew the most interesting critique on this ladder. Schaeffer and colleagues argued that apparent sharp jumps are frequently artefacts of the metric: score a multi-step task by exact match and you get a step function out of smooth progress, because a nearly-correct answer scores zero. Change to a metric with partial credit and the discontinuity often flattens out.

What makes this worth reading carefully is that the critique and the mechanism above are the same observation from opposite directions. The compounding argument says exact success on a k-step task is a harsh function of smooth underlying skill; the critique says exact-match scoring is a discontinuous metric. Both are true, and together they resolve the question — the jumps are real as measurements of task completion, and they are not evidence of a discontinuity in the underlying model. Which of those you care about depends on whether you are studying models or shipping products, and confusing the two is where most of the overclaiming happened.

How it held up: the dense architecture did not. A 540B model activating every parameter per token is precisely what the sparse ladder exists to avoid, and nothing at the frontier has been built that way since. The infrastructure work that made it trainable across many accelerators was genuinely influential, and the emergence question — reframed as "which metrics are we fooling ourselves with" — is now a standing methodological concern rather than a phenomenon.

PaLM helped cement scale as a deliberate research strategy: keep making models bigger and qualitatively new capabilities keep appearing. It also sharpened a debate that's still live — how much of "emergence" is a real phase change versus an artifact of using harsh, all-or-nothing metrics. The compositional exponent above is the strongest version of the skeptical view: use a continuous metric and the cliff often becomes a smooth ramp. Both can be true — the underlying skill improves smoothly, while the useful composed task genuinely flips from unusable to usable.

Practically, the takeaway outlived the model. Emergence is why techniques like chain-of-thought matter so much: by breaking a hard composed task into steps the model can each do reliably, you keep per-step accuracy high and dodge the p^^k collapse. PaLM itself was soon surpassed and Google moved to the Gemini line, but its lesson — that where you set the metric determines whether progress looks smooth or sudden — is now part of how the field reads every scaling curve.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Four papers that are reports, not methods: they state capability and withhold recipe. Read for the evaluation design and the omissions, not the architecture.

1. 1PaLM2022
2. 2GPT-4 Technical Report2023
3. 3Gemini 1.52024
4. 4GPT-5.6 System Card2026

All 110 papers, in 19 ladders →