PaLM,
explained.

PaLM is a 540-billion-parameter dense transformer, trained across thousands of TPU chips with Google's Pathways system (which the model is named after). It set state-of-the-art on a broad sweep of benchmarks. But its most quoted contribution isn't a number on a leaderboard — it's a shape of curve: the demonstration that certain abilities are emergent, near-absent below a scale and present above it.

With chain-of-thought prompting, PaLM solved multi-step arithmetic and commonsense problems that smaller models failed outright; it could even explain why a novel joke is funny. Plotted against scale, these tasks stay pinned at chance and then leap. That discontinuity provoked a lot of philosophizing about "phase transitions" in intelligence. The more grounded story is simpler and worth understanding, because it demystifies the whole phenomenon.

An emergent ability is one that's essentially at random-chance performance in smaller models and then rises, sometimes abruptly, once scale crosses a threshold. The defining feature is discontinuity: you scale and scale, the metric stays flat, and then it moves. Smooth quantities like loss keep improving predictably (that's scaling laws); emergent task metrics are the ones that look like a cliff.

PaLM made this vivid across many BIG-bench tasks. But "the model suddenly gained a new skill" is a strong claim — and mostly the wrong frame. What actually changed smoothly under the hood was the model's per-step competence; what looked discontinuous was a composed metric built on top of it. To see why, follow a multi-step task and count where it can go wrong.

A multi-step reasoning problem is an all-or-nothing chain: get any single step wrong and the final answer is wrong. So if a task needs k sequential steps and the model gets each one right with probability p, the whole task succeeds only when every step lands.

That single power is the whole trick. While per-step accuracy is modest, raising it to a high power crushes the task score toward zero — it looks like the model simply can't do the task. As per-step accuracy climbs past a point, the same power law snaps the task score upward. Nothing discontinuous happened to the model; a smooth curve got run through an exponent. Harder tasks (bigger k) need higher per-step accuracy before they clear the bar, so they emerge later — exactly the ordering observed.

Let per-step accuracy p rise smoothly with scale, and let a task require k correct steps. Task accuracy is:

A tiny per-step failure rate compounds: at p=0.8, an 8-step task scores 0.8⁸ ≈ 0.17; at p=0.95 it's 0.95⁸ ≈ 0.66. A modest per-step gain nearly quadruples the task score.

The amplification grows with the chain length. Compare two per-step accuracies; the task-level advantage of the better model is the per-step ratio raised to k:

The same per-step edge (say 0.8→0.9) is a 1.8× task advantage at k=5 but a ~10× advantage at k=20. So the longer the reasoning chain, the more a smooth per-step improvement looks like an abrupt emergence — and the later (in scale) it kicks in. The runnable version below turns a smooth per-step curve into the sharp task jump.

Beyond the emergence story, PaLM was a milestone in scale and engineering:

The joke-explanation demo is the one people remember: given a joke it hadn't seen, PaLM could articulate why it lands. Whether you call that "understanding" or sophisticated pattern completion, it was a concrete signal that qualitatively new behaviors were showing up as scale grew — which is precisely why the emergence framing stuck.

PaLM helped cement scale as a deliberate research strategy: keep making models bigger and qualitatively new capabilities keep appearing. It also sharpened a debate that's still live — how much of "emergence" is a real phase change versus an artifact of using harsh, all-or-nothing metrics. The compositional exponent above is the strongest version of the skeptical view: use a continuous metric and the cliff often becomes a smooth ramp. Both can be true — the underlying skill improves smoothly, while the useful composed task genuinely flips from unusable to usable.

Practically, the takeaway outlived the model. Emergence is why techniques like chain-of-thought matter so much: by breaking a hard composed task into steps the model can each do reliably, you keep per-step accuracy high and dodge the p^k collapse. PaLM itself was soon surpassed and Google moved to the Gemini line, but its lesson — that where you set the metric determines whether progress looks smooth or sudden — is now part of how the field reads every scaling curve.