Kimi K2,
explained.

K2's architecture is a knowing descendant of DeepSeek-V3: a fine-grained mixture-of-experts with multi-head latent attention for a small KV cache — scaled up to ~1 trillion total parameters with only ~32B active per token. More, smaller experts (384 versus V3's 256) push sparsity further: an even bigger knowledge store, an even smaller per-token bill.

The sparsity scaling law behind that choice is the quiet thesis of the model: at fixed compute, more total parameters with fewer active keeps winning. K2 is what happens when a lab takes that curve seriously and rides it to thirteen digits.

Every frontier model you'd heard of before K2 was trained with Adam-family optimizers. Muon — born in the open-source speedrunning community — updates weight matrices with orthogonalized momentum (via Newton-Schulz iterations), spending each update more evenly across a matrix's directions instead of letting a few dominant ones soak it up. The result: measurably more learning per token — exactly what matters when quality data, not compute, is the binding constraint.

The open question was whether Muon survived scale. Nobody had bet a trillion-parameter, multi-month run on it. Moonshot did — after finding and fixing the one thing that broke.

What broke was attention. In big runs, query-key dot products can grow until softmax saturates and the loss spikes — sometimes fatally, always expensively (rewind to checkpoint, skip data, pray). Muon's update geometry made these attention logit explosions worse, and at K2's scale a single bad spike could cost days.

QK-clip is the seatbelt: monitor each attention head's maximum logit, and when one crosses a threshold, rescale that head's query/key projection weights directly — at the source, in the weights, not with a bandage on the activations. The receipt is the single most striking line in the report: zero loss spikes across the entire 15.5T-token pretraining run. Boring, at trillion scale, is a superpower.

The clip itself is one rule. When a head's max logit crosses the threshold τ, rescale its query and key projections so their product — and thus the logit — is capped right at the source:

Left alone, a drifting logit blows up (softmax collapses to one-hot, gradients die); capped, attention stays differentiable. The runnable version below shows a logit exploding to millions vs held flat at τ.

Most models learn tool calling as a garnish: pretrain on text, sprinkle function-calling examples during fine-tuning. K2's subtitle — Open Agentic Intelligence — announces the opposite bet. Moonshot built a large-scale agentic data synthesis pipeline: thousands of real and synthetic tools, simulated multi-step episodes (plan → call → observe → recover → finish), user simulators to create realistic friction, and an LLM judge filtering the episodes for quality before they ever reach training.

On top: reinforcement learning on verifiable tasks — code that runs, answers that check — plus a critic for the fuzzy ones. The result is a model whose tool use feels native rather than prompted: it decomposes, calls, reads errors, and retries as learned behavior. This is the training-side mirror of everything the agent patterns world builds at inference time.

The benchmark story concentrated exactly where the training diet aimed: software engineering and multi-step tool tasks. On knowledge trivia it was merely very good; on doing things, it led the open field — which was precisely the point.

K2 validated two ideas the field now treats as obvious. Optimizers are not settled science: Muon at 1T ended Adam's decade of unquestioned default status and reopened optimizer research as a frontier lever. And agentic ability is a pretraining decision: synthesized tool-use at scale beats bolting function calling onto a text model — a lesson every serious lab's data pipeline absorbed within months.

Strategically, K2 extended what V3 started: open weights competing at the front, this time on the agent workloads that define the current product era. For builders it meant something concrete — the model driving your agent loop could now be one you download, inspect, fine-tune and serve yourself.