Kimi K2,
explained.

Rungs 1 to 3 are models you talk to. This rung is built for a model that acts — calling tools, running multi-step tasks, recovering from a failed call — and the interesting claim is about where that capability comes from. Not a scaffold around the model and not a prompt, but large-scale synthesis of agentic trajectories as training data: generated sequences of tool calls, results and recoveries, at a volume nobody could author by hand. It is the bootstrapping idea from the post-training ladder, applied to behaviour rather than to instructions.

The architecture is a mixture of experts at about a trillion total parameters with roughly 32B active, which puts it at the shape the sparse ladder established, released with open weights — a combination that had not previously been available at that size.

The training claim is the one worth carrying, because it is about a problem every large run has and few report on. Very large runs suffer loss spikes — sudden divergences that waste days and are usually handled by rolling back and skipping data. This release reports training through 15.5 trillion tokens with none, attributed to combining the Muon optimiser with a clip on the attention logits, which is where the instability originates: a growing dot product between query and key drives the softmax toward saturation, and bounding it directly removes the mechanism rather than damping the symptom.

K2's architecture is a knowing descendant of DeepSeek-V3: a fine-grained mixture-of-experts with multi-head latent attention for a small KV cache — scaled up to ~1 trillion total parameters with only ~32B active per token. More experts of the same width (384 versus V3's 256) push sparsity further: an even bigger knowledge store, an even smaller per-token bill.

The sparsity scaling law behind that choice is the quiet thesis of the model: at fixed compute, more total parameters with fewer active keeps winning. K2 is what happens when a lab takes that curve seriously and rides it to thirteen digits.

Every frontier model you'd heard of before K2 was trained with Adam-family optimizers. Muon — born in the open-source speedrunning community — updates weight matrices with orthogonalized momentum (via Newton-Schulz iterations), spending each update more evenly across a matrix's directions instead of letting a few dominant ones soak it up. The result: measurably more learning per token — exactly what matters when quality data, not compute, is the binding constraint.

The open question was whether Muon survived scale. Nobody had bet a trillion-parameter, multi-month run on it. Moonshot did — after finding and fixing the one thing that broke.

What broke was attention. In big runs, query-key dot products can grow until softmax saturates and the loss spikes — sometimes fatally, always expensively (rewind to checkpoint, skip data, pray). Muon's update geometry made these attention logit explosions worse, and at K2's scale a single bad spike could cost days.

QK-clip is the seatbelt: monitor each attention head's maximum logit, and when one crosses a threshold, rescale that head's query/key projection weights directly — at the source, in the weights, not with a bandage on the activations. The receipt is the single most striking line in the report: zero loss spikes across the entire 15.5T-token pretraining run. Boring, at trillion scale, is a superpower.

The clip itself is one rule. When a head's max logit crosses the threshold τ, rescale its query and key projections so their product — and thus the logit — is capped right at the source:

Left alone, a drifting logit blows up (softmax collapses to one-hot, gradients die); capped, attention stays differentiable. The runnable version below shows a logit exploding to millions vs held flat at τ.

Most models learn tool calling as a garnish: pretrain on text, sprinkle function-calling examples during fine-tuning. K2's subtitle — Open Agentic Intelligence — announces the opposite bet. Moonshot built a large-scale agentic data synthesis pipeline: thousands of real and synthetic tools, simulated multi-step episodes (plan → call → observe → recover → finish), user simulators to create realistic friction, and an LLM judge filtering the episodes for quality before they ever reach training.

On top: reinforcement learning on verifiable tasks — code that runs, answers that check — plus a critic for the fuzzy ones. The result is a model whose tool use feels native rather than prompted: it decomposes, calls, reads errors, and retries as learned behavior. This is the training-side mirror of everything the agent patterns world builds at inference time.

K2 validated two ideas the field now treats as obvious. Optimizers are not settled science: Muon at 1T ended Adam's decade of unquestioned default status and reopened optimizer research as a frontier lever. And agentic ability is a pretraining decision: synthesized tool-use at scale beats bolting function calling onto a text model — a lesson every serious lab's data pipeline absorbed within months.

Strategically, K2 extended what V3 started: open weights competing at the front, this time on the agent workloads that define the current product era. For builders it meant something concrete — the model driving your agent loop could now be one you download, inspect, fine-tune and serve yourself.

Kimi K2 comes from Moonshot AI, released in July 2025 with a technical report titled "Kimi K2: Open Agentic Intelligence". The released model is a mixture of experts with about 1.04T total parameters and 32B active per token: 61 layers, multi-head latent attention, 384 routed experts plus one shared expert, and 8 routed experts chosen per token.

Two architecture choices from scaling experiments

Moonshot's sparsity experiments found that, at a fixed number of active parameters, adding more total experts kept lowering loss, so K2 uses a sparsity of 48, 384 experts with 8 active, against DeepSeek-V3's 256. It also halves the attention heads to 64: doubling them to 128 improved validation loss only slightly while adding a large inference cost at long context, which matters for agents that read long tool outputs.

Pretraining with MuonClip

1. Muon update. For each weight matrix, keep a momentum of the gradients, orthogonalize it with a few Newton–Schulz iterations so the update has roughly equal strength in every direction, rescale it to the size an AdamW update would have, and apply it with weight decay.
2. Measure attention logits. During the forward pass, record for every attention head the largest query-key logit in the batch.
3. Clip at the source. After the optimizer step, any head whose maximum logit exceeded the threshold τ, set to 100 for K2, has its query and key projection weights scaled down by the factor that brings that maximum back to τ. Heads under the threshold are untouched, and the step's forward and backward passes are unchanged.
4. Respect MLA's shared key. In multi-head latent attention, part of the rotary key is shared by all heads, so scaling it would change heads that never misbehaved. QK-Clip therefore scales only the head-specific components.
5. Train on 15.5T tokens. A constant learning rate of 2×10^⁻⁴ for the first 10T tokens, then cosine decay to 2×10^⁻⁵ over 5.5T, weight decay 0.1 and a global batch of about 67M tokens, followed by an annealing and long-context phase that reaches 128K tokens with YaRN.

In K2's run the clip was active early, capping some heads at 100, and the maximum logits later settled below the threshold without further clipping. The report records no loss spikes across the whole pretraining run.

Data and post-training

To get more learning out of limited high-quality text, Moonshot rephrased knowledge data in varied styles and perspectives, chunk by chunk for long documents and with a fidelity check, and rewrote math documents into a study-note style. Post-training starts with supervised fine-tuning on a large synthetic agentic corpus: thousands of real tools served over the Model Context Protocol, collected from GitHub, plus tens of thousands of synthetic tools; generated agents and tasks with rubrics; multi-turn trajectories against simulated users and tool simulators, with real execution sandboxes for code; and an LLM judge that keeps only trajectories meeting their rubrics. Reinforcement learning then combines verifiable rewards for math, code and instruction following with a self-critique rubric reward for open-ended tasks, in which the model judges its own outputs against rubrics and the critic is itself refined on the verifiable tasks.

For a weight matrix W of shape A × B with gradient G_t, Muon as used for K2 updates:

After the step, QK-Clip checks each attention head h:

• NewtonSchulz(M) — a few iterations of an odd matrix polynomial that push M's singular values toward 1, approximating the orthogonal factor U V^^⊤ of M = U Σ V^^⊤
• √max(A, B) · 0.2 — rescales the orthogonalized update to roughly AdamW's update size, so AdamW-tuned learning rates and weight decay carry over
• τ — the logit cap, 100 for K2
• γh — 1 for a healthy head, so nothing changes; below 1 only for a head over the cap

Splitting γ_h as √γ_h on the query and √γ_h on the key scales their product, and so the head's logits, by exactly γ_h. With multi-head latent attention, the head-specific query and key parts take √γ_h, the head-specific rotary query takes the full γ_h, and the shared rotary key is left alone, so one head's clip never touches another head.

The report evaluates Kimi-K2-Instruct as a non-thinking model, without extended reasoning, against open models such as DeepSeek-V3 and Qwen3-235B-A22B with thinking off, and closed ones such as Claude Sonnet 4, Claude Opus 4 and GPT-4.1.

Among open non-thinking models these led at release, with the clearest margins on agentic coding and tool use, the tasks the synthetic data targeted. On training stability the evidence is the loss curve itself: 15.5T tokens with no spikes.

Ablations and negative results. The clip exists because of a failure: in mid-scale runs with plain Muon, maximum attention logits grew past 1,000 and training became unstable, and the report says the explosion appeared more often with Muon than with AdamW. QK-Norm, the usual fix, was ruled out because MLA never materializes full keys at inference, and logit soft-capping was judged insufficient because the dot products can still grow before the cap acts. For data, rephrasing knowledge text ten times and training one epoch beat ten epochs on the raw text on SimpleQA. The report also lists limits: on hard reasoning tasks or with unclear tool definitions, K2 can generate overly long outputs that end up truncated, and enabling tools when they are not needed can lower performance on some tasks.

The stability claim is the most substantive and the hardest to verify from outside: nobody else can inspect that loss curve, and a negative result of this kind — the absence of spikes — is exactly what cannot be independently reproduced without a comparable budget. The mechanism is checkable and plausible, and bounding attention logits has since been discussed as a general stabiliser. Treat the claim as credible and attributed.

The agentic data synthesis is the part to be most careful about, for a reason rung 3 of the post-training ladder already establishes: a generated corpus inherits the generator's distribution, and generated trajectories inherit its idea of what a sensible tool call looks like. A model trained largely on synthesised agent behaviour is good at the agent behaviour someone could synthesise, and how far that extends to a tool it has never seen in an environment that breaks in an unanticipated way is not something a benchmark answers.

How it held up: within its window it became the open model people reached for when they wanted an agent rather than a chat assistant, which is the most meaningful validation available for a release of this kind — practitioners choosing it over alternatives they could also run. The optimiser-plus-clip combination is the part most likely to outlive the specific model.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Five releases you can actually download, in the order each one answered the last. This is where "what does a frontier recipe look like" is documented in public.

1. 1Llama 22023
2. 2Llama 32024
3. 3Qwen32025
4. 4Kimi K22025
5. 5Kimi K32026

All 110 papers, in 19 ladders →