Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

~9 min readMoonshot AI · 2026

Kimi K3,
explained.

Every paper so far fought the KV cache's memory. K3 goes after the other bill — decode speed. Softmax attention has to re-read every token it has ever seen to write the next one, so generation gets slower the longer the context grows. Kimi Delta Attention throws that model out: keep one fixed-size state, update it per token with a correction that overwrites instead of piling up, and every decode step costs the same whether you're 1,000 tokens in or a million.

The paper

Kimi K3

Moonshot AI · unverified illustrative explainer (KDA comes from Moonshot's 2025 Kimi Linear report)

Contents 15 sections

The shape: the largest open model yet The decode wall: O(n) per token KDA: trade the cache for a fixed-size state The delta Attention residuals: keep the sharp lookups The receipt Why it matters How it works The math The evidence Since then Solve it yourself Questions Check yourself Where this sits

The shape: the largest open model yet

K3 is the sequel to Kimi K2, and it goes bigger: roughly 2.8T total parameters in a sparse mixture-of-experts, released with open weights — reported as the largest open model to date — carrying a 1M-token context. K2's contribution was training stability (MuonClip taming attention-logit blowups). K3's is a different axis entirely: the cost of generating into that million-token window.

Scale and context length are the easy headlines. The hard one is hidden in the word "1M": at that length, the thing that kills you is not storing the context — it's the fact that softmax attention re-reads all of it on every single token it generates. That is the wall K3's architecture is built to knock down.

The decode wall: O(n) per token

To generate token number n+1, softmax attention scores the new query against the key of every previous token and mixes their values. That is a read over the whole KV cache — and the cache has n entries. So the work per decode step grows linearly with how much you've already written. Multiply by a million tokens and decode, not memory, becomes the bottleneck.

softmax decode: read n keys/values per step → cost per token ∝ n (grows without bound)

The cache you must scan keeps getting longer as the model talks. Every new token is more expensive than the last.

This is the same wall Mamba and the linear-attention family were built to climb. K3's answer, KDA, is in that lineage — but with a twist that fixes linear attention's oldest weakness.

KDA: trade the cache for a fixed-size state

Kimi Delta Attention (introduced in Moonshot's 2025 Kimi Linear report) replaces the growing cache with a single fixed-size recurrent state — a matrix S of about d×d. Each token folds itself into S; to read, you multiply S by the query. The state is the same size no matter how long the context is, so every decode step does the same constant amount of work.

Softmax reads the whole past · KDA folds it into one fixed state

Token n+1 query

→

S ← update(S, k, v)
fixed d×d state

→

o = S·q
read is O(d²), not O(n)

→

Same cost at 1M
as at 1k

KDA decode: read d² state elements per step → cost per token = constant in n

Fixed state, fixed work. The runnable below shows the softmax read growing with context while the KDA read never moves.

The delta: overwrite, don't accumulate

Plain linear attention has a fatal flaw: it accumulates. Write key k with value v, then write the same k with a new v', and the state holds v + v' — a smear of every value ever associated with that key. Over a long context the memory turns to mud.

The delta rule fixes this. Before writing, it subtracts out what the state currently returns for that key, so the update is a correction toward the new value, not a pile-on:

S ← S − β·(S·k − v)·kᵀ · with β=1 and a clean key: read(k) returns v', the latest — not v+v'

The name is literal: it writes the delta between what's stored and what should be. That correction is what lets a fixed-size state stay accurate instead of blurring.

plain linear: write (k,a) then (k,b) → read k = a+b (smeared)
delta rule: write (k,a) then (k,b) → read k = b (overwritten, correct)

Attention residuals: keep the sharp lookups

Even with the delta correction, a fixed-size state gives up something: exact recall at extreme range. Softmax attention can point at one specific token a million steps back with perfect precision; a compressed recurrent state can blur it. So K3 doesn't go all-in on KDA.

It keeps a minority of full-attention layers (and residual paths) interleaved with the KDA bulk — a hybrid. The KDA layers give constant-cost decode across most of the network; the few softmax layers preserve the sharp, exact lookups that pure linear attention loses. You pay full-attention cost only where it earns its keep.

The key idea

Linear attention isn't a drop-in replacement for softmax — it's a budget. Spend it on the layers where approximate mixing is fine (most of them), and keep exact attention for the few layers that need to find a needle. The hybrid is what makes a 1M-token model both fast to decode and still able to remember precisely.

The receipt

Item	Reported
Total parameters	≈ 2.8T (MoE) — largest open model to date
Context length	1M tokens
KDA decode speedup @ 1M	≈ 6.3×
Attention	KDA (linear, delta rule) + residual full-attention layers
Weights	open

The 6.3× is an end-to-end decode figure at a million tokens — smaller than the raw asymptotic gap between a fixed state and an n-length cache, because the residual full-attention layers still pay the softmax price, and because real decode is more than attention. The runnable model below reproduces the mechanism: constant KDA read versus a softmax read that climbs with every token, and the delta correction that keeps the fixed state honest.

Why it matters

K3 is the clearest sign that linear attention has graduated from research curiosity to frontier production. For years the story was "softmax attention is expensive but irreplaceable." K3 replaces most of it — keeping just enough for exact recall — and ships it at the largest open scale yet. Paired with DeepSeek-V4, which keeps softmax attention but compresses and sparsifies it, you can see the field forking into two answers to the same long-context question.

The K2 → K3 arc is telling: K2 made attention stable at scale; K3 makes it cheap at length. And the delta rule at its core is a decades-old idea from associative memory, revived — a reminder that the linear-attention lineage running through Mamba keeps finding new ways to make a fixed-size state do a growing cache's job.

How Kimi Delta Attention works, step by step

Kimi Delta Attention, or KDA, was introduced by Moonshot AI in its Kimi Linear report in late 2025, as a refinement of the gated delta rule from Gated DeltaNet. This page's K3 figures are labeled illustrative and are not tied to a published report, so this section describes KDA as published, and anything specific to K3 is attributed to this page.

Kimi Delta Attention per token: decay each key channel, erase what the key currently retrieves, write the new value, read with the query. The state never grows, so the read costs the same at any context length.

Project the token. Each token produces a query q_t, a key k_t and a value v_t, as in any attention layer. In the DeltaNet family the query and key are L2-normalized, which keeps the erase-and-write step stable and makes a full-strength write an exact overwrite.
Compute two gates. A write strength β_t between 0 and 1, and a forget gate α_t with one value between 0 and 1 for every channel of the key dimension. The per-channel forget gate is what KDA adds. Gated DeltaNet uses a single scalar per head, so the whole memory fades at one rate; KDA lets each channel keep or drop its contents at its own rate.
Decay. Multiply the state S, a d_v × d_k matrix per head, by the diagonal of α_t along the key dimension. Old associations fade channel by channel.
Erase. Read what the decayed state currently returns for this key, S k_t, and remove a β_t share of it. This is the delta rule: the memory is corrected toward the new value instead of having the new value piled on top of the old one.
Write. Add β_t v_t k_t^^⊤, storing the new value under this key. Erase and write together amount to S ← S − β_t(S k_t − v_t) k_t^^⊤, applied to the decayed state.
Read. The output is S q_t, the value the memory associates with the query, which is then projected back to the model dimension. The cost is proportional to d_v × d_k per head, however long the context already is.

Training in chunks

Run token by token, the recurrence would leave a GPU idle. Training instead splits the sequence into chunks. Inside a chunk, the product of the per-token decay-and-erase matrices is rewritten in a compact form, the WY representation also used for DeltaNet, so most of the work becomes dense matrix multiplication. Between chunks only the state is carried forward. Kimi Linear describes its kernel as a constrained form of diagonal-plus-low-rank transitions that is cheaper than the general form, which is how the per-channel gate fits without giving up the parallel algorithm.

Hybrid layers

A fixed-size state cannot store an unbounded transcript exactly, so KDA models keep some full-attention layers. In Kimi Linear most layers are KDA and a minority are full attention, which provide exact retrieval over the whole context. Those layers still keep a KV cache that grows with length, so the end-to-end decode speedup at long context is smaller than the raw gap between a fixed state and a full cache. This page describes K3 as following the same pattern: KDA in most layers, a minority of full-attention layers, and a 1M-token context.

The math

The gated delta rule that KDA computes

Per head, with the state S_t of size d_v × d_k and a per-channel forget gate α_t ∈ (0, 1)^d_k, KDA updates and reads:

S_t = S_t−1 Diag(α_t) ( I − β_t k_t k_t^^⊤ ) + β_t v_t k_t^^⊤
o_t = S_t q_t

Diag(α_t) — decay: each key channel keeps its own fraction of what it stored
I − β_t k_t k_t^^⊤ — erase: removes a β_t share of whatever the decayed state returns for k_t
β_t v_t k_t^^⊤ — write: stores the new value under the key
o_t — read: what the memory associates with the query

The special cases show where KDA sits. With every α equal to 1 the decay disappears and this is the plain delta rule of DeltaNet; with α a single scalar per head it is Gated DeltaNet; KDA's change is letting α differ by channel. With β_t = 0 nothing is written and the state only fades. With β_t = 1 and a unit-length key, reading k_t right after the write returns exactly v_t, the overwrite this page's runnable example shows. Drop the erase factor and the rule becomes gated linear attention, which accumulates: with no decay, two writes to the same key return their sum.

The evidence

What has been measured, and what this page only claims

This page cites no primary source for Kimi K3 and labels its model figures illustrative. The evidence that exists for the mechanism comes from the work KDA is built on, so the table keeps the two apart.

Claim	Source	Status
About 2.8T total parameters, mixture of experts	This page	Illustrative, unsourced
1M-token context	This page	Illustrative, unsourced
About 6.3× faster decoding at 1M tokens	This page	Illustrative, unsourced
KDA: delta rule with a per-channel forget gate	Kimi Linear report, Moonshot AI, 2025	Published, open weights
A KDA and full-attention hybrid matches or beats full attention under the same training recipe	Kimi Linear report	Vendor-reported
Much smaller KV cache and several-fold faster decoding at 1M tokens	Kimi Linear report	Vendor-reported
Delta-rule updates can be trained in parallel over chunks	DeltaNet, Yang and colleagues, 2024	Published, open implementation

The negative result that shapes every design here. Fixed-state models have a documented weakness on exact recall. Zoology (Arora and colleagues, 2023), on associative recall, and Repeat After Me (Jelassi and colleagues, 2024), on copying, found that recurrent and state-space models with a fixed-size state fall behind attention on tasks that require retrieving a specific earlier span, and that the gap grows with how much has to be held. The delta rule targets interference, which helps, but it does not change how much information a fixed state can hold. That is why hybrids keep full-attention layers, and why any long-context claim for a KDA model should be tested on retrieval and multi-hop tasks, not on decode speed alone.

Since then

Where this stands, and what would settle it

Very recent, open weights, vendor-reported numbers. The mechanism is the part to reason about, and the question to ask it is the one the context ladder has asked every fixed-state design: what happens to exact recall. A delta-rule update is a genuinely better answer than decay — it targets interference, which is the specific way a fixed state degrades — and it does not change the information budget. A state of fixed size cannot hold an unbounded transcript, and no update rule makes it able to.

Which is why the interesting detail in designs of this family is usually the proportion: how many layers remain full attention. Pure linear attention has not beaten hybrids on recall-heavy work, and a released model's practical long-context behaviour depends more on that ratio than on the mechanism's name.

What would settle it: independent long-context evaluation on tasks that are not retrieval of a copyable span — multi-hop reasoning and aggregation across a long document — plus the decode numbers measured at realistic batch sizes rather than at the configuration that shows the multiple best. The structural claim stands regardless of how those land: the decode cost of long context is a first-order product constraint, and a constant-cost path is worth a large amount of architectural risk.

SOLVE IT YOURSELF

A fixed state that overwrites — and never gets slower

KDA's two claims are checkable in a dozen lines. First, the delta rule: write a key/value into a fixed-size state, then overwrite the same key — a correct memory returns the latest value, while plain linear attention would hand back the sum of both. Second, constant-cost decode: the state is d×d regardless of context, so a KDA read costs the same at a million tokens as at a thousand, while softmax's read grows with every token. Change the state dim, the keys, the values.

YOUR TASK

Write the body of delta_write. Everything else in the file is already written; this is the function the paper is about. Press ▶ Run to check it against the 5 assertions below — they start red.

WHAT IT HAS TO SATISFY — 5 ASSERTIONS

Write a key then read it back — the delta state returns that value
The delta rule OVERWRITES: writing a key twice keeps the latest value
Distinct keys do not interfere in the shared state
The recurrent state is a fixed d*d, independent of context length
Softmax decode grows with context; the KDA state read does not

CPython · WebAssembly

# Kimi Delta Attention as a fixed-size associative memory.
# State S is a d x d matrix; keys are one-hot here so S is an exact lookup table.
D = 4

def onehot(i, d=D):
    v = [0.0] * d; v[i] = 1.0; return v
def zeros():
    return [[0.0] * D for _ in range(D)]
def read(S, k):                                  # o = S . k
    return [sum(S[r][c] * k[c] for c in range(D)) for r in range(D)]

def delta_write(S, k, v, beta=1.0):              # S <- S - beta*(S.k - v).k^T   (corrective)
    # TODO: write this. Everything else on the page is already here —
    # this is the one function the paper is actually about.
    raise NotImplementedError

def linear_write(S, k, v):                       # plain linear attention: S <- S + v.k^T (accumulates)
    return [[S[r][c] + v[r] * k[c] for c in range(D)] for r in range(D)]

# --- decode cost: fixed state vs a cache that grows with context ---
def state_elems(d):        return d * d          # KDA: fixed recurrent state
def softmax_kv_reads(n, d):return 2 * n * d      # softmax: read K and V for all n tokens, every step
def delta_reads(d):        return d * d          # KDA: read only the fixed state

k1, k2 = onehot(1), onehot(2)
a, b = [9.0, 8.0, 7.0, 6.0], [1.0, 2.0, 3.0, 4.0]

S = delta_write(delta_write(zeros(), k1, a), k1, b)   # write k1=a, then overwrite k1=b
print("delta  read(k1):", read(S, k1), " -> latest value b (old a gone)")
L = linear_write(linear_write(zeros(), k1, a), k1, b)
print("linear read(k1):", read(L, k1), " -> a+b, smeared\n")

print(f"{'context n':>12}{'softmax reads':>15}{'KDA reads':>12}{'speedup':>10}")
for n in (1_000, 128_000, 1_000_000):
    sm = softmax_kv_reads(n, 64)
    print(f"{n:>12,}{sm:>15,}{delta_reads(64):>12,}{sm // delta_reads(64):>9,}x")

Questions

Questions people ask about Kimi K3

How does the gated delta rule update its memory?

The gated delta rule keeps a fixed-size matrix memory and, for each new token, does three things. It decays the memory by a forget gate, erases part of what the memory currently returns for the token's key, and writes the token's value under that key. The erase step is the delta rule, an idea that goes back to the Widrow and Hoff learning rule of 1960: the memory is corrected toward the target instead of having new values piled on old ones. The gate adds the ability to drop stale information quickly. Gated DeltaNet uses one gate value per head; Kimi Delta Attention uses one per key channel. Together, the gate and the correction keep a fixed-size state accurate for longer.

How is Kimi Delta Attention different from Gated DeltaNet?

In the forget gate. Gated DeltaNet decays its whole memory by one scalar per head at each token, so every stored association fades at the same rate. Kimi Delta Attention, introduced in Moonshot AI's Kimi Linear report in 2025, replaces that scalar with a vector holding one gate value per channel of the key dimension, so different parts of the memory can be kept or discarded at different rates. The delta-rule erase and write steps are the same. Moonshot also wrote a dedicated chunked training kernel, described as a constrained diagonal-plus-low-rank form that is cheaper than the general one, so the finer gate does not give up parallel training.

Why does linear attention decode at constant cost per token?

Because it never looks back at the individual past tokens. Softmax attention compares each new query with the stored key of every earlier token, so the work and memory traffic per generated token grow with the length of the context. A linear-attention layer, including KDA, instead folds each token into a fixed-size state matrix as it arrives, and reading that state for a new query costs the same whether the context is a thousand tokens or a million. The trade is exactness: the state compresses the past rather than storing it, which is why practical models interleave a minority of full-attention layers that keep a growing cache for precise retrieval.

Why do KDA models keep some full-attention layers?

Because a fixed-size state cannot hold an arbitrarily long context exactly. Research on recall, such as the Zoology study of associative recall and the Repeat After Me study of copying, found that models with a fixed state fall behind attention on tasks that require retrieving a specific earlier span, and the gap grows with how much has to be remembered. Full-attention layers keep every key and value, so they can find one token far back with precision. Interleaving a minority of them with KDA layers keeps most of the decode savings while restoring exact retrieval. The share of full-attention layers is the main design dial, and the long-context speedup shrinks as that share rises.

How is KDA trained efficiently if it is a recurrence?

By splitting the sequence into chunks. Inside each chunk, the product of the per-token decay-and-erase matrices can be rewritten in the WY form, a compact way to represent a product of such rank-one corrections, which turns most of the computation into ordinary matrix multiplications that GPUs run well. Only the state passes from one chunk to the next, so the sequential part is short. This chunkwise algorithm was worked out for DeltaNet by Yang and colleagues in 2024 and extended to gated variants; Kimi Linear adds a kernel specialized to KDA's per-channel gate. At inference the same layer runs as a plain recurrence, one constant-cost state update per generated token.

Check yourself

4 questions to test what you just read

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

A KDA state stores value a under a unit-length key k. With no decay and β = 1, the model writes value b under the same key. What does reading k return, and what would plain linear attention have returned?

Show answer

The erase removes what S returns for k, which is a, and the write adds b, so reading k returns b. Plain linear attention skips the erase and just adds b k^^⊤, so the read returns a + b, the smear the delta rule exists to prevent.

A head has d_k = d_v = 128. How many numbers does its KDA state hold, and how does that compare with a softmax head's cache of keys and values after 1,000,000 tokens?

Show answer

The state holds 128 × 128 = 16,384 numbers whatever the length. A softmax head caches a key and a value per token, 2 × 128 = 256 numbers each, so a million tokens means 256 million, about 15,600 times as many, and every new token's read scans all of them.

One key channel has α near 1 and another has α = 0.5. After 10 tokens with no new writes, how much of what each channel stored is left?

Show answer

The first keeps nearly all of it, since 1^¹⁰ = 1. The second keeps 0.5^¹⁰ ≈ 0.001, about a thousandth. That is the per-channel gate at work: one channel can hold long-range information while another forgets within a few tokens, which a single scalar gate per head cannot express.

This page reports about a 6.3× decode speedup at 1M tokens, while a fixed state against a 1M-entry cache suggests a far larger gap. Why should the end-to-end number be much smaller?

Show answer

Because the hybrid's full-attention layers still scan a cache that grows with length, and decoding also spends time on parts that attention does not touch, such as the feed-forward and expert layers and the output projection. Only the KDA layers become constant-cost, so the overall speedup is capped by everything that does not change.

Where this sits

The open-weights line

Five releases you can actually download, in the order each one answered the last. This is where "what does a frontier recipe look like" is documented in public.

All 110 papers, in 19 ladders →

Finished this one? 0 / 110 Paper Breakdowns done

NextAttention Is All You Need Closest paper breakdown to this one

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

LLM Engineering

Kimi K3,explained.

The shape: the largest open model yet

The decode wall: O(n) per token

KDA: trade the cache for a fixed-size state

The delta: overwrite, don't accumulate

Attention residuals: keep the sharp lookups

The receipt

Why it matters

How Kimi Delta Attention works, step by step

Training in chunks

Hybrid layers

The gated delta rule that KDA computes

What has been measured, and what this page only claims

Where this stands, and what would settle it

A fixed state that overwrites — and never gets slower

Questions people ask about Kimi K3

How does the gated delta rule update its memory?

How is Kimi Delta Attention different from Gated DeltaNet?

Why does linear attention decode at constant cost per token?

Why do KDA models keep some full-attention layers?

How is KDA trained efficiently if it is a recurrence?

4 questions to test what you just read

A KDA state stores value a under a unit-length key k. With no decay and β = 1, the model writes value b under the same key. What does reading k return, and what would plain linear attention have returned?

A head has d_k = d_v = 128. How many numbers does its KDA state hold, and how does that compare with a softmax head's cache of keys and values after 1,000,000 tokens?

One key channel has α near 1 and another has α = 0.5. After 10 tokens with no new writes, how much of what each channel stored is left?

This page reports about a 6.3× decode speedup at 1M tokens, while a fixed state against a 1M-entry cache suggests a far larger gap. Why should the end-to-end number be much smaller?

The open-weights line

Explore the topic

More Paper Breakdowns

Kimi K3,
explained.