Kimi K3 (reported July 2026) is an open-weight mixture-of-experts language model of roughly 2.8T total parameters — described as the largest open model released to date — with a 1M-token context. Its headline architectural feature is Kimi Delta Attention (KDA), a linear-attention variant that decodes at constant per-token cost regardless of context length, reported at about 6.3× faster decode at a 1M-token context.

What is Kimi Delta Attention (KDA)?

KDA is a linear-attention mechanism built on the delta rule. Instead of caching a key and value for every past token and reading the whole cache each decode step (softmax attention, O(n) per token), it keeps a single fixed-size recurrent state matrix that it updates per token. The delta rule makes the update corrective: writing a key overwrites its old associated value rather than accumulating on top of it, which is what lets a fixed-size state stay accurate.

Why does linear attention decode faster at long context?

Softmax attention must read every stored key and value to produce the next token, so per-step decode cost grows linearly with the number of tokens already generated — at a million tokens that read dominates. KDA's recurrent state is a fixed size (roughly d×d) no matter how long the context is, so each decode step does the same constant amount of work. The saving grows with context length.

Paper Breakdowns / Kimi K3

Paper 103~9 min readMoonshot AI · 2026

Paper Breakdown

Kimi K3,
explained.

Q: What are attention residuals?

Pure linear attention trades some exact-recall ability for its speed — a fixed-size state can blur precise lookups over very long contexts. K3 keeps a minority of full-attention layers (or residual paths) alongside the KDA layers, so the model retains sharp recall where it matters while getting constant-cost decode from the bulk of its layers. It is a hybrid, not a wholesale replacement of softmax attention.

Every paper so far fought the KV cache's memory. K3 goes after the other bill — decode speed. Softmax attention has to re-read every token it has ever seen to write the next one, so generation gets slower the longer the context grows. Kimi Delta Attention throws that model out: keep one fixed-size state, update it per token with a correction that overwrites instead of piling up, and every decode step costs the same whether you're 1,000 tokens in or a million.

Video breakdown

The animated walkthrough is in production.

Read the full breakdown below in the meantime ↓

Prefer video? Browse published breakdowns →

The shape: 2.8T open MoE The decode wall Kimi Delta Attention The delta correction Attention residuals The receipt Why it matters

The shape: the largest open model yet

K3 is the sequel to Kimi K2, and it goes bigger: roughly 2.8T total parameters in a sparse mixture-of-experts, released with open weights — reported as the largest open model to date — carrying a 1M-token context. K2's contribution was training stability (MuonClip taming attention-logit blowups). K3's is a different axis entirely: the cost of generating into that million-token window.

Scale and context length are the easy headlines. The hard one is hidden in the word "1M": at that length, the thing that kills you is not storing the context — it's the fact that softmax attention re-reads all of it on every single token it generates. That is the wall K3's architecture is built to knock down.

The decode wall: O(n) per token

To generate token number n+1, softmax attention scores the new query against the key of every previous token and mixes their values. That is a read over the whole KV cache — and the cache has n entries. So the work per decode step grows linearly with how much you've already written. Multiply by a million tokens and decode, not memory, becomes the bottleneck.

softmax decode: read n keys/values per step → cost per token ∝ n (grows without bound)

The cache you must scan keeps getting longer as the model talks. Every new token is more expensive than the last.

This is the same wall Mamba and the linear-attention family were built to climb. K3's answer, KDA, is in that lineage — but with a twist that fixes linear attention's oldest weakness.

KDA: trade the cache for a fixed-size state

Kimi Delta Attention replaces the growing cache with a single fixed-size recurrent state — a matrix S of about d×d. Each token folds itself into S; to read, you multiply S by the query. The state is the same size no matter how long the context is, so every decode step does the same constant amount of work.

Softmax reads the whole past · KDA folds it into one fixed state

Token n+1 query

→

S ← update(S, k, v)
fixed d×d state

→

o = S·q
read is O(d²), not O(n)

→

Same cost at 1M
as at 1k

KDA decode: read d² state elements per step → cost per token = constant in n

Fixed state, fixed work. The runnable below shows the softmax read growing with context while the KDA read never moves.

The delta: overwrite, don't accumulate

Plain linear attention has a fatal flaw: it accumulates. Write key k with value v, then write the same k with a new v', and the state holds v + v' — a smear of every value ever associated with that key. Over a long context the memory turns to mud.

The delta rule fixes this. Before writing, it subtracts out what the state currently returns for that key, so the update is a correction toward the new value, not a pile-on:

S ← S − β·(S·k − v)·kᵀ · with β=1 and a clean key: read(k) returns v', the latest — not v+v'

The name is literal: it writes the delta between what's stored and what should be. That correction is what lets a fixed-size state stay accurate instead of blurring.

plain linear: write (k,a) then (k,b) → read k = a+b (smeared)
delta rule: write (k,a) then (k,b) → read k = b (overwritten, correct)

Attention residuals: keep the sharp lookups

Even with the delta correction, a fixed-size state gives up something: exact recall at extreme range. Softmax attention can point at one specific token a million steps back with perfect precision; a compressed recurrent state can blur it. So K3 doesn't go all-in on KDA.

It keeps a minority of full-attention layers (and residual paths) interleaved with the KDA bulk — a hybrid. The KDA layers give constant-cost decode across most of the network; the few softmax layers preserve the sharp, exact lookups that pure linear attention loses. You pay full-attention cost only where it earns its keep.

The key idea

Linear attention isn't a drop-in replacement for softmax — it's a budget. Spend it on the layers where approximate mixing is fine (most of them), and keep exact attention for the few layers that need to find a needle. The hybrid is what makes a 1M-token model both fast to decode and still able to remember precisely.

The receipt

Item	Reported
Total parameters	≈ 2.8T (MoE) — largest open model to date
Context length	1M tokens
KDA decode speedup @ 1M	≈ 6.3×
Attention	KDA (linear, delta rule) + residual full-attention layers
Weights	open

The 6.3× is an end-to-end decode figure at a million tokens — smaller than the raw asymptotic gap between a fixed state and an n-length cache, because the residual full-attention layers still pay the softmax price, and because real decode is more than attention. The runnable model below reproduces the mechanism: constant KDA read versus a softmax read that climbs with every token, and the delta correction that keeps the fixed state honest.

Why it matters

K3 is the clearest sign that linear attention has graduated from research curiosity to frontier production. For years the story was "softmax attention is expensive but irreplaceable." K3 replaces most of it — keeping just enough for exact recall — and ships it at the largest open scale yet. Paired with DeepSeek-V4, which keeps softmax attention but compresses and sparsifies it, you can see the field forking into two answers to the same long-context question.

The K2 → K3 arc is telling: K2 made attention stable at scale; K3 makes it cheap at length. And the delta rule at its core is a decades-old idea from associative memory, revived — a reminder that the linear-attention lineage running through Mamba keeps finding new ways to make a fixed-size state do a growing cache's job.

A fixed state that overwrites — and never gets slower

KDA's two claims are checkable in a dozen lines. First, the delta rule: write a key/value into a fixed-size state, then overwrite the same key — a correct memory returns the latest value, while plain linear attention would hand back the sum of both. Second, constant-cost decode: the state is d×d regardless of context, so a KDA read costs the same at a million tokens as at a thousand, while softmax's read grows with every token. Change the state dim, the keys, the values.

CPython · WebAssembly

# Kimi Delta Attention as a fixed-size associative memory.
# State S is a d x d matrix; keys are one-hot here so S is an exact lookup table.
D = 4

def onehot(i, d=D):
    v = [0.0] * d; v[i] = 1.0; return v
def zeros():
    return [[0.0] * D for _ in range(D)]
def read(S, k):                                  # o = S . k
    return [sum(S[r][c] * k[c] for c in range(D)) for r in range(D)]

def delta_write(S, k, v, beta=1.0):              # S <- S - beta*(S.k - v).k^T   (corrective)
    Sk  = read(S, k)
    err = [Sk[r] - v[r] for r in range(D)]       # what's stored minus what we want
    return [[S[r][c] - beta * err[r] * k[c] for c in range(D)] for r in range(D)]

def linear_write(S, k, v):                       # plain linear attention: S <- S + v.k^T (accumulates)
    return [[S[r][c] + v[r] * k[c] for c in range(D)] for r in range(D)]

# --- decode cost: fixed state vs a cache that grows with context ---
def state_elems(d):        return d * d          # KDA: fixed recurrent state
def softmax_kv_reads(n, d):return 2 * n * d      # softmax: read K and V for all n tokens, every step
def delta_reads(d):        return d * d          # KDA: read only the fixed state

k1, k2 = onehot(1), onehot(2)
a, b = [9.0, 8.0, 7.0, 6.0], [1.0, 2.0, 3.0, 4.0]

S = delta_write(delta_write(zeros(), k1, a), k1, b)   # write k1=a, then overwrite k1=b
print("delta  read(k1):", read(S, k1), " -> latest value b (old a gone)")
L = linear_write(linear_write(zeros(), k1, a), k1, b)
print("linear read(k1):", read(L, k1), " -> a+b, smeared\n")

print(f"{'context n':>12}{'softmax reads':>15}{'KDA reads':>12}{'speedup':>10}")
for n in (1_000, 128_000, 1_000_000):
    sm = softmax_kv_reads(n, 64)
    print(f"{n:>12,}{sm:>15,}{delta_reads(64):>12,}{sm // delta_reads(64):>9,}x")

Frequently asked

Quick answers

What is Kimi K3?

A ~2.8T-parameter open MoE with a 1M-token context — reported as the largest open model to date — whose headline feature is Kimi Delta Attention, a linear-attention variant that decodes at constant per-token cost (~6.3× faster at 1M tokens).

What is Kimi Delta Attention?

A linear-attention mechanism built on the delta rule: a fixed-size recurrent state replaces the growing KV cache, and each write corrects the state toward the new value (overwrite) instead of accumulating — so a small state stays accurate.

Why is decode faster at long context?

Softmax attention reads the whole KV cache each token, so per-step cost grows with context. KDA's state is a fixed d×d, so every decode step does the same constant work no matter how long the context is.

What are attention residuals?

A hybrid: K3 keeps a few full-attention layers among the KDA layers, so it retains sharp exact recall (which pure linear attention blurs) while getting constant-cost decode from the bulk of the network.

Finished this one? 0 / 108 Paper Breakdowns done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

LLM Engineering

Kimi K3,
explained.

The shape: the largest open model yet

The decode wall: O(n) per token

KDA: trade the cache for a fixed-size state

The delta: overwrite, don't accumulate

Attention residuals: keep the sharp lookups

The receipt

Why it matters

A fixed state that overwrites — and never gets slower

Quick answers

What is Kimi K3?

What is Kimi Delta Attention?

Why is decode faster at long context?

What are attention residuals?

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Kimi K3,explained.

The shape: the largest open model yet

The decode wall: O(n) per token

KDA: trade the cache for a fixed-size state

The delta: overwrite, don't accumulate

Attention residuals: keep the sharp lookups

The receipt

Why it matters

A fixed state that overwrites — and never gets slower

Quick answers

What is Kimi K3?

What is Kimi Delta Attention?

Why is decode faster at long context?

What are attention residuals?

Go deeper on long-context attention

Kimi K2 →

DeepSeek-V4 →

Mamba →

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Get the next one in your inbox.

Kimi K3,
explained.