The whole design, in writing
Learn AI system design by designing an AI code editor like Cursor around the edit problem. An interactive guide covering why retyping 4,000 tokens to change 30 costs 80 seconds, the plan model and the apply model, why the apply step rewrites the whole file instead of writing a diff, speculative edits that turn 80 seconds into 2.2, a context index kept fresh by a Merkle tree, the Tab model priced against silence, the agent loop with checkpoints, and the freshness check that catches a right-looking edit made against a file that has moved.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
"Design an AI code editor" — why is this even hard?
You select a line and type make the timeout 30 seconds, and a second later the file has changed and nothing else has. A model did that by writing text, in small pieces called tokens. Say the file is 400 lines at about 10 tokens: that is 4,000 tokens, and at 20 ms a token that is 80 seconds, to change about 30 of them. And you will be watching.
Never pay for what did not change. Plan first, rewrite the whole file so there are no line numbers to get wrong, use the old file as the draft so copying is nearly free, find files through an index that only re-embeds what changed, speak only when a suggestion beats silence, loop with checkpoints, and check the file has not moved.
What the new pieces do
- Youclient
- The person typing. “Make the timeout 30 seconds” is the whole input; everything else is the editor deciding how to turn it into a change that lands in one file without disturbing the rest.
- The Filestore
- The 400-line file being changed. About 4,000 tokens, of which about 30 actually change. Retyped at 20 ms a token it would take 80 seconds.
Step 1 · Plan and apply
Two models, two jobs
One request, and the model has to both decide what to change and write the file. A big chat model is good at the first and slow at the second. If one model does both, which of the two do you give up?
How should the editor split the work of making a change?
It is smart but slow, so you wait for the large model to produce every token of a file that is mostly unchanged.
Planning needs intelligence and applying needs speed. Splitting them lets each do what it is good at.
Choosing what to change across a codebase is the hard part, and a small model will often choose badly.
The Planner, a big chat model, sketches the change in prose and fragments. The Apply Model, smaller and fast, turns that sketch into the real file. The hand-off is the price.
Why this piece earns its place
A plan is cheap to produce and cheap to be wrong about, so it belongs to the model that is good at deciding. Writing out a four-thousand-token file token by token is a different skill: it wants speed and exactness, not breadth. Splitting the two means the expensive model never has to emit the whole file, and the fast model is trained for exactly one thing, turning a sketch plus the old file into the new file. The trade is honest and worth stating: a hand-off between two models can lose a line, because the plan is a sketch and the applier has to fill in what the sketch left out. The rest of the design is mostly about making the second half of that hand-off fast and exact.
- 2models
- 1plan
- 1real file
What the new pieces do
- Plannerplan
- A large chat model that sketches the change: what to alter and roughly where. It is smart and slow, so it never writes the final file.
Step 2 · The rewrite
The whole file, not a diff
The apply model has to hand back a changed file. The compact way to describe a change is a diff: line 212, remove this, add that. It is a few tokens instead of four thousand. Why would anyone choose to write the whole file?
What should the apply model output?
It is small, but it carries line numbers, and models are poor at counting lines. Three edits carry six numbers.
That leaves a person or a script to guess where they go, which is the same line-number problem in another form.
Models see far more whole files than diffs, and a rewrite carries nothing to miscount.
The apply model rewrites the whole file. A diff of 3 edits carries 6 line numbers; at 2% wrong each, 1 − 0.98⁶ is 11.4% wrong. A rewrite carries none, at the price of producing or verifying every token.
Why this piece earns its place
The diff looks like the smart answer and fails for a quiet reason: it asks the model to do something it is bad at. Counting lines in a long file is exactly the kind of exact bookkeeping a language model gets wrong a small percentage of the time, and a small percentage compounds: three edits are six numbers, and six chances at two percent is about one edit in nine landing in the wrong place, without any error being raised. Models have also seen vastly more whole files than diffs, and a rewrite gives the answer more forward passes to get right. The cost is that you now owe every token, four thousand of them, and eighty seconds at twenty milliseconds each. That bill is exactly what the next step is for.
- 6line numbers in a 3-edit diff
- 11.4%wrong at 2% each
- 0numbers in a rewrite
What the new pieces do
- Apply Modelmodel
- A smaller, fast model that turns the plan into the real file. It rewrites the whole file instead of writing a diff, because models are poor at line numbers.
Step 3 · Speculative edits
The old file is the draft
The rewrite is right and slow: 4,000 tokens, 80 seconds, and 3,970 of them are exactly what was already there. The model generates one token per pass. How do you stop paying a pass for each token it already knows?
How do you make a whole-file rewrite fast?
Copying is nearly free in bulk: only the changed tokens need real generation.
A smaller model is faster per token, but it is still paying a pass for every one of the 4,000 tokens.
The model has to confirm they are unchanged, and a copied run that was silently altered is the failure you were avoiding.
Speculative edits: the model checks the old file as a draft, about 50 tokens per pass. 30 new tokens plus about 80 copy passes is 110 passes: 2.2 seconds against 80, about 36×. Cursor reports around 1,000 tokens a second and about 13× over standard serving; that is theirs, not ours.
Why this piece earns its place
A language model is slow because it generates one token per pass. It can, however, verify many tokens in one pass: feed it a guess and it tells you how many in a row match what it would have written. The old file is a perfect guess for nearly everything, so the model accepts runs of fifty at a time and generates fresh only where the edit is. That is where the arithmetic comes from: thirty new tokens, plus about eighty passes for the other 3,970, a hundred and ten passes at twenty milliseconds. The output is identical to a full rewrite, which is the whole point. The honest limit is just as important: the speedup is only as good as the match. Rewrite most of the file and nothing matches, so there is nothing to copy and the time returns to eighty seconds.
- 110passes, not 4,000
- 2.2 sagainst 80 s
- 36×faster, same file
What the new pieces do
- Speculationspeed
- Uses the old file as the draft and verifies it about fifty tokens at a time in one pass. Only the changed tokens cost real time: about 110 passes instead of 4,000.
Step 4 · The context index
Finding the file, cheaply and freshly
So far you pointed at one file. A real request touches code you have never opened. The editor has to find the right chunks, which means vectors for every chunk of a repo, and a repo changes all day. How do you keep that index current without redoing all of it?
A 10,000-file repo has 50,000 chunks and you edited 12 files. What should re-embed?
At 500 chunks a second that is 100 seconds of work for a 12-file change.
A Merkle tree finds the changed files without comparing everything: 60 chunks, about 0.12 seconds.
Then every question starts with a search through ten thousand files, which is the cost the index exists to avoid.
The Context Index: code is cut into chunks, embedded as vectors and stored with masked paths. A Merkle tree of file hashes finds what changed, so 12 files is 60 chunks, 0.12 s, not 50,000 chunks and 100 s. The price: it is stale for minutes between syncs.
Why this piece earns its place
Retrieval is what lets a request about “the auth flow” find the three files that matter. The index is a store of vectors, one per chunk, so that meaning, not just spelling, can be searched. Its problem is upkeep. Re-embedding everything on every edit is wasteful by four orders of magnitude, so each file gets a hash, each folder a hash of its children’s hashes, up to one root. Compare roots and you descend only where they differ, finding the twelve changed files without touching the other nine thousand nine hundred eighty-eight. Public write-ups describe a sync roughly every five minutes and only vectors and masked paths stored server-side, not source; I attribute those to the write-up. That gap, minutes, is the cost, and it is the setup for the last step.
- 50,000chunks in the repo
- 60chunks re-embedded
- 0.12 sagainst 100 s
What the new pieces do
- Context Indexstore
- Vectors for every code chunk, kept fresh by a Merkle tree of file hashes so only changed files are re-embedded. It is stale for minutes between syncs.
Step 5 · Tab
Speak only when it beats silence
Tab predicts your next edit while you type. Every suggestion that appears and gets rejected costs you a glance and a keystroke. Showing nothing costs nothing. What should decide whether it speaks?
Rewards are +0.75 accepted, −0.25 rejected and 0 not shown. When is a suggestion worth showing?
Only when it is accepted often enough. Below the break-even the average is negative.
That throws away a lot of useful suggestions. The threshold follows from the reward, not from a round number.
0.75p − 0.25(1 − p) = p − 0.25, which is zero at exactly 25%.
Tab shows a suggestion only when its expected reward beats silence. With +0.75, −0.25 and 0, the break-even is exactly 25%: at 40% a suggestion is worth +0.15, at 20% it is worth −0.05, so it stays quiet. Cursor reports 21% fewer suggestions and a 28% higher accept rate; attributed, not endorsed.
Why this piece earns its place
The reward arithmetic is the whole design. If an accepted suggestion is worth three-quarters of a point and a rejected one costs a quarter, the expected value of showing is p times 0.75 minus one-minus-p times 0.25, which simplifies to p minus a quarter. It is positive above twenty-five percent and negative below, so the threshold is not a tuning knob, it follows from what you decided accepts and rejects are worth. Training the model online on real accepts and rejects teaches it where that line is for you. The cost is the thing it learns from: what you click, which is not always what is right, and a model trained to be welcome can learn to be agreeable.
- +0.75accepted
- −0.25rejected
- 25%break-even
What the new pieces do
- Tabservice
- The next-edit suggestion model. It is trained on accepts and rejects and shows a suggestion only when its expected reward beats silence.
Step 6 · The agent loop
Many files, and a way back
One request, ten files. The editor has to plan, edit, run, read the result and go again, many times. Each step can be wrong, and your own work is somewhere in those files. What keeps a long edit from being a leap of faith?
Each step of an agent is right 95% of the time. How often is a 10-step run right end to end?
The steps compound: you need all of them to be right.
Ten steps at 95% each multiply, which is why a long run needs review and an undo.
Five wrong steps in ten would be a 50% error rate per step. Here each is 5%, but they compound.
The Agent Loop carries one request across many files. Checkpoints are local snapshots of its changes, so any of it can be undone, and you read a diff. Ten steps at 95% is about 60% right end to end, which is why the diff is for reading.
Why this piece earns its place
An agent turns a request into a sequence: find the files, plan an edit, apply it, run something, read the error, go again. The arithmetic of a sequence is unkind, because the end-to-end success is the product of each step, and ninety-five percent ten times is about sixty. That is not an argument against loops; it is an argument for making each step reviewable and reversible. A checkpoint is a local snapshot taken before the agent’s edits, so one click returns to before. Two honest limits: checkpoints are not version control, and they do not track edits you made yourself, so they undo the agent’s work and nothing else. The diff is what a human reads at the end, which is where a person is worth the most.
- 0.95¹⁰is about 60%
- localsnapshots
- 1diff you read
What the new pieces do
- Agent Loopservice
- Carries one request across many files in many steps. Errors compound: ten steps at 95% each is about 60%.
- Checkpointsundo
- Local snapshots of the agent’s changes so a multi-file edit can be undone. Not version control, and they do not track your own manual edits.
Step 7 · Freshness
The hidden slot: is the file still the file?
Six of six parts are built and every check is green. But the plan was made from an index that is minutes old, and someone moved a function in between. The edit lands, the file compiles, and the change sits on a line that is no longer the line you meant. What is the slot you only find by running it?
The edit is right against the index and the index is a few minutes old. What should happen before it lands?
A mismatch means the file moved, so the plan is re-read instead of applied blindly.
Minutes is long enough for a file to move, and the failure raises no error.
That is the 100-second cost the Merkle tree was built to avoid.
The Freshness Check: hash the file as it is now before an edit lands, run the same checks as a human edit, and show a diff you read. A mismatch means the file moved, so re-read it rather than apply a plan made against yesterday’s file.
Why this piece earns its place
The failure here is not a crash, and that is what makes it the interesting one. A right-looking edit, made against a file that has moved, applies cleanly and passes a glance. Every earlier part was about writing less or seeing more; this one is about asking whether what was seen is still true. The index is stale by design, because syncing continuously costs more than it is worth, so the staleness has to be caught at the last moment, where it is cheap: one hash of one file. Running the same checks a human edit would face, formatting, types, tests, means the editor’s change is held to the same bar as yours. The diff is the last guard, and the cheapest, because a person reads it before it is kept.
- 1hash per edit
- samechecks as a human edit
- 1diff you read
What the new pieces do
- Freshness Checkgate
- The hidden seventh slot. The index is minutes old and files move, so hash the file as it is now before an edit lands, run the same checks as a human edit, and show a diff you read.
The payoff
You designed an AI code editor
From “retype the file” to a build ladder: a planner and an apply model, a rewrite instead of a diff, speculative edits from the old file, a context index kept fresh by a Merkle tree, a Tab model priced against silence, an agent loop with checkpoints, and a freshness check, because the index is always a little old.
Now ask for a diff and watch one edit in nine land in the wrong place, rewrite everything and watch the speedup vanish, then skip the freshness check. Those three are the whole trade-off: not paying for what did not change buys speed, and every part has a price.
Everything you assembled, in order
- Plan and apply — a big model sketches, a fast one writes the file
- Rewrite — the whole file, no line numbers; 6 numbers at 2% is 11.4% wrong
- Speculation — old file as draft; 110 passes, 2.2 s, not 4,000 and 80 s
- Context index — vectors + Merkle tree; 60 chunks, not 50,000
- Tab — shown only above 25% accept chance
- Agent loop — checkpoints and a diff; ten steps at 95% is 60%
- Freshness check — hash the file now; same checks as a human edit
- The numbers — 80 → 2.2 s · 100 → 0.12 s · 25% break-even
