What's the difference between memory and weight improvement?

Memory improvement stores concrete experiences — a solved task, a learned skill — and recalls them exactly when a similar situation returns. It's instant to add, instant to remove, but it doesn't generalize: an unseen task gets no benefit. Weight improvement fine-tunes the model on filtered successful traces, which generalizes to unseen similar tasks — but it bakes what you trained on into the model, so a bad trace is hard to undo. Memory is a notebook; weights are a habit.

What safety rails does a self-improvement loop need?

Filter traces to verified-correct only; prefer reversible memory updates over irreversible weight updates when you can; canary-test a newly updated model before rolling it out; keep the ability to roll back; and watch for reward hacking and distribution drift. The key asymmetry is that you can forget a bad memory instantly but you cannot cleanly un-train a bad weight update — so the higher the permanence, the higher the bar for what you let through.

Handbooks / Self-Improving Agents

Handbook~14 min readAgentsworked math + runnable code

Handbook

Self-improving
agents.

Q: What is a self-improving agent?

A self-improving agent gets better from its own experience — it runs tasks in production, keeps a record of what worked, and folds that back into future behavior. Improvement happens along two axes: updating an external memory (storing successful trajectories or skills to reuse) or updating the model's weights (fine-tuning on filtered good traces). The first is fast and reversible; the second is slower but generalizes.

Q: Why filter traces before training?

Because a self-improvement loop trained on its own output can amplify its own mistakes. Filtering keeps only verified-correct traces, so the model learns from successes rather than confidently-wrong runs. Skipping the filter is how a loop drifts or collapses. It's the same verifier-quality problem as reinforcement learning: what you reinforce is what you get, so you reinforce only what you can confirm is right.

The dream is an agent that gets better the more you use it — one that runs in production, notices what worked, and folds it back in. It's real, and it's not magic: improvement happens on exactly two axes. An agent can update its memory (write down what it learned) or update its weights (retrain on what it did). One is a notebook, instant and erasable; the other is a habit, powerful and permanent. Knowing which to use — and how to keep the loop from teaching itself the wrong thing — is the whole discipline.

The improvement loop Memory: fast and reversible Weights: slow and general The trade-off & safety rails Why it matters

The loop: act, keep what worked, improve

Every self-improving agent runs the same cycle: act on real tasks, evaluate which attempts succeeded, and fold the successes back into future behavior. This isn't new — it's the mechanism family behind STaR (finetune on self-generated correct rationales), Voyager (grow a skill library), and Reflexion (write lessons to memory after a failure). What's new is doing it online, on production traces, on a fast cadence.

The design decision that everything hinges on is where the improvement lands. "Fold it back in" can mean two very different things — write to a memory store, or update the model's weights — and they have opposite properties. Get the choice right and the loop compounds; get it wrong and it either fails to generalize or quietly corrupts the model.

Memory: instant, exact, erasable — but doesn't generalize

The first axis is an external memory: store a solved trajectory, a working snippet, a learned skill, and retrieve it when a similar situation returns. Its virtues are speed and reversibility — you can add a memory in one write and delete a bad one just as fast, with no retraining.

memory: recall(store, task) = 1 if task ∈ store else 0 · add/forget = instant, reversible

But it's exact, not generalizing: a stored solution helps that task (and near-duplicates) and does nothing for a genuinely unseen one. Memory is a lookup, not a lesson.

That makes memory the right tool for specific, recurring knowledge — this customer's preferences, this repo's build command, this API's quirk — and for anything you might need to revoke. When the improvement should be surgical and undoable, it goes in memory.

Weights: generalizing, permanent — and risky

The second axis updates the model itself: fine-tune on filtered successful traces. Unlike memory, a weight update generalizes — train on enough solved instances of a task type and the model gets better at unseen instances of it too. That's the improvement memory can't buy.

weights: accuracy climbs on unseen similar tasks · bounded by data quality · hard to reverse

The catch is permanence. You can forget a memory; you cannot cleanly un-train a weight update. Whatever passed your filter is now baked into the model's habits.

Which is why the filter is everything. Train on verified-correct traces only — the STaR move — and the loop bootstraps toward a ceiling. Skip the filter and you're training on your own confident mistakes, and the loop can drift or collapse. It's the same lesson as RL environment design: what you reinforce is what you get, so reinforce only what you can confirm.

The trade-off, and the rails that keep it safe

	Memory update	Weight update
Speed	instant	slow (a training run)
Generalization	none (exact recall)	yes (to unseen similar tasks)
Reversibility	instant (forget)	hard (bake-in)
Best for	specific, recurring, revocable	broad skills worth making permanent

The safety rails follow directly from that last asymmetry — permanence raises the stakes:

Filter first. Only verified-correct traces enter any update. This is the single most important rail.
Prefer reversible. When memory can do the job, use it — a mistake you can delete beats one you can't.
Canary before rollout. Test a freshly-updated model on a held-out set; keep the ability to roll back.
Watch for drift & gaming. A loop optimizing its own signal can find and exploit its holes — monitor for it.

The key idea

Match permanence to confidence. Reversible memory for anything you're unsure about or might revoke; permanent weight updates only for filtered, verified, broadly-useful behavior you'd stake the model on. The runnable model below makes the two axes' properties concrete.

Why it matters

Self-improvement is moving from research to production — coding agents that retrain on a short cadence from their own accepted edits, assistants that accumulate skills across sessions. The payoff is compounding: an agent that's a little better every day. The risk is also compounding: a loop that learns from its own errors degrades every day. The difference between the two is entirely in the discipline above — filter, prefer reversible, canary, monitor.

The clean mental model to carry: an agent has a notebook and a set of habits. Write to the notebook freely; change habits only with proof. Everything else — online RL, skill libraries, trace pipelines — is machinery in service of that one judgment call about where a lesson belongs.

Two axes of improvement, side by side

The whole design choice is in the properties. Memory recalls a stored task exactly and can be added or forgotten instantly — but gives an unseen task nothing. Weights improve unseen similar tasks (they generalize) but bake in whatever passed the filter, which is why you keep only verified-correct traces and the flywheel climbs toward a ceiling. Watch memory recall a task and then forget it, watch weights lift an unseen task where memory can't, and watch the safety filter drop the bad traces.

CPython · WebAssembly

# Two axes of self-improvement, with opposite properties.

# MEMORY: an external store. Exact recall, instant add/remove, NO generalization.
def mem_success(store, task):  return 1.0 if task in store else 0.0
def remember(store, task):     return store | {task}
def forget(store, task):       return store - {task}

# WEIGHTS: fine-tune on FILTERED good traces. Generalizes to unseen similar tasks; bounded; hard to reverse.
def filtered_good(traces):     return [t for t in traces if t["correct"]]   # safety rail: keep only correct
def weights_accuracy(good_count, base=0.5, lr=0.1, ceiling=0.95):
    return min(ceiling, base + lr * good_count)   # improvement generalizes across the task population

# --- memory: exact but narrow ---
store = remember(set(), "invoice_task")
print("memory: recall stored task   ->", mem_success(store, "invoice_task"))
print("memory: recall UNSEEN task    ->", mem_success(store, "refund_task"), "(no generalization)")
print("memory: forget then recall    ->", mem_success(forget(store, "invoice_task"), "invoice_task"), "(reversible)\n")

# --- weights: general but permanent, and only as good as the filter ---
traces = [{"correct": True}, {"correct": False}, {"correct": True}, {"correct": True}, {"correct": False}]
good = filtered_good(traces)
print(f"weights: {len(traces)} traces -> {len(good)} pass the correctness filter")
print(f"weights: accuracy on UNSEEN similar tasks after {len(good)} good traces -> {weights_accuracy(len(good)):.2f}")
print(f"weights: flywheel is bounded -> {weights_accuracy(100):.2f} ceiling (you can't un-train a bad trace)")

Frequently asked

Quick answers

What is a self-improving agent?

One that gets better from its own experience — running tasks, keeping what worked, and folding it back into future behavior via memory updates or weight updates. The first is fast and reversible; the second generalizes but is permanent.

Memory vs weight improvement?

Memory stores concrete experiences for exact recall — instant, erasable, no generalization. Weights fine-tune on filtered traces — generalizes to unseen similar tasks, but bakes in what you trained on. A notebook versus a habit.

Why filter traces before training?

A loop trained on its own output amplifies its own mistakes. Filtering to verified-correct traces means the model learns from successes, not confident errors — the same what-you-reinforce-is-what-you-get rule as RL.

What safety rails are needed?

Filter to correct-only; prefer reversible memory over irreversible weights; canary-test and keep rollback; watch for drift and reward hacking. You can forget a bad memory but can't cleanly un-train a bad weight update.

Finished this one? 0 / 160 Handbooks done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

AI Agents & Tools

Self-improving
agents.

The loop: act, keep what worked, improve

Memory: instant, exact, erasable — but doesn't generalize

Weights: generalizing, permanent — and risky

The trade-off, and the rails that keep it safe

Why it matters

Two axes of improvement, side by side

Quick answers

What is a self-improving agent?

Memory vs weight improvement?

Why filter traces before training?

What safety rails are needed?

Explore the topic

More Handbooks

Explore more from Vibe Engines

Self-improvingagents.

The loop: act, keep what worked, improve

Memory: instant, exact, erasable — but doesn't generalize

Weights: generalizing, permanent — and risky

The trade-off, and the rails that keep it safe

Why it matters

Two axes of improvement, side by side

Quick answers

What is a self-improving agent?

Memory vs weight improvement?

Why filter traces before training?

What safety rails are needed?

Go deeper on agents that learn

STaR →

Voyager →

Reflexion →

Explore the topic

More Handbooks

Explore more from Vibe Engines

Get the next one in your inbox.

Self-improving
agents.