18 stations. 3 tracks. From the messages API and tool calling to memory, subagents, and evals — build the runtime behind every agent, at your own pace.
A harness talks to a model through one primitive: a list of messages with roles (system, user, assistant, tool) exchanged over request and response. Learn the shape of a completion call, how turns accumulate, and why everything the model knows this turn is exactly what you put in the array. This is the socket the whole harness is built on.
Roles & message turns
Request / response shape
Stateless calls
The assistant turn
Build itTrace one agent turn as a raw messages array. What is in it, and what does the model return — text, or a tool call?
✓ CheckpointExplain what the message list actually is on each turn, and why the model has no memory beyond what you resend.
2
Tokens, Context & Streaming
F2 · Beginner · 55 min
The model reads and writes tokens, not characters, inside a fixed context window — and streams them back one at a time. Learn how text becomes tokens, why the window is a hard budget every message competes for, and how streaming lets a harness react before generation even finishes.
Free-form text is unusable as an API. Force the model to emit valid JSON against a schema — through JSON mode, tool calls, or constrained decoding — so your code can parse it every time. Learn to design output schemas and to recover gracefully when the model still drifts.
JSON mode
Output schemas
Constrained decoding
Parse & repair
Build itA model must return {intent, args}. Design the schema and one guard for when it returns prose instead of JSON.
✓ CheckpointExplain why asking for JSON is not the same as guaranteeing JSON, and what your harness does on the turn it returns prose.
4
System Prompt & Steering
F4 · Intermediate · 60 min
The system prompt is the harness's constitution — the standing instructions, tools, format rules and persona the model obeys every turn. Learn to write one that is specific, ordered, and stable enough to cache, and why most agent misbehaviour is a prompt bug, not a model failure.
The same prompt can give different answers. Learn temperature, top-p and top-k — the dials that shape the model's randomness — and how seeds and low temperature buy the reproducibility a harness needs for tools and tests, without going brittle.
Temperature, top-p, top-k
Greedy vs sampled
Seeds & reproducibility
Determinism for tools
Build itYour agent must call a tool the same way every time for a given input. What sampling settings get you there — and what do you lose?
✓ CheckpointExplain why temperature 0 is not the same as deterministic, and what else varies.
Every turn re-sends the same long prefix — system prompt, tools, history. Prompt-prefix caching lets the provider skip re-processing that stable head, cutting latency and cost dramatically. Learn what makes a prefix cacheable and why cache-friendliness shapes how you order a prompt.
Prefix caching
Cache-friendly ordering
Latency & cost wins
Cache invalidation
Build itReorder a prompt so the stable parts sit first and the volatile parts last. Why does that one change slash the bill?
✓ CheckpointExplain what has to be stable at the FRONT of a prompt for caching to help, and what invalidates it.
Tool calling is what turns a text model into an agent: you describe functions, the model decides when to call one and emits structured arguments, your code runs it and feeds the result back. Learn the full round-trip — and why the model never runs anything itself, the harness does.
Tool definitions
Model-emitted calls
Executing & returning results
Multi-tool turns
Build itGive a model a get_weather tool. Walk the round-trip: what the model emits, what your code does, what goes back into the messages.
✓ CheckpointExplain why validating tool arguments matters more than the model's confidence, and what happens the first time it invents a plausible parameter.
Plan, act, observe, repeat. The agent loop feeds the model its own tool results and lets it decide the next step until the task is done. Learn loop control — stopping conditions, step limits, and detecting when the model is stuck or looping — the beating heart of every harness.
Plan / act / observe
Stopping conditions
Step limits
Loop & stuck detection
Build itDesign the stop condition for a research agent. How do you tell "done" from "gave up" from "looping forever"?
✓ CheckpointState the stopping conditions of your loop. If the only one is a step cap, the agent does not know when it has succeeded.
The tool schema is the contract the model codes against; a vague one is the top cause of tool-calling failures. Learn to write names, descriptions, typed parameters and required flags that make the model pick the right tool and fill it correctly — and to keep the toolset small and distinct.
Names & descriptions
Typed params & enums
Required vs optional
Minimal, distinct toolsets
Build itA model keeps calling the wrong tool between two similar ones. Rewrite their schemas so the choice is unambiguous.
✓ CheckpointExplain why a tool description is a prompt, and what a vague one does to the model's choices.
The window fills as the loop runs — tool outputs, history, retrieved docs — and once it overflows the agent forgets. Learn compaction: summarizing old turns, trimming tool noise, and deciding what to keep so the agent stays coherent over long tasks without blowing the budget.
Compaction & summarization
Trimming tool output
What to keep vs drop
Windowing history
Build itA 40-step agent overflows its window at step 25. Design a compaction rule that keeps it coherent all the way to step 40.
✓ CheckpointExplain why appending everything degrades the agent rather than informing it, and what you compact first.
Context is short-term memory that dies with the session; a harness needs more. Learn long-term memory — writing facts to a store and retrieving the relevant ones back into context — and the split between scratchpad, episodic, and semantic memory that lets an agent learn across runs.
Short vs long-term memory
Write & retrieve
Scratchpad / episodic / semantic
Memory into context
Build itAn agent should remember a user's preferences across sessions. What do you store, and how does it come back into the next prompt?
✓ CheckpointExplain what you evict first when memory exceeds budget, and why recency alone is a weak rule.
The moment an agent runs code or commands, safety becomes the harness's problem. Learn to sandbox execution — isolation, resource limits, network and filesystem boundaries — so a model's mistake, or an injection, can't escape. The harness, not the model, owns the blast radius.
Isolation & containers
Resource limits
Network / FS boundaries
Containing failure
Build itYour agent executes model-written Python. List three things the sandbox must prevent — and how each is enforced.
✓ CheckpointExplain what a sandbox must deny by default, and why an allowlist of commands is not enough on its own.
Not every action should run unattended. Learn to gate dangerous tools behind human approval, scope what an agent may touch, and design the permission model — allow, ask, deny — that lets a harness act autonomously where it is safe and pause where it is not.
Allow / ask / deny
Human-in-the-loop
Action scoping
Approval UX
Build itAn agent can read files freely but must ask before deleting. Design the permission check that enforces that split.
✓ CheckpointExplain which actions should require approval and why “anything destructive” is harder to define than it sounds.
One agent, one context, hits limits. Learn to spawn subagents — each with its own window and tools — and orchestrate them: delegating a subtask, collecting the result, and keeping the parent's context clean. The pattern behind planners, workers, and multi-agent systems.
Subagents & delegation
Isolated sub-contexts
Result aggregation
Planner / worker patterns
Build itA big task blows one context window. Split it across subagents — what does each get, and how does the parent recombine them?
✓ CheckpointJustify a subagent by naming the coordination cost you accepted. If you cannot name it, one agent was enough.
Models rate-limit, tools time out, JSON comes back malformed. A production harness expects failure: learn retries with backoff, timeouts and fallbacks, and how to feed an error back to the model as an observation it can recover from — instead of crashing the loop.
Retries & backoff
Timeouts & fallbacks
Error-as-observation
Graceful degradation
Build itA tool returns a 500 mid-loop. Show two recovery paths: one your code handles, one the model handles.
✓ CheckpointExplain why a retry without idempotency can double-charge, and where the key belongs.
4
Evals in the Loop
P4 · Advanced · 60 min
You cannot ship what you cannot measure. Learn to evaluate a harness — offline test suites, LLM-as-judge scoring, and in-loop self-checks where the agent verifies its own work before finishing. Evals are the regression test that keeps a prompt change from silently breaking everything.
Offline eval suites
LLM-as-judge
In-loop self-check
Regression on prompt changes
Build itYou tweak the system prompt. What eval tells you — before users do — whether the agent got better or worse?
✓ CheckpointExplain why an eval that runs only before release tells you nothing about the run that just failed.
When an agent does something weird, you need the tape. Learn to trace every turn — prompts, tool calls, results, tokens, latency, cost — so you can replay a run, find where it went wrong, and debug behaviour that only shows up in production.
Per-turn tracing
Prompt & tool logging
Token / latency / cost metrics
Replay & debugging
Build itAn agent gave a bad answer once, in production. What must your trace have captured for you to reconstruct why?
✓ CheckpointExplain what a trace must capture for you to reconstruct why the agent chose a tool three days later.
A harness that is correct but slow or expensive will not ship. Learn the levers — model routing (a cheap model for easy turns), caching, trimming context, and parallelizing tool calls — that keep an agent fast and affordable at scale without dropping quality where it counts.
Model routing
Caching & context trimming
Parallel tool calls
Quality vs cost tradeoffs
Build itYour agent averages 12 model calls per task at frontier prices. Name three changes that cut cost without hurting results.
✓ CheckpointCompute the cost of one agent run from tokens and price, then name the step you would cut first.
The common questions before you start — how long it takes, whether to follow it in order, and how it stays current.
How long does this roadmap take?
It runs 18 stations across 3 tracks — roughly ~16h of focused learning, plus the time you spend actually building. It is self-paced, so most people work through it over a few weeks, an evening or a single station at a time.
Do I have to follow the stations in order?
The tracks are ordered so each station builds on the one before, and following them start to finish is the intended path. But every station also stands alone — if you already have the foundations, jump straight to the part you need.
Is it free?
Yes. The whole roadmap, and every handbook, lab, and challenge it links to, is free and open — no sign-up and no paywall.
How is the roadmap kept current?
It teaches the durable fundamentals of the role first, then the tooling and the AI-era shifts on top — so most of it stays relevant as individual tools churn, and it is revised as the role itself changes.
Who is this roadmap for?
Anyone stepping into or leveling up in the AI Harness role — whether you are switching in, early-career, or a senior filling gaps. Start where you are; the ladder shows what is left.
Finished this one? 0 / 31 Roadmaps done
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.