The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What makes a tutor "adaptive"?
Bolt an LLM onto a fixed list of practice problems and you've built a chatbot with homework attached — every student gets the same order, the same explanations, no matter what they already know or what specifically confused them. What actually makes a tutor adaptive isn't the LLM at all — it's a model of what this learner knows, driving what to ask next and how to explain it.
We'll build the pieces most "AI tutor" demos skip: a per-skill mastery model, a grading path that reaches for an exact checker before it ever reaches for an LLM, explanations grounded in a vetted curriculum instead of free generation, and hints that deliberately don't just hand over the answer.
What the new pieces do
- Studentclient
- The learner — attempting problems, asking questions, sometimes getting stuck and needing a nudge, not the answer.
Step 1 · The baseline
A fixed list of problems
Simplest version: every student gets problems in the same fixed order, chapter by chapter. What actually goes wrong with this for real learners?
Every student sees the same fixed problem order. What's the core failure?
A single fixed order can't simultaneously be right for a student who already knows the first three chapters AND a student who needs to slow down on chapter one — "good for everyone" and "one fixed order" are in direct tension.
A fixed order treats every learner identically regardless of their actual knowledge state. A student who has already mastered a skill is bored repeating it; a student who hasn't mastered a prerequisite is pushed forward into material built on a foundation they don't have yet.
A list of practice problems is tiny; storage isn't remotely the constraint here. The failure is pedagogical, not infrastructural.
A fixed order can't be right for every learner at once — it's simultaneously too slow for students who've already mastered a skill and too fast for students missing a prerequisite. Real personalization needs the system to actually know what this specific student knows.
Why this piece earns its place
The baseline is worth building deliberately rather than treating it as a placeholder, because what a fixed order really encodes is not the list — it is a prerequisite structure flattened into a line. Someone decided fractions come after division. Step 2 discards the line, and discards that judgement with it, unless the judgement was written down separately as edges between skills: this one depends on that one. That graph is authored content rather than code, and it is what later lets a wrong answer point at a missing prerequisite instead of only at the skill that happened to be on screen. The second thing fixed here is the problem bank's metadata. Every piece that follows — mastery, grading routes, review scheduling — reads a per-item label saying which skill this problem exercises and roughly how hard it is. Nothing can be personalized against items that carry no skill tags, and retro-tagging thousands of authored problems is content work, not engineering work, so the cost lands on the people least able to absorb it. Tag at authoring time and it is invisible. The operational tell that you got this wrong: item analytics that can only report one global pass rate, because there is no dimension to group by.
What the new pieces do
- Tutor APIgateway
- Orchestrates a tutoring session: which problem to show, how to grade the response, when to call the LLM.
- Problem Selectorservice
- Decides what to ask next — starts as a fixed curriculum order, becomes mastery-aware from Step 2.
Step 2 · The mastery model
Track what this student actually knows
You need to pick the next problem to be neither trivially easy nor hopelessly hard for THIS student, on THIS skill, right now. What decides that?
How does the system decide the right next problem for a specific student's current level?
Self-reported skill is notoriously unreliable — students (especially struggling ones) are often poor judges of their own gaps, which is part of why they need a tutor in the first place.
One data point is noisy — a lucky guess or a careless slip shouldn't swing the whole selection; you want an estimate that accumulates evidence over multiple attempts.
A mastery model accumulates evidence over many attempts into a probabilistic estimate per skill, and problem selection targets difficulty just above what the student can already reliably do — hard enough to stretch them, not so hard it's just frustration, and not so easy it wastes their time.
Maintain a per-skill mastery model — a probabilistic estimate (Bayesian Knowledge Tracing or similar) that updates after every attempt. The Problem Selector uses it to target each student's zone of proximal difficulty: challenging enough to be worth doing, achievable enough not to be discouraging.
Why this piece earns its place
Treat the stored estimate as a derived view, not the record. The durable thing is the attempt log — who answered what, when, and how it was graded — and mastery is recomputed from it. That distinction is what lets you change the model at all. Retuning a skill's guess and slip parameters, or moving from one estimator to another, silently changes the meaning of every number already sitting in the table; with raw attempts you replay and rebuild, without them you have a column whose units shifted mid-corpus and no way to tell which rows are which. Version the estimator beside the values it wrote. The other thing to say out loud is cold start. A brand-new student has no evidence at all, so their first few problems are really a placement probe, and the priors you seed matter more than the estimator does — seed everyone as a beginner and strong students sit through a tedious first session, seed optimistically and weak students meet material built on a prerequisite they never had. Neither shows up as an error.
- per-skillnot one global level
- updatesafter every attempt
- targetszone of proximal difficulty
What the new pieces do
- Mastery Modeldata
- A per-skill estimate of what this specific learner actually knows — the thing that makes the tutor "adaptive" instead of just a quiz app.
Step 3 · Grading: symbolic first, LLM second
Don't use an LLM to check 2+2
A student submits an answer — sometimes a number, sometimes code, sometimes a written explanation. Should everything route through the LLM for grading?
Answers range from "42" to a paragraph explaining a concept. Should an LLM grade all of it?
LLM grading is probabilistic and can be inconsistent run to run, and it's slower and more expensive than a symbolic check — using it for something with one exactly-correct answer is strictly worse on cost, speed, AND reliability than a deterministic checker.
A symbolic checker for "does this equal 42" or "do these unit tests pass" is instant, deterministic, and free of any grading-inconsistency risk. The LLM is reserved for exactly the cases that NEED judgment — evaluating whether a written explanation demonstrates real understanding — where no exact check exists.
String matching fails immediately on anything with multiple valid forms (2+2, 2 + 2, four) or any open-ended written response — it's only viable for the narrowest structured cases, not a general grading strategy.
Build a hybrid grader: structured answers (numeric, symbolic math, code with test cases) go to a deterministic exact checker — fast, free, perfectly consistent. Only genuinely open-ended answers (written explanations, proofs, reasoning) are routed to the LLM, which is the one grading tool actually suited to judgment calls.
Why this piece earns its place
Two implementation details decide whether this stays clean. First, the route belongs to the item, not to the answer. The problem declares its answer type at authoring time, so a numeric item is always checked symbolically; sniffing the submitted string at runtime means a student who types a sentence into a numeric field quietly falls through to the expensive path, and LLM spend then grows with the weirdness of your users rather than the shape of your content. Second, the LLM path should return a rubric, not a score. Which criterion did the answer miss — the reasoning, the notation, the conclusion? A bare wrong is enough to move mastery and nothing else; a named failing criterion is what Step 5 hands the explainer, so the hint can address this student's actual misconception instead of re-teaching the topic generically. There is a quieter payoff too, though only if you go and get it. The two paths never see the same submissions — the route is fixed by the item — so symbolic correctness cannot by itself reveal that the LLM grader drifted after a model change. What it hands you is free labels: shadow-grade a frozen sample of structured items with the LLM, off the student's path, and that is your regression set. Open-ended answers need a held-out set someone scored by hand against the rubric.
- instant + freesymbolic checker path
- LLMonly for open-ended judgment
- 0 grading varianceon the structured path
What the new pieces do
- Hybrid Graderservice
- Grades structured answers (math, code) with an exact symbolic checker; routes only genuinely open-ended answers to the LLM.
Step 4 · Close the loop
Every grade updates the mastery model
A grading result — right, wrong, partially right — is only useful if it actually changes what happens next.
Every graded attempt feeds back into the mastery model immediately: a correct answer nudges that skill's estimate up, an incorrect one nudges it down (and, ideally, flags a related prerequisite skill to revisit). The next problem the Selector picks reflects the updated estimate — this closes the loop between "how did the student just do" and "what should they see next," which is the actual mechanism of adaptivity, not a one-time setup step.
Why this piece earns its place
The interesting bug here is credit assignment. Most real problems exercise more than one skill — a word problem about fractions is also arithmetic, also reading — so a wrong answer has to be attributed to something before any estimate can move, and attributing it to the wrong skill is completely invisible. Nothing errors. The student is simply sent to practise something they already do well while the real gap goes untouched, and the dashboards show a healthy loop throughout. Prefer items tagged with one primary skill, use the grader's rubric to disambiguate where you have one, and be willing to record a low-confidence attribution as weaker evidence rather than forcing a clean choice. The second detail is that this is a read-modify-write on one row per student per skill, so it needs an attempt id and an idempotency rule. Retries, a double-tapped submit button, or an offline client syncing yesterday's work will otherwise count the same answer twice, and after the fact a doubled attempt is indistinguishable from genuine evidence.
Step 5 · Ground the explanations
Don't let the LLM freely teach facts
When a student gets something wrong, they need an explanation — and ideally one tailored to their specific misconception. Should the LLM generate that explanation purely from its own training?
The LLM explains a concept the student got wrong. Generate freely, or ground it in something?
A capable LLM is USUALLY right on well-known material, but "usually" is a dangerous bar for something explicitly being taught as fact to a learner who has no way to verify it — and curriculum-specific conventions (a syllabus's exact definitions, notation, or scope) aren't something the LLM's general training reliably matches.
Grounding constrains what the LLM is allowed to assert as fact to vetted, curriculum-accurate source material, while still letting it do what LLMs are genuinely good at: adapting HOW the explanation is delivered to this student's specific error, in natural, conversational language.
A fully static explanation misses the actual value an LLM adds — tailoring the explanation to the SPECIFIC way this student went wrong, which a pre-written generic paragraph can't do.
Retrieve relevant content from a vetted Curriculum Knowledge Base (RAG) before generating an explanation, and constrain the LLM to ground its answer in that retrieved material — it can still tailor the phrasing, examples, and framing to the student's specific mistake, but the underlying facts come from vetted source material, not free generation.
Why this piece earns its place
Grounding fixes what the tutor asserts; it does not fix what the tutor reaches for. Retrieval also has to be filtered to where this student actually is — the curriculum's own explanation of a concept can be perfectly accurate and still sit three chapters ahead, written in notation they have not met yet. That failure looks nothing like a hallucination, and no amount of vetting the knowledge base prevents it: the passage is correct, it is just the wrong passage for this learner. Put the constraint in the retrieval query, alongside the skill and the level the mastery model already knows, rather than asking the model to simplify afterwards. What this buys you elsewhere is a repair loop, but only if you record what went into each explanation. A hint is synthesised from several retrieved passages together with the student's mistake and the grader's verdict, so a teacher reporting a confusing hint is pointing at a set, not a sentence, and nobody can tell which passage misled it unless the generation logged the ids it was handed. Log them beside the hint. A bad explanation then becomes an edit to vetted source material that improves every future explanation drawing on it, rather than a prompt someone rewords and hopes about.
What the new pieces do
- Tutor LLMservice
- Generates hints and explanations — grounded in the curriculum, and deliberately gated to avoid just handing over the answer.
- Curriculum KBdata
- The vetted, syllabus-accurate knowledge the LLM retrieves from, instead of freely generating explanations from parametric memory alone.
Step 6 · Hints, not answers
The Socratic gate
A stuck student asks "just tell me the answer." Should the tutor comply immediately?
A struggling student directly asks for the answer. What should the tutor do?
Immediately handing over the answer optimizes for the student's immediate comfort at the cost of the actual learning outcome — the productive struggle of almost-getting-there is often where real understanding forms.
A hint ladder gives the student every chance to have the "aha" themselves first, which is where durable learning happens, while still guaranteeing they're never permanently stuck — the direct answer remains available as the last rung, not the first response.
Refusing forever, even after a student has made genuine repeated attempts, just produces frustration and abandonment — the goal is productive struggle with a safety net, not struggle without end.
Build a hint ladder: the first response to "I'm stuck" is a pointer toward the relevant concept, not the answer. Further requests escalate — a more specific hint, then the exact next step — and only after genuine attempts have been made does the tutor give the full direct answer with explanation. This is a deliberate pedagogical gate, not a capability limitation.
Why this piece earns its place
The rung is server-side state, and that is the part people miss. Which hint level this student has reached on this problem has to persist — across a page reload, a second device, a new conversation — or the ladder is decorative. Students find the reset fast: if a fresh session restarts at rung one, asking three times in three tabs is quicker than thinking. What goes with it is your own measurement of the gate. Hint requests per problem is the number you would reach for to judge whether the ladder is working at all, and three tabs look like three learners needing help rather than one learner routing around the gate. The other thing to pin down is what counts as a genuine attempt, since that is what the last rung waits for. The only honest source is a graded submission — Step 3 already produces one — not a count of messages sent. Advance on turns instead and typing "I am stuck" three times walks the student to the bottom of the ladder with no work in between, which is the same outcome as having no gate, arrived at politely.
Step 7 · Remember across sessions
Learning happens over months, not one sitting
A student logs back in three weeks later. Should the tutor start from scratch, or remember that they were shaky on fractions and haven't practiced since?
Maintain a long-term Learner Profile, separate from any single session's working state — persistent mastery trends, known weak spots, and a forgetting-curve estimate per skill (skills genuinely decay without practice). The Problem Selector consults this profile to bring back a skill for review right around when it's predicted to be forgotten — spaced repetition — rather than only reacting to what happens inside today's session.
Why this piece earns its place
This store changes the product's obligations, not just its features. A months-long record of exactly what a named learner keeps getting wrong is sensitive in a way session state never was. If any of those learners are minors — and the moment you sell into a school they are — retention windows, export, deletion, and who may read it (the student, a parent, a teacher, a district) stop being policy questions and become requirements that land on this store specifically. Build deletion as a supported path and check the tutor still works afterwards, degraded to session-only behaviour rather than broken. The design decision to watch is arbitration. Two things now want to choose the next problem: the mastery model reaching for the right difficulty, and the profile saying a skill is due for review. With no explicit policy the review queue wins whenever a student has been away, so the session after a holiday is all revision and no progress — precisely when motivation is thinnest. Cap the share of review in a session and make that a tunable, not an emergent behaviour.
What the new pieces do
- Learner Profiledata
- Tracks this learner's weak spots and forgetting curve across weeks and months — separate from the single session's working state.
Step 8 · The sharp edges
Cost, latency, and staying on-curriculum
Hint generation is on the interactive path — a student waiting 8 seconds for a hint is a bad experience. Mastery-model and profile updates don't need to be instant. And an LLM asked an off-topic or inappropriate question needs to stay in its lane.
Split by latency requirement: hint/explanation generation uses a fast, cheap model tuned for low latency, since it's directly in the student's interactive loop; mastery-model recomputation and profile updates can run slightly deferred/batched, since a few seconds of lag there is invisible to the student. Layer guardrails on the LLM path specifically for the tutoring context: refuse to just do the student's homework verbatim, redirect off-curriculum or inappropriate requests, and never let a jailbreak attempt bypass the hint-ladder's "don't just give the answer" gate.
Why this piece earns its place
The guardrails are the one part of this design whose behaviour you do not own. The orchestrator picks which rung is due, so no model swap can move a student down the ladder — but the wording at that rung is generated, and a newer, more helpful model will cheerfully fold the answer into what is logged as a level-one hint. That failure is invisible from both ends: the ladder state says rung one, the student got the answer, and nothing in your metrics moves. Keep a frozen set of real tutoring transcripts, including the jailbreak attempts you have already seen, and replay it before any model or prompt change ships — scoring each reply on whether it withheld what it was meant to withhold, not on whether it sounds helpful. The cost picture is worth naming too, because it is inverted here. Inference spend scales with engagement, so your best students are your most expensive ones, and a flat per-student cap punishes exactly the behaviour the product exists to produce. Budget per session instead, and decide in advance what degrades when it runs out — shorter explanations, fewer LLM-graded open responses, symbolic-only practice — rather than discovering the answer in the month the bill arrives.
You did it
You just designed an adaptive AI tutor.
Everything you assembled, in order
- A fixed problem order can't be right for every learner — real adaptivity requires tracking a per-skill mastery estimate per student.
- The mastery model targets each student's zone of proximal difficulty, and alone (even without an LLM) is already a legitimate adaptive tutor.
- Grading routes structured answers to a deterministic symbolic checker and reserves the LLM only for genuinely open-ended judgment calls.
- Every graded attempt feeds back into the mastery model, closing the loop that actually makes selection adaptive over time.
- Explanations are grounded in a vetted curriculum knowledge base (RAG), not freely generated — a confidently wrong hint is worse than a slow one.
- Hints escalate through a Socratic ladder instead of immediately giving the direct answer — a deliberate pedagogical gate, not a limitation.
- A long-term learner profile, separate from session state, drives spaced repetition of skills predicted to be forgotten.
- Latency-sensitive hint generation and latency-tolerant mastery/profile updates are split, and tutoring-specific guardrails keep the LLM on-curriculum.