The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Prompts change fast — the platform has to keep up
A prompt gets tweaked far more often than a model gets retrained — sometimes many times a day, often by someone who isn't a software engineer. Treat every prompt change like an emergency code deploy and iteration grinds to a halt; treat it too casually and an unvalidated change reaches every user instantly.
We'll build the platform layer for prompts specifically: externalized, versioned config decoupled from deploys; offline eval before anything reaches users; live A/B testing because offline eval alone misses real effects; guardrail metrics that must never regress; and a review gate suited to non-engineer editors.
What the new pieces do
- Prompt Engineerclient
- Often not a software engineer — a PM, researcher, or specialist iterating on wording, examples, and instructions, wanting to test changes safely without needing a code deploy.
Step 1 · The baseline
Prompts hardcoded in application code
Simplest version: the prompt text lives directly in the application's source code. What happens when someone wants to test a wording change?
Prompts are hardcoded in application source code. What's the cost of testing a wording change?
A code change routine for an engineer is a genuinely high barrier for the people who most often want to iterate on prompt wording — and even for an engineer, it means a full build/test/deploy cycle for what might be a two-word tweak.
Coupling prompt text to code deployment means the iteration speed on prompts is bottlenecked by the SLOWEST part of the stack (code review, build, deploy) even though prompt wording is conceptually a much lighter-weight thing to change — and it puts prompt editing behind a skill barrier (writing and deploying code) that doesn't match who actually has the best judgment about prompt wording.
Editing prompt text has nothing to do with the model's weights — this conflates prompt engineering with model training, which are entirely different operations with different costs.
Hardcoded prompts couple a lightweight text change to a heavyweight code-deploy cycle, slowing iteration and locking out the people (often PMs or prompt specialists, not software engineers) who most directly understand prompt intent. Prompts need to be decoupled from code the way configuration typically is.
Why this piece earns its place
The cost people quote is deploy latency, but the expensive part is attribution. When prompt text lives in source, a prompt edit arrives inside a release alongside a batch of unrelated commits, so when quality drops on Tuesday the only honest answer is that something in that release did it, and you end up bisecting a deploy to find a two-word change. Externalizing is what gives a prompt its own change event, with its own timestamp and author — and every later step on this page reads from that event. Hardcoding does something else too: it multiplies the text. The same system instruction gets pasted into a batch job, a retry path and an internal tool, and the three copies drift, so a fix lands in one of them and the bug survives in the other two. When you would skip it: one prompt, one call site, and the only editor is the engineer who deploys. At that size a registry is ceremony. The moment there is a second editor or a second call site, the copies have already started.
What the new pieces do
- App Runtimeclient
- The live application making real LLM calls on behalf of real users — starts with prompts baked directly into its own code.
- LLM Callservice
- Where the assembled prompt actually gets sent to a model — the same call site regardless of which prompt version supplied the text.
Step 2 · Externalize as versioned config
Edit the prompt, not the code
Decouple prompt text from application code entirely.
Prompts should be editable without a code deploy. What's the actual mechanism?
A spreadsheet has no versioning discipline, no eval gate, no audit trail of who changed what and when — it solves the "no code deploy" problem but reintroduces every quality and safety problem a proper registry is meant to prevent.
A registry gives prompts the same treatment as any other production configuration: versioned, auditable, and referenced by the application at runtime rather than baked into its source. Publishing a new prompt version becomes a fast, reviewable registry operation, completely independent of the application's own deploy cycle.
Baking multiple variants into code still requires a deploy to add, remove, or adjust any of them — this doesn't actually solve the decoupling problem, it just hardcodes more options instead of one.
Introduce a Prompt Registry: every prompt is versioned config, and the application fetches the currently assigned version at runtime rather than having it compiled into its source. Publishing a new version is a registry operation — fast, auditable, and entirely decoupled from the application's own code-deploy cycle.
Why this piece earns its place
Fetching config at runtime puts the registry in the request path, so the first real question is what the app does when the registry is unreachable. The answer is a local cache with a last-known-good copy baked into the deploy: the app boots, serves what it had, and a registry outage degrades publishing rather than answering. That cache is also where rollback speed actually lives — repointing a version is instant in the registry and takes a TTL to become true everywhere, which makes the TTL a recovery-time decision, not a performance one. Two knobs belong in different places. Which version is active for a use case is registry state, owned by the platform. An override that pins one request to a specific version belongs in the request, so an engineer can exercise a draft against real inputs without publishing it — but it is an internal, authenticated path a caller cannot reach, and requests carrying it are tagged and dropped from every experiment and guardrail number. A bypass that also quietly polluted the evidence would be the worst of both. And every model call should log the exact version id it used, because without that the measurements taken later have nothing to attribute themselves to.
What the new pieces do
- Prompt Registrygateway
- The source of truth for every prompt version — decoupling "what prompt is live" from "what code is deployed," so edits don't require a redeploy.
Step 3 · Offline evaluation before rollout
Score it before anyone sees it
A new prompt version is drafted. Should it go straight to real users, or be validated first?
A new prompt version is ready. Should it go directly to production traffic?
Fast iteration and validated iteration aren't in conflict — skipping validation entirely just means users are the first to discover a regression, which is a worse trade than a quick, automated pre-check.
An offline eval gate catches obvious regressions (broken output format, degraded task performance on known cases) cheaply and fast, before any real user is exposed to a bad candidate — the same "validate before it goes near production" principle used for model registry promotion, applied here to prompts.
Manual spot-checking by one person catches only what that person happens to think to try — it's not a systematic, repeatable, or scalable substitute for running against a defined evaluation set.
Add an Offline Eval Gate: every candidate prompt version is scored against a fixed evaluation set before it's eligible for any real traffic, and the gate fails closed — no unvalidated candidate can be promoted while it's unavailable. This is the same eval-gate principle used for model promotion, applied to the much-higher-frequency artifact of prompts.
Why this piece earns its place
The gate is only as trustworthy as the set behind it, and prompt eval sets rot in a specific way. Most cases here are reference-based: the expected output is known, so scoring is a comparison, which is cheap, deterministic, and the reason the gate can afford to run on everything. But plenty of what you actually want to check has no single correct string: whether the tone held, whether a refusal was the right call, whether the answer stayed inside policy. Those need a model reading the output against a rubric, and the moment you add that mode the judge is itself a prompt, with its own version. Change the judge and every historical score moves, so a candidate can pass today and identical text fail next month with nothing about the candidate having changed. Version the judge alongside the set and freeze both, or today's score is not comparable to yesterday's and the gate is theatre. Cost follows the same split: a reference case is one model call, a judged case is two, and the bill is paid per candidate — while candidates arrive many times a day here rather than on a retraining cadence. A gate slow enough to interrupt someone gets routed around. So run a small smoke subset on every draft save and reserve the full frozen set for the moment a version asks for traffic.
What the new pieces do
- Offline Eval Gateservice
- Scores a candidate prompt against a fixed evaluation set before it's allowed anywhere near real production traffic.
Step 4 · A/B testing on live traffic
Offline eval alone can still miss real effects
The candidate passed offline evaluation. Is that sufficient confidence to fully replace the current production prompt?
A prompt candidate passed the offline eval set. Is that enough to fully roll it out to all users?
An offline eval set is a fixed, historical snapshot — real user inputs, phrasing, and edge cases in live traffic can differ meaningfully from what the eval set captured, the same gap that makes shadow/canary deployment necessary for models, now applied to prompts.
A/B testing exposes the candidate to actual live user inputs and measures actual outcomes, catching real-world effects — including ones the offline eval set never anticipated — before the candidate earns full rollout. This is the exact same principle as shadow/canary validation for models, reapplied here because the underlying reason (offline metrics aren't the whole story) is the same regardless of what's being tested.
Repeating an identical evaluation against an identical static dataset can't reveal anything about real-world behavior the dataset itself doesn't already capture — it adds no new information.
Run genuine A/B tests on live traffic via the A/B Traffic Split engine: split a percentage of real requests between the current production prompt and the candidate, and measure real outcome metrics — not just the offline eval score — before considering full rollout.
Why this piece earns its place
The detail that decides whether a prompt test means anything is the assignment unit. Split per request and one conversation flips versions between turns, which breaks the experience and wrecks the measurement — each user contributes half an observation to each arm, the arms converge, and a real difference reads as noise. Assign sticky, by user or session, and analyse at the unit you assigned at. The second thing is statistical power. Task success is usually the common outcome rather than a rare one, so rarity is not the limit — the limit is that a wording change moves the rate by less than the traffic you have can resolve, and the metrics further down the same list, conversion especially, are rare enough to need far more traffic again. A test with no pre-committed stopping rule then gets stopped when the number looks good, which manufactures wins reliably. Decide the smallest effect worth shipping before the test starts, and accept that plenty of wording tweaks fall below it and should ship on judgment rather than on evidence that is not there. Also watch for concurrent experiments on the same call site: two tests over one prompt are testing the product of both, and neither result is clean unless assignment is coordinated.
What the new pieces do
- A/B Traffic Splitservice
- Runs two prompt versions concurrently on a split of real traffic, comparing real-world outcomes offline eval alone can't reveal.
Step 5 · Guardrail metrics, not just the headline number
A "win" on one metric can be a loss overall
An A/B test shows Variant B has a meaningfully better task-success rate than the current prompt. Should it be promoted based on that result alone?
A variant wins on the primary metric (task success). Is that alone sufficient to promote it?
A variant could improve task success while simultaneously tripling cost per request, degrading latency, or increasing unsafe outputs — none of which the primary metric alone would reveal, and any of which could make the "win" a net negative in practice.
Guardrail metrics catch exactly the failure mode where a prompt genuinely improves on what it's being optimized for while quietly regressing something equally important that wasn't the explicit target — cost, latency, and safety are common candidates for guardrails specifically because they're easy to overlook when focused on one headline number.
A subjective feel-check isn't a repeatable or reliable substitute for tracking the actual metrics that matter — it's exactly the kind of ad-hoc judgment a rigorous testing platform is meant to replace.
Track guardrail metrics — cost, latency, safety/refusal rate — in the Outcome + Guardrail Metrics store alongside the primary metric being optimized. A variant is only eligible for promotion if it improves (or holds) the primary metric AND doesn't meaningfully regress any guardrail, regardless of how good the primary result looks in isolation.
Why this piece earns its place
A guardrail is only enforceable if it is computable per request and attributable to a version. Cost and latency are, because both fall out of what the call site already records — tokens in, tokens out, wall time. Refusal rate nearly is: the stop reason or moderation flag comes back with the completion, and a refusal has a shape you can match on, so counting it is a cheap check over data you already hold. Safety is the expensive one — deciding whether an output was actually harmful means running a classifier or another model over it, which makes it the one guardrail with a per-request bill, and therefore the one that quietly gets sampled down to nothing. Sample it deliberately and write the rate down, rather than discovering after an incident that the safety line on the dashboard was drawn from a handful of requests. Then define the denominator. Cost per request and cost per completed task are different metrics and can move in opposite directions — a variant that asks one clarifying question costs more per call while costing less per outcome, and which of those you wrote down decides whether that change reads as a breach or a win. Evaluate the guardrails continuously while the test runs, not once at the end: the damage accrues per request served.
- primary metricwhat the test aims to improve
- guardrail metricsmust never regress
- both requiredfor promotion
What the new pieces do
- Outcome + Guardrail Metricsdata
- Tracks both the metric a test is trying to improve AND the metrics (cost, latency, safety) that must not get worse regardless of the primary result.
Step 6 · Routine, instant rollback
Not an emergency procedure — a daily tool
A promoted prompt version turns out to underperform in ways the test window didn't catch. How fast can you revert?
Because every prompt version in the Registry is immutable and versioned, rollback is repointing the "currently active" reference back to the previous version — the same version-pointer mechanism as model registry rollback, but exercised far more routinely given how often prompts change. This isn't a rare emergency procedure here; it's treated as a normal, low-friction operation a prompt engineer might use several times in a week.
Why this piece earns its place
The hard half is the residue. Moving the pointer changes what the next request gets; it does nothing about what the bad version already produced and something else kept — cached responses, conversation summaries, extracted fields written to a table, memory carried into the next session. All of it is still wrong after the revert, and that is what separates a prompt rollback from a code rollback. So reverting is two operations — move the pointer, then decide what to invalidate — and the second is only possible if caches and derived records are keyed by the prompt version that produced them. Add that key early; retrofitting it during an incident is a migration. The other rule is that the way back must not run through the gates guarding the way forward. If reverting needs an approval and an eval run, it inherits their latency at the exact moment you cannot afford it. Any version that was previously active has already passed both, so returning to one is a pointer write while anything new is a fresh candidate — that asymmetry is what keeps rollback routine instead of an escalation.
Step 7 · A review gate built for non-engineers
Safe editing without requiring a code-review habit
Prompt engineers are often not software engineers, and may not have the instinct to flag "this edit accidentally removed the safety instruction" the way a code reviewer would catch a risky diff. What keeps an accidental unsafe edit from reaching real users?
Require every prompt version to pass through a Review / Approval Gate before it can be eligible for eval and rollout — a lightweight, prompt-specific review step (not a full code review) that a peer or an automated check can use to catch obviously risky changes (removed safety instructions, changed output-format contracts other systems depend on) before they even reach the Eval Gate.
Why this piece earns its place
The hard part of reviewing a prompt is that a word-level diff of a long block of prose shows churn, not risk, and a human reading the whole thing every time is the bottleneck. Structure fixes it: keep the prompt in named sections — role, safety rules, output contract, examples — so the gate can report which section a candidate touched and route on that. An edit inside the examples is not the same risk as one that deletes a line from the safety block, and only a structured prompt lets a machine tell them apart. The checks worth automating are the ones with a machine-readable answer: run the candidate over a few fixed inputs and diff the parsed shape of the output, which catches a broken format contract that a downstream service depends on in seconds, with no human in the loop. One detail that bites: record the approval against the immutable version id, never the prompt name. Approve by name and a later edit inherits that approval and ships unreviewed — the exact failure the gate was installed to prevent.
What the new pieces do
- Review / Approvalservice
- A required approval step before a prompt edit can reach real users — a safety net specifically because prompts are often edited by people without a code-review habit.
Step 8 · The sharp edges
Injection regressions and prompt sprawl
Two subtler risks: a seemingly-innocuous prompt edit that inadvertently makes the model MORE susceptible to prompt injection, and, after months of active testing, genuine confusion about what prompt version is actually live for a given use case.
Include a dedicated security/injection-resistance eval suite in the Eval Gate, distinct from task-success metrics — a prompt edit can pass every functional check while quietly weakening resistance to adversarial input, so security needs its own explicit test, not an assumption that functional correctness implies it. For prompt sprawl, the Registry maintains one clear, queryable single source of truth for "what's the currently active prompt version for this use case," regardless of how many draft versions or past experiments exist alongside it.
Why this piece earns its place
Injection resistance cannot be gated the way task success is, because the other side moves. A frozen adversarial suite decays into a regression test for attacks you already defeated — useful, and not the same thing as being safe. The suite has to be fed: attempted injections found in real traffic become new cases, and a pass means no known attack got through, which is the only claim the evidence supports. Say it in those words, because a security suite reported as a score invites everyone to read it as one. Sprawl has an unglamorous second half nobody writes down, and that is retention. Keep every version that was ever active — those are the rollback targets, and the only way to reconstruct what a complaint from three weeks ago actually saw — and garbage-collect drafts that never passed the gate. And watch use-case keys shared across features: two call sites reading one key means a change made for the first ships silently to the second, which surfaces as an unexplained regression in a team that never touched a prompt.
You did it
You just designed a prompt management platform.
Everything you assembled, in order
- Hardcoded prompts couple a lightweight text change to a heavyweight deploy cycle — externalizing them as versioned config decouples iteration speed from code deployment.
- An offline Eval Gate, failing closed, validates every candidate against a fixed set before it's eligible for real traffic.
- Live A/B testing catches real-world effects an offline evaluation set — a fixed historical snapshot — structurally can't reveal.
- Guardrail metrics (cost, latency, safety) must never regress, even when a variant wins on the primary metric being optimized.
- Rollback is a routine, instant version-pointer change, exercised far more often than for models given how frequently prompts iterate.
- A dedicated, non-engineer-friendly review gate catches prompt-specific risks (like an accidentally-removed safety instruction) before rollout.
- Security/injection-resistance needs its own explicit eval, separate from task-success metrics, since the two can diverge.