The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
The gap between "it trained" and "it's safely serving"
A model finishes training. Turning that into a file that answers real production requests safely is a genuinely different problem than training it — versioning, validating, gradually rolling out, and being able to instantly undo it if something's wrong. Skip this layer and "which model is actually in production, and can we prove it's the one we validated" becomes an unanswerable question.
We'll build the governance layer: immutable versioning with full lineage, staged promotion through validation gates, shadow/canary deployment because offline metrics alone aren't enough, instant rollback, and ongoing production drift monitoring.
What the new pieces do
- ML Engineerclient
- Trains a candidate model and wants a safe, auditable path from "it trained" to "it's serving real traffic" — with a fast way back out if it isn't.
Step 1 · The baseline
A file gets deployed
Simplest version: someone trains a model, copies the resulting weights file to wherever the serving system reads from, and it's live. What breaks down as this scales beyond one person, one model?
Model files get deployed ad-hoc, with no registry or tracked lineage. What's the core problem?
"It works right now" says nothing about whether it can be reproduced, audited, rolled back to a known-good state, or even identified later — the problem shows up the moment something goes wrong or someone needs to answer "what exactly is running."
Without a registry and tracked lineage, the model in production is effectively an untraceable artifact: if it starts behaving badly, there's no reliable record of what to roll back TO, and no way to reproduce or audit exactly how it was trained.
File format efficiency is a minor, separate concern — the real problem is the complete absence of versioning, lineage, and a reliable rollback path.
Without a registry, nobody can reliably answer which model version is live, what produced it, or how to get back to a known-good state if it fails — a reproducibility and rollback blind spot that gets worse with every additional model and every additional engineer.
Why this piece earns its place
Worth being honest that this baseline is fine for a while. One model, one owner, a weekly batch job — a registry is pure overhead, and the trigger to build one is the second engineer or the second model, not a rule. What makes it stop working is quieter than it looks. Serving does not load a version, it loads a path, and a path like prod/model.pkl is a mutable pointer with no history. Replace the file during a rolling restart and half the fleet keeps the old weights in memory while the other half picks up the new ones, so for the length of the rollout you are serving two different models behind one endpoint and your dashboards show the average of two populations — a metric shift with no deploy event to blame it on. Nothing errors, and nothing in the telemetry names a version, because there are no versions to name. The cheap first move is a checksum and a version string that serving reports on startup and stamps on every prediction; everything later in this design is what you get once that string means something.
What the new pieces do
- Production Servingservice
- What actually answers real requests — starts as whatever file someone happened to deploy; becomes governed by every stage this design adds.
Step 2 · Immutable versioning and lineage
Every model, tagged with what produced it
Fix the blind spot with a registry.
Introduce a Model Registry backed by Versioned Artifacts storage: every trained model is registered as an immutable version, tagged with exactly which training data version, code commit, and hyperparameters produced it. Past versions are never overwritten — a new training run is always a NEW version, never a mutation of an old one.
Why this piece earns its place
Immutable has to be enforced by the store, not agreed in a code review. Write-once keys, object versioning, a checksum verified on load — otherwise the day someone re-uploads to fix a packaging bug is the day every rollback target quietly points at different bytes than the run that was evaluated, and nothing tells you. Split what you keep, too: the version row is small, queryable metadata and belongs in a database you can index by model, stage and time; the weights are large blobs and belong in object storage keyed by version id. The reason is not that a listing would read them — every serious engine stores large values out of line — it is that blobs ride along in every backup, every restore, every replica stream and every table rewrite, so the cheapest thing in the registry ends up hostage to the largest. The consequence teams hit is storage. Every training run registers a version and most are never promoted, so the set grows with no obvious owner. Retention has to be stage-aware rather than age-based: dev versions can be collected aggressively, while a version that ever served production traffic carries a floor — the longer of the audit period you have committed to and the window in which it is still a plausible rollback target. Both are finite. Writing keep production versions forever is how a storage problem becomes a permanent one.
What the new pieces do
- Model Registrygateway
- The source of truth for every model version, its lineage, and which stage (dev/staging/prod) it currently occupies.
- Versioned Artifactsdata
- Immutable storage for every model version ever registered, tagged with exactly what training data, code commit, and hyperparameters produced it.
Step 3 · Staged promotion
Dev → staging → production, not a straight line
A newly-registered model version exists. Should it be eligible to serve production traffic immediately?
A model just finished training and got registered. Should it be immediately eligible to serve production traffic?
Training succeeding (the loss converged, the process completed) says nothing about whether the model actually performs well enough, safely enough, and without regression relative to what's currently serving real users.
Staged promotion means a model has to earn its way to production through explicit checkpoints, rather than a successful training run alone being sufficient — this mirrors how a code change progresses through environments before reaching users.
A single offline holdout test, while useful (see Step 4), doesn't capture everything a staged rollout with real production-adjacent validation (Step 5) can catch — one static check isn't equivalent to a staged process.
Give every model version an explicit stage tracked by the registry — dev, staging, production — and require an explicit, gated promotion action to advance between them. A model doesn't default into serving production traffic just by existing; it has to be deliberately promoted, with checks along the way.
Why this piece earns its place
Hold a stage as a slot rather than a label: for a given model name, exactly one version occupies production at a time, and promoting is a swap. That constraint is what lets serving ask one question instead of running a search, and it is what makes "what was production during that bad week" answerable later — provided you keep the history of the slot and not only its current value, which is the column most teams discover they never wrote. The decision that actually bites is when serving resolves the slot. Resolve it once at process start and a promotion does nothing until someone restarts the fleet, so the registry holds an intent that disagrees with what is answering requests — the deploy-shaped problem this step exists to end. Resolve it live and the change lands on every replica at once, which is what you want in one direction and not the other: on the way down, the whole fleet should leave a bad version together; on the way up, a candidate should reach users gradually. So put the rollout on promotion and let a reversal skip it. Skip the ladder entirely for a model nobody serves live — a batch scorer has one stage that matters, the one the job reads — but keep the explicit promotion, because that is what dates the change.
Step 4 · Automated evaluation gate
Block promotion on regression, automatically
Before a candidate model can be promoted, it should be validated against the currently-serving model. Should that validation be a manual, ad-hoc judgment call each time?
A candidate model needs validation before promotion. Manual judgment call, or something else?
Manual review doesn't scale as promotion frequency grows, is inconsistent between reviewers, and — critically — is easy to skip under deadline pressure, which is exactly when a regression is most likely to slip through unnoticed.
Automating the gate makes validation consistent, fast, and impossible to accidentally skip — every promotion attempt runs the same objective comparison against the current production baseline, and a regression is blocked by default rather than relying on someone remembering to check.
Evaluating a candidate's absolute metrics without comparing to the CURRENT production baseline misses the actual question that matters for a promotion decision: is this better or worse than what's live right now?
Add an automated Eval Gate: every promotion attempt runs the candidate against a held-out evaluation set, computes the same metrics for the candidate and the current production baseline, and blocks promotion by default on any meaningful regression. This makes validation consistent and impossible to skip under pressure.
Why this piece earns its place
The gate is only as good as its evaluation set, and the way that set goes bad is invisible. If it is refreshed from the same pipeline that produces training data, a candidate trained on overlapping rows scores beautifully and is promoted into a regression. Freeze the set and version it apart from the training data, and you get a second benefit: verdicts stay comparable over months, because a candidate measured in March and one measured in September only mean anything against each other if the ruler did not move. Then define the word doing the work in the solution. Meaningful has to be a measured number, not zero: two evaluations of the same model differ — sampling, nondeterministic kernels, batch effects — so a threshold set at nothing blocks on noise, and a gate that blocks on noise is a gate people learn routes around. Measure the run-to-run spread of one version against one set, set the block threshold outside that band, and set it per metric, because accuracy and tail latency do not wobble by the same amount. Make an override a recorded action with a named reason attached to the version, so the gate bends somewhere the next audit can see rather than being widened in a config file.
- same metricscandidate vs current baseline
- default: blockedon a meaningful regression
- 0 manual stepsto skip
What the new pieces do
- Eval Gateservice
- Runs a candidate against held-out data and compares it to the currently-serving model — blocks promotion on a meaningful regression.
Step 5 · Shadow and canary deployment
Offline metrics aren't the whole story
The candidate passed the Eval Gate against a held-out dataset. Is that sufficient confidence to fully replace the production model right now?
A candidate passed offline evaluation against a held-out set. Is that enough to fully deploy it to 100% of production traffic?
A held-out evaluation set is necessarily a SNAPSHOT of past data — real production traffic can have a meaningfully different distribution (new user behavior, edge cases the eval set didn't capture, seasonal shifts), which offline metrics alone can't reveal.
Shadow/canary deployment exposes the candidate to actual current production traffic — the real distribution, not a static historical snapshot — while limiting user-facing risk to zero (shadow) or a small, bounded slice (canary). This catches real-world behavior gaps that a held-out offline eval set, by definition, can't.
Re-running the identical offline evaluation against the identical static dataset produces the identical result — it can't reveal anything about how the model behaves on data it hasn't been evaluated against, which is exactly the gap shadow/canary deployment closes.
Run the candidate through Shadow or Canary deployment before full promotion: shadow mode mirrors live traffic to the candidate with results never shown to users (zero risk, real data); canary mode actually serves a small percentage of real traffic (bounded risk, real user-facing signal). Only after this real-world validation does the candidate earn full production promotion.
Why this piece earns its place
Shadow and canary are not two strengths of the same thing, and knowing why you need both is the answer to the follow-up. Shadow measures everything that lives inside the prediction — agreement with the current model, latency, error rate, how the output distribution shifts. It cannot measure anything that requires a user to act on the prediction, because nobody ever sees it: a ranking the shadow model produces is never clicked, so click-through, conversion and escalation rate are unmeasurable there by construction. Those need a canary, where a real slice of traffic is actually served. Shadow is also not free. Mirroring means paying for a second full inference on every mirrored request, so mirror a sample rather than the whole stream unless the model is cheap. And the shadow path has to be side-effect free. If inference writes anywhere — a cache, a computed feature, an event stream that feeds the next training set — the mirrored copy writes too, and you have let an unpromoted model edit production state, or train its own successor on its output.
What the new pieces do
- Shadow / Canaryservice
- Runs the candidate on live (or a slice of live) traffic without it actually serving responses users depend on — catches what offline eval can't.
Step 6 · Rollback
A version pointer, not a fire drill
The new model is fully promoted and serving. Something's wrong in production that wasn't caught earlier. How fast can you get back to the previous version?
Because every past version is immutably stored in Versioned Artifacts, rollback is simply repointing production serving at the previous version id — no retraining, no re-deriving what the old model was, because it was never overwritten in the first place. This is the direct payoff of the immutability decision made back in Step 2.
Why this piece earns its place
Repointing serving is the fast part, and it is not the recovery. What a rollback restores is the model; what it cannot restore is whatever the bad version already caused while it was live. So the number worth managing here is detection time, not switch-over time — the pointer move costs the same either way, and the size of the correction scales only with how long the regression ran. A rollback plan that names a version id and stops is half a plan; the other half names who owns the correction and how its scope gets bounded. The second trap is that rollback compatibility expires. The old version expects the input schema, feature set and preprocessing it was trained against, and all of those keep moving after it stops serving, so the version you assumed was your floor quietly stops loading — or worse, loads and scores confidently on a field that has since been renamed. Pin the input contract with the version rather than only the weights, and rehearse a rollback on a schedule instead of discovering the mismatch mid-incident. Then mark the version you rolled away from as bad in the registry: move only the pointer and nothing stops the next promotion walking straight back into it.
Step 7 · Reproducibility and audit
Prove exactly what produced this model
Months later, someone needs to know exactly how the currently-serving model was produced — for debugging an issue, or for a compliance/audit requirement.
Because every version's lineage (training data version, code commit, hyperparameters) was recorded at registration time, the registry can answer exactly what produced any given model version — and, with the same data and code pinned by hash/version rather than "latest," the training run can genuinely be reproduced if needed. This lineage record is often as important operationally as the model weights themselves.
Why this piece earns its place
Be precise about what reproducibility promises, because bit-exact retraining is mostly not achievable and claiming it will get you caught. Nondeterministic GPU kernels, library and driver versions, data ordering, thread counts — the same inputs rarely produce identical weights. So pin the container image by digest alongside the data and the code, and state the claim as a model statistically equivalent on the frozen evaluation set, not one that hashes the same. Lineage is also only worth its weakest reference. Pinning the code commit while recording the training data as a query against a live warehouse table is decorative: the commit is fixed, the table is not, and the run is unreproducible through the one input that matters most. Materialize a snapshot, or use a store that can address a table as of a timestamp. And the lineage has to be written by the training job itself, from the values it actually ran with, at registration — the field a human fills in afterwards is the field that is wrong, and it is wrong in the direction of what someone believed they had configured.
Step 8 · Production drift
Valid at deploy time isn't valid forever
A model passed every gate and is performing well in production. Six months later, has anything changed?
A model passed every validation gate at deploy time. Does that guarantee it stays good indefinitely?
Real-world data distributions shift over time (user behavior changes, new categories of input appear, the world itself changes) — a model trained and validated on data from six months ago can quietly degrade as the live distribution drifts away from what it learned, with no code or model change required to trigger it.
Deploy-time validation only proves the model was good AT THAT MOMENT, against data available then. A Drift Monitor watching the SERVING model's real-world performance continuously is what catches gradual degradation that no amount of pre-deployment gating could have predicted.
Model performance can degrade with ZERO code or model changes at all, purely because the real-world data distribution shifted — waiting for a code change to trigger re-validation misses this entirely.
Add a Drift Monitor watching the currently-serving model's real-world performance continuously — not just at deploy time. When production metrics degrade meaningfully from their validated baseline, it flags the registry, triggering either an automatic rollback to the previous known-good version (Step 6) or an alert for human review, depending on the severity and the organization's risk tolerance for that model.
Why this piece earns its place
The awkward part of watching a model in production is that the metric you care about needs labels, and labels are late or absent. A fraud call is settled by a chargeback weeks out; a recommendation may never get ground truth at all. So monitoring is layered by latency, and the layers say different things. Input distributions and the model’s own output distribution are available on the next request, and they are proxies — they tell you the world moved, not that you got worse. Accuracy against real outcomes is the measurement that decides, and it lands on a delay you design around: a fast path that alerts on the proxies, a slower one that confirms or clears each alert once labels arrive. That delay is the part people skip, and it sets two things. It sets the shortest regression you can detect at all, because a window narrower than the label lag will always look empty. And it forces the baseline to be recomputed on the same delay, from a cohort as fully settled as the one being judged — otherwise you compare a finished month against an unfinished week and call the gap drift. Compare against what the canary actually produced rather than the offline score, since that is the last measurement taken on real traffic.
What the new pieces do
- Drift Monitordata
- Watches the SERVING model's real-world performance continuously — because a model validated at deploy time can still degrade later as real data shifts.
You did it
You just designed a model registry.
Everything you assembled, in order
- A model artifact alone is nearly meaningless without lineage — exactly what data, code, and hyperparameters produced it.
- Immutable versioning is the foundation everything else depends on: rollback, reproducibility, and safe evaluation all require past versions to be permanent.
- Models progress through explicit, gated stages (dev/staging/production) rather than defaulting to serving traffic just by existing.
- An automated Eval Gate compares a candidate against the current production baseline and blocks promotion by default on a meaningful regression — removing reliance on manual review.
- Shadow/canary deployment validates against real, current production traffic, catching what a static offline evaluation set structurally can't.
- Rollback is a version-pointer change, not a fire drill, because past versions were never overwritten.
- Ongoing production drift monitoring catches degradation that happens with zero code or model change, purely from real-world data shifting over time.