The big idea

When prompting isn’t enough, fine-tune

Prompting and RAG get you far: they steer a model with instructions and context at inference time. But sometimes you need the model to internalize a style, a format, a domain, or a skill — reliably, without a giant prompt every call. That means changing the weights. How do you turn "we have examples of the behavior we want" into a better model, safely and repeatably?

Build a fine-tuning pipeline: an offline, repeatable path from curated data through training and a hard evaluation gate to a versioned, deployable model — with a feedback loop that turns production into the next dataset. It’s an ML system, so it lives or dies on data quality and honest evaluation.

How to read this: Each step opens with a real design decision — you make the call before I show you what ships. Watch the diagram grow, hover any box, replay the flow. At the end, leak the eval set to feel training’s quiet failure. Hit Begin.

Step 1 · The skeleton

Data → train → eval → register → deploy

Fine-tuning isn’t "run a training script." It’s a pipeline with stages that must be reproducible and gated. What’s the minimal shape that keeps a model change safe?

Design decision: What’s the right shape for a fine-tuning pipeline?

The call: Curate data → train → evaluate against a gate → register the versioned artifact → deploy behind a canary. — Each stage is reproducible and traceable: a versioned dataset trains a model, an eval gate decides promotion, the registry records lineage, and deployment is gradual and reversible.

The pipeline is a chain of reproducible, gated stages: curate a versioned dataset → train → pass an evaluation gate → register the artifact with full lineage → deploy behind a canary. Every arrow is traceable, so you always know what a model learned from and can roll back to any prior version.

Reproducibility is the point: A fine-tuned model is only trustworthy if you can say exactly which base, data, and settings produced it — and reproduce it. The pipeline’s structure exists to make model changes auditable and reversible.

Step 2 · Garbage in, garbage model

Dataset curation is the real work

The instinct is "more data = better model." But a fine-tuned model mimics its training data exactly — including its noise, duplicates, and mistakes. What actually makes a good fine-tuning dataset?

Design decision: How do you build a fine-tuning dataset that helps?

The call: Clean, deduplicate, filter for quality, format consistently, and hold out an eval set. — Curation dominates outcomes: remove junk and near-duplicates, filter for correctness, format into consistent instruction→response pairs, and carve off a strictly separate held-out eval set.

Curate: clean and filter for correctness, deduplicate (including near-duplicates), format into consistent examples, strip PII, and — critically — split off a held-out eval set that never touches training. Version the result. Quality and diversity beat raw volume; the model will imitate exactly what you feed it.

The model copies your data: Fine-tuning is imitation. Every flaw in the dataset becomes a flaw in the model. That’s why curation — not the training loop — is where most of the quality is won or lost, and why the held-out split must be pristine.

Step 3 · Adapt, don’t retrain

Base model + parameter-efficient fine-tuning

You have data. Do you retrain a model from scratch? Update all of a pretrained model’s billions of weights? Both are expensive and risky. What’s the practical way to specialize a model?

Design decision: How do most teams actually fine-tune a large model?

The call: Start from a pretrained base and train a small adapter (LoRA/PEFT), freezing most weights. — Parameter-efficient fine-tuning trains a tiny set of new parameters (low-rank adapters) on top of a frozen base — far cheaper, faster, less prone to catastrophic forgetting, and adapters can be swapped per task.

Start from a pretrained base model and, in most cases, use parameter-efficient fine-tuning (LoRA/PEFT): freeze the base and train small adapters. It’s cheaper, faster, resists catastrophic forgetting, and lets you keep many task-specific adapters over one shared base — hot-swappable at serving time.

PEFT is the default: Full fine-tuning is a big hammer; LoRA/PEFT is the everyday tool. Small adapters over a frozen base capture the task while preserving the base’s general strength — and make deployment (adapter swap) trivial.

Step 4 · The training loop

Batches, loss, checkpoints, at scale

Now you run the actual training. Data and base feed the trainer; out comes a model. What has to be right about the loop — and what breaks when the model is too big for one GPU?

Design decision: What does the training loop need to be reliable and scalable?

The call: Batched forward/backward passes, a monitored loss, periodic checkpoints, and data/tensor parallelism across GPUs. — Train in batches over epochs, watch the loss (and eval loss) for over/underfitting, checkpoint often so a crash doesn’t lose everything, and shard across GPUs when the model or data won’t fit on one.

The Trainer runs batched forward/backward passes over epochs, monitors training and eval loss (to catch over/underfitting), checkpoints regularly, and shards across GPUs with data/tensor parallelism when the model or batch won’t fit on one device. Hyperparameters (learning rate, epochs, batch size) are logged for reproducibility.

Watch eval loss, not train loss: Falling training loss with rising eval loss is overfitting — the model is memorizing. Early-stopping on held-out eval, frequent checkpoints, and logged hyperparameters keep a long, expensive run recoverable and honest.

Step 5 · The gate that matters

Evaluate before you promote

Training finished; you have a checkpoint. Is it actually better than what’s in production? Ship it untested and you might deploy a regression. What decides whether a model is allowed out?

Design decision: What should gate a fine-tuned model’s promotion?

The call: Scores on a held-out eval set, compared to the base, against a threshold — with no regressions. — The eval gate runs the model on truly held-out data, compares to the current/base model, and blocks promotion on a regression or a missed bar. Objective, repeatable, and the last line of defense.

The Eval Gate scores the fine-tuned model on the held-out eval set, compares it to the base/current model, and blocks promotion on a regression or a missed threshold. This is the pipeline’s most important stage — and it’s only trustworthy if the eval set was never contaminated by training data (the failure you’ll trigger).

The gate is only as honest as the split: Every number the gate reports assumes the eval data is unseen. Break that assumption — leak eval into training — and the gate cheerfully approves a model that just memorized the test. Isolation of the eval set is sacred.

Step 6 · Remember everything

Model registry, lineage, versioning

A model passed the gate. Six weeks later it misbehaves and you need to know exactly what produced it — or roll back. How do you make every model fully traceable and reversible?

Design decision: What must you record about every trained model?

The call: The artifact plus its lineage: base model, dataset version, hyperparameters, and eval scores. — A model registry stores the artifact with full provenance — which base, which dataset version, which hyperparameters, which eval results — so any model is reproducible, comparable, and rollback-able.

Store every promoted model in a Model Registry with full lineage: base model, dataset version, hyperparameters, training run, and eval scores. Now models are versioned, comparable, reproducible, and instantly rollback-able — the audit trail that makes fine-tuning safe to iterate on.

A model is its provenance: The weights are just one artifact; the value is knowing how they were made. Lineage turns "a mystery checkpoint" into a reproducible, governable version you can trust and revert.

Step 7 · Ship it carefully

Canary deploy, rollback, adapter swap

The gate passed offline — but offline eval never perfectly predicts production. How do you deploy a new model so that if it’s worse in the real world, few users feel it and you recover instantly?

Design decision: How do you roll out a fine-tuned model safely?

The call: Canary to a small % of traffic, watch live metrics, then ramp — with instant rollback. — Serve the new model to a slice of traffic, compare live quality/latency/cost to the incumbent, ramp up if it holds, and roll back instantly if not. For PEFT, hot-swap the adapter without redeploying the base.

Deploy behind a canary: route a small slice of traffic to the new model, compare live quality, latency and cost to the incumbent, then ramp — with instant rollback via the registry. With PEFT you hot-swap the adapter onto the shared base, making deploys and rollbacks cheap. Offline eval starts the decision; production finishes it.

Offline eval is necessary, not sufficient: The gate catches obvious regressions; the canary catches what the gate can’t model — real distribution, real users. Gradual, reversible rollout is how you ship model changes without betting production on an offline number.

Step 8 · Close the loop

The production data flywheel

The model is live. Its best feature is that it’s now generating exactly the data that would make the next version better — real prompts, real outcomes, real corrections. How do you turn production into fuel without poisoning the pipeline?

Design decision: How does production improve the next model?

The call: Collect interactions, ratings and corrections, curate them, and feed them into the next dataset. — A feedback loop harvests production signals (thumbs, edits, escalations), curates them through the same cleaning/dedup/held-out discipline, and rolls them into the next training set — a compounding data advantage.

A Feedback Loop collects production interactions, ratings, and corrections, then routes them back through curation into the next Training Set. Each release generates the data for the next — a compounding data flywheel — as long as that data gets the same cleaning, dedup, and held-out discipline as everything else. Never train on raw output unchecked.

Your data moat: Production interaction data — especially corrections — is the hardest asset for competitors to copy. The flywheel is fine-tuning’s long-term payoff, but only if fed through the same gates; raw self-training drifts.

The payoff

You built a fine-tuning pipeline

From "we have examples of what we want" to a governed model factory: curated, versioned datasets; a pretrained base adapted with LoRA/PEFT; a monitored distributed training loop; a hard held-out eval gate; a registry with full lineage; canary deploys with rollback; and a production flywheel feeding the next round.

Now leak the eval set into training and watch every offline number light up green while the deployed model is no better than the base — because the metrics measured memorization, not generalization. That’s why the held-out split is sacred, why dedup must catch near-duplicates, and why time-based splits and re-verification protect the one gate that decides what ships.

• Pipeline shape — curate → train → eval gate → register → canary — reproducible, reversible
• Data curation — clean, dedup, format, hold out — the model imitates exactly what you feed it
• Base + PEFT — adapt a pretrained base with small LoRA adapters, not from scratch
• Training loop — batched, checkpointed, distributed; watch eval loss, not train loss
• Eval gate — held-out scores vs base — the honest test that gates promotion
• Registry — artifact + lineage → reproducible, versioned, rollback-able
• Canary — gradual rollout with instant rollback; hot-swap PEFT adapters
• Flywheel — curated production data fuels the next model; never self-train raw
• The failure — eval contamination inflates metrics silently — isolate the split