ML
Ops.

F1 · Beginner · 45 min

A model is never "done" — it lives in a loop: collect data, train, evaluate, deploy, monitor, and retrain. MLOps is the discipline of making that loop reliable and repeatable. Everything downstream rests on one habit: reproducibility, so any result can be recreated exactly.

• The train→deploy→monitor loop
• Reproducibility
• Environments vs artifacts
• Why notebooks don't ship

Build itTake a notebook model and list every hidden dependency that would stop a teammate reproducing your result — data snapshot, random seed, library versions, hardware.

✓ CheckpointExplain which part of your train-deploy-monitor loop is currently manual, and what that costs when a model degrades.

F2 · Beginner · 45 min

Code is versioned; data usually is not — and that is where reproducibility dies. Versioning datasets and feature definitions means every model can be traced back to the exact inputs it saw, so "it worked last week" becomes a fact you can check, not a mystery.

• Dataset versioning (DVC-style)
• Feature definitions
• Immutable snapshots
• Data lineage basics

Build itSnapshot a dataset, train on it, then change a few rows and re-snapshot. Prove you can reproduce the original model bit-for-bit from the first snapshot.

✓ CheckpointExplain why versioning the code without versioning the data leaves a run unreproducible.

F3 · Beginner · 45 min

Which hyperparameters produced your best model? Without tracking, that answer is lost. Experiment tracking logs params, metrics, and artifacts for every run, so you can compare hundreds of experiments and reproduce any one of them on demand.

• Logging params & metrics
• Run comparison
• Artifact storage
• Reproducible runs

Build itInstrument a training script to log its hyperparameters and final metrics. Run it 5 times with different settings and rank the runs by validation score.

✓ CheckpointExplain why changing two things between runs makes both results uninterpretable.

F4 · Intermediate · 45 min

Once you have good models, you need a source of truth for them: which version is in staging, which is in production, what data trained it, and how to roll back. A model registry gives every model a version, a stage, and a lineage you can audit.

• Model versioning
• Staging → production
• Lineage & metadata
• Rollback

Build itDesign the metadata you would store for each registered model version — enough to answer "what is in production and how do I roll it back?" in one query.

✓ CheckpointExplain what a model registry gives you that a file in object storage does not.

F5 · Intermediate · 45 min

"Works on my machine" is the enemy of production ML. Containers (Docker) and pinned dependencies package your code, libraries, and runtime so training and serving behave identically on a laptop, a CI runner, and a GPU cluster.

• Docker for ML
• Dependency pinning
• Reproducible builds
• GPU base images

Build itContainerize a training script with pinned dependencies. Run it on two machines and confirm the output is identical.

✓ CheckpointExplain why pinning dependencies matters more for ML than for most services — “it worked last month” is a real failure mode here.

F6 · Intermediate · 60 min

ML deserves the same automation as software — plus a few extras. A CI/CD pipeline runs tests, validates data, trains or promotes a model, and deploys it, so a change goes from commit to production without manual, error-prone steps.

• Automated tests
• Data validation gates
• Model promotion
• Automated deploys

Build itSketch a CI/CD pipeline for a model: what checks run on every commit, what gates a deploy, and what happens if data validation fails?

✓ CheckpointExplain what a data validation gate blocks that a unit test never will.

T1 · Intermediate · 60 min

A real training job is a graph of steps — ingest, validate, featurize, train, evaluate, register — that must run on a schedule, retry on failure, and be observable. Orchestrators (Airflow, Kubeflow, Prefect) turn a fragile notebook into a dependable DAG.

• DAGs & orchestration
• Scheduling & retries
• Step isolation
• Pipeline observability

Build itDecompose one training script into a 5-step DAG. For each step, name its inputs, outputs, and what should happen when it fails.

✓ CheckpointExplain why step isolation in a pipeline makes a failure cheaper to recover from.

T2 · Advanced · 60 min

The classic MLOps bug: features computed one way in training and another way in serving, so the model sees different inputs in production. A feature store computes features once and serves the same values to both — killing training/serving skew.

• Training/serving skew
• Online vs offline features
• Feature reuse
• Point-in-time correctness

Build itPick a feature (e.g. "orders in last 7 days"). Explain how it could be computed differently in training vs serving, and how a feature store prevents the mismatch.

✓ CheckpointExplain training-serving skew with a concrete example, and how a feature store prevents it.

T3 · Intermediate · 60 min

A trained model is useless until something can call it. Packaging wraps it behind a versioned API with a defined input/output contract; serving runs that API reliably, with health checks, autoscaling, and safe rollouts.

• Model as an API
• I/O contracts
• Health checks
• Safe rollout

Build itDefine the request/response contract for a model API. What do you return on a bad input, a timeout, or a model-not-loaded error?

✓ CheckpointExplain what an I/O contract pins down, and what breaks silently when it changes without a version bump.

T4 · Intermediate · 45 min

Not every prediction needs to be real-time. Batch inference precomputes predictions in bulk on a schedule (cheap, high-throughput); online inference answers per request in milliseconds (costly, low-latency). Choosing right shapes your whole architecture.

• Batch scoring
• Online / real-time serving
• Latency vs throughput
• Caching predictions

Build itFor three use cases (recommendations email, fraud check, search ranking) decide batch or online, and justify the latency and cost tradeoff.

✓ CheckpointDecide batch or online for one real prediction, and justify it by what is waiting on the result.

T5 · Advanced · 60 min

Under load, serving becomes a systems problem: autoscaling replicas, batching requests, and squeezing GPU utilization so you are not paying for idle silicon. The goal is meeting latency targets at the lowest cost per prediction.

• Autoscaling
• Request batching
• GPU utilization
• Cost per prediction

Build itFor a service at 1,000 requests/second with a 100ms budget, reason through replicas, batch size, and GPU count. Where is the bottleneck?

✓ CheckpointExplain why request batching improves GPU utilisation and hurts tail latency at the same time.

T6 · Advanced · 60 min

LLMs add a loop on top of classic MLOps: prompts are versioned artifacts, evals gate releases, and fine-tuning is its own pipeline. LLMOps is MLOps plus prompt management, eval-driven deployment, and the cost/latency realities of giant models.

• Prompt versioning
• Eval-gated releases
• Fine-tuning pipelines
• Cost & latency control

Build itSketch the extra pipeline stages an LLM feature needs that a classic model does not — prompt versioning, eval gates, guardrails — and where each sits.

✓ CheckpointExplain what an eval-gated release means in practice, and why prompt versioning is a deployment concern.

P1 · Advanced · 60 min

Models rot. The world shifts under them — user behaviour changes, upstream data changes — and accuracy quietly decays with no error thrown. Monitoring watches for data drift and prediction drift so you catch decay before your users do.

• Data drift
• Prediction / concept drift
• Baseline distributions
• Alerting on decay

Build itPick a feature and define a drift check: what distribution do you compare against, and what change would trigger an alert?

✓ CheckpointExplain how you would notice a model degrading before a user complains.

P2 · Advanced · 60 min

Never trust a new model blindly. Shadow it against the old one, canary it to a slice of traffic, and A/B test to prove it actually improves the metric that matters. Rollout strategy is how you ship models without betting the whole system.

• Shadow deployment
• Canary releases
• A/B testing
• Guardrail metrics

Build itDesign the rollout for a new ranking model: shadow → canary → full. What metric decides promotion, and what triggers an automatic rollback?

✓ CheckpointExplain what a shadow deployment tells you that a canary does not.

P3 · Intermediate · 45 min

When a prediction goes wrong at 2am, logs are all you have. Trace each request, log inputs, outputs, model version, and latency, and alert on quality and error rates. Observability turns "the model is acting weird" into a debuggable timeline.

• Request tracing
• Prediction logging
• Latency & error alerts
• Debuggability

Build itDesign a logging schema for a prediction service: which 8 fields do you capture, and what condition fires an alert? Write it as JSON.

✓ CheckpointExplain what you would log to diagnose a bad prediction three days later — and what you would refuse to log.

P4 · Intermediate · 45 min

ML infrastructure bills add up fast — idle GPUs, oversized instances, redundant recomputation. Right-sizing compute, caching predictions, spot instances, and turning off what you are not using keeps the system affordable at scale.

• Right-sizing compute
• Caching & spot instances
• Idle-resource cleanup
• Cost attribution

Build itAudit a serving setup for waste: where would you cache, where would you downsize, and where is a GPU sitting idle you could reclaim?

✓ CheckpointExplain what makes a workload safe for spot instances, and what it must tolerate.

P5 · Intermediate · 45 min

In regulated or high-stakes settings you must answer: what data trained this model, who approved it, and why did it make that decision? Lineage and governance track the full provenance of every model and prediction for audit and compliance.

• Model & data lineage
• Audit trails
• Access control
• Compliance & documentation

Build itFor a model in a regulated domain, list what you would need to store to answer an auditor: training data, approvals, and decision traceability.

✓ CheckpointExplain what an audit trail must contain to answer “why did the model decide this, in March”.

P6 · Advanced · 60 min

The loop closes here. When monitoring flags a regression, you need a runbook: roll back to a known-good version, diagnose the cause, and trigger retraining on fresh data. Automated retraining pipelines turn decay from a crisis into a routine.

• Rollback runbooks
• Root-cause diagnosis
• Automated retraining
• Closing the loop

Build itWrite an incident runbook: a model's accuracy drops 10% overnight — what are your first five steps, in order?

✓ CheckpointExplain why automated retraining without a gate can make an incident worse rather than better.