POC → Production Checklist

The gate a demo must clear to become a system people depend on: reliability, observability, security, and a real handoff.

A proof of concept shows that something can work; production means it keeps working when inputs are messy, dependencies fail, traffic spikes and the person who built it is on vacation. The gap is rarely the core logic — it is everything around it: failure handling, visibility, security, releases and ownership.

Use this as a gate review before customers depend on the system. Each row is a question with a yes-or-no answer, and every “no” gets an owner and a date, or a written decision to accept the risk.

Reliability

Handle the sad path: Every external call can fail. Timeouts, retries (with jitter), and a graceful fallback.
Idempotency: Re-running a job / re-delivering a webhook shouldn’t double-charge or double-write.
Backpressure & limits: Bounded queues, rate limits, and a dead-letter path for what you can’t process.
Data validation: Reject/quarantine bad input at the edge; don’t let it corrupt downstream.
Timeouts everywhere: every network call has one, and together they fit inside the caller’s own deadline
Circuit breaker: stop calling a failing dependency for a while instead of piling up retries
Graceful degradation: decide in advance what the product does when the model or a dependency is down

Performance & capacity

Load test at 2–3× peak: find the breaking point before customers do; record p50, p95 and p99 latency
Latency target: e.g. p95 under 2 seconds for the main flow, measured from the user’s side
Connection pooling: reuse database and HTTP connections — a new connection per request exhausts servers
Caching: cache expensive repeatable results, with explicit expiry and invalidation rules
Async for slow work: queue long jobs and return a job id (202) instead of holding the request open

Observability

Structured logs: With a request-id / correlation-id you can trace end to end.
Metrics + alerts: Track the one metric that means "it’s working"; alert when it moves.
Health check: A real readiness/liveness endpoint — not just "process is up".
Know when it’s broken: Before the customer tells you. Wire alerts into their channel too.

Data

Reversible migrations: each schema change has a tested way back, or ships as expand-then-contract
Restored backups: a backup only counts once a test restore has worked — put one on the calendar
Retention and deletion: know how long data is kept and how a deletion request reaches every copy
PII out of logs: redact personal data before it reaches logs, traces or prompt captures

Security & config

Secrets out of code: Vault/KMS + env. No keys in the repo, no keys in logs.
Least-privilege access: Scope every credential to what the job needs.
Config, not constants: Per-environment config; nothing hardcoded to one customer.
Repeatable deploy: One command / pipeline to deploy + roll back. No snowflake servers.
Dependency scanning: CVE scans on libraries and container images, with a deadline for patching
Audit log: who did what and when, for every admin action and data access

Release & rollback

CI on every change: tests, lint and a security scan must pass before anything merges
Staging like production: the same configuration shape and infrastructure as code, at a smaller scale
Canary or blue-green: send a small share of traffic to the new version first and watch the key metric
Feature flags: ship code dark and switch it on per customer — the cheapest rollback there is
Rollback drill: practice a rollback before launch day, and time it
Pinned dependencies: lockfiles and image digests, so a rebuild cannot quietly change behavior

AI-specific

Eval gate in CI: a golden set that must stay above threshold before a prompt or model change ships
Pin model versions: call a dated model snapshot, not a moving alias, and track deprecation dates
Prompts in version control: reviewed like code, with the prompt version logged on every call
Validate outputs: schema-check structured output; handle refusals and truncated responses explicitly
Fallback path: a second model, a cached answer or a human queue for when the model fails
Token and cost budget: per-request token caps and spend alerts — one runaway loop can burn a month’s budget
Human review for high stakes: route money, legal and medical outputs to a person before anything acts on them

Operations

On-call owner: a named rotation with access, alerts routed to it, and an escalation path
SLO and error budget: 99.9% availability allows 43.2 minutes of downtime in a 30-day month
Golden-signal dashboards: traffic, errors, latency and saturation for every service
Incident process: who declares an incident, who communicates, and a blameless review afterwards

Handoff

Runbook: How to deploy, roll back, rotate secrets, and fix the top 3 failures.
Docs their team can use: Architecture + integration points + who to call. Not just your commit history.
Train the owners: Walk their engineers through it; watch them operate it once.
Define the exit: Who owns it now, what you still support, and for how long.

Recipes

Error-budget math for common SLOs

30-day month = 43,200 minutes

SLO        downtime allowed per 30 days
99%        432 min    (7.2 hours)
99.5%      216 min    (3.6 hours)
99.9%      43.2 min
99.95%     21.6 min
99.99%     4.32 min

Each extra nine cuts the budget tenfold and costs far more than tenfold to achieve. Pick the SLO from what users actually notice, and spend the unused budget on shipping changes.

A production-readiness review

[ ] the metric that means "it works" is on a dashboard, with an alert on it
[ ] every external call has a timeout, retries with jitter, and a fallback
[ ] a load test at 3x expected peak passed; p95 latency is recorded
[ ] secrets live in the vault; none in the repo, CI output or application logs
[ ] deploy and rollback are one command each; the rollback was rehearsed and timed
[ ] backups exist, and a restore was tested this month
[ ] evals gate prompt and model changes in CI
[ ] the runbook covers the top five failures, and their on-call has read it
[ ] a named owner on the customer side has accepted the system

Run the review with the team that will own the system. Every unchecked box leaves the meeting with an owner and a date, or with a written decision to accept the risk.

Retries with exponential backoff and jitter

import random, time

def call_with_retries(fn, attempts=5, base=0.2, cap=10.0):
    for attempt in range(attempts):
        try:
            return fn()
        except TransientError:              # timeouts, 429, 503 - not 400 or 401
            if attempt == attempts - 1:
                raise
            delay = random.uniform(0, min(cap, base * 2 ** attempt))   # "full jitter"
            time.sleep(delay)

Retry only errors that can succeed on a second try, and only operations that are idempotent or carry an idempotency key. Without jitter, every client retries in lockstep and hammers the service as it recovers.

A runbook entry someone can follow at 3 a.m.

Alert:     extraction queue depth > 500 for 10 minutes
Impact:    new documents wait; users see "processing" for longer than usual
Check:     dashboard "Extraction": worker count, model error rate, p95 latency
Likely:    the model provider is rate-limiting us (429s in the worker logs)
Fix:       kubectl -n app scale deploy/extractor --replicas=2      # cut concurrency
           then raise the provider quota, or turn on the fallback-model flag
Escalate:  after 30 minutes, page platform on-call and post in #incidents

Write for someone who has never seen the system: exact dashboard names, exact commands, and a clear point at which to stop and escalate.

Roll out a change behind a canary

1. deploy v2 next to v1 and route 5% of traffic to v2
2. compare v2 with v1 for 30 minutes: error rate, p95 latency, the business metric
3. healthy? go to 25%, then 50%, then 100%, comparing again at each step
4. any regression: send 100% back to v1 (the rollback) and investigate
5. for prompt or model changes, also compare eval scores and a sample of real outputs

A canary only helps if someone, or something, is watching the comparison. Automate the check and the rollback once the process has been run by hand a few times.

Questions about POC → Production Checklist

What does it take to move a proof of concept to production?

Mostly the work around the core feature. Production systems need failure handling for every dependency, with timeouts, retries and fallbacks; monitoring and alerts on the metric that proves the system works; secrets management and least-privilege access; repeatable deployments with a rehearsed rollback; backups that have actually been restored; and load testing against realistic peaks. AI systems also need an evaluation suite that gates changes, pinned model versions and cost limits. Finally, someone must own it: a runbook, an on-call rotation and a named owner on the customer side.

What is an SLO and an error budget?

A service level objective is a target for how reliable a service should be, measured from the user’s point of view, such as 99.9 percent of requests succeeding within two seconds over 30 days. The error budget is the failure that target allows: at 99.9 percent, about 43 minutes of downtime, or one failed request in a thousand, each month. Teams spend the budget deliberately on releases and experiments, and when it runs out they slow down and fix reliability first. This turns arguments about reliability versus speed into a shared, measurable rule.

Why do AI proofs of concept often stall before production?

Usually for reasons that have little to do with the model. Common ones are a success metric that was never written down, so no one can say whether the pilot worked; demos built on clean sample data that collapse on real, messy inputs; integrations with authentication, data access and security review left until the end; no evaluation set, so every prompt change is a gamble; costs and latency that were never measured at production volume; and no owner on the customer side to run the system after the builders leave. Each is cheap to fix early and expensive late.

What is a runbook?

A runbook is a written, step-by-step guide for operating a system and responding to its known problems. Each entry typically names the alert or symptom, its impact on users, what to check first, the most likely causes, the exact commands or switches that fix it, and when to escalate and to whom. Good runbooks are written for someone who did not build the system and may be paged at night, and they are tested by having that person actually follow them. Keep them next to the alerts that link to them, and update them after every incident.

Go deeper

FDE playbook →Deployment targets →