Forward Deployed Engineer  /  POC → Production
FDE Track 09~13 min readIntermediate
From Demo to Depended-On

Turn the demo into
something they trust.

A demo runs once, on a good day, with you watching. Production runs thousands of times against systems that fail intermittently, with no one watching but the customer. The gap between the two is where a lot of otherwise-successful deployments quietly die. Hardening is how you cross it — deliberately.

01

The gap nobody demos

Here’s the trap that catches good engineers: the demo went great, the customer is excited, so it must be nearly done. It isn’t. A demo is a single run of the happy path under ideal conditions with you standing by. Production is thousands of runs, including every unhappy path, against systems that fail on their own schedule, with no one watching. The distance between those two is enormous, and it’s invisible from the demo.

Naming that gap is the first act of hardening. A prototype optimized for the demo has, by design, none of the machinery needed to survive real traffic — and that’s fine, until the moment a real user depends on it. Hardening is the deliberate work of building that machinery, and treating it as a real project (not a weekend of cleanup) is what separates deployments that last from ones that flame out three weeks after the applause.

02

Reliability: assume everything fails

Production is a hostile environment. A handful of patterns turns a fragile demo into a system that shrugs off the failures a demo never sees.

Every external call can time out, rate-limit, error, or silently double-deliver. Design as if it will. These patterns — each with a hands-on challenge in the track — are the core of reliability.

Sad pathHandle the failure of every external call: timeouts, retries with jitter, and a graceful fallback instead of a crash.
IdempotencyA re-run job or re-delivered webhook must not double-charge or double-write. Dedupe by a key.
BackpressureBounded queues, rate limits, and a dead-letter path for what you can’t process — don’t let a spike take you down.
ValidationReject or quarantine bad input at the edge, before it corrupts anything downstream.

None of this is exotic, and you don’t need all of it everywhere — you need it on the paths that touch the real world. If you internalize one thing, make it idempotency: it’s the pattern that makes every retry safe, and retries are how a distributed system survives a flaky one.

03

Observability: know before they tell you

The demo has perfect observability — you’re watching it live. Production has none unless you build it, and on a locked-down deployment you may never see the running system at all. So you instrument it to report on itself.

The minimum

  • Structured logs with a request/correlation id — trace one operation end to end.
  • Metrics + alerts on the one number that means “it’s working”; alert when it moves.
  • A real health check — readiness/liveness, not just “the process is up.”

The goal

  • Learn it broke before the customer tells you.
  • Diagnose a failure without shell access to the box.
  • Wire alerts into their channel too, so the right people know.

The bar to clear is simple and unforgiving: could you diagnose a 2 a.m. failure without touching the machine? If the answer is no, you don’t have enough observability yet — and on a deployment where you can’t reach the system, that gap is the difference between a five-minute fix and a lost account.

04

Security and config hygiene

Demos cut corners that production can’t afford: a key in the code, a value hardcoded to one customer, a deploy you did by hand and couldn’t repeat. Hardening closes them.

Demo shortcutProduction requirement
Secret in the code or a .env in the repoSecrets in a vault/KMS + env injection; nothing in logs
A broad admin credentialLeast-privilege access scoped to what the job needs
Values hardcoded to one customerPer-environment config; nothing baked to a single tenant
A hand-run deployOne command / pipeline to deploy and roll back — no snowflake servers

The last row is the sleeper. A deployment you can’t reliably reproduce or roll back is a deployment that will eventually strand you — a bad release with no way back, on a system you may not even be able to reach. A repeatable, reversible deploy isn’t polish; it’s the safety net the whole engagement rests on. (The full gate is the POC → production checklist.)

05

Harden, or rebuild?

Before you harden, ask whether you should. A prototype was built to be disposable, and its speed came from shortcuts. If those shortcuts are localized — a hardcoded config here, a stub there — hardening in place is efficient. If they’re woven through everything — no error handling anywhere, state smeared across globals, a structure that fought you even in the demo — a clean rebuild informed by everything you learned is often faster and always cleaner.

→ The honest default

Engineers over-attach to code they’ve written. Ask: “If I were starting today, knowing what I now know, would I build it this way?” If the answer is a clear no, rebuild. The prototype already did its job — it taught you the answer.

06

Hardening ends in handoff

A hardened system isn’t the finish line — a system the customer’s team can run without you is. So the last act of hardening points at the handoff: the reliability, observability, and repeatable deploys you just built are exactly what make it operable by someone else. Pair them with a runbook (how to deploy, roll back, rotate secrets, and fix the top few failures) and documentation their engineers can actually use, and you’ve turned a demo into an asset they own.

This is the difference between an FDE who ships and one who becomes a permanent, unpaid support team for everything they ever built. Harden it so it survives, then hand it off so it thrives — the checklist covers both, and the discipline of building for handoff from the start is what keeps you free to go solve the next customer’s problem.

Deep dive

The hardening gate, written as assertions

“Production-ready” is a conversation until it is a list someone can fail. Here is the list, in the order things actually break.

The gap between a working proof of concept and something a customer’s team depends on is not mostly code — it is the set of failure modes the POC was allowed to ignore. Writing them as assertions rather than as a document does two things: it makes the gate refusable, and it turns “is this ready?” from a judgement call into a check anyone can run.

what a POC is allowed to skip, and what production is not

Happy path only

The POC handled the cases you chose. Production meets the malformed PDF, the empty file, the 14MB attachment and the record from 2011.

You in the loop

The POC worked because you were watching. Production runs at 02:00 while their operator reads a runbook you have not written.

One data source

The POC read the clean export. Production reads the live system, which is being written to while you read it.

Each of the three has a fix that is cheap now and a migration later.
# hardening-gate — run before anyone says "production", fails loudly

correctness
  assert eval.overall >= human_baseline       # measured, not claimed
  assert eval.high_cost_error_rate <= budget
  assert eval.regressions == []              # the one that earns its keep

failure
  assert handles(malformed_input, empty_input, oversized_input)
  assert upstream_down_returns_a_message_not_a_timeout()
  assert every_write_is_idempotent()          # retries WILL happen
  assert rollback_tested_in_their_environment()

operability
  assert health_endpoint_names_the_failing_dependency()
  assert errors_say_what_to_do_not_just_what_happened()
  assert runbook_covers(last_five_incidents, symptom_first=True)
  assert logs_redaction_safe_and_exportable_by_them()

identity & evidence
  assert no_static_credentials_in_config()
  assert deactivating_a_user_revokes_sessions_and_tokens()
  assert audit_log_reconstructs_a_decision_six_months_later()

ownership
  assert their_engineer_has_deployed_it_once, "with you watching"
  assert named_owner_for(state, change_approval, first_response)

The assertion people argue about is the last one, and it is the one that decides whether this is a deployment or a dependency. An engineer of theirs who has deployed the system once, with their credentials, through their pipeline, while you watched, has surfaced every piece of undocumented knowledge you still hold — and that hour is worth more than the document you would otherwise have written instead.

→ The self-explaining error is the highest-leverage line

source system returned 403 — check the service-account credential expiry converts a 2am page into a self-service fix, permanently, for a team that is not you. It is worth more than any dashboard, and it costs one string. Every incident you handle should add one of these and one runbook entry — see incidents, on-call & SLAs.

Frequently asked

Quick answers

What does it mean to harden a proof of concept?

Hardening turns a demo into a system people can depend on: adding reliability (handling failures, retries, idempotency, validation), observability (structured logs, metrics, alerts, health checks), and security and config hygiene (secrets out of code, least privilege, repeatable deploys). A POC proves the value; hardening makes it survive real traffic against systems that fail — which a demo never has to.

Why do proofs of concept fail in production?

Because a demo only ever runs the happy path once, while production hits every sad path repeatedly: an external call times out, a webhook is delivered twice, a bad record arrives, the customer’s API rate-limits you. A POC built for the demo has none of the retries, idempotency, validation, or monitoring needed to survive that — so it works beautifully until real traffic hits, then fails quietly.

What observability does a deployed system need?

At minimum: structured logs with a request/correlation id so you can trace an operation end to end, metrics on the one number that means "it’s working" plus alerts when it moves, and a real health check. On a locked-down deployment where you can’t reach the system, this is doubly important — it’s how you learn it broke before the customer tells you.

Should you harden the prototype or rebuild it?

Harden it when the core is genuinely sound and its shortcuts are localized; rebuild when the prototype’s hardcoding and hacks are woven throughout, because untangling them is often slower than a clean rebuild informed by everything you learned. Treat it as a deliberate decision, not a default — and remember that a prototype was built to be disposable, so rebuilding is often the honest, faster path.

From POC to Production: Hardening · part of the FDE track · Vibe Engines · 2026
Finished this one? 0 / 208 Handbooks done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Handbooks