Forward Deployed Engineering
The trending customer-facing engineering role: discover the problem on site, prototype fast, integrate with the customer’s systems, deploy in their environment, demo, harden, and hand off — everything for becoming a Forward Deployed Engineer.
Handbooks 24
The Agent Evaluations Handbook
A self-contained handbook on evaluating AI agents — theory, interactive widgets, and practical guidance. Trajectory evals, tool-use scoring, LLM-as-judge, observability, and reliability for PMs, engineers, and founders.
The Agent Patterns Handbook
The design patterns behind every LLM agent — the ReAct Thought–Action–Observation loop, tool/function calling, plan-and-execute, reflection, memory, multi-agent orchestration, and the failure modes (loops, hallucinated tools, recovery, human-in-the-loop) that break agents in production.
The Kubernetes Handbook
The one idea under all the YAML — declare desired state, and a control loop makes reality match. Covers the orchestration problem, pods, deployments and replicasets, services and networking, the reconciliation loop and self-healing, the scheduler, config/secrets and health probes, autoscaling, and when you actually need Kubernetes.
The Observability Handbook
Seeing inside production — monitoring vs observability, the three pillars (metrics, logs, traces) and what each answers, structured logging, metric types and the cardinality trap, distributed tracing, the golden signals and SLIs/SLOs/error budgets, alerting on symptoms not causes, and correlating all three during an incident.
The AI Security Handbook
Your app now reads the internet and believes it. Why prompt injection is a design problem filters cannot solve, the lethal trifecta (private data + untrusted content + exfiltration), defense in depth from least privilege to output validation, where PII actually leaks (logs, embeddings, weights), MCP/model supply-chain trust, and the red-team eval suite that gates CI.
The Docker Handbook
A container is a lie told by the kernel — namespaces (what a process sees) plus cgroups (what it uses), not a VM. Image layers and the cache-ordering rule that halves build times, multi-stage Dockerfiles that ship small, volumes vs bind mounts, name-based networking, Compose, and the handoff to Kubernetes.
The API Design Handbook
An API is a promise you keep for years. Choosing REST vs gRPC vs GraphQL honestly, resource-modeling REST so consumers can guess it, contract-first gRPC and protobuf evolution, the retry-safe semantics that survive real networks (idempotency keys, cursor pagination, structured errors), versioning without breaking clients, and why agent tool schemas are the newest API surface.
RAG vs Fine-Tuning
The most common wrong turn in AI engineering — reaching for fine-tuning when you needed retrieval, or vice versa. The one distinction that decides it (knowledge vs behavior), a head-to-head table, when to use each, and why the strongest systems combine them: fine-tune for behavior, RAG for knowledge.
MCP vs Function Calling
Pitted against each other, but they live at different layers — like comparing USB-C to sending data. Function calling is the model capability to emit a structured tool request; MCP is the open standard for discovering and connecting to tool servers. How MCP uses function calling, and when each matters.
The LLM Observability Handbook
Tracing an AI request as a tree of spans. Why total latency is the critical path — the max end time, not the sum of durations, because parallel spans overlap — while cost is the sum across every span, how to find the bottleneck span, and what to log for every LLM call (including percentiles over averages). With worked math and runnable code.
The Evals in CI Handbook
Turn LLM quality into a regression gate, like unit tests for code. A golden set scored on every change, and why the gate needs both an aggregate threshold AND per-case no-regression — because a healthy average can hide a broken case (mean stays up while one case collapses). How to wire it into the PR pipeline, handle non-determinism with a tolerance band, and avoid a flaky gate. With worked math and runnable code.
The OAuth & Auth Deep Dive
How "Sign in with Google" lets an app act for you without ever seeing your password. OAuth swaps your credentials for a scoped, revocable token, delivered via a short-lived authorization code that's worthless on its own — turning it into a token needs a back-channel exchange the browser can't make. Two parameters guard the flow: state (a random value echoed back, blocking forged/CSRF responses) and PKCE (challenge = SHA-256(verifier), so a stolen code can't be redeemed without the secret). Plus authentication vs authorization, OIDC/ID tokens, and the traps (implicit flow, unvalidated tokens, wide scopes, token leaks). With worked math and runnable code.
The Forward Deployed Engineer: Role & Mindset
The fastest-growing engineering title in AI, demystified: what an FDE actually does (owns the customer outcome, not the ticket), how it differs from a software engineer, solutions engineer, and consultant, the mindset shifts it demands, a week in the life, and how to break in.
Customer Discovery & Scoping for FDEs
How FDEs find the real problem before building the wrong one: the wrong-problem tax, watching users instead of asking, the questions that surface truth, defining success as one number, scoping the smallest slice that proves value, and the red flags that mean a deployment will stall.
Customer Integration Patterns for FDEs
Integration is where POCs die. The FDE playbook: the five integration families (REST/GraphQL, webhooks, files/SFTP, database/CDC, queues), auth into customer systems (API keys, OAuth, mTLS), reliability at the edge (idempotency, retries with jitter, webhook verification, dead-letter), and mapping messy customer data.
Rapid Prototyping for FDEs
How FDEs build a convincing prototype in days, not months: throwaway-first thinking, the smallest demoable vertical slice, choosing a boring fast stack, faking the non-core parts honestly (hardcode / stub / Wizard-of-Oz), demo-driven development, and knowing when to throw it away versus harden.
Deploying in Customer Environments
Ship where the customer can actually run it: the four deployment targets (your SaaS, the customer’s cloud/VPC, on-prem, and air-gapped), what changes as you move right — data egress, updates, model hosting, secrets, observability, and your own access — plus packaging portably and the air-gapped update path.
The Demo & Storytelling Playbook
A demo is a performance, not a code review. How FDEs prep (their data, rehearse, a fallback recording), structure the story (open with the problem, show-then-explain, land the "so what"), recover from a live failure without debugging on stage, read the audience (exec / IT / user), and follow up while it’s hot.
Security & Compliance for FDEs
Enough security to earn a customer’s "yes": what SOC 2, HIPAA, GDPR and PCI actually ask, how to handle data and secrets, what a security review checks, and the AI-specific risks (prompt injection, data leakage to model providers, PII in prompts) an FDE has to manage.
Solutions Engineering & Stakeholder Management
The non-code half of the FDE job: mapping the room (champion, blocker, decision-maker, user), managing scope creep with a visible later-list, saying no kindly, setting expectations and surfacing risk early, and multi-threading so one person can’t sink the deployment.
From POC to Production: Hardening
The gap between "it demoed" and "people depend on it": how FDEs harden a proof of concept into a real system — reliability (sad-path handling, idempotency, backpressure, validation), observability (structured logs, metrics, health checks), security and config hygiene, and the harden-versus-rebuild decision.
Handoff & Enablement for FDEs
How FDEs hand off a deployment without becoming a permanent, unpaid support team: why the exit is the goal, what to hand off (runbook, docs, architecture, who-to-call), how to write a runbook that survives 2 a.m., enabling the customer’s team hands-on, and defining the exit up front.
Learning a Customer’s Domain Fast
How FDEs get dangerous in a new field in a week: why domain fluency is a superpower, learning from the customer’s real data, running SME interviews that teach you (the naive-question license), building a glossary of their terms, finding the 80/20 of a domain, and how fluency compounds trust.
AI App Patterns for FDEs
The AI toolkit a modern FDE deploys: RAG over the customer’s documents, agents that drive their workflows, and evals as the acceptance test — plus how to choose between prompting, RAG, agents, and fine-tuning, and how to ship AI honestly with a good-enough bar and a human in the loop.
Roadmaps 1
System Designs 4
Design Webhook Ingestion at Scale
Build a reliable webhook ingestion pipeline — the backbone of every integration. Learn signature verification and replay protection, acknowledging fast with a durable queue, idempotent processing that turns at-least-once delivery into effectively exactly-once, bounded retries with a dead-letter queue, replay after a fix, and monitoring consumer lag and DLQ depth.
Design a Customer Data Integration Pipeline
Build the pipeline that ingests a customer’s messy data on almost every deployment. Learn source connectors (API, SFTP, database, change-data-capture), staging, validation and mapping to a canonical schema, quarantining bad rows instead of dropping them, incremental sync by checkpoint, idempotent upsert loads with backfill, and reconciliation that proves the data landed.
Design Multi-Tenant Customer Isolation
Serve many customers from one system without ever leaking data between them. Learn tenant context propagation, the data-isolation spectrum (row-level, schema-per-tenant, database-per-tenant), preventing noisy neighbors with per-tenant quotas, per-tenant encryption keys and config, and audit plus automated cross-tenant tests that make isolation provable.
Design an Audit Logging & Compliance Trail
Build the audit system that passes a customer’s security review. Learn structured capture of who-did-what-when, an append-only immutable (WORM) store, tamper-evidence with hash chaining, retention and archival for compliance windows, and access-controlled querying that redacts sensitive data and audits the auditors.
AI System Designs 2
Deploy an LLM in a Customer’s Environment
Deploy a large language model inside a customer’s own environment — the constraint a forward deployed engineer meets when a regulated customer won’t send data to a public API. See why a hosted endpoint is a non-starter, right-sizing the model and precision to the fixed hardware they actually own, serving efficiently on limited GPUs, running with no network egress (air-gapped, no phone-home), shipping model and security updates as signed bundles into a locked-down environment, and getting observability out without exfiltrating customer data.
Design Secure Document Ingestion + RAG
Build a retrieval-augmented assistant over a customer’s sensitive internal documents, where security — not relevance — is the hard constraint. See why naive RAG leaks documents across users, classifying and tagging access on ingest, permission-aware retrieval that intersects relevance with what the user may see, redacting PII and secrets before the prompt and logs, defending against prompt injection carried inside untrusted documents, and citing sources plus auditing every access so the pipeline can pass a security review.
Coding Challenges 9
Retry with Exponential Backoff
Tools time out and APIs rate-limit — a production agent retries without hammering. Exponential backoff waits longer after each failure (1s, 2s, 4s, 8s…) up to a cap. Compute the delay schedule: delay[i] = min(cap, base·2ⁱ). Solve it in Python or TypeScript.
Verify a Webhook Signature
Every real integration sends signed webhooks — and every FDE has to verify them. Recompute the signature over the payload, compare in constant time, and reject stale events to stop replay attacks. Solve it in Python or TypeScript, with hidden tests.
Idempotency Key Handler
Networks retry. A webhook fires twice; a payment is re-sent after a timeout. Without idempotency you double-charge or double-write. The fix: dedupe by a client-supplied key and always return the first result. Solve it in Python or TypeScript, with hidden tests.
CSV → Schema Mapper
Every customer hands you a messy export. Before it flows into your system, each row must be coerced to a schema — strings to ints, "true" to booleans — and the rows that don’t fit quarantined, not silently dropped. Solve it in Python or TypeScript, with hidden tests.
Data Reconciliation Diff
The customer swears the migration worked. You reconcile: compare source records against what landed in the target, keyed by id, and report exactly what was added, removed, or changed. Solve it in Python or TypeScript, with hidden tests.
PII Redactor
Before customer data touches a log line — or a third-party LLM — the personal bits have to go. Mask emails, SSNs, credit cards, and phone numbers with pattern matching, leaving the rest readable. Solve it in Python or TypeScript, with hidden tests.
Config Validator
Half of "it doesn’t work at the customer" is a bad config — a missing key, a string where an int belongs, an env that isn’t allowed. Validate it up front and fail loudly with a clear list instead of crashing three layers deep. Solve it in Python or TypeScript, with hidden tests.
API Pagination Collector
The customer’s API returns 100 rows at a time behind a cursor. You need all of them. Follow the "next" pointer page by page until it runs out — the loop behind every "sync everything" integration. Solve it in Python or TypeScript, with hidden tests.
Exponential Backoff with Full Jitter
When a customer’s API throws 503s, everyone retrying on the same doubling schedule stampedes it in sync. Full jitter spreads the retries randomly across the window so the herd disperses. Compute the delay schedule. Solve it in Python or TypeScript, with hidden tests.
Labs 5
The Deployment Target Chooser
Don't guess where a customer solution should run — answer the constraints and watch the recommendation fall out. The FDE question that reshapes an architecture most is "where can this run?". Toggle whether data can leave, whether it's regulated, whether there's internet, and how fast you must ship updates, and see how the four targets — SaaS, customer VPC, on-prem, air-gapped — score, with the reasoning behind the winner. Made playable, with theory and a quiz.
The Integration Reliability Lab
Don't read about reliable integrations — watch a flaky endpoint drop and duplicate events, then fix it. Send a stream of deliveries through an endpoint that sometimes fails or times out, and toggle the three patterns that make it survive real traffic: retries (fewer losses, but new duplicates), idempotency (kills the duplicates), and a dead-letter queue (turns loss into something recoverable). Watch the exactly-once delivery rate climb — made playable, with theory and a quiz.
The Data Mapping Lab
Don't read about data mapping — do it, and watch rows pass or fail. Every customer hands you a messy export; before it flows in, each column must be mapped to a canonical field and coerced to a type, with the rows that don't fit quarantined, not silently dropped. Pick which column feeds each field and watch validation light up green and red, row by row — a wrong mapping fails everything, a right one quarantines only the bad values. Made playable, with theory and a quiz.
The Rollout Strategy Lab
Don't read about rollout strategies — ship a bug and watch the blast radius. How you roll out a change to a customer's users decides how many people a bad release hits before you catch it: big-bang exposes everyone, a canary exposes 1% first. Pick a strategy, ship a buggy or healthy release, and see the blast radius — and the rollout speed you trade for safety. Made playable, with theory and a quiz.
The Scoping Simulator
Don't read about scoping — cut it under a deadline. You have a two-week budget and a wishlist. Pick the features for a proof of concept and watch the verdict: does it include the one vertical slice that proves value, and does it fit the timebox? Overload it and you ship nothing; skip the core and you prove nothing. Find the smallest slice that makes the customer believe — made playable, with theory and a quiz.