One engagement,
nine finished artifacts

Three intake teams (A: 14 adjusters, 450 claims a week; B: 11, 420; C: 9, 330) share one queue. The engagement runs 12 weeks with one FDE. Everything you will read sits on that spine:

Current workflow, observed (12 claims shadowed)

1. Claim email lands in the shared inbox with 1–9 attachments (PDF forms, phone photos, contractor estimates).
2. An adjuster opens each attachment and re-keys policy number, loss date, cause and amounts into the case system.
3. If evidence is missing, they email the policyholder and park the claim until a reply.
4. They route by severity and type to a specialist queue.

Evidence heard

• Fact: 1,200 claims a week (system report, last 8 weeks, range 1,090–1,310).
• Fact: baseline from a 150-claim sample over two weeks: median triage 22 minutes, p90 51 minutes, 31% bounced for missing evidence on first contact.
• Assumption to validate: “most time is reading PDFs.” Partly true: 41% is reading, but 26% is chasing evidence a checklist could have requested on day one.
• Assumption to validate: that handwritten contractor estimates are rare. Team C says they are most of their work.

Open questions

• What share of claims carry a handwritten estimate? (Answer in week 3: 22% overall, 58% of Team C’s.)
• Who is allowed to switch an automated feature off at 09:00 on a Monday?
• Is any adjuster time actually removable, or is the value cycle time? Finance must say before week 10.

Owner and approver: claims director approves scope; the Team A lead owns the pilot; the security lead holds a veto on architecture. Decision date: go/no-go at the end of week 4 on the first 100 live claims.

Context and constraints

Claim attachments hold names, addresses, policy numbers and photos of homes. Security forbids sending them to an unapproved model vendor. Processing must stay in the customer’s region and support deletion on request. Procurement for new infrastructure takes 14 weeks; the engagement has 12.

Options considered

Decision and why

Option B. It is the only option that passes security and fits the calendar. We accept a smaller model menu and pay for it with an eval gate (Artifact 5) so a model swap is a measured change, not a leap of faith.

Data flow and trust boundaries

1. Mail gateway → ingest. The endpoint acknowledges immediately and enqueues; it never does extraction inline. Every message carries an idempotency key (claim id + attachment hash).
2. Redaction. Names, addresses and policy numbers are replaced with tokens before any model call. The token map stays in the claim store.
3. Extraction by the in-tenant model, one claim at a time, with no tools and no memory across claims.
4. Deterministic validator. Policy active on the loss date, loss date inside coverage period, line items sum to the stated total. Failing fields are flagged, never silently corrected.
5. Router. Confidence below 0.8, or any validator failure, goes to a human with the reason. Otherwise it suggests a route.
6. Claim store keyed by claim id; 30-day retention; a delete API removes the claim, tokens and logs.

Failure modes and rollback

A feature flag has three states: on, shadow (processes and logs, shows adjusters nothing) and off. Shadow exists so that “disable now” never means “lose the queue.”

Threats and mitigations

Questions for security review

• Is the token map in the same trust zone as the claim store, or does it need a separate key?
• Does 30-day retention satisfy the regulator’s record-keeping rule, or must the originals be kept elsewhere?

Setup and test path (what a reviewer does)

1. Run the seed script; 40 claims appear in the staging queue.
2. Open claim 17 (clean PDF): fields extracted, route suggested, no flags.
3. Open claim 31 (phone photo, blurry): low-confidence flag, human route with the reason shown.
4. Open claim 36 (handwritten estimate): fields partly extracted, amounts flagged for human entry.
5. Set the flag to shadow: the queue stops showing suggestions within one minute; ingest continues.

Failure behaviour

Known limitations (stated, not hidden): no languages other than English; photos under 300×300 pixels are always routed to a human; handwritten estimates are extracted at low accuracy and mostly routed to a human (see Artifact 5).

Dataset: golden-v3, 240 claims, versioned in the repository, provenance recorded (sampled from week 3–6 live claims, anonymised, labelled by two adjusters with disagreements settled by the Team A lead). Sliced by attachment type, because that is where the harm hides.

Metrics, thresholds and why

• Silent error rate (wrong and not flagged) is the headline, not accuracy: a wrong answer a human sees costs seconds; a wrong answer nobody sees costs a mis-routed claim. v0 had 89 silent errors (37.1%); v2 has 10 (4.2%). The gate is ≤5% overall and ≤8% on every slice.
• Injection slice must pass 15/15. A single failure blocks the release.
• No slice may regress by more than 2 points against the last released version, so a model swap that lifts the average by hiding a worse photo performance cannot ship.

Error analysis

Of the 25 v2 errors, 11 are handwritten estimates (the digit 1 read as 7 in contractor script, and amounts split across two columns). Eight of those are caught by the validator because the line items stop summing to the total; the three silent ones all have a total that the model also misread consistently. That points at one fix: for handwritten estimates, extract line items and total independently and compare them. It is the next experiment.

CI gate

Every change to a prompt, a model version, the redaction rules or the validator runs the full 240 cases on pull request. The job fails if any threshold above is missed and posts the slice table in the pull request. Cost per full run: about 240 × $0.18 = $43, so it runs on change, not on every commit.

Severity: SEV2. Detected: 09:12. Resolved: 10:46. Duration: 94 minutes.

Customer impact

About 310 claim tasks entered the queue between 09:00 and 10:00 (Monday peak). 25 of them (8%) were duplicates, each assigned to an adjuster, who opened and began re-keying a claim another adjuster already held. No claim was mis-decided; no policyholder was contacted twice. Adjuster time lost: roughly 25 duplicate tasks × 12 minutes = 5 hours.

Detection and response timeline

Contributing factors (not blame)

• The architecture record (Artifact 3) listed an idempotency key as a mitigation. It was written, then deferred in week 4 as “low risk on staging volume” and not tracked. The control existed on paper only.
• Ingest did inline extraction, so any latency spike turned into gateway retries.
• The deploy window met Monday peak with cold workers. The runbook had no warm-up step.
• The disable switch had two states, so the first instinct was the destructive one.

What worked and what did not

Worked: the shadow state (created in week 3 for exactly this) turned a demand to disable into a 5-minute mitigation; the dashboard showed duplicate rate directly. Did not: the first customer message was sent after 80 minutes; nobody had been told who sends it.

Corrective actions

Original hypothesis

Pre-extracting facts and flagging missing evidence at intake will cut median triage time from 22 to 14 minutes or less and cut first-contact missing-evidence from 31% to 20% or less.

Baseline and measurement method

The same 150-claim sampling method as week 2, repeated in weeks 10–11 over claims handled through the tool and, separately, claims handled the old way. Medians, not means; time stamped from claim opened to route confirmed.

Outcome and adoption

• Median triage, tool-handled: 12 minutes (baseline 22; target ≤14). Met. Claims handled the old way: still 22.
• First-contact missing-evidence, tool-handled: 19% (baseline 31%; target ≤20%). Met. Across all claims it is 24%, because 46% of claims never touch the tool.
• Guardrails: high-severity mis-routes 0.3% (limit 0.5%); zero automated decisions; silent error rate in production sample 3.9% (limit 5%).

Why Team C is at 12%

Five interviews and the flag log agree: 58% of Team C’s claims carry a handwritten estimate, and the tool routes most of those to a human with the reason shown. To Team C the tool looks like an extra step that mostly says “do it yourself.” The eval predicted this in week 6; the adoption gap is the product behaving as designed in front of the wrong population.

Costs

Run cost: about $0.18 per claim × 643 = $116 a week. Build: one FDE for 12 weeks. Change management: two training sessions and the Team A lead’s time.

The argument with finance, kept honest

The naive claim: 643 claims × 10 minutes saved = 6,430 minutes = 107 hours a week. At a loaded $38 an hour that is about $4,070 a week against $116 of run cost. Finance’s objection was correct: no headcount line moves, so 107 hours is capacity, not cash. The review therefore states the value as faster first response (22 to 12 minutes on half the claims) and fewer evidence bounces, and shows capacity separately, labelled as capacity. It does not put 107 hours in the savings column.

Risks and confounders

• Team A was the pilot and has the cleanest claims; its 12-minute median may not transfer to Team C’s mix.
• Weeks 10–11 were quiet; peak-week behaviour is unmeasured.
• Adjusters may have timed themselves differently once they knew they were measured.

Service owner: operations manager (new). Backup: Team A lead.

What it does and does not do

Reads incoming claim emails, extracts six fields, flags missing evidence, suggests a route. It does not decide coverage, pay anything, or contact policyholders. A suggestion is never final until an adjuster confirms.

Routine operation

• Morning: open the dashboard; queue depth under 300 and duplicate rate near 0 is normal.
• Weekly: run the 240-case eval on the pinned version; file the slice table.
• Never deploy between Sunday 22:00 and Monday 12:00.

Common failures and diagnosis

Escalation and disabling

Sev1 (wrong decisions or exposed data): set off, call the security lead, then the vendor contact. Sev2 (duplicates, slowness): set shadow first. Shadow keeps intake running and is reversible in one click; off stops assistance but intake still works. Comms owner sends the first message within 30 minutes.

Handoff test

Operator: the new operations manager. Date: Thursday of week 12, FDE in the room, silent. Task: read the dashboard, diagnose a seeded latency fault, set shadow, restore, and explain the state to a manager. Result: passed in 6 minutes. Gaps found (2): the alert names did not match the dashboard panels, and “shadow” was undefined in the glossary. Both were fixed the same day. The demo video (7 minutes) shows the happy path, a flagged photo, and the shadow switch.

Reusable problem pattern

“Intake with scattered evidence”: a person re-keys facts from semi-structured attachments and discovers missing evidence late. Any insurer, lender or claims-like workflow has it. The reusable parts are the redaction-then-extraction pipeline, the validator-and-router shape, the shadow state, and the slice-based eval gate.

Customer-specific constraints

• The 14-week procurement that ruled out self-hosting.
• The 30-second gateway retry that caused the incident.
• Handwritten estimates as a dominant input for one team.
• The coverage-period validator, which encodes this insurer’s policy rules.

Product versus configuration

Evidence still needed

One engagement is one data point. We have not shown the pipeline works for a second insurer’s forms, in another language, or at 10× volume.

Next-customer validation plan

Run discovery at a second insurer with the same 150-claim baseline method; deploy in shadow only for two weeks and compare silent error against this engagement’s 4.2%. Productise only if two of the four product items above survive unchanged.

Productisation risks

• A generic “extraction product” invites the claims-agent scope this engagement rejected.
• Maintaining golden datasets across customers is a cost, not a feature.
• Handwriting accuracy may be the whole market, and it is the weakest result in the page.

Common mistakes when you write your own

• No baseline. You finish with a nice number and nothing to compare it with.
• One accuracy figure. It hides the slice a customer team lives in.
• A threat model nobody tracks. Mitigations without tickets are decoration.
• Claiming hours as savings. Finance will ask which budget line moved.
• A runbook written, not tested. The first unassisted operator is the test.

Where this shows up in interviews

The take-home defence, the decomposition round and the behavioural “tell me about a failure” all reward the same evidence: a baseline, a slice, an owned miss and a number you can defend. Bring one of these artifacts to the loop.