CODING CHALLENGE · N°76

Ship or Stop

Medium AI EngineeringAgentsEvaluationSelf-Improvement

A self-improvement loop finished another round and every number on the dashboard went up. Write the gate that decides whether it ships: six stop conditions over two rounds of measurements, plus a canary nobody is allowed to read early.

The problem

Implement ship_or_stop(prev, curr, canary). prev and curr are one round of measurements each, with keys verifier_pass, human_pass, frozen, distinct_per_task, memory_rows, memory_hit_rate, can_roll_back, weights_updated and training_set_reproducible. canary is {"runs": int, "wins": int, "min_runs": int}. Return {"ship": bool, "stops": [name, …]} where ship is True only when stops is empty, and stops holds these names in exactly this order, each appended only if its condition holds: "reward-hacking", "frozen-set-down", "diversity-collapse", "memory-bloat", "unrollable", "unreproducible", "canary-too-small", "canary-not-ahead".

EXAMPLE 1

Input a round where every measurement improved and the canary ran 200 times, winning 130

Output {'ship': True, 'stops': []}

nothing fired — this is the only shape that ships

EXAMPLE 2

Input verifier_pass 0.61 → 0.74 while human_pass 0.48 → 0.49

Output {'ship': False, 'stops': ['reward-hacking']}

the number you control moved and the one you do not stayed flat

EXAMPLE 3

Input frozen 0.540 → 0.539, everything else improved

Output {'ship': False, 'stops': ['frozen-set-down']}

no tolerance at all — one tenth of a point down is a stop

EXAMPLE 4

Input a clean round, canary {"runs": 20, "wins": 11, "min_runs": 200}

Output {'ship': False, 'stops': ['canary-too-small']}

ahead, and on far too few runs to know it — read it early and the noise will flatter you eventually

CONSTRAINTS

reward-hacking: curr["verifier_pass"] - prev["verifier_pass"] > 0.01 and the same delta for human_pass is <= half of it.
frozen-set-down: curr["frozen"] < prev["frozen"]. Any drop. Equal is not a drop.
diversity-collapse: curr["distinct_per_task"] < prev["distinct_per_task"] * 0.8.
memory-bloat: the store grew (memory_rows up) and memory_hit_rate did not (<= the previous one).
unrollable: curr["can_roll_back"] is falsey.
unreproducible: curr["weights_updated"] is truthy and curr["training_set_reproducible"] is falsey. A round that updated no weights never fires this.
canary-too-small: canary["runs"] < canary["min_runs"]. Check this before dividing — a canary with 0 runs must not raise.
canary-not-ahead: only when the canary is big enough, and then wins / runs <= 0.5.

SOLVE IT YOURSELF

Your turn — write it

Edit the stub, hit Run (or ⌘/Ctrl + Enter), and watch the hidden tests. Stuck? the hints are right above and Reveal solution is one click away.

YOUR TASK

Implement ship_or_stop(prev, curr, canary) — six stop conditions over two rounds of measurements plus two canary guards, appended in the order given, returning {"ship": bool, "stops": [...]}.

HINTS — 5 IDEAS

Build one list and append in the statement’s order. Never sort it — the caller reads the first entry as the most urgent thing to look at.
reward-hacking needs BOTH halves: the verifier delta must exceed 0.01 AND the human delta must be at most half of it. A verifier gain the humans matched is not hacking.
frozen-set-down uses a bare `<`. An unchanged frozen set is not a drop, and there is no "significant" threshold to apply.
unreproducible is guarded by weights_updated. A round that only changed a prompt cannot fire it, however little paperwork it left.
Do the canary last, and check runs < min_runs BEFORE computing wins / runs, or a canary that has not started raises ZeroDivisionError instead of reporting canary-too-small.

CPython · WebAssembly

Approach, complexity & discussion — open after you solve

The approach

Check the conditions in the order the statement lists them and append as you go — the order is part of the contract, so do not sort at the end. Every condition compares curr against prev, because none of these numbers means anything as a single reading. Do the canary last, and guard the division: a canary with zero runs must report canary-too-small, not raise.

Complexity

Time and space O(1) — a fixed number of comparisons. The difficulty is entirely in the thresholds and the edge cases, which is the point.

Common mistakes

Sorting or de-duplicating the stop list. The order is specified; a caller reads the first entry as the most urgent.
Using <= on the frozen set. It has no tolerance: any drop is a stop, and an unchanged frozen set is not.
Computing wins / runs before checking runs, so a canary that has not started yet raises ZeroDivisionError.
Firing unreproducible when no weight update happened. The condition is about a weight update you cannot re-derive, not about missing paperwork on a prompt change.
Treating a round with identical measurements as a stop. Nothing got worse, so nothing fires.

Where this shows up

This is the gate between “the loop ran” and “the loop shipped”. Every condition here exists because some number that went up was mistaken for an agent that got better: a verifier agreeing with itself more, a model narrowing onto its own habits, a canary read early because it happened to be ahead. Writing it as code rather than a wiki page is what makes it a gate instead of a good intention.

Finished this one? 0 / 76 Challenges done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

AI Agents & Tools