Vibe Engines Learn
CODING CHALLENGE · N°65

Write a Verifier

Medium AI EngineeringRLVRVerificationAgents

In RL from verifiable rewards, the verifier IS the reward. A gameable verifier is worse than no verifier — every false positive is a lie the model happily learns to reproduce. Write one that a reward-hacking agent cannot fool: recompute the score from ground truth, demand exact coverage, and compare with types intact.

The problem

Implement verify(submission, answer_key). answer_key maps each task_id to the correct answer. submission is what the agent hands back: {"outputs": {task_id: value, …}, "reported_reward": float}. Return {"reward": float, "accepted": bool} under three rules that close three reward hacks: (1) never trust reported_reward — recompute the reward yourself from outputs vs answer_key; (2) accepted is False (reward 0.0) unless outputs covers exactly the answer-key task ids — nothing missing, nothing extra; (3) a task counts correct only on a type-strict match (1 is not True, "1" is not 1). When accepted, reward is (# correct) / (# tasks).

EXAMPLE 1
Input submission={"outputs": {"t1": 4, "t2": 9}, "reported_reward": 1.0}, answer_key={"t1": 4, "t2": 9}
Output {'reward': 1.0, 'accepted': True}
every output matches with the right type — full reward
EXAMPLE 2
Input submission={"outputs": {"t1": 0, "t2": 0}, "reported_reward": 1.0}, answer_key={"t1": 4, "t2": 9}
Output {'reward': 0.0, 'accepted': True}
the agent SAYS it scored 1.0 — recompute and ignore it. Outputs are wrong → 0.0
EXAMPLE 3
Input submission={"outputs": {"b1": True}, "reported_reward": 1.0}, answer_key={"b1": 1}
Output {'reward': 0.0, 'accepted': True}
True == 1 in Python — but type-strict, a bool is not an int, so it does NOT count
EXAMPLE 4
Input submission={"outputs": {"t1": 4}, "reported_reward": 1.0}, answer_key={"t1": 4, "t2": 9}
Output {'reward': 0.0, 'accepted': False}
partial coverage — answering only the easy tasks is rejected outright, not given partial credit
CONSTRAINTS
  • Recompute the reward from outputs vs answer_key. reported_reward is the agent’s self-report — never let it influence the result.
  • Coverage must be exact: the set of outputs keys must equal the set of answer_key keys. A missing task OR an extra task ⇒ accepted=False, reward=0.0.
  • Correct means a type-strict match: same value AND same type. In Python, type(got) is type(expected) before == (so True≠1, "1"≠1).
  • When accepted, reward = correct / len(answer_key). An empty answer key is a vacuous accept: {"reward": 0.0, "accepted": True}.
SOLVE IT YOURSELF

Your turn — write it

Edit the stub, hit Run (or ⌘/Ctrl + Enter), and watch the hidden tests. Stuck? the hints are right above and Reveal solution is one click away.

YOUR TASK

Implement verify(submission, answer_key) — recompute the reward from ground truth, reject anything but exact task coverage, and score only type-strict matches. Return {"reward": float, "accepted": bool}.

HINTS — 4 IDEAS
  1. Start with coverage: if set(submission["outputs"]) != set(answer_key), return {"reward": 0.0, "accepted": False} — before you score anything.
  2. Never read submission["reported_reward"] into your result. It exists only to tempt you.
  3. Type-strict: for each task, count it correct only when type(got) is type(expected) AND got == expected. This is what stops True from passing as 1.
  4. reward = correct / len(answer_key) once accepted. Guard the empty key first: an empty answer_key is a vacuous accept with reward 0.0.
CPython · WebAssembly
Approach, complexity & discussion — open after you solve

The approach

Trust nothing the submission tells you. Recompute the reward yourself from outputs vs answer_key, and gate acceptance on two structural checks before you even score: exact coverage (the output keys equal the answer-key keys — nothing missing, nothing extra) and a type-strict match per task. Score only what survives both. Every rule here is a variation on “verify, don’t trust.”

Complexity

Time O(n) in the number of tasks; space O(n) for the key sets. A single pass — the cost is discipline, not computation.

Common mistakes

  • Reading reported_reward into the result — it exists only to tempt you; recompute instead.
  • Comparing with == and no type check, so True passes as 1 (in Python 1 == True) and "1" nearly passes as 1.
  • Giving partial credit on partial coverage — a submission that answers only the easy tasks must score 0, not a fraction.
  • Forgetting the empty answer-key case (a vacuous accept with reward 0).

Where this shows up

This is the heart of RLVR (reinforcement learning from verifiable rewards): the verifier is the reward, so a gameable one is worse than none — every false positive is a lie the model learns to reproduce. The same “recompute from ground truth, demand exact coverage, compare strictly” discipline runs every autograder, CI test harness, and payments reconciler that scores untrusted input.

Explore the topic

See this challenge alongside everything else on the same subject — handbooks, system designs, algorithms and tools, in one place.

AI Agents & Tools