Vibe Engines Learn
CODING CHALLENGE · N°65

Write a Verifier

Medium AI EngineeringRLVRVerificationAgents

In RL from verifiable rewards, the verifier IS the reward. A gameable verifier is worse than no verifier — every false positive is a lie the model happily learns to reproduce. Write one that a reward-hacking agent cannot fool: recompute the score from ground truth, demand exact coverage, and compare with types intact.

The problem

Implement verify(submission, answer_key). answer_key maps each task_id to the correct answer. submission is what the agent hands back: {"outputs": {task_id: value, …}, "reported_reward": float}. Return {"reward": float, "accepted": bool} under three rules that close three reward hacks: (1) never trust reported_reward — recompute the reward yourself from outputs vs answer_key; (2) accepted is False (reward 0.0) unless outputs covers exactly the answer-key task ids — nothing missing, nothing extra; (3) a task counts correct only on a type-strict match (1 is not True, "1" is not 1). When accepted, reward is (# correct) / (# tasks).

EXAMPLE 1
Input submission={"outputs": {"t1": 4, "t2": 9}, "reported_reward": 1.0}, answer_key={"t1": 4, "t2": 9}
Output {'reward': 1.0, 'accepted': True}
every output matches with the right type — full reward
EXAMPLE 2
Input submission={"outputs": {"t1": 0, "t2": 0}, "reported_reward": 1.0}, answer_key={"t1": 4, "t2": 9}
Output {'reward': 0.0, 'accepted': True}
the agent SAYS it scored 1.0 — recompute and ignore it. Outputs are wrong → 0.0
EXAMPLE 3
Input submission={"outputs": {"b1": True}, "reported_reward": 1.0}, answer_key={"b1": 1}
Output {'reward': 0.0, 'accepted': True}
True == 1 in Python — but type-strict, a bool is not an int, so it does NOT count
EXAMPLE 4
Input submission={"outputs": {"t1": 4}, "reported_reward": 1.0}, answer_key={"t1": 4, "t2": 9}
Output {'reward': 0.0, 'accepted': False}
partial coverage — answering only the easy tasks is rejected outright, not given partial credit
CONSTRAINTS
  • Recompute the reward from outputs vs answer_key. reported_reward is the agent’s self-report — never let it influence the result.
  • Coverage must be exact: the set of outputs keys must equal the set of answer_key keys. A missing task OR an extra task ⇒ accepted=False, reward=0.0.
  • Correct means a type-strict match: same value AND same type. In Python, type(got) is type(expected) before == (so True≠1, "1"≠1).
  • When accepted, reward = correct / len(answer_key). An empty answer key is a vacuous accept: {"reward": 0.0, "accepted": True}.
SOLVE IT YOURSELF

Your turn — write it

Edit the stub, hit Run (or ⌘/Ctrl + Enter), and watch the hidden tests. Stuck? the hints are right above and Reveal solution is one click away.

YOUR TASK

Implement verify(submission, answer_key) — recompute the reward from ground truth, reject anything but exact task coverage, and score only type-strict matches. Return {"reward": float, "accepted": bool}.

HINTS — 4 IDEAS
  1. Start with coverage: if set(submission["outputs"]) != set(answer_key), return {"reward": 0.0, "accepted": False} — before you score anything.
  2. Never read submission["reported_reward"] into your result. It exists only to tempt you.
  3. Type-strict: for each task, count it correct only when type(got) is type(expected) AND got == expected. This is what stops True from passing as 1.
  4. reward = correct / len(answer_key) once accepted. Guard the empty key first: an empty answer_key is a vacuous accept with reward 0.0.
CPython · WebAssembly

Explore the topic

See this challenge alongside everything else on the same subject — handbooks, system designs, algorithms and tools, in one place.

AI Agents & Tools