Write a Verifier

Start from brute force

The obvious verifier loops over what the agent submitted: count the outputs that equal the key with ==, and divide by the number of outputs. It works on honest, complete submissions, and it costs O(n) time and O(1) extra space, so speed is not the problem. It is gameable three ways. A submission of only {"t1": 4} against a two-task key scores 1.0, because the denominator shrinks with the answers. {"b1": True} against {"b1": 1} also scores 1.0, because True == 1 in Python. And any code path that reads reported_reward lets the agent write its own grade.

The approach

The reward must be a function of the answer key and the outputs alone, with the checks ordered so that every hack is closed before scoring starts. The reference reads outputs = submission.get("outputs", {}) and never touches reported_reward. First, coverage: set(outputs.keys()) != set(answer_key.keys()) returns {"reward": 0.0, "accepted": False}, so a missing task and an extra task both fail before anything is scored. Second, an empty key returns the vacuous accept, which also keeps the division safe. Third, the loop runs over answer_key.items(), not over the outputs, and counts a task only when type(got) is type(expected) and got == expected; the type test is what stops True, whose type is bool, from passing as 1. The reward is correct / len(answer_key), a denominator the agent cannot change. In TypeScript, === never coerces, so typeof got === typeof exp && got === exp gives the same strictness.

Walk through an example

The first example, two correct ints against a two-task key, with the self-reported 1.0 ignored throughout.

Complexity

Reference: O(n) time for n tasks: building and comparing the two key sets is O(n), and scoring does one type check and one == per task (longer for big string or list answers). Space is O(n) for the two sets. Naive loop: also O(n) time with O(1) extra space; it is not slower, only gameable.

Edge cases to test

• An extra task smuggled in: outputs {"t1": 4, "t2": 9, "t3": 1} against {"t1": 4, "t2": 9} gives {"reward": 0.0, "accepted": False}. A verifier that only checks that every key task is present accepts it.
• A string for an int: {"s1": "1"} against {"s1": 1} is accepted, since coverage is exact, and scores 0.0.
• A float for an int: {"t1": 4.0} against {"t1": 4} counts as wrong in the Python reference, because float is not int, but as correct in the TypeScript reference, where both are number. Decide which one your verifier means and test it.
• An empty answer key with empty outputs: {"reward": 0.0, "accepted": True}, the vacuous accept. Dividing first raises ZeroDivisionError in Python and yields NaN in TypeScript.
• No "outputs" field at all: submission.get("outputs", {}) treats it as empty, so it is rejected unless the answer key is empty too.

Common mistakes

• Reading reported_reward into the result — it exists only to tempt you; recompute instead.
• Comparing with == and no type check: in Python True == 1, so a bool passes as an int, and in JavaScript loose == also lets "1" pass as 1.
• Using isinstance(got, type(expected)) as the strict check: isinstance(True, int) is true, because bool subclasses int, so the bool gets through again.
• Giving partial credit on partial coverage — a submission that answers only the easy tasks must score 0, not a fraction.
• Dividing by len(outputs) instead of len(answer_key): the agent controls that denominator, so answering one easy task scores 1.0.
• Forgetting the empty answer key: correct / len(answer_key) raises ZeroDivisionError in Python and gives NaN in TypeScript, where the spec wants a vacuous accept with reward 0.

Follow-ups an interviewer asks next

Answers can be lists or nested dicts.

A top-level type check is no longer enough: expected [1, 2] and output [True, 2] are both lists and compare equal in Python, so the reference counts it correct. Compare recursively instead: the same type at every level, the same length or the same keys, then recurse into each element, and compare leaves with == only after their types match.

Some answers are free-form text or rounded numbers.

Strict equality turns formatting into failure: " 42" and 42.0 both miss 42. Canonicalize both sides with one fixed function, by stripping whitespace, parsing numbers or folding case, and compare floats with a tolerance such as math.isclose. Every relaxation is new surface for reward hacking, so the verifier chooses the comparison and the submission never does.

The verifier runs the agent’s code instead of comparing values.

Then the submission can attack the verifier itself. Run the code in a sandbox with no network and with time and memory limits, keep the answer key and the expected results outside it, and compare in a separate process the code cannot reach. If the code can write to whatever the score is read from, it can simply report success.

Could partial coverage earn partial credit safely?

Yes, if the denominator stays fixed: divide by len(answer_key) and count every missing task as wrong, so {"t1": 4} against a two-task key scores 0.5 instead of 1.0. Skipping hard tasks then never helps. This challenge rejects instead, which also catches extra task ids, a sign of a malformed or adversarial submission.

Where this shows up

This is the heart of RLVR (reinforcement learning from verifiable rewards): the verifier is the reward, so a gameable one is worse than none — every false positive is a lie the model learns to reproduce. The same “recompute from ground truth, demand exact coverage, compare strictly” discipline runs every autograder, CI test harness, and payments reconciler that scores untrusted input.

Reference solution

def verify(submission, answer_key):
    outputs = submission.get("outputs", {})
    if set(outputs.keys()) != set(answer_key.keys()):
        return {"reward": 0.0, "accepted": False}
    if not answer_key:
        return {"reward": 0.0, "accepted": True}
    correct = 0
    for task_id, expected in answer_key.items():
        got = outputs[task_id]
        if type(got) is type(expected) and got == expected:
            correct += 1
    return {"reward": correct / len(answer_key), "accepted": True}


AK = {"t1": 4, "t2": 9, "t3": 16, "t4": 25}
print(verify({"outputs": {"t1": 4, "t2": 9, "t3": 16, "t4": 25}, "reported_reward": 1.0}, AK))
# -> {'reward': 1.0, 'accepted': True}
print(verify({"outputs": {"t1": 0, "t2": 0, "t3": 0, "t4": 0}, "reported_reward": 1.0}, AK))
# -> {'reward': 0.0, 'accepted': True}  (self-reported 1.0 ignored)

type Val = number | string | boolean;
type Submission = { outputs: Record<string, Val>; reportedReward: number };
type Verdict = { reward: number; accepted: boolean };

function verify(submission: Submission, answerKey: Record<string, Val>): Verdict {
  const outKeys = Object.keys(submission.outputs);
  const keys = Object.keys(answerKey);
  const sameCoverage = outKeys.length === keys.length && keys.every((k) => k in submission.outputs);
  if (!sameCoverage) return { reward: 0, accepted: false };
  if (keys.length === 0) return { reward: 0, accepted: true };
  let correct = 0;
  for (const k of keys) {
    const got = submission.outputs[k];
    const exp = answerKey[k];
    if (typeof got === typeof exp && got === exp) correct++;
  }
  return { reward: correct / keys.length, accepted: true };
}

const AK = { t1: 4, t2: 9, t3: 16, t4: 25 };
console.log(verify({ outputs: { t1: 4, t2: 9, t3: 16, t4: 25 }, reportedReward: 1 }, AK));
// -> { reward: 1, accepted: true }
console.log(verify({ outputs: { t1: 0, t2: 0, t3: 0, t4: 0 }, reportedReward: 1 }, AK));
// -> { reward: 0, accepted: true }  (self-reported 1 ignored)