LLM-as-Judge Rubric Builder

Once a system produces open-ended text there is no exact string to test against, so teams grade it with another model and a rubric. This builder writes that judge prompt for you from a task description, a list of criteria, a scale and a mode, and keeps the parts that make judges more reliable: named criteria, a fixed output schema and reasoning before verdicts.

What it computes

There is no arithmetic here; the output is a structured prompt assembled in a fixed order:

1. A role line: an impartial evaluator who will score one response or compare two.
2. # Task: your description of what is being evaluated.
3. # Evaluation criteria: a numbered list of each criterion’s name and description.
4. # Inputs: placeholders, {{INPUT}} plus {{RESPONSE}}, or {{RESPONSE_A}} and {{RESPONSE_B}} in compare mode, each wrapped in a named tag.
5. # Instructions: the scale, a rule to judge substance rather than length, a rule that response order must not matter in compare mode, and the rationale rule if enabled.
6. # Output (JSON): in score mode, one key per criterion, its name lowercased with spaces turned into underscores, each holding a rationale and a score; in compare mode, a rationale and a winner of A, B or tie.

The scale options are Pass/Fail (written as 0 or 1), 1–3, 1–5 (the default, labeled Likert) and 1–10. Your harness replaces the placeholders with real data before each call.

A worked example

The defaults evaluate a grounded question-answering system: the task is “Answer a user question using only the provided context.”, the criteria are Faithfulness, Relevance and Completeness, the mode is Score one, the scale is 1–5 and the rationale is required. The generated prompt ends like this:

# Inputs
<user_input>{{INPUT}}</user_input>
<response>{{RESPONSE}}</response>

# Instructions
- Score the response on each criterion using the scale: 1–5 (Likert).
- Judge on substance, not length or verbosity.
- For each criterion, give a one-line rationale before its score.

# Output (JSON)
{
  "faithfulness": { "rationale": "<one line>", "score": <number> },
  "relevance": { "rationale": "<one line>", "score": <number> },
  "completeness": { "rationale": "<one line>", "score": <number> }
}

Switch to Compare two and the scale disappears. The inputs become a response A and a response B, the instructions add that order is random and must not influence the decision, and the output becomes a single object with a rationale of two or three sentences and a winner.

Assumptions and limits

• No context slot. The default task mentions provided context, but the inputs hold only the user input and the response. For faithfulness checks, add the retrieved context to {{INPUT}} or extend the prompt with its own tag.
• Scale points are not defined. The prompt names the range but does not describe what a 3 means. Add one-line anchors per score in the criterion descriptions for steadier grading.
• Keys come straight from names. Only spaces are replaced, so punctuation survives in JSON keys and two criteria with the same name collide. Use short, unique names.
• Instructions are not controls. Telling the judge that order is random does not randomize it; your harness must swap positions, ideally judging each pair both ways.
• Unvalidated by default. The tool cannot tell you whether a judge agrees with humans on your data.

How to use the result

• Calibrate first. Label 50 to 100 examples by hand, run the judge on them and measure agreement before relying on its scores.
• Parse strictly. Reject outputs that are not valid JSON or miss a key, and retry, rather than guessing a score.
• Report per criterion. A single averaged score hides which dimension regressed.
• Freeze the judge. Pin the model version and prompt, because changing either changes every score you compare against.

When to use it, and when not

Use a judge where the output is open-ended and the volume is too high to read: grading hundreds of answers after a prompt change, comparing two models on the same inputs, or tracking one system week by week. It earns its keep once it has been checked against human labels on your own data.

Do not use one where an exact check exists. Valid JSON, a matching label, a numeric answer or code that passes its tests are cheaper, faster and deterministic to verify in code. Do not let a judge make final calls where a wrong verdict is costly; use it to sort the queue a human then reviews. And where you can, avoid judging a model with itself.

Trade-offs and limits

The deeper limit is that a judge measures agreement with its own reading of your rubric. If the rubric misses what users care about, a perfectly consistent judge reports the wrong thing with confidence.

Common mistakes

• Reporting raw agreement. On a set where humans pass 80% of answers, a judge that passes everything agrees 80% of the time and knows nothing. Report agreement beyond chance, such as Cohen’s kappa.
• Calibrating on easy cases. A set of clear passes and clear failures flatters any judge. Include the borderline answers where humans themselves hesitate.
• Dropping failed parses. Outputs that are not valid JSON are often the long or odd responses. Removing them from the denominator quietly raises the pass rate; count them separately.
• Overlapping criteria. Relevance and Completeness both reward covering the question, so one miss is penalized twice. Each criterion should fail for a reason no other criterion catches.
• Optimizing against the judge alone. Tune prompts to raise the judge’s score and you will also find its biases, such as a taste for longer answers. Keep a human spot check on every release.

Check yourself