Production agents
LLM-as-judge rubrics you can defend in CI
LLM-as-judge is fine when the rubric is boring, structured, and checked by a human. Here is how we score Claude agents without turning CI into a $12 hobby.
June 22, 2026/2 min read/Claude Certified Engineers
LLM-as-judge is a second model scoring the first against a written rubric. It is useful when the right answer is not a string match. It is expensive and sloppy when you ask it to decide whether the prose "sounds good."
On this bench the judge is a component of the eval harness, not a replacement for one. Deterministic checks run first. The judge only sees cases that still need a human-shaped opinion.
Make the rubric boring
Give the judge:
- The case id and the invariants from the golden set.
- Two or three pass and fail examples.
- A forced schema:
pass,score(0-100),failed_invariants[],rationalecapped at two sentences.
Do not let it invent extra criteria. If "tone" is not in the engagement brief, it is not in the rubric.
Humans for two weeks, then a sample
A certified engineer should spot-check every disagreement in the first two weeks. If the judge and the human diverge on the same five cases, the rubric is wrong. Fix the rubric. Do not "tune" the prompt until the numbers look pretty.
After that, sample. Ten percent of diffs is enough to catch drift. Store the judge rationale next to the trace so a later reviewer can see why CI went red.
Do not spend $12 scoring a $0.02 call
Token budgets belong on the judge too. If scoring costs more than serving, you will stop running the suite. That is how notebooks come back.
We wire this into CI on Eval and Guardrail Harness work so model quality is a number leadership can read, not an argument in a meeting.
Questions
- Is LLM-as-judge reliable enough for CI?
- Yes, if the rubric is short, the output is structured, and a human reviews disagreements for the first batches. A judge asked to "rate helpfulness 1-10" will green-light regressions that matter.
- What should the judge return?
- A forced object with pass, a numeric score, and the invariant IDs that failed. Not a paragraph of vibes.
- When should we not use a judge?
- IDs, amounts, JSON schema, and PII leaks should be deterministic. The judge scores the rest.