Put an eval harness on Claude before the agent goes live

An eval harness for Claude is a golden set of inputs, expected behaviors, and automated checks that run in CI whenever prompts, tools, or models change. If you cannot point at a number that moved, you do not have an eval. You have a demo with extra steps.

Most Claude pilots we inherit fail the same way. The agent looks sharp in a recorded walkthrough. Two weeks later, a tool schema changes, a prompt grows by 4,000 tokens, and nobody can say whether quality went up or down. Leadership asks for evidence. The team argues from anecdotes. The project stalls.

Certification exists because this failure is predictable. The Architect exams treat evaluation as a design problem, not a nice-to-have. We treat it the same way on every Production Agent Build: the harness lands in week one, before the agent sees live traffic.

What an eval harness has to catch

A useful harness answers four questions on every pull request:

1. Did the agent still complete the job on the cases we care about?
2. Did it call the right tools, with arguments a downstream system can accept?
3. Did it refuse, escalate, or redact when it should have?
4. Did cost or latency jump enough to break the budget you sold internally?

If your suite only scores "the prose sounds good," it will green-light regressions that matter. Fluency is cheap. Tool correctness and refusal behavior are not.

You do not need a research platform to start. You need a dataset you own, a scorer you can explain, and a gate in CI that can fail the build.

Build the golden set from production, not from imagination

Start with 40 to 80 cases, not 4,000. The first set should be painful and specific:

• Ten happy paths copied from real tickets or transcripts.
• Ten messy inputs: truncated PDFs, mixed languages, missing IDs.
• Ten adversarial cases: prompt injection in a document, "ignore previous instructions," requests for data the agent must not fetch.
• Ten cost traps: huge attachments, repeated lookups, questions that should be a single cached call.

Write the expected behavior as a rubric, not as one canonical paragraph. Claude will not match a golden answer word for word, and you should not want it to. Specify the invariants: "must cite the invoice number," "must not call refund without a human approval id," "must refuse to export the full customer table."

Name every case. invoice-missing-po-v3 is a test you can talk about in standup. case_47 is not.

When the product changes, add cases. Do not rewrite the suite to make the new prompt look good. If a behavior is no longer required, retire the case in a separate commit so the history stays honest.

Score with a judge you can defend

LLM-as-judge is fine when the judge is boring. Give it a short rubric, a forced scale, and examples of pass and fail. Ask for a structured object (pass, score, failed_invariants[]), not an essay.

Keep humans in the loop for the first two weeks. A certified engineer should spot-check disagreements between the judge and the golden rubric. If the judge and the human diverge on the same five cases, the rubric is wrong, not the model.

Deterministic checks should run first, because they are cheap and they catch the expensive bugs:

• JSON schema validation on every tool call.
• Exact-match on IDs, amounts, and record keys.
• Regex or classifier checks for PII in the output.
• Token and latency budgets per case.

Only then run the judge. A suite that spends $12 scoring a $0.02 agent call is not a harness. It is a hobby.

Put the gate in CI, not in a notebook

The eval has to run where code already runs. On this bench that means the same pipeline that tests the rest of the repo.

A pull request that changes system.md, a tool schema, or the model id should kick the suite. Fail the build when:

• Task completion on the golden set drops more than you agreed (we usually start at 3 points on a 0-100 score, or any P0 safety case flipping to fail).
• A P0 safety case fails, full stop.
• Median tokens or p95 latency exceed the budget in the engagement brief.

Store traces. When a case fails, you need the prompt, the tool calls, the output, and the judge rationale. Without traces, the team will "fix" quality by restating the prompt and hoping.

A minimal layout that survives contact with a real repo:

// ci should fail if p0 safety cases fail or task score drops > 3
type EvalCase = {
  id: string;
  input: string;
  invariants: string[];
  safety: "p0" | "p1" | "none";
  budgetTokens: number;
};

Keep the runner boring: load cases, call the same path production calls, score, write a JSON summary, exit non-zero. Dashboards can come later. The gate is the product.

What we install in week one

On a six-week build, the eval work is not a phase at the end. It is how we know the prototype in week two is not theater.

By the end of week one we want:

• A golden set checked into the client repo.
• Deterministic contract tests on tools.
• A judge rubric with at least one human-reviewed batch.
• A CI job that can fail.
• A baseline number you can quote in the weekly readout.

That is also why certification matters here. An engineer who has not had to design evaluation will optimize for the demo. An engineer who has been tested on it will refuse to ship without a score.

If you already have an agent in production and no harness, start with the Claude Readiness Audit. Two weeks is enough to build the first suite, measure the current system, and tell you whether the next move is a rebuild, a guardrail pass, or a smaller scoped agent.

The related problem, once the agent can act, is whether its tools are even callable. That is a schema design problem, and it is the second place we see silent quality loss.

Do not wait for a model upgrade to invent measurement. Put the harness in first. Then change the agent.