Claude Code
CI eval gates for Claude Code changes
If Claude Code can edit prompts and tools, CI must be able to fail those edits. Here is how we gate agentic coding the same way we gate application code.
August 17, 2026/2 min read/Claude Certified Engineers
A CI eval gate is a job that scores the golden set and fails the pull request when quality, safety, or spend regresses. If Claude Code can edit system.md and merge on green unit tests alone, you have taught the agent that model quality is optional.
We wire the gate on Eval and Guardrail Harness work and on enablement so your engineers inherit the same loop.
What should trigger it
- Prompts and CLAUDE.md policy that affects runtime agents.
- Tool JSON Schema and MCP server behavior.
- Model id and caching breakpoints.
- Retrieval corpus selection.
Unit tests still run. They do not measure whether Claude called refund on the wrong id. That is the eval harness.
Local first
Put the exact command in CLAUDE.md. Claude Code should run a smoke subset before it opens a PR. Full suite on CI. If local takes twenty minutes, split a fast P0 pack from the rest.
Fail closed on P0
Safety cases flipping to fail always block. Score drops use the threshold in the brief (we usually start at 3 points on a 0-100 score). Token ceilings use the budget you already sold. See LLM-as-judge for how we keep the judge cheap enough that people leave the gate on.
Store traces with the failed case id. A red X with no prompt, tool calls, or judge rationale trains people to retry CI and hope. The runner should write JSON the same way unit tests write JUnit. Dashboards can wait.
What Claude Code will do without a gate
It will edit system.md to "be more helpful," merge, and you will find out in production that refund no longer requires a ticket. Unit tests will still be green. That is the whole point of the harness: the failures that matter are not type errors.
On a six-week build this gate is not a phase at the end. It is how week two's prototype stays honest through week five's tool changes. If you already have Claude Code in the repo and no gate, that is Eval and Guardrail Harness work, not a style comment on the next PR.
Questions
- Should Claude Code changes run the eval suite?
- Yes, whenever the change touches system prompts, tool schemas, retrieval, or model ids. Formatting-only diffs can skip. Policy diffs cannot.
- What does a failed gate look like?
- Non-zero exit, a JSON summary of failed case ids, and stored traces. A screenshot of a notebook is not a gate.
- Will this slow developers down?
- It slows bad merges. Local commands in CLAUDE.md keep the loop on the laptop. CI is the backstop, not the first time anyone ran the suite.