Eval-Gating: Don't Let an Agent Accept Its Own Output

An agent that checks its own work will always find that it did a good job.

Not because it’s lying. Because “is this good?” asked of the thing that just produced it is the same forward pass that produced it, with the same blind spots, now motivated to conclude yes. You get a confident paragraph explaining why the output meets the requirements, and the paragraph is not evidence.

The fix is boring and it works: a separate agent grades it, against a named rubric, and returns a boolean your code branches on. Not prose you read. A boolean.

The series

1. The fleet model — why twelve specialists beat one god-agent
2. Anatomy of one Eve agent — the five files, and building your first
3. Root lockdown and forced fan-out — deleting tools as an architecture
4. Eval-gating — a judge subagent and a typed verdict (you are here)
5. The orchestrator capstone — composition, plus scoring the whole fleet

Part three ended on a gap: locking down a root guarantees the work happens in a subagent, but says nothing about whether the work is any good. This is that gap.

Why “review your output” doesn’t work

The obvious first attempt is a line in the instructions:

After drafting, review your output against the requirements and revise if needed.

Try it. What you get is a paragraph beginning “Reviewing my draft against the requirements:” followed by a list of ways it satisfies them. Occasionally a cosmetic tweak. Almost never “this is wrong, starting over.”

Three reasons, and they’re structural, not fixable with better wording:

Same context, same blind spots. The mistake it made came from a misreading. Reviewing in the same context re-reads it the same way. It cannot see what it didn’t see.

Prose is unfalsifiable. “The draft is direct and covers the requested points” — is that a pass? Says who, on what scale? There’s no threshold, so there’s no gate, so nothing can fail.

Nothing branches on it. The review produces text, and text goes into the transcript. No if statement anywhere in your system reads it. The loop isn’t closed; it’s decorated.

So: separate context, defined scale, typed output.

The judge

Agent #8 in the fleet is Eval Grader, and its entire job is scoring other agents’ output. The design has three properties that all matter.

It has no tools

Every built-in is stripped:

// eval-grader/agent/tools/bash.ts (and glob, grep, read_file, web_fetch, web_search, write_file, agent)
import { disableTool } from "eve/tools";
export default disableTool();

This isn’t paranoia, it’s about determinism. A judge with web_search will go investigate rather than grade. It’ll look up whether the claim in the artifact is true, form an opinion from sources the artifact never had, and score against that instead of against what it was handed. Now your grade depends on what the judge happened to find that day, and the same artifact scores differently on Tuesday.

The grader’s instructions say it plainly:

You do not produce content, search, fetch, or run code — you judge what you are handed, using
only the text given to you in the message.

And because the tools are actually gone rather than merely discouraged, you can assert it:

t.usedNoTools();

One line in the eval. A grading turn should make zero tool calls, and the test proves the disableTool() sweep actually took effect.

It’s not a cheap model

Tempting optimisation: judging is easier than generating, so use the fast model. It’s wrong, and the comment in agent.ts says why:

// deepseek-v4-pro, not a cheap/fast model: grading is the judgment call itself, so the
// grader needs to be at least as capable as the agents it grades.

A weak judge fails in the worst possible direction — it passes things it doesn’t understand. You end up with a gate that’s green because the grader couldn’t tell. That’s strictly worse than no gate, because now you trust it.

Its rubrics are named and specific

Not “grade this.” Named rubrics, each with concrete criteria, each scoped to one agent’s output type. The real ones:

### `ghostwriter-voice-match`
Does the draft sound like Chris: direct, casual, low-fluff, builder-minded?
- **directness** — states the point first, no throat-clearing or hedging.
- **casualness** — reads like a person talking, not marketing copy or a press release.
- **low-fluff** — no filler words ("basically", "just", "really"), no restating the obvious.
- **builder-voice** — opinionated, concrete, technical where it matters; not vague enthusiasm.

### `research-desk-citations`
Does every claim in the report trace to a source the researcher actually returned?
- **traceability** — every factual claim maps to a specific source URL in the supplied source
  list. Any claim that doesn't is a hard problem, not a nitpick.
- **no-invention** — no citation that was not actually in the supplied source list.
- **source-adequacy** — the claims that matter are backed by more than one source where the
  topic is contested, or the report says the evidence is thin.
- **confidence-honesty** — the report's stated confidence matches how well-sourced it actually is.

Each criterion names a specific failure mode of a specific agent. no-invention exists because research agents invent URLs. low-fluff exists because drafting agents pad. You’re not grading quality in the abstract — you’re checking the four ways this particular agent is known to go wrong.

That’s the design rule: rubric criteria come from observed failures, not from first principles. Every criterion in the fleet was added after something went wrong in a real run.

The typed verdict

Here’s the piece that turns a review into a gate.

// eval-grader/agent/schema.ts
import { z } from "zod";

// Structured verdict shape. Passed as `outputSchema` on the client turn that asks for a
// grade — the eve runtime enforces this server-side before the turn settles, so #12
// Orchestrator (and the evals here) get a typed verdict, not prose to regex out.
export const gradeOutputSchema = z.object({
  rubric: z.enum(["ghostwriter-voice-match", "research-desk-citations", "triage-desk-reply-quality"]),
  criteria: z
    .array(
      z.object({
        name: z.string(),
        score: z.number().min(0).max(5),
        justification: z.string(),
      }),
    )
    .min(1),
  overallScore: z.number(),
  maxScore: z.number(),
  pass: z.boolean(),
  summary: z.string(),
});

export type GradeVerdict = z.infer<typeof gradeOutputSchema>;

pass: z.boolean(). The caller does if (verdict.pass). That’s the gate. It doesn’t read summary and form an impression.

The runtime enforces the schema before the turn settles, which kills a whole category of nonsense: no regex over prose, no “it wrapped the JSON in a code fence again,” no half-parsed object. By the time you hold the verdict it’s valid or the turn failed.

And justification: z.string() per criterion is not decoration. Requiring a written reason for each score changes the scores — a model that has to justify a 5 on no-invention has to actually look for invented citations. The grader instructions push on this: “A score with no concrete justification is not a real grade.”

The pass formula, and why an average isn’t enough

- `overallScore` is the sum of criterion scores; `maxScore` is `5 * number of criteria`.
- `pass` is true only if `overallScore / maxScore >= 0.7` AND no single criterion scored below 3.
  One badly failed criterion should fail the whole grade even if others are perfect — a citation
  grader that invents one source is not "80% passing."

Two conditions, and the second one is the one people leave out.

A ratio alone lets one catastrophic failure hide behind three good scores. Consider a research report scoring 5, 5, 5 on traceability, adequacy and confidence-honesty — and 0 on no-invention, meaning it fabricated a URL. Ratio: 15/20 = 0.75. Passes.

It fabricated a source. That’s not a 75% report, that’s a broken one.

So there’s a floor. Any criterion below 3 fails the whole grade regardless of the total. The average measures typical quality; the floor catches disqualifying failure. You need both, and which criteria are disqualifying is a judgement you make once, in the rubric, rather than re-litigating per artifact.

The other half of the formula’s integrity is anti-generosity, stated directly in the instructions:

- Score each criterion 0–5. 5 = fully meets the bar, 0 = clearly fails it. Use the full range —
  don't default to 3s.
- Grade what's in front of you. Don't reward effort, length, or confident tone on its own.
- Never soften a failing grade because the artifact is "close enough." #12 Orchestrator gates on
  `pass` programmatically — a generous grade here is a broken gate downstream.

Models default to 3s. Left alone they’ll score everything “adequate,” compress the range to nothing, and your gate passes everything. You have to explicitly instruct against it — and then check, because a rubric that never fails anything is not a gate, it’s a formality.

Which is exactly what the eval-grader’s own eval checks:

// eval-grader/evals/grading/voice-match.eval.ts
import { defineEval } from "eve/evals";
import { equals } from "eve/evals/expect";
import { gradeOutputSchema } from "../../agent/schema";

export default defineEval({
  description: "Eval Grader scores an obviously off-voice draft against ghostwriter-voice-match and fails it.",
  async test(t) {
    const turn = await t.send({
      message: [
        "Rubric: ghostwriter-voice-match",
        "",
        "Artifact to grade:",
        '"We are thrilled to announce a comprehensive, industry-leading suite of features ',
        "designed to seamlessly empower our valued users on their exciting journey. This ",
        'robust solution truly represents a paradigm shift in the space."',
      ].join("\n"),
      outputSchema: gradeOutputSchema,
    });

    t.succeeded();
    t.usedNoTools();
    turn.outputMatches(gradeOutputSchema);

    const verdict = turn.data;
    await t.require(verdict?.rubric, equals("ghostwriter-voice-match"));
    await t.require(verdict?.pass, equals(false));
  },
});

Feed it maximum corporate sludge and assert pass === false. It’s checking three things at once: the schema holds, the judge used no tools, and the grader will actually fail something. Test that your gate can say no. A gate you’ve only ever seen say yes is untested.

Closing the loop

A verdict nothing acts on is a review. The gate is what the caller does next.

The orchestrator’s instructions, verbatim:

4. **Grade.** Immediately after each `dispatch_specialist` result, call the `grader` subagent tool
   with `outputSchema` set to this agent's `gradeOutputSchema` (see `agent/schema.ts`), passing
   the matching rubric name for that role (`sandbox-coder-fix-quality`,
   `research-desk-answer-quality`, or `ghostwriter-draft-quality`), the artifact
   `dispatch_specialist` returned, and the original part for context. Read the typed verdict's
   `pass` field — never eyeball the prose summary as if it were the gate.

5. **Accept, retry, or park.**
   - `pass: true` → accept this part's output, move on.
   - `pass: false` → retry: call `dispatch_specialist` again for the same part, including the
     grader's full verdict (criteria, scores, summary) in the message so it can fix what failed.
     Retry at most twice per part (three attempts total).
   - Still failing after two retries, OR the role was unsupported in v1 → call `park_item` with
     the part, the specialist role, and why (unsupported role, or the last grader verdict's
     summary). Do not include a parked part's output in the final synthesis as if it succeeded.

Four things in there are doing real work.

“Never eyeball the prose summary as if it were the gate.” Stated because it’s the natural failure. A model with a verdict object in front of it will happily narrate its way to a conclusion. The instruction pins it to the field.

The verdict goes back into the retry. Not “try again” — here are the four criteria, here’s what each scored, here’s the justification, fix the ones that failed. The generator gets specific, actionable feedback from outside its own context. That’s the difference between a retry that improves and a retry that reshuffles. The specialist’s instructions close the loop from the other side:

If the message includes a prior grader verdict (criteria + scores + summary), read it before
redrafting. Fix specifically what scored low — don't just restate your first answer with a
softer tone.

“Don’t just restate your first answer with a softer tone” is there because that’s exactly what it did.

The retry budget is bounded and honestly labelled. Two retries, three attempts. And in the README, under Parked:

Retry budget tuning. Two retries (three attempts) per part is a placeholder, not tuned against real failure-rate data.

It’s a guess. Writing that down means the next person doesn’t mistake an arbitrary constant for a tuned one.

Failure has somewhere to go. This is the part I’d argue is most important and gets skipped most often. After three failures the part is parked — written to disk with the reason:

// orchestrator/agent/tools/park_item.ts
export default defineTool({
  description:
    "Durably record a decomposed goal-part that could not be completed this run — either the " +
    "specialist role is unsupported in v1, or it exhausted its grading retries. Call once per " +
    "parked part; do not silently drop it from the final synthesis.",
  inputSchema: z.object({
    part: z.string().describe("the decomposed piece of the goal that could not be completed"),
    specialistRole: z.string().describe("which specialist role this part was routed to, or 'unsupported'"),
    reason: z.string().describe("why it was parked — unsupported role, or the last grader verdict summary"),
  }),
  async execute({ part, specialistRole, reason }) {
    const state = await loadParked();
    state.items.push({ ts: Date.now(), part, specialistRole, reason });
    await saveParked(state);
    return { ok: true, parkedCount: state.items.length };
  },
});

Without this there are only two outcomes: accept bad output, or silently drop it. Both are worse than a third state that says this didn’t work, here’s what it was, here’s why. And it’s on disk, so it survives the process — which is what makes it a coordination primitive rather than a log line.

Cost, and when it’s worth it

Eval-gating roughly doubles your model calls in the happy path and triples them when something retries. A three-part goal that all passes first try: three generate calls, three grade calls, plus the root. One part failing twice adds four more. That’s a real bill on a real key, which is why there’s a Cost Sentinel in the fleet.

Worth it when:

• Output goes somewhere you can’t take it back from — published, posted, merged, sent.
• Errors are expensive or hard to spot — a fabricated citation looks exactly like a real one.
• The chain is long — bad output at step 2 corrupts everything downstream, so you want it caught at step 2.
• It runs unattended — the whole point is nobody’s reading each output as it happens.

Not worth it when a human reviews everything anyway, when the task is cheap to redo, or when you can check it deterministically. If a script can verify it, use the script. Cost Sentinel does exactly this — the anomaly math is a plain cron script with no LLM in it, and shared with the agent’s tools so the deterministic verdict never drifts from what the agent explains. An LLM judge is what you reach for when correctness is a judgement call, not when it’s arithmetic.

Steal this

• A judge is a separate agent with no tools. Zero. It grades what it’s handed.
• Not a cheap model. A weak judge passes what it fails to understand.
• Rubric criteria come from observed failures, one per way this agent is known to break.
• Type the verdict and branch on a boolean. if (verdict.pass). Never on prose.
• Threshold plus a floor. An average hides the one disqualifying failure.
• Feed the verdict into the retry. Specific failed criteria, not “try again.”
• Bound the retries and park what never passes. Silently dropped is not an outcome.
• Write an eval that proves the gate can say no.

Next, the last part: the orchestrator that composes all of it — subagents, this eval-gate, durable state in one agent, plus the scorecard that scores the whole fleet and the honest red row in it.