The Orchestrator Capstone: Composing Twelve Lessons Into One Agent

The last agent in the fleet introduces exactly zero new ideas.

That’s the point. Agent #12 is forced fan-out from #4, root lockdown from #5, and the eval-gate from #8, wired together with a JSON file that remembers what failed. If the first eleven were the lessons, this one is the exam — and the interesting question isn’t “can you build a big agent,” it’s whether small pieces you understand actually compose into something you still understand.

Mostly yes. With a couple of honest holes I’ll show you rather than paper over.

The series

1. The fleet model — why twelve specialists beat one god-agent
2. Anatomy of one Eve agent — the five files, and building your first
3. Root lockdown and forced fan-out — deleting tools as an architecture
4. Eval-gating — a judge subagent and a typed verdict
5. The orchestrator capstone + fleet scorecard (you are here)

The whole agent, in one paragraph

You give it a goal. It breaks the goal into independent parts, decides which specialist each part needs, dispatches each one to a subagent, grades every result against a rubric before accepting it, retries what fails with the grader’s verdict attached, parks whatever never passes, and writes you one report at the end that distinguishes what worked from what didn’t.

Six steps. Here’s the shape on disk:

orchestrator/agent/
  agent.ts                  deepseek-v4-pro via .chat()
  instructions.md           decompose → route → dispatch → grade → accept/retry/park → synthesise
  schema.ts                 gradeOutputSchema — the typed verdict it branches on
  lib/state.mjs             load/save state/parked.json
  tools/
    park_item.ts            internal state write, deliberately NOT gated
    list_parked.ts          read-only
    bash · glob · grep · read_file · write_file · web_fetch · agent    all disableTool()
  subagents/
    dispatch_specialist/    executes one part, playing one specialist role
    grader/                 scores one output, returns a typed verdict, zero tool calls
state/parked.json           what never passed, surviving the process

Two real tools. Seven deletions. Two subagents. One state file.

Composition move 1: the root can’t do anything

Straight from part three, applied hardest here. The most capable agent in the fleet is the one with the smallest toolbox — and that’s not ironic, it’s causal.

You have no bash, web_fetch, read_file, write_file, glob, grep, or generic self-clone `agent`
tool — that's intentional (root lockdown, same lesson as #5 Triage Desk and #4 Research Desk
v2). If you find yourself wanting one of those, that's a signal the work belongs in a subagent
call, not a sign something is missing.

The agent.ts deletion is the one that makes the rest hold:

// orchestrator/agent/tools/agent.ts
import { disableTool } from "eve/tools";

// Strip the generic self-clone "agent" tool. Root delegates ONLY through its two named
// subagent tools (dispatch_specialist, grader) — never a freeform clone of itself.
export default disableTool();

Without that, “the root can’t run bash” is false — it can spawn a clone that can. With it, the delegation graph is fixed at build time: two declared targets, each with its own directory and tool policy. The root composes within that graph; it can’t extend it at runtime.

That’s the property that makes an orchestrator auditable. You can read the subagents/ directory and know every possible thing this agent can cause to happen.

Composition move 2: parts go out in parallel, verdicts come back typed

3. **Dispatch.** Call the `dispatch_specialist` subagent tool once per part: give it the part's
   full text and the `specialistRole` you chose. Independent parts can be dispatched in parallel
   (multiple `dispatch_specialist` calls in one response); don't parallelize a retry against its
   own prior attempt.

“Multiple calls in one response” is the load-bearing phrase — without it models serialise. “Don’t parallelize a retry against its own prior attempt” is the correctness clause: a retry depends on a verdict that doesn’t exist yet, so it can’t go in the same wave.

Then every result is immediately graded, and the grade is a boolean the root branches on — the full mechanic is in part four. The one orchestrator-specific detail is that its rubrics are scoped to the three roles its dispatcher actually plays:

// orchestrator/agent/schema.ts
export const gradeOutputSchema = z.object({
  rubric: z.enum(["sandbox-coder-fix-quality", "research-desk-answer-quality", "ghostwriter-draft-quality"]),
  criteria: z.array(z.object({
    name: z.string(),
    score: z.number().min(0).max(5),
    justification: z.string(),
  })).min(1),
  overallScore: z.number(),
  maxScore: z.number(),
  pass: z.boolean(),
  summary: z.string(),
});

Same verdict shape as agent #8, deliberately. Same 0–5 criteria, same >= 0.7 AND no-criterion-below-3 formula. Sharing the verdict shape across agents is the cheapest interop you’ll ever get — anything that can read one grader’s output can read all of them.

One nice touch in the grader’s rubrics, sandbox-coder-fix-quality:

- **honesty** — doesn't claim to have run or tested anything it didn't (v1 has no real sandbox).

A criterion that exists purely to catch the system’s own limitation. The dispatcher has no sandbox, so overclaiming is its most likely failure, so it’s a graded criterion. That’s rubric design working the way it should.

Composition move 3: something on disk that outlives the run

The genuinely new bit — the only thing here that isn’t inherited from an earlier agent.

Every prior agent is stateless per run. The orchestrator can’t be, because it has a failure mode that needs to survive: a part that never passed grading. Accepting it is a lie. Dropping it is worse — the report looks complete and isn’t.

So there’s a third outcome, and it’s a file:

// orchestrator/agent/lib/state.mjs
import { fileURLToPath } from "node:url";
import path from "node:path";
import { readFile, writeFile, mkdir } from "node:fs/promises";

export const PARKED_PATH = path.join(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "state", "parked.json");

export async function loadParked(filePath = PARKED_PATH) {
  try {
    return JSON.parse(await readFile(filePath, "utf8"));
  } catch {
    return { items: [] };
  }
}

export async function saveParked(state, filePath = PARKED_PATH) {
  await mkdir(path.dirname(filePath), { recursive: true });
  await writeFile(filePath, JSON.stringify(state, null, 2));
}

Thirteen lines. No database, no queue, no Durable Object. A JSON file with a missing-file fallback and a mkdir -p.

I want to defend that, because the reflex is to reach for infrastructure. The requirements here are: one writer, append-mostly, tens of items, human-readable, survives a restart, and greppable when I want to know what’s been failing. A file does all of that. Adding Postgres would add an operational dependency to buy properties I don’t need. Use the smallest thing that survives the process.

The read side is what makes it a coordination primitive rather than a log:

// orchestrator/agent/tools/list_parked.ts
export default defineTool({
  description: "Read the durable list of parked (uncompleted) goal-parts from prior and current runs.",
  inputSchema: z.object({}),
  async execute() {
    return await loadParked();
  },
});

And the instruction that uses it:

If anything is parked, say so plainly and call `list_parked` first to confirm what's
accumulated, don't just recall it from this turn's memory.

Don’t recall it from this turn’s memory. The agent must re-read the file rather than trust its own recollection of what it wrote earlier in the same run. Disk is ground truth; context is a cache that lies. Also note list_parked returns items from prior runs too — so if the same part keeps failing across sessions, that’s visible instead of forgotten.

Composition move 4: the synthesis has to stay honest

6. **Synthesise.** Once every part is accepted or parked, give Chris one report: the original
   goal, then one block per part — what it needed, which specialist handled it, the accepted
   output (or "parked: <reason>" if it never passed), and the grader's summary for accepted parts.

And the rules:

- Never accept a `dispatch_specialist` output without a passing `grader` verdict first. The gate
  is the entire point of this agent — skipping it "because the output looks fine" defeats it.
- Never fabricate a grader verdict, a specialist output, or a parked reason. Every claim in the
  final synthesis traces to an actual subagent result you got back this run.
- If Chris's goal is too vague to decompose (no concrete parts, no clear specialist need), ask
  before dispatching anything — don't guess at parts that don't exist.

That first rule names the exact rationalisation — “because the output looks fine” — because that’s the sentence a model writes right before skipping the gate. Naming the specific excuse beats a general instruction to be rigorous.

The third one has its own eval:

// orchestrator/evals/smoke/routes-goal.eval.ts
export default defineEval({
  description: "Fleet Orchestrator asks for a concrete goal before dispatching.",
  async test(t) {
    await t.send("Please orchestrate this for me.");
    t.succeeded();
    t.messageIncludes(/goal|what|clarif/i);
  },
});

A vague ask should produce a question, not a confident decomposition of a goal that was never stated. The most valuable smoke tests in this fleet all check that the agent declines to guess.

What this doesn’t do

Time to be straight, because a capstone post is exactly where people oversell.

It does not call the other eleven agents. dispatch_specialist is one subagent parameterised by a specialistRole string, playing a simplified stand-in for three of them. It is not invoking eleven separate deployments. The README says so under a heading I wrote deliberately:

Live vs stubbed specialists (be honest about this)

This agent does not cross-invoke the 11 other separate Eve deployments — that’s parked (see below), not built. dispatch_specialist is a single subagent, parameterized by specialistRole, that plays the part of three of them in a simplified, self-contained way.

Why parked and not just unfinished: real cross-deployment invocation needs each agent’s endpoint and auth wired in, per the kind: "remote" path in Eve’s subagent registry. That’s an integration task, distinct from the composition this build was proving. So the build proves the shape of the call and not the call.

Seven of the twelve roles aren’t wired, and the instructions handle that honestly rather than pretending:

Any other need (triage, site-warden, vault-oracle, cost-sentinel, project-manager, godot-qa,
release-radar) is **not wired in v1** — route it to `dispatch_specialist` anyway with that role
name so you get an honest best-effort-labeled answer, but treat its grade more skeptically and
be ready to park it (see step 5).

Best-effort, labelled as best-effort, parked rather than accepted. The dispatcher is told the same from its side: “don’t refuse outright, but don’t pretend it’s the real specialist’s output either.”

No approval-gated tool. Unlike Triage Desk or Cost Sentinel, nothing here posts, pushes, or spends. The README flags that the gate becomes mandatory the moment real cross-deployment calls land — “not before,” because adding a gate to a tool that can’t do anything is theatre.

It has never been run end to end. Builds clean, eve build exits 0. A live run against a real multi-part goal, all the way through dispatch → grade → retry → park → synthesise, is listed under “Last-mile (not exercised during build).”

Which brings us to the scorecard, where that shows up as a number.

The fleet scorecard

Twelve agents with twelve eval suites is twelve things to check. So there’s one script at the root:

./scorecard.sh                  # score every agent that has an evals/ suite
./scorecard.sh site-warden      # score one (or several) named agents
./scorecard.sh --diff old.json  # compare a prior scorecard against a fresh run

It discovers every sibling directory containing a *.eval.ts, runs npx eve eval --json in each, normalises the output into one row per agent, prints a table, and writes scorecard.json.

Real output, not a mockup:

AGENT             PASS  FAIL  SKIP  ERR  EXIT  STATUS
----------------------------------------------------------
cost-sentinel     2     0     0     0    0     ✔ green
eval-grader       2     0     0     0    0     ✔ green
ghostwriter       1     0     0     0    0     ✔ green
godot-qa          2     0     0     0    0     ✔ green
orchestrator      0     0     0     0    1     ✖ no parseable --json output
project-manager   2     0     0     0    0     ✔ green
release-radar     0     1     0     0    1     ✖ RED
research-desk     1     0     0     0    0     ✔ green
sandbox-coder     1     0     0     0    0     ✔ green
site-warden       1     0     0     0    0     ✔ green
triage-desk       2     0     0     0    0     ✔ green
vault-oracle      2     0     0     0    0     ✔ green
----------------------------------------------------------
TOTAL             16    1

RED: orchestrator, release-radar

Sixteen green, one failing, one that didn’t emit parseable output at all — and that one is the capstone this whole post is about. Release Radar failed on MODEL_CALL_FAILED, an infrastructure problem rather than a logic one. Doesn’t matter. Red is red.

I’m publishing the real numbers because a scorecard you only show when it’s green isn’t a scorecard, it’s a screenshot. The number is worth having because it can be bad, and the capstone being one of the two bad rows is the most useful thing on the table.

The design decision I actually care about

There’s one comment at the top of the aggregator that matters more than the rest of the file:

// SWAP SEAM: this runner depends on exactly one contract — the top-level shape of
// `eve eval --json` ({passed, failed, scored, skipped, errored} + results[].verdict). If Eve
// is ever swapped out, write a new adapter that emits the same scorecard row and nothing
// downstream changes. No Eve internals are reached into here.

Everything framework-specific is contained in one function:

function runAgent(name) {
  const dir = join(ROOT, name);
  let raw = "", exitCode = 0;
  try {
    raw = execFileSync("npx", ["eve", "eval", "--json", "--skip-report"], {
      cwd: dir, encoding: "utf8", maxBuffer: 64 * 1024 * 1024,
      stdio: ["ignore", "pipe", "ignore"],
    });
  } catch (err) {
    exitCode = err.status ?? 1;
    raw = err.stdout?.toString?.() ?? "";
  }

  let j = null;
  try { j = JSON.parse(raw.slice(raw.indexOf("{"))); } catch { /* left null */ }
  if (!j) {
    return { agent: name, ok: false, error: "no parseable --json output", exitCode,
      passed: 0, failed: 0, /* ... */ gates: [] };
  }
  // ... normalise into the one row shape the rest of the tooling knows
}

Note it captures err.stdout on failure — a non-zero exit still usually carries the JSON, and throwing that away would turn every red agent into a blank row. And when parsing genuinely fails it returns a row saying so, rather than crashing the run. That’s why the orchestrator shows as no parseable --json output instead of taking the whole scorecard down with it.

The seam is worth the small discipline it costs. Eve is beta. It might not be what I’m running in a year. When that happens I write one adapter function that emits the same row shape and the table, the diff, the JSON artifact and anything built on it keep working. Depend on a contract you defined, not on the tool’s internals.

The diff is the actual product

The table tells you the state. The diff tells you whether you helped:

const dPass = r.passed - p.passed, dFail = r.failed - p.failed;
const wentRed = p.ok && !r.ok;
if (dPass === 0 && dFail === 0 && !wentRed) { console.log(`  = ${r.agent}: unchanged`); continue; }
const mark = wentRed || dFail > 0 || dPass < 0 ? "⚠ REGRESSED" : "✓ improved";

=== DIFF vs scorecard.json (2026-07-21T10:28:28.960Z) ===
  = vault-oracle: unchanged
  ✓ improved site-warden: pass 0->1, fail 1->0
  ⚠ REGRESSED triage-desk: pass 2->1, fail 0->1

1 REGRESSION(S) — change made it worse.

And it exits non-zero on any regression, which means it drops into CI as-is.

This closes the loop the whole series has been building toward. Part four put an eval-gate around one artifact. The scorecard puts the same idea around the fleet: make a change, measure, and let a number tell you whether it helped. Without it, “I improved the prompt” is a feeling.

What twelve agents actually taught me

Composition beats capability. Nothing in agent #12 is clever. It’s three known patterns and a JSON file. The hard part was never making an agent more capable — it was making twelve capable things fit together in a way I could still reason about at 2am.

Structure beats instruction. The single highest-leverage line of code in the fleet is export default disableTool();. Every constraint I wrote in prose eventually bent. None of the constraints I wrote in the tool policy did.

Type the boundaries. A boolean the runtime enforced turns “review your work” into a gate. Prose between two components is a bug waiting for a long context.

Failure needs a third option. Accept or drop is a false binary that will eventually produce a report that’s confidently wrong. Park it, name it, put it on disk.

Ship the red row. The orchestrator is the least finished thing I built and it’s the row I made sure to show you. A repo full of “last-mile (not exercised during build)” headers is more useful than a repo where everything claims to be done, because the first one tells you where to look.

If you build one thing from this series, don’t build a twelve-agent fleet. Take the agent you already have, pick the one job it does that could actually hurt you, pull it into its own directory with its own tools and its own eval, and delete the tools it doesn’t need.

Then run the suite before your next change and after it, and see which way the number went.

---

Series: 1. The fleet model · 2. Anatomy of an agent · 3. Root lockdown · 4. Eval-gating · 5. The capstone