Orchestration Quality Patterns

What You’ll Learn

Session 27 gave you the Workflow tool’s mechanics: agent(), pipeline(), parallel(), schemas, resume. This session is about what you build with those mechanics. Spawning twenty agents does not make an answer correct — the structure of the workflow does.

By the end, you’ll know:

• The seven canonical quality patterns: adversarial verify, perspective-diverse verify, judge panel, loop-until-dry, multi-modal sweep, completeness critic, and no-silent-caps
• Why loop-until-dry must dedup against everything seen, not just what was confirmed
• How to scale workflow structure to the ask, and let a token budget shape the run instead of crashing it
• How to compose the patterns into one exhaustive-review script

The Problem

A naive workflow fans out N agents, collects their answers, and merges. It feels rigorous. It has three failure modes that no amount of fan-out fixes:

1. Plausible-but-wrong results. A single agent asked “is this a bug?” tends to say yes convincingly. Merging twelve such reports gives you twelve confident maybes.
2. Unknown-size discovery. If you don’t know how many bugs exist, one pass tells you nothing about completeness. “The agents came back” is not “the search is done.”
3. Silent truncation. A stage that keeps “the top 10 findings” or samples a subset throws away coverage without telling anyone. The report looks complete because nothing in it says otherwise.

The fix is structural. You don’t prompt your way out — you arrange agents so that wrong findings get killed, discovery proves its own exhaustion, and every cap is visible.

How It Works

Adversarial Verify

For each candidate finding, spawn N independent skeptics whose prompt is to refute it — not to review it, not to score it, to prove it wrong. If a majority refute, the finding dies.

The prompt direction matters. An agent asked “verify this” anchors on agreement; an agent asked “this is probably wrong — show why” actually re-derives the claim. Independent skeptics (each in its own context, unaware of the others) can’t collude on the same mistake. This is the direct countermeasure to plausible-but-wrong results.

Perspective-Diverse Verify

A variant: instead of N identical refuters, each verifier gets a distinct lens — correctness, security, performance, does-it-reproduce. You trade some redundancy for coverage: three identical skeptics might all miss a race condition that a lens explicitly assigned to concurrency would catch. Use identical refuters when the question is “is this claim true?”; use lenses when the question is “is this claim true and is it the whole story?”

Judge Panel

For generation tasks (designs, fixes, explanations) rather than verification tasks: run N independent attempts from different angles, score them with parallel judges, then synthesize from the winner plus the best ideas of the runners-up. The synthesis step is what separates this from “pick the best of three” — losing attempts often contain one insight the winner missed.

Loop-Until-Dry

For unknown-size discovery, keep spawning finder rounds until K consecutive rounds return nothing new. That termination condition is the only honest signal of exhaustion — a fixed round count is just a guess wearing a loop.

The subtle rule: dedup against everything seen, not just what was confirmed. Suppose round 1 finds candidate X, and your judges reject it. If your dedup list only holds confirmed findings, round 2’s finders will “discover” X again, the judges will reject it again, and the loop never goes dry — judge-rejected findings reappear forever. The seen set must record every finding ever reported, whatever its fate.

Multi-Modal Sweep

Parallel agents each searching a different way: by container, by content, by entity, by time. Ten agents running the same search strategy share the same blind spots; four agents with orthogonal strategies cover each other’s. In a code review this means one finder sweeps file-by-file, one traces data flow across module boundaries, one follows changed functions to their call sites, one inspects error paths.

Completeness Critic

After the main pass, a final agent gets one question: “what’s missing?” Its job is not to find more instances but to find uncovered categories — “nobody checked the migration scripts,” “no finder looked at concurrency.” Its findings seed the next discovery round. This is the workflow-level version of the error-recovery reflex from Session 20: assume the process failed somewhere and go look.

No Silent Caps

If coverage is bounded anywhere — top-N selection, sampling, a budget-forced early exit — log what was dropped. A log() line costs nothing; an invisible cap costs trust in the entire report. This is the same observability discipline you applied to production setups in Session 24, pointed at the workflow itself.

Scaling to the Ask

Not every request deserves the full arsenal. “Find bugs in this diff” is a bounded ask: one multi-modal sweep plus adversarial verify is proportionate. “Audit this thoroughly” is an exhaustiveness ask: loop-until-dry, a judge panel, and a completeness critic are the point, not overkill. Ultracode-style runs are explicitly optimized for the most exhaustive correct answer rather than the cheapest one — but that scale is opt-in, and matching structure to intent is the orchestrator’s first judgment call.

Budget-Driven Scaling

When the user gives a token target (like +500k in the prompt), the script sees a budget object: budget.total (a number, or null when no target was given), budget.spent(), and budget.remaining(). The pool is shared across the main loop and all workflows in the turn, and the target is a hard ceiling — once spent reaches total, further agent() calls throw. So don’t treat the budget as decoration: check remaining() at loop boundaries and stop discovery gracefully, logging the rounds you dropped. A workflow that dies mid-round with an exception reports nothing; one that exits early with a logged cap still reports everything it confirmed.

Hands-On Example

Here is the composed script: multi-modal finders, dedup against seen, a three-lens judge panel, looping until two consecutive dry rounds — with a budget guard and no silent caps.

export const meta = {
  name: 'exhaustive-review',
  description: 'Multi-modal bug hunt that loops until dry, judged by a three-lens panel',
  phases: [{ title: 'Hunt' }, { title: 'Report' }],
};

const findingSchema = {
  type: 'object',
  properties: {
    findings: {
      type: 'array',
      items: {
        type: 'object',
        properties: {
          file: { type: 'string' },
          summary: { type: 'string' },
        },
        required: ['file', 'summary'],
      },
    },
  },
  required: ['findings'],
};

// Multi-modal sweep: four DIFFERENT search strategies, not four clones.
const modes = [
  'Sweep the changed files one by one for logic errors',
  'Trace data flow across module boundaries for contract mismatches',
  'Follow each changed function to its call sites and check the callers',
  'Inspect error paths and edge cases the changes touch',
];

const lenses = ['correctness', 'security', 'performance'];

const seen = new Set(); // every finding ever reported — confirmed or not
const confirmed = [];
let dryRounds = 0;
let round = 0;

phase('Hunt');

while (dryRounds < 2) {
  round += 1;

  // Budget-driven scaling: exit before the hard ceiling makes agent() throw.
  if (budget.total !== null && budget.remaining() < budget.total * 0.15) {
    log(`Budget cap: stopped discovery after round ${round - 1}; later rounds dropped.`);
    break; // no silent caps — the report will say discovery was bounded
  }

  const known = [...seen].join('\n');

  const batches = await parallel(modes.map((mode, i) => () =>
    agent(
      `${mode}. Report bugs in the current diff of this repository.\n` +
      `Do NOT repeat findings already known:\n${known}`,
      {
        label: `finder-r${round}-m${i + 1}`,
        phase: 'Hunt',
        model: 'haiku',   // mechanical fan-out: cheap tier,
        effort: 'low',    // low effort
        schema: findingSchema,
      },
    ),
  ));

  // Dedup against SEEN, not confirmed — otherwise judge-rejected
  // findings get "rediscovered" every round and the loop never dries.
  const fresh = batches
    .filter(Boolean)
    .flatMap((b) => b.findings)
    .filter((f) => {
      const key = `${f.file} :: ${f.summary}`;
      if (seen.has(key)) return false;
      seen.add(key);
      return true;
    });

  if (fresh.length === 0) {
    dryRounds += 1;
    log(`Round ${round}: dry (${dryRounds} consecutive).`);
    continue;
  }
  dryRounds = 0;
  log(`Round ${round}: ${fresh.length} new candidates.`);

  // Perspective-diverse judge panel: three lenses per finding, each
  // prompted to REFUTE. Majority refutation kills the finding.
  const verdicts = await parallel(fresh.map((f) => async () => {
    const votes = await parallel(lenses.map((lens) => () =>
      agent(
        `You are a skeptical reviewer. Lens: ${lens}.\n` +
        `Assume this finding is WRONG and try to refute it:\n` +
        `${f.file}: ${f.summary}`,
        {
          label: `judge-${lens}`,
          phase: 'Hunt',
          model: 'fable',  // judgment stage: top tier,
          effort: 'high',  // high effort
          schema: {
            type: 'object',
            properties: {
              refuted: { type: 'boolean' },
              reason: { type: 'string' },
            },
            required: ['refuted', 'reason'],
          },
        },
      ),
    ));
    const refutals = votes.filter(Boolean).filter((v) => v.refuted).length;
    return refutals >= 2 ? null : f;
  }));

  for (const f of verdicts.filter(Boolean)) confirmed.push(f);
  // A completeness critic would slot in here: ask "what's missing?"
  // and feed uncovered categories into the next round's finder prompts.
}

phase('Report');
log(`Confirmed ${confirmed.length} of ${seen.size} candidates seen.`);

await agent(
  'Write the final review report, ordered by severity, for these ' +
  'confirmed findings:\n' +
  confirmed.map((f) => `- ${f.file}: ${f.summary}`).join('\n'),
  { label: 'synthesizer', phase: 'Report', effort: 'high' },
);

Walk through the load-bearing choices:

• seen vs confirmed are separate collections. Dedup keys go into seen the moment a finding appears; confirmed only grows after the panel votes. Collapse them and the loop can never terminate honestly.
• Termination is dryRounds < 2 — two consecutive rounds with zero new findings. One dry round can be luck; two is evidence.
• Finders run on haiku at low effort; judges run on fable at high. Cheap models for mechanical fan-out, the top tier for verification — the tier-mixing pattern from Session 25. The synthesizer omits model entirely and inherits the session model, the right default when you’re not sure a different tier fits.
• parallel() thunks that throw resolve to null, and a skipped or terminally-failed agent() returns null — so every collection step runs .filter(Boolean) before use. A crashed judge becomes a missing vote, not a crashed workflow.
• Both caps are logged. The budget exit and the final confirmed ... of ... seen line mean the report’s coverage is auditable from the run log alone.
• Concurrency is handled for you. Each round launches 4 finders, then up to 3 × fresh judges; anything beyond the per-workflow cap of min(16, CPU cores − 2) simply queues, and the 1,000-agent lifetime cap backstops a loop that refuses to dry.

What Changed

Naive fan-out	Quality-pattern composition
N agents, one pass, merge everything	Rounds continue until the search proves dry
Findings accepted as reported	Findings survive a refutation panel or die
N clones of the same search	Orthogonal search modes cover each other’s blind spots
Completeness assumed	Completeness interrogated by a critic
Caps and sampling invisible	Every bound logged; coverage is auditable
Budget overrun crashes mid-run	Budget checked at loop boundaries; graceful, logged exit
Same structure for every ask	Structure scaled to “find bugs” vs “audit thoroughly”

Key Insight

Answer quality is a structural property of the harness, not just the model.

The same model produces different-quality answers depending on the structure it’s embedded in. One agent asked “find the bugs” produces plausible output. The same model split into finders that search four different ways, skeptics forced to refute, a loop that must prove exhaustion, and logs that admit every cap produces output you can defend — because every claim survived an attack and every gap is on the record.

That’s why these patterns have canonical names. They aren’t prompt tricks; they’re architecture. When the ask justifies the scale, you reach for the pattern the same way a backend engineer reaches for a retry-with-backoff — not invented per incident, composed per need.

Next Session

Session 29 covers Persistent Memory — the file-based memory directory that lets Claude Code carry curated knowledge across sessions: one fact per file, a MEMORY.md index, four memory types, and the hygiene rules that keep recall trustworthy.