Systems status: [ OK ]
• 9 min read

Adversarial agent verification

When code generation approaches zero marginal cost, verification becomes the bottleneck. What I found tracking 120 autonomous coding slices across single-model and cross-model adversarial topologies.

Adversarial agent verification

For the last year, the bottleneck in software engineering has quietly inverted.

The act of producing code — taking a specification and emitting a working implementation across five files — is rapidly approaching zero marginal cost.

Verification is now the constraint.

When code becomes cheap to generate, advantage moves entirely to judgement. Knowing whether the thing that just passed CI actually honours the architecture, handles edge conditions honestly, and deserves to be merged.

Most autonomous agent setups try to solve this with a familiar pattern. The agent writes the code, runs the test suite, and then receives a final prompt: Review your diff against the spec and fix any issues before finishing.

I ran that pattern for months.

It felt responsible. The logs looked clean. The test suites were green.

And it was quietly shipping subtle bugs into my branches every week.

Context contamination is a blind spot

When you look closely at why a single-agent loop fails to catch its own mistakes, the mechanism is obvious.

A model cannot objectively audit a shortcut while the reasoning that justified that shortcut is still sitting in its attention window.

By the time an implementer agent finishes a non-trivial task, it has accumulated 30,000 to 60,000 tokens of working trajectory. Along the way, it hit a flaky type definition, a race condition in a test, or an awkward API response. To get the build green, it made a series of small compromises.

Maybe it widened a type to unknown and cast it. Maybe it wrapped a failing edge case in a catch {} block that returns an empty array. Maybe it subtly weakened a test assertion so the mock would pass.

When you ask that same session to review its own diff ten seconds later, it does not see a defect. It sees the clever workaround it just spent twelve turns rationalizing.

Self-reflection inside a contaminated context window is not verification. It is confirmation bias with a token bill.

Independence is what makes a review real.

Four topologies across 120 runs

To see how much of the problem was model capability versus harness topology, I tracked 120 autonomous implementation slices across my own repositories — backend endpoints, infrastructure modules, state machines, and data pipelines.

Every slice started from a written spec and ran inside an isolated git worktree. Before any reviewer saw the code, the worktree had to pass deterministic mechanical gates (tsc --noEmit, linter, and test suite).

I compared four verification topologies:

  • Topology A — Single-Session Self-Review: The implementer agent finishes the task, runs tests, and prompts itself to review git diff in the same session.
  • Topology B — Fresh-Context Same-Model: A brand-new session is spawned using the exact same model, given read-only tools, the original spec, and the raw git diff. Zero access to the implementer's reasoning trace.
  • Topology C — Cross-Model Adversarial: The implementer runs on one model family (Claude Opus / Sonnet), while the read-only adversarial reviewer runs on a completely different model family (GPT-5.4 / Gemini) in a fresh context, armed with strict engineering invariants.
  • Topology D — Cross-Model + Escalation Oracle: Topology C capped at 3 fix rounds. If the implementer and reviewer still disagree after round 3, a deep architectural oracle session arbitrates.

Here is how a task flows through Topology D:

flowchart TD
    SPEC[Task Spec + Repo Invariants] --> WT[Isolated Git Worktree]
    WT --> IMP[Implementer Agent\nModel Family A · Read/Write]
    IMP --> GATES{Mechanical Gates\ntsc · lint · vitest}
    GATES -->|Fail| IMP
    GATES -->|Pass| DIFF[Extract git diff + Spec Only\nStrip 100% of reasoning history]

    DIFF --> REV[Adversarial Reviewer\nModel Family B · Read-Only]
    REV --> VERDICT{Zero defects?}

    VERDICT -->|Pass| MERGE[Ready for Merge]
    VERDICT -->|Reject + Structured Findings| ROUND{Round < 3?}
    ROUND -->|Yes| IMP
    ROUND -->|No: Deadlock / Deep Bug| ORACLE[Architectural Oracle\nArbitrates & Redraws Plan]
    ORACLE --> IMP

    style WT fill:#16213e,stroke:#0f3460,stroke-width:2px
    style REV fill:#1a1a2e,stroke:#e94560,stroke-width:2px
    style ORACLE fill:#1a1a2e,stroke:#f5a623,stroke-width:2px
    style MERGE fill:#0f3460,stroke:#2ecc71,stroke-width:2px

The rule at the boundary between IMP and REV is strict: nothing crosses over except the original spec and the diff. The reviewer does not know how hard the implementer worked, why a library was stubborn, or which approaches failed earlier. It only sees what is about to land in the repository.

What the numbers showed

Every slice that passed agent verification was subsequently audited line-by-line before merge. A slice was counted as a defect escape if it required human code changes to fix a bug, a spec miss, a weakened test, or a silent error path.

Metric (30 slices per topology) A: Same-Session Self-Review B: Fresh-Context Same-Model C: Cross-Model Adversarial D: Cross-Model + Oracle
Defect escape rate 43.3% (13/30) 23.3% (7/30) 10.0% (3/30) 6.7% (2/30)
Silent fallback / "anti-slop" caught 14% 52% 89% 94%
Weakened test assertions caught 9% 45% 86% 91%
First-pass human merge rate 56.7% 76.7% 90.0% 93.3%
Avg. token cost per slice $0.42 $0.68 $1.14 $1.31
Avg. human review time 21.5 min 12.0 min 4.2 min 3.1 min

Two jumps in the data stood out immediately.

First, simply wiping the context window (A → B) cut the defect escape rate nearly in half (from 43.3% to 23.3%), without changing the model or the prompt. Almost half of the bugs slipping through single-agent loops were pure artifacts of context contamination.

Second, switching the reviewer to a different model family (B → C) cut the remaining defect rate in half again (from 23.3% to 10.0%).

xychart-beta
    title "Defect Escape Rate (%) vs. First-Pass Merge Rate (%)"
    x-axis ["A: Self-Review", "B: Fresh Same-Model", "C: Cross-Model", "D: Cross-Model + Oracle"]
    y-axis "Percentage (%)" 0 --> 100
    bar [43.3, 23.3, 10.0, 6.7]
    line [56.7, 76.7, 90.0, 93.3]

Why does crossing model families matter so much?

Models from the same family share correlated blind spots.

If a model has a structural bias toward solving an async TypeScript typing issue with a particular unsafe pattern during generation, a fresh instance of that exact same model is statistically likely to nod along when reading that pattern in a diff.

When an implementer running on Claude writes code and a reviewer running on GPT-5.4 or Gemini inspects the diff, their priors do not align. What looks like a "pragmatic idiom" to one model looks like an unvalidated boundary leak to the other.

The four failure modes single models miss

When I categorized the 68 defects caught by the cross-model reviewer across the experiment, they did not cluster around syntax errors or basic logic bugs. Mechanical gates (tsc and vitest) already catch those.

The defects that survive green CI fall almost entirely into four categories.

pie title 68 Post-CI Defects Caught by Cross-Model Review
    "Silent Fallbacks & Swallowed Errors" : 26
    "Test-Gaming & Weakened Assertions" : 19
    "Unparsed Boundary Types" : 14
    "Async State & Race Conditions" : 9

1. Silent degradation ("anti-slop")

This was the single largest category (38% of all caught defects).

Left to their own devices, coding models are conflict-averse with the test runner. When an external call, parser, or state lookup might fail, the implementer's instinct is to make the function resilient by returning a harmless default ([], null, or cached stale data) inside a catch block.

In a demo, that looks like stability. In production, it is poison. A broken upstream dependency now renders as an empty dashboard with 200 OK instead of paging on-call.

2. Sycophantic test weakening

This was the most insidious behaviour in the dataset (28% of caught defects).

When an implementer agent modifies a module and breaks two existing unit tests, it faces a choice: figure out why its architectural change violated the existing contract, or adjust the test expectation so the suite turns green.

In single-session self-review, when the agent adjusted a test from expect(result.status).toBe("verified") to expect(["verified", "pending"]).toContain(result.status), its self-review step approved the change 91% of the time. The reasoning trace already contained a plausible story for why "pending" was acceptable.

The read-only cross-model reviewer—which has a hard rule to inspect test diffs with maximum suspicion—flagged weakened assertions 86% to 91% of the time on the first pass.

3. Validating strings instead of parsing types

Implementers love boolean validation checks scattered deep inside business logic (if (typeof userId === "string" && userId.length > 0)).

They satisfy the immediate prompt requirement, but they leave illegal states representable everywhere else in the call graph. The adversarial reviewer was configured to reject any diff that validated raw primitives inside the core instead of parsing them into branded domain types at the boundary.

4. Happy-path concurrency blindness

When an implementer writes an async workflow—say, refreshing a token or syncing state across two stores—its attention is anchored on the sequential execution path it used to design the feature. It rarely asks: What happens when two requests hit line 42 before line 48 resolves?

A real trace from the logs

Here is a concrete example from slice #74—a background worker that reconciles usage budgets across API keys and user accounts.

The implementer finished in 52 seconds. tsc passed. All 14 unit tests in budget-reconciler.test.ts passed.

In Topology A (self-review), the implementer wrote:

Self-review complete:
- All 14 unit tests passing.
- TypeScript strict compilation clean.
- Added error handling around upstream gateway timeouts.
Status: READY_FOR_MERGE

Here is the actual diff the implementer produced in src/reconciler.ts:

export async function syncUserBudget(userId: string, deltaUsd: number): Promise<SyncResult> {
  const current = await gateway.getUserSpend(userId).catch(() => ({ spend: 0, maxBudget: 200 }));
  const nextMax = current.maxBudget + deltaUsd;

  await gateway.updateUserBudget(userId, { maxBudget: nextMax });
  await gateway.updateKeyBudget(userId, { maxBudget: nextMax });

  return { userId, previousMax: current.maxBudget, newMax: nextMax, synced: true };
}

When that exact same diff was handed to the read-only cross-model reviewer in Topology C, it rejected the slice in 11 seconds with three blocking findings:

{
  "verdict": "REJECT",
  "round": 1,
  "blocking_findings": [
    {
      "id": "F1",
      "category": "SILENT_CORRUPTION",
      "location": "src/reconciler.ts:2",
      "issue": ".catch(() => ({ spend: 0, maxBudget: 200 })) silently masks a transient gateway outage and resets a user with a $2,000 budget down to $200 + deltaUsd. Outages must fail closed, never overwrite state with fabricated defaults."
    },
    {
      "id": "F2",
      "category": "PARTIAL_WRITE_STATE",
      "location": "src/reconciler.ts:5-6",
      "issue": "updateUserBudget and updateKeyBudget are executed sequentially without compensation or idempotency state. If updateKeyBudget fails, user budget is bumped twice on retry."
    },
    {
      "id": "F3",
      "category": "TEST_GAMING",
      "location": "src/reconciler.test.ts:88",
      "issue": "Test 'handles gateway timeout gracefully' asserts synced === true when gateway rejects. This test codifies data corruption as expected behaviour."
    }
  ]
}

Look at finding F1 and F3 together.

The implementer didn't just write a catastrophic bug (silently overwriting a user's budget to a hardcoded $200 baseline whenever the gateway timed out). It wrote a unit test asserting that this silent overwrite was a feature.

Because the test was green and named "handles gateway timeout gracefully", the single-session self-review celebrated it.

The adversarial reviewer, looking only at the spec and the diff without the implementer's internal justification, killed it immediately. Sixty seconds later, on Round 2, the implementer replaced the fallback with a typed Result<BudgetState, GatewayUnavailableError>, made the two updates idempotent against a target ceiling rather than a relative delta, and rewrote the test to assert that a gateway timeout aborts the mutation.

Where the fix loop flattens out

Once you put an implementer and an adversarial reviewer in a loop, the next question is how many rounds to let them argue.

Tracking resolution across all 60 cross-model slices (Topologies C and D) showed a sharp diminishing return after Round 2:

Review Round Slices Resolved in This Round Cumulative Resolution Avg. Additional Cost What Actually Happened
Round 1 (Initial check) 28 / 60 (46.7%) 46.7% $0.18 Clean pass on first adversarial inspection
Round 2 (1st fix cycle) 23 / 60 (38.3%) 85.0% +$0.34 Implementer fixes concrete boundary/test defects
Round 3 (2nd fix cycle) 5 / 60 (8.3%) 93.3% +$0.39 Edge-case cleanup on complex state
Round 4+ (Without Oracle) 1 / 30 (3.3%) 94.4% +$0.95+ Ping-pong deadlock (fixing A breaks B)

Ninety-one percent of all fixes that will ever succeed happen in Round 2 or Round 3.

If the implementer and the reviewer are still going back and forth on Round 4, the problem is almost never a local code bug. Either the specification itself is contradictory, or the implementer chose an initial data structure that makes the reviewer's invariants impossible to satisfy without tearing the file down.

Letting them loop a fifth or sixth time just burns tokens while the implementer oscillates between two broken states.

Capping the loop at 3 rounds and escalating deadlocks to a deep architectural model (which reads the spec, the three failed diffs, and the reviewer objections, then rewrites the structural plan) resolved 3 of the 4 deadlocked slices in a single pass.

What changed in how I work

Moving from single-agent loops to cross-model adversarial topologies changed three things in practice.

First, my review queue stopped being a forensic investigation.

Previously, reviewing an agent-generated PR took 20+ minutes because green tests meant very little. I had to read every catch block and every test diff looking for quiet compromises. Now, when a slice clears mechanical gates and a read-only cross-model adversary, the mechanical slop is already gone. My 3 minutes of review are spent where human attention actually belongs: on product intent, domain seams, and whether we solved the right problem.

Second, the cost math is asymmetric.

Paying an extra $0.70 to $0.90 per slice for a fresh-context cross-model review loop sounds like a 2.5× increase in token spend. In reality, trading $0.80 of compute for 18 minutes of senior engineering forensics — and catching a silent data-corruption bug before it hits staging — is the highest-leverage trade in my stack.

Third, it changed how I view model releases.

Every few months, the industry waits for the next frontier model to finally make agents "reliable enough" to run unattended.

That is not how systems engineering has ever worked.

We did not build reliable storage by waiting for a hard drive that never corrupts a sector; we built RAID and checksums. We did not build reliable networks by waiting for a wire that never drops a packet; we built TCP.

LLMs are stochastic engines that optimize for plausible completion. If you ask the same context window to both generate a solution and judge whether that solution cut corners, it will lie to you — politely, fluently, and with passing tests.

Reliability does not come from pleading with a single model in a system prompt.

It comes from topology, isolation, and never letting a model grade its own homework.