Does a Second Model Catch More Bugs?

Executive summary

The prevailing bet across the AI industry is that stacking more models on a problem, more agents, more debate, more deliberation, yields better answers. This study put a specific, falsifiable version of that bet to the test in the domain where it is easiest to check: finding real bugs in real code.

Two configurations reviewed the same twenty-five historical software defects, each with a known, maintainer-verified fix as ground truth. The first, the council, paired Anthropic's Claude as defender with OpenAI's Codex as an adversarial critic, arguing under an evidence-enforced protocol. The second, the control, ran a single model performing the same structured adversarial review against itself. The only intended difference between them was the identity of the critic.

Success was defined in advance and frozen with a cryptographic hash before the first task ran: the council had to detect at least 20% more real defects than the control, without a meaningfully worse false-alarm rate, at no more than triple the cost. It failed the first and central gate, and it failed in the opposite direction from the hypothesis. The council detected fewer defects than the lone model, nine against fourteen, and there was not a single task where the two-model council caught a bug that self-critique missed. A blind preference test, conducted without knowing which review came from which arm, favored the single-model reviews nine times out of ten.

This is a negative result, and because the experiment was pre-registered, isolated against contamination, and graded blind by two independent models, it is a trustworthy negative result. The recommendation it produces was also fixed in advance: do not build the two-model council. Ship the simpler, cheaper, structured single-model review instead. The remainder of this document records the thesis, the system, the method, the numbers, and the two honest caveats that keep the finding from being read as more than it is.

The question, stated so it could lose

Every capable AI reviewer shares a weakness with every other: where its blind spots overlap, no amount of self-examination reveals them. The intuitive fix is a second set of eyes, and specifically a second set trained by a different lab, on the theory that two models with different lineages fail in different places. If that theory holds, an adversarial exchange between them should surface defects that either alone would miss.

That is a real hypothesis, and it is worth money. It underwrites a growing category of products, and it is the implicit promise of every "heavy" or multi-agent mode now shipping. But most claims in its favor are demonstrated against weak baselines: two models beating a lazily-prompted single model proves little. The honest test pits the council against a single model given every structural advantage the council has, the same evidence rules, the same rounds, the same discipline, so that the only variable left is whether a second, independent model earns its seat.

Does structured adversarial review between two frontier models detect measurably more real defects than one model running an equally rigorous self-critique, by enough to justify the added cost and complexity? The frozen research question

The wording matters. It could lose. A result that cannot come back negative is not a measurement, and the entire value of this project rests on the fact that it was built to be able to disappoint its own author.

From one-shot plugin to adjudicated debate

The project began with an existing tool. OpenAI's codex-plugin-cc lets a developer summon Codex from inside Claude's workflow to review code, and it ships with an adversarial mode in which Codex attacks a change as hard as it can. But that review is one-shot and one-directional: Codex issues its critique once, Claude relays it verbatim, and the exchange ends. There is no rebuttal, no chance for a defender to answer with evidence, no iteration toward agreement, and no impartial verdict, just a single volley.

That gap is where the theory came from. If a lone critique already helps, would a genuine structured debate, two different models forced to argue under evidence rules until a neutral referee declares the matter settled, surface defects that neither a single pass nor a model examining itself would catch? The reused foundation supplied the hard part: a working channel between the two models. Everything that turns a one-shot critique into a trial was built on top of it, and that added layer is precisely what the benchmark set out to test.

What was actually built

The result is a small, deterministic debate machine with three roles. A critic attacks a piece of code, tasked to break confidence in it. A defender answers every attack, conceding, rebutting, or partially accepting. And a referee, an ordinary program with no intelligence of its own, enforces the rules, owns the record, and decides when the debate ends. The referee cannot be persuaded, which is the entire point of separating it from the two parties who can.

Three rules do the real work. First, no claim counts without checkable evidence: a finding that cites no file, no line, no reproducible behavior is stamped unsupported and struck from the verdict. Second, no rejection counts without counter-evidence: "I disagree" is bounced back until it points at the guard clause or the test that proves the point. Third, nothing settled can be quietly reopened: once a finding is accepted or withdrawn, the referee refuses any later message that tries to rewrite it. Every exchange is written to a ledger on disk that neither model can edit directly. The models argue; the file remembers.

The output is not a better answer in the abstract. It is a verdict, computed mechanically from the ledger, of the form ship, ship-with-fixes, or do-not-ship, accompanied by the full evidentiary record of what was contested and why. That auditable trail, not the verdict alone, is what distinguishes this design from a black-box deliberation that simply hands you a synthesized conclusion.

A system designed by the process it tests

The architecture was not authored by one party. It was negotiated across roughly seven rounds of adversarial exchange between Claude and ChatGPT, each critiquing the other's design, each required to concede only with reasons. The record of that negotiation is itself a small demonstration of the method, and an honest one, because it cuts against the eventual result.

Over those rounds, the second model found three genuine defects in the first model's implementation, including a state-machine flaw that would have let a verdict be quietly laundered from do-not-ship to clean. The first model, in turn, caught the second contradicting itself and misattributing positions it had never taken. Later, a fresh instance of the first model caught a specification contradiction that both prior parties had reviewed and approved. No participant reviewed its own work successfully; every real defect was caught by someone who had not written it.

The experiment

Twenty-five real defects were drawn from the histories of active open-source projects, each one a bug that maintainers later fixed, so that the correct answer is known but was hidden from the reviewers. The set was balanced across defect classes, concurrency and race conditions, error and retry handling, resource leaks, boundary and null-state errors, data-integrity failures, and injection flaws, and split across two languages. Bugs with public security advisories were deliberately excluded to reduce the chance either model had memorized the fix.

Each defect was reviewed twice, once by each arm:

• Control (Arm A): a single model performing structured adversarial self-critique, playing critic and defender in separate, isolated passes under the identical protocol, schema, evidence rules, and round structure.
• Council (Arm B): the full two-model debate, Claude as defender, Codex as the independent critic, over the identical artifact.

The referee verified that both arms received byte-for-byte identical inputs before either began, so any difference in outcome could be attributed to the one variable of interest. Three success criteria were then written down and frozen:

The interpretations were fixed at the same time: pass all three, and the council is worth building out. Fail S1, and the architecture simplifies to single-model review. Those rules were committed, with the specification's cryptographic hash recorded in the commit, before the first task ran. Nothing about the outcome could move the goalposts, because the goalposts were nailed down while the field was still empty.

Guarding against fooling ourselves

An AI reviewer that can see the fix is not detecting anything; it is reading. The bulk of the engineering in this project went into ensuring neither arm could reach the answer key, and much of it was prompted by contamination the system caught mid-flight and refused to ignore.

The code handed to reviewers was rebuilt with its version history amputated, so the corrective commit was physically absent from anything they could reach, and an independent audit re-derived that guarantee both ways, verifying that the fix was gone and that nothing legitimate had been stripped with it. The session that orchestrated the run had, by necessity, seen the task list, so it was quarantined: it was never permitted to argue or to grade, only to move files. Every reviewer and grader ran as a fresh context with no memory beyond the artifact, the rules, and the ledger.

Mid-run, a reviewer disclosed that a build step had pulled a published release of the project under review into its workspace, a channel that could place fixed source beside the buggy slice. The affected tasks were rebuilt and re-run in frozen, dependency-isolated environments, and the exposure was logged and remediated rather than buried. Separately, the graders caught a reviewer that returned a detailed report claiming to have written and validated a file that did not exist, the one integrity property never checked mechanically, now demonstrably fallible, and exactly why grading verifies against ground truth rather than any reviewer's word.

Grading itself was doubled and blind. Two independent frontier models scored all fifty debates against the fixes, with no knowledge of which arm produced which review. Where the two graders diverged, the disagreements were settled by seven systemic scoring rulings and three blocking-dispute adjudications, every one of them made without visibility into which arm a finding came from, so no ruling could favor a preferred answer.

What the numbers said

The central result is not close, and it does not point where the hypothesis predicted. The council detected fewer of the twenty-five known defects than the single model did, and every defect the council caught, the control had already caught alone. There is no task in the set where a second model found something self-critique missed.

A finer cut sharpens the picture. Counting only what each arm's critic surfaced independently, the control found fourteen and the council's external critic found four; the council's other five detections came from the shared defender, not from the second model at all. On the axis the experiment was built to measure, the independent contribution of the second model, the number is four against fourteen.

The two secondary gates passed but tell us less than their checkmarks suggest. Cost favored the council heavily, it ran at roughly a quarter of the control's cost, but a cheaper reviewer that finds less is not a bargain. And the false-positive gate passed vacuously: across 231 total findings, neither arm produced any verified false positives at all. The gate meant to catch "noise dressed as rigor" caught nothing, because the structured protocol kept both arms disciplined regardless of how many models were in the room. That is a real finding about the machinery; it is silent on the value of the second model.

Finally, the human preference test, conducted blind, agreed with the machine graders without being told their verdict. Presented ten pairs of reviews with all labels and costs stripped, the author preferred the single-model review in nine of ten cases, gravitating consistently toward the tighter, higher-signal output over the longer, broader one.

Two caveats, placed in front of the verdict

Intellectual honesty cuts toward the loser as readily as the winner, so two qualifications belong before any conclusion, not after it.

The comparison was tilted, and it tilted toward the winner

The two arms differed by slightly more than the identity of the critic. The control's critic operated with a fuller operational brief, the protocol's state-transition rules, output schema, and known environmental hazards, that the council's external critic did not receive. That asymmetry biases exactly the axis S1 measures, in the control's favor. This run therefore does not establish that the second model is a weaker critic in general, only that under these conditions it did not help, and the pre-registered rule says to ship the simpler thing when it does not. A cleaner future test would hand both critics an identical brief. That is a different experiment; this one is closed.

The bottleneck was judgment, not search

The most consequential diagnostic sits in the failures themselves. On five tasks, an arm reached the exact ground-truth code and did not file it: a probe that printed the defect was logged as a negative result; a mechanism was named at the correct lines across consecutive rounds and then withheld on a technicality. These were not failures of search. More critics, more rounds, or more models would not have fixed them, because the evidence had already been gathered and the eyes had already seen it. What failed was inference on evidence in hand.

What survives the result

The headline conclusion is unambiguous and was fixed in advance: on this evidence, a two-model adversarial council is not worth building over structured single-model review for code defect detection. The simpler configuration found more, cost less, and was preferred blind. That is the recommendation, and the discipline of pre-registration is what makes it safe to follow rather than argue away.

But the result retires a hypothesis, not the machinery. Three things survive it intact. The protocol itself, evidence-enforced, ledger-backed adversarial review, produced clean, zero-false-positive, disciplined output in both arms; its value was demonstrated, just not its dependence on a second model. The auditable record it generates, a defensible trail of what was checked and why, is a genuine differentiator for any setting where a reviewer must show its work, not merely assert a conclusion. And the method proved out on the object level: it is possible to take a fashionable industry premise, test it against a strong baseline under conditions that could embarrass you, and get an answer you can trust.

The practical shape of the survivor is therefore "one model, structured adversarial self-review, with an audit trail," which is cheaper and simpler to run than the council that was tested and, on this evidence, at least as effective. It also resolves a live industry question in passing: where same-model structured deliberation is good enough, the cross-model premium is not automatically paid. It has to be earned, task by task, and here it was not.

What this study does not claim

The result is narrow by construction, and stating its boundaries is part of reporting it honestly. It concerns code defect detection against known ground truth; it says nothing directly about open-ended design, strategy, or writing tasks, where evidence is softer and a second perspective may pay differently. Twenty-five tasks is enough to settle a single business decision with a clear margin, not to publish a population-level law. The critic-brief asymmetry described above means the specific pair tested was not compared under perfectly identical conditions. And the two models involved share substantial training data, so "independence" between them was always partial, an argument for expecting convergence, and one this result is consistent with.

None of these caveats rescues the council. They bound what the finding proves: not that a second model can never help, but that under a fair-to-strong single-model baseline, on this task, at this scale, it did not, and the honest, pre-committed response is to build the simpler thing.

A negative result, earned on purpose

The industry's reflex is to answer hard problems with more intelligence in the room. This study spent real effort to check that reflex in the one domain where checking is cheap, and found that, against a disciplined single model, the extra intelligence did not pay. The council found fewer bugs, the lone model found more, and a blind human preferred the lone model nine times in ten.

The value here is not the disappointment; it is the confidence. A freeze, adversarial audits, blind dual grading, and caveats disclosed in front of the verdict rather than behind it exist so that when the answer arrives negative, it can be believed and acted on instead of rationalized away. Most people never get a clean no. This project spent the compute to earn one, and a trustworthy no is worth more than a hopeful maybe.

What this was built on

This work would not exist without OpenAI's codex-plugin-cc, the open-source plugin that first made Claude and Codex talk to each other from inside a single workflow. Its broker and transport layer, the machinery that lets one model drive the other over a local connection, were reused here directly, and its adversarial-review prompt and structured finding schema were the seed the council's debate protocol extended. The plugin is released under the Apache License 2.0; the reused components retain their original license and notices in the study repository, and the files derived from it are marked as modified.

The debate system was co-designed and executed in collaboration with Claude (Anthropic) and Codex (OpenAI). The experimental design, the pre-registration and its enforcement, the interpretation, and any errors in this report are the author's own.