Multi-Agent Debate: What Happens When AI Models Disagree

Two models receive the same question. One answers A, the other denies A and argues for B. Instead of a dead end, they begin iteratively revising their positions — each model sees the other's arguments and must respond. After a few rounds, they may converge on a stronger answer than either produced alone.

Intuition says disagreeing models are a problem. There is one correct answer — one of them is wrong, or both are. Multi-Agent Debate (MAD) inverts this intuition. In research by Du et al. (2023), models engaged in iterative debate: each model saw the others' responses and could revise its position based on their arguments. In many settings, this kind of structured adversarial feedback improves factuality and reasoning compared to a single-shot answer.

Adversarial pressure, which looks like a problem, works as a correction mechanism. This article explains why.

Claims Framework

• What this article claims: Iterative debate between models (Multi-Agent Debate) reduces hallucinations and improves factuality compared to single-shot answers. Heterogeneous models from different vendors produce better results than homogeneous sets. Adversarial pressure works analogously to scientific peer review or adversarial legal proceedings.
• What it is based on: Du et al. (2023) -- experiments with iterative debate between models; Dhuliawala et al. (2023) -- Chain-of-Verification as a related principle; Nemeth et al. (2001) -- psychological research on the devil's advocate role.
• Where it simplifies: The article presents MAD as a generally effective technique, but the actual degree of improvement depends on task type and model selection. Analogies to peer review and juries are simplified -- models lack the motivation and accountability of human participants. Claims about TruthfulQA are general, without citing specific numbers.

How MAD Works: Three Phases of Iterative Debate

Multi-Agent Debate is not simultaneous generation of multiple answers followed by averaging. It is an iterative protocol with structured revision phases.

Phase 1 — Independent response: Each model generates its answer to the question without seeing what others said. This phase ensures initial positions are genuinely independent — not influenced by another model's response.

Phase 2 — Adversarial revision: Each model sees the others' responses and must react. It can accept an opponent's argument and revise its position, reject it with an explanation of why, or propose a modified synthesis. Crucially, the model must explicitly justify its position — saying "I agree" isn't enough; it must explain why.

Phase 3 — Repetition: Phase 2 repeats until models converge (reach agreement) or hit a preset round limit.

Du et al. report that agreement can increase over rounds, and that debate can outperform simple "sample many answers and vote" baselines in some tasks. The difference isn't in the number of attempts — it's in the interaction structure.

Example: Models are asked a factual question with an uncertain answer. Model A says 1956, Model B says 1958. In the debate, B presents an argument — a reference to a specific database or logical reasoning chain. Model A must either defend its claim with concrete counter-arguments, or revise. The result isn't an average (1957) — it's an iteratively justified answer with explicit reasoning.

For functional MAD, three things are essential: explicit revision rounds, visibility of arguments (not just conclusions), and a clear termination mechanism.

Why Adversarial Pressure Reduces Hallucinations

Hallucinations in LLMs persist without external challenge. When a model generates a response by itself, it has no mechanism for detecting its own uncertainty. It generates words with the highest probability — not the highest factual accuracy. Confidence and accuracy are decoupled in LLM outputs.

An adversarial partner changes this. If Model B presents a counter-argument to Model A's claim, Model A must either defend its claim with explicit arguments (revealing weaknesses if they exist), or revise its position. The mechanism is analogous to scientific peer review: an author who must defend claims before a skeptical reviewer identifies weaknesses they would otherwise overlook.

Dhuliawala et al. (Chain-of-Verification / CoVe) show that structured verification questions can reduce factual errors in multiple settings. MAD operates on the same principle — the opponent's objections function as external verification pressure.

Key insight: adversarial pressure doesn't reduce hallucinations by "teaching models the truth." It reduces them by exposing unsubstantiated claims. A model can still hallucinate, but is less likely to propagate a hallucination into the final answer if it has passed through adversarial review.

Concrete example: A claim about a research study. Model A states: "Study X found Y with high confidence." Model B asks: "What year was the study published? Was it a randomized controlled trial or observational?" Model A must either supply supporting details (strengthening the claim) or reveal it doesn't know those details (weakening or retracting the claim). Without adversarial pressure, the claim would pass unchallenged.

Heterogeneous vs. Homogeneous Debaters — Why Composition Matters

MAD with identical models creates an echo chamber. The real benefit comes from models with genuinely different approaches to the problem.

Intuition says: more equally good models equals better results. Research says otherwise. If two instances of GPT-4 debate, both share similar training data, similar RLHF (Reinforcement Learning from Human Feedback) values, and similar architecture. Their "disagreement" in the first round is sampling noise — not genuine perspective divergence. They quickly converge to an average GPT-4 answer, not to the truth.

Du et al. compared homogeneous debates (same model family) with heterogeneous ones (different model families). Heterogeneous configurations were generally stronger on factual tasks. Different models have different blind spots — what one overlooks due to training data or RLHF alignment bias, another catches.

Example — ethical question: GPT-4 and Claude have different alignment values. Claude is generally cautious, GPT-4 is generally helpfulness-oriented. In a heterogeneous debate, Claude will raise concerns that GPT-4 underweights, and conversely, GPT-4 will surface practical aspects that Claude considers less. The result covers more relevant perspectives.

For MAD implementation: model selection is not a trivial detail — it's a critical design decision. The principle: maximize epistemic diversity of debaters, not their average benchmark score. Different vendor is better than different scale from the same vendor.

When Debate Converges to the Right Answer — and When It Doesn't

MAD doesn't work everywhere. On certain problem types, it generates a self-reinforcing error instead of a correction.

MAD works well on factual questions with verifiable claims, logical reasoning, mathematical problems, and analysis with clear criteria.

MAD performs poorly on value judgments (where there is no "correct" answer in the sense of verifiable truth), questions requiring proprietary or highly specialized knowledge, and — most critically — problems where all available models share a correlated bias.

The TruthfulQA benchmark (designed to test resistance to common misconceptions) showed less consistent improvement with debate than with pure mathematical tasks in some reports. Why? Because common misconceptions are present in the training data of many models. If all debaters share the same misconception, the debate converges to a confidently presented error — not the truth.

Example of correlated bias: we ask about a scientific consensus that is systematically distorted in popular sources. All five debaters share the distorted version from training data. The debate concludes with a confident but wrong answer — because there was no one to challenge the shared belief.

MAD is a tool for reducing random variance and idiosyncratic errors of individual models. It is not a tool for correcting systematic bias shared across all models. For detecting shared bias, you need external verification.

MAD Beyond AI — Adversarial Pressure in Human Processes

Multi-Agent Debate reproduces a principle that humans have long used in the most important decision-making contexts.

Scientific peer review is structured debate: an author presents claims, a reviewer raises objections, the author revises or defends. It works because the reviewer has an incentive to find weaknesses — not to confirm the author's conclusions. Adversarial legal proceedings are MAD in a legal context: defense and prosecution each present the strongest version of their case, each side seeking weaknesses in the other's arguments. The result is more robust than unilateral assessment.

M&A due diligence works the same way: a bull case team argues for the opportunity, a bear case team identifies risks. Their argumentation is structured debate — the result is a more complete picture than a single team would produce.

Psychological research confirms this principle. Nemeth et al. (2001) demonstrated that groups with an explicit "devil's advocate" — a member whose role is to challenge others' conclusions — achieve higher decision quality. The effect isn't that the devil's advocate is right. It's that they force others to explicitly articulate and defend assumptions that otherwise remain implicit.

Adding an adversarial model to an AI workflow — a model whose explicit role is to challenge others' claims — is the digital equivalent of the devil's advocate role. The most effective model isn't the one with the strongest arguments, but the one that systematically identifies weaknesses in others' arguments.

Limits of MAD: Latency, Cost, and Groupthink

MAD is an effective technique, but it has real tradeoffs.

Latency: Three debate rounds mean at least 3× more API calls. For real-time applications — chatbots, live assistance — MAD is impractical. For analytical tasks where accuracy outweighs speed, the overhead is justifiable.

Cost: More debate rounds means more tokens and higher cost. On factually rich tasks where MAD delivers benefit, the cost per accuracy improvement is calculable. On simple questions, the benefit doesn't justify the cost.

Groupthink: The most fundamental limit. If models share a correlated bias, debate amplifies it rather than correcting it. MAD needs genuinely heterogeneous debaters — same architecture with different temperature settings is insufficient.

MAD is most valuable for analytical tasks with verifiable claims, room for iteration, and tolerance for latency. It is not a universal upgrade over single-model approaches.

Practical Conclusions

Multi-Agent Debate is not a complex technique — it is structured iterative argument exchange. Four principles for application:

Identify suitable tasks. Factual questions with concrete claims, analytical tasks with verifiable conclusions, decisions with high error cost and tolerance for latency. Not value judgments, not real-time interactions.

Maximize debater heterogeneity. Different model vendors, different scales. GPT-4 + Claude + Gemini outperforms three instances of GPT-4 — because genuine perspective diversity depends on differences in training data and alignment philosophies.

Watch convergence, not just the conclusion. If models converge quickly and unanimously on a niche question — either it's a trivial case, or they share a correlated bias. Fast unanimous debate is a warning signal.

Separate disagreement types. Factual disagreement in debate adds value. Value-based disagreement displays perspectives, but a "correct result" doesn't exist as verifiable truth. Each type requires different interpretation.

Tools like CrossChat implement adversarial workflows in a structured way — Multi-Agent Debate is available as a predefined workflow where heterogeneous models go through iterative rounds of argumentation, and the output shows the history of position revisions along with the consensus score.

Sources

• Du, Y. et al. (2023). Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv:2305.14325. DOI: 10.48550/arXiv.2305.14325.
• Dhuliawala, S. et al. (2023). Chain-of-Verification Reduces Hallucination in Large Language Models. arXiv:2309.11495. DOI: 10.48550/arXiv.2309.11495.
• Nemeth, C. J. et al. (2001). The liberating role of conflict in group creativity. Journal of Personality and Social Psychology.
• Lin, S. et al. (2021). TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958. DOI: 10.48550/arXiv.2109.07958.
• Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155. DOI: 10.48550/arXiv.2203.02155. (InstructGPT / RLHF baseline.)

Editorial History

Concept: Claude Code + Anthropic Sonnet 4.6 Version 1: Claude Code + Anthropic Sonnet 4.6 Version 2: Codex + GPT-5.2 Quality audit (2026-03-23, Claude Code + Claude Opus 4.6): added Claims Framework, verified sources, language polish.