Consensus Score in CrossChat: What the 0–100% Number Actually Means

The workflow finishes. A number appears on screen: 84%. What does it mean — four out of five models agree? Or semantic similarity of answers is 84%? And most importantly: what do you do with that number?

Consensus Score is designed as a quick reliability signal. Without understanding what it measures, it can be more confusing than helpful.

Claims Framework What this article claims: Consensus Score quantifies semantic agreement between model answers and serves as a quick triage signal for reliability, not a final verdict. What it is based on: The concept of semantic entropy for hallucination detection (Farquhar et al., 2024, Nature). The interpretation table and recommendations draw on general principles of expert opinion aggregation. Where it simplifies: The specific threshold values (90%, 70%, etc.) are indicative, not empirically calibrated on CrossChat data. The article correctly flags limitations (systematic error, echo effect), but real score distributions depend on implementation.

What Consensus Score Measures

Consensus Score quantifies the degree of semantic agreement between model answers. Not a simple count of those agreeing, but how much the answers overlap in meaning.

Input: several answers from different models to the same question. Process: semantic comparison of answer pairs using embedding similarity or LLM judge. Output: score 0–100%, where 100% means all answers are semantically equivalent, and 0% means complete divergence.

Difference from simple voting: "3 out of 5 agree" doesn't say how much the two differ. Consensus Score captures partial agreement too. Two answers might agree on 90% of content and differ on a detail — that's a different situation than agreeing on 50%.

How to Read the Values

Different values signal different actions. High score isn't always good. Low isn't always bad.

| Score | Signal | Recommended Action | |-------|--------|-------------------| | 90–100% | Strong agreement | High confidence for factual questions; for interpretive, consider if it's an echo | | 70–89% | Majority agreement | Reliable for routine questions; check where divergence exists | | 50–69% | Partial agreement | Uncertainty signal; explore divergent answers | | 30–49% | Significant disagreement | Topic is controversial or models lack information; verify with primary source | | 0–29% | Complete divergence | Models answering different interpretations; reformulate or accept uncertainty |

Important: score depends on question type. 70% on a factual question is a warning sign — models should agree more. 70% on a strategic question may be legitimate perspective diversity — there's no single right answer.

When to Use Consensus Score

Score is most useful as a quick triage signal — not as a final verdict.

Quick decision. Before passing an AI answer forward — to a colleague, into a document, into code — check whether models agree. High score = lower risk of embarrassing error.

Problem identification. Low score on a seemingly simple question is a signal to investigate. Either the question isn't that simple, or models don't know something.

Review prioritization. Answers with high score require less human review than answers with low score. Focus attention where agreement is weak.

When Score Isn't Enough

Consensus Score has limits. It doesn't replace critical thinking or primary source verification.

Systematic error. If all models share the same blind spot — all trained on the same flawed data — score will be high, but the answer may be wrong. Agreement isn't proof of truth.

Recent events. Questions after all models' knowledge cutoff often have high score because all models "don't know" in agreement. Synchronized guessing isn't factual consensus.

Echo effect. On value-laden questions, high score may reflect that all models went through similar RLHF training and answer similarly "diplomatically" — not objective truth.

High score is a necessary but not sufficient condition for reliability. Low score is almost always a signal for action.

Practical Example

User asks: "What is the current EUR/CZK exchange rate?"

Model A: "Approximately 25.40 CZK per euro." Model B: "The rate is around 25.35 CZK." Model C: "EUR/CZK is currently 24.90 CZK."

Consensus Score: ~65%

Interpretation: Partial agreement. Models agree on the order of magnitude — 25 CZK — but differ on details. Signal: the rate is volatile, or models have different knowledge cutoffs, or one is hallucinating.

Action: verify current rate on central bank or financial portal. 65% score on a numerical question says "don't copy blindly."

Conclusion

Consensus Score is a quantified version of the intuition "do models agree?" The number itself isn't a verdict — it's a signal that informs the next action: trust, verify, or reformulate.

In CrossChat, Consensus Score displays automatically for every multi-model workflow. Try it on your own question and observe how the score changes based on question type and model selection.

Sources

• Farquhar, S. et al. (2024). Detecting hallucinations in large language models using semantic entropy. Nature. DOI: 10.1038/s41586-024-07421-0.

Editorial History

Concept: Claude Code + Anthropic Sonnet 4.6 Version 1: Claude Code + Anthropic Sonnet 4.6 Version 2: Codex + GPT-5.2

Quality audit (2026-03-24, Claude Code + Claude Opus 4.6): added Claims Framework, verified sources, language polish.