When Is a Multi-Agent Code Judge Actually Grounded? Two Label-Free Measurements, and a Judge That Declines to Guess

The paper addresses a core failure mode in LLM-as-judge systems: when one language model judges whether another's code is correct, it does not report the absence of evidence, instead returning a confident verdict with reasoning that is indistinguishable from a verdict it actually had grounds for. Multi-agent verification is a promising response, decomposing a judgment into checkable claims and verifying each against evidence, and it works well when that evidence is a set of retrieved documents.

The authors argue such methods require two things of their evidence: it must be independent of the answer under review, and it must differ between the two candidates being compared. The second condition holds automatically with retrieved documents but stops holding in code judging. To test this, they run MARCH, a published framework, unmodified over 80 condition-by-cell measurements on two code judging benchmarks.

MARCH declares both solutions equally good on 78% to 95% of comparisons, reaching 4.4% accuracy where the same model asked directly reaches 43.7%. Neither easier problems nor a larger judge changes this outcome. Two measurements taken from the pipeline's own logs explain the behavior without needing labels.

Gating on one of those measurements, the pipeline declines the comparisons it cannot make and raises its accuracy from 20.7% to 36.9% while still answering half of all comparisons. The contribution is not a more accurate judge, but a label-free way to tell when a judge has no basis for its answer.