InjazAI

A new arXiv paper finds that multi-agent code judging frameworks like MARCH fail to distinguish between code solutions in 78-95% of comparisons, and proposes two label-free measurements to detect when a judge lacks grounds for its answer.

1 min read

Researchers Reveal When Multi-Agent Code Judges Lose Their Grounding

What the paper reveals

FAQ

What is the MARCH framework in code judging?

MARCH is a published multi-agent verification framework that decomposes a judgment into checkable claims and verifies each against evidence, originally designed for retrieved document verification.

Why does multi-agent verification fail at code judging?

Because the second condition — evidence differing between compared candidates — holds automatically with retrieved documents but stops holding in code judging, so the framework declares both solutions equal instead of distinguishing them.

Can teams in the MENA region benefit from these findings?

Yes, as it provides a label-free way to identify when a code judge lacks grounds, helping teams avoid relying on ungrounded verdicts in production pipelines.

Did a larger judge model improve accuracy?

No, neither a larger judge nor easier problems changed the outcome, suggesting the problem is structural in the nature of the evidence rather than in model capability.

Source: arXiv cs.AI

AI-assisted summary of the source above. High-confidence stories publish automatically; others wait for review. Facts that are not in the source are not added.