AlphaDiana: New Measurement System Shows Agent Harnesses Don't Always Improve Reasoning Accuracy

It is a research measurement system with modular interfaces for datasets, agents, environments, and scorers. It preserves harness-native control loops while controlling file reuse and memory retention separately, linking answer correctness and execution status to traces and resource records.

According to the findings, not consistently. Gains on questions the model failed alone can be outweighed by losses on questions it answered correctly without tools.

The results cover three open-source models and three open-source harnesses on specific benchmarks, so they are best read as a call to measure agent performance internally rather than assume tools or memory improve outcomes automatically.

Because the paper notes that low token entropy can accompany wrong answers, and less output per execution can mean more output tokens per correct answer when accuracy falls.