After Benchmark Saturation: Study Finds Model Transcripts Still Carry Capability Signals

It is the point at which several models approach perfect scores on a benchmark, so the benchmark loses its ability to differentiate between them and to inform safety cases and frontier policies.

The study tests three approaches: pairwise judging between two transcripts, single-transcript absolute scoring, and predictions from deterministic transcript features or embeddings. Pairwise judging achieved the best results.

Not always. Predictive power is affected by distribution shifts, and the protocols underestimated the capabilities of the first reasoning models introduced in early 2025.

The study is preliminary and uses retrospective simulation, so it is a research signal rather than a production-ready tool. Evaluation teams can track it for planning internal benchmark lifecycles, but independent validation is needed before relying on it.