EvalDetectBench: A New Benchmark for Measuring Evaluation Awareness in Frontier Language Models

EvalDetectBench is an open-source benchmark designed to measure how well large language models can recognize they are being evaluated, a concept known as 'evaluation awareness', and it works with any Inspect-compatible evaluation.

Because if models behave differently during evaluation than in actual deployment, it undermines the validity of evaluation results, which are crucial components of current AI safety frameworks.

EvalDetectBench uniquely addresses methodological biases found in prior research, such as the impact of the generator model's identity, and offers a correction mechanism via per-model calibration and generator harmonisation, making it more accurate.

Yes, especially for organizations relying on model evaluations for safety and performance decisions, as the benchmark provides a reliable tool to ensure results reflect true performance rather than mere test-taking behavior.