BenchMIRT: Are LLM Benchmarks Measuring Real Performance or Memorization?

BenchMIRT is a new benchmark from the Allen Institute for AI that measures LLMs' true reasoning ability by modifying questions to prevent memorization.

While MMLU relies on static questions that may appear in training data, BenchMIRT uses modified questions requiring deep understanding, showing performance drops of up to 40%.

Yes, integrating BenchMIRT with other benchmarks is recommended before deploying models in sensitive applications like healthcare or government services.