Study Maps 14,767 LLM Benchmarks: Do They Measure Capability or Reproduce Model Bias?

It revealed a shift in evaluation criteria toward measuring action, interaction, and professional applications, alongside growing model participation in scoring, raising concerns about evaluation independence.

Because it may reproduce the evaluating model's preferences and blind spots, turning evaluation from independent evidence into a reflection of the model's own biases.

Yes. Regional teams rely on global benchmarks to select models. Understanding evaluation bias helps them build Arabic-context benchmarks instead of depending entirely on ready-made rankings.

Focus shifted from purely cognitive tasks to measuring action, interaction, and professional applications, with older and newer design elements coexisting.