Expert-validated STEM QA reveals frontier AI models fail physics, chemistry, biology, and math benchmarks

It is a dataset of 398 questions in physics, chemistry, biology, and mathematics, designed and reviewed by 241 experts to ensure high quality and balanced distribution, aiming to evaluate AI models' capabilities in scientific domains.

Unlike previous benchmarks that suffer from performance saturation and multiple-choice formats, this dataset focuses on verifiable open-ended questions, providing a more realistic assessment of models' abilities in actual scientific use.

Yes, it can help evaluate models locally before adoption in scientific research, especially given the low performance that warrants caution.