LitReview Arena Reveals: AI Models Lose to Humans in Scientific Literature Reviews

It's a battle-style evaluation platform designed specifically to assess the quality of AI-generated scientific literature reviews, where domain experts compare anonymized drafts across five tailored criteria.

According to the results, the strongest models win only 23% of decisive matches against human drafts, meaning humans remain significantly superior in overall quality and research utility.

LitJudge is an automated evaluator calibrated on human expert preferences, achieving much higher alignment than traditional methods (0.78 vs 0.467), making it a reliable tool for evaluating literature reviews without costly human intervention.

Yes, the code and data are publicly available on GitHub, allowing researchers and academic institutions in the region to adopt these standards for evaluating AI tools in scientific research.