smevals: An Open-Source Tool for Easy LLM Evaluation for MENA Developers

smevals is an open-source tool developed by Simon Willison in collaboration with Prime Radiant lab, allowing developers to create and run small evaluation suites to compare the performance of different AI language models, grade results, and display them easily.

MENA development teams can use smevals to evaluate language models before adopting them in their projects, especially those requiring Arabic language support, helping select the most efficient and cost-effective model.

The tool is simple and command-line based; any coding agent can learn it quickly. It only requires installing uvx and then running commands like uvx smevals run to execute evaluations and uvx smevals grade to grade results.

While designed for small to medium projects, large enterprises can use it as a preliminary evaluation tool before investing in more comprehensive solutions, saving time and resources.