InjazAI

Researchers released ESQ-Bench, an Oracle-first NL2SQL benchmark, showing closed models like GPT-4o and Claude Sonnet 4.6 significantly outperform open-weight models like Llama 3.2, with accuracy dropping as schema complexity rises.

1 min read

ESQ-Bench: New Benchmark Exposes NL2SQL Model Gaps in Enterprise Oracle Environments

Introduction

FAQ

What is the ESQ-Bench benchmark?

ESQ-Bench is a new benchmark for evaluating Natural Language to SQL (NL2SQL) models in enterprise Oracle environments, featuring 550 question-query pairs across three complexity tiers and measuring silent semantic divergence.

How does GPT-4o compare to Claude Sonnet 4.6 on this benchmark?

According to results, Claude Sonnet 4.6 outperforms GPT-4o on all tiers when using schema-linked prompting, achieving 87.4% vs 79.8% on Tier-1, 74.9% vs 60.3% on Tier-2, and 68.7% vs 57.2% on Tier-3.

Why is this benchmark important for MENA enterprises?

It helps enterprises in the region understand the real capabilities of NL2SQL models in their enterprise environments (especially Oracle), avoiding reliance on models that may fail on complex queries, leading to incorrect decisions.

Are open-weight models suitable for enterprise Oracle databases?

Currently no, as results show Llama 3.2 achieves only 13.3% execution accuracy, indicating a large gap with closed models, and may not be suitable for critical applications.

Source: arXiv cs.AI

AI-assisted content, human-reviewed.