MERIT Study: Long-Term Memory Lifts Tool-Using AI Agents to 0.55-1.00, But Agents Act Correctly Only 55% of the Time

MERIT is a benchmark and harness from arXiv:2609.05441 that measures the marginal utility of long-term memory in tool-using LLM agents under explicit token and dollar cost accounting, featuring episodic tasks, an automated leak check, and a difficulty ladder ending in updated-fact recall.

LoCoMo and LongMemEval measure question answering over dialogue history, whereas MERIT measures whether remembered facts actually change what a task-executing agent does, with per-operation cost metering.

Results suggest caution: embedding retrieval collapses unpredictably on updated facts (0.30-0.95), while structured fact stores and LLM summarization hold at 0.70-1.00, making the latter preferable for sensitive enterprise applications.

The 23,440 scored episodes cost $42.57, and full replay is never economical; the best condition per domain delivers 2.7-3.9x its marginal utility per dollar.