Study Finds 91 "Silent Failures" in AI Agent-Tool Interaction Across 15 Scientific Tools

It is a case where a tool invocation appears successful, but some or all of the information or functionality delivered through the API or wrapper is incomplete or missing, with no notification to the user or the agent. Both then treat the output as complete and correct.

The researchers built an audit mechanism to detect silent failures by examining 15 scientific tools along with their API and tool documentation inside the ToolUniverse environment. The study was structured around seven failure loci in the interaction chain, followed by manual validation of automatically discovered candidates.

Of the 91 failures, 51 occurred at the API layer and 25 at the wrapper layer, with the potential for silent failure amplification downstream.

The paper does not ship a production tool, but it offers a conceptual framework for testing, disclosure, and monitoring. Teams building agentic pipelines on external tools can adapt the audit methodology to verify data completeness before trusting outputs.