IBM Research Publishes Benchmark Exposing AI Agent Failure Modes

IBM Research has published VAKRA, a benchmark framework designed to systematically evaluate how AI agents reason, use tools, and fail during complex tasks, according to a paper released on the Hugging Face platform.

The benchmark provides empirical analysis of where agentic AI systems break down, moving beyond simple accuracy metrics to examine the specific failure modes that emerge when language models are asked to chain together reasoning steps and tool calls to accomplish goals.

VAKRA — which stands for Verification and Analysis of Knowledge, Reasoning, and Actions — tests agents across multiple dimensions including their ability to select appropriate tools, sequence operations correctly, and recover from errors mid-task. The research found that even high-performing models exhibit predictable failure patterns when task complexity increases.

Among the key findings, agents frequently struggle with what the researchers describe as “reasoning-action misalignment,” where a model correctly identifies what needs to be done but fails to translate that understanding into proper tool calls. The benchmark also documents cases where agents enter repetitive loops, repeatedly attempting the same failed approach without adapting their strategy.

The work arrives as the AI industry pushes aggressively toward agentic systems capable of operating with minimal human oversight. Companies including Anthropic, OpenAI, Google, and Microsoft have all released or announced agent frameworks in recent months, raising questions about reliability and safety in production deployments.

“Understanding failure modes is prerequisite to building reliable agents,” the research team noted, arguing that current evaluation methods do not adequately capture the ways agents can fail silently or compound errors across multi-step tasks.

The benchmark is available as an open resource on Hugging Face, allowing other researchers and developers to test their own agent implementations against the framework. IBM has also released the underlying dataset and evaluation code.

The publication adds to a growing body of work focused on agent evaluation, an area that has lagged behind the rapid deployment of agentic capabilities across the industry.