The Agent Retrieved the Evidence and Then Failed to Use It

An engineer reviewing an AI agent's logs sees what looks like a successful run. The agent needed a fact, called a search tool, and got the relevant evidence back. But its next reasoning step missed the information sitting in the result.

That gap is the subject of Making Every Tool Call Count, a vision-language agent study submitted to arXiv on September 3. Its central finding is useful for anyone evaluating tool-using agents: a successful tool call does not establish that the call was necessary or that the agent used what came back.

The researchers tested an 8B vision-language agent that could crop images, perform reverse-image searches, and search text across seven image-grounded benchmarks. The study does not test enterprise deployments, but the failure it measures is relevant to systems where agents depend on external evidence.

Traditional outcome-based training can reward a trajectory when the final answer is correct. That creates a blind spot. An agent can search repeatedly before reaching the right answer, while another can retrieve exactly what it needs and then fail to extract the decisive information.

Both can look similar if the evaluator only checks the final answer.

How the Mechanism Works

The researchers introduce Necessary Tool-Evidence Paths, or NTEPs, to make those intermediate failures measurable.

Instead of prescribing an exact reasoning sequence, an NTEP identifies what evidence the agent needs and which tool should obtain it. Each call is evaluated twice: did the agent pursue a still-missing piece of evidence with the appropriate tool, and after the result returned, did its next reasoning step capture the required information?

The reward mechanism, NTEP-R, combines those checks with final-answer correctness. It also tracks completed evidence goals and penalizes unnecessary attempts to retrieve them again.

More searching is not necessarily better searching.

Analysis

The paper's most interesting result appears when accuracy and tool behavior are measured separately.

Across 4,417 evaluation examples, full NTEP-R reached 69.7% overall macro-average accuracy with 1.55 tool calls on average. A model trained with answer reward alone reached 69.6%, but averaged 3.11 calls. The paper cautions that this is not a fully controlled comparison because the answer-only model used a smaller historical training set.

The score barely moved. The agent's behavior did.

A separate analysis of 300 tool-using trajectories per model found that 78.8% of NTEP-R calls were both necessary and successfully used, compared with 69% for SenseNova-MARS-8B, the strongest same-harness reinforcement-learning agent in the paper's main comparison. Removing the mechanism discouraging repeated evidence goals pushed redundant calls from 1% to 13.5%.

NTEP-R did not eliminate evidence-use failures. In 3.8% of its calls, relevant evidence returned but was not captured in subsequent reasoning, compared with 2.2% for SenseNova. Its clearest gains appeared elsewhere: more calls were useful, fewer were repeated, and the agent more often stopped when it had enough evidence.

Implications for Enterprises

For teams building agents, tool-call success should be treated as an operational event, not proof of successful reasoning.

An observability system that records only whether a search, retrieval, or API call succeeded can miss several failures. The agent may pursue the wrong evidence, choose the wrong tool, retrieve useful information but fail to extract it, or continue searching after it already has enough.

Those distinctions affect more than accuracy. They affect latency, inference cost, external API usage, and the ability to diagnose why an agent failed.

For the engineer looking at that original run, a green check beside the search call would no longer be the end of the story.

Risks and Open Questions

The measurement has limits. NTEPs are constructed by a teacher model from agent trajectories, while Qwen3-VL-Plus judges whether calls align with evidence goals and whether returned information appears in subsequent reasoning. A blinded audit found 97% agreement with humans across 100 judge-involved cases, but that does not independently validate every generated evidence path.

The experiments also cover only three tools and predominantly English retrieval. It remains unclear how the approach performs with larger tool inventories, contradictory evidence, changing APIs, or enterprise data.

There is a deeper measurement problem too. Seeing evidence appear in subsequent reasoning does not prove that it causally determined the final answer.

For engineering teams, the immediate takeaway is more practical: agent observability may need an evidence layer between tool execution and final output. Teams need to know what the agent was looking for, what came back, whether it extracted what mattered, and whether it knew when to stop.

Further Reading

• Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Models
• MMSearch
• InfoSeek
• V* Bench
• HR-Bench
• Search-R1