New tool catches AI agents citing the wrong source in 138 of 139 cases

The quiet error in AI answers

Imagine an AI assistant tells you a refund policy exists, but it pulls that fact from a general document instead of your specific account record. The fact is true, but the source is wrong. This is a subtle error that standard fact-checkers often miss because they only look at whether the information exists somewhere in the data pool.

Introducing ProvenanceGuard

Multiverse Computing released a new verification layer called ProvenanceGuard. It sits on top of AI agents that use the Model Context Protocol, or MCP, to call various tools. The system checks not just if a claim is true, but if the agent attributed it to the correct tool output.

How it works

The tool reads the agent's trace, which includes the specific outputs from each tool it called. It breaks the final answer into individual claims. For each claim, it identifies the most relevant source and checks if that source supports the statement. If the answer implies a different source, the system flags it.

Test results in medicine

The team tested this on a medical agent that used patient records and research articles. They analyzed 281 real traces. Human experts reviewed 361 claims from 40 answers. They identified 139 claims that should have been blocked due to wrong attribution. ProvenanceGuard caught 138 of them, letting only one slip through.

A cautious approach

The system is designed to be conservative. It held 67 claims that experts considered supported, sending them for review. This preference for a second look over a fast answer suits sensitive fields like healthcare, where a wrong source can be as damaging as a wrong fact.

Handling similar sources

In a harder test with similar sources, the tool correctly identified the exact source in 50.3% of claims. It scored 0.846 F1 for deciding which claims to block. While it caught all 50 cases where the source was clearly swapped, distinguishing between very similar sources remains a challenge.

Repairing blocked answers

When an answer is blocked, the system can attempt a repair. In one run, it resolved all 173 blocked answers. However, 144 of these ended in fallback text rather than a substantive rewrite. This means the system chose to avoid an unverifiable answer rather than inventing one.

Practical overhead

The verification process adds a modest delay. On the local configuration used for testing, it takes roughly half a second per answer. The core checks run in tens of milliseconds, making it feasible for real-time applications.

Why it matters for teams

For teams building agents that use multiple tools, this distinction is critical. A "grounded" answer should mean supported by the specific source named, not just any tool output. This prevents cross-source conflation, where a fact from one tool is incorrectly credited to another.

Limitations and next steps

The current setup uses local models for controlled testing. While the approach can be adapted to cloud services, it requires its own calibration. The system does not yet handle the nuance of choosing among many highly similar sources with high accuracy, which remains an area for improvement.

Does your current evaluation check which specific tool provided a fact, or just that the fact exists somewhere?