AI evaluation vendor AgentOracle deletes a 'Verified Claims' metric it says was never real

AgentOracle, a startup that sells automated confidence scores telling AI agents whether to trust retrieved information, has deleted a published metric after disclosing that the numbers behind it were never real.

The company's site offers a "Reputation Score" for domains such as reuters.com and arxiv.org, meant to tell an agent how much to trust content from that source before acting on it. Until Sept. 22, the same table also carried a "Verified Claims" column with figures such as 142,881 listed for reuters.com. A note added to the page that day says the column was removed because "those numbers were static text in this file, not values returned by any endpoint or backed by a real counter — there is no claim-count tracking anywhere in the service." The company says it chose to delete the figures "rather than corrected, since no real count exists yet to show."

AgentOracle markets its product as infrastructure letting an AI agent decide whether to "act," "verify" or "reject" a claim based on a numeric confidence score, charging $0.01 per claim evaluated and settling payment in USDC on Base through the x402 protocol. The reputation scores it continues to publish are also not drawn from a track record: the site describes them as "a starting value assigned to each source, adjusted as real queries accumulate."

The episode is notable mainly because the company disclosed it. AI Dispatch's index tracks agent-trust and evaluation vendors that publish self-reported performance figures with no visible methodology; most make no such correction when a number cannot be substantiated.

AgentOracle is listed in the index as a candidate entity. Its trust-gap scorecard, unrelated to this episode since both predate it, currently shows a 1 out of 3 on audit trail, a 0 on insurance and indemnity, and a 1 on outcome settlement — all within the published rubric's trust-gap range of 0-1.