trending
Visa has launched a new industry benchmark for agentic commerce that suggests artificial intelligence is still some distance from consistently making reliable purchasing decisions on behalf of consumers.
Visa Benchmark Exposes Reliability Gap
The Visa Agentic Commerce Benchmark tested 22 AI models from 12 laboratories across 15 realistic shopping scenarios, including the purchase of televisions, laptops and personal care appliances.
Across all individual test runs, the models achieved a 57.2% pass rate, meaning 42.8% failed to meet Visa’s full criteria. Success required the AI to uncover every relevant customer requirement and then make the correct buy or no-buy decision.
The strongest-performing models passed 82.2% of attempts, while the weakest managed just 28.9%.
Discovery Proves Critical to Shopping Accuracy
Visa’s results suggest the quality of the conversation is central to performance.
The five models that uncovered the greatest number of hidden buying requirements achieved an average pass rate of 70.2%. The five that identified the fewest passed only 44.4% of the time.
In one test, an AI helping a consumer choose a television had to uncover requirements that were not initially disclosed, including budget, video-format compatibility, HDR10+ support and gaming needs.
The exercise highlights a fundamental difference between conventional search and agentic commerce. A successful shopping agent must not simply retrieve products; it must establish what the customer actually needs before acting.
Consistency Remains the Bigger Problem
Repeatability was weaker still. Each model was given three attempts at the same scenario, and all three attempts succeeded in only 34.2% of cases.
That means a model could complete a shopping task correctly once but fail when repeating the same exercise.
More concerningly, 54% of failed runs still ended with the AI making a purchase. In those cases, the product either failed to meet the customer’s identified requirements or the model had not discovered enough information to justify the transaction.
Trust Will Determine Whether Agentic Commerce Scales
Visa’s own research suggests consumer confidence remains limited. While 72% of US consumers surveyed had used an AI assistant, only 23% trusted AI to handle payment transactions on their behalf.
The company plans to expand the benchmark into checkout, payment and order placement, moving beyond product selection into areas where mistakes could have direct financial consequences.
The findings suggest the obstacle to agentic commerce is no longer whether AI can shop, but whether it can do so reliably enough to earn permission to spend. Before autonomous purchasing scales, the industry will need to solve consistency, explainability and control — not simply model intelligence.
Comments