How should investors view financial advice from artificial intelligence (AI) models based on large language models, which have neither finance credentials nor investing track records? In their October 2026 paper entitled "Judge for Yourself: Confidence Beats Accuracy in AI Financial Advice", Golnaz Bahrami, Anthony Cookson and Austin Moss study advice preferences on Chatbot Arena, where users vote on answers to same question from two randomly assigned and anonymous AI models. Model anonymity guarantees that users do not vote based on AI reputation. Votes may indicate a decisive preference between the two answers, or they may result in a tie (both answers good or bad). For each AI answer, they task Claude Sonnet 5 to: (1) grade its accuracy based on Certified Financial Planner Board practice standards and life cycle prescriptions; and, (2) assess AI confidence in and actionability of the answer. Using votes on 3,060 financial questions (2,212 of which are decisive) from two public releases of Chatbot Arena full conversations during June to August 2024 and April to July 2025 and Claude answer assessments, they find that:
Subscribe to Keep Reading
Get the research edge serious investors rely on.
- 1,200+ research articles
- Monthly strategy signals
- 20+ years of backtested analysis
Cancel anytime