Can large language models (LLM) be trusted for economic/financial forecasts during periods within their training data? In their April 2025 paper entitled "The Memorization Problem: Can We Trust LLMs' Economic Forecasts?", Alejandro Lopez-Lira, Yuehua Tang and Mingyin Zhu evaluate use of ChatGPT 4o (knowledge cutoff October 2023) for economic/financial forecasting via:
- Forecasts of variables before and after knowledge cutoff.
- Explicit instructions to ignore knowledge during periods before the cutoff.
- Masking of inputs (anonymized firm names or dates) to mitigate use of memorized data in forecasts before knowledge cutoff.
Using data for major economic indicators, stock index levels, individual stock returns/conference calls and Wall Steet Journal (WSJ) headlines during December 1989 through February 2025, they find that:
Subscribe to Keep Reading
Get the research edge serious investors rely on.
- 1,200+ research articles
- Monthly strategy signals
- 20+ years of backtested analysis
$17.99
/month
Cancel anytime
