Predicting Short-term Market Returns with LLM-generated Market Sentiment

Does financial news sentiment as interpreted by large language models (LLM) such as ChatGPT and BARD predict short-term stock market returns? In their September 2023 paper entitled "Large Language Models and Financial Market Sentiment", Shaun Bond, Hayden Klok and Min Zhu separately test the abilities of ChatGPT and BARD to predict daily, weekly and monthly S&P 500 Index returns based on sentiments they extract from daily financial news summaries. ChatGPT is trained on information available on the web through September 2021. In contrast, BARD is connected to the web and updates itself on live information. The authors:

1. Ask each of ChatGPT and BARD to summarize the most important news from the Thomson Reuters News Archives for each trading day starting in January 2000.
2. Consolidate each set of daily summaries.
3. Ask each of ChatGPT and BARD to use their respective set of summaries to quantify market sentiment each day on a scale from 1 (weakest) to 100 (strongest) and separately evaluate the sentiment as positive, neutral or negative.
4. Relate via regressions each set of daily sentiment measurements to next-day, next-week and next-month S&P 500 Index returns. These regressions control for same-day index return, VIX, short-term credit risk and the term spread (plus additional variables when predicting monthly returns). 

For ChatGPT, analysis extends through September 2021 (the end of its training period). For BARD, analysis continues through July 2023. As benchmarks, they consider sentiment measurements from two traditional dictionary methods and two simple transformer classifiers. To estimate economic value of predictions, they compute certainty equivalent returns (CER) for a mean-variance investor who allocates between the S&P 500 Index and a risk-free asset each day according to out-of-sample sentiment measurements starting in 2006. Using Thomson Reuters News Archives and daily, weekly and monthly S&P 500 Index returns since January 2000, they find that: