Guru Stock Market Forecasting Grades
Can experts, whether self-proclaimed or endorsed by others (publications), provide reliable stock market timing guidance? Do some experts clearly show better intuition about overall market direction than others?
We have accumulated reviews of the public U.S. stock market forecasts of various investing/trading experts for more than two years. With over 4,650 measurements for 52 gurus, including bulls and bears and technicians and fundamentalists, we have critical mass for: (1) assessing the forecasting acumen of the stock market gurus as a group; and, (2) ranking experts according to the accuracy of their past forecasts. This kind of forecasting ability is different from, but may be related to, stock picking expertise.
Note that the overall assessment of the stock market forecasting ability of experts in aggregate is far more reliable, based on sample size and duration, than the evaluations of individuals.
Note also that this study is not a test of whether the outputs of the experts are interesting, stimulating or useful in ways other than predicting the behavior of the overall U.S. stock market.
We update this page and the underlying guru forecast database approximately biweekly.
Snapshot - Notes - Individual Gurus - Q&A
We restrict reviews to publicly available material (freely available on the web), putting ourselves in the place of an individual investor trying to locate value in the marketplace, and mindful of concerns about copyright and trade secrets. Sometimes we find public records on the web sites of the experts themselves and sometimes on web sites of other parties (for example, the business media). This approach helps keep a level playing field for reviews, and it allows readers to check easily the context of forecasts and the reasonableness of our judgments.
The following table (last updated 6/13/09) summarizes the results of our reviews of the publicly available forecasts of 52 experts regarding the future direction of the overall U.S. stock market. The chart has two sections:
- Active Reviews are those for which at least one U.S. stock market forecast less than one year old is available from the indicated source(s).
- Dormant Reviews are those for which no U.S. stock market forecasts are available from the indicated source(s) within the past year, either because the source or the individual gurus no longer publish such forecasts. We retain dormant reviews and include them in overall forecast accuracy calculations to avoid survivorship bias.
For each expert, the table shows the total number of measurable forecasts, the numbers of forecasts we judge to be essentially right and essentially wrong, and the accuracy rate. Accuracies range from a low of 19% to a high of 68%. The overall accuracy of the group based on both raw forecast count and on the average of forecaster accuracies (weighting each individual equally) is 48% (48%).
See the Notes section below for discussion of the grading process and for cautions on interpreting both the aggregate and the individual expert accuracy rates. See the Individual Gurus section below for links to the detailed evaluations for all these experts.
If you believe that past forecasting performance indicates future accuracy (skill, wisdom), you might want to keep tabs on the forecasts of the leaders. If you believe in reversals of fortune, you might follow those gurus with the lowest current accuracies. On the third hand, if you expect mean reversion in forecasting (pure luck), you might prefer the "cone of silence". A fourth alternative is to apply Bayesian updating and weight the predictions of experts according to their evolving track records.
In summary, stock market experts as a group do not reliably outguess the market. Some experts, though, may be better than others.
For a simple analysis of the relationship between attention paid to individual gurus and their forecasting accuracy, see our blog entry of 4/17/09. For another way to measure interest in individual gurus, see our blog entry of 6/10/09.
For an unscientifically collected list of reasons why gurus go wrong, see our blog entry of 10/14/08. For additional comic relief, see our blog entries of 8/2/07 and 8/3/07.
Please note these points regarding the results in this table:
We selected experts to be graded based on extensive web searches for public archives offering at least marginal stock market forecast sample sizes. (Readers helped identify some commentators.) There may be data availability bias in the aggregation; some types of commentators may be more likely to offer frequent public commentary than others.
Many of the samples for individual experts are small, rendering confidence in the associated accuracies low. The aggregate sample, however, is large.
Assessing stock market forecasts requires judgment (ours) because they sometimes contain ambiguities, equivocations and/or conditional statements. We expect that our judgment errors tend to cancel each other, but we may have biases. Our detailed judgments are available for inspection via the links at the bottom of this page. Further, each individual review provides a link to source commentaries and articles, so that readers may decide whether the essential forecast language properly reflects commentary context.
When an expert commentary is too vague to assess or does not include a stock market forecast, we either do not include it or (if it is of background interest) we include but do not label it either "right" or "wrong." For experts with small samples, we make a few exceptions for commentaries that include no market direction forecast but do offer some other significant and reasonably testable market-related prediction or recommendation.
When an expert issues weekly (monthly) commentaries, we tend to focus on the behavior of the market in the next week (month), unless the forecast specifies some other timeframe.
When judging whether a forecast is correct, we keep in mind empirical benchmarks based on weekly S&P 500 index data for January 1950-March 2008, as listed below. For example, if a guru says investors should be bullish over the next six months (26 weeks), and the market is up by only 2% over that interval, we would judge the call incorrect. Conversely, if a guru says investors should short the market over the next month, and the market is down only 1%, we would again judge the forecast incorrect (a losing position after trading and carrying costs). In summary, the grading process has normalizing or detrending effects such that the aggregate accuracy should probably be around 50%.
- About 50% of all one-week returns are greater than +0.1%.
- About 50% of all four-week returns are greater than +0.6%.
- About 50% of all 13-week returns are greater than +1.9%.
- About 50% of all 26-week returns are greater than +3.9%.
- About 50% of all 52-week returns are greater than +8.4%.
The tabular records of S&P 500 index performance for each individual guru are based on closing levels that commence as of the close on the forecast publication date. There is some looseness in this methodology because it does not take into account the publication timestamp of a forecast and the intraday (or opening) level of the index for that timestamp. The extreme case of looseness would be the different treatment of two forecasts timestamped just before and just after midnight (ET). In fact, many forecasters do not use timestamps. The grading of individual forecasts that focus on the short term takes into consideration this looseness in methodology
Assessments of different experts cover different timeframes according to the data available. This fact arguably weakens the case for ranking the experts. An expert who is stuck on bullish (bearish) would tend to outperform in a rising (declining) stock market. This effect should eventually cancel for the entire sample across all experts.
The private (for example, paid subscription) forecasts of experts may be more timely and more accurate than the forecasts recorded publicly. As noted above, we restrict reviews to publicly available material to: (1) maintain a level playing field for experts reviewed; and, (2) let readers check the reasonableness of our judgments.
With exceptions as noted, these assessments generally address forecasts of overall market direction, not the performance of stock picks. Most stocks, however, exhibit substantial co-movement with the overall market.
For further reading on the performance of experts in general, and what might distinguish a good market forecaster from a bad one, see Blog Synthesis: The Wisdom of Analysts, Experts and Gurus. Consider also also research on how individuals recollect and process advice as summarized in some of the items at Blog Synthesis: Individual Investing.
The following names link to detailed performance evaluations. Most of these gurus are included in the snapshot above, but some do not fit the methodology and are therefore not comparable.
Some of these individuals have contested our evaluations of their commentaries. In such cases, individual evaluations note the disputes and provide links to details.
A reader who is a financial advisor with a large investment services company sent a series of questions/comments regarding Guru Grades. In case others have the same questions, here are the queries and responses...
Question 1: I'm guessing that you get paid by the gurus you hype, no?
Response 1: No. Neither CXO Advisory Group LLC, nor any of its members personally, receive any payments or other compensation from the gurus reviewed on the site. The reviews are attempts to measure the ability of experts to forecast stock market behavior, not "hype."
Question 2: I'm flabbergasted by your top "gurus", as I am familiar with them as well, and they are very poor performers. How can you tout a marketing company (Fisher) as a top money management company? Do you research these firms at all?????
Response 2: The site does not "tout" any companies. It presents reviews of research, some models and some original analyses. The reviews and analyses presented are exactly as described on the site. It is up to readers to decide whether and how this information translates into action for them, or merely flabbergasts them.
Question 3: Why do you compare what people say in articles/periodicals? It means nothing for professionals/investors! Their actual management/performance would certainly be more beneficial to "private investors and financial advisors."
Response 3: As stated above, the material cited addresses the following questions: "Can experts, whether self-proclaimed or endorsed by others (publications), provide reliable stock market timing guidance? Do some experts clearly show better intuition about overall market direction than others?" A reader suggested the concept of comparing multiple experts in this way some time ago. The discussion there further notes that: "This kind of forecasting ability is different from, but may be related to, stock picking expertise." Results are not conclusive that any individual guru has special ability to forecast market behavior. The distribution of the accuracies of individual gurus about the mean is possibly normal. If the reviews of guru stock market forecasting records mean nothing to professionals/investors, then professionals/investors will pay no attention to them. Note that Blog Synthesis: The Wisdom of Analysts, Experts and Gurus catalogs considerable formal research on portfolio performance for different categories of investment managers, including: newsletter writers, hedge fund managers, mutual fund managers and brokerage firm analysts. Guru Grades is just one corner of the site, examining one aspect of investing expertise.
Another reader sent the following suggestion:
Suggestion: While your guru backtesting offers some fun in the way of balloon popping, it does not really offer much in terms of actual investment techniques. I would say the vast majority of your loyal audience is far more interested in [quantitative testing of trading rules] than in testing whether Guru X is or not another snake oil merchant. (Usually he/she is.) I have noticed that your tests of trading rules are widely cited in other places, but I have never seen anybody referencing your popping of another self-appointed guru. Reason? There is no surprise. We do know these guys offer zero. I humbly petition that you stop doing the Guru Grades.
Response: There are two aspects to learning in all fields – what to do, and what not to do. In investing/trading, following the advice of gurus falls mostly into the second category. Many seasoned investors/traders understand that stock market gurus are, by and large, self-promoting and unable to provide accurate financial market forecasts and winning trades with significant reliability. However, other investors are in learning mode regarding guru expertise. Following the advice of experts is an investment technique. Hopefully, Guru Grades is not just fun but also helps these latter investors discover quickly how much attention (and money) they should devote to this technique. Reviews of log files indicate:
Blog entries on quantitative tests of trading rules and academic studies tend to draw a moderate number of readers immediately as posted and then withdraw to a quiet retirement, occasionally picking up a little traffic later via searches or links.
Blog entries on gurus tend to draw more readers initially and have more staying power, especially those covering well-known individuals such as Bob Brinker, Jim Cramer, Ken Fisher, Bill Fleckenstein and the Fast Money experts. Follow-up traffic again comes via searches and links from message boards. If you follow the "Site Hot Spots" feature in the right margin of site pages, you will see that several guru reviews (unlike any specific quantitative analysis or academic study) are consistently among the top 25 site pages in terms of unique visitors each month. This indicates a steady supply of readers seeking information on the credibility of guru pronouncements. The scope of the Guru Grades effort lends support to its conclusions. Nevertheless, the frequency of adding new gurus has dwindled as the number tracked has grown. It is a lot of work to compile a forecast/recommendation history.
Guru Grades is probably not of interest to all segments the CXOadvisory.com audience, but site traffic logs indicate that there is a fairly large segment with continuing interest in this section.
Two readers (one a tracked guru) have suggested that we drop David Nassar from the the list of gurus because: (1) his record is inactive, with no additional forecasts being added; and, (2) his record is relatively short (about 18 months).
Response: As stated, the objectives for Guru Grades are to answer the questions:
"Can experts, whether self-proclaimed or endorsed by others (publications), provide reliable stock market timing guidance?"
"Do some experts clearly show better intuition about overall market direction than others?"
Results indicate that, on average, experts do not provide accurate market forecasts. Dropping gurus from the study as they become inactive might introduce survivorship bias into these results. Several other gurus in the table are at least temporarily absent from the public forecasting forum. A bigger, longer study is probably necessary to determine whether level of accuracy relates to survival.
It is much more difficult to measure whether the dispersion of individual accuracies is random or derived from genuine ability. In either case, it appears that about two-thirds correct may be the best one should expect. David Nassar's sample is relatively short in duration. However, his forecasting style appears to rely on short-term market trends (not broad bull or bear market views), such that there are a fair number of independent forecasts in his sample. In any case, individual reviews take sample size into consideration, and the discussion above includes several cautions regarding the scores of individuals, as follows:
"Note that the overall assessment of the stock market forecasting ability of experts in aggregate is far more reliable, based on sample size and duration, than the evaluations of individuals."
"Many of the samples for individual experts are small, rendering confidence in the associated accuracies low. The aggregate sample, however, is large."
"Assessments of different experts cover different timeframes according to the data available. This fact arguably weakens the case for ranking the experts. An expert who is stuck on bullish (bearish) would tend to outperform in a rising (declining) stock market. This effect should eventually cancel for the entire sample across all experts."
A reader asked: "If you are not a subscriber to a service such as Prechters Elliott Wave then how can you judge them?"
Response: Guru Grades rates experts based on their public forecasts for the U.S. stock market, offered either systematically on their own web sites or as reported by commentary aggregators and financial media such as MarketWatch or Forbes. Use of public information allows transparency in the rating process by quoting forecasts and showing the grade for each forecast. Readers can adjust the subjective grades as they choose.
A subscriber to an investment advisory service can evaluate the accuracy of market forecasts provided by the service, but the terms and conditions of such services generally forbid publication of contents by subscribers. A third party therefore has great difficulty assessing the fairness/completeness of an evaluating subscriber’s grading method.
Should one have to pay for a forecasting service over many years to get a reasonably reliable reading of its accuracy? Free trials do not help much, because randomness dominates results for short measurement periods.
The primary objective of Guru Grades is to measure whether experts can accurately forecast the market. How much attention should investors/traders give to the pronouncements of gurus? The Guru Grades method suggests that experts on average offer little or no forecasting ability. A ranking of experts on forecasting accuracy is a less reliable output. While there is dispersion in the accuracy rates among individual experts, it is not obvious whether differences are due to skill (i.e., persistence over very long periods) or randomness. Sample durations, frequencies and time periods vary among rated experts. See, for example, "Converging Guru Accuracies", which indicates that very long sample durations may be needed to detect true guru forecasting ability. "Expert Political Judgment: How Good Is It? How Can We Know? (Chapter-by-Chapter Review)" summarizes an example of a very long study with some similarities to Guru Grades. Some of the lessons there likely apply to Guru Grades.
Note that there are alternative accuracy measurement systems for gurus, with priority on distinguishing among individual experts, such as (no affiliations):
TimerTrac, which states: "We Track Market Timing Professionals". The gurus themselves must subscribe and submit calls to this service.
Hulbert Financial Digest/Hulbert Interactive, self-described as "a completely independent, impartial, and authoritative rating service that arms you with the facts about stock and mutual fund investment newsletter performance." This service subscribes to investment newsletters and constructs portfolios that seek to mirror both timing and stock-picking advice. It does not evaluate other types of gurus.
