Guru Grades

Mark Hulbert's Stock Newsletter Sentiment Index (Updated 5/22/09 to Reference a Robust Study of the Predictive Power of Various Sentiment Indicators, Including HSNSI)

A reader suggested that we review the stock market commentary of Mark Hulbert, editor of the Hulbert Financial Digest and Hulbert Interactive, both of which track the recommendations of a wide range of investing newsletters. Mr. Hulbert is also a regular columnist at MarketWatch.com. Because Mr. Hulbert uses his Hulbert Stock Newsletter Sentiment Index (HSNSI) as a principal quantitative tool in formulating his market outlook, we evaluate the usefulness of that index in predicting stock market returns rather than his qualitative commentary. HSNSI "reflects the average recommended stock market exposure among a subset of short-term market timers tracked by the Hulbert Financial Digest." Mark Hulbert regards HSNSI as a contrarian signal for future stock returns; when HSNSI is high (low), he views the outlook for stocks as generally bearish (bullish). Using a sample of 195 values of HSNSI over the period 7/22/02-10/30/08 (generated by searching MarketWatch.com for "HSNSI" and its predecessor "HSSI"), we find that:

The average value of HSNSI for the sample is 22.9%, with a standard deviation of 26.2% (very high volatility). The high for the sample is 67.8%, and the low is -36.1%. In terms of size alone, assuming normal variable distributions, the sample appears large enough to test predictive power with reasonable reliability for short-term stock returns (a few trading days or weeks), but is not large enough to test HSNSI reliably as a long-term stock market indicator. The sample may be biased due to selection and clustering, because Mark Hulbert may be more likely to cite HSNSI, and cite it frequently, when either HSNI or the stock market behavior is extreme. We address these possible biases later.

The following chart superimposes the HSNSI sample on a plot of the S&P 500 index. The blue dashed line is the average HSNSI for the sample. As noted, HSNSI has high variability. Visual inspection suggests that HSNSI, like many other sentiment indicators, may tend to be high (low) during market peaks (dips).

To add precision to that observation, we examine how HSNSI relates to future changes in the S&P 500 index.

The following scatter plot relates the return on the S&P 500 index over the next five trading days (one week) to HSNSI. The Pearson correlation for the two series is -0.04, much less different from zero than in past measurements. The R-squared statistic is 0.00, indicating no relationship between HSNSI and stock returns over the next five days. The recent stock market plunge has added data mostly in the lower-left quadrant, substantially reducing the previously measured strength of correlation.

Note that the sample may have some selection/clustering biases at this point. Clustering leads to modest overlap of five-day test intervals, effectively reducing sample size and perhaps overweighting extreme values of HSNSI.

The next scatter plot relates the return on the S&P 500 index over the next 21 trading days (one month) to HSNSI. The Pearson correlation for the two series is -0.06. The R-squared statistic is again 0.00, indicating no relationship between HSNSI and stock returns over the next 21 days. The recent stock market plunge has added data mostly in the lower-left quadrant, substantially reducing the previously measured strength of correlation.

Note that the sample may have some selection/clustering biases at this point. Clustering leads to some overlap of 21-day test intervals, effectively reducing sample size and perhaps overweighting extreme values of HSNSI.

The third scatter plot relates the return on the S&P 500 index over the next 63 trading days (three months) to HSNSI. The Pearson correlation for the two series is -0.24, indicating some tendency for the stock market to rise (fall) over the next three months when HSNSI is low (high). The R-squared statistic suggests that HSNSI explains 6% of stock price movement over the next 63 days. Only one new data point from the recent stock market plunge is reflected in this plot. When available, additional data may move both the correlation and the R-squared statistic toward zero.

Selection bias in the sample, along with substantial overlap of 63-day test intervals, have potential to mislead.

Next, we address the selection/clustering biases and examine whether particular intervals of HSNSI values have exceptional predictive power.

To eliminate test interval overlap and alleviate clustering bias, we winnow sample points so that no two surviving points are within five trading days of each other. The revised sample size is 146 (we delete 49 closely clustered sample points). Then we order the revised sample from lowest to highest HSNSI and relate the return on the S&P 500 index over the next five trading days to HSNSI. Ordering helps determine whether some ranges of HSNSI might be more predictive than others. The following chart shows the results. Note that the horizontal axis is not time-sequential. The Pearson correlation between the two series is still -0.11, and the R-squared statistic suggests that HSNSI explains about 2% of stock price movement over the next five days.

Visual inspection suggests the possibility of two return regimes below and above an HSNSI value of about 20%, and a possible relationship between low values of HSNSI and high stock return volatility. For the 88 (58) instances when HSNSI is greater (less) than 20%, the average return on the S&P 500 index over the next five trading days is 0.6% (-0.2%), compared to 0.1% for the entire winnowed sample. Recent data have diminished the difference in returns between the <20% and >20% regimes.

One interpretation of these results is that low HSNSI relates to periods during which investors are scrambling out of or into the stock market in mass, and there is not enough liquidity to accommodate their correlated (panicky) actions. Slight predictive power may arise from the existence of two regimes rather than a linear relationship between HSNSI and future 5-day returns.

To eliminate test interval overlap and alleviate clustering bias for a longer test interval, we winnow sample points so that no two surviving points are within about 21 trading days of each other. The revised sample size is 61. We again order the revised sample from lowest to highest HSNSI and relate the return on the S&P 500 index over the next 21 trading days to HSNSI. The following chart shows the results. The Pearson correlation between the two series is 0.02, and the R-squared statistic is 0.00, indicating no relationship between HSNSI and next-month stock returns.

Visual inspection mildly suggests two return regimes below and above an HSNSI value of about 30%. The average return on the S&P 500 index over the next 21 trading days is 0.7% when HSNSI is below 30% and -0.5% when HSNSI is above 30%. The average 21-day return for all days in the winnowed sample is 0.1%. Recent data have diminished the difference in returns between the <30% and >30% regimes.

Finally, to refine analysis for an even longer test interval, we winnow sample points so that no two surviving points are within about 63 trading days of each other. The revised sample size is just 23. We again order the revised sample from lowest to highest HSNSI and relate the return on the S&P 500 index over the next 63 trading days to HSNSI. The following chart shows the results. The Pearson correlation between the two series is 0.17, and the R-squared statistic is just 0.01, suggesting that HSNSI may not useful as an indicator for 63-day stock market returns. This case especially would benefit from a longer-duration sample.

Results for all of the above cases indicate that new data from a relatively short period (the extremes of the past three months) can substantially alter the relationship between HSNSI and future stock returns. As maintained by Nassim Taleb, the fat tails of the distribution of stock returns undermine the usefulness of "normal" statistical measures such as correlation and R-squared.

HSNSI appears to be more strongly related to past than future stock returns. The Pearson correlations between HSNSI and S&P 500 index returns over the past 5, 21 and 63 trading days (uncorrected for clustering bias) are 0.33, 0.62 and 0.71, respectively. When the stock market advances (declines), HSNSI tends to rise (fall). For returns over the future 5, 21 and 63 trading days, as noted above, the correlations are -0.04, -0.06 and -0.24. In other words, it may be easier to predict the future value of HSNSI based on past stock returns than it is to predict future stock returns based on the value of HSNSI.

In summary, simple statistics indicate that the Hulbert Stock Newsletter Sentiment Index has little or no predictive power for stock returns over the short and intermediate terms. There may be regimes of low HSNSI values that weakly indicate elevated short-term stock returns.

See below for Mark Hulbert's comments on these results.

See Guru Grades for a snapshot of the accuracies of various experts in forecasting the direction of the U.S. stock market, including links to detailed individual evaluations. We do not include Mark Hulbert in the snapshot because the above review is fundamentally different from the reviews of guru commentaries.

For research on other sentiment indicators, see Blog Synthesis: Sentimental Journey.

On 10/31/08, Mark Hulbert wrote:

Thanks for your rigorous analysis of the Hulbert Stock Newsletter Sentiment Index. I am sorry you had such limited data with which to conduct your analysis, which -- as you conceded -- greatly reduces the potential power of any test results. We have found--using our data set back to 1985, and correcting for various statistical biases, including the ones you mention--markedly greater statistical significance than you found from your more limited sample.

Our response:

The sample used derives from the nearly 200 mentions of the Hulbert Stock Newsletter Sentiment Index in MarketWatch columns over the past 6+ years. It may be that there is some skew in the published data, or that the predictive power of the index varies over time (and has diminished). If you have made public your study of the index, we will be glad to cite it. Or, if you make public the complete dataset, we will be glad to run analyses on the complete set, as well as subsets to test robustness over time.

[See our blog entry of 5/22/09 for a summary of a study of several sentiment indicators, including the Hulbert Stock Newsletter Sentiment Index. Results of this study suggest that poll-based sentiment indicators such as the Hulbert Stock Newsletter Sentiment Index are proxies for simple price trend and that they add minimal value to price-based indicators.]