Jack Schannep, editor of The DowTheory.com, has several times requested that we reconsider parts of our assessment of his specific stock market forecasts, as follows:
Jack Schannep wrote on 12/23/05:
I had a minute so I thought I would check out if the ‘grading’ of my Letter changed from the April 10th “pedestrian” to the current “tentatively good,” but as you know the earlier one is not still posted. So I looked in on the recent one to see if I agreed with your pluses and minuses, and the results are . . . . I DO agree with you on most, particularly when you look to intermediate-term or short-term trading, but a half dozen I would submit you might review (with an eye to considering as pluses). They are:
9/3/03 “A ways to go” WAS up 4.3% and 9.3% after 63 and 253 days???
3/9/04 “Stick with the major (bullish) trend” – a year later +5.2%
4/8/04 “Still in a bull market” – yes, and we STILL are.
6/4/04 “Strong earnings growth is more of a positive factor than…” +6.4% a year later.
7/9/04 “the second half of the year will see even more highs”. Written at 10435 Dow, 1141 S&P, and 6605 NYSE (you’ll note I always include market levels in my Letters for later accountability), the prior ’04 highs had been 10737, 1157, and 6780. In the second half they rose to “even more highs” of 10783, 1211, and 7250. (63 days +2.6%, 253 days +9.8%).
I know, I know, I’m nit-picking. However, an extra six pluses changes the positive/negative ratio from 60/40 to 80/20!!! Either way, it’s your website, your judgments, and I have no complaints.
Our response:
Our 12/6/05 updated assessment of Jack’s forecasts added a “winning streak” of five straight pluses to his record of 4/10/05. Hence, we upgraded the overall assessment from “pedestrian” to “tentatively good.” Jack’s point that changing six items from minus to plus would improve his accuracy rate from 60% to 80% is an important one. This sensitivity emphasizes that the forecast sample we have for him is small, accounting for our use of the word “tentatively” in the overall assessment.
As noted at Guru Grades, in performing assessments of forecasts we rely on public information and try to put ourselves in the place of an individual investor reading them. How would such an individual react to each forecast? When an expert issues forecasts monthly, as does Jack, we tend to focus on the behavior of the market over the next month as the most important measure of the value of the forecast. With that in mind, here are our point-by-point responses to the items Jack questions:
The following is the assessment of Jack’s 9/3/03 forecast. For all of these items the four percentages represent the performance of the S&P 500 index over the 5, 21, 63 and 254 trading days after the publication date. We understand Jack’s comment, but in this case we reacted to “All…indicators are in full BUY mode.” It was not a good week or month to buy indices. We are leaving this one as a minus.
Next is the assessment of Jack’s 3/9/04 forecast. In this case, we would rather have been told to get out and stay out for a while. We think the risk-adjusted return on the index here does not justify being in stocks. We are leaving this one as a minus
Next is the assessment of Jack’s 4/8/04 forecast. We understand Jack’s literal point in his comment on this one, but there is substantial erosion of capital here after a month and still after three months. We would have felt like we were moving backward. Even after a year, the return on the index is anemic. We are leaving this one as a minus.
Next is the assessment of Jack’s 6/4/04 forecast. During 2004, S&P 500 aggregate operating earnings rose more than 20% year-over-year. The Federal Reserve began the present cycle of interest rate hikes in June 2004. One month and three months after the forecast, the index was down. We think many would agree that rising interest rates/inflation fears likely trumped the good earnings news over most of the year. We are leaving this one as a minus.
Next is the assessment of Jack’s 7/9/04 forecast. On this one, we yield. The S&P 500 index was 1113 on 7/9/04 and finished 2004 above 1200. The timeframes in the chart do not capture late 2004. We will change this one to a plus.
In checking on his comments, we found that we failed to rate Jack’s 5/6/04 forecast (next table element) during the December update of the review of his forecasts, even though sufficient S&P 500 index data was available. As noted above, S&P 500 aggregate operating earnings in 2004 rose more than 20% year-over-year. The Federal Reserve began the present cycle of interest rate hikes in June 2004. We think that interest rate hikes/fears likely dampened the stock market’s response to these strong earnings. We are making this one a minus.
The net effect of these changes is that Jack’s positive/negative ratio increases from 60/40 to 61/39. We have updated Guru Grades accordingly.
Jack Schannep wrote on 10/16/06:
I can’t believe you still use the 5-day timeframe when judging my letter, particularly since that is not my focus and you usually don’t receive the synopsis of my monthly letter within five days of publication. It is posted on the last day of each month and dated the next day, the 1st of the coming month.
I guess the 21-day timeframe is fair although that isn’t my focus either. The 5-day part of the evaluation process would seem to be the basis for the ‘minus’ on several dates (see 1/7/05, 3/4/05, and 4/8/05), if so I would hope you will re-evaluate them using the more appropriate timeframes??
I WILL acknowledge that the ‘minus’ for my cautiousness on your 8/12/06 issue IS warranted (and well deserved!). Incidentally, I reversed that outlook last Thursday.
One final thought — You marked down my 12/2/05 letter because (I suppose) 10940.55 was not reached in December (it was reached three (3) trading days later on 1/6). The rest of the quote was/is correct. That warrants a ‘minus’???
Our response:
For the reasons Jack cites, we have not considered the 5-day stock market return in grading his forecasts. However, because his forecasts are no more frequent than monthly, we have redone his grades with a 126-day (6-month) return substituted for the 5-day return. Depending on the specific forecast, the 126-day return may significantly affect the evaluation.
The following is the revised assessment of Jack’s 1/7/05 forecast. For all of these items the four percentages represent the performance of the S&P 500 index over the 21, 63, 126 and 254 trading days after the publication date. We are leaving this one as a minus because the Dow Jones Industrial Average (DJIA) (Jack’s focus) did not come close to setting an all-time high during 2005, for which this entry offered an outlook. In fact, staying out of broad market indexes was not a bad idea for most of 2005. The “no new highs” for the S&P 500 and the NASDAQ was not a challenging call.
The following is the revised assessment of Jack’s 3/4/05 forecast. We are leaving this one as a minus because broad market returns are negative at one, three and six months, and the 12-month return is well below average.
The following is the revised assessment of Jack’s 4/8/05 forecast. We are switching this one to a plus, but it is a close call. Returns at three and 12 months are acceptable, but returns at one and six months are not.
The following is the revised assessment of Jack’s 12/2/05 forecast. We are also switching this one to a plus. Jack is correct that he missed the timing of the first DJIA target by only one month, and the index reached the second target this month. The zero six-month return from the broad market gives pause, but the forecast is reasonably specific.
The net effect of these specific changes, plus other changes from reviewing all of Jack’s other forecasts with the six-month return data, is an increase in his positive/negative ratio from 60/40 to 62/38. We have updated Guru Grades accordingly.
Jack Schannep wrote on 1/30/10:
The recent update seems to show the same numbers for me (52,33,19) as last month, even as an entry is now rated that was not previously….my count comes up to 50,32,18 for 64%. I see one unrated (11/7/06) that should be a plus and two of the recent unrated ones (too soon to judge?). Is your adding calculator lying to you or am I missing something?
Our response:
The new accuracy rate (as of 1/30/10) is 33/52 = 63.46%, up from 32/51 = 62.75%. Excel rounds both to 63%.
The entry from 11/7/06 is probably of interest to those reviewing your past comments, but it is not gradable per the usual Guru Grades time-scale. It is mostly right now, pending the Dow hitting 16,000 now that the recession is over.
The bullish forecasts implied by the 6/12/09 and 10/16/09 entries need more seasoning. There are quite a few open comments among other experts like the one from 6/12/09, either calling a low in March 2009 or calling a resumption of the bear market to a new low.
As always, we invite readers to make their own judgments.
We appreciate Jack Schannep’s taking the time to review evaluation details.
Frequently Asked Questions
What is a reclama in the context of grading stock market forecasts?
A reclama is a request by a forecaster to reconsider how their predictions were assessed. In this case, DowTheory.com editor Jack Schannep repeatedly asked CXO Advisory to review its grades of his stock market forecasts (2003–2006), and CXO re-examined specific items and timeframes in response.
Did re-grading Jack Schannep's stock market forecasts make them look more accurate?
Re-grading modestly improved Jack Schannep's forecast record, but not dramatically. CXO Advisory's positive/negative ratio rose from 60/40 to 61/39 after one round of review (one 2004 forecast changed to a plus, another rated a minus), then to 62/38 after substituting a 126-day return for the 5-day return. Schannep argued six changes would lift the ratio to 80/20, but CXO accepted only a few of them.
How does CXO Advisory grade a stock market forecast as a plus or minus?
CXO Advisory grades forecasts using public information, trying to react as an individual investor reading the forecast would. It compares S&P 500 returns over set trading-day windows after publication: originally 5, 21, 63 and 254 days, later 21, 63, 126 and 254 days. For monthly forecasters like Schannep, the next month's market behavior is weighted as the most important measure.
Does a forecast being literally correct over the long run make it a good forecast for investors?
Not necessarily. CXO Advisory kept a minus on Schannep's 4/8/04 call that the market was 'still in a bull market' because the S&P 500 showed capital erosion after one and three months and an anemic one-year return. CXO judged forecasts by what an investor would have experienced, including risk-adjusted returns, not only by whether the stated view was eventually right.
What are the limitations of grading an expert's market forecasts like this?
The grading is sensitive to sample size and to the chosen timeframes. CXO Advisory noted that changing six items would swing Schannep's accuracy from 60% to 80%, showing his small forecast sample, which is why the overall assessment was labeled 'tentatively' good. Results also depend on judgment calls, such as which return windows are used and whether the timeframes capture later market moves, as with the 7/9/04 forecast whose late-2004 gains fell outside the charted windows.

