Blog - Investing Notes

June 2, 2006 - A Reader's Out-of-sample Test of Some Technical Trading Research

Reader Wes Williams of Stafford TX offered the following observations and recommendation:

I read your blog entry of 4/11/05 on "Technical Trading Thoroughly Tested", carefully read (and re-read) the 27-page paper of Hsu and Kuan and then wrote some TradeStation programs to translate their results into 2003-present for the Russell 2000 index.

The results were quite disappointing. I then wondered why Hsu and Kuan did not publish more specifics about the parameters used in their moving averages and learning programs. In their conclusion on page 20 of 27 of their paper, their out-of-sample results were bad enough for me to avoid the entire methodology. After transaction costs in 2002, they report a 48% loss in comparison with a 22% loss for buy-and-hold on the Russell 2000! Their 10-year in-sample result cannot count since it is, after all, a sampling to set parameters for the next two years.

There is, of course, much more to be said on this subject. I used to be a Hedge Fund Manager in a partnership and have written thousands of such strategies and walk-forward-tested them. I applaud them for applying scientific disciplines - if only more people would - but trading a system with a 48% drawdown in a first sample of a walk-forward test? I wouldn't be able to sleep at night wondering if the system simply doesn't work any longer. I want to write them about how they conducted their learning methodology since it makes sense that it should *theoretically* work, but that is time I do not have right now.

I bring this to your attention because you reference this study in your blog entry of 4/11/05, which states: "The best simple rules are: (1) for the DJIA, a momentum strategy in volume rule (NOT SIGNIFICANT); (2) for the S&P 500, a contrarian on balance volume rule (NOT SIGNIFICANT); (3) for the NASDAQ, a two-day moving average rule (HIGHLY SIGNIFICANT); and, for the Russell 2000, a two-day moving average rule (HIGHLY SIGNIFICANT)." I first interpreted your comments as your own conclusions. I recommend that you consider a disclaimer, unless you endorse their conclusions, since my reproduction of their test indicates that it has stopped working.

But so far, very good stuff on your site. My strength is in the technicals and programming disciplines and I appreciate your work on the economics.

Our response is as follows:

Regarding a disclaimer:

There is a general disclaimer on the site in the footer that appears on each page. It is brief but, we think, fairly all-encompassing. There are pointers to the disclaimer (actively linked to it) in the text on the home page and at Stock Market Status and Positions. The latter two places seem the most likely places that readers might be tempted to act on site content.

For the sake of succinctness, we do not use a lot of hedge words in blog write-ups. For research such as that by Hsu and Kuan, we assume that readers sophisticated enough to implement it will understand (as you do) the source of the methodology, and the ambiguities and uncertainties of complex financial markets in general. We could add further disclaiming verbiage to each research summary, but entries would then become more cumbersome (if not unreadable).

For some other disconfirming out-of-sample tests, see our blog entries of 2/9/06, 7/14/05 and 7/6/05.

An important lesson of all these out-of-sample tests is that investors/traders must assume substantial doubt regarding the assertions even of academic experts regarding financial markets. The theoretical foundation of financial markets (and, more generally, economics) is very weak compared to those of the hard sciences. The field has few reliable ways to relate empirical data. And, there are a lot of academics in the field who must publish to survive.

See also the relevant reflections of Emanuel Derman in our blog entry of 11/19/05, and in the book (My Life as a Quant, Reflections on Physics and Finance) that it highlights.

Finally, see Blog Synthesis: Some Trading Indicators for analyses of the usefulness of other technical indicators.