Chapter 3: Avoiding or Mitigating Snooping Bias
December 6, 2013
Snooping bias, also called mining bias and more loosely benefit of hindsight, is a notorious artificial booster of backtest performance. It takes multiple forms:
- Picking the best of many rules/indicators (strategies, models) for a given data sample
- Optimizing rule parameters for a given data sample
- Restricting a data sample to find favorable performance of a given rule
- Running an investment contest among many individuals
A sentiment shared among researchers in stochastic fields is: “If you torture the data long enough, it will confess to anything.” Because returns are noisy (substantially random), trying many combinations of rules, parameter settings and data samples will generate strategies that outperform benchmarks by extreme good luck. A prosecutor (an investor) satisfied with false confessions is likely to lose in court (the market).
To illustrate, Figure 3-0 depicts the net cumulative values of $1.00 initial investments in each of 12 variations of the simple asset class momentum strategy introduced in Figure 1-1. This strategy shifts each month to the one of nine asset class proxies with the highest total return over a past return measurement (ranking) interval. Most of the proxies are exchange-traded funds (ETF). The 12 variations differ by the length of the ranking interval, from one to 12 months. All variations impose a switching friction of 0.25% whenever the strategy switches funds.
Does the top-performing variation (dotted line) represent a premium earned by extracting truly valuable information from market prices, or just the payout from being the lucky winner of a lottery? The following sections address this question.
Figure 3-0: Performance of 12 Asset Class Momentum Strategy Variations
Chapter 2: Making the Strategy Logical
November 29, 2013
Making an investment/trading strategy logical essentially means making it testable and implementable, with inputs, outputs and rules clearly defined, properly sequenced and inclusive of all material factors. Clearly defined inputs, outputs and rules enable verification and extension. Definitions that require subjective interpretation are not clear. Properly sequenced inputs, outputs and rules fit the real world, representing an analysis and implementation scenario available to an investor in real time. Some strategies are more forgiving of tight sequencing than others. Including all material factors means accounting for all significant contributions to (capital gains, dividends, interest) and debits from (costs of data, trading frictions, cost of shorting, cost of leverage) investment outcome. The materiality of factors varies with strategy specifics.
How can investors make sure their strategies are logical? (more…)
Chapter 1: Some Statistical Practices that Make Sense
November 22, 2013
Financial systems, such as stock markets, involve a large number of interacting decisions based on many different time-varying levels of knowledge, processing capabilities, motivations and financial resources. Due to this complexity, theories of financial system behavior cannot determine future prices and returns. Said differently, the models termed “financial theories” are actually just working hypotheses generally formed retrospectively (empirically) to fit the past.
Lack of solid theories leaves researchers to explore a jungle of empirical data via statistical inference, constructing samples and looking for past conditions (indicators) that relate strongly to future outcomes (returns) within those samples. Investors then make the leap (despite limitations in empirical research and changes in the market conditions) that future data is enough like past data to apply findings from such inferences to investment decisions.
How should investors generate and interpret research findings in such an environment? (more…)
Avoiding Investment Strategy Flame-outs
November 16, 2013
”…markets always eventually outwit us. Even if markets are not strictly random, their vagaries are too rich to capture in a few sentences or equations. So die the dreams of financial theories. Only imperfect models remain. …Given that finance’s best tools are shaky models, the best strategy is to use models as little as possible, and to replicate making as little assumptions as you can. …Every financial axiom is pretty much wrong; the practical question is: how wrong, and can you still make use of it?”
”The world of markets doesn’t exactly match the ideal circumstances a model assumes, but a robust model allows a savvy user to qualitatively adjust for those mismatches. A user should know what has been assumed when he uses the model, and he should know exactly what has been swept out of view.”
”Financial modelers must therefore compromise, must firmly decide what small part of the financial world is of greatest current interest, decide on its key features, and make a mock-up of only those. …A successful financial model must have limited scope; you must work with simple analogies…”
- Emanuel Derman in “Metaphors, Models & Theories”
Why do investment/trading strategies that test well on historical data flame out when put to actual use? Are there steps investors can take to improve the odds that strategies they develop will perform as tested? This book draws upon reviews of hundreds of academic and practitioner studies that seek to predict asset prices and exploit the predictions. It focuses on widespread weaknesses and limitations in these studies to help investors: (1) avoid or mitigate the weaknesses in developing their own strategies; and, (2) perform due diligence on strategies offered by others. (more…)
October 14, 2013
Weekly Email Summary
August 8, 2013
To receive a weekly summary of our research findings, please add your name and preferred email address to the form below. Upon signing up, you’ll automatically be added to our list to receive the email. You may unsubscribe from receiving the email at anytime (this does not affect your CXO subscription). Have questions? Let us know.
test — rss-subscriber – weekly
August 7, 2013
July 6, 2013
May 29, 2013
May 16, 2013
The inflation rate may be the most fundamental determinant of the discount rate used to calculate the present value of an investment. Changes in the inflation rate therefore affect stock market valuation. What is the best way to forecast the inflation rate? How reliable is inflation forecasting? The following discussion provides the CXO Advisory Group LLC forecasts for U.S. total and core (excluding food and energy) inflation rates, along with the method for constructing the forecast and the rationale for the methodology. (more…)