Should investors worry that investment strategies available in the marketplace may derive from optimization via intensive backtesting? In the September 2013 update of their paper entitled “Backtest Overfitting and Out-of-Sample Performance”, David Bailey, Jonathan Borwein, Marcos Lopez de Prado and Qiji Zhu examine the implications of overfitting investment strategies via multiple backtest trials. Using Sharpe ratio as the measure of strategy attractiveness, they compute the minimum backtest sample length an investor should require based on the number of strategy configurations tried. They also investigate situations for which more backtesting may produce worse out-of-sample performance. Based on interpretations of mathematical derivations, they conclude that:
- Researchers can generate high in-sample Sharpe ratios by backtesting a relatively small number of alternative strategy configurations. However, the more configurations tried, the greater the probability of overfitting (data snooping bias). And, the more complex the strategy (more parameters), the more configurations a researcher can generate.
- A given sample has limited backtest “carrying capacity” (see the chart below). For example, testing more than 45 strategy configurations on a five-year sample is almost guaranteed to identify configurations with an in-sample annualized Sharpe ratio of one, but an expected out-of-sample annualized Sharpe ratio of zero.
- The more trials a researcher runs, the higher the in-sample Sharpe ratio an investor should demand.
- It is not possible to estimate the likelihood of overfitting for a strategy when the researcher does not report the number of alternatives tried in developing it.
- When a data series exhibits memory, there is a point at which further in-sample optimization via strategy backtesting actually worsens out-of-sample performance. Examples of memory in data are mean reversion and serial correlation (autocorrelation). A researcher can introduce memory into a data series via periodic renormalization.
The following chart, taken from the paper, depicts the relationship between number of strategy alternatives considered (N) and the minimum backtest length needed to prevent skill-less strategies (those with out-of-sample annualized Sharpe ratio zero) from generating an in-sample annualized Sharpe ratio of one.

In summary, investors should understand that generating high in-sample Sharpe ratios via iterative backtesting of strategy alternatives is easy to do, often difficult to detect and potentially harmful for out-of-sample performance.
Cautions regarding conclusions include:
- Tracking all strategy alternatives considered requires considerable discipline and is not clearly in the interest of strategy sellers.
- It is also difficult to detect derived overfitting, whereby research done by others (who do not document the strategy alternatives they considered) stimulates selection of a strategy for further testing. In other words, imprecise communications among researchers makes the number of models tested against a given data set essentially impossible to know.
- Assumptions (including use of Sharpe ratio) and derivations generally assume tame variable distributions. To the extent that these distributions are wild, statistical interpretations break down.
See also “Stock Return Model Snooping” and “Taming the Factor Zoo?”.
Frequently Asked Questions
What is backtest overfitting in investment strategy development?
Backtest overfitting occurs when a researcher tries many strategy configurations on the same historical data and selects the one with the best in-sample results, which reflects data snooping bias rather than skill. The resulting strategy can show a high in-sample Sharpe ratio while having a much lower expected out-of-sample Sharpe ratio. The more configurations tried, and the more parameters a strategy has, the greater the probability of overfitting.
Can backtesting many strategy variations produce a high Sharpe ratio even if the strategy has no skill?
Yes, intensive backtesting can produce a high in-sample Sharpe ratio for a skill-less strategy. Bailey et al. show that testing more than 45 strategy configurations on a five-year sample is almost guaranteed to find one with an in-sample annualized Sharpe ratio of one but an expected out-of-sample annualized Sharpe ratio of zero. This is a theoretical result based on mathematical derivations.
How do you determine the minimum backtest length needed to avoid overfitting?
The minimum backtest length is calculated from the number of strategy configurations (N) tried, using the Sharpe ratio as the performance measure. Bailey et al. derive the shortest sample for which a skill-less strategy (out-of-sample annualized Sharpe ratio of zero) is unlikely to produce an in-sample annualized Sharpe ratio of one. The more trials a researcher runs, the longer the required sample and the higher the in-sample Sharpe ratio an investor should demand.
Can more in-sample optimization make out-of-sample performance worse, and does it depend on the data?
Yes, more in-sample optimization can worsen out-of-sample performance when the data series exhibits memory, such as mean reversion or serial correlation (autocorrelation). In that case, there is a point beyond which further backtest optimization hurts rather than helps. A researcher can also introduce memory into a series through periodic renormalization.
What are the limitations of using the number of backtest trials to detect overfitting?
The likelihood of overfitting cannot be estimated when the researcher does not report how many alternatives were tried, and tracking all of them takes discipline that may not serve strategy sellers' interests. Derived overfitting, where others' undocumented research prompts selection of a strategy for further testing, makes the total number of models tested on a data set essentially impossible to know. The derivations, including the use of the Sharpe ratio, also assume tame variable distributions, and statistical interpretations break down to the extent distributions are wild.