Why don’t machines rule the financial world? In his September 2017 presentation entitled “The 7 Reasons Most Machine Learning Funds Fail”, Marcos Lopez de Prado explores causes of the high failure rate of quantitative finance firms, particularly those employing machine learning. He then outlines fixes for those failure modes. Based on more than two decades of experience, he concludes that:
- To address the complexities of discovering a true investment strategy, developers should not work in isolation (a portfolio of individuals), but as specialists within assembly line teams. They should focus on expected recovery time for drawdowns of any size, because recovery time failures indicate a latent strategy weakness.
- Do not rely exclusively on asset returns for strategy development. Use a combination of returns (series that are stationary but retain no path memory) and prices (series that retain path memory but are non-stationary).
- Do not use time intervals in constructing samples. Instead, use intervals that measure information content such as number of trades, volume of trades, dollar value traded, order imbalance or entropy impact of trading.
- Do not use testing methods that ignore the path followed by prices, thereby missing conditions that would have stopped out positions (such as fund illiquidity, margin calls or risk intolerance). Include profit-taking limits, stop-loss limits and loss-of-patience (strategy expiration) limits.
- Do not assume that series of test observations are independent and identically distributed, because strategy parameters often rely on overlapping data. Instead, weight each observation as a function of absolute log returns uniquely attributable to it.
- Do not assume that training (in-sample) and testing (out-of-sample) data sets are separable by a simple date. Purge from the training set all observations whose strategy parameter measurements overlap with those in the testing set.
- Do not ignore backtest overfitting, which generates false positives (strategies that are lucky within the training data set). To mitigate, use a deflated Sharpe ratio (DSR) for backtest evaluation. In addition to return average and standard deviation, DSR considers non-normality of returns, length of the training sample, number of independent trials and intensity of data snooping.
In summary, investment strategy developers should be experts in the meaning of financial data and the effects of modeling assumptions on that meaning.
Cautions regarding conclusions include:
- The presentation is a statement of beliefs. The author does not prove that his recommendations are reliably sufficient for discovering persistently successful investment strategies.
- Implementing the recommendations implies costs to be debited from the performance of strategies so discovered.
Frequently Asked Questions
What is a deflated Sharpe ratio in quantitative investing?
The deflated Sharpe ratio (DSR) is a backtest evaluation metric that adjusts a strategy's Sharpe ratio for the risk of false positives from overfitting. Beyond the return average and standard deviation, it accounts for non-normality of returns, length of the training sample, number of independent trials and intensity of data snooping. Marcos Lopez de Prado recommends it to mitigate backtest overfitting.
Why do most machine learning investment funds fail?
Most machine learning investment funds fail, according to Marcos Lopez de Prado's September 2017 presentation, because of seven failure modes: developers working in isolation rather than in teams, ignoring drawdown recovery time, relying only on returns, sampling by fixed time intervals, using path-ignorant tests, assuming independent and identically distributed observations or separable training and testing sets, and ignoring backtest overfitting. His conclusions draw on more than two decades of experience. He does not prove that his fixes reliably produce persistently successful strategies.
How do you prevent overlapping data from contaminating a backtest's training and testing sets?
To prevent overlap contamination, Lopez de Prado recommends purging from the training set all observations whose strategy parameter measurements overlap with those in the testing set, rather than splitting by a simple date. He also advises weighting each observation as a function of the absolute log returns uniquely attributable to it, instead of assuming observations are independent and identically distributed. This addresses the fact that strategy parameters often rely on overlapping data.
Should quant strategies be built on returns or prices, and what about time bars versus volume bars?
Lopez de Prado recommends combining returns and prices rather than relying only on returns, because returns are stationary but retain no path memory while prices retain path memory but are non-stationary. He also advises against sampling by time intervals, favoring intervals that measure information content such as number of trades, volume of trades, dollar value traded, order imbalance or entropy impact of trading.
What are the limitations of Lopez de Prado's recommendations for avoiding machine learning fund failure?
Lopez de Prado's recommendations are a statement of beliefs rather than a proven framework, since he does not show they are reliably sufficient for discovering persistently successful investment strategies. Implementing them also carries costs that must be debited from the performance of any strategies discovered. The presentation is conceptual and not an empirical test.