Are the backtests provided for alternative beta investment products representative of their future live performance? In their March 2016 paper entitled “Quantifying Backtest Overfitting in Alternative Beta Strategies”, Antti Suhonen, Matthias Lennkh and Fabrice Perez compare the backtested and live performances of alternative beta products offered by investment banks. The strategies underlying these products are formulaic and non-discretionary, designed to extract risk factor/style premiums (such as value, momentum, carry or term) or exploit some financial market anomaly (such as turn-of-the-month or mean reversion). Specifically, they:
- Present an overview of alternative beta products offered by investment banks.
- Compare backtested and live performance data for these products.
- Compare backtested and live factor exposures four four strategy families (equity value, equity volatility, fixed income term and currency exchange carry).
Using daily returns in U.S. dollars of 215 alternative beta strategies across five asset classes and 11 strategy types offered by 15 investment banks as available during January 1990 through early March 2015, they find that:
- The average length of backtest (live) sample periods is 10.7 (4.6) years. During 2005-2009 (2010-2014), 96 (125) strategies go live.
- The median annualized Sharpe ratio for live products (0.31) is 73% lower than that of associated backtests (1.20). (See the first chart below.)
- 18 of 215 products have live Sharpe ratios at least as large as their backtested Sharpe ratios.
- 65 of 215 products have a negative live Sharpe ratio.
- While 50% or higher for all asset classes, the magnitude of backtested-to-live Sharpe ratio deterioration varies (see the second chart below). Deterioration is:
- Most pronounced in currencies as an asset class and among event-driven, value and carry strategy types.
- Least pronounced among multi-asset and fixed income classes and volatility and trend-following strategy types.
- About 30% stronger for the most complex strategies (many trading rules, filters and parameters) than the simplest ones.
- Regarding exposures to targeted factors during backtested and live samples for four strategy types:
- All four strategy types have significantly positive targeted alphas in backtests, but these alphas fall by half in live samples.
- Live targeted alphas are significantly positive for equity volatility, interest rate term and currency exchange carry strategy types.
- Live targeted alpha for the equity value strategy type is not significant.
- More recent products appear to be better at retaining targeted factor exposures in live samples than older products.
The following chart, taken from the paper, illustrates the difference between backtested performance and live performance for one alternative beta product that goes live in October 2010. This strategy is representative of the median of all 215 products, with a backtested (live) Sharpe ratio 1.42 (0.36) for a deterioration of 75%.
The next chart, also from the paper, summarizes median alternative beta product Sharpe ratio deterioration (backtested Sharpe ratio minus live Sharpe ratio, divided by backtested Sharpe ratio) overall and by asset class for:
- Full sample: complete available backtested and live periods.
- 3 year pre/post: the last three years of backtest performances and the first three years of live performances.
- 1 year pre/post: the last year of backtest performances and the first year of live performances.
The magnitude of deterioration varies somewhat across asset classes and measurement intervals, but is at least 50% for all asset classes using full samples. Based on these results, the industry practice of derating backtested Sharpe ratios by 50% is not conservative enough.
In summary, evidence from live performance data supports belief that that alternative beta products generally suffer from backtest overfitting and unreliable factor/anomaly exposure.
Cautions regarding findings include:
- More rigorous controls for data snooping may close the gap between backtested and live performance. See “Navigating the Data Snooping Icebergs”, “Snooping Bias Accounting Tools” and “Taming the Factor Zoo?”.
- Backtested-to-live performance deterioration may derive from market adaptation instead of/in addition to data/model snooping.
Frequently Asked Questions
What are alternative beta products in investing?
Alternative beta products are investment bank offerings built on formulaic, non-discretionary strategies that aim to capture risk factor or style premiums (such as value, momentum, carry or term) or exploit market anomalies (such as turn-of-the-month or mean reversion). A study of 215 such strategies across five asset classes and 11 strategy types from 15 investment banks examined how they perform.
Do alternative beta backtests accurately predict live performance?
Alternative beta backtests do not accurately predict live performance. Across 215 strategies from 15 investment banks, the median live Sharpe ratio (0.31) was 73% lower than the median backtested Sharpe ratio (1.20), only 18 products matched or beat their backtests, and 65 had negative live Sharpe ratios.
How is backtest overfitting measured in alternative beta strategies?
Backtest overfitting in alternative beta strategies was measured as Sharpe ratio deterioration: backtested Sharpe ratio minus live Sharpe ratio, divided by backtested Sharpe ratio. The study used daily U.S. dollar returns from January 1990 to early March 2015, with average backtest and live periods of 10.7 and 4.6 years, and compared full samples, the last/first 3 years, and the last/first 1 year around launch. It also compared backtested and live targeted factor alphas for four strategy families.
Is the industry practice of cutting backtested Sharpe ratios by 50% enough to estimate real-world results?
A 50% haircut to backtested Sharpe ratios is not conservative enough based on the live data for alternative beta products. Full-sample deterioration was at least 50% for every asset class, and the median was 73%. Deterioration was most pronounced in currencies and in event-driven, value and carry strategies, and least pronounced in multi-asset and fixed income classes and volatility and trend-following strategies.
What are the caveats in attributing alternative beta underperformance to overfitting?
The gap between backtested and live alternative beta performance may not be due solely to overfitting. More rigorous controls for data snooping might narrow the gap, and the deterioration may also reflect market adaptation instead of or in addition to data/model snooping. Complexity also matters: deterioration was about 30% stronger for the most complex strategies than for the simplest ones.