Big Ideas

What happens out-of-sample to stock portfolios with weights derived from extreme in-sample fitting? In their February 2016 paper entitled “Stock Portfolio Design and Backtest Overfitting”, David Bailey, Jonathan Borwein and Marcos Lopez de Prado examine backtest overfitting in the context of designing a stock portfolio/fund. Their test approach is:

1. Construct split-adjusted, dividend-reinvested price series for all S&P 500 components as of January 22, 2016 with continuous monthly prices during 1991 through 2015 (277 stocks).
2. Select a target performance profile, including annualized return (6%, 8%, 10%, 12% 15% or 18%) and “shape” of return (principally, steady increase every month).
3. Apply an optimization program to determine the fixed stock price weights (0.1% increments) that achieve target performance profile in-sample during 1991-2005 (requiring monthly rebalancing of the portfolio to those weights).
4. Apply these stock price weights during 2006-2015 (again, with monthly portfolio rebalancing) to measure out-of-sample performance.

In initial tests, they allow negative weights (shorting). Because of the risks of shorting, they repeat analyses with a long-only constraint. They note that their in-sample fitting process considers “an inconceivably large set” of possible weights. They use the S&P 500 Total Return Index as a benchmark. Using adjusted monthly prices for the specified stocks from the end of December 1990 through the beginning of January 2016, they find that: Keep Reading

How can investors easily estimate the degradation from optimized in-sample Sharpe ratio to out-of-sample expected Sharpe ratio? In their February 2016 paper entitled “Noise Fit, Estimation Error and a Sharpe Information Criterion”, Dirk Paulsen and Jakob Sohl derive a simple correction for the upward bias in an optimized in-sample Sharpe ratio. The upward bias derives from fitting: (1) random noise within the backtest sample; and, (2) peculiarities in the backtest sample that make it less than perfectly representative of the entire (unknowable) series. In other words, even if no predictability exists, fitting noise “discovers” some. And, even if predictability exists, predictability within a backtest sample will likely be different from predictability in the entire series. Based on derivations addressing quantification of these two sources of bias, they conclude that: Keep Reading

How easy is overfitting of investment strategy parameters and how much does overfitting inflate expectations? In their February 2016 paper entitled “Backtest Overfitting in Financial Markets”, David Bailey, Jonathan Borwein, Marcos Lopez de Prado, Amir Salehipour and Qiji Zhu introduce two online backtest overfitting tools:

1. Backtest Overfitting Demonstration Tool – BODT simulates the overfitting of seasonal strategies (typical of technical analysis) to find the optimal strategy within a simulated sample of prices or actual S&P 500 Index levels by varying entry day, holding period, long or short, and stop-loss level. It runs a “movie” showing the progression of Sharpe ratio optimization. BODT then tests the optimal strategy on new (out-of-sample) data. It also provides a deflated in-sample Sharpe ratio based on the number of variations tested.
2. Tenure Maker Simulation Tool – TMST simulates the overfitting of econometric strategies (typical of academic journals) by varying forecasting equation parameters to maximize predictive power within a random (unpredictable) time series. It also runs a “movie” showing progression of Sharpe ratio optimization.

By overfitting, they mean repetitive use of an historical set of market data to identify the best of many variations of a strategy. Such optimality tends to target idiosyncrasies of the historical sample rather than any general market behavior. Their goals are to show how easy it is to overfit an investment strategy and how much overfitting may inflate investment performance expectations. Based on outputs of the two simulation tools, they conclude that: Keep Reading

Has the Moore’s Law-driven advance in financial information technology strengthened the hand of Murphy’s Law in markets? In the January 2016 version of his paper entitled “Moore’s Law vs. Murphy’s Law in the Financial System: Who’s Winning?” Andrew Lo reviews big unintended consequences of technology-leveraged finance including fire sales, flash crashes, botched initial public offerings (IPO), catastrophic algorithmic trading errors and access failures. He then discusses the counterbalancing roles of technology in elevating and suppressing financial system risk. Based on a survey of recent financial system breakdowns and his experience, he finds that: Keep Reading

Early in the first chapter of their 2015 book, Superforecasting: The Art and Science of Prediction, Philip Tetlock and Dan Gardner state: “…forecasting is not a ‘you have it or you don’t’ talent. It is a skill that can be cultivated. This book will show you how.” Based on the body of research on forecasting (with focus on Philip Tetlock’s long-term studies), they conclude that: Keep Reading

Wesley Gray, Jack Vogel and David Foulke preface their 2015 book, DIY Financial Advisor: A Simple Solution to Build and Protect Your Wealth, by stating that: “This book is a synopsis of our research findings developed while serving as a consultant and asset manager for large family offices. …Our book is meant to be an educational journey that slowly builds confidence in one’s own ability to manage a portfolio. In our book, we explore a potential solution that can be applicable to a wide variety of investors, from the ultra-high-net-worth to middle-class individual, all of whom are focused on similar goals  of preserving and growing their capital over time.” Based on their research, they conclude that: Keep Reading

What are pros and cons of extending the definition of financial index beyond conventional market capitalization (buy-and-hold) weighting? In the October 2015 draft of his paper entitled “What Is an Index?”, Andrew Lo proposes that any portfolio satisfying three properties should be considered an index: (1) transparent (public and verifiable); (2) investable (realistic and liquid benchmark); and, (3) entirely rules-based (allowing no judgment/discretion). He calls indexes that are not weighted by market capitalization dynamic indexes (requiring frequent rebalancing). He distinguishes between active investing and active risk management. He also addresses the elevated risk of snooping bias as dynamic indexes proliferate. Based on a broader perspective on indexes, he concludes that: Keep Reading

Robert Carver introduces his 2015 book, Systematic Trading: A Unique New Method for Designing Trading and Investing Systems, by stating that: “I don’t believe there is any magic system that will automatically make you huge profits, and you should be wary of anyone who says otherwise, especially if they want to sell it to you. Instead, success in systematic trading is mostly down to avoiding common mistakes such as over complicating your system, being too optimistic about likely returns, taking excessive risks, and trading too often. I will help you avoid these errors. This won’t guarantee returns, but it will make failure less likely. My framework…can be adapted to meet your needs. …Each element of the framework has been carefully designed… I’ll explain the available options, which I prefer, and why.” Based on his experience as a trader/portfolio manager and specific research, he concludes that: Keep Reading

What does a large online repository of research on financial markets say about community interactions? In the August 2015 version of his article entitled “Recent Trends in Empirical Finance”, Marcos Lopez de Prado measures trends in level of research activity, topical emphasis, level of interest as measured by downloads and level of collaboration. Based on data for 128,897 research papers by 72,070 authors posted on SSRN’s Financial Economics Network (as of June 4, 2015), he finds that: Keep Reading

How does growth in capitalization-weighted equity index investing affect the stock market? In the December 2014 update of their paper entitled “Indexing and Stock Price Efficiency”, Nan Qin and Vijay Singal examine the relationship between equity index investing (driven passively by liquidity trading and index changes, not actively by information) and stock price efficiency. They estimate equity index (passive) investing from holdings of 591 equity index mutual funds, enhanced index mutual funds, exchange-traded funds and closet indexers. They measure each stock’s passive (non-passive) ownership as the percentage of shares held by these funds (other funds) at the end of each quarter, with the lower bound of passive (non-passive) trading volume the absolute quarterly change in holdings of these (other) funds. They measure stock price efficiency by: (1) magnitude of post-earnings announcement drift (response to new information); and, (2) intraday and daily deviations of price from a random walk. Each quarter, they relate these measures of price inefficiency to level of index ownership across stocks. Using intraday and daily return, earnings announcement and quarterly fund holdings data for S&P 500 Index stocks and size/turnover-matched stocks during 2002 (post-decimalization) through 2013, they find that: Keep Reading