Big Ideas

How should investors assess systematic trading programs? In his August 2014 paper entitled “Evaluation of Systematic Trading Programs”, Mikhail Munenzon offers a non-technical overview of issues involved  in evaluating systematic trading programs. He defines such programs as automated processes that generate signals, manage positions and execute orders for exchange-listed instruments or spot currency rates with little or no human intervention. He states that the topics he covers are not exhaustive but should be sufficient for an investor to initiate successful relationships with systematic trading managers. Based on his years of experience as a systematic trader and as a large institutional investor who has evaluated many diverse systematic trading managers on a global scale, he concludes that: Keep Reading

How can researchers account for the snooping bias derived from testing of multiple strategy alternatives on the same set of data? In the July 2014 version of their paper entitled “Evaluating Trading Strategies”, Campbell Harvey and Yan Liu describe tools that adjust strategy evaluation for multiple testing. They note that conventional thresholds for statistical significance assume an independent (single) test. Applying these same thresholds to multiple testing scenarios induces many false discoveries of “good” trading strategies. Evaluation of multiple tests requires making significance thresholds more stringent. In effect, such adjustments mean demanding higher Sharpe ratios or, alternatively, applying “haircuts” to computed strategy Sharpe ratios according to the number of strategies tried. They consider two approaches: one that aggressively excludes false discoveries, and another that scales avoidance of false discoveries with the number of strategy alternatives tested. Using mathematical derivations and examples, they conclude that:

Keep Reading

Are widely used volatility-adjusted investment performance metrics, such as Sharpe ratio, robust to different measurement intervals? In the July 2014 version of their paper entitled “The Divergence of High- and Low-Frequency Estimation: Implications for Performance Measurement”, William Kinlaw, Mark Kritzman and David Turkington examine the sensitivity of such metrics to the length of the return interval used to measure it. They consider hedge fund performance, conventionally estimated as Sharpe ratio calculated from monthly returns and annualized by multiplying by the square root of 12. They also consider mutual fund performance, usually evaluated as excess return divided by excess volatility relative to an appropriate benchmark (information ratio). Finally, they consider Sharpe ratios of risk parity strategies, which periodically rebalance portfolio asset weights according to the inverse of their return standard deviations. Using monthly and longer-interval return data over available sample periods for each case, they find that: Keep Reading

Is there some tractable investment performance metric that corrects weaknesses commonly encountered in financial markets research? In the July 2014 version of their paper entitled “The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality”, David Bailey and Marcos Lopez de Prado introduce the Deflated Sharpe Ratio (DSR) as a tool for evaluating investment performance that accounts for both non-normality and data snooping bias. They preface DSR development by noting that:

• Many investors use performance statistics, such as Sharpe ratio, that assume test sample returns have a normal distribution.
• Fueled by high levels of randomness in liquid markets, testing of a sufficient number of strategies on the same data essentially guarantees discovery of an apparently profitable, but really just lucky, strategy.
• The in-sample/out-of-sample hold-out approach does not eliminate data snooping bias when multiple strategies are tested against the same hold-out data.
• Researchers generally publish “successes” as isolated analyses, ignoring all the failures encountered along the road to statistical significance.

The authors then transform Sharpe ratio into DSR by incorporating sample return distribution skewness and kurtosis and by correcting for the bias associated with the number of strategies tested in arriving at the “winning” strategy. Based on mathematical derivations and an example, they conclude that:

Keep Reading

Is there a practical way to mitigating data snooping bias while exploring optimal parameter values? In his February 2014 paper entitled “Know Your System! – Turning Data Mining from Bias to Benefit through System Parameter Permutation” (the National Association of Active Investment Managers’ 2014 Wagner Award winner), Dave Walton outlines System Parameter Permutation (SPP) as an alternative to traditional in-sample/out-of-sample cross-validation and other more complex approaches to compensating for data snooping bias. SPP generates distributions of performance metrics for rules-based investment strategies by systematically collecting outcomes across plausible ranges of rule parameters (see the figure below). These distributions capture typical, optimal and worst-case outcomes. He explains how to apply SPP to estimate both long-run and short-run strategy performance and to test statistical significance. He provides an example that compares conventional in-sample/out-of-sample testing and SPP as applied to an asset rotation strategy based on relative momentum. Using logical arguments and examples, he concludes that: Keep Reading

How should investors interpret findings of statistical significance in academic studies of financial markets? In the March 2014 draft of their paper entitled “Significance Testing in Empirical Finance: A Critical Review and Assessment”, Jae Kim and Philip Ji review significance testing in recent research on financial markets. They focus on interplay of two types of significance: (1) the probability of a Type I error (the probability of rejecting a true null hypothesis), with significance threshold usually set at 1%, 5% (most used) or 10%; and, (2) the probability of a Type II error (the probability of accepting a false null hypothesis). They consider the losses associated with the significance threshold, and they assess the Bayesian method of inference as an alternative to the more widely used frequentist method associated with conventional significance testing. Based on review of past criticisms of conventional significance testing and 161 studies applying linear regression recently published in four top-tier finance journals, they conclude that: Keep Reading

A subscriber flagged an apparently very attractive exchange-traded fund (ETF) momentum-volatility-correlation strategy that, as presented, generates a optimal compound annual growth rate of 45.7% with modest maximum drawdown. The strategy chooses from among the following seven ETFs:

ProShares Ultra S&P500 (SSO)
SPDR EURO STOXX 50 (FEZ)
iShares MSCI Emerging Markets (EEM)
iShares Latin America 40 (ILF)
iShares MSCI Pacific ex-Japan (EPP)
Vanguard Extended Duration Treasuries Index ETF (EDV)
iShares 1-3 Year Treasury Bond (SHY)

The steps in the strategy are, at the end of each month:

1. For the first six of the above ETFs, compute log returns over the last three months and standard deviation (volatility) of log returns over the past six months.
2. Standardize these the monthly sets of past log returns and volatilities based on their respective means and standard deviations.
3. Rank the six ETFs according to the sum of 0.75 times standardized past log return plus 0.25 times past standardized volatility.
4. Buy the top-ranked ETF for the next month.
5. However, if at the end of any month, the correlation of SSO and EDV monthly log returns over the past four months is greater than 0.75, instead buy SHY for the next month.

The developer describes this strategy as an adaptation of someone else’s strategy and notes that he has tested a number of systems. How material might the implied secondary and primary data snooping bias be in the performance of this system? To investigate, we examine the fragility of performance statistics to variation of each strategy parameter separately. As presented, the author substitutes other ETFs for the two with the shortest histories to extend the test period backward in time. We use only price histories as available for the specified ETFs, limited by EDV with inception January 2008. Using monthly adjusted closing prices for the above seven ETFs and for SPDR S&P 500 (SPY) during January 2008 through February 2014 (74 months), we find that: Keep Reading

Here is this Friday’s installment of Avoiding Investment Strategy Flame-outs, a short book we are previewing for subscribers. With this post, the book preview is complete.

Chapter 9: “Getting Expert Advice (Delegating Strategy Development)”

“Section 8-2 examines in detail the attractiveness of a short-term trading strategy offered in the quasi-advisory (“educational”) marketplace. Assessing this strategy entails considerable work, only to find that it is not attractive. This chapter covers more broadly the delegation of investment strategy development, ranging from following an expert’s public advice on market timing to deposit of funds for professional management. Such practices relieve investors (at a cost) of some or all of the burdens of learning, data collection/analysis, strategy design and disciplined implementation.

“However, such delegation entails agency issues (conflicts of interest). Potentially more than they want to help their readers/subscribers/clients earn exceptional investment returns:

• Media that present investing advice want subscription fees or attention to advertisements. Media company interest in the usefulness of what they present is arguably secondary to attracting attention. In general, contributors to free media also have motives that bias what they present (attracting their own subscribers or clients).
• Academics studying financial markets want employment (and tenure) and funding of future research. They therefore must attract the attention of peers and publishers. They often have no stake in whether their research findings are useful to investors. They do have an incentive to attract the attention of investors when making a transition to investment management.
• Expert equity analysts want employment by brokers and asset managers, and access to industry sources. The interests of their bosses may not always coincide with the interests of the clients of their bosses or other investors.
• Newsletter sellers want subscription fees. Getting the attention of potential subscribers is essential to their business model. They sometimes seek attention by uncritically presenting snooped, gross trading system results as an “educational” service.
• Financial advisors want advisory fees. They must attract the attention of potential clients. As with newsletter sellers, the font used for marketing copy is much larger than that used for the legal disclaimer.
• Investment managers, mutual fund managers and hedge fund managers want management fees, normally as a percentage of account balance. They have to get the account before they can debit the balance. They have to get the attention of a potential clients before they get the account.

“A common motive across the range of investment service providers is attention-seeking, which tends to drive offerors toward extreme representations (possible but low-probability scenarios, the tails of the distribution of potential outcomes). The most extreme representations offer the “holy grail” of amazingly large and reliable returns (appealing to investor greed) or the “safety of Noah’s ark” from impending doom (appealing to investor fear).

“Conflict-of-interest materiality persists because investors have great difficulty distinguishing luck from skill when outcomes involve a high degree of randomness.”

Here is this Friday’s installment of Avoiding Investment Strategy Flame-outs, a short book we are previewing for subscribers. Chapter previews will continue for one more Friday.

Chapter 8: “Two Analysis Regimes”

“This chapter steps through two analysis regimes via examples to illustrate avoidance and mitigation of the issues covered in Chapters 1 through 6. The first example involves a widely used technical indicator, the 10-month simple moving average, but with an investigation of whether there is more information in the average than conventionally extracted. The second example constructs in detail a portfolio-level view of a short-term trading strategy offered in the quasi-advisory (“educational”) marketplace. The purpose of the examples is to illustrate different ways that most investors can use to analyze investment strategies.

“The analysis tool is Microsoft Excel. Some or all of the steps in the examples may be useful in analyzing other potentially useful asset return indicators.”

Here is this Friday’s installment of Avoiding Investment Strategy Flame-outs, a short book we are previewing for subscribers. Chapter previews will continue for the next two Fridays.

Chapter 7: “Thinking about Taxes”

“To the extent that governments tax different kinds of income/investment returns (interest, dividends, long-term capital gains and short-term capital gains) differently, taxes may be decisive for some individuals in designing a strategy or selecting one strategy over another.

“Taxes are more personal than other investment frictions. Relevant questions include:

1. What is the investor’s expected marginal tax rate?
2. Does the investor have any capital losses carried forward from prior years that may offset future gains?
3. Are the funds in a tax-advantaged account, such as (in the U.S.): a conventional Individual Retirement Account (IRA), subject to tax rates at the time of withdrawal (whatever they may be); or, a Roth IRA, subject to no taxes at withdrawal (as the rules stand now)?

“An obvious risk to long-term strategies including assumptions about taxes is that governments may change the rules at any time.

“The next two sections explore how incorporating tax avoidance into an investment strategy might impact returns.”