Brute Force Stock Trading Signal Discovery

How serious is the snooping bias (p-hacking) derived from brute force mining of stock trading strategy variations? In their August 2017 paper entitled “p-Hacking: Evidence from Two Million Trading Strategies”, Tarun Chordia, Amit Goyal and Alessio Saretto test a large number of hypothetical trading strategies to estimate an upper bound on the seriousness of p-hacking and to estimate the likelihood that a researcher can discover a truly abnormal trading strategy. Specifically, they:

• Collect historical data for 156 firm accounting and stock price/return variables as available for U.S. common stocks in the top 80% of NYSE market capitalizations with price over $3.
• Exhaustively construct about 2.1 million trading signals from these variables based on their levels, changes and certain combination ratios.
• Calculate three measures of trading signal effectiveness:
  1. Gross 6-factor alphas (controlling for market, size, book-to-market, profitability, investment and momentum) of value-weighted, annually reformed hedge portfolios that are long the value-weighted tenth, or decile, of stocks with the highest signal values and short the decile with the lowest.
  2. Linear regressions that test ability of the entire distribution of trading signals to explain future gross returns based on linear relationships.
  3. Gross Sharpe ratios of the hedge portfolios used for alpha calculations.
• Apply three multiple hypothesis testing methods that account for cross-correlations in signals and returns (family-wise error rate, false discovery rate and false discovery proportion.

They deem a signal effective if it survives both statistical hurdles (alpha t-statistic 3.79 and regression t-statistic 3.12) and has a monthly Sharpe ratio higher than that of the market (0.12). Using monthly values of the 156 specified input variables during 1972 through 2015, they find that:

(more…)