Stopping Tests after Lucky Streaks?

Might purveyors of trading strategies be presenting performance results biased by stopping them when falsely successful? In other words, might they be choosing lucky closing conditions for reported positions? In the December 2018 revision of their paper entitled “p-Hacking and False Discovery in A/B Testing”, Ron Berman, Leonid Pekelis, Aisling Scott and Christophe Van den Bulte investigate whether online A/B experimenters bias results by stopping monitored commercial (marketing) experiments based on latest p-value. They hypothesize that such a practice may exist due to: (1) poor training in statistics; (2) self-deception motivated by desire for success; or, (3) deliberate deception for selling purposes. They employ regression discontinuity analysis to estimate whether reaching a particular p-value causes experimenters to end their tests. Using data from 2,101 online A/B experiments with daily tracking of results during 2014, they find that:

(more…)