Evidence-based investing research
Value Investing Strategy (Strategy Overview)
Allocations for November 2026 (Final)
Cash TLT LQD SPY
Momentum Investing Strategy (Strategy Overview)
Allocations for November 2026 (Final)
1st ETF 2nd ETF 3rd ETF

The Significance of Statistical Significance?

Steve LeCompte | | Posted in: Big Ideas

Key Insight

Kim and Ji argue conventional significance thresholds overstate statistical significance with massive samples, finding only 32% of published finance studies survive Bayesian analysis (161 studies from top-tier journals; frequentist vs Bayesian comparison).

How should investors interpret findings of statistical significance in academic studies of financial markets? In the March 2014 draft of their paper entitled “Significance Testing in Empirical Finance: A Critical Review and Assessment”, Jae Kim and Philip Ji review significance testing in recent research on financial markets. They focus on interplay of two types of significance: (1) the probability of a Type I error (the probability of rejecting a true null hypothesis), with significance threshold usually set at 1%, 5% (most used) or 10%; and, (2) the probability of a Type II error (the probability of accepting a false null hypothesis). They consider the losses associated with the significance threshold, and they assess the Bayesian method of inference as an alternative to the more widely used frequentist method associated with conventional significance testing. Based on review of past criticisms of conventional significance testing and 161 studies applying linear regression recently published in four top-tier finance journals, they conclude that:

  • Use of massive samples, insensitively to a balance between Type I and Type II errors, tends to overstate statistical significance.
    • When sample size is massive (small), researchers should tighten (relax) the threshold for statistical significance. For example, researchers should use 0.5% or 0.1% thresholds for massive samples rather than the exclusively used 1%, 5% and 10% thresholds. One way to optimize the significance threshold is through explicit balancing of the probabilities of Type I and Type II errors. Another way is to minimize the potential economic loss driven by the significance threshold.
    • Alternatively, researchers could use the Bayesian method of inference, effectively tightening the significance threshold as sample size grows. Only 32% of published studies (all frequentist) offer strong evidence of an effect/anomaly under Bayesian analysis. For the subset of these studies with sample size under 1,000, 61% survive Bayesian analysis.
  • Leading financial journals appear to be biased in favor of studies that demonstrate conventional levels of statistical significance. Such biased feedback may drive researchers toward massive samples that make finding conventional significance easier.
  • It appears that replication of published studies is problematic. Out of email requests to 50 authors: four responded with data and elaboration to facilitate replication; 12 declined to share their data (for reasons such as copyright and confidentiality); and 34 have not responded.
  • Simple visual representations of the data sometimes suggest predictions that outperform those based on the R-squared statistic or the t-statistic.

In summary, significance testing methods exhibited in formal (published) finance research are conducive to spurious statistical significance and economically incorrect decisions.

Cautions regarding conclusions include:

Frequently Asked Questions

What is statistical significance and what are Type I and Type II errors in finance research?

Statistical significance in finance research is judged by the probability of a Type I error (rejecting a true null hypothesis), with thresholds usually set at 1%, 5% (most used) or 10%. A Type II error is the opposite mistake: accepting a false null hypothesis. Kim and Ji examine how these two error types interact when studies test for market effects or anomalies.

Do published finance studies with statistically significant results hold up under Bayesian analysis?

Most published finance studies with conventional statistical significance do not hold up under Bayesian analysis: only 32% of the 161 reviewed studies from four top-tier finance journals offer strong evidence of an effect or anomaly. Among the subset with sample sizes under 1,000, 61% survive. Massive samples tend to overstate significance when the threshold is not adjusted.

How can researchers choose a better significance threshold when sample sizes are very large?

Researchers can optimize the significance threshold by explicitly balancing the probabilities of Type I and Type II errors, or by minimizing the potential economic loss driven by the threshold. They should tighten the threshold for massive samples, such as 0.5% or 0.1% rather than the conventional 1%, 5% or 10%, and relax it for small samples. Bayesian inference is an alternative that effectively tightens the threshold as sample size grows.

How does Bayesian inference compare to conventional frequentist significance testing?

Bayesian inference differs from conventional frequentist significance testing in that it effectively tightens the significance threshold as sample size grows, whereas frequentist testing uses fixed thresholds of 1%, 5% or 10%. In Kim and Ji's review of 161 studies, all of which were frequentist, only 32% showed strong evidence under Bayesian analysis. This gap suggests frequentist testing with massive samples tends to overstate significance.

What are the limitations of the critique of significance testing in empirical finance?

Kim and Ji's critique does not address data snooping bias and its implications for significance testing, nor does it address the failure of many formal studies to account for implementation frictions, which are essential for assessing economic significance. Their findings also rest on 161 linear regression studies from four top-tier finance journals. Replication checks were limited as well: of 50 authors emailed, only four provided data, 12 declined and 34 did not respond.

Guru Grades

1,206 predictions by 42 market gurus since 2022: 41% right. Always saying stocks will go higher: 62%.

As of the Nov 19, 2026 close

  1. 1 Ed Yardeni A 71%
  2. Always bullish B 62%
  3. 2 Jim Paulsen B 61%
  4. 3 Sam Stovall B 58%
  5. 4 Adam Turnquist B 58%
  6. 5 Ryan Detrick B 56%

Full leaderboard ›