We occasionally get requests from readers to review the claims stated by an online investing/trading advisory service or investment manager. Most such web sites do not provide enough information to perform a quantitative review. Here is compilation of key points to consider in evaluating the claims of advisory services:
Read the general disclaimer (often linked at the bottom of site pages). Does it say something like: “It should not be assumed that the trading recommendations or advice will be profitable or that they will not result in losses.” Do you believe the marketers or the lawyers?
Find the assumptions made about claimed returns:
Are claimed returns cherry-picked or do they comprise steady outperformance over a reasonably long period of time? (There is often not enough information to tell.)
Does past performance commence with a strong return burst (perhaps based on backtesting) and then moderate (perhaps in real trading)? Such a scenario may indicate picking a start date at the beginning of a non-repeating lucky streak. At the end of the streak, the luck is gone.
If the offeror manages multiple services, are they all outperforming, or just the one being promoted? The latter case suggests the possibility of luck rather than skill in the promoted fund.
Are claimed returns from real trading or from a hypothetical backtest? Look for disclaiming statements such as: “Hypothetical performance results have many inherent limitations… One of the limitations is that they are generally prepared with the benefit of hindsight.” Exhaustive backtesting (data mining) likely discovers non-repeating lucky streaks rather than reliable patterns. Moreover, hypothetical trade timing may assume favorable exit points that a real-life trader is very unlikely to achieve systematically.
What are the assumptions about trading frictions? Look for disclaiming statements such as: “Performance reflects gross profit or loss and is exclusive of commissions, trading fees and subscription costs.” Assumptions about trading frictions (including about buy and sell prices with respect to the bid-ask spread) can have very large impacts on profitability for strategies that involve frequent trading and/or options. Also, in general, the less money you have to invest, the more trading frictions depress returns.
Are the claimed returns clearly tied to an amount of capital required to implement them, or are capital requirements hazy? If the frequency of an offeror’s recommended trades is sometimes so slow that assumed position sizes leave you with cash, or so fast that you do not have the cash to exploit them all, then your portfolio returns may be substantially lower than the offeror’s per-trade statistics.
Are the claimed returns for closed positions only, suggesting the possibility of a bow wave of bad trades kept open and excluded from analysis.
How do the offeror’s claims square with your beliefs about the level of financial markets efficiency and about human nature? Specifically:
Why have market makers, hedge fund managers and other sophisticated investors not discovered and extinguished the related market inefficiency?
If the offeror can reliably generate very large returns, why is he bothering to sell and administer an advisory service?
Is there credible research that supports the reliable outperformance of the type of strategy employed by the offeror?
Is there credible research that supports the existence, in general, of reliable returns of the magnitude claimed? In other words, does it seem too good to be true?
In summary, this compilation of suggestions and questions can help investors/traders assess the credibility of returns claimed by advisory services.
Frequently Asked Questions
What is an investing or trading advisory service?
An investing or trading advisory service is an online offering or investment manager that provides trading recommendations or advice, often supported by claims of past returns. Because most such sites do not provide enough information for a quantitative review, readers must judge the credibility of the claimed returns using the service's disclaimers, return assumptions, and consistency with market efficiency.
Can you trust the performance claims made by trading advisory services?
Performance claims by trading advisory services should be treated with skepticism unless they can be verified, because most sites do not provide enough information for a quantitative review. Claims may reflect cherry-picked start dates, hypothetical backtests, or gross returns that exclude costs. Claims that seem too good to be true should prompt questions about why sophisticated investors have not already eliminated the supposed inefficiency.
How do you evaluate the claimed returns of an investment advisory service?
Evaluating an advisory service's claimed returns starts with reading the general disclaimer and then examining the assumptions behind the numbers. Check whether returns come from real trading or a hypothetical backtest, whether they are cherry-picked or show steady outperformance over a long period, and whether other services run by the same offeror also outperform. Also check assumptions about commissions, fees and bid-ask spreads, the capital required, and whether returns include only closed positions.
What is the difference between backtested results and real trading performance?
Backtested (hypothetical) results differ from real trading performance because they are prepared with the benefit of hindsight and may assume favorable exit points that a real trader is very unlikely to achieve systematically. Exhaustive backtesting (data mining) likely finds non-repeating lucky streaks rather than reliable patterns. A track record that begins with a strong return burst and then moderates may indicate backtested results followed by real trading, or a start date chosen at the beginning of a lucky streak.
Why do trading frictions and capital requirements make advisory service returns hard to replicate?
Trading frictions and capital requirements make advisory service returns hard to replicate because claimed returns often reflect gross profit before commissions, trading fees, and subscription costs. Assumptions about bid-ask spreads can have very large effects on strategies involving frequent trading or options, and the less money an investor has, the more frictions depress returns. If recommended trade frequency leaves an investor holding cash or lacking cash to follow every trade, portfolio returns may be substantially lower than the offeror's per-trade statistics.