What does Backtest Auditor check?

Seven ways a backtest misleads you: lookahead bias, survivorship bias, ticker reuse, unrealistic costs, overfitting and multiple testing, sub-period instability, and concentration.

Details

1. Lookahead bias: using data that wasn't available when the trade would have been made, such as a day's closing price to trade at that day's open. See lookahead bias.
2. Survivorship bias: testing only on companies that still exist today leaves out the ones that failed or were delisted.
3. Ticker reuse: symbols get recycled and companies get renamed; joining on the symbol alone can splice two companies into one history. See delisted stocks and tickers.
4. Unrealistic costs: commissions, spreads and slippage add up. A high-turnover strategy can look fine before costs and lose money after them.
5. Overfitting and multiple testing: try enough variations and one will look good by chance. You say how many variants you tried, and the Deflated Sharpe Ratio raises the bar to match. See number of variants.
6. Sub-period instability: the test period is split into thirds and years and the pieces compared.
7. Concentration: if a handful of days or tickers produce most of the gains, the result is fragile.

Each check in a report is marked "Nothing found", "Possible issue" or "Red flag". The report ends with a verdict (Major red flags, Some red flags, or No red flags found) and always the same caveat: no red flags doesn't mean a strategy will make money; it means these checks didn't find the common errors.

The checks miss things. On code it hadn't seen before, the code check caught roughly half to two thirds of the serious leaks (47% and 67% in two internal tests).