These checks come from the process we used on our own models, which found real data errors along the way. None of the models we tested passed our own pre-registered standards.
Backtest Auditor · In development
Red-team your backtest before you trust it with money.
Run it on your strategy code, return series or trade log and get a report on the errors that most often make a backtest look better than it is. Version 0.1 is an MCP server and Python library that runs on your own machine. We're testing it before a public release; a hosted web check is planned.
Sign-ups aren't open yet. Want a heads-up? Email hello@nothingbutalpha.app.
What we check
Seven ways a backtest misleads you.
Lookahead bias
Using data that wasn't available when the trade would have been made, such as a day's closing price to trade at that day's open. It's one of the most common ways a backtest looks better than it should.
Survivorship bias
Testing only on companies that still exist today leaves out the ones that failed or were delisted, which tilts results upward.
Ticker reuse
Symbols get recycled and companies get renamed. Joining data on the symbol alone can splice two different companies into one price history.
Unrealistic costs
Commissions, spreads and slippage add up. A high-turnover strategy can look fine before costs and lose money after them.
Overfitting and multiple testing
Try enough variations and one will look good by chance. You tell us how many variants you tried, and the Deflated Sharpe Ratio raises the bar to match.
Sub-period instability
A result that comes from one stretch of years may not hold in others. We split the test period into thirds and years and compare the pieces.
Concentration
If a handful of days or tickers produce most of the gains, the result is fragile. We measure how much comes from the best days and the biggest positions.
The tools
Four MCP tools, one report format.
Once released, you'll be able to ask your AI assistant to run them, or call the same functions from Python.
check_codePython strategy source code
Lints the code for lookahead and survivorship leaks, missing costs and similar mistakes. Each finding has a rule id, the line, why it inflates results and how to fix it.
audit_returnsA return series, as CSV or JSON
Checks costs and the break-even cost, Sharpe significance, the Deflated Sharpe Ratio for the number of variants you tried, sub-period stability, concentration, drawdown and a benchmark comparison.
audit_trade_logA CSV trade log: date, ticker, side, quantity, price
Rebuilds daily results from the trades, runs the same checks as audit_returns, and adds ticker concentration and survivorship hints.
explainA check id
Says what a check does, why it matters, how to fix it and where it comes from.
Every report ends with a verdict: Major red flags, Some red flags, No red flags found. No red flags doesn't mean a strategy will make money; it means these checks didn't find the common errors.
Limits and your data
What it misses, and what we keep.
- It misses things. Static checks catch many common leaks but miss others. On leaky code the checks hadn't seen before, the code check caught roughly half to two thirds of the leaks (47% and 67% in two internal tests).
- Your code is never executed. The Auditor reads it with static analysis only.
- Hosted uploads (planned) will be deleted within 24 hours. When the hosted check opens (planned), uploaded code, returns and trade logs will be deleted within 24 hours, never used to train models and never shared. Run locally, your files never reach us.
Sample report
What a report will look like.
A layout preview for a made-up strategy file. It describes no real strategy.
Illustrative sample: a made-up strategy, not a real audit
Verdict: Major red flagsNo red flags doesn't mean a strategy will make money; it means these checks didn't find the common errors.
- Lookahead bias Red flagThe signal for each day uses that day's close, but trades fill at that day's open.
- Survivorship bias Possible issueThe universe is today's index members. Names that left the index are missing.
- Ticker reuse Nothing foundNo symbol in the trade log was traded before its current company used it.
- Unrealistic costs Red flagNo trading costs are modeled, and turnover is high.
- Overfitting and multiple testing Possible issueA parameter grid was scored on the full sample. Only the best result is reported.
- Sub-period instability Nothing foundResults have the same sign in each third of the test period.
- Concentration Nothing foundGains aren't dominated by a few days or tickers.
This is a statistical and code review of a hypothetical backtest. It is not investment advice, a recommendation to trade any strategy or security, or a prediction of future results. It is not tailored to your finances, goals or risk tolerance. “No red flags found” means these checks didn't find these common errors; it doesn't mean a strategy will make money. The review depends on the information you provided, including the number of variants you tried. Reports are produced by software and can contain errors. Nothing But Alpha is not a registered investment adviser or broker-dealer.
Where the checks come from
We ran them on ourselves first.
FAQ
Questions
What formats does it support?
Version 0.1 reads Python source code for the code check, return series as CSV or JSON, and trade logs as CSV. It doesn't read Pine Script, notebooks or other languages yet.
Is this investment advice?
No. The Auditor checks a backtest for common errors. It doesn't tell you whether to trade a strategy, and it isn't tailored to your finances or goals.
What do the verdicts mean?
Every report ends with one of three verdicts: Major red flags, Some red flags, No red flags found. Each is followed by the same line: No red flags doesn't mean a strategy will make money; it means these checks didn't find the common errors.
Will it catch every leak in my code?
No. Static checks catch many common leaks but miss others. In two internal tests on leaky code the checks hadn't seen before, the code check caught 15 of 32 (47%) and 8 of 12 (67%) of the serious leaks. Treat a clean code check as weak evidence, not a clean bill of health.
Do you support MCP?
Version 0.1 is an MCP server with four tools: check_code, audit_returns, audit_trade_log and explain. It runs on your own machine. We're testing it internally; it isn't publicly available yet.
What happens to my code and data?
Your code is never executed: the Auditor only reads it. When you run it on your own machine, nothing reaches us, though anything you paste into an AI assistant goes to that assistant's provider under its terms. For the hosted check (planned), uploaded code, returns and trade logs will be deleted within 24 hours, never used to train models and never shared.
When do you launch?
We haven't set a date.