Lookahead bias in pandas: 8 ways a backtest sees the future
If a backtest looks too good, check it for lookahead before you check anything else. Lookahead bias means the strategy uses information on a given day that wasn't available until later. In pandas it rarely looks like cheating. It looks like one argument with the wrong sign, a fill method, or a scaler fitted on the whole table.
Below are eight patterns that come up again and again, each with the leaky line and the fix. Every example runs on synthetic data, and every one comes with the same check: change the data after some date t, and make sure nothing computed up to t changes. If it does, the code is reading the future.
All results on this page are hypothetical and come from simulated prices. They show how the bug behaves, not how any strategy performs.
Setup: a market with nothing to find
The prices below are a random walk, so no strategy can predict them. Anything that "works" on this data is leaking.
import numpy as np
import pandas as pd
rng = np.random.default_rng(42)
n = 1000
dates = pd.bdate_range("2020-01-01", periods=n, name="date")
close = 100 * np.exp(np.cumsum(rng.normal(0, 0.01, n)))
open_ = np.r_[100, close[:-1]] * np.exp(rng.normal(0, 0.003, n))
df = pd.DataFrame({"open": open_, "close": close}, index=dates)
t = dates[600] # the "today" we test againstAnd the test. It multiplies every number after t by random noise, recomputes the feature, and compares everything up to t:
def leaks(make_feature, data, t, seed=0):
"""True if anything computed for dates <= t changes when only data after t changes."""
noise = np.random.default_rng(seed)
future = data.index > t
cols = data.select_dtypes("number").columns
changed = data.copy()
changed.loc[future, cols] = changed.loc[future, cols] * noise.uniform(0.5, 1.5, (future.sum(), len(cols)))
before = make_feature(data).loc[:t].to_numpy(dtype=float)
after = make_feature(changed).loc[:t].to_numpy(dtype=float)
return not np.allclose(before, after, equal_nan=True)1. shift(-1) in a feature
A negative shift pulls tomorrow's value into today's row.
leaky = lambda d: d["close"].pct_change().shift(-1) # tomorrow's return, today
fixed = lambda d: d["close"].pct_change() # known at today's close
leaks(leaky, df, t), leaks(fixed, df, t) # -> (True, False)Negative shifts are the normal way to build a label, the thing you're trying to predict. They belong in the label column only, and that column must stay out of the features.
2. Trading on the bar that made the signal
This one passes the test above, because nothing after t is used. The problem is timing: the signal is computed from today's close, and the strategy is credited with the return that ends at that same close.
ret = df["close"].pct_change()
signal = np.sign(ret) # decided at today's close
def sharpe(r):
return np.sqrt(252) * r.mean() / r.std()
round(sharpe(signal * ret), 1) # -> 20.5 (leaky; hypothetical, random walk)
round(sharpe(signal.shift(1) * ret), 1) # -> 1.1 (fixed; still noise, random walk)On a random walk, the leaky version shows an annualized Sharpe ratio of about 20 (hypothetical, on simulated data). The fixed version holds yesterday's position through today's return. Its 1.1 (also hypothetical) is noise, not skill: with about four years of daily data, the standard error of an annualized Sharpe ratio is roughly 0.5, so a value near 1 on pure noise is unremarkable. (Try enough variants and one will look far better; that's what the deflated Sharpe ratio corrects for.) That gives a second test: run your strategy on several random walks; a value as large as the leaky one should never appear.
If your signal uses the close, the realistic fill is the next open. A label built that way may look forward, as long as it stays a label:
h = 5
entry = df["open"].shift(-1) # buy at the next open
exit_ = df["close"].shift(-h) # sell at the close h bars later
label = exit_ / entry - 1 # a label may look forward; features may not3. rolling(..., center=True)
A centered window averages the rows on both sides of today, so half of it is in the future.
leaky = lambda d: d["close"].rolling(20, center=True).mean()
fixed = lambda d: d["close"].rolling(20).mean() # trailing window, the default
leaks(leaky, df, t), leaks(fixed, df, t) # -> (True, False)Smoothing that looks great on a chart is often centered. Check the default before you copy a plotting snippet into a strategy.
4. bfill()
Back-filling copies a later value into an earlier gap. Forward fill carries the last known value instead.
gappy = df.copy()
gappy.loc[t, "close"] = np.nan # a missing price on day t
leaky = lambda d: d["close"].bfill() # fills day t from day t+1
fixed = lambda d: d["close"].ffill() # carries day t-1 forward
leaks(leaky, gappy, t), leaks(fixed, gappy, t) # -> (True, False)interpolate() has the same problem: a linear interpolation through a gap uses the point after it.
5. Normalizing on the full sample
A z-score with the full-sample mean and standard deviation tells every row something about the whole history, including the part that hadn't happened yet.
x = lambda d: d["close"].pct_change()
leaky = lambda d: (x(d) - x(d).mean()) / x(d).std()
fixed = lambda d: (x(d) - x(d).expanding(min_periods=60).mean()) / x(d).expanding(min_periods=60).std()
leaks(leaky, df, t), leaks(fixed, df, t) # -> (True, False)The same applies to scikit-learn. Split by time first, then fit the scaler on the training window only:
from sklearn.preprocessing import StandardScaler
X = pd.DataFrame({"ret": df["close"].pct_change(), "ret_5": df["close"].pct_change(5)}).dropna()
train, test = X[X.index <= t], X[X.index > t]
scaler = StandardScaler().fit(train) # not .fit(X)
test_scaled = scaler.transform(test)A Pipeline refitted inside each walk-forward step does this for you.
6. Fundamentals keyed by period end
Quarterly results "dated" December 31 weren't public on December 31. They became public when the company filed, often weeks later. Merge on the date the data became available, not the period it describes.
prices = pd.DataFrame({"date": pd.to_datetime(["2021-01-15", "2021-03-01"]), "ticker": "XYZ"})
funds = pd.DataFrame({
"ticker": "XYZ",
"period_end": pd.to_datetime(["2020-09-30", "2020-12-31"]),
"accepted_date": pd.to_datetime(["2020-10-29", "2021-02-19"]),
"eps": [1.10, 1.35],
})
leaky = pd.merge_asof(prices, funds.sort_values("period_end"),
left_on="date", right_on="period_end", by="ticker")
fixed = pd.merge_asof(prices, funds.sort_values("accepted_date"),
left_on="date", right_on="accepted_date", by="ticker")
leaky["eps"].tolist(), fixed["eps"].tolist() # -> ([1.35, 1.35], [1.1, 1.35])On January 15 the leaky merge already knows the Q4 result that was filed on February 19. Keep the default direction="backward"; "forward" and "nearest" can attach a later record. If your data source restates numbers, you also need the value as originally reported, not today's corrected one. Point-in-time fundamentals covers filing times, acceptance times and restatements in more detail.
7. Back-adjusted price levels as features
Split- and dividend-adjusted prices are rewritten every time a new split or dividend happens. An adjusted price for 2015 that you download today depends on every split since 2015. Returns computed from adjusted prices don't have this problem; levels do.
def back_adjust(raw, split):
"""Divide each price by the product of all splits after that day."""
later = split[::-1].cumprod()[::-1] / split
return raw / later
px = pd.DataFrame({"raw": df["close"], "split": 1.0})
split_day = dates[800] # a 2-for-1 split, after t
px.loc[px.index >= split_day, "raw"] /= 2
px.loc[split_day, "split"] = 2.0
levels = lambda d: back_adjust(d["raw"], d["split"])
returns = lambda d: back_adjust(d["raw"], d["split"]).pct_change()
leaks(levels, px, t), leaks(returns, px, t) # -> (True, False)A rule like "price below $20" or a feature like a 52-week high in adjusted dollars quietly uses future corporate actions. Use returns and ratios, or rebuild adjusted prices as of each date.
8. Aggregations on the wrong rows
Two variations of the same mistake: a summary computed over rows it shouldn't see.
Resample labels. A weekly bar contains the whole week. If it's stamped with the start of the week and joined back to daily rows, every day of that week sees Friday's close.
weekly = lambda d, label: d["close"].resample("W-FRI", label=label).last().reindex(d.index, method="ffill")
leaky = lambda d: weekly(d, "left") # bar stamped with the Friday before the week
fixed = lambda d: weekly(d, "right") # bar stamped with the Friday that ends it
leaks(leaky, df, t), leaks(fixed, df, t) # -> (True, False)For weekly, month-end and quarter-end rules pandas defaults to label="right". For most others, including intraday rules like "5min", the default is closed="left", label="left": each bar is stamped with its start time and contains trades after that stamp. See the pandas resample documentation, and set label explicitly.
Cross-sectional ranks. Ranking a stock's return against every row of a long panel compares it with other days, including future ones. Rank within each date.
wide = pd.DataFrame(rng.normal(0, 0.01, (n, 4)), index=dates, columns=["AAA", "BBB", "CCC", "DDD"])
wide.columns.name = "ticker"
panel = wide.stack().rename("ret").reset_index(level="ticker")
leaky = lambda p: p["ret"].rank(pct=True) # against every day, past and future
fixed = lambda p: p.groupby(level="date")["ret"].rank(pct=True) # within each day
leaks(leaky, panel, t), leaks(fixed, panel, t) # -> (True, False)Put the test in your test suite
The leaks check is easy to run on every change. Here it is as a pytest file. It assumes your feature code has one entry point that takes prices and returns a DataFrame indexed by date:
# file: test_no_lookahead.py
import numpy as np
import pandas as pd
import pytest
from features import build_features # your code: prices in, features out
def make_prices(n=500, seed=0):
rng = np.random.default_rng(seed)
dates = pd.bdate_range("2020-01-01", periods=n, name="date")
close = 100 * np.exp(np.cumsum(rng.normal(0, 0.01, n)))
return pd.DataFrame({"open": close * (1 + rng.normal(0, 0.002, n)), "close": close}, index=dates)
@pytest.mark.parametrize("cut", [100, 250, 400])
def test_features_ignore_the_future(cut):
prices = make_prices()
t = prices.index[cut]
future = prices.index > t
changed = prices.copy()
changed.loc[future] = changed.loc[future] * np.random.default_rng(1).uniform(0.5, 1.5, (future.sum(), 2))
pd.testing.assert_frame_equal(build_features(prices).loc[:t], build_features(changed).loc[:t])It covers patterns 1 and 3 to 8. Pattern 2 needs the random-walk check, because the leak is in how returns are booked, not in the features.
Walk-forward splits need a gap
A label that looks h bars ahead overlaps the next h rows. If the training window ends right before the test window starts, the last training labels contain test-period prices. Leave a gap at least as long as the label horizon (López de Prado calls this purging and embargo; see Advances in Financial Machine Learning, chapter 7). scikit-learn's TimeSeriesSplit has a gap parameter for this:
from sklearn.model_selection import TimeSeriesSplit
h = 5 # label horizon in bars
for train_idx, test_idx in TimeSeriesSplit(n_splits=5, gap=h).split(X):
assert train_idx.max() + h < test_idx.min() # no training label reaches into the test windowNever shuffle time-series rows into folds. train_test_split shuffles by default; pass shuffle=False.
Checking code for these patterns
Backtest Auditor (in development) looks for patterns 1 to 6 in Python code, without running it, and explains each finding with the line and a suggested fix. Of pattern 8 it covers only intraday resamples stamped at the bar start; it doesn't look for pattern 7, weekly or monthly resample labels, or pooled cross-sectional ranks. It will be available after release. A static check misses some leaks, so the tests above are worth keeping either way.
Further reading: Kaufman, Rosset, Perlich and Stitelman, "Leakage in Data Mining: Formulation, Detection, and Avoidance", ACM Transactions on Knowledge Discovery from Data 6(4), 2012. Related on this site: survivorship bias and reused tickers, the deflated Sharpe ratio in Python, and our help articles.