Why Most Backtests Lie

Backtests do not fail randomly. They fail in one direction — optimistic — and they do so for five structural reasons that are all easy to introduce by accident and hard to notice once introduced.

1. Survivorship bias#

Your universe is the instruments that still exist. The ones that were delisted, acquired or went to zero are missing, and they are precisely the ones that would have hurt.

Detection: count the instruments in your universe as of 2015 in your data, and compare with how many actually existed then. If the numbers match, your data is survivorship-biased.

Cost: typically 1–4 percentage points of annual return on equity strategies. Enough to turn a losing strategy into a winning one on paper.

2. Look-ahead bias#

Using information that was not available at decision time. It is rarely as obvious as reading tomorrow's price. It usually looks like:

• Fundamental data stamped with the period it describes rather than its publication date.
• Prices that were later revised, used as if the revision was known.
• Index membership as of today applied to historical dates.
• Computing a normalisation constant — a mean, a standard deviation — over the whole sample.

the common onepython
# Wrong: z-score uses the full history's statistics, including the future.
df = df.with_columns(
    ((pl.col("close") - pl.col("close").mean()) / pl.col("close").std()).alias("z")
)

# Right: expanding window uses only what was known at each point.
df = df.with_columns(
    (
        (pl.col("close") - pl.col("close").rolling_mean(252))
        / pl.col("close").rolling_std(252)
    ).alias("z")
)

Detection: shift every feature forward by one bar and rerun. If results collapse, you had look-ahead.

3. Overfitting#

Testing forty parameter combinations and reporting the best one is not research; it is selecting a maximum from a noise distribution. With enough variants, something will look excellent on any history.

Detection: walk-forward. Optimise on a rolling in-sample window, evaluate strictly out of sample, and report the retention ratio. Below roughly 30% retention, you are looking at a curve fit.

The honest addition: report how many variants you tried. That number is a fact about the research process and it belongs in the result.

4. Ignoring costs#

Zero-cost backtests do not merely overstate returns — they change which strategies appear to work. High-turnover strategies look best precisely because turnover is free in the simulation.

Model commission, half-spread, and market impact that scales with participation. A strategy that survives realistic costs is a different strategy from the one that wins without them.

5. Regime luck#

A decade of data contains far fewer independent market regimes than it appears to. A strategy tested on 2013–2023 was tested largely on one long trending environment with two sharp interruptions.

Detection: report results per regime — trending, ranging, high volatility, crisis — separately. Many strategies work in exactly one and are flat or negative elsewhere. An aggregate number hides that entirely.

What an honest report contains#

None of this makes a strategy work. It makes the report tell you the truth about whether it does — which is the only thing a backtest was ever able to offer.