On a finished chart every good setup looks obvious and every bad one looks avoidable. That is hindsight bias, and it is the main reason manual backtests on static charts come out better than the same rules traded live.
Bar-by-bar replay fixes the obvious version of it: you cannot see what happens next because it has not been drawn yet. It does not fix everything. Why bar-by-bar replay beats chart scrollback covers what it does fix. This post is about what it does not.
Leak 1: you remember the chart
Hide the future bars of the 2022 FTX collapse and most crypto traders still know how it ends. The same goes for COVID in March 2020, the May 2021 crash, any headline move you lived through. Once you recognise the period, you are no longer deciding under uncertainty — you are recalling.
Memory leaks in quieter ways too. A familiar price level, a date in the corner of the screen, the symbol name next to a year you remember as a bull market — any of these can tilt a decision without your noticing.
Fix: test on periods you cannot place. Backcandle's blind replay picks the instrument and window at random, hides the symbol, and shifts every timestamp into a distant future year. Weekday, time of day and session gaps are kept, so the market's rhythm stays real. The real symbol and dates are revealed when the session ends.
Leak 2: the rules move while you test
The second leak is subtler. You start testing a rule, lose three trades in a row, and add a filter that would have avoided them. Then another. By the end you have a rule set fitted to the sample you just watched — the manual version of curve-fitting.
Fix:
- Freeze the rules before the first bar. Write them down. If you want to change them, finish the sample first, then start a new sample with the new rules.
- Log the skipped trades. If the rules fired and you passed, that is a discretionary filter you have not written down.
- Keep a holdout. Develop on one set of periods; confirm on another you have not looked at.
Leak 3: you choose the periods
Picking windows where the setup "should" work is selection bias, and it is easy to do without meaning to. Choose periods by date before looking at them, or let blind replay choose.
A quick self-check
If your replay results are much better than your live results on the same rules, assume one of these leaks before assuming bad luck. Running the same rules in blind sessions is the cleanest comparison: if the edge survives without the symbol and the dates, it is more likely to survive the market.
For the structure of a test session, see how to structure a replay session.