Backtesting trading strategies
Why are backtest numbers too good?
Backtesting trading strategies produces a claim about what would have happened. Almost every backtest is more optimistic than what does happen, and the reasons are known rather than mysterious. The useful skill is not producing one, it is knowing how a backtest lies and correcting for it.
Expect live results to trail a study meaningfully. If a change to your method makes the backtest look dramatically better, the most likely explanation is that you introduced one of the errors below rather than found an edge.
Which errors inflate backtesting results?
Fills nobody actually gets
Assuming the mid price, or assuming a fill at the price that triggered the signal, is the largest single source of overstatement. Real fills pay the spread, and on anything with multiple legs or thin quotes they pay considerably more. A cost model that reflects what you would actually pay produces worse and far more useful numbers.
Look-ahead bias
Using information the strategy could not have had at the moment it decided. This creeps in subtly through revised data, through survivorship in the instrument list, and, with language models, through knowledge of the period being tested. It is the error most likely to be invisible.
Selecting the period that flatters
Any approach looks good over some window. Choosing the window after seeing the results is not testing, it is illustration, and it is extremely easy to do without intending to.
Tuning until it passes
Adjusting parameters against the same data until performance improves fits the history rather than the market. The improvement is real in the study and absent afterwards.
Ignoring what the strategy could not have traded
Instruments that were illiquid, halted, or not yet listed at the time still appear in a naive test. Sizes that could not have been filled do too.
How should you approach backtesting trading strategies?
Hold data back and mean it
Validation on data the process has never touched is the only meaningful check, and it stops being meaningful the moment you iterate against it. Once you have looked, it is no longer out of sample.
Test the method, not the result
The question is whether the process for finding approaches produces things that hold up, not whether one particular approach scored well. Individual results are noisy enough to be uninformative.
Reproduce it on demand
A backtest you cannot rerun from a single command is a screenshot. Being able to reproduce it is what allows anyone, including you, to check whether it still holds after a change.
Expect and accept worse numbers
A study that gets less impressive as you add realistic cost and honest validation is a study getting more truthful. That direction of travel is the signal you want.
How do you judge someone else's backtest?
Ask what the fills assumed
Asking what the fills assumed separates most serious research from most sales material. If the answer is the mid price, the numbers do not describe anything achievable.
Ask whether you can rerun it
On your hardware, from one command. A backtest you have to take on trust is marketing regardless of who produced it.
Ask what was excluded and why
Every study excludes something. The exclusions are where the flattering assumptions hide, and a serious answer will be specific.
Compare it against live behaviour
If a system is running, its actual record can be checked against what the study predicted. Divergence is the most informative measurement available and almost nobody offers it.
What can backtesting not tell you?
Whether you could have held it
A study shows a drawdown as a line. Living through it with your own capital is a different experience, and most abandonment happens for reasons no backtest models.
How the market will change
A backtest describes a period that has ended. Approaches decay, and nothing in a historical study anticipates the regime it was not tested in.
What happens when the system fails
Outages, bad data and rejected orders do not appear in a clean historical replay, and they are ordinary in live operation.
Common questions
- Why do backtests look better than live trading?
- Several known reasons compound: assumed fills that ignore the spread and market impact, look-ahead bias from information the strategy could not have had, periods chosen after seeing results, and parameters tuned against the same data until performance improved. Expect live results to trail a study meaningfully rather than to match it.
- What is look-ahead bias in backtesting?
- Using information at a decision point that would not have been available then. It enters through revised data that differs from what was published at the time, through instrument lists that exclude things which later failed, and, when language models are involved, through the model already knowing what happened in the period being tested. It is the error most likely to remain invisible.
- How do I know if a vendor's backtest is trustworthy?
- Ask what the fills assumed, since mid-price assumptions describe nothing achievable. Ask whether you can rerun it yourself on your own hardware from one command. Ask what was excluded and why, because exclusions are where flattering assumptions hide. A backtest you have to take on trust is marketing, whoever produced it.
- What can a backtest not tell you?
- Whether you could have held the approach through its drawdown, which is where most abandonment happens and which no study models. How the market will change, since a backtest only describes a period that has ended. And what happens during outages, bad data and rejected orders, which are ordinary in live operation and absent from a clean historical replay.
TradeAgentic is an autonomous trading desk for macOS and Windows, licensed to operators and firms who intend to run it themselves.
Request licensing terms