Backtesting trading strategies
Why are backtest numbers too good?
Backtesting trading strategies produces a claim about what would have happened. Almost every backtest is more optimistic than what does happen, and the reasons are known rather than mysterious. The useful skill is not producing one, it is knowing how a backtest lies and correcting for it.
Expect live results to trail a study meaningfully. If a change to your method makes the backtest look dramatically better, the most likely explanation is that you introduced one of the errors below rather than found an edge.
Which errors inflate backtesting results?
Fills nobody actually gets
Assuming the mid price, or assuming a fill at the price that triggered the signal, is the largest single source of overstatement. Real fills pay the spread, and on anything with multiple legs or thin quotes they pay considerably more. A cost model that reflects what you would actually pay produces worse and far more useful numbers.
Look-ahead bias
Using information the strategy could not have had at the moment it decided. This creeps in subtly through revised data, through survivorship in the instrument list, and, with language models, through knowledge of the period being tested. It is the error most likely to be invisible.
Selecting the period that flatters
Any approach looks good over some window. Choosing the window after seeing the results is not testing, it is illustration, and it is extremely easy to do without intending to.
Tuning until it passes
Adjusting parameters against the same data until performance improves fits the history rather than the market. The improvement is real in the study and absent afterwards.
Ignoring what the strategy could not have traded
Instruments that were illiquid, halted, or not yet listed at the time still appear in a naive test. Sizes that could not have been filled do too.
How should you approach backtesting trading strategies?
Hold data back and mean it
Validation on data the process has never touched is the only meaningful check, and it stops being meaningful the moment you iterate against it. Once you have looked, it is no longer out of sample.
Test the method, not the result
The question is whether the process for finding approaches produces things that hold up, not whether one particular approach scored well. Individual results are noisy enough to be uninformative.
Reproduce it on demand
A backtest you cannot rerun from a single command is a screenshot. Being able to reproduce it is what allows anyone, including you, to check whether it still holds after a change.
Expect and accept worse numbers
A study that gets less impressive as you add realistic cost and honest validation is a study getting more truthful. That direction of travel is the signal you want.
How do you judge someone else's backtest?
Ask what the fills assumed
Asking what the fills assumed separates most serious research from most sales material. If the answer is the mid price, the numbers do not describe anything achievable.
Ask whether you can rerun it
On your hardware, from one command. A backtest you have to take on trust is marketing regardless of who produced it.
Ask what was excluded and why
Every study excludes something. The exclusions are where the flattering assumptions hide, and a serious answer will be specific.
Compare it against live behavior
If a system is running, its actual record can be checked against what the study predicted. Divergence is the most informative measurement available and almost nobody offers it.
What should a good backtest report show?
| What to look for | Why it matters | Red flag |
|---|---|---|
| Cost assumptions | Costs decide most short-horizon strategies | Frictionless fills, or no mention of costs |
| Research and validation periods | Shows whether results held on unseen data | One continuous period with no hold-out |
| How many variations were tried | More attempts make a lucky result likelier | Silence on the number of attempts |
| The worst stretch | Tells you what you would have had to sit through | Only annual or total figures |
| Kinds of market covered | Different markets reward different things | Only a rising market |
| Reproducibility | Evidence you can check yourself | A chart and nothing else |
Backtesting, paper trading and live trading compared
| Backtesting | Paper trading | Live trading | |
|---|---|---|---|
| What it tests | The idea against history | Behavior in today's market | Everything, with real money |
| Real fills | No | No | Yes |
| Speed | Years in minutes | Real time | Real time |
| Cost of being wrong | Nothing | Nothing | Real losses |
| What it cannot show | How holding it feels | Real execution cost | Nothing hidden, but expensive |
Each stage answers a question the previous one cannot, and a strategy should earn its way through all three in order. The mechanics of how to backtest a trading strategy, step by step, are set out on their own page.
Which backtesting mistakes depend on the strategy?
Trend following
Results often depend on a handful of long moves. Remove a few of them and see what is left. If the answer is nothing, the strategy is a bet on those moves recurring.
Mean reversion
Many small wins and the occasional large loss. Backtests tend to understate the large loss, because the period tested did not happen to contain one bad enough.
Options strategies
Quotes can be wide and stale, and a backtest using midpoints assumes fills nobody gets. Assignment and expiration need modeling, not assuming.
Crypto
Prices differ between venues and histories are short. A strategy tested on one exchange's data may not survive another's, and weekend liquidity is thinner.
What can backtesting not tell you?
Whether you could have held it
A study shows a drawdown as a line. Living through it with your own capital is a different experience, and most abandonment happens for reasons no backtest models.
How the market will change
A backtest describes a period that has ended. Approaches decay, and nothing in a historical study anticipates the regime it was not tested in.
What happens when the system fails
Outages, bad data and rejected orders do not appear in a clean historical replay, and they are ordinary in live operation.
Common questions
- Why do backtests look better than live trading?
- Several known reasons compound: assumed fills that ignore the spread and market impact, look-ahead bias from information the strategy could not have had, periods chosen after seeing results, and parameters tuned against the same data until performance improved. Expect live results to trail a study meaningfully rather than to match it.
- What is look-ahead bias in backtesting?
- Using information at a decision point that would not have been available then. It enters through revised data that differs from what was published at the time, through instrument lists that exclude things which later failed, and, when language models are involved, through the model already knowing what happened in the period being tested. It is the error most likely to remain invisible.
- How do I know if a vendor's backtest is trustworthy?
- Ask what the fills assumed, since mid-price assumptions describe nothing achievable. Ask whether you can rerun it yourself on your own hardware from one command. Ask what was excluded and why, because exclusions are where flattering assumptions hide. A backtest you have to take on trust is marketing, whoever produced it.
- What can a backtest not tell you?
- Whether you could have held the approach through its drawdown, which is where most abandonment happens and which no study models. How the market will change, since a backtest only describes a period that has ended. And what happens during outages, bad data and rejected orders, which are ordinary in live operation and absent from a clean historical replay.
- What is overfitting in backtesting?
- Tuning a strategy until it fits historical data so closely that it captures noise rather than anything repeatable. It shows up as a backtest that looks excellent and a live result that does not. Holding data back from the research and testing on it once is the main defense.
- How much historical data do you need to backtest?
- Enough to include several different kinds of market: rising, falling, calm and volatile, including at least one sharp decline. A long period containing only one kind of market says less than a shorter one that contains several.
- Is backtesting enough before trading live?
- No. A backtest tests the idea against history. Paper trading tests behavior in the current market. Only live trading reveals real execution costs. Each stage answers a question the previous one cannot.
TradeAgentic is an autonomous trading desk for macOS, licensed to operators and firms who intend to run it themselves.
Request licensing terms