The factor research pipeline ends with a test I could only run once. Before looking at any result I set aside the last five years of data. I ran six ways of combining eleven published factors on everything before that, picked the best by a rule fixed in advance, committed the choice, and then ran it on the five years.
| In sample | Holdout | |
|---|---|---|
| Period | August 1969 to December 2020 | January 2021 to December 2025 |
| Net Sharpe ratio | 1.50 | −0.37 |
| 95% bootstrap interval | 1.12 to 1.90 | −1.20 to 0.40 |
| Annual return | 20.0% | −5.4% |
| Deflated Sharpe ratio | 1.000 | |
| Probability of backtest overfitting | 0.071 |
The two intervals do not overlap. This is more than five noisy years.
What the statistics measure
The deflated Sharpe ratio asks: given that I tried six configurations, how likely is it that the best one has a true Sharpe ratio above what the best of six with no skill would show? The probability of backtest overfitting asks: if I pick the winner on one half of the sample, how often does it land in the bottom half on the other?
Both are about selection. They answered correctly. The result was not an accident of picking one of six: all six had positive Sharpe ratios in sample, and the in-sample winner stayed in the top half out of sample on 93% of splits.
Neither asks whether the backtest describes something that could have been traded, or whether the next five years resemble the last fifty.
What they do not measure
Before the holdout was run I wrote down why the in-sample numbers looked too good. None of these is a selection problem.
- Costs. Turnover was above 100% of the book every month, charged at a flat ten basis points from 1963 to 2020. Trading was far more expensive in the earlier decades.
- An input that may not be tradable. One-month reversal was among the strongest factors. It is often attributed in part to bid-ask bounce: the closing price a backtest trades at is not a price anyone could get.
- No market impact and no borrow fee.
- Stationarity. Every factor came from a published paper. A backtest that starts before the papers were written measures something the market may since have priced.
A statistic computed from the backtest's own returns cannot see any of this. If the returns are wrong in the same way in every configuration, comparing configurations tells you nothing about the error.
Why I did not fix them first
I could have raised the cost or dropped reversal before running the holdout. I did not, because by then I had seen the in-sample results, and a change made after seeing them changes what the holdout tests. The honest version of the experiment was the one already written down. A fair objection is that I spent the only look I had on a configuration I already doubted.
Why I do not know which one it was
After a result like this the natural next step is to break the five years down by year, by factor, by method. Each of those is another look at the same data, and after enough of them the holdout is just more in-sample data with a story attached. So the cause is not established. The list above is what I believed beforehand, and it remains untested.
What I take from it
- Selection statistics are necessary and they are not a verdict. They rule out one way of fooling yourself.
- The assumptions of a backtest need their own evidence: a cost model that matches the period, and a check that each input can be traded at the price the backtest uses.
- A holdout only works if the design is frozen first and the result is reported whatever it is. This one was, and it is.
Calculated based on data from CRSP US Stock Database, © Center for Research in Security Prices (CRSP), The University of Chicago Booth School of Business, and from Compustat, © S&P Global Market Intelligence.