The tests are sharp and specific. Running the reference engine and the eight single-bug mutants against all eight traps gives a diagonal matrix: the reference engine passes everything, and each mutant fails its own trap and no other.
A second engine passes. An example vectorized engine that works in money per asset with numpy, and shares no code with the reference engine, passes all eight traps. On a 60-day market with gapping opens and costs it matches the reference engine to 1e-9.
How much each bug moves a backtest, as final NAV on the trap's own scenario:
| Trap | Bias | Correct engine | With the bug | Change |
|---|
T1 | Price lookahead | −12.6% | +211.5% | +256.6% |
T2 | Signal/execution misalignment | +4.7% | +0.8% | −3.7% |
T3 | Fundamentals not point-in-time | −15.1% | −14.2% | +1.0% |
T4 | Survivorship | −12.1% | −3.1% | +10.2% |
T5 | Zero transaction costs | −2.1% | +0.0% | +2.1% |
T6 | Corporate actions unadjusted | +0.0% | −39.4% | −39.4% |
T7 | Universe membership lookahead | +0.3% | +41.9% | +41.4% |
T8 | Free and unlimited shorting | −0.5% | +0.0% | +0.5% |
These are sizes on small synthetic markets, not estimates for real data. T3's +1.0% is noise: the synthetic fundamentals do not predict returns, so the leak changes which stock is held without making it a better one. It is caught on the first day either way.