Backtesting & Risk

This is a comprehensive guide to common pitfalls in the "Backtesting and Risk Control" series: Backtesting is the most self-deceptive stage in quantitative research. This article lists all the traps that "look beautiful but lose money" in one go, with real cases from this site following each trap.

One-Sentence Conclusion

Trap	One-Sentence Summary	Site Case
Look-Ahead Bias	Using data that didn't exist "at that time"	Data Normalization
Overfitting	Parameters are tuned, not selected	Beyond Out-of-Sample Testing
Survivorship Bias	Only looking at stocks that survived	Low Turnover Factor
Costs	What remains after deducting fees	Picking Nickels in Front of a Steamroller
Executability	Daily bar executability ≠ You can execute	Volatility Trading

I. Look-Ahead Bias: Most Common, Yet Most Concealed

The Sharpe Ratio Greater Than 4 Strategy details a real case: a model suddenly performed far better than historical norms, with the only change being the normalization method—global Z-score leaked "future" information into the "past." The correction method is using rolling windows (Point-in-Time): each day can only use data from that day and prior.

Self-check checklist: Use announcement dates for financial reports (not report periods), use当日 factors for price adjustment, and use the constituent list at that time. If any item uses something "known only after the fact," the backtest result is void.

II. Overfitting: More Parameters, Faster Death

Beyond Out-of-Sample Testing, What Other Overfitting Detection Methods Are There systematically explains three sets of detection methods: out-of-sample testing, white noise, and label shuffling. Remember two iron rules:

1. Parameter tuning counts towards degrees of freedom: A beautiful curve found after trying 20 parameter sets will likely fail out-of-sample.
2. Out-of-sample testing can only be used once: Using it a second time turns it into in-sample data.

The title of the Low Turnover Factor post is quite honest—"How could such a simple Alpha calculation be wrong?" A single error in details (alignment, adjustment timing, missing value handling) can reverse the conclusion.

III. Costs and Executability: Paper Profits Discounted

Picking Nickels in Front of a Steamroller discusses the temptation and traps of high-frequency, low-risk strategies: nominal Sharpe ratios are high, but after deducting impact costs, slippage, and stock borrowing fees, very little remains. In the A-share market, costs start at a bilateral 2.5 basis points; the higher the turnover, the more severe the cost erosion.

KS Test and Bottom-Fishing the Shanghai Composite demonstrates that "statistically significant ≠ tradable"—no matter how well the distribution fits, it's a different story when it hits the real order book.

IV. Risk Control: The Last Gate Before Strategy Launch

• Position Sizing: Maximum drawdown of a single strategy × 2 is the psychological loss you must prepare for.
• Circuit Breakers: Set one at the strategy level (underperforming the benchmark for N consecutive days) and one at the account level (total drawdown line).
• Attribution: Answer every day after close: "Where did today's profits come from?" If you can't answer, it means you don't understand your own strategy.
• Interview Question Self-Test: The Probability Trap of Random Points on a Circle—Those who can't even calculate basic probabilities correctly cannot write reliable backtests.

V. Further Reading

• Data Normalization Pitfall Avoidance — The first case of look-ahead bias
• Overfitting Detection Methods — Beyond out-of-sample testing
• Low Turnover Factor — Details determine life and death
• Picking Nickels in Front of a Steamroller — Cost perspective
• Random Points on a Circle — Probability fundamentals
• Volatility Trading — Executability

There is no silver bullet in backtesting, only one sentence: Assume all "too good to be true" results are your own fault first, then prove it's the market's fault.