Chapter 7 · Modern Statistical Learning in Practice
Section 7.6 · Chapter 7 · Learning Statistics with Python
Modern Statistical Learning in Practice
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
Every Sharpe ratio in this course was the result of a search: over features, targets, thresholds, models. The search is the part the Sharpe ratio never reports. You will backtest 42 technical rules on the S&P 500, find the best, and then put it on trial four ways — a bootstrap interval, the Deflated Sharpe Ratio, the probability of backtest overfitting, and a false-discovery correction across the whole grid — before checking whether the Dow agrees.
17 moving-average crossovers, 9 RSI mean-reversion rules, 8 Bollinger breakouts and their 8 mirror images: 42 daily P&L series over 2 266 days, each shifted so that a position decided at the close of \(t-1\) earns \(r_t\). The mirrors are deliberate — a chart-reader who “tries both directions” has doubled the trials, and the statistics below must count them.
Best: Bollinger mean reversion, 20 days, k = 2 — Sharpe 0.68, from a rule that is in the market 9 % of the time and whose P&L has skew 7 and kurtosis 154 (a few large days). Median rule 0.15, buy-and-hold 0.64, 71 % of rules positive because the index rose. A grid this size always has a winner; the question is whether 0.68 is more than the winner of 42 coin flips.
A colleague reports: “the best of my 40 rules has Sharpe 1.1 over ten years.” Is that evidence the rule works?
Expected maximum of \(N\) trials with true Sharpe 0 and cross-trial standard deviation \(\sigma_{SR}\) (Bailey & López de Prado, 2014): \[\mathbb E[\max_N SR] \approx \sigma_{SR}\Big[(1-\gamma)\,\Phi^{-1}\!\big(1 - \tfrac1N\big) + \gamma\,\Phi^{-1}\!\big(1 - \tfrac{1}{Ne}\big)\Big], \qquad \gamma = 0.5772.\]
A bootstrap interval for the rule you picked; a Sharpe deflated by the number of rules you tried; the probability that the in-sample winner loses out of sample; and a false-discovery correction across the grid. Each answers a different question — the rule must pass all four.
Politis & Romano (1994): resample the P&L in blocks of random length (geometric, mean 20 days) so that autocorrelation and volatility clusters survive; recompute the Sharpe on each pseudo-sample.
95 % interval [0.18, 1.11], only 1.7 % of resamples at or below zero. Taken alone, the interval says “significant”. But it answers the wrong question: given that we picked this rule, how uncertain is its Sharpe? It knows nothing about the 41 rules we did not pick.
Bailey & López de Prado (2014). First, the Sharpe a selection would produce by luck alone, \(SR_0\) (the expected-maximum formula, with \(\sigma_{SR}\) estimated across the \(N\) trials). Then the probability that the observed Sharpe exceeds it, allowing for non-normal returns:
\[DSR = \Phi\!\left[\frac{(\widehat{SR} - SR_0)\sqrt{T-1}}{\sqrt{1 - \gamma_3 \widehat{SR} + \frac{\gamma_4 - 1}{4}\widehat{SR}^2}}\right],\]
with \(\widehat{SR}\) per period (daily, not annualised), \(\gamma_3\) the skewness and \(\gamma_4\) the kurtosis of the P&L.
Against a benchmark of zero the rule looks real: PSR 0.990. Against the benchmark the search deserves — an expected best-of-42 of 0.57 — the Deflated Sharpe Ratio is 0.637: a 36 % chance that a Sharpe this large comes from a grid of 42 worthless rules, before the fat tails are even the point. Ten years of the S&P 500 do not distinguish this rule from luck.
Bailey, Borwein, López de Prado & Zhu (2017). Split the \(T \times N\) P&L matrix into \(S\) blocks of consecutive days. For every way of choosing \(S/2\) blocks as in-sample (the rest is out-of-sample):
PBO = 0.533: in 24 of 45 splits the rule that won in-sample was below the median out of sample, and it finished first out of sample in only 2. This is the signature of a grid with no edge — the best in-sample rule is a different rule each time and it carries nothing across the boundary.
§3.2 gave each hypothesis its own p-value. With 42 of them, control the false discovery rate: sort the p-values, find the largest \(k\) with \(p_{(k)} \le 0.05\,k/N\), reject the \(k\) smallest.
Two rules clear 0.05 on their own p-value — the Bollinger breakout and its mirror, which share one p-value (0.043) because they are the same bet with the sign flipped. Benjamini–Hochberg needs the best of 42 below 0.0012; it is 36 times too large. Zero rules survive.
Same 42 rules on dji.csv, 1985–2020: 8 774 days, nearly four times the sample.
On the Dow the S&P’s winner earns Sharpe −0.00 and ranks 32nd of 42. The Dow’s own best — RSI-21 at 30/70 — has Sharpe 0.23, DSR 0.58, PBO 0.78, no BH survivor; and the rule that won 1985–2002 (a 50/250 crossover, 0.25) earned 0.02 afterwards. Thirty-five years, two indices, four statistics: no technical rule in this grid survives the search that found it.
Add a 43rd rule to the S&P grid — 63-day time-series momentum, new_rule below — as column "TSMOM63" of pnl2, then recompute the best rule and its Deflated Sharpe Ratio. Does one more trial change the verdict?
The momentum rule earns 0.18 and does not displace the winner; DSR moves from 0.637 to 0.642 — one more mediocre trial barely changes the benchmark. It is the number of trials and their spread that set \(SR_0\), and 42 was already enough to explain a 0.68.
Warning
Quantopian hosted hundreds of thousands of amateur and professional quants who backtested strategies on a free platform and could be allocated capital if their algorithms looked good. In 2016 its own research team (Wiecki, Campbell, Lent & Stauth, “All That Glitters Is Not Gold”) took 888 algorithms that users had deployed to paper or live trading and compared each one’s backtest with what it did afterwards, out of sample.
The in-sample Sharpe ratio had essentially no power to predict the out-of-sample Sharpe ratio; the ranking of strategies by backtest was close to useless. The one variable that did predict out-of-sample failure was the amount of backtesting the author had done — the more times a user had re-run and tuned a strategy, the worse it performed live. Quantopian’s community platform closed in November 2020, four years after its own paper had measured the problem.
Lesson for this chapter: every re-run is a trial. A backtest reported without the number of trials behind it — and without the DSR, PBO and false-discovery rate that number implies — is a description of the search, not of the strategy.
To: Head of Research, systematic equities From: <Your name>, quant research Subject: Go / no-go on the rank-target boosting signal (§7.1) and the Bollinger mean-reversion rule (§7.6) Date: 2026-10-05
Recommendation: No-go on the Bollinger rule. Paper-trade the cross-sectional rank signal at zero risk for 24 months; re-evaluate against the four statistics below.
Evidence: - Cross-sectional rank signal, walk-forward 2019–24: rank IC +0.017 (t = 1.0), top-20 − bottom-20 Sharpe 0.64; DML partial effect of momentum +0.23 % a month, t = 1.4 clustered. Five losses tried — DSR against five trials would put the best Sharpe near 0.5. - Bollinger rule, best of 42: Sharpe 0.68, bootstrap CI [0.18, 1.11] — but DSR 0.64, PBO 0.53, 0 BH survivors, and Sharpe 0.00 on the Dow. Every statistic that counts the search says noise. - Conformal coverage on the S&P band fell from 0.86 in January to 0.18 in March 2020; any live signal needs the coverage alarm attached.
Caveats: the stock file is survivorship-biased in the signal’s favour (§4.5); all numbers are before costs; 71 months cannot distinguish a Sharpe of 0.6 from 0. Ledoit–Wolf min-variance (Sharpe 1.01, turnover 0.70) or HRP (0.84, long-only) should size whatever survives — never the sample-covariance optimiser.
Next step: freeze the rank signal’s code and the list of trials in a registry; paper-trade with Ledoit–Wolf sizing and conformal width limits; require DSR > 0.95 and PBO < 0.2 on the live record before capital.
Pitfall: a copilot that builds triple-barrier labels will happily refit each January on all earlier rows. Ask what the last label in the training set depends on; if the answer includes any test-period return, it has leaked.
| Method | Question it answers | Tool |
|---|---|---|
| Learning to rank | Which stocks are in the top 20 — in what order, not by how much? | within-month rank(pct=True); pairwise logistic on \(x_i - x_j\); HistGradientBoostingRegressor on the rank; rank IC, spread Sharpe |
| Quantile + conformal | How wide is tomorrow’s range, with what guarantee — and has the world changed? | loss="quantile" at 0.05 / 0.5 / 0.95; calibration scores \(s_i\), \(\hat q\) at rank \(\lceil (n+1)(1-\alpha) \rceil\); monthly coverage; \(w = \min(\text{target}/\text{width}, w_{\max})\) |
| Double ML | What does this signal do, holding many controls fixed in any functional form? | KFold cross-fitting, two boosting nuisance fits, residual-on-residual \(\hat\theta\), influence-function SE clustered by month |
| Shrinkage + HRP | Which portfolio survives estimation error in \(\Sigma\)? | np.linalg.cond, LedoitWolf().shrinkage_, linkage on \(\sqrt{(1-\rho)/2}\), leaves_list, recursive bisection; walk-forward vol / Sharpe / DD / turnover |
| Triple barriers + meta-labeling | When did the trade end, and how much should I bet on the primary signal? | vectorised first-touch over \(\pm m\hat\sigma_t\) and \(h\); embargo of \(h\) days; LogisticRegression → \(b = \max(0, 2(p - \tfrac12))\) |
| Backtest statistics | Is the best of \(N\) trials more than luck? | stationary bootstrap; \(SR_0\) and DSR; CSCV / PBO with \(S = 8\); Benjamini–Hochberg |
The message of the course: the tools grew from a Series to a transformer and a deflated Sharpe ratio, and the discipline never changed — a fixed seed, a held-out window, a baseline to beat, and a count of everything you tried.
The rank-target trees scored IC +0.017 and the MSE trees −0.015 on identical features and folds. A colleague concludes “ranking losses work”. Write the two-sentence reply, with the standard error, and describe the experiment (data, horizon, number of stocks) that would settle it.
Conformal coverage on the S&P band fell to 0.18 in March 2020 and the rolling re-calibration only recovered in May. Design a coverage-based risk rule (threshold, window, action) and state what it would have done on 27 February 2020 — and what it would have cost in 2022, when coverage dipped to 0.62 twice.
DML gave momentum a partial effect of +0.23 % a month with t = 1.4 after clustering by month. Name two unmeasured variables that could drive both momentum and next-month return, and say which §4.4 design (DiD, instrument) could address each — or why neither can.
The Bollinger rule has a bootstrap interval that excludes zero, a DSR of 0.64, a PBO of 0.53 and no BH survivor. Rank the four statistics by how much you trust them for this question, and explain what a research head should require before a fifth statistic is added to the list.
Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python