Learning Statistics with Python
Chapter 7 · Learning Statistics with Python
Six methods that have moved from research papers into everyday practice, each built on what earlier chapters taught: learning to rank, quantile and conformal prediction, double machine learning, covariance shrinkage and cluster-based allocation, event-based labels and meta-labeling, and the statistics that decide whether a discovered pattern is real.
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
Why this matters
Chapters 3 and 5 taught the walk-forward protocol and showed that honest numbers are small. This chapter is what a practitioner does next: choose the loss the decision needs, quantify the uncertainty, isolate the effect, allocate across correlated risks, size the action — and then prove, against the number of things tried, that any of it is real.
The demonstrations use market data because it is the hardest test bed; every method is in daily use elsewhere — learning to rank in search and recommendation, conformal intervals in demand and medical forecasting, double machine learning in policy and marketing evaluation, shrinkage covariances in genomics and sensor networks, meta-labeling in fraud and screening systems, and selection statistics in large-scale A/B experimentation.
| Section | Concept | Tool |
|---|---|---|
| §7.1 Learning to rank | Rank IC, rank targets, pairwise logistic loss, boosting on ranks | rank(pct=True), LogisticRegression(fit_intercept=False) on pair differences, HistGradientBoostingRegressor |
| §7.2 Quantile and conformal | Pinball loss, split conformal, coverage as a regime alarm, sizing by width | HistGradientBoostingRegressor(loss="quantile"), calibration quantile \(\hat q\) |
| §7.3 Double machine learning | Partialling out, cross-fitting, clustered standard errors | KFold, two boosting nuisance models, residual-on-residual OLS |
| §7.4 Shrinkage and HRP | Condition number, Ledoit–Wolf, correlation distance, quasi-diagonalisation, recursive bisection | LedoitWolf, scipy.cluster.hierarchy.linkage, leaves_list |
| §7.5 Labels and meta-labeling | Triple barriers, embargo, meta-model, bet size from probability | vectorised barrier search, LogisticRegression, HistGradientBoostingClassifier |
| §7.6 Backtest statistics | Stationary bootstrap, Deflated Sharpe, CSCV / PBO, Benjamini–Hochberg | numpy, scipy.stats.norm, itertools.combinations |
Data: the §4.5 monthly cross-section (213 stocks), S&P 500 and Dow daily closes, 22 ETFs 2015–2024. Every experiment is walk-forward; every number is before costs; the stock file is survivorship-biased (§4.5).
In §4.5 you trained a boosting model to minimise squared error on next month’s return and then built a top-20 / bottom-20 portfolio from it. The portfolio never used the level of the prediction — only its order. You will show that a monotone transform of the prediction leaves the portfolio untouched while multiplying the MSE a hundredfold, then train three models that target the order directly and put all of them through the §4.5 walk-forward.
Standardise per month exactly as in §4.5 (median / std within the month, clipped at ±3). New this time: the target’s within-month percentile rank, centred at zero.
ret_next has mean 0.013 and std 0.082 with a range from −0.735 to +0.805 — a few meme months dominate any squared error. y_rank runs from −0.495 to +0.5 with std 0.289 in every month: the rank target has thrown away the level and kept the order.
Take a prediction \(\hat y\) and replace it by \(3\tanh(50\hat y)\) — strictly increasing, very different values. What happens to the top-20 / bottom-20 portfolio and to the MSE?
Same 20 names long, same 20 short, same spread — and an MSE 110 times larger. A squared-error model spends its capacity on the level of returns, which is almost all noise (§6.2); a ranking model spends it on the order, which is all the portfolio uses.
The scoreboard of §4.5 was the rank IC — Spearman correlation between prediction and realised return across the month’s stocks. Before running: is it, like the portfolio, blind to the monotone transform?
spearmanr(p1, y)[0] and spearmanr(p2, y)[0] for October 2022, to 4 dp — the same number or different?
-0.3316 -0.3316
Spearman −0.3316 for both; Pearson moves from −0.338 to −0.333. October 2022 was a month in which momentum lost — the high-momentum names fell hardest — and the rank IC records that with one number that no monotone transform can touch. It is the score that matches the trade, and the first column on every slide that follows.
Same walk-forward as §4.5: test months from 2019-01, refit every 12 months on all earlier rows, two scores per month.
71 test months, 6 refits, 10 212 training rows for the first fit. Rank IC is the Spearman correlation between prediction and realised return across the 213 stocks in a month — a score that, like the portfolio, sees only the order. The spread’s Sharpe is \(\bar s / \sigma_s \cdot \sqrt{12}\), before costs.
A linear model with the MSE loss — the loss of every regression in Chapter 4.
Mean IC +0.001 (t = 0.06): the linear MSE model ranks no better than a coin, although its extremes earn a spread of +8.5 % a year (Sharpe 0.34). A one-sd move in momentum is worth +0.26 % a month, in volatility +0.26 %, in price-to-high −0.22 % — and last-month return gets nothing (−0.004): the loss, dominated by the ±80 % months, sees no reversal at all.
Replacing ret_next by its within-month percentile changes…
IC −0.004 (t = −0.17), spread +5.0 % a year, Sharpe 0.24. Not better — with five price-based characteristics on 213 survivors there is little order to learn, whichever target you pick. Note the coefficient signs: last-month return negative (short-term reversal), 12-1 momentum positive — the two best-known cross-sectional anomalies, present but faint.
Ranking is a statement about pairs: within month \(t\), stock \(i\) should score above stock \(j\) when \(r_{i} > r_{j}\). RankNet (Burges et al., 2005) models
\[P(i \succ j) = \sigma\big(f(x_i) - f(x_j)\big), \qquad \sigma(u) = \frac{1}{1 + e^{-u}},\]
and maximises the log-likelihood of the observed orderings. With a linear scorer \(f(x) = w^\top x\) the difference is \(w^\top (x_i - x_j)\): a logistic regression, without intercept, on pair differences, label \(\mathbb 1[r_i > r_j]\).
IC −0.005 (t = −0.21), spread −1.8 % a year, Sharpe −0.09. The weights again say reversal (−0.031) and momentum (+0.026), and the model is about as good as the other two — that is, indistinguishable from zero. The pairwise loss is the right loss for a ranking problem; it cannot manufacture order that five characteristics on 213 stocks do not contain.
Same specification as §4.5 (max_depth=2, learning_rate=0.05, max_iter=200, min_samples_leaf=50); the only change is the target column.
The §4.5 number returns: IC −0.015 (t = −1.04) with the MSE loss. On the rank target the same trees give IC +0.017 (t = +0.98) and the best spread of the five, +9.3 % a year, Sharpe 0.64. The sign of the IC flipped with the target — which tells you the difference between the two is noise, not that ranks are magic.
Every mean IC lies within one standard error (0.02) of zero and the five cumulative spreads fan out from the same 2020 whipsaw. The ranking losses did what they promise — they changed which order the model learns: the MSE trees’ monthly ICs correlate only 0.31 with OLS and 0.20 with the pairwise model, while the three rank-based models move together (0.87–0.92) — but 71 months of a survivorship-biased file cannot tell 0.6 from 0. Report the protocol, not the best line.
The pairwise loss you just fitted is a preference model: the data are statements “\(i\) was better than \(j\) this month” and the score \(f(x)\) is a latent utility that rationalises them. Push the idea further and it becomes a research programme:
Note
Nothing in this slide changes the verdict of the previous one: on five characteristics and 213 survivors, no loss finds a reliable order. The loss decides what you estimate; the data decide whether there is anything to find.
evaluate builds a top-K / bottom-K spread with K=20. Re-score the rank-target boosting predictions with K=10 (more concentrated) or K=40 (more diversified) and compare the Sharpe ratio with the K = 20 value of 0.64. Which direction improves the Sharpe, and is the change larger than the noise you saw across models?
K = 10: spread +20.7 % a year, Sharpe 1.10; K = 40: +8.9 %, Sharpe 0.75. A Sharpe of 1.1 from the ten most extreme names looks like a discovery — hold that thought for §7.6, where you will learn what the best of three values of K is worth. Over 71 months none of the three is separable from the others, or from zero.
Next: §7.2 — stop predicting a number; predict an interval, and check that it keeps its promise.
§6.2’s boosting model produced one number per day and was worse than predicting zero. A trading desk rarely needs the number; it needs to know how wide the range is today — for sizing, for stops, for the risk report. You will fit three quantile models on the §6.2 features, calibrate them with a split-conformal step that comes with a coverage guarantee, then watch the guarantee fail in March 2020 and turn that failure into an alarm.
The pinball (check) loss at level \(\tau\):
\[\ell_\tau(y, q) = \begin{cases} \tau\,(y - q) & y \ge q \\ (1-\tau)\,(q - y) & y < q \end{cases}\]
Which constant \(q\) minimises \(\mathbb E[\ell_\tau(y, q)]\)?
HistGradientBoostingRegressor(loss="quantile", quantile=τ) boosts trees on this loss. Three fits — \(\tau\) = 0.05, 0.5, 0.95 — give a lower band, a median and an upper band that all depend on the features.
Same eleven lagged and rolling features as §6.2, every one shifted so that day \(t\) uses data to \(t-1\). New: a calibration window between training and test.
753 training days from January 2016, 252 calibration days, 1 258 test days that include the March 2020 crash, the 2022 bear market and two calm years. The calibration year is never used to fit a tree — that is what makes the guarantee on the next slides valid.
The median is 0.05 % — as useless as §6.2’s point forecast. The bands are not: on the quietest day of the sample (12 October 2017) the model says [−0.52 %, +0.71 %]; on the stormiest (27 December 2018) [−2.31 %, +2.01 %]. Width tracks the 21-day volatility feature with correlation 0.82: the quantile model has rediscovered GARCH’s message from §6.5 — the size is forecastable, the sign is not.
The 90 % band \([\hat q_{0.05}, \hat q_{0.95}]\) should contain \(y\) on 90 % of days. Where will empirical coverage be lowest?
0.895 in training, 0.853 on the calibration year, 0.773 on the test years — one day in four falls outside a band that promised one in ten. A quantile model’s band is a prediction, not a guarantee. The fix does not retrain anything: it measures the miss on data the model never saw, and widens the band by that much.
The quantile model’s band is a good shape and a wrong size. Conformal prediction keeps the shape and fixes the size with one number measured on data the model never saw — and comes with a proof.
For each calibration day compute the non-conformity score — how far outside the band the truth fell (negative if inside):
\[s_i = \max\big(\hat q_{0.05}(x_i) - y_i,\; y_i - \hat q_{0.95}(x_i)\big), \qquad i = 1,\dots,n.\]
Take \(\hat q\) = the \(\lceil (n+1)(1-\alpha) \rceil\)-th smallest score and report \([\hat q_{0.05}(x) - \hat q,\; \hat q_{0.95}(x) + \hat q]\).
The guarantee (Vovk; Romano, Patterson & Candès 2019)
If the calibration and test days are exchangeable, then \(P\big(y_{\text{new}} \in \text{band}\big) \ge 1 - \alpha\) — for any underlying model, however badly fitted. The model decides the shape of the band (wide on stormy days); the calibration score decides its size.
The catch is the word exchangeable. Daily returns in 2019 and daily returns in March 2020 are not draws from one urn — and the next slides measure exactly how far that assumption breaks.
The 228-th smallest of 252 scores is +0.18 %: the 2019 misses say “widen each side by 0.18 %”. Coverage on 2020–24 rises from 0.773 to 0.830 — better, still short of 0.90. By year: 0.73 in 2020, 0.87 in 2021, 0.79 in 2022, 0.89 and 0.87 after. The guarantee held where 2019 resembled the future and failed where it did not.
The band widens as the std21 feature catches up — a half-width of ±1.0 % in January, ±2.6 % by 17 March — but the returns of late February and March run ahead of it: 18 misses in March alone, 11 in April. A band built from the features of the last 21 days is a lagging description of a crash. Count the misses month by month and the lag becomes an alarm.
February 0.68, March 0.18, April 0.48: in March 2020 the band caught fewer than one day in five — 45 % of days broke through the floor and 36 % through the ceiling. The 21-day rolling coverage fell to 0.14 on 30 March. A band whose coverage collapses is telling you the distribution has moved, with a lag of a few days — a regime detector that needs no regime model (§6.4), only the counts of hits and misses.
Coverage fell to 0.18 in March 2020. What failed?
A trailing 252-day calibration lifts coverage to 0.893 — on target on average — but March 2020 is still 0.18: the window learns the new width only after the misses have happened (April 0.67, May 0.90). Conformal prediction gives you an honest average; it cannot see a regime before it arrives, only report it faster than a quarterly review.
§6.6 held \(w_t = \min(\sigma^\ast / \hat\sigma_t, w_{\max})\) of the index. Replace \(\hat\sigma_t\) by the conformal band’s width — a model-free measure of tomorrow’s range.
Weight 0.37 in March 2020, the 1.5 cap in calm months, 0.76 on average. Cumulative 41.9 % against 59.9 % for buy-and-hold, Sharpe 0.66 vs 0.56, drawdown −21.2 % vs −41.4 %. The §6.6 overlay (Sharpe 0.77, −22.6 %) did slightly better with a GARCH forecast; the interval width gets most of the way with no parametric model and a coverage guarantee attached.
Set alpha_new = 0.20 (an 80 % band) and re-run the calibration. How much narrower is the band, and what happens to test coverage? Then try 0.05.
\(\alpha\) = 0.20: \(\hat q\) = −0.14 % (the band shrinks — the raw 5/95 band already over-covers 80 %), coverage 0.714, width 2.31 %. \(\alpha\) = 0.05: \(\hat q\) = +0.39 %, coverage 0.887, width 3.35 %. Each step of coverage costs about 0.5 % of width — the price list a risk manager should know before choosing the confidence level.
loss="quantile" turns the §6.2 boosting model into a band whose width tracks volatility (correlation 0.82 with std21) while its median stays useless (0.05 %).Next: §7.3 — not “what will happen” but “what does this signal do”, holding everything else fixed.
§4.4 showed a budgeting tool that predicted repayment perfectly and did nothing — until the confounder was controlled for. Controls in a regression only work if their functional form is right. You will break a linear control with a nonlinear confounder, repair it with two boosting models and a cross-fitting trick that takes twenty lines, then ask the real question: does 12-1 momentum move next month’s return once every other characteristic and the sector are held fixed?
The Frisch–Waugh–Lovell theorem: in \(Y = \theta D + X\beta + \varepsilon\), the OLS estimate \(\hat\theta\) is the regression of the residual of \(Y\) on \(X\) on the residual of \(D\) on \(X\). Controls remove from \(Y\) and \(D\) whatever \(X\) explains; \(\theta\) is estimated from what is left.
\[\tilde Y = Y - \mathbb E[Y \mid X], \qquad \tilde D = D - \mathbb E[D \mid X], \qquad \hat\theta = \frac{\sum \tilde D_i \tilde Y_i}{\sum \tilde D_i^2}.\]
OLS with \(X\) as controls assumes…
Double machine learning (Chernozhukov et al., 2018) keeps the FWL recipe and replaces the two linear regressions by any learner — here the §4.3 boosting machine.
Three covariates; the confounder \(g(X) = 1.5\sin(2x_0) + 1.5\sin(2x_1)\) drives both the signal \(D\) and the outcome \(Y\). True effect \(\theta = 0.5\).
Naive 1.264, with linear controls 1.211 — still two and a half times the truth, with a standard error of 0.02 that says the wrong number is very precise. A linear fit explains 14 % of the confounder, so the controls closed a seventh of the backdoor and left the rest to \(\theta\). This is the §4.4 lesson in its harder form: the confounder is measured and the regression still fails.
Why it works. The residual-on-residual moment is Neyman-orthogonal: first-order errors in \(\hat m\) and \(\hat\ell\) cancel, only their product enters the bias. Two learners that are each moderately good give an estimate that is very good.
Why cross-fit. Step 2 fits on other folds so that the residual of row \(i\) never comes from a model that saw row \(i\). The next-but-one slide shows what happens if you skip this.
0.496 ± 0.023, on top of the truth, from two boosting models with depth-2 trees. The boosting nuisance models absorbed the sine curves that the linear controls missed; the residual of \(D\) kept 71 % of its spread — the part of the signal that the confounder does not explain, which is the only part that can identify \(\theta\).
Cross-fitting introduces a random element — which rows land in which fold. A method whose answer moved with the seed would be worthless.
Re-run with \(K = 2\) folds and seed 0, then \(K = 2\) and seed 1. Will both estimates stay within two standard errors (0.046) of the \(K = 5\) answer, 0.496?
yes — 0.515 and 0.525, both within 0.03 of 0.496
0.515 and 0.525 against 0.496: the split moves the estimate by about one standard error, never more. With \(K = 2\) each nuisance model sees only half the data and is a little rougher — the cost shows up as a slightly higher estimate (the residual confounding of a cruder fit), not as instability. Five folds is the usual compromise; the paper’s recommendation is to average over several splits.
Fit \(\hat m\) and \(\hat\ell\) on all \(n\) rows with a learner flexible enough to memorise, then compute \(\hat\theta\) on the same rows. The estimate will be…
Same rows: 0.377 — a quarter below the truth — because the memorising learner left \(D\) with 31 % of its spread, and that remainder is contaminated by the fit that produced it. The cross-fitted recipe gave 0.496. The bias is not in the learner; it is in letting one row be both teacher and witness — the §4.3 train/test rule applied inside an estimator.
\(D\) = the per-month z-score of 12-1 momentum; \(Y\) = next month’s return in percent; \(X\) = the other four characteristics plus sector dummies. The §4.5 panel, 25 335 stock-months.
Momentum is far from independent of the controls: correlation 0.51 with price / 252-day high (a stock near its high has usually risen over the year) and 0.10 with dollar volume. Whatever “momentum effect” a naive regression finds is partly the 52-week-high effect wearing a different label — exactly the §4.4 confounder, now with 14 of them at once.
Standard errors are clustered by month: the 213 stocks in one month share the market’s move, so treating them as independent overstates precision.
Naive +0.16 % per standard deviation of momentum; with linear controls +0.20 %; DML +0.23 % a month (2.7 % a year). Controlling raised the estimate — the 52-week-high effect was masking momentum, not manufacturing it. The controls explain 41 % of momentum’s variance. But look at the standard errors: 0.09 under the iid assumption, 0.16 clustered by month; with clustering, t = 1.4. Twenty-five thousand rows are only 119 independent months.
| Naive OLS | OLS + controls | DML | |
|---|---|---|---|
| Closes the backdoor through measured \(X\) | no | only if linear | yes, any shape |
| Robust to a badly fitted nuisance model | — | — | yes, to first order |
| Valid standard errors | with clustering | with clustering | with clustering |
| Closes the backdoor through unmeasured confounders | no | no | no |
| Turns a characteristic into a treatment | no | no | no |
The honest reading
DML answers “what is the partial association of momentum with next-month return, holding fourteen controls fixed in whatever functional form they take?”. It does not say what would happen if you made a stock’s momentum higher — there is no such intervention — and a characteristic you did not measure (analyst coverage, index membership, news flow) can still drive both. The §4.4 hierarchy stands: controls, then DiD, then an instrument.
Next: §7.4 — from one signal to a whole portfolio, and why the covariance matrix needs help before you invert it.
| ## Covariance Shrinkage and Cluster-Based Allocation (Hierarchical Risk Parity) {.divider #mod-7-4} |
| The minimum-variance portfolio has a closed form, \(w \propto \Sigma^{-1}\mathbf 1\), and everything fragile about it lives in the inverse. You will measure how ill-conditioned a 21-asset sample covariance is, watch its optimal weights swing from one year to the next, shrink the matrix with Ledoit–Wolf, and then build a portfolio that never inverts anything — hierarchical risk parity — before putting all four rules through a nine-year walk-forward. |
2 515 days, 21 assets from 6.7 % (IEF, 7–10-year Treasuries) to 40 % (USO, oil). SHY is excluded on purpose: a minimum-variance optimiser offered a 1.5 %-vol asset will buy 120 % of it and short everything else — a correct answer to an uninteresting question. The 21 that remain are the risky universe.
For 21 correlated ETFs on 251 daily returns, the condition number \(\lambda_{\max}/\lambda_{\min}\) of the sample covariance will be of the order of…
Sample: 6 373 in 2018, 5 010 in 2019. Ledoit–Wolf: 415 and 299 — fifteen times better-conditioned — from a shrinkage intensity \(\delta\) of only 0.03. The next slide shows what that does to the weights.
Sample min-variance in 2018: 75 % IEF, −29 % TLT, +10 % SPY; a year later −27 % TLT and −22 % SPY — SPY flipped from long to short on nothing but a new year of data. Total weight change 0.99 for the sample, 0.52 for Ledoit–Wolf, whose SPY weight stays near zero. Both under-predict next year’s volatility (1.5 % promised, 2.0 % delivered): the optimiser has found the noise in \(\Sigma\) and bet on it.
\[\hat\Sigma_{LW} = (1 - \delta)\, S + \delta\, \mu I, \qquad \mu = \frac{\operatorname{tr}(S)}{N}, \quad \delta^\ast = \arg\min_\delta \mathbb E\lVert \hat\Sigma_{LW} - \Sigma \rVert^2.\]
sklearn.covariance.LedoitWolf computes \(\delta^\ast\) from the data (Ledoit & Wolf, 2004); \(\delta\) = 0.03 sounds tiny, but it is applied where the matrix is weakest.Note
Alternatives in the same family: shrinkage toward a constant-correlation target, toward a one-factor (CAPM) covariance, and non-linear shrinkage that adjusts each eigenvalue separately. All exist because \(S^{-1}\) is the problem, not \(S\).
López de Prado (2016): the instability lives in \(\Sigma^{-1}\), so never compute it. Cluster the assets, order them so that neighbours are alike, and split capital down the tree two clusters at a time — estimation error stays local.
The recipe: never invert \(\Sigma\). Cluster the assets on the correlation distance \(d_{ij} = \sqrt{\tfrac12(1 - \rho_{ij})}\), reorder them so that similar assets sit together, then split the capital top-down, two clusters at a time.
Two assets with \(\rho = 0.5\) — what is their distance \(d\), to 2 dp? And for \(\rho = -1\)?
0.5 1.0
The tree puts the dollar and the metals first, then the three bond funds, then the defensive sectors, energy, credit and financials, and finally the equity indices with their sector funds. Adjacent assets correlate 0.63 on average against 0.22 in alphabetical order: the correlation matrix is now nearly block-diagonal.
Right-hand panel: a red equity block in the bottom-right, a small bond block, and the blue stripe where bonds meet equities. Recursive bisection will walk down this order and, at every split, give more capital to the half with the lower variance.
Start with all assets in tree order and weight 1. Split the list in half; compute each half’s variance as an inverse-variance portfolio; give the left half the share \(\alpha = 1 - v_L / (v_L + v_R)\); recurse into each half.
Sum 1, every weight positive, largest 0.31 (UUP, the dollar — lowest volatility in its cluster), then LQD and IEF at 0.17, gold 0.12. No short, no inverse, no eigenvalue. Estimation error stays local: a noisy correlation between two energy funds changes their split, not the weight of Treasuries.
Each January, estimate on the previous calendar year, hold for the year. Four rules: equal weight, min-variance (sample), min-variance (Ledoit–Wolf), HRP.
Before running: which rule will have the lowest volatility, and which the highest turnover?
Equal weight: vol 12.6 %, Sharpe 0.67, drawdown −34.5 %, no trading. Min-variance on the sample matrix: vol 3.2 %, Sharpe 0.82 — but it re-trades 148 % of the book every January and holds a 102 % position. Ledoit–Wolf: vol 3.6 %, Sharpe 1.01, drawdown −7.6 %, turnover 0.70 — the best Sharpe with half the trading. HRP: vol 5.4 %, Sharpe 0.84, drawdown −13.9 %, turnover 0.56, long-only, largest weight 0.51. Sharpe ratios are on raw returns, before costs, over nine years (standard error ≈ 0.33): LW’s edge over the others is suggestive, its lower turnover is certain.
Next: §7.5 — before you can train on a trade, you have to decide what “the trade succeeded” means.
Every classifier in Chapter 4 needed a label, and every label so far was “the return over the next \(h\) days was positive”. A trader who set a stop-loss would not recognise that label. You will build López de Prado’s triple-barrier label on the S&P 500, scaled by the §6.5 volatility; give a moving-average rule the job of choosing the side; and train a second model — the meta-model — to decide the size.
The fixed-horizon label \(\mathbb 1[r_{t \to t+10} > 0]\) ignores…
Triple-barrier label (López de Prado, 2018). From entry at close \(t\), three barriers: an upper profit-take at \(+m_{pt}\,\hat\sigma_t\), a lower stop-loss at \(-m_{sl}\,\hat\sigma_t\), and a vertical barrier at \(t + h\). Label +1 if the upper barrier is touched first, −1 if the lower, and the sign of the return at \(t+h\) if neither. \(\hat\sigma_t\) is the §6.5-style volatility estimate at \(t\) — barriers are wider in stormy markets.
60 % of entries are labelled +1 against 64 % for the fixed-horizon label; 76 % of entries hit a horizontal barrier within ten days, on average after six. The two labels disagree on 11.5 % of days — those are the trades where the path and the end point tell different stories, and they are the ones a stop-loss decides.
Entering on 24 February 2020 with \(\hat\sigma\) = 1.22 %: the stop at −2.44 % is hit on day 1 (label −1) — the fixed-horizon label agrees here, but the barrier label closed the trade nine days earlier. The vertical barrier caps how long a bet is allowed to be wrong; the horizontal ones cap how wrong.
A 50/200-day moving-average crossover — one of the §7.6 family — goes long when the fast average is above the slow one and short otherwise. The meta-label asks: did the side the primary chose pay, in the triple-barrier sense?
2 307 days from October 2015. The primary is long 79 % of the time and its trade “works” on 55.7 % of days — 60 % when long, 38 % when short. Seven features, all known at the close of \(t\), including the side itself: the meta-model may learn that the primary is only worth backing in some states.
Refitting each January on all earlier rows — what leaks?
1 500 test days, AUC 0.545 — the meta-model separates good from bad primary trades a little better than a coin (0.5), which is what one should expect from seven price features (§6.2). It says “back the trade” (p > 0.5) on 71 % of days.
Bet size \(b_t = \max\big(0,\; 2(p_t - \tfrac12)\big)\): nothing when the meta-model is unsure, a full position at \(p = 1\). Position \(= \text{side}_t \times b_t\), applied to the next day’s return.
Primary alone: Sharpe 0.20, drawdown −46 % (the 2020 whipsaw: the crossover went short in April and stayed short through the recovery; 2020 Sharpe −0.68). Meta-labelled and sized: Sharpe 0.77, drawdown −3.9 %, but cumulative return only 13.9 % because the mean bet is 0.16. The on/off version (full size when p > 0.5) earns 63.5 % at Sharpe 0.74 — level with buy-and-hold (0.73) at half the drawdown. The gain is the meta-model declining the 2020 shorts (2020: −0.68 → +0.79); 2022 got worse (−0.22 → −1.11). Meta-labeling improves a signal’s quality; it cannot create a signal.
Replace the logistic meta-model by HistGradientBoostingClassifier(max_depth=2, learning_rate=0.05, max_iter=150, min_samples_leaf=50, random_state=0). Does a more flexible meta-model improve the AUC or the Sharpe ratio?
Boosting: AUC 0.528, Sharpe 0.14, drawdown −11.3 % — worse than the logistic model’s 0.545 and 0.77. Seven features and about 800 training days per class are not enough for trees to find a stable state dependence, and the meta-model’s Sharpe swings by 0.6 with the choice of learner. That swing is the honest error bar on the previous slide.
Next: §7.6 — the crossover was one of 42 rules we could have picked. What is the best of 42 worth?
Every Sharpe ratio in this course was the result of a search: over features, targets, thresholds, models. The search is the part the Sharpe ratio never reports. You will backtest 42 technical rules on the S&P 500, find the best, and then put it on trial four ways — a bootstrap interval, the Deflated Sharpe Ratio, the probability of backtest overfitting, and a false-discovery correction across the whole grid — before checking whether the Dow agrees.
17 moving-average crossovers, 9 RSI mean-reversion rules, 8 Bollinger breakouts and their 8 mirror images: 42 daily P&L series over 2 266 days, each shifted so that a position decided at the close of \(t-1\) earns \(r_t\). The mirrors are deliberate — a chart-reader who “tries both directions” has doubled the trials, and the statistics below must count them.
Best: Bollinger mean reversion, 20 days, k = 2 — Sharpe 0.68, from a rule that is in the market 9 % of the time and whose P&L has skew 7 and kurtosis 154 (a few large days). Median rule 0.15, buy-and-hold 0.64, 71 % of rules positive because the index rose. A grid this size always has a winner; the question is whether 0.68 is more than the winner of 42 coin flips.
A colleague reports: “the best of my 40 rules has Sharpe 1.1 over ten years.” Is that evidence the rule works?
Expected maximum of \(N\) trials with true Sharpe 0 and cross-trial standard deviation \(\sigma_{SR}\) (Bailey & López de Prado, 2014): \[\mathbb E[\max_N SR] \approx \sigma_{SR}\Big[(1-\gamma)\,\Phi^{-1}\!\big(1 - \tfrac1N\big) + \gamma\,\Phi^{-1}\!\big(1 - \tfrac{1}{Ne}\big)\Big], \qquad \gamma = 0.5772.\]
A bootstrap interval for the rule you picked; a Sharpe deflated by the number of rules you tried; the probability that the in-sample winner loses out of sample; and a false-discovery correction across the grid. Each answers a different question — the rule must pass all four.
Politis & Romano (1994): resample the P&L in blocks of random length (geometric, mean 20 days) so that autocorrelation and volatility clusters survive; recompute the Sharpe on each pseudo-sample.
95 % interval [0.18, 1.11], only 1.7 % of resamples at or below zero. Taken alone, the interval says “significant”. But it answers the wrong question: given that we picked this rule, how uncertain is its Sharpe? It knows nothing about the 41 rules we did not pick.
Bailey & López de Prado (2014). First, the Sharpe a selection would produce by luck alone, \(SR_0\) (the expected-maximum formula, with \(\sigma_{SR}\) estimated across the \(N\) trials). Then the probability that the observed Sharpe exceeds it, allowing for non-normal returns:
\[DSR = \Phi\!\left[\frac{(\widehat{SR} - SR_0)\sqrt{T-1}}{\sqrt{1 - \gamma_3 \widehat{SR} + \frac{\gamma_4 - 1}{4}\widehat{SR}^2}}\right],\]
with \(\widehat{SR}\) per period (daily, not annualised), \(\gamma_3\) the skewness and \(\gamma_4\) the kurtosis of the P&L.
Against a benchmark of zero the rule looks real: PSR 0.990. Against the benchmark the search deserves — an expected best-of-42 of 0.57 — the Deflated Sharpe Ratio is 0.637: a 36 % chance that a Sharpe this large comes from a grid of 42 worthless rules, before the fat tails are even the point. Ten years of the S&P 500 do not distinguish this rule from luck.
Bailey, Borwein, López de Prado & Zhu (2017). Split the \(T \times N\) P&L matrix into \(S\) blocks of consecutive days. For every way of choosing \(S/2\) blocks as in-sample (the rest is out-of-sample):
PBO = 0.533: in 24 of 45 splits the rule that won in-sample was below the median out of sample, and it finished first out of sample in only 2. This is the signature of a grid with no edge — the best in-sample rule is a different rule each time and it carries nothing across the boundary.
§3.2 gave each hypothesis its own p-value. With 42 of them, control the false discovery rate: sort the p-values, find the largest \(k\) with \(p_{(k)} \le 0.05\,k/N\), reject the \(k\) smallest.
Two rules clear 0.05 on their own p-value — the Bollinger breakout and its mirror, which share one p-value (0.043) because they are the same bet with the sign flipped. Benjamini–Hochberg needs the best of 42 below 0.0012; it is 36 times too large. Zero rules survive.
Same 42 rules on dji.csv, 1985–2020: 8 774 days, nearly four times the sample.
On the Dow the S&P’s winner earns Sharpe −0.00 and ranks 32nd of 42. The Dow’s own best — RSI-21 at 30/70 — has Sharpe 0.23, DSR 0.58, PBO 0.78, no BH survivor; and the rule that won 1985–2002 (a 50/250 crossover, 0.25) earned 0.02 afterwards. Thirty-five years, two indices, four statistics: no technical rule in this grid survives the search that found it.
Add a 43rd rule to the S&P grid — 63-day time-series momentum, new_rule below — as column "TSMOM63" of pnl2, then recompute the best rule and its Deflated Sharpe Ratio. Does one more trial change the verdict?
The momentum rule earns 0.18 and does not displace the winner; DSR moves from 0.637 to 0.642 — one more mediocre trial barely changes the benchmark. It is the number of trials and their spread that set \(SR_0\), and 42 was already enough to explain a 0.68.
Warning
Quantopian hosted hundreds of thousands of amateur and professional quants who backtested strategies on a free platform and could be allocated capital if their algorithms looked good. In 2016 its own research team (Wiecki, Campbell, Lent & Stauth, “All That Glitters Is Not Gold”) took 888 algorithms that users had deployed to paper or live trading and compared each one’s backtest with what it did afterwards, out of sample.
The in-sample Sharpe ratio had essentially no power to predict the out-of-sample Sharpe ratio; the ranking of strategies by backtest was close to useless. The one variable that did predict out-of-sample failure was the amount of backtesting the author had done — the more times a user had re-run and tuned a strategy, the worse it performed live. Quantopian’s community platform closed in November 2020, four years after its own paper had measured the problem.
Lesson for this chapter: every re-run is a trial. A backtest reported without the number of trials behind it — and without the DSR, PBO and false-discovery rate that number implies — is a description of the search, not of the strategy.
To: Head of Research, systematic equities From: <Your name>, quant research Subject: Go / no-go on the rank-target boosting signal (§7.1) and the Bollinger mean-reversion rule (§7.6) Date: 2026-10-05
Recommendation: No-go on the Bollinger rule. Paper-trade the cross-sectional rank signal at zero risk for 24 months; re-evaluate against the four statistics below.
Evidence: - Cross-sectional rank signal, walk-forward 2019–24: rank IC +0.017 (t = 1.0), top-20 − bottom-20 Sharpe 0.64; DML partial effect of momentum +0.23 % a month, t = 1.4 clustered. Five losses tried — DSR against five trials would put the best Sharpe near 0.5. - Bollinger rule, best of 42: Sharpe 0.68, bootstrap CI [0.18, 1.11] — but DSR 0.64, PBO 0.53, 0 BH survivors, and Sharpe 0.00 on the Dow. Every statistic that counts the search says noise. - Conformal coverage on the S&P band fell from 0.86 in January to 0.18 in March 2020; any live signal needs the coverage alarm attached.
Caveats: the stock file is survivorship-biased in the signal’s favour (§4.5); all numbers are before costs; 71 months cannot distinguish a Sharpe of 0.6 from 0. Ledoit–Wolf min-variance (Sharpe 1.01, turnover 0.70) or HRP (0.84, long-only) should size whatever survives — never the sample-covariance optimiser.
Next step: freeze the rank signal’s code and the list of trials in a registry; paper-trade with Ledoit–Wolf sizing and conformal width limits; require DSR > 0.95 and PBO < 0.2 on the live record before capital.
Pitfall: a copilot that builds triple-barrier labels will happily refit each January on all earlier rows. Ask what the last label in the training set depends on; if the answer includes any test-period return, it has leaked.
| Method | Question it answers | Tool |
|---|---|---|
| Learning to rank | Which stocks are in the top 20 — in what order, not by how much? | within-month rank(pct=True); pairwise logistic on \(x_i - x_j\); HistGradientBoostingRegressor on the rank; rank IC, spread Sharpe |
| Quantile + conformal | How wide is tomorrow’s range, with what guarantee — and has the world changed? | loss="quantile" at 0.05 / 0.5 / 0.95; calibration scores \(s_i\), \(\hat q\) at rank \(\lceil (n+1)(1-\alpha) \rceil\); monthly coverage; \(w = \min(\text{target}/\text{width}, w_{\max})\) |
| Double ML | What does this signal do, holding many controls fixed in any functional form? | KFold cross-fitting, two boosting nuisance fits, residual-on-residual \(\hat\theta\), influence-function SE clustered by month |
| Shrinkage + HRP | Which portfolio survives estimation error in \(\Sigma\)? | np.linalg.cond, LedoitWolf().shrinkage_, linkage on \(\sqrt{(1-\rho)/2}\), leaves_list, recursive bisection; walk-forward vol / Sharpe / DD / turnover |
| Triple barriers + meta-labeling | When did the trade end, and how much should I bet on the primary signal? | vectorised first-touch over \(\pm m\hat\sigma_t\) and \(h\); embargo of \(h\) days; LogisticRegression → \(b = \max(0, 2(p - \tfrac12))\) |
| Backtest statistics | Is the best of \(N\) trials more than luck? | stationary bootstrap; \(SR_0\) and DSR; CSCV / PBO with \(S = 8\); Benjamini–Hochberg |
The message of the course: the tools grew from a Series to a transformer and a deflated Sharpe ratio, and the discipline never changed — a fixed seed, a held-out window, a baseline to beat, and a count of everything you tried.
The rank-target trees scored IC +0.017 and the MSE trees −0.015 on identical features and folds. A colleague concludes “ranking losses work”. Write the two-sentence reply, with the standard error, and describe the experiment (data, horizon, number of stocks) that would settle it.
Conformal coverage on the S&P band fell to 0.18 in March 2020 and the rolling re-calibration only recovered in May. Design a coverage-based risk rule (threshold, window, action) and state what it would have done on 27 February 2020 — and what it would have cost in 2022, when coverage dipped to 0.62 twice.
DML gave momentum a partial effect of +0.23 % a month with t = 1.4 after clustering by month. Name two unmeasured variables that could drive both momentum and next-month return, and say which §4.4 design (DiD, instrument) could address each — or why neither can.
The Bollinger rule has a bootstrap interval that excludes zero, a DSR of 0.64, a PBO of 0.53 and no BH survivor. Rank the four statistics by how much you trust them for this question, and explain what a research head should require before a fifth statistic is added to the list.
Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python