Chapter 7 · Learning Statistics with Python
Order, intervals, partial effects, allocation, event labels and selection uncertainty — followed by a revisable coding reference.
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
The market examples also illustrate shortlist decisions, demand forecasting, policy evaluation and two-stage screening.
Follow the numbered parts in each section: concept → prediction → example → explanation → check.
The 3.5-hour class selects examples, group discussion and notebook work. The complete Slides and Book support self-study.
Each setup links to its earlier introduction. Change inputs in the current cell before the operation you test.
| Section | Concept | Tool |
|---|---|---|
| §7.1 Learning to rank | Rank IC, rank targets, pairwise logistic loss, boosting on ranks | rank(pct=True), LogisticRegression(fit_intercept=False) on pair differences, HistGradientBoostingRegressor |
| §7.2 Quantile and conformal | Pinball loss, split conformal, coverage as a regime alarm, sizing by width | HistGradientBoostingRegressor(loss="quantile"), calibration quantile \(\hat q\) |
| §7.3 Double machine learning | Partialling out, cross-fitting, clustered standard errors | KFold, two boosting nuisance models, residual-on-residual OLS |
| §7.4 Shrinkage and HRP | Condition number, Ledoit–Wolf, correlation distance, quasi-diagonalisation, recursive bisection | LedoitWolf, scipy.cluster.hierarchy.linkage, leaves_list |
| §7.5 Labels and meta-labeling | Triple barriers, embargo, meta-model, bet size from probability | vectorised barrier search, LogisticRegression, HistGradientBoostingClassifier |
| §7.6 Backtest statistics | Stationary bootstrap, Deflated Sharpe, CSCV / PBO, Benjamini–Hochberg | numpy, scipy.stats.norm, itertools.combinations |
| §7.7 Revisable coding reference | Retrieve, inspect, correct and test source-linked evidence | TF–IDF, cosine similarity, source fingerprints; notebook lab |
Data: 213-stock monthly cross-section (§4.5), S&P 500 and Dow daily closes, 22 ETF price columns (21 retained return series), 2015–2024. Performance examples are chronological and before costs; the stock panel is survivorship-biased. Random-row DML folds and CSCV serve different purposes and are not chronological walk-forward validation.
| Part | Question |
|---|---|
| 7.1.1 Ranking objectives and evaluation | Does the decision require an accurate level or ordering? |
| 7.1.2 Pointwise and pairwise learners | How can a model learn the desired ordering? |
| 7.1.3 Common-protocol comparison and concentration | Does a changed objective improve the same decision? |
Question: Does the decision require an accurate level or ordering?
Result: Define date-level rank IC and a top-minus-bottom comparison.
Standardise within each month as in §4.5: subtract the cross-sectional median, divide by sample SD, clip at ±3 and fill missing features with zero. Add rank(pct=True) − 0.5 for the realised target; it is approximately centred.
Raw returns span −0.735 to +0.805 (mean 0.013, SD 0.082). Rank targets have comparable bounded scale; finite groups and averaged ties prevent an exactly uniform continuous distribution.
Take a prediction \(\hat y\) and replace it by \(3\tanh(50\hat y)\) — strictly increasing, very different values. What happens to the top-20 / bottom-20 portfolio and to the MSE?
The scoreboard of §4.5 was the rank IC — Spearman correlation between prediction and realised return across the month’s stocks. Before running: is it, like the portfolio, blind to the monotone transform?
spearmanr(p1, y)[0] and spearmanr(p2, y)[0] for October 2022, approximately — the same number or different?
-0.3316 -0.3316
Same walk-forward as §4.5: test months from 2019-01, refit every 12 months on all earlier rows, two scores per month.
Rank IC compares ordering; the equal-weight spread compares realised extreme-group returns. Annualised spread Sharpe is \(\bar s/s_s\sqrt{12}\), before costs; serial dependence can invalidate the usual scaling.
Question: How can a model learn the desired ordering?
Result: Compare raw-return, rank-target and pairwise losses.
Builds on Ranking objectives and evaluation.
A linear model with the MSE loss — the loss of every regression in Chapter 4.
OLS: mean IC +0.0014 (t = 0.06), gross spread +8.5%/yr, Sharpe 0.34. Final-fit coefficients per z-unit: momentum +0.26%, volatility +0.26%, price/high −0.22%, last return −0.004%. These weak diagnostics do not establish useful ranking performance.
Replacing ret_next by its within-month percentile changes…
Rank-target OLS: IC −0.004 (t = −0.17), gross spread +5.0%/yr, Sharpe 0.24. Final coefficients show negative last-month return and positive momentum, but these data do not establish reliable effects.
Ranking is a statement about pairs: within month \(t\), stock \(i\) should score above stock \(j\) when \(r_{i} > r_{j}\). RankNet (Burges et al., 2005) models
\[P(i \succ j) = \sigma\big(f(x_i) - f(x_j)\big), \qquad \sigma(u) = \frac{1}{1 + e^{-u}},\]
With \(f(x)=w^\top x\), use a no-intercept logistic regression on \(x_i-x_j\), with label \(\mathbb 1[r_i>r_j]\). LogisticRegression fits these pair-order probabilities with its default L2 penalty (C=1). Swapping a non-tied pair negates the margin and reverses the label.
Pairwise logit: IC −0.005 (t = −0.21), gross spread −1.8%/yr, Sharpe −0.09. Final weights include reversal −0.031 and momentum +0.026. The objective targets pair order; it does not ensure a useful forecast.
Same specification as §4.5 (max_depth=2, learning_rate=0.05, max_iter=200, min_samples_leaf=50); the only change is the target column.
HGB-MSE: IC −0.015 (t = −1.04), gross spread +7.7%/yr, Sharpe 0.47. HGB-rank: IC +0.017 (t = +0.98), +9.3%/yr, Sharpe 0.64. A sign change is not evidence of superiority; compare paired monthly differences and account for selection.
Question: Does a changed objective improve the same decision?
Result: Compare learners on matched dates and test portfolio concentration.
Builds on Pointwise and pairwise learners.
The average IID SE rounds to 0.02; HGB-MSE has |t|=1.04, so “all within one own SE” is not exact. Rank-model monthly IC correlations are 0.81–0.92; MSE-tree correlations with the others are 0.20–0.31. Report the matched protocol and all results, including dependence, survivor bias and costs.
The score \(f\) acts as latent utility. Pairwise comparisons can extend from linear \(w\) to the §4.5 attention scorer; listwise losses can emphasise the top of an order. Preference-based asset-pricing research can combine estimation and portfolio objectives. Required score calibration depends on the actual decision.
Note
This small survivor-panel demonstration does not establish a reliable ordering. Changing the objective changes the estimation target; data quality, protocol and uncertainty determine what the evidence supports.
evaluate builds a top-K / bottom-K spread with K=20. Re-score the rank-target boosting predictions with K=10 (more concentrated) or K=40 (more diversified) and compare the Sharpe ratio with the K = 20 value of 0.64. Which direction improves the Sharpe, and is the change larger than the noise you saw across models?
The best of three tested K values is selected evidence (§7.6). Comparisons are paired across months and require uncertainty; a higher observed Sharpe does not establish superiority.
Next: §7.2 — stop predicting a number; predict an interval, and check that it keeps its promise.
| Part | Question |
|---|---|
| 7.2.1 Conditional quantiles and interval checks | How do we predict a range instead of a point? |
| 7.2.2 Conformal calibration | How does held-out calibration adjust an interval? |
| 7.2.3 Coverage under drift and risk applications | What happens when future data differ from calibration? |
Question: How do we predict a range instead of a point?
Result: Fit quantiles and measure interval coverage and width.
The pinball (check) loss at level \(\tau\):
\[\ell_\tau(y, q) = \begin{cases} \tau\,(y - q) & y \ge q \\ (1-\tau)\,(q - y) & y < q \end{cases}\]
Which constant \(q\) minimises \(\mathbb E[\ell_\tau(y, q)]\)?
HistGradientBoostingRegressor(loss="quantile", quantile=τ) uses this loss. Fits at 0.05, 0.5 and 0.95 supply conditional lower, median and upper predictions; check for crossing.
Same eleven lagged and rolling features as §6.2, every one shifted so that day \(t\) uses data to \(t-1\). New: a calibration window between training and test.
The 90 % band \([\hat q_{0.05}, \hat q_{0.95}]\) should contain \(y\) on 90 % of days. Where will empirical coverage be lowest?
Coverage deteriorates across these samples. Estimation error, temporal dependence and changing conditions require investigation; the loss does not guarantee unseen coverage.
Question: How does held-out calibration adjust an interval?
Result: A calibration quantile with its exchangeability assumptions stated.
Builds on Conditional quantiles and interval checks.
Conformal calibration changes both edges by a held-out score quantile. Its coverage statement is marginal and depends on exchangeability, rather than proving each day’s band correct.
For each calibration day compute the non-conformity score — how far outside the band the truth fell (negative if inside):
\[s_i = \max\big(\hat q_{0.05}(x_i) - y_i,\; y_i - \hat q_{0.95}(x_i)\big), \qquad i = 1,\dots,n.\]
Take \(\hat q\) as the \(\lceil(n+1)(1-\alpha)\rceil\)-th smallest calibration score. Report \([\hat q_{0.05}(x)-\hat q,\hat q_{0.95}(x)+\hat q]\); use \(+\infty\) if the required rank exceeds \(n\).
The guarantee (Vovk; Romano, Patterson & Candès 2019)
With the model held fixed after a disjoint fit, exchangeable calibration/new scores give marginal \(P(y_{new}\in\text{band})\ge1-\alpha\). This is not a guarantee for each date, feature group or realised test period.
Exchangeability is an assumption, not a property established by the chronological split. Dependence and distribution change can invalidate the ordinary guarantee.
Test coverage rises from 0.773 to 0.830, below 0.90. By year: 0.727, 0.869, 0.789, 0.892 and 0.873. Realised rates do not prove that a marginal guarantee held or failed in a particular year.
The interval responds to lagged features. January–June misses: 3, 6, 18, 11, 6 and 5 days. Count a miss only after the target return arrives.
Question: What happens when future data differ from calibration?
Result: Diagnose time-varying coverage before using width for sizing.
Builds on Conformal calibration.
21-day coverage reaches 0.143 on 30 March. This is a retrospective monitoring signal, not proof of a changed distribution or a forecast of the next regime.
Coverage fell to 0.18 in March 2020. What failed?
Observed average coverage rises to 0.893; March stays 0.18, April 0.67 and May 0.90. Rolling calibration adapts after observations arrive; it does not restore the ordinary guarantee under arbitrary drift.
As in §6.6, inverse forecast risk sets exposure. Here use \(w_t=\min(0.02/\text{width}_t,1.5)\): target a two-percentage-point interval width. Its ingredients are known before the target return; it is not a model-free or guaranteed-profit rule.
Mean exposure 0.76. Additive log-return diagnostics: sum 41.9% vs 59.9%, annualised Sharpe 0.66 vs 0.56, drawdown of cumulative log-return sum −21.2 vs −41.4 percentage points. Exact NAV comparison also needs simple returns, financing, costs and opening wealth.
Set alpha_new = 0.20 (an 80 % band) and re-run the calibration. How much narrower is the band, and what happens to test coverage? Then try 0.05.
Next: §7.3 — not “what will happen” but “what does this signal do”, holding everything else fixed.
| Part | Question |
|---|---|
| 7.3.1 Controls and residualisation | What variation remains after controlling for covariates? |
| 7.3.2 Cross-fitting and orthogonal estimation | How do we reduce bias from flexible nuisance fits? |
| 7.3.3 Empirical interpretation and identification limits | Which causal assumptions still require justification? |
FWL is exact sample OLS algebra. Causal interpretation needs assumptions beyond residualisation.
FWL: regress both \(Y\) and \(D\) on the same control matrix, including an intercept, then regress their residuals. This gives exactly the multiple-OLS coefficient on \(D\). The population analogue replaces linear projections with conditional means:
\[\tilde Y = Y - \mathbb E[Y \mid X], \qquad \tilde D = D - \mathbb E[D \mid X], \qquad \hat\theta = \frac{\sum \tilde D_i \tilde Y_i}{\sum \tilde D_i^2}.\]
The choices refer to treating the linear nuisance fits as correctly specified conditional means.
OLS with \(X\) as controls assumes…
DML estimates the two conditional means with flexible learners, then uses cross-fitting and an orthogonal moment (§7.3.2). Nuisance quality and identification assumptions still matter.
Three covariates; the confounder \(g(X) = 1.5\sin(2x_0) + 1.5\sin(2x_1)\) drives both the signal \(D\) and the outcome \(Y\). True effect \(\theta = 0.5\).
Naive 1.264; linear controls 1.211 versus truth 0.5. The linear fit explains 13.9% of \(g\) variance; this is not a measured fraction of a causal backdoor closed.
Cross-fitting supplies residuals from models trained on other rows; the next recipe gives the moment and its uncertainty calculation.
For held-out row \(i\), \(\tilde D_i=D_i-\hat m_{-k(i)}(X_i)\) and \(\tilde Y_i=Y_i-\hat\ell_{-k(i)}(X_i)\). Combine all folds:
\[\hat\theta=\frac{\sum_i\tilde D_i\tilde Y_i}{\sum_i\tilde D_i^2},\qquad \psi_i=\tilde D_i(\tilde Y_i-\hat\theta\tilde D_i),\qquad \widehat{se}_{iid}=\frac{\sqrt{\sum_i\psi_i^2}}{\sum_i\tilde D_i^2}.\]
Neyman orthogonality means the population moment’s nuisance derivative is zero at the truth. Its second-order remainder includes \(\mathbb E[\delta m(\delta\ell-\theta\delta m)]\), so both product and squared treatment-nuisance errors matter. A sufficient rate condition is \(\|\delta m\|_2(\|\delta\ell\|_2+\|\delta m\|_2)=o(n^{-1/2})\); two errors merely \(O(n^{-1/4})\) do not ensure negligible bias. Independent-row splitting, nonzero residual variation and regularity conditions are also required.
KFold supplies row splits; HistGradientBoostingRegressor is the histogram boosting learner used for the nuisance fits. Both are from scikit-learn; the split is explained in §7.3.2 and boosting in §4.3.
DML retains 71% of raw \(D\) standard deviation. If residual \(D\) is identically zero, the slope is unidentified, not an estimate with an infinite standard error. Shuffled folds suit these independent simulated rows.
Compare two two-fold seeds with the existing five-fold estimate. These are split-sensitivity diagnostics for the same simulated sample.
Re-run with \(K = 2\) folds and seed 0, then \(K = 2\) and seed 1. Will both estimates stay within two standard errors (0.046) of the \(K = 5\) answer, 0.496?
yes — 0.515 and 0.525, both within 0.03 of 0.496
Both estimates are close on this sample. This does not establish seed stability generally or explain their differences as residual confounding. Repeat splits and report their variation; the DML paper uses median aggregation.
Read the choices for the seeded learner below; overfitting has no universal bias direction.
Scope: the fitted same-row learner in this chapter’s seeded example. The direction of bias is not a universal property of same-row residuals.
Fit \(\hat m\) and \(\hat\ell\) on all \(n\) rows with a learner flexible enough to memorise, then compute \(\hat\theta\) on the same rows. The estimate will be…
The same-row learner gives 0.377 here. The comparison also changes learner flexibility, so it does not isolate cross-fitting alone; bias direction is not universal and exact memorisation can make the slope undefined.
Question: Which causal assumptions still require justification?
Result: Compare estimates without treating machine learning as identification.
Builds on Cross-fitting and orthogonal estimation.
\(D\) = the per-month z-score of 12-1 momentum; \(Y\) = next month’s return in percent; \(X\) = the other four characteristics plus sector dummies. The §4.5 panel, 25 335 stock-months.
25 335 stock-months, 14 controls and 119 months. Momentum correlates 0.51 with price relative to the 252-day high and 0.10 with dollar volume; correlation alone does not identify confounding or explain a coefficient change. The input is median-centred, divided by the within-month standard deviation, clipped and missing values set to zero: a coefficient unit is one pre-clipping input-scale unit, not necessarily one realised standard deviation.
Month clustering includes within-month dependence by summing influence contributions before squaring. The displayed formula has no finite-cluster adjustment; independence across months is a further assumption.
Naive +0.156%, linear-adjusted +0.201%, DML +0.226% per input-scale unit. The latter is about 2.7 percentage points under simple ×12 annual scaling, not a realised annual return. The nuisance fit leaves 59% of \(D\) variance (about 77% of its standard deviation). A coefficient increase alone does not attribute the change to one control.
The panel still uses shuffled row folds. Month clustering does not make the 119 months independent or correct cross-fold stock/month dependence; credible inference needs dependence-aware splits and covariance assumptions. Cluster-robust DML treats splitting and inference together.
OLS uses a chosen linear control space; DML estimates flexible nuisance functions. Neither guarantees adequate fit or identification. The iid or clustered SE is valid only under its matching sampling and rate conditions.
The honest reading
Read the panel coefficient as a fitted partial association. A causal effect requires a meaningful intervention, consistency, relevant residual variation and justified no-unmeasured-confounding assumptions. DiD and instruments have their own assumptions; they are alternative designs, not an automatic hierarchy of credibility.
Next: §7.4 — from one signal to a whole portfolio, and why the covariance matrix needs help before you invert it.
| Part | Question |
|---|---|
| 7.4.1 Covariance instability and shrinkage | Why can small covariance errors create extreme weights? |
| 7.4.2 Hierarchical risk parity | How does a correlation tree determine an allocation? |
| 7.4.3 Walk-forward allocation comparison | Which trade-offs survive a common evaluation period? |
Minimum variance permits shorts here: \(w=\Sigma^{-1}\mathbf1/(\mathbf1^\top\Sigma^{-1}\mathbf1)\). Leverage and estimation risk must be assessed separately.
2 515 daily observations, 2015–2024, across 21 risky ETFs after excluding SHY, a low-volatility short-Treasury fund rather than literal cash. The universe covers Treasuries, credit, currencies, metals, oil, equity indices and sectors; annualised sample volatility ranges from 6.7% (IEF) to 40.1% (USO). The exclusion is a modelling choice, not a universal allocation rule.
For 21 correlated ETFs on 251 daily returns, the condition number \(\lambda_{\max}/\lambda_{\min}\) of the sample covariance will be of the order of…
The archive gives condition numbers 6 373 → 415 in 2018 and 5 010 → 299 in 2019 under Ledoit–Wolf. Intensities are 0.029 and 0.034. These are conditioning diagnostics, not a fixed conversion from 1% covariance error into a stated inverse error.
For 2018 targets, predicted versus 2019 realised annualised weighted-log volatility is 1.54% → 2.00% (sample) and 1.82% → 2.39% (LW). Estimation error and a changing covariance can both contribute; these realised gaps do not isolate their causes.
\[\hat\Sigma_{LW} = (1 - \delta)\, S + \delta\, \mu I, \qquad \mu = \frac{\operatorname{tr}(S)}{N}, \quad \delta^\ast = \arg\min_\delta \mathbb E\lVert \hat\Sigma_{LW} - \Sigma \rVert^2.\]
\(\mu I\) uses one estimated average variance and zero off-diagonals; it still has estimation error. LedoitWolf estimates the shrinkage intensity from data. Its empirical \(S\) uses denominator \(T\); the earlier np.cov uses \(T-1\), so trace preservation applies to the matching base matrix. Estimator definition.
Note
Alternatives in the same family: shrinkage toward a constant-correlation target, toward a one-factor (CAPM) covariance, and non-linear shrinkage that adjusts each eigenvalue separately. All exist because \(S^{-1}\) is the problem, not \(S\).
Question: How does a correlation tree determine an allocation?
Result: Trace distance, clustering and recursive bisection into weights.
Builds on Covariance instability and shrinkage.
HRP avoids a full covariance inverse: distance → single-linkage tree → leaf order → bisection of the ordered list. Its weights still depend on estimated covariances and the tree.
Correlation distance is \(d_{ij}=\sqrt{(1-\rho_{ij})/2}\). Single-linkage merges the nearest pair across two clusters; leaves_list returns their ordered leaves.
Two assets with \(\rho = 0.5\) — what is their distance \(d\), approximately? And for \(\rho = -1\)?
0.5 1.0
The original 2018 tree places dollar/metals, bond funds and equity-sector groups near one another. Adjacent mean correlation is 0.63 versus 0.22 in file order; reordering preserves every pair correlation and does not make off-block correlations vanish.
The matrix shows estimated blocks and negative relationships on the same 2018 sample. The next operation bisects its leaf list, which need not follow each dendrogram merge.
Start with all assets in tree order and weight 1. Split the list in half; compute each half’s variance as an inverse-variance portfolio; give the left half the share \(\alpha = 1 - v_L / (v_L + v_R)\); recurse into each half.
The original weights sum to one and are positive: UUP 0.308, LQD and IEF about 0.17, GLD 0.12. Positivity requires well-defined positive cluster variances. A covariance change can affect ancestor splits or the tree, so errors are not guaranteed to stay local.
Question: Which trade-offs survive a common evaluation period?
Result: Compare risk, return and concentration without selecting on one metric.
Builds on Hierarchical risk parity.
Four rules share the same dates and universe. The code multiplies each daily asset log return by fixed annual target weights: a fixed-weight log-return approximation, not an exact buy-and-hold portfolio accounting model. Predict the observed ordering before running; no rule guarantees lower realised risk.
Before running: which rule will have the lowest volatility, and which the highest turnover?
The table’s turnover/yr is mean annual \(\sum_i|w_{new,i}-w_{old,i}|\): equal targets give zero but maintaining equal weights can require trading. It excludes drift, initial funding, fees and borrowing. maxDD is a decline in cumulative weighted log returns, with opening zero omitted; it is not compounded NAV drawdown. The Sharpe ordering is descriptive; dependence and nine years of data limit precision.
Next: §7.5 — before you can train on a trade, you have to decide what “the trade succeeded” means.
| Part | Question |
|---|---|
| 7.5.1 Triple-barrier labels | Which event determines a trade’s outcome? |
| 7.5.2 Primary signals and the meta-model | Should we act on an existing directional signal? |
| 7.5.3 Probability, position size and comparison | How does the second-stage probability affect exposure? |
Question: Which event determines a trade’s outcome?
Result: Trace upper, lower and time barriers and their label end times.
The fixed-horizon label \(\mathbb 1[r_{t \to t+10} > 0]\) ignores…
Triple-barrier label: at entry close \(t\), freeze volatility \(\hat\sigma_t\) and barriers \(+m_{pt}\hat\sigma_t\), \(-m_{sl}\hat\sigma_t\), \(t+h\). First observed upper/lower touch gives +1/−1; otherwise use the return sign at the vertical barrier. Here observations are daily closes, in log-return units.
All rates use complete ten-day paths with a known entry volatility. Different labels describe different decisions; a close-only crossing is not an intraday stop execution.
The code records crossing dates, not executable fills. Gaps, intraday moves and slippage can produce losses beyond a stated stop level.
Question: Should we act on an existing directional signal?
Result: Separate direction from acceptance and respect overlapping label horizons.
Builds on Triple-barrier labels.
The primary moving-average rule supplies long/short direction. The meta-target is sign agreement with the realised event label; the meta-model predicts acceptance or size.
Seven features are known by close \(t\). A positive vertical exit can also give meta_y = 1; the target is not restricted to horizontal profit touches.
Refitting each January on all earlier rows — what leaks?
The code fits the scaler and classifier only on retained earlier rows. AUC 0.545 is a sample ranking result, with dependent overlapping labels; it is not a calibrated trading probability.
Question: How does the second-stage probability affect exposure?
Result: A stated probability-to-size rule checked on later observations.
Builds on Primary signals and the meta-model.
Bet size \(b_t = \max(0,\,2(p_t-\tfrac12))\); signed daily exposure is \(\text{side}_t b_t\). The next-day calculation is a gross mark-to-market proxy, distinct from the up-to-ten-day event label.
The sizing rule is a heuristic: calibration, costs and feasible execution need separate checks. The code sums weighted log returns and measures cumulative-log drawdown; these are not compounded net NAV statistics. Sized exposure averages 0.16.
Replace the logistic meta-model by HistGradientBoostingClassifier(max_depth=2, learning_rate=0.05, max_iter=150, min_samples_leaf=50, random_state=0). Does a more flexible meta-model improve the AUC or the Sharpe ratio?
Observed learner sensitivity is 0.77 versus 0.14 Sharpe. This comparison is another research choice, not an estimated sampling standard error or proof that trees cannot work.
Next: §7.6 — the crossover was one of 42 rules we could have picked. What is the best of 42 worth?
| Part | Question |
|---|---|
| 7.6.1 The search space and winner’s bias | How many alternatives were tried before choosing a winner? |
| 7.6.2 Uncertainty and selection-adjusted statistics | Does the apparent winner survive different evidence checks? |
| 7.6.3 External comparison and research revision | Does the conclusion survive another stated comparison? |
Question: How many alternatives were tried before choosing a winner?
Result: A complete rule grid and an explanation of selection bias.
All 42 configurations enter the recorded search, including mirrors. Their dependence means the raw configuration count is not an estimated number of independent trials. Positions use yesterday’s close for today’s gross return.
Winner: BBmr20 k=2.0, annualised Sharpe 0.676; active on 9.1% of days, skew 7.1 and raw kurtosis 154. A maximum exists even without skill. Establish selection-adjusted evidence before using its gross return.
Benchmark for the choices: 40 independent, zero-mean Normal Sharpe estimates, each with standard deviation 0.3. Actual correlated trials require a qualified selection benchmark.
A colleague reports: “the best of my 40 rules has Sharpe 1.1 over ten years.” Is that evidence the rule works?
Expected maximum under independent Normal trial estimates, true Sharpe 0 and common spread \(\sigma_{SR}\) (Bailey & López de Prado, 2014): \[\mathbb E[\max_N SR] \approx \sigma_{SR}\Big[(1-\gamma)\,\Phi^{-1}\!\big(1 - \tfrac1N\big) + \gamma\,\Phi^{-1}\!\big(1 - \tfrac{1}{Ne}\big)\Big], \qquad \gamma = 0.5772.\]
Question: Does the apparent winner survive different evidence checks?
Result: Distinguish bootstrap uncertainty, deflation, overfitting probability and FDR.
Builds on The search space and winner’s bias.
The four diagnostics answer different questions: uncertainty for the selected series, a search benchmark, rank stability and family-level discoveries. Their assumptions overlap; passing them is not a validated net-investment decision.
Random geometric block lengths have mean 20 days. Copying runs preserves local dependence, subject to stationarity and block-length assumptions.
The interval keeps the selected rule fixed. It does not repeat the search or supply a selection-adjusted significance test.
Bailey & López de Prado (2014). First, the Sharpe a selection would produce by luck alone, \(SR_0\) (the expected-maximum formula, with \(\sigma_{SR}\) estimated across the \(N\) trials). Then a Normal-approximation one-sided statistic against that benchmark, with skew and kurtosis in its standard-error expression:
\[DSR = \Phi\!\left[\frac{(\widehat{SR} - SR_0)\sqrt{T-1}}{\sqrt{1 - \gamma_3 \widehat{SR} + \frac{\gamma_4 - 1}{4}\widehat{SR}^2}}\right],\]
with \(\widehat{SR}\) per period (daily, not annualised), \(\gamma_3\) the skewness and \(\gamma_4\) the kurtosis of the P&L.
\(\gamma_4\) is raw kurtosis. For a positive Sharpe, greater kurtosis or more negative skew widens the denominator, other terms fixed. Use daily Sharpe and independent-trial assumptions; this Normal approximation does not correct serial correlation. \(DSR=\Phi(z)\) is a one-sided tail complement, not posterior odds of skill. Method.
Under the reported 42-trial assumption: PSR 0.990 versus zero, DSR 0.637 versus expected maximum 0.57 annualised. The difference shows benchmark sensitivity; 1 − DSR is not a 36% probability that all rules are worthless.
Bailey, Borwein, López de Prado & Zhu (2017). Split the \(T \times N\) P&L matrix into \(S\) blocks of consecutive days. For every way of choosing \(S/2\) blocks as in-sample (the rest is out-of-sample):
\(S=8\) gives \(\binom84=70\) possible subsets; the cell samples 45 without replacement (seed 0), after dropping two trailing days. Each subset selects a winner and its complement ranks it. Arbitrary subsets can put later blocks in-sample and earlier ones out-of-sample: CSCV is a symmetric selection diagnostic, not chronological walk-forward. Method.
24/45 sampled logits are nonpositive; the selected rule is OOS-best in 2/45. This diagnoses unstable selection in these samples, not proof that every rule’s population edge is zero.
§3.2 gave each hypothesis its own p-value. With 42 of them, control the false discovery rate: sort the p-values, find the largest \(k\) with \(p_{(k)} \le 0.05\,k/N\), reject the \(k\) smallest.
The code finds no crossing anywhere in this 42-test family, so it rejects none. Mirrored rules share a two-sided p-value. Valid null p-values and independence/PRDS are required for nominal BH control; these naive dependent-return p-values do not establish that guarantee.
Question: Does the conclusion survive another stated comparison?
Result: An updated search record and a decision that includes null evidence.
Builds on Uncertainty and selection-adjusted statistics.
Same 42 rules on dji.csv, 1985–2020: 8 774 days, nearly four times the sample.
The frozen S&P winner ranks 32/42 on Dow; selecting Dow’s own winner is another search. These indices overlap in dates and constituents, so this is an external descriptive comparison, not an independent replication. Neither sample establishes net returns after costs.
Add a 43rd rule to the S&P grid — 63-day time-series momentum, new_rule below — as column "TSMOM63" of pnl2, then recompute the best rule and its Deflated Sharpe Ratio. Does one more trial change the verdict?
The momentum rule earns 0.18 and does not displace the winner; DSR moves from 0.637 to 0.642 — one more mediocre trial barely changes the benchmark. It is the number of trials and their spread that set \(SR_0\), and 42 was already enough to explain a 0.68.
Warning
Wiecki, Campbell, Lent and Stauth studied 888 filtered Quantopian algorithms, backtested over 2010–2015 with at least six months after code creation reserved as out-of-sample simulation. Backtest Sharpe had little predictive value (\(R^2 < 0.025\)); risk and portfolio-construction features added information. More backtesting was associated with a larger backtest–out-of-sample discrepancy. This was neither a test of live fills nor evidence that backtesting was the only informative variable. Original study.
Lesson: record the search history and assess genuinely held-out evidence; one selected Sharpe does not describe the full search.
| Part | Question |
|---|---|
| 7.7.1 Evidence, provenance and correction | Which source supports the proposed coding pattern? |
| 7.7.2 Retrieve, inspect and detect stale evidence | Is the retrieved example current and eligible? |
| 7.7.3 Adaptation, checks and evaluation | Does the adapted code meet this task’s requirements? |
Question: Which source supports the proposed coding pattern?
Result: Define retrieval, provenance, eligibility and the correction loop.
This reference lab connects the statistical methods to reusable, cited code. It adds no new assignment or grading component.
A colleague asks for a function that collects a value. Two calls with no list supplied should be independent. A search finds a slide with items=[] in the function definition.
That slide introduces the mutable-default trap. Copying its first code example would reproduce the problem it was written to teach.
The retrieved passage has the highest similarity. What should the assistant do next?
Retrieval-augmented generation (RAG) supplies searched passages to the assistant before it adapts code. The collection stays outside the model’s trained parameters.
The index is derived from the source. Correct the original chapter or notebook, review it, rebuild, then retrieve again. Retain older snapshots for audit and refuse retrieval when the active index is stale.
| Material | How to use it |
|---|---|
| Current canonical Slides and Book | Default reference, with headings and code explanations |
| Current project notebooks | Retain cell locations and assumptions; ignore saved outputs as proof |
| Intentional bugs and exercise starters | Label as teaching cautions; exclude from solution retrieval by default |
| Older material and narration | Historical context, explicitly requested |
| Instructor answers | Private index and explicit instructor access only |
| Data, credentials, student submissions, generated duplicates | Outside the coding-reference corpus |
A PDF’s extracted text does not capture all diagrams or scanned pages. Check the original page when the visual matters. “All course material” means an organised inventory with these boundaries, not a claim that every file is a verified answer.
Each unit carries its file and location, source hash and index revision, status and audience, and validation (NOT_TASK_TESTED until checked for this task).
Question: Is the retrieved example current and eligible?
Result: Inspect cited chunks and reject an outdated index.
Builds on Evidence, provenance and correction.
The browser lab uses six curated teaching cards, not the instructor’s private index. Two describe a retired or unfinished default-list example; four are current references.
read_json loads one record per card into a DataFrame. status tells us whether a card is eligible; revision identifies its version. The source column points back to the course explanation.
TF–IDF gives a word more weight when it is frequent in a passage but uncommon across the collection. Cosine similarity compares the directions of the resulting vectors. Here, larger means more shared weighted vocabulary; it is not a probability of correctness.
fit_transform learns the vocabulary and encodes cards; transform uses the fitted vocabulary for the query. ravel flattens the one-row score matrix. Sorting alone retains ineligible cards.
query selects current material; head(3) limits the evidence; iloc[0] selects its first row.
The full local tool combines semantic embeddings with keyword search. Embeddings help match different wording; keyword search helps preserve API names such as merge and validate. It filters the candidate set before ranking and returns source-linked evidence. The browser baseline filters an already-computed full ranking, which has the same eligible ordering here.
Predict first: select “Source corrected; not rebuilt”. Why must retrieval stop? Compare successful and failed rebuilds; identify the returned revision and the task checks still needed.
The result is False. This cell demonstrates change detection, not a full rebuild. The local tool hashes actual source files, checks new and deleted files, rebuilds passages and vectors into a new revision, then switches the active pointer. A failed rebuild keeps the previous snapshot and refuses stale retrieval.
Question: Does the adapted code meet this task’s requirements?
Result: Task-specific checks and a measured evaluation of the reference workflow.
Builds on Retrieve, inspect and detect stale evidence.
None marks an omitted list. If the requirement says never mutate an input, adapt the supplied-list branch and add that test.
Task: make only current cards eligible.
The starter below includes all six cards. Replace the assignment with a DataFrame filter so the pool contains the four current references. The query vocabulary alone must not decide eligibility.
After your check passes, change the task to “explain the mutable-default bug”. Would you retrieve a teaching caution for explanation? Explain why that still would not make its buggy code a solution.
The course implementation indexes canonical chapters, project notebooks and separately labelled older/private sources. It runs locally with a cached embedding model and SQLite; it does not need an API credential. A connected cloud assistant may still send selected retrieved context to its provider.
python KnowledgeBase/course_rag.py search "pandas merge validate duplicate customer_id values"
python KnowledgeBase/course_rag.py status
# After correcting and checking the original course material:
python KnowledgeBase/course_rag.py syncThese are local terminal commands, run from the instructor’s course repository using its configured Python. The public notebook below provides a portable, smaller implementation; students do not need the private repository or its instructor sources.
Download the notebook lab. It implements retrieval, a stale-index guard, correction and rebuilding on public cards. Browser cells need no local installation; the notebook uses pandas and scikit-learn.
Retrieve current course guidance for this task and cite the file and location. State the data schema, prediction time and assumptions. Adapt the method, explain the changes, and test normal, missing and boundary cases. If the course evidence is insufficient or contradictory, say so and check current official documentation.
An MCP connection makes search_course available as a tool; it does not guarantee that every assistant turn invokes it. Ask for retrieval explicitly and inspect the citations. Existing chat answers do not update automatically when a course file changes.
Also test edits and deletions, status filtering and access controls. Bug examples and private answers must remain ineligible for ordinary solution retrieval.
A working index and a good retrieval score alone do not establish fewer coding errors. RAG can retrieve an incorrect source, omit a condition or encourage an unsuitable adaptation. CodeRAG-Bench studies these retrieval and generation difficulties for coding tasks.
Further reading: Sentence Transformers semantic search, SQLite FTS5, and Codex MCP connections.
The decision memo applies the chapter’s evidence and qualifications to a research decision.
To: Head of Research, systematic equities From: <Your name>, quant research Subject: Go / no-go on the rank-target boosting signal (§7.1) and the Bollinger mean-reversion rule (§7.6) Date: 2026-10-05
Recommendation: No-go on the Bollinger rule. Paper-trade the cross-sectional rank signal at zero risk for 24 months; re-evaluate against the four statistics below.
Evidence: - Cross-sectional rank signal, walk-forward 2019–24: rank IC +0.017 (t = 1.0), top-20 − bottom-20 spread Sharpe 0.64. Five learners were compared; no selection-adjusted DSR was computed for this comparison. The separate DML association is +0.23 % a month, t = 1.4 clustered, subject to identification and dependence assumptions. - Bollinger rule, best of 42: gross log-P&L Sharpe 0.68, conditional bootstrap CI [0.18, 1.11], DSR 0.64, CSCV PBO 0.53, 0 BH survivors and Sharpe about 0.00 on the Dow. These comparisons do not establish a robust selected signal. - Conformal coverage on the S&P band fell from 0.86 in January to 0.18 in March 2020; any live signal needs the coverage alarm attached.
Caveats: the stock file is survivorship-biased (§4.5); numbers are before costs and 71 monthly observations provide limited precision. Ledoit–Wolf (Sharpe 1.01, target-weight change 0.70) and HRP (0.84, long-only) are allocation candidates to evaluate under the same protocol, rather than guaranteed solutions.
Next step: freeze the signal, trial registry and evaluation plan; record 24 months of simulated forward outcomes. Evaluate costs, outcome maturity, coverage and allocation under a reconciled account. Any proposed DSR > 0.95 or PBO < 0.2 threshold is a preregistered research choice, not a guarantee or a substitute for adequate data.
Pitfall: a copilot that builds triple-barrier labels will happily refit each January on all earlier rows. Ask what the last label in the training set depends on; if the answer includes any test-period return, it has leaked.
| Method | Question it answers | Tool |
|---|---|---|
| Learning to rank | Which stocks are in the top 20 — in what order, not by how much? | within-month rank(pct=True); pairwise logistic on \(x_i - x_j\); HistGradientBoostingRegressor on the rank; rank IC, spread Sharpe |
| Quantile + conformal | How wide is tomorrow’s range, with what guarantee — and has the world changed? | loss="quantile" at 0.05 / 0.5 / 0.95; calibration scores \(s_i\), \(\hat q\) at rank \(\lceil (n+1)(1-\alpha) \rceil\); monthly coverage; \(w = \min(\text{target}/\text{width}, w_{\max})\) |
| Double ML | What is the partial association, and what assumptions support an effect? | KFold cross-fitting, two boosting nuisance fits, residual-on-residual \(\hat\theta\), influence-function SE clustered by month |
| Shrinkage + HRP | How sensitive are weights to estimation of \(\Sigma\)? | np.linalg.cond, LedoitWolf().shrinkage_, linkage on \(\sqrt{(1-\rho)/2}\), leaves_list, recursive bisection; walk-forward vol / Sharpe / DD / turnover |
| Triple barriers + meta-labeling | When did the trade end, and how much should I bet on the primary signal? | vectorised first-touch over \(\pm m\hat\sigma_t\) and \(h\); embargo of \(h\) days; LogisticRegression → \(b = \max(0, 2(p - \tfrac12))\) |
| Backtest statistics | Is the best of \(N\) trials more than luck? | stationary bootstrap; \(SR_0\) and DSR; CSCV / PBO with \(S = 8\); Benjamini–Hochberg |
Use a fixed seed, a stated information clock, an appropriate baseline, a held-out comparison and a record of all trials.
The rank-target trees scored IC +0.017 and the MSE trees −0.015 on identical features and folds. A colleague concludes “ranking losses work”. Write the two-sentence reply, with the standard error, and describe the experiment (data, horizon, number of stocks) that would settle it.
Conformal coverage on the S&P band fell to 0.18 in March 2020 and the rolling re-calibration only recovered in May. Design a coverage-based risk rule (threshold, window, action) and state what it would have done on 27 February 2020 — and what it would have cost in 2022, when coverage dipped to 0.62 twice.
DML gave momentum a partial effect of +0.23 % a month with t = 1.4 after clustering by month. Name two unmeasured variables that could drive both momentum and next-month return, and say which §4.4 design (DiD, instrument) could address each — or why neither can.
The Bollinger rule has a bootstrap interval that excludes zero, a DSR of 0.64, a PBO of 0.53 and no BH survivor. Rank the four statistics by how much you trust them for this question, and explain what a research head should require before a fifth statistic is added to the list.
ISOM 5650 · Prof. Xuhu Wan · HKUST Business School