Chapter 7 · Modern Statistical Learning in Practice
Section 7.3 · Chapter 7 · Learning Statistics with Python
Modern Statistical Learning in Practice
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
§4.4 showed a budgeting tool that predicted repayment perfectly and did nothing — until the confounder was controlled for. Controls in a regression only work if their functional form is right. You will break a linear control with a nonlinear confounder, repair it with two boosting models and a cross-fitting trick that takes twenty lines, then ask the real question: does 12-1 momentum move next month’s return once every other characteristic and the sector are held fixed?
The Frisch–Waugh–Lovell theorem: in \(Y = \theta D + X\beta + \varepsilon\), the OLS estimate \(\hat\theta\) is the regression of the residual of \(Y\) on \(X\) on the residual of \(D\) on \(X\). Controls remove from \(Y\) and \(D\) whatever \(X\) explains; \(\theta\) is estimated from what is left.
\[\tilde Y = Y - \mathbb E[Y \mid X], \qquad \tilde D = D - \mathbb E[D \mid X], \qquad \hat\theta = \frac{\sum \tilde D_i \tilde Y_i}{\sum \tilde D_i^2}.\]
OLS with \(X\) as controls assumes…
Double machine learning (Chernozhukov et al., 2018) keeps the FWL recipe and replaces the two linear regressions by any learner — here the §4.3 boosting machine.
Three covariates; the confounder \(g(X) = 1.5\sin(2x_0) + 1.5\sin(2x_1)\) drives both the signal \(D\) and the outcome \(Y\). True effect \(\theta = 0.5\).
Naive 1.264, with linear controls 1.211 — still two and a half times the truth, with a standard error of 0.02 that says the wrong number is very precise. A linear fit explains 14 % of the confounder, so the controls closed a seventh of the backdoor and left the rest to \(\theta\). This is the §4.4 lesson in its harder form: the confounder is measured and the regression still fails.
Why it works. The residual-on-residual moment is Neyman-orthogonal: first-order errors in \(\hat m\) and \(\hat\ell\) cancel, only their product enters the bias. Two learners that are each moderately good give an estimate that is very good.
Why cross-fit. Step 2 fits on other folds so that the residual of row \(i\) never comes from a model that saw row \(i\). The next-but-one slide shows what happens if you skip this.
0.496 ± 0.023, on top of the truth, from two boosting models with depth-2 trees. The boosting nuisance models absorbed the sine curves that the linear controls missed; the residual of \(D\) kept 71 % of its spread — the part of the signal that the confounder does not explain, which is the only part that can identify \(\theta\).
Cross-fitting introduces a random element — which rows land in which fold. A method whose answer moved with the seed would be worthless.
Re-run with \(K = 2\) folds and seed 0, then \(K = 2\) and seed 1. Will both estimates stay within two standard errors (0.046) of the \(K = 5\) answer, 0.496?
yes — 0.515 and 0.525, both within 0.03 of 0.496
0.515 and 0.525 against 0.496: the split moves the estimate by about one standard error, never more. With \(K = 2\) each nuisance model sees only half the data and is a little rougher — the cost shows up as a slightly higher estimate (the residual confounding of a cruder fit), not as instability. Five folds is the usual compromise; the paper’s recommendation is to average over several splits.
Fit \(\hat m\) and \(\hat\ell\) on all \(n\) rows with a learner flexible enough to memorise, then compute \(\hat\theta\) on the same rows. The estimate will be…
Same rows: 0.377 — a quarter below the truth — because the memorising learner left \(D\) with 31 % of its spread, and that remainder is contaminated by the fit that produced it. The cross-fitted recipe gave 0.496. The bias is not in the learner; it is in letting one row be both teacher and witness — the §4.3 train/test rule applied inside an estimator.
\(D\) = the per-month z-score of 12-1 momentum; \(Y\) = next month’s return in percent; \(X\) = the other four characteristics plus sector dummies. The §4.5 panel, 25 335 stock-months.
Momentum is far from independent of the controls: correlation 0.51 with price / 252-day high (a stock near its high has usually risen over the year) and 0.10 with dollar volume. Whatever “momentum effect” a naive regression finds is partly the 52-week-high effect wearing a different label — exactly the §4.4 confounder, now with 14 of them at once.
Standard errors are clustered by month: the 213 stocks in one month share the market’s move, so treating them as independent overstates precision.
Naive +0.16 % per standard deviation of momentum; with linear controls +0.20 %; DML +0.23 % a month (2.7 % a year). Controlling raised the estimate — the 52-week-high effect was masking momentum, not manufacturing it. The controls explain 41 % of momentum’s variance. But look at the standard errors: 0.09 under the iid assumption, 0.16 clustered by month; with clustering, t = 1.4. Twenty-five thousand rows are only 119 independent months.
| Naive OLS | OLS + controls | DML | |
|---|---|---|---|
| Closes the backdoor through measured \(X\) | no | only if linear | yes, any shape |
| Robust to a badly fitted nuisance model | — | — | yes, to first order |
| Valid standard errors | with clustering | with clustering | with clustering |
| Closes the backdoor through unmeasured confounders | no | no | no |
| Turns a characteristic into a treatment | no | no | no |
The honest reading
DML answers “what is the partial association of momentum with next-month return, holding fourteen controls fixed in whatever functional form they take?”. It does not say what would happen if you made a stock’s momentum higher — there is no such intervention — and a characteristic you did not measure (analyst coverage, index membership, news flow) can still drive both. The §4.4 hierarchy stands: controls, then DiD, then an instrument.
Next: §7.4 — from one signal to a whole portfolio, and why the covariance matrix needs help before you invert it.
Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python