Chapter 7 · Modern Statistical Learning in Practice
Section 7.2 · Chapter 7 · Learning Statistics with Python
Modern Statistical Learning in Practice
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
§6.2’s boosting model produced one number per day and was worse than predicting zero. A trading desk rarely needs the number; it needs to know how wide the range is today — for sizing, for stops, for the risk report. You will fit three quantile models on the §6.2 features, calibrate them with a split-conformal step that comes with a coverage guarantee, then watch the guarantee fail in March 2020 and turn that failure into an alarm.
The pinball (check) loss at level \(\tau\):
\[\ell_\tau(y, q) = \begin{cases} \tau\,(y - q) & y \ge q \\ (1-\tau)\,(q - y) & y < q \end{cases}\]
Which constant \(q\) minimises \(\mathbb E[\ell_\tau(y, q)]\)?
HistGradientBoostingRegressor(loss="quantile", quantile=τ) boosts trees on this loss. Three fits — \(\tau\) = 0.05, 0.5, 0.95 — give a lower band, a median and an upper band that all depend on the features.
Same eleven lagged and rolling features as §6.2, every one shifted so that day \(t\) uses data to \(t-1\). New: a calibration window between training and test.
753 training days from January 2016, 252 calibration days, 1 258 test days that include the March 2020 crash, the 2022 bear market and two calm years. The calibration year is never used to fit a tree — that is what makes the guarantee on the next slides valid.
The median is 0.05 % — as useless as §6.2’s point forecast. The bands are not: on the quietest day of the sample (12 October 2017) the model says [−0.52 %, +0.71 %]; on the stormiest (27 December 2018) [−2.31 %, +2.01 %]. Width tracks the 21-day volatility feature with correlation 0.82: the quantile model has rediscovered GARCH’s message from §6.5 — the size is forecastable, the sign is not.
The 90 % band \([\hat q_{0.05}, \hat q_{0.95}]\) should contain \(y\) on 90 % of days. Where will empirical coverage be lowest?
0.895 in training, 0.853 on the calibration year, 0.773 on the test years — one day in four falls outside a band that promised one in ten. A quantile model’s band is a prediction, not a guarantee. The fix does not retrain anything: it measures the miss on data the model never saw, and widens the band by that much.
The quantile model’s band is a good shape and a wrong size. Conformal prediction keeps the shape and fixes the size with one number measured on data the model never saw — and comes with a proof.
For each calibration day compute the non-conformity score — how far outside the band the truth fell (negative if inside):
\[s_i = \max\big(\hat q_{0.05}(x_i) - y_i,\; y_i - \hat q_{0.95}(x_i)\big), \qquad i = 1,\dots,n.\]
Take \(\hat q\) = the \(\lceil (n+1)(1-\alpha) \rceil\)-th smallest score and report \([\hat q_{0.05}(x) - \hat q,\; \hat q_{0.95}(x) + \hat q]\).
The guarantee (Vovk; Romano, Patterson & Candès 2019)
If the calibration and test days are exchangeable, then \(P\big(y_{\text{new}} \in \text{band}\big) \ge 1 - \alpha\) — for any underlying model, however badly fitted. The model decides the shape of the band (wide on stormy days); the calibration score decides its size.
The catch is the word exchangeable. Daily returns in 2019 and daily returns in March 2020 are not draws from one urn — and the next slides measure exactly how far that assumption breaks.
The 228-th smallest of 252 scores is +0.18 %: the 2019 misses say “widen each side by 0.18 %”. Coverage on 2020–24 rises from 0.773 to 0.830 — better, still short of 0.90. By year: 0.73 in 2020, 0.87 in 2021, 0.79 in 2022, 0.89 and 0.87 after. The guarantee held where 2019 resembled the future and failed where it did not.
The band widens as the std21 feature catches up — a half-width of ±1.0 % in January, ±2.6 % by 17 March — but the returns of late February and March run ahead of it: 18 misses in March alone, 11 in April. A band built from the features of the last 21 days is a lagging description of a crash. Count the misses month by month and the lag becomes an alarm.
February 0.68, March 0.18, April 0.48: in March 2020 the band caught fewer than one day in five — 45 % of days broke through the floor and 36 % through the ceiling. The 21-day rolling coverage fell to 0.14 on 30 March. A band whose coverage collapses is telling you the distribution has moved, with a lag of a few days — a regime detector that needs no regime model (§6.4), only the counts of hits and misses.
Coverage fell to 0.18 in March 2020. What failed?
A trailing 252-day calibration lifts coverage to 0.893 — on target on average — but March 2020 is still 0.18: the window learns the new width only after the misses have happened (April 0.67, May 0.90). Conformal prediction gives you an honest average; it cannot see a regime before it arrives, only report it faster than a quarterly review.
§6.6 held \(w_t = \min(\sigma^\ast / \hat\sigma_t, w_{\max})\) of the index. Replace \(\hat\sigma_t\) by the conformal band’s width — a model-free measure of tomorrow’s range.
Weight 0.37 in March 2020, the 1.5 cap in calm months, 0.76 on average. Cumulative 41.9 % against 59.9 % for buy-and-hold, Sharpe 0.66 vs 0.56, drawdown −21.2 % vs −41.4 %. The §6.6 overlay (Sharpe 0.77, −22.6 %) did slightly better with a GARCH forecast; the interval width gets most of the way with no parametric model and a coverage guarantee attached.
Set alpha_new = 0.20 (an 80 % band) and re-run the calibration. How much narrower is the band, and what happens to test coverage? Then try 0.05.
\(\alpha\) = 0.20: \(\hat q\) = −0.14 % (the band shrinks — the raw 5/95 band already over-covers 80 %), coverage 0.714, width 2.31 %. \(\alpha\) = 0.05: \(\hat q\) = +0.39 %, coverage 0.887, width 3.35 %. Each step of coverage costs about 0.5 % of width — the price list a risk manager should know before choosing the confidence level.
loss="quantile" turns the §6.2 boosting model into a band whose width tracks volatility (correlation 0.82 with std21) while its median stays useless (0.05 %).Next: §7.3 — not “what will happen” but “what does this signal do”, holding everything else fixed.
Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python