Chapter 7 · Modern Statistical Learning in Practice
Section 7.4 · Chapter 7 · Learning Statistics with Python
Modern Statistical Learning in Practice
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
The minimum-variance portfolio has a closed form, \(w \propto \Sigma^{-1}\mathbf 1\), and everything fragile about it lives in the inverse. You will measure how ill-conditioned a 21-asset sample covariance is, watch its optimal weights swing from one year to the next, shrink the matrix with Ledoit–Wolf, and then build a portfolio that never inverts anything — hierarchical risk parity — before putting all four rules through a nine-year walk-forward.
2 515 days, 21 assets from 6.7 % (IEF, 7–10-year Treasuries) to 40 % (USO, oil). SHY is excluded on purpose: a minimum-variance optimiser offered a 1.5 %-vol asset will buy 120 % of it and short everything else — a correct answer to an uninteresting question. The 21 that remain are the risky universe.
For 21 correlated ETFs on 251 daily returns, the condition number \(\lambda_{\max}/\lambda_{\min}\) of the sample covariance will be of the order of…
Sample: 6 373 in 2018, 5 010 in 2019. Ledoit–Wolf: 415 and 299 — fifteen times better-conditioned — from a shrinkage intensity \(\delta\) of only 0.03. The next slide shows what that does to the weights.
Sample min-variance in 2018: 75 % IEF, −29 % TLT, +10 % SPY; a year later −27 % TLT and −22 % SPY — SPY flipped from long to short on nothing but a new year of data. Total weight change 0.99 for the sample, 0.52 for Ledoit–Wolf, whose SPY weight stays near zero. Both under-predict next year’s volatility (1.5 % promised, 2.0 % delivered): the optimiser has found the noise in \(\Sigma\) and bet on it.
\[\hat\Sigma_{LW} = (1 - \delta)\, S + \delta\, \mu I, \qquad \mu = \frac{\operatorname{tr}(S)}{N}, \quad \delta^\ast = \arg\min_\delta \mathbb E\lVert \hat\Sigma_{LW} - \Sigma \rVert^2.\]
sklearn.covariance.LedoitWolf computes \(\delta^\ast\) from the data (Ledoit & Wolf, 2004); \(\delta\) = 0.03 sounds tiny, but it is applied where the matrix is weakest.Note
Alternatives in the same family: shrinkage toward a constant-correlation target, toward a one-factor (CAPM) covariance, and non-linear shrinkage that adjusts each eigenvalue separately. All exist because \(S^{-1}\) is the problem, not \(S\).
López de Prado (2016): the instability lives in \(\Sigma^{-1}\), so never compute it. Cluster the assets, order them so that neighbours are alike, and split capital down the tree two clusters at a time — estimation error stays local.
The recipe: never invert \(\Sigma\). Cluster the assets on the correlation distance \(d_{ij} = \sqrt{\tfrac12(1 - \rho_{ij})}\), reorder them so that similar assets sit together, then split the capital top-down, two clusters at a time.
Two assets with \(\rho = 0.5\) — what is their distance \(d\), to 2 dp? And for \(\rho = -1\)?
0.5 1.0
The tree puts the dollar and the metals first, then the three bond funds, then the defensive sectors, energy, credit and financials, and finally the equity indices with their sector funds. Adjacent assets correlate 0.63 on average against 0.22 in alphabetical order: the correlation matrix is now nearly block-diagonal.
Right-hand panel: a red equity block in the bottom-right, a small bond block, and the blue stripe where bonds meet equities. Recursive bisection will walk down this order and, at every split, give more capital to the half with the lower variance.
Start with all assets in tree order and weight 1. Split the list in half; compute each half’s variance as an inverse-variance portfolio; give the left half the share \(\alpha = 1 - v_L / (v_L + v_R)\); recurse into each half.
Sum 1, every weight positive, largest 0.31 (UUP, the dollar — lowest volatility in its cluster), then LQD and IEF at 0.17, gold 0.12. No short, no inverse, no eigenvalue. Estimation error stays local: a noisy correlation between two energy funds changes their split, not the weight of Treasuries.
Each January, estimate on the previous calendar year, hold for the year. Four rules: equal weight, min-variance (sample), min-variance (Ledoit–Wolf), HRP.
Before running: which rule will have the lowest volatility, and which the highest turnover?
Equal weight: vol 12.6 %, Sharpe 0.67, drawdown −34.5 %, no trading. Min-variance on the sample matrix: vol 3.2 %, Sharpe 0.82 — but it re-trades 148 % of the book every January and holds a 102 % position. Ledoit–Wolf: vol 3.6 %, Sharpe 1.01, drawdown −7.6 %, turnover 0.70 — the best Sharpe with half the trading. HRP: vol 5.4 %, Sharpe 0.84, drawdown −13.9 %, turnover 0.56, long-only, largest weight 0.51. Sharpe ratios are on raw returns, before costs, over nine years (standard error ≈ 0.33): LW’s edge over the others is suggestive, its lower turnover is certain.
Next: §7.5 — before you can train on a trade, you have to decide what “the trade succeeded” means.
Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python