Chapter 3 · Reshaping Statistics
Section 3.5 · Chapter 3 · Learning Statistics with Python
Reshaping Statistics
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
Averages have the central limit theorem; maxima have Fisher–Tippett–Gnedenko. You will first see that matching mean and variance does not match the tail, then collect the worst day of every 20-trading-day block on the Dow (1985–1990, so Black Monday is in the sample), fit a GEV by maximum likelihood, check it with a Q-Q plot, score each block’s rarity, and read off the extreme VaR — the highest pain threshold.
Draw 4 309 Normal returns with Apple’s mean and standard deviation (seed 71). They match on the first two moments. Predict the 0.1 % quantile comparison.
With mean and sd matched, Apple’s 0.1 % quantile versus the Normal’s is…
Same mean, same sd — and a 0.1 % quantile of −10.5 % against −5.8 %, a 1 % CVaR of −7.8 % against −5.1 %, and 25 days beyond four standard deviations where the Normal has none. Variance does not measure tail risk. Two portfolios with identical Sharpe ratios can have wildly different blow-up probabilities.
If the standardised maximum \(M_n\) of \(n\) i.i.d. draws converges at all, it converges to the GEV family:
\[F(s) = \begin{cases} \exp\!\big(-e^{-s}\big) & \xi = 0 \\[4pt] \exp\!\big(-(1 + \xi s)^{-1/\xi}\big) & \xi \ne 0 \end{cases} \qquad Q(p) = \begin{cases} -\ln(-\ln p) & \xi = 0 \\[4pt] \dfrac{1}{\xi}\big((-\ln p)^{-\xi} - 1\big) & \xi \ne 0 \end{cases}\]
with \(s = (x - \mu)/\sigma\). Three sub-families, indexed by the tail index \(\xi\).
Which sign of \(\xi\) means a heavy, unbounded tail?
scipy’s sign convention
scipy.stats.genextreme uses shape c with c = −ξ. Every fit below reports ξ = −c.
With \(\mu = 0, \sigma = 1\) the support ends at \(\mu - \sigma/\xi\): \(= 3\) for \(\xi = -1/3\) (right end), \(= -3\) for \(\xi = 1/3\) (left end). The Fréchet curve is the one that keeps going.
Which date is the worst single day in 1985–1990, and roughly what log return? (Think October 1987.)
1987-10-19, −0.2563
rolling(20).min() gives the running 20-day minimum; taking every 20th row (iloc[::20]) makes the blocks non-overlapping. Predict the number of blocks.
Roughly 1 500 trading days in 1985–1990. How many 20-day block minima?
One red dot per block, whatever happened in it: a calm month contributes a minimum of −0.5 %, October 1987 contributes −25.6 %. Block maxima treat both as one observation each — the method is rate-limited by design.
POT throws away the calendar and keeps every exceedance. Predict the order of magnitude: how many of the 1 496 days fell below −1.5 %?
How many days in 1985–1990 have a log return below −0.015?
The dots now cluster in October 1987 and October 1989 — POT captures clusters of stress that block minima flatten to one point. Its price is choosing \(u\): too high and the sample is tiny, too low and the GPD approximation fails.
Before real data, check the theorem on a case where you know the answer (the notebook’s i.i.d. Normal experiment).
10 000 standard-Normal draws, 100 blocks of 100, take each block’s maximum, fit a GEV. Which \(\xi\) should you expect?
Minima become maxima by a sign flip: fit to \(-\text{Min20}\), the block loss. Predict the support bound.
For \(\xi > 0\) the GEV support starts at \(\mu - \sigma/\xi\). With \(\mu \approx 0.0136\), \(\sigma \approx 0.0070\), \(\xi \approx 0.354\), what is the bound?
−0.0063
\(\xi \approx 0.35 > 0\): Fréchet. The 20-day worst loss of the Dow has a polynomial tail; moments of order above \(1/\xi \approx 2.8\) do not exist.
One point will sit far above the 45° line at the top right. Which block is it?
Score \(= 1 - F_{\text{GEV}}(\text{loss}) = P(\text{a block is worse than this one})\). Small score = rare block.
Which date has the smallest score, and how many blocks score below 0.10?
1987-10-19, 6
Black Monday’s score is 0.0007: the fitted GEV says a worse 20-day block arrives once in about 1 400 blocks — 110 years. Two others (January 1988, October 1989) are 1-in-50 events.
Exceedances over a high threshold follow a GPD (Pickands–Balkema–de Haan): \(\;P(X - u > y \mid X > u) \approx \big(1 + \xi y/\beta\big)^{-1/\xi}\), with the same \(\xi\) as the GEV. The POT quantile is
\[\text{VaR}_q = u + \frac{\beta}{\xi}\Big[\Big(\tfrac{n}{N_u}(1-q)\Big)^{-\xi} - 1\Big].\]
91 exceedances give \(\xi = 0.44\) against the GEV’s 0.35 — the same heavy-tail verdict from a different sample; agreement between the two estimators is your sanity check. The GPD’s 99 % one-day VaR (3.3 %) sits just above the empirical 1 % quantile (3.1 %): with 1 496 days the empirical quantile is fine at 1 %. The GPD earns its keep at 0.1 %, where there is nothing left to count.
A Fréchet tail is a power law, \(P(L > x) \sim x^{-\alpha}\) with \(\alpha = 1/\xi\). Hill (1975) reads \(\alpha\) off the top \(k\) order statistics: \(\hat\alpha_k = \big[\tfrac{1}{k}\sum_{i\le k}\ln(L_{(i)}/L_{(k+1)})\big]^{-1}\). Predict which \(k\) agrees with the GPD’s \(\xi = 0.44\).
Where will the Hill estimate of \(\xi\) form a plateau near 0.44?
\(\hat\xi\) = 0.47, 0.42, 0.44 for \(k\) = 25, 50, 100 — the plateau — then 0.63 at \(k = 200\), where the threshold (0.8 %) is an ordinary day. Picking \(k\) is the craft, exactly as picking \(u\) was for POT: too few points is noise, too many is the body of the distribution.
The \(T\)-day return level solves \(P(L > x_T) = 1/T\). Two estimators: the empirical quantile, and the fitted GPD extrapolated: \(x_T = u + \frac{\beta}{\xi}\big[(\tfrac{k}{n}T)^{\xi} - 1\big]\). Predict the empirical method’s fatal limit.
Why can’t an empirical quantile give a 1-in-2 500-day loss from 1 496 days?
At \(T = 250\) (one year) the two agree: 4.9 % against 4.7 %. At \(T = 1000\) the GPD says 8.9 % while the empirical 7.8 % rests on the two worst days in the sample; at \(T = 2500\) (ten years) only the GPD answers — 13.2 %, and Black Monday’s 25.6 % is the reminder that even that is one draw from the tail. Extrapolation with a fitted shape is what turns six years of data into a ten-year estimate.
Invert the fitted GEV at probability \(\alpha\): the loss that only a fraction \(\alpha\) of 20-day blocks will exceed.
\[\text{EVaR}_\alpha = -\Big(\mu + \frac{\sigma}{\xi}\big[(-\ln(1-\alpha))^{-\xi} - 1\big]\Big)\]
With \(\xi = 0.354\), \(\mu = 0.0136\), \(\sigma = 0.0070\), what is EVaR at \(\alpha = 5\,\%\) (as a return)?
−0.0506
A one-day 5 % VaR says “−1.6 %”. The extreme VaR says: in one 20-day block out of twenty, expect a day worse than −5.1 % — and 4.1 % of the 74 blocks did. That is the number a risk limit should be sized to.
Compute the 1 % extreme VaR from the fitted GEV into evar01 (use EVaR), then count how many of the 74 block minima fell below it.
c = −ξ.Three prompts that make an LLM useful for this chapter — and the pitfall each one guards against.
n, the standard deviation, and the excess kurtosis; if kurtosis is above 3, tell me which of the tests you propose assume Normality and give me a bootstrap or permutation alternative.” A copilot will run ttest_1samp on 20 fat-tailed returns and report a p-value to four decimals. The number is real; the assumption behind it is not.kstest against norm, confirm the data are standardised first (or pass args=(mean, sd)), and when you fit genextreme, state the sign convention and report ξ = −c.” Both are places where the code runs, returns a number, and the number is wrong — D = 0.47 on raw returns is a scale mismatch, not a shape test.Warning
In the second week of August 2007, quantitative equity funds run by Goldman Sachs, Renaissance, AQR and others lost between 10 % and 30 % in a few days as crowded long-short factor positions unwound together. Goldman’s Global Equity Opportunities fund fell about 30 % in a week and received a US$3 billion injection. Explaining it, CFO David Viniar told the Financial Times (13 August 2007): “We were seeing things that were 25-standard-deviation moves, several days in a row.”
Under a Normal model a 25σ event has probability of order \(10^{-137}\) — it should not happen once in the life of the universe, let alone on consecutive days. The models were not unlucky; they were mis-specified. A Fréchet tail with \(\xi \approx 0.35\), the number you fitted to the Dow, assigns such moves probabilities measured in years, not eons.
Lesson for this chapter: counting sigmas presumes a Gaussian. The tail index, block maxima and the extreme VaR exist precisely because “how many σ” is the wrong question about the tail.
To: Chief Risk Officer From: <Your name>, risk analytics Subject: Replace the daily-VaR-based stop with an extreme-VaR limit Date: 2026-09-15
Recommendation: Size the index book’s hard loss limit to the 5 % extreme VaR of the 20-day block minimum: −5.1 % of notional per block, reviewed quarterly.
Evidence: - GEV fitted by MLE to 74 block minima (DJI 1985–1990): ξ = 0.35, μ = 0.0136, σ = 0.0070; Q-Q diagnostic straight apart from 19 Oct 1987. - EVaR(5 %) = −5.1 % vs the daily 5 % quantile of −1.6 %: the current stop is sized to an ordinary bad day, not to an ordinary bad month. - Six blocks in six years scored below 0.10; the limit would have been breached 3 times (4.1 % of blocks), all in identified stress episodes.
Caveats: - 74 observations: the ξ confidence interval is wide (bootstrap it, §3.2, before sign-off). - Historical window ends 1990; re-estimate on 2008 and 2020 to test stability of ξ. - GPD on 91 exceedances gives ξ = 0.44 and the Hill plateau 0.42–0.47 vs the GEV’s 0.35; agreement in sign and size is the sanity check, not proof.
Next step: Re-fit on rolling 6-year windows to 2024; report EVaR(5 %) and EVaR(1 %) with bootstrap bands.
| Concept | Tool |
|---|---|
| Population vs sample | sample, std(ddof=1) |
| Empirical distribution | ECDF via np.sort and arange(1, n+1)/n, DKW bound, KernelDensity, Silverman’s \(h\) |
| Parametric fit and diagnostics | stats.t.fit, probplot, bimodality coefficient |
| Bootstrap | rng.integers(0, n, (B, n)), percentile CI, BCa (\(z_0\), \(a\) from the jackknife) |
| Tests | shapiro, kstest (studentise first), ks_2samp, ttest_1samp(alternative=), permutation via rng.shuffle, rolling \(\hat t\) |
| Experimental design | \(n = 2\sigma^2(z_\alpha+z_\beta)^2/\tau^2\), peeking simulation, \(\alpha\)-spending, ε-greedy / UCB1 / Thompson |
| Association | pearsonr, spearmanr, kendalltau, distance correlation via pdist, tail co-exceedance |
| Extreme values | genextreme.fit (ξ = −c), block minima, genpareto.fit, Hill estimator, return levels, extreme score, EVaR |
Next: Chapter 4 — Statistical Predictive Models.
Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python