Chapter 3 · Reshaping Statistics
Section 3.1 · Chapter 3 · Learning Statistics with Python
Reshaping Statistics
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
A population parameter is a fact; a sample statistic is an estimate with a distribution of its own. You will build the empirical CDF from five numbers, read Apple’s fat tails off it, estimate a density two ways (bandwidth vs a formula), learn to see a Student-t in a Q-Q plot, and then let Shapiro–Wilk and Kolmogorov–Smirnov put numbers on what the eye saw.
The notebook’s 63 numbers are the whole population. We draw 10 without replacement (seed 1).
Why does pandas compute std() with ddof=1 by default?
ddof=0Log return \(r_t = \ln S_t - \ln S_{t-1}\). Before running: is the excess kurtosis of daily returns positive or negative?
The excess kurtosis of Apple’s daily returns will be…
Excess kurtosis 5.99 and skew −0.28. Half the days sit between the two red lines — a band about 2 % wide — yet the axis runs from roughly −20 % to +14 %. That gap between the middle and the extremes is what the next slides quantify.
Textbooks hand you a density. Real work hands you a sample. The bridge between them is one line of arithmetic — you will build it from five numbers and then read Apple’s tails off it.
You have the sample [3, 1, 4, 1, 5]. Define \(\hat F(x)\) = (fraction of the sample \(\leq x\)). Before any code, predict two values:
The object you built has a name. Given an i.i.d. sample \(X_1,\dots,X_n\), the empirical CDF is \[\hat F_n(x) = \frac{1}{n}\sum_{i=1}^{n}\mathbf 1\{X_i \le x\}.\] A step function. No assumptions about shape. It is the non-parametric maximum-likelihood estimate of the true CDF — and you computed it by hand.
A sorted sample’s \(i\)-th value sits at height \(i/n\). Studentise the 4 309 returns (\(z = (r - \bar r)/s\)) and sort them.
The ECDF height at the 2 155th of 4 309 sorted points — what is \(2155/4309\) to three decimals?
0.5
The median studentised return is −0.008, essentially zero. But 76.9 % of days lie within one standard deviation (a Normal says 68.3 %) and only 98.5 % within three (a Normal says 99.7 %): a taller peak and fatter tails. That is excess kurtosis 5.99, read off a staircase.
The notebook’s mixture sample (seed 7) is two Normals stuck together, one at −10 and one at +5. Predict what its ECDF looks like before you draw it.
The data is bimodal (two clusters). What will the ECDF curve look like?
The curve climbs steeply through −10, then flattens between about −5 and 0 (the first quartile is −10.0, the median −5.2, the third quartile 5.0), then climbs again around +5. The rule you discovered: the ECDF has no tuning knob, so it cannot lie about shape. The flat stretch is the bimodality.
The ECDF is built from \(n\) random points, so it is itself random. Will it land on the truth as \(n\) grows? Predict the worst-case gap for Apple’s \(n = 4\,309\).
The DKW bound on \(\Pr(\sup_x|\hat F_n - F| > 0.02)\) is \(2e^{-2n(0.02)^2}\). Evaluate it at \(n = 4309\), rounded to 3 decimals.
0.064
Glivenko–Cantelli (1933): \(\sup_x|\hat F_n(x) - F(x)| \xrightarrow{a.s.} 0\) — the ECDF converges uniformly to the truth. The DKW inequality is the finite-\(n\) version: with 4 309 days, the whole Apple staircase is within 0.02 of the true CDF with probability at least 93.6 %. This theorem is what the bootstrap (§3.2), the KS test and the KDE all lean on.
The ECDF is a staircase. To get a smooth density, smear each point into a small bump and add them up. The width of the smear, \(h\), is the only choice that matters — and you will discover how much it matters.
A kernel density estimate puts a small bump on every observation and adds them up. The bandwidth \(h\) is the only knob. Predict its effect on the notebook’s two-component mixture.
For a two-mode sample, what does a very small bandwidth do?
Silverman’s rule of thumb gives a starting bandwidth: \(h^\star \approx 1.06\,\hat\sigma\,n^{-1/5}\). Predict it for the studentised Apple returns (\(\hat\sigma = 1\), \(n = 4309\)).
Compute Silverman’s \(h\) for \(z\) (sd 1, \(n = 4309\)), rounded to 3 decimals.
0.199
\(h^\star \approx 0.199\) on the studentised scale — a fifth of a standard deviation — or 0.0041 on raw returns. But Silverman assumes one bump: on the mixture above it would over-smooth. Always plot a sweep; never trust one number.
Studentise the returns, then let scipy.stats.t.fit choose the degrees of freedom. Predict the meaning of the number before you see it.
If the fitted degrees of freedom come out near 3, what does that say?
The notebook fits the same \(t\) to TSLA, CVX and BAC (df between 3 and 6 for every large stock). Do it for Tesla in 2020.
Studentised 2020 Tesla returns are in z_tsla. Set nu_tsla to the degrees of freedom returned by stats.t.fit (its first element).
A Q-Q plot puts sample quantiles against a reference distribution’s quantiles. A perfect match traces the diagonal. Apple’s returns have df ≈ 3.3 tails; predict what that does to the ends against a Normal reference.
The sample is heavier-tailed than Normal. The Q-Q plot will…
A Q-Q plot compares sample quantiles with a reference distribution’s quantiles. Straight line = same shape.
Against the normal the tails peel away from the line on both sides — the classic S. Against the fitted \(t\) the points hug the line until the very last few, where the right tail is shorter than a \(t_{3.3}\) predicts.
The Q-Q plot showed where the departure is; a test puts a number on it. The notebook runs kstest(returns, stats.norm.cdf) and gets \(D \approx 0.47\). Predict why it is that large.
Why is the KS statistic 0.47 when raw returns are compared with stats.norm.cdf?
Reading the tests
Null hypothesis: the population is normal. A tiny p-value rejects it: Shapiro–Wilk \(p \approx 10^{-38}\); KS on the studentised returns \(D = 0.075\), \(p \approx 10^{-21}\). Against the fitted \(t_{3.3}\) the KS test does not reject (\(D = 0.015\), \(p = 0.27\)) — that is what “the t fits” means in numbers. Shapiro–Wilk is the more powerful normality test; KS is the more general one — any CDF, or another sample (§3.2 uses the two-sample form).
Before fitting anything, the bimodality coefficient \(b = (g_1^2 + 1)/(g_2 + 3)\) flags trouble when \(b > 5/9 \approx 0.555\). Predict the verdict on the mixture, then on Apple.
For the seed-7 mixture, \(g_1 = 0.381\) and \(g_2 = -1.214\). Is \(b\) above 0.555?
True
Mixture: \(b = 0.641\), flagged — negative kurtosis (−1.21) is the fingerprint of two separated humps. Apple: \(b = 0.120\), not flagged — one peak, fat tails. The detector tells you which problem you have before you choose a density.
ddof=1 corrects the downward bias of the sample variance.norm on raw returns tests scale, not shape — studentise first (\(D = 0.075\)); the fitted \(t_{3.3}\) is not rejected (\(D = 0.015\), \(p = 0.27\)).Next: §3.2 — how sure are you of any of those numbers? Resample, and test.
Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python