5.3 — Robust Regression with Fat Tails

Chapter 5 · Rethinking Statistics with Bayesian Methods

Prof. Xuhu Wan

Section 5.3 · Chapter 5 · Learning Statistics with Python

Robust Regression with Fat Tails

Rethinking Statistics with Bayesian Methods

Prof. Xuhu Wan

ISOM, HKUST Business School · 2026 Edition

Robust Regression with Fat Tails

Three planted outliers will drag an OLS slope from 5 to 4.4. You will predict the direction, then swap the likelihood for a Student-t and watch the slope come back — first on simulated data, then on GOOG’s market beta.

Plant the outliers

\(y = 0.5 + 5x + N(0, 0.5^2)\) on 100 points, plus three points at \((0.1, 8), (0.15, 6), (0.2, 9)\) — far above the line on the left.

How will OLS respond to three high points at small \(x\)?

Slope up, intercept down
Slope down, intercept up
Both unchanged — three points cannot matter out of 103
Slope up, intercept up

The normal likelihood fails

OLS [1.033, 4.403] — intercept doubled, slope off by 0.6. Under a normal likelihood a residual of 7 costs \(7^2/2 = 24.5\) log-units; the fit bends to reduce that at everyone else’s expense.

In Colab

with pm.Model() as model:
    beta0 = pm.Normal("b0", 0.0, 1.0); beta1 = pm.Normal("b1", 0, 1.2)
    std = pm.Exponential("sd", 1.0)
    output = pm.Normal("y", mu=beta0 + beta1 * data2["x"], sigma=std, observed=data2["y"])
# robust version: replace the last line by
    likelihood = pm.StudentT("y", mu=mu, sigma=std, nu=3, observed=data2["y"].values)

Same priors, one word changed: Normal → StudentT. Let’s see what that word does.

Student-t likelihood by maximum likelihood

No sampler needed to see the effect: minimise the negative t log-likelihood with scipy.optimize.minimize, \(\nu = 3\).

Robust [0.492, 5.167], scale 0.457 — the truth is 0.5, 5, 0.5. The green dashed line goes through the cloud; the red one is tugged up-left by three points.

Why does the t-likelihood ignore them?

The t log-density’s derivative gives each point an implicit weight \(w_i = \dfrac{\nu + 1}{\nu + r_i^2}\), where \(r_i\) is the standardised residual.

The outlier at \((0.2, 9)\) has standardised residual ≈ 16.5 under the robust fit. With \(\nu = 3\), what is its weight (3 dp)?

round((3 + 1) / (3 + 16.5**2), 3)

0.015

Outliers get weights 0.017, 0.036, 0.015; a typical point gets 1.13. The t-likelihood is a self-weighting least squares — it down-weights exactly the points that do not fit. Nobody chose which points to drop.

Posterior of the robust slope by Metropolis

Same sampler as §5.1, new log-posterior: Prior II from §5.2, \(\sigma \sim \text{Exponential}(1)\), t(3) likelihood.

Slope posterior mean 5.06, credible interval [4.64, 5.41]. OLS’s 4.40 lies outside it. The model is telling you the outliers are not from the same process.

Real data: GOOG’s beta on the S&P 500

Daily returns 2015-10 to 2020-10, 1 258 days including the March 2020 crash. Two likelihoods, one question: what is \(\beta\)?

Predict the direction

OLS gives \(\beta \approx 1.06\). What will a Student-t likelihood give?

Exactly the same — beta is beta
Much lower, because crash days are removed
Slightly different in a direction you cannot predict in advance
Exactly 1 — the CAPM says so

Robust beta, with ν estimated

OLS β 1.064, robust β 1.094, \(\nu = 3.35\) — close to the 2.7 the return model found in §5.1. The point estimates differ by 0.03; the difference in inference is larger: a normal model with kurtosis-11 residuals reports standard errors that are fiction.

Where the two betas disagree

Twelve days with \(|r_{SP}| > 5\%\), nine of them in March 2020. Those twelve points — 1 % of the sample — carry 37 % of \(\sum x^2\), the leverage that drives OLS. The t model weights each by its surprise, not by its size.

Your turn: rolling robust regression

250-day windows, stepped every 25 days. The OLS beta is done; replace the placeholder so b_rob is the t(3) slope from minimize (hint: mimic nll_t with xx, yy). Then compare the two paths.

What you discovered

Three outliers in 103 points moved the OLS slope from 5 to 4.40; the Student-t fit gave 5.17.

The t-likelihood is self-weighting: \(w_i = (\nu+1)/(\nu + r_i^2)\) gave the outliers weights of 0.02 without anyone deleting a row.

Changing one word (Normal → StudentT) is the whole robust model; minimize or Metropolis fits it in under a second.

GOOG’s residuals have kurtosis 11; the robust beta 1.094 vs OLS 1.064 differ little in level, a lot in the honesty of their uncertainty.

Next: §5.4 — stop refitting from scratch; let yesterday’s posterior become today’s prior.