Chapter 6 · Learning Statistics with Python

Time Series Models for Volatility and Risk

Model how risk changes over time: volatility clustering → ARCH/GARCH → asymmetric shocks → variance forecasts → VaR and position sizing. Use stationarity, mean models, chronological validation and hidden states to support that risk workflow.

Prof. Xuhu Wan

ISOM, HKUST Business School · 2026 Edition

What This Chapter Builds

• Recognise volatility clustering: distinguish dependence in returns from dependence in squared returns and mean dynamics from conditional variance.
• Build and interpret volatility models: ARCH, GARCH(1,1), persistence, mean reversion, multi-step forecasts and GJR asymmetry.
• Check the risk forecast: convergence, parameter constraints, standardised residuals, chronological comparisons and held-out VaR exceptions.
• Use forecasts in decisions: convert variance into tail thresholds and capped positions, with explicit units, information dates and model limitations.

The central question

How large could the next move be, given what we know now? A volatility forecast estimates conditional spread. It does not by itself predict whether the next return will be positive or negative.

The message of this chapter

\[r_t=\mu_t+\epsilon_t,\qquad \epsilon_t=\sigma_t z_t,\qquad E(z_t\mid\mathcal F_{t-1})=0,\quad \mathrm{Var}(z_t\mid\mathcal F_{t-1})=1.\]

Object	Question	Role in this chapter
Conditional mean \(\mu_t\)	Where is the next return centred?	Fit/check a mean model before interpreting its residuals.
Conditional variance \(h_t=\sigma_t^2\)	How wide is the next return distribution?	Main focus: estimate, forecast and diagnose changing volatility.
Conditional tail	How bad could a loss be?	Combine location, volatility and a declared innovation distribution.
Risk decision	How much exposure can we carry?	Apply constraints and evaluate later risk, costs and outcomes.

The volatility modelling workflow

1 · Diagnose →

Returns and squared returns
Mean residuals
ARCH evidence

2 · Model →

ARCH / GARCH
GJR asymmetry
Innovation distribution

3 · Forecast →

Next variance and horizon
Standardised residuals
Later forecast evidence

4 · Decide

VaR and exceptions
Capped risk exposure
Stress and model limits

Why keep the other time-series tools?

§6.1 supplies stable transformations and mean-model diagnostics; §6.2 supplies dated forecast evaluation; §6.3 retains relationship and hedge-risk analysis; §6.4 introduces changing states and variance regimes. All six sections remain assigned.

How to use the numbered learning path

Follow the numbered parts in order within each section. Each part states a question and a result. Read the concept, predict the example, run it, and explain the output before attempting the check.

The 3.5-hour class selects examples, group discussion and notebook work. The complete Slides and Book support self-study.

Worked examples run automatically when the page opens or refreshes. Each cell includes required earlier imports, data and definitions under Previously introduced setup. Run Code includes that setup; Copy complete code copies the whole cell. Practice and repair cells run when you choose. Make input changes in the current cell before the operation you are testing.

Roadmap

Section	Concept	Tool
§6.1 Foundations	Stationarity, unit roots, ACF/PACF, ARIMA baseline	kpss, adfuller, plot_acf, ARIMA
§6.2 Forecast evaluation	Forecast scoring, walk-forward, TS features for GBM	.shift, .rolling, HistGradientBoostingRegressor
§6.3 Relationship and hedge risk	Cointegration, pairs trading	coint, rolling hedge ratio, z-score rule
§6.4 Changing risk states	Kalman filter, time-varying beta, regime switching	UnobservedComponents, MarkovRegression
§6.5 Volatility I	ARCH, GARCH(1,1), forecasts, VaR	het_arch, scipy.optimize, stats.t
§6.6 Volatility II	GJR leverage, ARMA+GARCH, volatility targeting	LR test, ARIMA(...).resid, target/σ̂

Data: Apple minute bars, AAPL, S&P 500, GOOG, Dow, NVDA/SPY, airline passengers, FRED-MD.

Stationarity, Dependence and ARIMA

Part	Question
6.1.1 Stationarity and unit-root tests	Which properties can reasonably be treated as stable?
6.1.2 Autocorrelation and model order	How does the present relate to its own past?
6.1.3 ARIMA and seasonal baselines	What does a simple time-series model forecast?

Stationarity and unit-root tests

Question: Which properties can reasonably be treated as stable?

Result: Distinguish trends and unit roots and reconcile ADF with KPSS.

Begin with the definitions and assumptions; then trace the worked example.

One day of Apple, minute by minute

390 one-minute bars for 2 November 2020. Load the close price and look at it.

389 rows after dropping the first return. The price opens near 109.6, climbs above 110.5, then drifts down to 108.8 by the close. Wherever it is at 10:00 tells you a lot about where it will be at 10:01 — the level has memory.

Which of the two is a model allowed to touch?

Which series looks stationary — constant mean and variance over the day?

• The close price — it stays between 108 and 111 all day
• The minute return — it oscillates around zero with a stable spread
• Both — the day is too short for either to drift
• Neither — intraday data is never stationary

Mean -0.000018, std 0.00108: a flat cloud around zero. The price is the cumulative sum of these; summing a stationary series is what manufactures a wandering level.

Same shocks, two values of a

\(X_t = a\,X_{t-1} + \varepsilon_t\). With \(a = 1\), \(\text{Var}(X_t) = t\sigma^2\) grows forever — the random walk, a unit root. With \(|a|<1\) it settles at \(\sigma^2/(1-a^2)\). Feed identical shocks into \(a = 1\) and \(a = 0.8\) (seed 1, 200 steps). Predict which ends further from zero.

Same 200 shocks: the random walk wanders to -15.8 (std 5.27); the \(a=0.8\) series never leaves [-3.1, 3.1] (std 1.15). The difference between \(a = 0.99\) and \(a = 1\) is the difference between a model and a lottery.

Two flavours of non-stationary, two cures

Trend-stationary \(y_t = g(t) + X_t\): cure by detrending. Difference-stationary \(y_t = y_{t-1}+\varepsilon_t\): cure by differencing.

\(y_t = 0.5\,t + 3\varepsilon_t\). You difference it instead of detrending. What do you get?

• A random walk — differencing creates a unit root
• A stationary series, but with a larger variance than the detrended one (over-differenced)
• Exactly the same residual as detrending
• A series with a deterministic trend of slope 1

Detrending recovers the shock scale (2.77 ≈ 3); differencing the same series gives 4.06 ≈ \(3\sqrt2\). Differencing the random walk gives 0.93: exactly the shocks. Match the cure to the disease.

KPSS: the null is “stationary”

kpss(x, regression="c") tests stationarity around a constant; "ct" around a linear trend. Small p ⇒ reject stationarity. Run it on the trend series with both nulls and predict which one rejects.

Around a constant: stat 2.102, p 0.01 — rejected (the trend is not a constant mean). Around a trend: stat 0.058, p 0.1 — not rejected. KPSS reports p only inside [0.01, 0.1]; 0.01 means “≤ 0.01”.

ADF: the null is “unit root”

adfuller estimates \(\Delta y_t = \alpha + \beta t + \gamma\,y_{t-1} + \dots\) and tests \(\gamma = 0\) (i.e. \(a = 1\)). Small p ⇒ reject the unit root.

The trend series fails ADF with a constant (p 0.95) but rejects the unit root once a trend is allowed (p 0.0000). The random walk: ADF p 0.81, KPSS p 0.01 — both agree it is non-stationary. AR(0.8): ADF p 0.0000, KPSS p 0.1 — both agree it is stationary.

The decision table

	ADF rejects (p < 0.05)	ADF does not reject
KPSS does not reject (p ≥ 0.05)	Stationary — model it	Not enough data to tell (low power)
KPSS rejects (p < 0.05)	Conflict: trend-stationary or structural break — try "ct"	Unit root — difference it

Important

Always run both. ADF alone cannot distinguish “stationary” from “too little data to reject”; KPSS alone cannot distinguish “unit root” from “deterministic trend”. The pair, with "c" and "ct", covers the cases.

Apple’s minute prices: a conflict to read

For the close price: KPSS(c) p = 0.01, ADF(c) p = 0.035. What is the honest verdict?

• Stationary — ADF rejects at 5 %
• Unit root — KPSS rejects
• Conflict: near-unit-root; do not model the level, model the returns
• The tests are broken on intraday data

Return: KPSS p 0.1, ADF stat -25.07, p 0.0000 — unambiguous. The price is the conflict cell; the return is the “model it” cell.

Your turn: is US inflation stationary? Unemployment?

FRED-MD, monthly 1959–2022. The first row holds transformation codes — drop it. Test the CPI level and the unemployment rate.

p_cpi is done. Compute p_unrate, the ADF p-value (constant only) for UNRATE, and compare. Which one is stationary in levels?

CPI p 0.9986 — a unit root (with a trend too: "ct" gives 0.77). UNRATE p 0.011 — stationary: unemployment goes up and comes back down. A feature with a unit root goes into a model only after differencing — remember this in §6.2.

Autocorrelation and model order

Question: How does the present relate to its own past?

Result: Read ACF and PACF patterns and state their limits.

Builds on Stationarity and unit-root tests.

\(\rho_k = \text{corr}(y_t, y_{t-k})\). The ACF plots \(\rho_k\) against \(k\); the PACF plots the correlation at lag \(k\) after removing lags \(1,\dots,k-1\).

ACF and PACF of the airline passengers

For monthly airline passengers (144 months, 1949–1960), which is larger: the ACF at lag 6 or at lag 12?

• Lag 6 — closer in time is always more correlated
• Lag 12 — same month last year beats opposite season
• They are equal — the series is a random walk
• Both are ≈ 0 — passengers are unpredictable

ACF: 0.95, 0.68, 0.76 — slow decay (trend) with a bump at 12 (season). PACF: 0.95, -0.23, 0.04 — a single spike then nothing: once you know last month, the month before adds little. AR-like level, seasonal echo.

How do you recognise AR versus MA — and the ACF of returns

Pattern	ACF	PACF	Model
AR(\(p\))	decays geometrically	cuts off after lag \(p\)	past values
MA(\(q\))	cuts off after lag \(q\)	decays	past errors
White noise	all inside \(\pm 2/\sqrt n\)	same	nothing to model

Dow log returns 2015–2019: all ten autocorrelations inside ±0.056 — the mean has no memory. But \(|r|\) has ACF 0.28, 0.26, 0.27, … — volatility remembers. That second panel is §6.5. It is the whole chapter in one figure.

ARIMA and seasonal baselines

Question: What does a simple time-series model forecast?

Result: Compare an ARIMA baseline with a seasonal example and inspect errors.

Builds on Autocorrelation and model order.

ARIMA(p, d, q): a 9-fit grid on Dow returns

Difference \(d\) times, then fit ARMA(\(p,q\)): \(\Delta^d y_t = c + \sum\phi_i\,\Delta^d y_{t-i} + \varepsilon_t + \sum\theta_j\,\varepsilon_{t-j}\). Log prices need \(d=1\); returns are already stationary, \(d=0\).

\(p, q \in \{0,1,2\}\), \(d=0\), chosen by AIC on the Dow returns above. Which order wins?

• (0, 0, 0) — a constant; the market has no linear memory worth two parameters
• (1, 0, 0) — yesterday’s return matters
• (2, 0, 2) — the richest model always fits best
• (0, 0, 1) — bid–ask bounce shows up as an MA(1)

Winner: (0, 0, 0), AIC -8409.8. That is not a failure of ARIMA; it is a fact about the market — and the reason §6.2 exists.

Where ARIMA does work: forecast the passengers

Train on log passengers 1949–1959 (132 months), hold out 1960. Grid \(p, q \in \{0,1,2\}\) with \(d = 1\).

Best by AIC: (2, 1, 1), AIC −228.9. RMSE over 1960: ARIMA(2,1,1) versus the seasonal-naive forecast (1959’s value for the same month). Who wins?

• ARIMA, easily — it has a likelihood
• Roughly a tie
• Seasonal-naive, by a wide margin — the ARIMA has no lag-12 term
• Neither can forecast 12 months ahead

See the flat forecast

ARIMA RMSE 89.4, seasonal-naive 50.7. The forecast settles between 427 and 440 with bands widening from ±84 to ±188. The bands are honest; the model is blind to the season. Order selection is not validation.

Give it eyes: one seasonal term

SARIMA adds a seasonal difference and a seasonal MA at lag 12: order=(0,1,1), seasonal_order=(0,1,1,12) — the classic “airline model”.

RMSE 18.6 — almost five times better than the non-seasonal fit (89.4) and nearly three times better than the seasonal-naive. Same two-parameter budget, one extra idea. AIC agrees: -441.3 versus -228.9. Where a series has structure, the classical model is unbeatable per parameter.

What you discovered

• Prices remember; returns forget. A stationary series has time-invariant mean, variance and autocorrelation — the random walk (\(a = 1\)) has none of those.

• Two diseases, two cures: detrend a trend-stationary series, difference a unit-root series (std 4.06 vs 2.77 when you pick wrong).

• KPSS (null: stationary) and ADF (null: unit root) must be read together; Apple’s minute price landed in the conflict cell, its returns did not. CPI has a unit root (p 0.999); unemployment does not (p 0.011).

• Dow returns: every ACF inside ±0.056; AIC picks ARIMA(0,0,0). \(|r|\) remembers — that is §6.5.

• On passengers, ARIMA(2,1,1) won by AIC but lost the 1960 hold-out to seasonal-naive (89.4 vs 50.7); the airline SARIMA scored 18.6.

Next: §6.2 — score a mean forecast honestly, and see what a gradient-boosting model does with the same series.

Forecast Benchmarks and Walk-forward Learning

Part	Question
6.2.1 Forecast errors and simple benchmarks	Does the model beat a clear baseline?
6.2.2 Chronological features and model comparison	Can every feature be built at the forecast date?
6.2.3 Extensions: sequence and multivariate models	What changes when we use sequences or multiple series?

Forecast errors and simple benchmarks

Question: Does the model beat a clear baseline?

Result: Score naive, zero-change and moving-average forecasts consistently.

Begin with the definitions and assumptions; then trace the worked example.

Four ways to score a forecast

With actual \(y_t\) and forecast \(\hat y_t\) over \(n\) periods:

\[\text{MAE} = \tfrac1n\sum|y_t - \hat y_t|, \quad \text{RMSE} = \sqrt{\tfrac1n\sum (y_t-\hat y_t)^2}, \quad \text{MAPE} = \tfrac{100}{n}\sum\left|\tfrac{y_t-\hat y_t}{y_t}\right|,\]

\[\text{DA} = \tfrac1n\sum \mathbf 1\{\text{sign}(y_t) = \text{sign}(\hat y_t)\}.\]

Which metric is unusable for forecasting minute-by-minute price changes?

• MAE — absolute errors are not differentiable
• RMSE — it over-penalises large errors
• MAPE — it divides by actual values that are near or exactly zero
• Directional accuracy — the sign of a change is meaningless

The naive benchmark: “tomorrow’s change = today’s”

Target Y is the next minute’s price change; the forecast Yhat is the current change.

388 forecasts: MAE 0.1157 dollars, RMSE 0.1694, directional accuracy 0.466 — worse than a coin. Minute changes are slightly negatively autocorrelated (bid–ask bounce), so copying the last change points you the wrong way.

Can “predict nothing” beat “predict the last change”?

Forecast zero for every minute. The RMSE of the zero forecast is just the std of Y around zero. Is it above or below the naive RMSE of 0.1694?

print(RMSE(d.Y, 0 * d.Y))

0.1057

0.1057 vs 0.1694. Copying the last change adds the variance of two changes; predicting the mean (≈ 0) adds none. Any forecast you build must beat 0.1057, not the naive number. That zero forecast is the first baseline in every ML pipeline too.

Simple moving average: average the last 40 changes

\(\hat y_{t+1} = \tfrac{1}{40}\sum_{i=0}^{39} \Delta P_{t-i}\) — .rolling(40).mean(). Predict: does averaging out the bounce beat the zero forecast (0.1057)?

On the common 349 rows: SMA RMSE 0.0993, zero 0.0986, naive 0.1428. The SMA crushes naive and ties the zero forecast; its only edge is a direction call (DA 0.521) that a zero forecast cannot make. And compare only on the same rows — the naive RMSE moved from 0.1694 to 0.1428 just by dropping 40 warm-up minutes.

EWMA: recent values count more

\(s_t = \alpha\,x_t + (1-\alpha)\,s_{t-1}\), \(s_1 = x_1\). As \(\alpha \to 1\) it is the naive forecast; as \(\alpha \to 0\) the long-run mean.

x = [10, 20, 30], α = 0.3, adjust=False. \(s_1 = 10\), \(s_2 = 0.3·20 + 0.7·10\), \(s_3 = 0.3·30 + 0.7·s_2\). What prints?

print(pd.Series([10., 20., 30.]).ewm(alpha=0.3, adjust=False).mean().tolist())

[10.0, 13.0, 18.1]

EWMA(0.3): RMSE 0.1075, DA 0.496 — between naive and SMA, because α = 0.3 remembers only ~2 minutes and inherits the bounce. The window is the model — and you tuned it by eye. Now let a likelihood choose.

The ARIMA trading test: fixed parameters, one step ahead

Fit ARIMA(2,0,1) — the AIC winner — on AAPL log returns 2020–2023, then feed 2024 through res.apply(new_data) for one-step-ahead forecasts without refitting. Trade the sign.

DA 0.480, RMSE 0.01451 — worse than forecasting the training mean (0.01421); Sharpe -0.20 against 1.16 for holding. The AIC winner in-sample is a loser out of sample. This is the result that moved mean forecasting off the classical desk.

Chronological features and model comparison

Question: Can every feature be built at the forecast date?

Result: A walk-forward learner with lagged inputs and an unchanged test period.

Builds on Forecast errors and simple benchmarks.

The pivot: what the ML pipeline inherits from time series

ML did not replace time series; it sits on top of it. Four things every return-prediction model in Chapter 4 quietly assumed:

Inherited discipline	From	Failure if skipped
Stationary features — returns, ratios, differences, never price levels	§6.1	tree splits on a level that never recurs; the model memorises 2021
Lag to avoid leakage — every feature at \(t\) uses data to \(t-1\)	.shift(1)	in-sample accuracy 0.9, live accuracy 0.5
Walk-forward validation — expanding window, refit per period	§6.2	random K-fold lets the future train the past
A baseline that must be beaten — zero forecast, ARIMA(0,0,0)	§6.1–5.2	a Sharpe with nothing to compare it to

Tip

The ML model is the last step of a time-series pipeline. Most of the value is in the first three rows.

Build the features — and spot the leak first

Target is y = r (today’s return). A colleague builds X[“std21”] = r.rolling(21).std() with no shift. What is wrong?

• Nothing — volatility is a legitimate feature
• The window at day \(t\) contains \(r_t\) itself: the target leaks into the feature
• Rolling std needs at least 60 days to be stable
• The feature is non-stationary

Eleven features, all stationary (returns, rolling moments, a ratio), all shifted. The first fold trains on 2016–2017 and tests on 2018; each later fold expands the training window by a year. That is the walk-forward split.

Gradient boosting, trained the way a desk would train it

Same spec as Chapter 4: HistGradientBoostingRegressor(max_depth=2, learning_rate=0.05, max_iter=200, min_samples_leaf=50). Refit every January on all data to date; predict the year; trade long/flat on the sign.

Honest numbers: DA 0.494, RMSE 0.01276 — worse than the zero forecast (0.01247), correlation -0.013; the rule’s Sharpe 0.31 against 0.57 for holding the index, in the market 59 % of days. No year’s DA clears 0.54. The same walk-forward discipline that exposed ARIMA exposes the GBM. This experiment supports keeping chronological validation and a common-date baseline. It does not establish that cross-sectional learning, deeper models or richer features will succeed on another task.

Extensions: sequence and multivariate models

Question: What changes when we use sequences or multiple series?

Result: Explain the extra structure and distinguish Granger prediction from causation.

Builds on Chronological features and model comparison.

What about LSTM and sequence models?

In Colab

import torch, torch.nn as nn
class LSTMReg(nn.Module):
    def __init__(self, n_feat, hidden=32):
        super().__init__(); self.lstm = nn.LSTM(n_feat, hidden, batch_first=True); self.head = nn.Linear(hidden, 1)
    def forward(self, x): out, _ = self.lstm(x); return self.head(out[:, -1])
model = LSTMReg(n_feat=11); opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for epoch in range(20):                       # X_seq: (batch, 21 days, 11 features), y: next-day return
    opt.zero_grad(); loss = ((model(X_seq) - y).pow(2)).mean(); loss.backward(); opt.step()

• Gu, Kelly & Xiu (RFS 2020): on the US stock cross-section, shallow trees and 2–3-layer nets beat deep ones; the gain over linear models comes from interactions and features, not from depth.
• Zero-shot foundation models (Chronos, TimesFM): no consistent edge over a GBM baseline on daily returns — there is too little predictable structure in returns for pre-training to transfer.
• Where sequence models do help: intraday order flow, the volatility surface, and joint modelling of many series at once.

Your LSTM beats the GBM by 4 % accuracy in-sample. What do you check first?

• Increase the hidden size — the LSTM is under-fitting
• Walk-forward out-of-sample on the same features and folds, plus a leakage audit of the sequence windows
• Switch the loss from MSE to directional accuracy
• Add more epochs

Aside: when you have several series — Granger causality

\(X\) Granger-causes \(Y\) if lags of \(X\) improve the forecast of \(Y\) beyond \(Y\)’s own lags — an F-test on the \(X\) coefficients in \(Y\)’s equation (a VAR is \(k\) such regressions on shared lags). Forecastability, not mechanism.

p_nvda_to_spy is done. Compute p_spy_to_nvda — the p-value that SPY’s past helps forecast NVDA — using granger_p with 2 lags.

NVDA → SPY 0.6835, SPY → NVDA 0.1257. Neither direction at 5 %: in daily returns the lead–lag is priced away within the day. On macro data (Stock–Watson, FRED-MD) Granger tests are informative; on liquid daily prices they mostly confirm efficiency.

What you discovered

• Score forecasts with MAE, RMSE and DA on the same rows; MAPE only away from zero. The naive forecast (RMSE 0.169) loses to predicting zero (0.106); SMA(40) ties zero (0.099 vs 0.099).

• The AIC-winning ARIMA on AAPL: 2024 DA 0.48, RMSE worse than the training mean, Sharpe −0.20 vs 1.16 for holding.

• What ML inherits: stationary features, .shift(1) before anything else, expanding-window walk-forward, and a baseline to beat.

• The GBM, trained walk-forward on eleven lagged/rolling features of the S&P 500: DA 0.494, Sharpe 0.31 vs 0.57. This experiment found no mean-forecasting advantage; it does not establish a universal ranking of learners or datasets.

• NVDA/SPY Granger causality: 0.68 and 0.13 — nothing at daily frequency.

Next: §6.3 — stop forecasting a direction; trade a stationary relationship instead.

Cointegration and Pairs Trading

Part	Question
6.3.1 Spurious regression and cointegration	Does a strong price regression imply a stable spread?
6.3.2 Formation and later-sample stability	Does the relationship survive the formation period?
6.3.3 Hedge ratios, signals and trading rules	How does a spread become a dated position?

Spurious regression and cointegration

Question: Does a strong price regression imply a stable spread?

Result: Test for cointegration and distinguish it from ordinary correlation.

Begin with the definitions and assumptions; then trace the worked example.

First, the trap: a regression of two strangers

Regress Apple’s log price (2015–2024) on the Dow’s log price from 1985–1995, lined up row by row. The two series have nothing to do with each other. Predict the R².

Apple 2015–24 on Dow 1985–95, 2 516 rows each. What will R² and the t-statistic look like?

• R² ≈ 0, t ≈ 0 — unrelated series
• R² ≈ 0.05, t ≈ 2 — mild chance correlation
• R² ≈ 0.3, t ≈ 5
• R² ≈ 0.8, t > 100 — a “highly significant” fiction

R² 0.804, t 101.7 — and an ADF p of 0.2138 on the residual: the “relationship” wanders. Rule: a regression between two unit-root series is only meaningful if the residual is stationary. That is the cointegration test.

The test: Engle–Granger in one call

1. Each series has a unit root (ADF fails).
2. Regress \(\log P^A_t = \alpha + \beta \log P^B_t + u_t\).
3. Test \(u_t\) for a unit root — with harsher critical values, because \(\hat\beta\) was chosen to make \(u_t\) look as stationary as possible.

statsmodels.tsa.stattools.coint(a, b) does steps 2–3 and returns (t_stat, p_value, critical_values).

You run a plain ADF on the OLS residual and get p = 0.024, but coint() reports p = 0.080. Which is right?

• ADF — coint() is just ADF with more decimals
• coint() — the residual’s β was estimated, so the critical values must be harsher
• Neither — average them
• ADF, because the sample is large

NVDA and SPY: tied by a rope?

Both levels keep their unit roots (NVDA p 0.23, SPY p 0.79). β = 4.38; the naive ADF on the residual says p 0.024, but Engle–Granger says p 0.080. Not cointegrated at 5 %, borderline at 10 %: the rope is loose. The residual plot shows the drunks drifting apart for months at a time.

Formation and later-sample stability

Question: Does the relationship survive the formation period?

Result: A dated formation/trading split with honest rejection criteria.

Builds on Spurious regression and cointegration.

Formation then trade: the discipline, on a pair with no rope

Estimate the rope on 2023 (formation), then watch the spread in 2024 (trading) — never the other way round.

The 2023 residual already fails (ADF p 0.32), and in 2024 the spread’s mean sits at -0.227 — 1.5 formation-sigmas below zero, and it never came back. A z-score would have fired 17 “entry” days on a spread with no anchor. No rope, no pairs trade. (AAPL vs the S&P over 2015–2024 fails too: coint p = 0.25.)

A rope that holds: Alphabet and the S&P 500, 2015–2020

Five years of daily closes, 1 259 days. Predict: will coint reject at 5 %?

Both have unit roots (p 0.85, 0.80). β = 1.271, Engle–Granger t = -3.800 against a 5 % critical value of -3.341: p = 0.0136. Cointegrated at 5 %. A deviation halves in 24.3 days — slow, but it comes back. This is the rope.

Why does in-sample cointegration overstate tradability?

The full-sample test rejects at p = 0.014. Split into formation (first 3 years) and trading (last 2). What do you expect on the formation window alone?

• The same p ≈ 0.01 — cointegration is a property of the stocks
• A smaller p — fewer points means less noise
• A larger p, possibly above 0.05 — the full-sample β used the trading window to fit itself
• p is undefined without both windows

Formation to Oct 2017: p 0.239; to Oct 2018: p 0.113. Neither rejects at 5 %, and the trading-window spread drifts to a mean of 0.071 (1.8 formation-sigmas) in the first case. The rope you can see in 2020 is not the rope you could have tied in 2017. So: re-estimate it as you go.

Hedge ratios, signals and trading rules

Question: How does a spread become a dated position?

Result: Trace a fitted hedge ratio into a z-score, entry and exit rule.

Builds on Formation and later-sample stability.

Walk-forward hedge ratio and the z-score

Every 63 trading days, refit \((\alpha, \beta)\) on the trailing 252 days and hold them for the next 63. The spread uses only ropes that were known at the time; the z-score uses a 40-day rolling mean and std.

16 refits, 1 007 trading days from October 2016. The hedge ratio moves from 0.569 to 1.601 — the rope is not a constant. 110 days have |z| > 2: those are the candidates. Now the rule.

The rule: enter at |z| > 2, exit at |z| < 0.5

Short the spread (short GOOG, long β·SPX) when \(z > 2\); long it when \(z < -2\); close when \(|z|\) falls below 0.5. Yesterday’s position earns today’s spread change.

In a trade 35 % of days, 52 entries and exits. Pairs: cumulative 30.5 %, Sharpe 0.75, max drawdown -11.6 %. Holding GOOG: 68.6 %, Sharpe 0.63, drawdown -36.8 %. Less money, better money — and market-neutral: the book made it through March 2020 with a third of the drawdown. Before costs.

Your turn: a looser trigger

Change entry to 1.5 (keep exit at 0.5) and read off sh_rule. More trades — but is it a better book, or just a busier one?

Entry 1.5: in a trade 47 % of days, 80 round trips, cumulative 53.0 %, Sharpe 1.09, drawdown -11.6 %. Better on paper — with 80 crossings the transaction costs you ignored are now 50 % larger. One tuned threshold on one pair is a hypothesis, not a book.

What you discovered

• Regressing two random walks gives R² 0.80 and t 102 from nothing. Only a stationary residual — coint, with its harsher critical values — makes a level regression meaningful.

• NVDA/SPY: p 0.080, and the 2024 spread never returned to its 2023 anchor. AAPL/S&P: p 0.25. No rope, no pairs trade.

• GOOG/S&P 500 2015–2020: t −3.80, p 0.014, half-life 24 days — cointegrated. But on the formation windows alone p is 0.24 and 0.11: the in-sample rope is always tighter than the one you could have tied.

• Walk-forward: refit the hedge ratio (0.57 → 1.60), z-score on 40 days, enter at 2, exit at 0.5: Sharpe 0.75, drawdown −11.6 % vs 0.63 and −36.8 % for GOOG alone.

Next: §6.4 — the hedge ratio as a state that moves every day (Kalman), and the market’s regime as a hidden state (Markov switching).

Kalman Filters and Regime Switching

Part	Question
6.4.1 Latent levels and recursive filtering	How do we estimate an unobserved state from noisy measurements?
6.4.2 Time-varying hedge ratios	What changes when beta evolves through time?
6.4.3 Regime probabilities and risk rules	Can changing market states inform a decision?

Latent levels and recursive filtering

Question: How do we estimate an unobserved state from noisy measurements?

Result: Trace prediction, innovation and correction through one Kalman update.

Begin with the definitions and assumptions; then trace the worked example.

A hidden level behind noisy data

Local level model: \[\mu_t = \mu_{t-1} + \eta_t,\ \eta_t\sim N(0,Q) \qquad y_t = \mu_t + \varepsilon_t,\ \varepsilon_t \sim N(0,R).\]

Keep two numbers: the estimate \(\hat\mu_{t|t}\) and its variance \(P_{t|t}\).

• Predict: \(\hat\mu_{t|t-1} = \hat\mu_{t-1|t-1}\), \(\;P_{t|t-1} = P_{t-1|t-1} + Q\).
• Update: gain \(K_t = \dfrac{P_{t|t-1}}{P_{t|t-1}+R}\), \(\;\hat\mu_{t|t} = \hat\mu_{t|t-1} + K_t\,(y_t - \hat\mu_{t|t-1})\), \(\;P_{t|t} = (1-K_t)P_{t|t-1}\).

Observations become much noisier (\(R\) ↑). The Kalman gain \(K_t\)…

• falls — trust the model more, the data less
• rises — noisy data needs bigger corrections
• is unchanged — \(K\) depends only on \(Q\)
• becomes negative

One update step by hand — six lines

Start \(\hat\mu = 0\), \(P = 1\), with \(Q = 0.5\), \(R = 1\). Observe \(y = 2\). Predict: \(P \to 1.5\); \(K = 1.5/2.5\); \(\hat\mu \to 0 + K\cdot 2\). What are K and mu (approximately)?

print(K, mu)

0.6 1.2

[1.2]: 60 % of the surprise (\(2 - 0\)) is absorbed. New estimate = old prediction + gain × prediction error. Six lines; that is the whole filter.

Filter Apple’s minute price — and let statsmodels estimate Q and R

sm.tsa.UnobservedComponents(y, level="local level") estimates \(Q\) (sigma2.level) and \(R\) (sigma2.irregular) by maximum likelihood, then filters.

\(\sqrt R\) = $0.048 — about a nickel of bid–ask bounce per minute; \(\sqrt Q\) = $0.096 — the true level moves twice as much. Steady-state gain ≈ 0.83: the filter follows the data closely because the level genuinely wanders. The six-line filter matches statsmodels to 0.005.

Time-varying hedge ratios

Question: What changes when beta evolves through time?

Result: Interpret a filtered hedge ratio and its uncertainty.

Builds on Latent levels and recursive filtering.

In §6.3 the rope moved from 0.57 to 1.60 and you re-tied it every quarter. A Kalman filter re-ties it every day: let \(\beta_t\) itself be a random walk.

A time-varying beta: the same filter with a regressor

\[\beta_t = \beta_{t-1} + \eta_t,\ \eta_t \sim N(0,Q) \qquad y_t = \beta_t\,x_t + \varepsilon_t,\ \varepsilon_t \sim N(0,R).\]

The only change: the observation is \(\beta_t x_t\), so the gain is \(K_t = P\,x_t/(x_t^2 P + R)\) and the update is \(\beta + K_t(y_t - \beta x_t)\).

Set \(Q = 0\). What does the filtered \(\beta_t\) become?

• Zero — no signal means no beta
• The rolling 60-day OLS beta
• Recursive least squares: it converges to the full-sample OLS beta and stops moving
• Undefined — the gain divides by zero

NVDA’s beta on SPY, one day at a time

Daily returns in percent, 2023–2024. Static OLS gives β = 2.00 in 2023 and 2.64 in 2024 — one number per year. Set \(R\) to the OLS residual variance and the signal-to-noise ratio \(Q/R = 0.001\).

Quarter-end betas 1.72, 2.28, 2.05, 1.73, 2.66, 2.67, 3.18, 1.73 around a static 2.307: the filter sees the 2024 rise and the late-2024 fall that an annual OLS would report a year late. Honest number: the hedge error is 2.475 with yesterday’s Kalman beta versus 2.463 with the look-ahead static one and 3.068 unhedged. On 500 days of a single beta, the gain is timeliness, not variance — no window to choose, no refit schedule.

Regime probabilities and risk rules

Question: Can changing market states inform a decision?

Result: Distinguish filtered and smoothed probabilities before evaluating an overlay.

Builds on Time-varying hedge ratios.

Calm and stormy are not two values of \(\sigma\) on a continuum; they are two states the market jumps between. Hamilton (1989): a Markov chain nobody observes drives the parameters.

Two regimes, switching variance: fit it to the S&P 500, 2015–2024

\[r_t = \mu_{S_t} + \sigma_{S_t}\,\varepsilon_t, \qquad P(S_t = j \mid S_{t-1} = i) = p_{ij}, \qquad S_t \in \{0, 1\}.\]

Six parameters: two means, two variances, two staying probabilities. The filter returns \(P(S_t = \text{calm} \mid \text{data to } t)\) every day; the smoother uses the whole sample.

If the staying probabilities are \(p_{00} = 0.98\) (calm) and \(p_{11} = 0.96\) (stormy), the expected durations are…

• 98 and 96 days
• 50 and 25 days — \(1/(1-p_{ii})\)
• 2 and 4 days
• Equal — a Markov chain has no memory

Calm: sd 0.602 % a day, mean +0.107, expected duration 54.8 days. Stormy: sd 1.778 %, mean -0.096, duration 26.6 days — three times the volatility, a negative drift, and the desk spends 68 % of its days in the calm state. One fit, 0.3 seconds.

Where were the storms?

34 of 120 months are mostly stormy: January 2016, February and the fourth quarter of 2018, March–June 2020, all of 2022 — and August 2024. The model has no calendar; it found the bear markets from the variance alone.

What a regime filter does to drawdown

Hold the index when yesterday’s filtered probability of calm exceeds 0.5. This uses one-sided states, but full-sample fitted parameters make it a retrospective illustration; a live test needs earlier parameter estimates.

In the market 67.9 % of days. Regime filter: cumulative 76.4 %, Sharpe 0.81, max drawdown -11.3 %. Buy-and-hold: 105.0 %, 0.59, -41.4 %. It gave up a quarter of the return to cut the worst drawdown by three quarters. Like every volatility tool in this chapter it forecasts turbulence, not direction.

What you discovered

• The Kalman filter is predict → update with gain \(K = P/(P+R)\); on Apple’s minute bars MLE found a nickel of noise and a dime of true movement, and six lines matched statsmodels to 0.005.

• With a regressor the same filter tracks a time-varying beta: NVDA’s beta swung 2.7 → 3.2 → 1.7 within 2024. \(Q\) is the memory dial; \(Q = 0\) is recursive least squares.

• Markov switching on the S&P 500: calm sd 0.60 % (55-day spells), stormy sd 1.78 % with negative drift (27-day spells). The model found 2016, 2018, 2020, 2022 and August 2024 without a calendar.

• A filter that holds only when \(P(\text{calm}) > 0.5\): Sharpe 0.81 vs 0.59, drawdown −11.3 % vs −41.4 % — for a quarter less return.

Next: §6.5 — the regime model said variance is what changes. Model it directly: ARCH and GARCH.

ARCH, GARCH and Conditional Risk

Part	Question
6.5.1 Volatility clustering and ARCH evidence	Does return magnitude depend on recent history?
6.5.2 GARCH estimation and variance forecasts	How does a fitted recursion produce future variance?
6.5.3 Value-at-Risk and exception testing	Do the risk forecasts have the claimed coverage?

Volatility clustering and ARCH evidence

Question: Does return magnitude depend on recent history?

Result: Read squared-return dependence and an ARCH-LM test.

Begin with the definitions and assumptions; then trace the worked example.

Which is easier to forecast: the sign of tomorrow’s return, or its size?

For daily Dow Jones returns, which is more predictable from yesterday?

• The sign (up or down)
• The size (how big the move is)
• Both equally
• Neither — returns are i.i.d.

The daily standard deviation was 0.417 % in 2017 and 6.381 % in March 2020 — a 15× swing in scale, a 234× swing in variance. The worst day, −13.84 % on 2020-03-16, sits inside a cluster of other huge days. Quiet begets quiet; storms beget storms.

Predict: the ACF of \(r_t\) versus the ACF of \(r_t^2\)

The white-noise band is \(\pm 2/\sqrt{1479} = \pm 0.052\).

Which pattern will acf(r) and acf(r**2) show at lags 1–5?

• Both inside the ±0.052 band — returns are white noise, so are their squares
• Both far outside the band — returns are strongly autocorrelated
• \(r_t\) mostly inside the band; \(r_t^2\) far outside at every lag
• \(r_t\) far outside; \(r_t^2\) inside

Squared returns: ACF 0.442, 0.551, 0.325, 0.324, 0.297 — every lag far outside ±0.052, and \(Q(10) = 1929\). Raw returns: −0.195, 0.157, −0.015, −0.080, 0.079 — small, though \(Q(10)=321\) is not zero either (the March 2020 reversals).

Engle’s ARCH and Bollerslev’s GARCH

\[\varepsilon_t = \sigma_t z_t,\ z_t \sim N(0,1),\qquad \text{ARCH}(q):\ \sigma_t^2 = \omega + \sum_{i=1}^{q}\alpha_i \varepsilon_{t-i}^2,\qquad \text{GARCH}(1,1):\ \sigma_t^2 = \omega + \alpha\,\varepsilon_{t-1}^2 + \beta\,\sigma_{t-1}^2 .\]

\(\varepsilon_t\) stays white noise (\(\mathbb E[\varepsilon_t\mid\text{past}] = 0\)) while \(\varepsilon_t^2\) follows an AR(\(q\)). Unconditional variance \(\omega/(1-\alpha-\beta)\); persistence \(\alpha+\beta\).

Compared with ARCH(\(q\)), the \(\beta\) term makes the ACF of \(\varepsilon_t^2\)…

• Cut off sharply after lag 1
• Decay geometrically at rate \(\alpha+\beta\), never cutting off — GARCH(1,1) is ARCH(\(\infty\))
• Zero at every lag
• Constant at every lag

The ARCH-LM test: one OLS you already know

Engle’s ARCH-LM test for order \(q\) regresses:

• \(r_t\) on \(r_{t-1},\dots,r_{t-q}\) and reports the F-test
• \(|r_t|\) on time and reports the slope
• \(\hat\varepsilon_t^2\) on \(\hat\varepsilon_{t-1}^2,\dots,\hat\varepsilon_{t-q}^2\) and reports \(LM = nR^2 \sim \chi^2_q\)
• \(\hat\varepsilon_t\) on \(\hat\varepsilon_t^2\) and reports \(R^2\)

\(R^2 = 0.362\) on a regression that “should” have \(R^2 = 0\): \(LM = 533.2\) against a \(\chi^2_5\) whose 1 % critical value is 15.1. het_arch reports the same 533.2 — you just re-implemented it. (On 1 479 i.i.d. normals the same code gives \(LM \approx 5\), \(p = 0.42\).)

Your turn: does Apple have ARCH effects too?

Load appl.csv, build percent log returns for 2018–2020 into ra, and run arch_lm(ra.values, q=5). The starter leaves LM_a as a placeholder; replace the marked line so that LM_a is Apple’s ARCH-LM statistic.

Expected once fixed: \(LM = 132.6\), \(p < 10^{-4}\) on 756 days, with squared-return ACF 0.319, 0.302, 0.159. Apple clusters like the index.

GARCH estimation and variance forecasts

Question: How does a fitted recursion produce future variance?

Result: Explain GARCH parameters, persistence and forecast horizons.

Builds on Volatility clustering and ARCH evidence.

In Colab: the notebook fits GARCH in one line — read the output carefully

In Colab

from arch import arch_model
re = dji["Return"]                       # decimal log returns, 1985 onward
GARCH11 = arch_model(re.iloc[0:252*5], p=1, q=1).fit(disp="off")
print(GARCH11.summary())
#   mu        1.0176e-03
#   omega     3.5235e-06   std err 4.250e-12
#   alpha[1]  0.1000       std err 5.500e-02
#   beta[1]   0.8800       std err 3.678e-02

\(\hat\alpha = 0.1000\) and \(\hat\beta = 0.8800\) to four decimals. What is the most likely explanation?

• The Dow really has \(\alpha=0.10\), \(\beta=0.88\) exactly
• arch rounds to two decimals
• These are the optimiser’s starting values — it never moved because the data were in decimals
• Only 1 260 observations, so the estimate is coarse

The estimator in under 20 lines

\(r_t = \mu + \varepsilon_t\). Gaussian log-likelihood of one observation: \(-\tfrac12\left[\ln(2\pi\sigma_t^2) + \varepsilon_t^2/\sigma_t^2\right]\). Sum, negate, minimise — in percent, so \(\omega\) is of order 0.01, not \(10^{-6}\).

Fit it to the Dow, 2015–2020

Equity indices typically show \(\alpha \approx 0.05\)–\(0.2\) and \(\alpha+\beta \approx 0.95\)–\(0.99\). Will alpha + beta land above or below 0.95?

print(alpha + beta)

0.975 — above

\(\hat\mu=0.082\), \(\hat\omega=0.037\), \(\hat\alpha=0.214\), \(\hat\beta=0.761\). Persistence 0.975 — a shock to variance halves in 27 days. Long-run sd 1.217 % vs sample 1.240 %. Well under a second.

The conditional volatility path

From 0.43 % on 2017-11-09 to 8.88 % on 2020-03-17 — 141 % annualised. Volatility is not a number; it is a time series, and the model tracks it one day at a time.

Forecast: one step, then many

Tomorrow is exact: \(\hat\sigma^2_{T+1} = \omega + \alpha\varepsilon_T^2 + \beta\sigma_T^2\). Beyond that, \(\mathbb{E}[\varepsilon^2_{T+h}] = \sigma^2_{T+h}\), so \(\hat\sigma^2_{T+h} = \omega + (\alpha+\beta)\,\hat\sigma^2_{T+h-1}\).

On 2020-11-13, \(\varepsilon_T = 1.283\) and \(\sigma_T^2 = 1.913\). What is \(\hat\sigma^2_{T+1}\) to 3 decimals?

0.0374 + 0.214*1.283**2 + 0.761*1.913

1.845

1.845 → 1.810 → 1.704 → 1.560, sliding toward the long-run 1.48. A GARCH forecast is a decay path, not a level: the further out, the less today matters.

Value-at-Risk and exception testing

Question: Do the risk forecasts have the claimed coverage?

Result: Translate variance into VaR and count held-out exceptions.

Builds on GARCH estimation and variance forecasts.

From variance to Value-at-Risk: normal or Student-\(t\)?

\(\text{VaR}_p = -\left(\hat\mu + \hat\sigma_{T+1}\, q_p\right)\), a positive loss. With \(q_{0.01} = -2.326\), \(\hat\mu = 0.082\), \(\hat\sigma_{T+1} = \sqrt{1.845} = 1.358\):

print(VaR_1pct) gives approximately which value?

-(0.082 + 1.358 * (-2.326))

3.077

Standardised residuals still have excess kurtosis 2.43; the fitted \(\nu = 12.7\). Tomorrow’s 1 % VaR: 3.08 % (normal) vs 3.23 % (\(t\)). In-sample the normal 1 % VaR is breached 2.5 % of days, the \(t\) 2.2 % — both too often. At 5 % both are close (5.8 %, 6.1 %). Fat tails survive GARCH; they are just thinner.

The honest test is out of sample: back-test with Kupiec

Fit on 2015–2018 (1 006 days), freeze the parameters, run the recursion forward through 2019–2020 (473 days), count breaches of the 1 % VaR. Compare with an unconditional VaR from the training mean and sd.

Over 2019–2020 the 1 % VaR should be breached about 5 times (473 × 0.01). Which is closer to 5?

• Unconditional VaR — a fixed number is more robust
• Both hit exactly 5
• GARCH — it re-scales within days; unconditional VaR is breached far more, mostly in March 2020
• Neither breaches at all

1 % VaR: GARCH 18 breaches (3.8 %, Kupiec \(p<0.001\)), unconditional 31 (6.6 %). 5 % VaR: GARCH 32 (6.8 %, \(p = 0.09\) — not rejected), unconditional 47 (9.9 %). GARCH-normal is still too thin at 1 %; the fixed-sd model is simply wrong. The unconditional line is crossed ten times in March 2020 alone; GARCH’s is crossed once — it was already at \(\sigma \approx 6\) %.

What you discovered

• Daily returns: unpredictable sign, predictable size. 2017 sd 0.42 %, March 2020 sd 6.38 %. ACF of \(r_t^2\): 0.44, 0.55, 0.33 …; ARCH-LM 533 on the Dow, 133 on Apple, 5 on white noise.

• GARCH(1,1) by MLE is 15 lines: a variance recursion, a Gaussian log-likelihood, minimize with bounds. Dow 2015–2020: \(\alpha=0.214\), \(\beta=0.761\), half-life 27 days. Work in percent or the optimiser stalls at its starting values.

• Conditional \(\sigma_t\) ranged 0.43 % → 8.88 %. Multi-step forecasts decay geometrically toward \(\omega/(1-\alpha-\beta)\).

• \(\text{VaR}_p = -(\hat\mu + \hat\sigma_{T+1} q_p)\). Standardised residuals keep excess kurtosis 2.4; the 1 % normal VaR is breached 3.8 % of days out of sample — the unconditional VaR 6.6 %.

Next: §6.6 — does bad news raise volatility more than good news? And what happens when you size positions by \(\hat\sigma_t\)?

Variance forecasts need their own scoreboard

Fit/select earlier; compare forecasts on the same later dates, horizon and return units.

Forecast target	Score / check	What it assesses
Return mean	RMSE; directional accuracy	The centre or sign, not changing variance.
Variance \(\hat h_t>0\)	MSE \(=\operatorname{mean}[(\epsilon_t^2-\hat h_t)^2]\)	Large variance errors receive more weight.
Variance \(\hat h_t>0\)	QLIKE \(=\operatorname{mean}[\log\hat h_t+\epsilon_t^2/\hat h_t]\)	Compare with constant variance and EWMA; smaller is better.
Conditional tail	VaR exception frequency and clustering	Tail calibration; coverage alone is incomplete.

\(\epsilon_t=r_t-\hat\mu_t\): use an available mean estimate. One squared daily residual is a noisy variance proxy, not the latent variance.

Asymmetric Volatility and Position Sizing

Part	Question
6.6.1 Asymmetric news and GJR-GARCH	Do negative and positive shocks have different effects?
6.6.2 Mean, variance and risk-timing rules	How do mean and variance forecasts enter a policy?
6.6.3 Volatility targeting	What position corresponds to a target risk level?

Asymmetric news and GJR-GARCH

Question: Do negative and positive shocks have different effects?

Result: Read an asymmetry coefficient, news-impact curve and nested comparison.

Begin with the definitions and assumptions; then trace the worked example.

Predict: does a −2 % day raise tomorrow’s variance more than a +2 % day?

Symmetric GARCH cannot tell them apart: \(\varepsilon_{t-1}^2\) is the same. Glosten, Jagannathan and Runkle (1993) add an indicator:

\[\sigma_t^2 = \omega + \left(\alpha + \gamma\,\mathbf 1[\varepsilon_{t-1}<0]\right)\varepsilon_{t-1}^2 + \beta\sigma_{t-1}^2 .\]

For a stock index, the sign of \(\gamma\) is expected to be:

• Negative — good news is more surprising
• Positive — bad news raises volatility more (the leverage effect)
• Zero — the square already captures size
• Depends on the seed

Fit GJR-GARCH and the nested symmetric model

Same likelihood, one more parameter. Setting the bound on \(\gamma\) to \((0,0)\) gives plain GARCH(1,1) — so one function fits both and the likelihood-ratio test is free.

Is \(\gamma\) real? Likelihood-ratio test

\(LR = 2(\ell_{GJR} - \ell_{GARCH})\), \(\chi^2_1\) under \(H_0: \gamma=0\) (5 % critical value 3.84). Log-likelihoods −1815.05 and −1836.45. What is \(LR\)?

2 * (-1815.05 - (-1836.45))

42.8

\(\hat\gamma = 0.254\), \(LR = 42.8\), \(p = 6\times10^{-11}\). A −2 % day feeds 0.335 of its square into tomorrow’s variance; a +2 % day only 0.081 — a 4.2× asymmetry. Persistence is unchanged at 0.974. The S&P 500 gives the same story (\(\hat\gamma = 0.272\), \(LR = 38.9\)).

The news-impact curve

Plot tomorrow’s variance as a function of today’s shock, holding \(\sigma_{t-1}^2\) at its long-run level.

After −3 %: variance 4.11; after +3 %: 1.82. The symmetric model splits the difference and is wrong on both sides — too calm after crashes, too nervous after rallies.

Mean, variance and risk-timing rules

Question: How do mean and variance forecasts enter a policy?

Result: Separate the two models and evaluate a dated risk threshold.

Builds on Asymmetric news and GJR-GARCH.

ARMA + GARCH: model the mean, then the variance

Everything so far assumed \(r_t = \mu + \varepsilon_t\). If the mean has ARMA structure, fit it first and hand the residuals to GARCH.

Two-step ARMA+GARCH means:

• Fit ARIMA on \(r_t\), then fit GARCH on the ARIMA residuals
• Fit GARCH on \(r_t\), then fit ARIMA on \(\sigma_t\)
• Fit both on \(r_t\) independently and average
• Fit ARIMA on \(r_t^2\)

In Colab

import armagarch as ag
mdl = ag.empModel(re.to_frame(), ag.ARMA(order={'AR':1,'MA':0}),
                  ag.garch(order={'p':1,'q':1}), ag.tStudent())
mdl.fit(); mdl.summary()     # notebook: AR = -0.066, alpha = 0.1, beta = 0.8 (1985-2020)

Run the two steps

\(\hat\phi = -0.195\) (se 0.008) — but look at the standardised residuals: ACF of \(z\) at lag 1 is +0.151. The AR(1) was pulled by March 2020’s giant reversals; once each day is weighted by \(1/\sigma_t\) the mean dynamics look different. ACF of \(z^2\) (0.03, −0.01, 0.02) is clean — the variance is done. That mismatch is why joint estimation (armagarch, weighted by \(\sigma_t\)) is preferred when the mean matters.

Timing: do calm days pay better per unit of risk?

Apple 2020–2024, 1 258 days. Fit GARCH(1,1), sort days into quartiles of its fitted conditional SD, and compare returns. Parameters and quartile boundaries use the full sample: this is a retrospective description, not a later forecast test.

Compared with the calmest quartile, the most volatile quartile will show:

• Much higher mean return — risk is rewarded day by day
• Similar mean return but roughly double the sd — worse return per unit of risk
• Negative mean return always
• Zero sd

Calm quartile: mean 0.104 %/day, sd 1.35; stormy: mean 0.049, sd 2.91. Return per unit of daily risk falls from 0.077 to 0.017. Apple’s \(\alpha=0.086\), \(\beta=0.885\) — more persistent than the Dow.

Your turn: the threshold rule

Position \(=1\) when \(\hat\sigma_t <\) thr, else cash. The starter has thr = 100 — always invested, i.e. buy-and-hold. Set a real threshold (try 2.0; the median \(\hat\sigma\) is 1.70).

Change thr so the rule is in the market on fewer than 99 % of days, then read off cumulative return, annualised Sharpe and maximum drawdown versus buy-and-hold.

At thr = 2.0: in the market 73 % of days; cumulative 95 % vs 123 % for buy-and-hold, Sharpe 0.89 vs 0.78, max drawdown −25.5 % vs −37.7 %. Less money, better risk-adjusted money. An on/off switch is crude, though — the industry version is continuous.

Volatility targeting

Question: What position corresponds to a target risk level?

Result: Trace a lagged volatility forecast into capped exposure and realised risk.

Builds on Mean, variance and risk-timing rules.

Instead of in-or-out, hold \(w_t = \min\!\left(\dfrac{\sigma^\ast}{\hat\sigma_t},\, w_{\max}\right)\) of the index: more when it is calm, less when it is stormy, so that realised volatility stays near \(\sigma^\ast\). This is an inverse-risk sizing rule; the realised target is not guaranteed.

Predict: what does vol targeting do to the S&P 500, 2020–2024?

Target \(\sigma^\ast = 1\) % a day, cap \(w_{\max} = 1.5\). Buy-and-hold over these five years realised 1.35 % a day with a −41 % drawdown.

Compared with buy-and-hold, the vol-targeted book should show…

• Higher return, higher drawdown — it levers up
• Lower return, same drawdown — scaling is a cost
• Realised sd near 1 %, roughly half the drawdown, higher Sharpe, similar total return
• Identical — scaling is a monotone transform

Run it

The recursion uses lagged shocks, but the parameters below are fitted on the full 2020–2024 sample. This is a retrospective sizing illustration. A forecast test requires an earlier fit or dated refits.

Realised sd 0.994 % against the 1 % target (buy-and-hold 1.350). Vol-target: cumulative 60.7 %, Sharpe 0.77, max drawdown -22.6 %; buy-and-hold 59.9 %, 0.56, -41.4 %. Same money, half the drawdown. The weight ran from 0.13 in March 2020 to the 1.5 cap in quiet months. Nothing here forecasts direction — the whole edge is the variance forecast from §6.5.

What you discovered

• One indicator term captures the leverage effect: Dow \(\hat\gamma = 0.254\), \(LR = 42.8\). Bad news moves variance 4.2× more than good news; −3 % → variance 4.11, +3 % → 1.82.

• Two-step ARMA+GARCH: ARIMA for the mean, GARCH on its residuals. If the mean matters, the two-step fit is distorted by heteroskedasticity — estimate jointly.

• Calm days pay 0.077 per unit of daily risk, stormy days 0.017. A “hold only if \(\hat\sigma_t < 2\) %” switch on Apple cut the drawdown from −37.7 % to −25.5 %.

• Volatility targeting on the S&P 500, 2020–2024: realised sd 0.99 % vs 1.35 %, Sharpe 0.77 vs 0.56, drawdown −22.6 % vs −41.4 %, same cumulative return. This is an illustrative risk comparison, not a guarantee of future performance.

Mistakes Library: the Swiss franc floor, 15 January 2015

Warning

From September 2011 the Swiss National Bank held EUR/CHF above 1.20, and for three years the exchange rate behaved like a textbook stationary series pinned to its floor: ADF rejected, KPSS did not, the spread to 1.20 mean-reverted within days. Traders and retail brokers treated the floor as a cointegrating rope between the franc and the euro, ran carry and short-volatility positions against it, and sized them on a variance estimated from those three quiet years.

On 15 January 2015 the SNB abandoned the floor without warning. EUR/CHF fell from 1.20 to 0.85 within minutes — a move of roughly 30 %, about 60 standard deviations of the pre-announcement daily change. FXCM, then the largest US retail FX broker, faced US$225 million of client losses in excess of collateral and needed a US$300 million rescue loan the next day; Alpari UK entered insolvency; Everest Capital’s US$830 million Global fund was wiped out.

Lesson for this chapter: a stationarity test tells you about the sample you fed it. A regime held in place by a policy is stationary until the policy changes, and no ADF statistic, GARCH \(\hat\sigma_t\) or smoothed regime probability can see the meeting at which it does. Test the spread, then ask who is holding the rope.

Decision Memo — is the volatility forecast ready for use?

To: Risk manager Subject: Evidence required before using a conditional-volatility forecast

Model: compare a fixed-variance baseline, an exponentially weighted variance and GARCH; test whether GJR adds useful asymmetry. Keep the mean specification, forecast horizon and return units explicit.

Validation: freeze fitting choices before the evaluation period. Compare forecasts on identical later dates; inspect standardised residuals, variance forecast losses and VaR exception frequency and clustering.

Decision: assess a capped volatility-targeting rule against a fixed-exposure baseline. Report realised risk, drawdown, exposure, turnover and costs. The existing illustrative gross-return comparisons do not authorise deployment.

Limit: a quiet fitting sample cannot rule out a structural break. Stress the position separately and state what would trigger review or refitting.

Working with an AI Copilot

1. Ask for the pipeline before the model. “Predict tomorrow’s S&P return with XGBoost” will get you a random K-fold split and features that include today’s close. Prompt: “List every feature with the date of the last observation it uses; use an expanding-window walk-forward split refit yearly; report the zero-forecast RMSE on the same rows.” An unexpectedly high directional score warrants a leakage audit; the score alone does not prove leakage.
2. Ask for units before code. “Fit GARCH(1,1) to these returns” will often produce arch_model(r) on decimal returns — the stalled fit in the notebook (alpha=0.1000, beta=0.8800). “Scale returns to percent first, and tell me the optimiser’s convergence flag.”
3. Separate the rope from the trade. A copilot that reports coint p = 0.01 and then back-tests a z-score rule on the same window has used the future twice. Ask it to state the formation window, the trading window, and the hedge ratio it used on each day — and to refuse a Sharpe on any spread whose formation-window test did not reject.

Pitfall: when a copilot reports a regime model or a Kalman beta, ask whether it used the smoothed (two-sided) or filtered (one-sided) estimate in the back-test. The smoothed one knows the future.

Chapter Summary

Role	Knowledge to retain	Main tools
Core: diagnose	Return versus squared-return dependence; ARCH evidence	ACF; ARCH-LM; mean residuals
Core: model	ARCH/GARCH; persistence; asymmetric news	Variance recursion; likelihood; GJR indicator; LR limits
Core: forecast	One/multi-step variance; innovation tails; residual checks	\(\alpha+\beta\); standardised residuals; Normal/standardised-t quantiles
Core: evaluate and use	Later risk forecasts, VaR coverage and bounded exposure	Common-date comparison; Kupiec; exception clustering; volatility targeting
Supporting foundations · §6.1	Stationarity, unit roots, ARIMA/seasonal models	ADF/KPSS; ACF/PACF; training AIC; forecast baseline
Supporting protocol · §6.2	Forecast errors, dated features and walk-forward learning	MAE/RMSE/DA; zero baseline; shifts; expanding fits; sequence/VAR extensions
Relationship and state models · §6.3–6.4	Cointegration, pairs, Kalman updates and regimes	Formation/trading separation; dated hedge ratios; filtered versus smoothed states

Main takeaway: conditional volatility is forecastable only to the extent supported by later evidence. A volatility model supplies a risk estimate, not a guaranteed return or a complete trading strategy.

Discussion Questions

1. The walk-forward boosting model did not beat its zero-return benchmark in this experiment. Why can weak predictability of the return mean coexist with dependence in squared returns? Explain which object ARCH/GARCH models, and what additional evidence is needed before claiming a useful volatility forecast.

2. GOOG/S&P 500 passed the Engle–Granger test on the full sample (p = 0.014) but not on either formation window (p = 0.24, 0.11). A colleague proposes trading it anyway because “the rope was there all along”. Write the two-sentence reply, with numbers, and say what evidence would change your mind.

3. The regime filter and the vol-targeting overlay both cut the S&P 500 drawdown by more than half, using only a variance model. Under what return-generating process would either of them raise the total return, not just the Sharpe ratio — and does the quartile table on Apple support that process?

4. The Kalman beta tracked NVDA’s 2024 swing (2.7 → 3.2 → 1.7) a year before annual OLS could, yet the hedge error was the same to two decimals. When does timeliness in a hedge ratio pay, and when is it just noise? Propose one experiment on the NVDA/SPY data that would tell the two apart.