6.4 — Structural Arbitrage II: Kalman Filters and Regime Switching

Chapter 6 · Time Series Models for Trading and Risk

Prof. Xuhu Wan

Section 6.4 · Chapter 6 · Learning Statistics with Python

Structural Arbitrage II: Kalman Filters and Regime Switching

Time Series Models for Trading and Risk

Prof. Xuhu Wan

ISOM, HKUST Business School · 2026 Edition

Structural Arbitrage II: Kalman Filters and Regime Switching

A price is a hidden “true level” plus noise; a hedge ratio is a hidden slope that drifts; the market is in a hidden calm or stormy regime. You never observe any of them — but each new data point lets you update your best guess. Six lines of Kalman, eight lines of a time-varying beta, and one MarkovRegression call that cuts a drawdown by three quarters.

A hidden level behind noisy data

Local level model: \[\mu_t = \mu_{t-1} + \eta_t,\ \eta_t\sim N(0,Q) \qquad y_t = \mu_t + \varepsilon_t,\ \varepsilon_t \sim N(0,R).\]

Keep two numbers: the estimate $\hat\mu_{t|t}$ and its variance $P_{t|t}$.

Predict: $\hat\mu_{t|t-1} = \hat\mu_{t-1|t-1}$, $\;P_{t|t-1} = P_{t-1|t-1} + Q$.
Update: gain $K_t = \dfrac{P_{t|t-1}}{P_{t|t-1}+R}$, $\;\hat\mu_{t|t} = \hat\mu_{t|t-1} + K_t\,(y_t - \hat\mu_{t|t-1})$, $\;P_{t|t} = (1-K_t)P_{t|t-1}$.

Observations become much noisier ($R$ ↑). The Kalman gain $K_t$…

falls — trust the model more, the data less
rises — noisy data needs bigger corrections
is unchanged — $K$ depends only on $Q$
becomes negative

One update step by hand — six lines

Start $\hat\mu = 0$, $P = 1$, with $Q = 0.5$, $R = 1$. Observe $y = 2$. Predict: $P \to 1.5$; $K = 1.5/2.5$; $\hat\mu \to 0 + K\cdot 2$. What are K and mu (3 dp)?

print(round(K, 3), round(mu, 3))

0.6 1.2

[1.2]: 60 % of the surprise ($2 - 0$) is absorbed. New estimate = old prediction + gain × prediction error. Six lines; that is the whole filter.

Filter Apple’s minute price — and let statsmodels estimate Q and R

sm.tsa.UnobservedComponents(y, level="local level") estimates $Q$ (sigma2.level) and $R$ (sigma2.irregular) by maximum likelihood, then filters.

$\sqrt R$ = $0.048 — about a nickel of bid–ask bounce per minute; $\sqrt Q$ = $0.096 — the true level moves twice as much. Steady-state gain ≈ 0.83: the filter follows the data closely because the level genuinely wanders. The six-line filter matches statsmodels to 0.005.

The hedge ratio as a hidden state

In §6.3 the rope moved from 0.57 to 1.60 and you re-tied it every quarter. A Kalman filter re-ties it every day: let $\beta_t$ itself be a random walk.

A time-varying beta: the same filter with a regressor

\[\beta_t = \beta_{t-1} + \eta_t,\ \eta_t \sim N(0,Q) \qquad y_t = \beta_t\,x_t + \varepsilon_t,\ \varepsilon_t \sim N(0,R).\]

The only change: the observation is $\beta_t x_t$, so the gain is $K_t = P\,x_t/(x_t^2 P + R)$ and the update is $\beta + K_t(y_t - \beta x_t)$.

Set $Q = 0$. What does the filtered $\beta_t$ become?

Zero — no signal means no beta
The rolling 60-day OLS beta
Recursive least squares: it converges to the full-sample OLS beta and stops moving
Undefined — the gain divides by zero

NVDA’s beta on SPY, one day at a time

Daily returns in percent, 2023–2024. Static OLS gives β = 2.00 in 2023 and 2.64 in 2024 — one number per year. Set $R$ to the OLS residual variance and the signal-to-noise ratio $Q/R = 0.001$.

Quarter-end betas 1.72, 2.28, 2.05, 1.73, 2.66, 2.67, 3.18, 1.73 around a static 2.307: the filter sees the 2024 rise and the late-2024 fall that an annual OLS would report a year late. Honest number: the hedge error is 2.475 with yesterday’s Kalman beta versus 2.463 with the look-ahead static one and 3.068 unhedged. On 500 days of a single beta, the gain is timeliness, not variance — no window to choose, no refit schedule.

Regime switching

Calm and stormy are not two values of $\sigma$ on a continuum; they are two states the market jumps between. Hamilton (1989): a Markov chain nobody observes drives the parameters.

Two regimes, switching variance: fit it to the S&P 500, 2015–2024

\[r_t = \mu_{S_t} + \sigma_{S_t}\,\varepsilon_t, \qquad P(S_t = j \mid S_{t-1} = i) = p_{ij}, \qquad S_t \in \{0, 1\}.\]

Six parameters: two means, two variances, two staying probabilities. The filter returns $P(S_t = \text{calm} \mid \text{data to } t)$ every day; the smoother uses the whole sample.

If the staying probabilities are $p_{00} = 0.98$ (calm) and $p_{11} = 0.96$ (stormy), the expected durations are…

98 and 96 days
50 and 25 days — $1/(1-p_{ii})$
2 and 4 days
Equal — a Markov chain has no memory

Calm: sd 0.602 % a day, mean +0.107, expected duration 54.8 days. Stormy: sd 1.778 %, mean -0.096, duration 26.6 days — three times the volatility, a negative drift, and the desk spends 68 % of its days in the calm state. One fit, 0.3 seconds.

Where were the storms?

34 of 120 months are mostly stormy: January 2016, February and the fourth quarter of 2018, March–June 2020, all of 2022 — and August 2024. The model has no calendar; it found the bear markets from the variance alone.

What a regime filter does to drawdown

Hold the index only when yesterday’s filtered probability of calm exceeds 0.5 (no look-ahead — the smoother is for the picture, the filter is for trading).

In the market 67.9 % of days. Regime filter: cumulative 76.4 %, Sharpe 0.81, max drawdown -11.3 %. Buy-and-hold: 105.0 %, 0.59, -41.4 %. It gave up a quarter of the return to cut the worst drawdown by three quarters. Like every volatility tool in this chapter it forecasts turbulence, not direction.

What you discovered

The Kalman filter is predict → update with gain $K = P/(P+R)$; on Apple’s minute bars MLE found a nickel of noise and a dime of true movement, and six lines matched statsmodels to 0.005.

With a regressor the same filter tracks a time-varying beta: NVDA’s beta swung 2.7 → 3.2 → 1.7 within 2024. $Q$ is the memory dial; $Q = 0$ is recursive least squares.

Markov switching on the S&P 500: calm sd 0.60 % (55-day spells), stormy sd 1.78 % with negative drift (27-day spells). The model found 2016, 2018, 2020, 2022 and August 2024 without a calendar.

A filter that holds only when $P(\text{calm}) > 0.5$: Sharpe 0.81 vs 0.59, drawdown −11.3 % vs −41.4 % — for a quarter less return.

Next: §6.5 — the regime model said variance is what changes. Model it directly: ARCH and GARCH.