Chapter 4 · Statistical Predictive Models
Section 4.5 · Chapter 4 · Learning Statistics with Python
Statistical Predictive Models
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
Every month a fund sees the same object: a matrix of stocks by characteristics, and next month’s returns as the target. You will standardise it without peeking at the future, replace the sector as “peer group” with one attention layer written in numpy, and put both through the walk-forward protocol of §4.3 — and discover that the protocol, not the architecture, is the lesson.
At month-end \(t\), \(X_t \in \mathbb{R}^{N \times K}\) holds \(N\) stocks × \(K\) characteristics and \(y_t \in \mathbb{R}^N\) holds next month’s returns. The prediction problem is cross-sectional: rank the \(N\) stocks, month by month.
25 335 rows = 119 months × 213 stocks in 11 sectors, 2015-01 … 2024-11. Five price-based characteristics: last-month return, 12-1 momentum, 21-day realised volatility, log dollar volume, price / 252-day high. ret_next at month \(t\) is exactly ret_1_0 at month \(t+1\) (0.100776 = 0.100776) — the target is the next row’s return.
The universe is today’s large caps, tracked back to 2015. Nothing that was large in 2015 and has since failed, shrunk or been delisted is in the file.
What does this survivorship bias do to a backtest run on the file?
Production rule: build the universe as of each month from a point-in-time index membership file. This file is fine for learning the mechanics — never quote its Sharpe ratio as evidence.
Why is (rvol_21d − rvol_21d.mean()) / rvol_21d.std() over the whole 2015–2024 file a look-ahead?
Median and std are computed within each month (and, for the _s columns, within each month × sector — the production invariant: a stock is judged against the peers it is compared with on the day). Sector medians are exactly 0 afterwards; values are clipped at ±3 so one meme month cannot dominate a tree split.
“Sector” is a label assigned by an index provider. In October 2022 NVDA’s behavioural peers — high volume, high volatility, far below the 52-week high — included AMD, but also Netflix, Alibaba and Tesla, none of them “Technology”.
Attention builds the peer group from the data. With queries \(Q = XW_q\), keys \(K = XW_k\), values \(V = XW_v\):
\[A = \text{softmax}_{\text{rows}}\!\left(\frac{QK^\top}{\sqrt{d}\,\tau}\right), \qquad C = AV\]
With \(W_q = W_k = I\), the weight stock \(i\) gives stock \(j\) is proportional to
NVDA’s attention row in October 2022: AMD 0.353, itself 0.263, NFLX 0.154, BABA 0.077, TSLA 0.055, AMZN 0.028 — six stocks carry more than 1 %, from three sectors. The context \(C\) is the peer-weighted characteristic vector: 12-1 momentum −1.49 for NVDA against −1.26 for its peers, dollar volume 3.0 against 2.63.
With identity weights and \(\tau \to \infty\) every score goes to 0. What does NVDA’s context vector collapse to?
the cross-sectional mean of X — every weight is 1/N
Temperature is the dial between two crude peer groups: \(\tau \to 0\) attends only to the most similar stock (at \(\tau = 0.05\) AMD carries 0.997 — NVDA is not even its own nearest point), \(\tau \to \infty\) gives the cross-sectional mean (0.02, 0.10, 0.19, 0.00, −0.14), which plain per-month standardisation already removes. \(\tau = 1\) sits in between: a handful of behavioural peers.
“Artificial Intelligence Asset Pricing Models” (KKMX, 2025) puts a transformer over the stock cross-section, not over words:
What we do instead
Weights stay fixed (\(W = I\); a seeded random \(W\) works the same way and is the random-features view). The attention output is used only as feature engineering: for every stock, the context \(C\) (peer-weighted characteristics) and the deviation \(X - C\) (“how I differ from my attention-weighted peers”). The learner on top is the boosting machine from §4.3.
Fifteen columns per stock-month: 5 market-relative characteristics, 5 peer contexts, 5 deviations. NVDA in October 2022: 12-1 momentum −1.49 against −1.38 for its peers, price / 52-week high −2.67 against −2.59 — deviations of −0.10 and −0.08. With a self-weight of 0.26 and AMD at 0.35, NVDA largely is its peer group that month. Only month-\(t\) rows enter month \(t\)’s attention: no look-ahead.
The row dated 2018-12-31 has ret_next = January 2019’s return. May it sit in the training set when we predict the 2019-01-31 row?
Expanding window from 2019-01: 71 test months, 6 refits (every 12 months, predicting the next 12 — a monthly refit would be 71 fits, same protocol, ten times the runtime). Two scores per month: rank IC (Spearman between prediction and realised return across the 213 stocks) and the top-20 minus bottom-20 equal-weight spread.
Which feature set will have the highest mean rank IC — and will the difference be significant?
Mean rank IC: −0.015 (t = −1.04) raw, −0.014 (t = −0.99) with sector features, −0.010 (t = −0.67) with attention. All three are zero within noise (standard error ≈ 0.015). The long-short spread is positive (+7.7 %, +9.3 %, +2.8 % a year; Sharpe 0.47, 0.62, 0.17, before costs) — the extremes rank slightly better than the middle, but 71 months cannot separate a Sharpe of 0.6 from 0. Attention features did not help here; the protocol that tells you so is the deliverable.
The t-statistic of a spread with Sharpe \(S\) over \(T\) months is \(S\sqrt{T/12}\): with \(T = 71\), the best set’s Sharpe of 0.62 gives t = 1.51; the raw set’s 0.47 gives 1.15; attention’s 0.17 gives 0.40. To reach t = 2 the sector set would need 124 months of the same performance (10 years), the raw set 18 years. Every gain in the plot comes with an equally plausible zero — and the file is survivorship-biased in the spread’s favour.
Change the temperature (tau_new = 3.0) or build the keys from a subset of characteristics (keys_new = ["ret_12_1_z", "rvol_21d_z"]) so that ic_new differs from the baseline attention IC. Does the peer-group definition move the IC by more than its standard error (about 0.015)?
\(\tau = 3\): IC −0.010 (unchanged); keys from momentum and volatility only: −0.025 (t = −1.64); \(\tau = 0.3\): −0.041 (t = −2.60); \(\tau = 10\): −0.008. The sign of the change flips with the setting, and the one “significant” number appears after four tries — that is multiple testing, not a signal. In KKMX the peer group is learned on a Sharpe objective with thousands of stocks and six decades; here it is a fixed kernel on 213 survivors.
Warning
Zillow Offers used the company’s Zestimate models to buy homes directly, renovate and resell them. The models were good predictors of sale prices on the market as observed. But once Zillow acted on them — bidding at the model’s price — the data-generating process changed: sellers accepted the offers the model had overpriced and walked away from the ones it had underpriced (adverse selection), and a 2021 shift in the housing market moved prices faster than the model updated.
In Q3 2021 Zillow wrote down its home inventory by $304 million, disclosed roughly 7 000 homes still to be sold, and on 2 November 2021 shut the business and cut about 25 % of its workforce.
Lesson for this chapter: a model tuned on test RMSE answers “how close is \(\hat y\) to \(y\) under the world that produced the training data?”. The moment your predictions become actions — offers, prices, approvals — you are asking a causal question, and the selection that follows your action is a confounder the test set never contained.
To: Head of Credit, consumer lending From: <Your name>, credit analytics Subject: Move the PD approval cutoff from 0.50 to 0.17 Date: 2026-09-15
Recommendation: approve applicants with predicted PD below 0.17; refer the rest for manual review.
Evidence: - Logistic PD model, 8 origination features, test AUC 0.70 (8 000-loan sample, 30 % held out). - Expected loss per bad loan ≈ EAD × LGD = 0.70 × 0.89 = 0.62 of principal; expected gain per good loan ≈ 0.10. - Profit-maximising cutoff (ROC slope = gain/loss) is 0.17: catches 78 % of defaulters at the cost of 50 % of good applicants, turning −3.8 cents per applicant into +1.4. At 0.50 the model flags only 7 % of defaulters.
Caveats: - LGD is effectively unpredictable (test adjusted R² < 0); the 0.89 mean is the model. - The cutoff is a prediction, not a causal estimate: tightening approvals changes the applicant pool (Zillow lesson). Pilot on a random 10 % of applications for one quarter.
Next step: A/B pilot; re-estimate AUC and the profit curve on the pilot’s realised outcomes.
| Concept | Tool |
|---|---|
| Default, EAD, LGD | map, ledger arithmetic on funded_amnt, total_rec_prncp, recoveries |
| Feature engineering | str.extract, pd.to_datetime(format=), fillna(360), pd.get_dummies(...).iloc[:, :-1] |
| Multiple regression | sm.OLS(y, sm.add_constant(X)).fit(), .rsquared_adj, test RMSE |
| Interactions, best subset | itertools.combinations, adjusted R² envelope over \(k\) |
| Inference | .tvalues ( |
| Trees and ensembles | DecisionTreeRegressor(max_depth, min_samples_leaf), RandomForestRegressor, GradientBoostingRegressor |
| Time-series prediction | lagged features, TimeSeriesSplit walk-forward, per-fold AUC |
| Classification | sm.Logit, confusion table, roc_curve, roc_auc_score, cutoff at ROC slope = gain/loss |
| Causal analysis | confounder adjustment, DiD interaction term, 2SLS via np.linalg.lstsq |
| Cross-sectional attention | per-month (sector) standardisation, Q @ K.T / sqrt(d) + row softmax in numpy, context \(C\) and \(X - C\) features, expanding-window rank IC and top − bottom spread |
Next: Chapter 5 — Rethinking Statistics with Bayesian Methods.
balance and install, and state what it would cost.Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python