Learning Statistics with Python
Chapter 4 · Learning Statistics with Python
Credit-risk case (default, EAD, LGD): feature engineering, multiple regression, model selection, inference, trees and boosting, logistic classification, and why prediction is not causation.
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
statsmodels.Why this matters
Every lending, pricing or marketing model ends in a decision. This chapter gives you the machinery to build the prediction and the discipline to know when the prediction answers the question actually being asked.
| Section | Concept | Tool |
|---|---|---|
| 4.1 Credit-risk data | default, EAD, LGD; feature engineering; OLS for EAD and LGD | map, get_dummies, sm.OLS, R², adjusted R², RMSE |
| 4.2 Model selection | adjusted R², interactions, best subset, residuals, t-tests, VIF | itertools.combinations, .tvalues, variance_inflation_factor |
| 4.3 Nonlinear models | trees, bagging, random forests, boosting, walk-forward stock prediction, logistic classification | DecisionTreeRegressor, RandomForestRegressor, GradientBoosting*, TimeSeriesSplit, sm.Logit, roc_curve |
| 4.4 Causal analysis | confounders, regression adjustment, DiD, instrumental variables | simulation, sm.OLS, 2SLS by hand |
| 4.5 Cross-sectional attention | per-month standardisation, attention as a data-driven peer group, context and deviation features, walk-forward rank IC | numpy softmax attention, HistGradientBoostingRegressor, spearmanr |
A lender’s loss on a loan is the product of three things — whether it defaults, how much is still owed when it does, and how little comes back. You will build all three from raw columns, engineer 25 features, and discover how much (and how little) a linear model explains.
LendingClub matches borrowers (debt consolidation, big purchases, medical bills) with funding. The platform underwrites each application with a credit-risk model, sets the rate and term, and either keeps the loan or sells it to investors.
Which statement best describes the platform’s economics?
\[\text{EL} = \underbrace{\text{PD}}_{\text{probability of default}} \times \underbrace{\text{EAD}}_{\text{exposure at default}} \times \underbrace{\text{LGD}}_{\text{loss given default}}\]
Which factor calls for a classification model rather than a regression?
This section builds EAD and LGD by regression. PD waits for §4.3.
The catalog says 20 % of loans are charged off. What does the second line print for Charged Off?
0.2
(8000, 29) — a random sample of the 150 000-loan original. Exactly 80 % Fully Paid, 20 % Charged Off. Every loan here has finished its life, so we know the outcome.
Three ledger columns carry the story: funded_amnt (what was lent), total_rec_prncp (principal repaid), recoveries (cash clawed back after charge-off).
For a Fully Paid loan, what is EAD = (funded_amnt − total_rec_prncp) / funded_amnt?
Among the 1 604 charged-off loans the mean EAD is 0.698 — on average 70 % of the principal was still outstanding — and mean LGD is 0.892: recoveries claw back barely a tenth. The 1e-4 guards the one loan with EADamount = 0. Six loans recovered more than the outstanding principal (late fees, collection interest), which is why the minimum LGD is −0.177 — real ledgers do this, and we keep the raw number.
Raw columns are strings, dates and NaNs. A regression needs numbers with a meaning. You will convert term and employment length, build a credit age, a FICO midpoint, two “time since trouble” clocks, and dummies for three categoricals.
emp_length holds values like "10+ years", "< 1 year", "3 years" and NaN. What does str.extract(r"(\d+)") return for "10+ years" and for "< 1 year"?
['10', '1']
"< 1 year" would extract as 1, so we override it to 0. The 484 loans with no employment record get employment = 0 and a flag employed = 0 — the flag lets the model treat “unknown” differently from “under a year”.
earliest_cr_line is like "Jan-1986". Days from 1 Jan 1986 to 29 Dec 2018 (the data’s end date)?
12050
format="%b-%Y" parses Jan-1986 directly — no guessing, no warnings. Credit histories run from 1 277 to 20 757 days (3.5 to 57 years). install is the monthly instalment as a share of annual income — a burden ratio, not a dollar amount. Three applicants report zero income: dividing by NaN rather than by 0 keeps their ratio missing instead of infinite.
mths_since_last_delinq is NaN for half the loans. What should NaN become?
balance (instalment-loan balance over income) is NaN when the borrower has no instalment loans — there the honest fill is 0.
Grade has 7 levels A–G. Why keep only 6 dummy columns in a regression with an intercept?
Home ownership also has ANY and OTHER (one loan each). We keep OWN, RENT, MORTGAGE as the notebook does — and in §4.2 the VIF will show why that was a trap.
Why permute the rows before taking the first 70 % as training data?
25 features. Six rows are dropped for a missing dti or revol_util — the three zero-income applicants among them. 5 595 / 2 399 loans train/test; of those, 1 146 / 458 defaulted — the EAD and LGD regressions see only these.
R² = 0.192. Longer term (t = 7.5) and higher int_rate (t = 3.5) mean more principal is still outstanding when the loan fails; a heavier instalment install (t = −2.7) means the borrower had paid more down before failing. fico and dti add nothing once the rest are in.
Same 25 features, target LGD. The R² will be roughly:
R² = 0.039, adjusted 0.018. The 150 000-loan notebook found 0.012. LGD is, for practical purposes, a constant near 0.89 — and a model that knows it is a constant is more useful than one that pretends otherwise.
\[R^2_{\text{adj}} = 1 - (1 - R^2)\,\frac{n-1}{n-k-1}, \qquad \text{RMSE} = \sqrt{\tfrac{1}{n}\sum (y_i - \hat y_i)^2}\]
EAD: train adjusted R² 0.174 → test 0.107. LGD: test adjusted R² is negative (−0.080) — worse than predicting the mean. Test RMSE 0.190 for EAD says a typical exposure forecast is off by 19 percentage points of principal.
You have all three factors on the training data. Put them together.
EL = PD × mean EAD × mean LGD, using the default rate in train and the mean EAD and LGD of the defaulted loans train_d. Fix the EL line.
About 12.7 cents per dollar lent — which is why the average interest rate in this book is 13 %.
"< 1 year" → 0, NaN delinquency → 360 (never), missing balance → 0, one dummy dropped per categorical.Next: §4.2 — can interactions and best-subset search raise that 0.107?
More variables always raise R². You will see why adjusted R² is the honest score, create interaction terms that reveal hidden structure, search all 255 subsets of 8 candidates, and then read t-tests and VIF to decide which coefficients mean anything.
Same seed 5650, same 1 146 / 458 defaulted rows, same adjusted R² 0.174 as §4.1.
We append 10 columns of pure noise (seed 1) to the 25 features and refit.
After adding 10 noise columns, R² and adjusted R² will:
R² 0.1920 → 0.1959; adjusted R² 0.1740 → 0.1706. Ten columns that cannot possibly matter “explained” 0.4 % more variance. Never select on raw R².
y = x1 * x2 + noise with independent standard-normal x1, x2. What are the three correlations corr(y,x1), corr(y,x2), corr(y,x1·x2)?
−0.027, −0.081, 0.887. A term-by-rate interaction in the loan data could hide the same way — so we add products to the candidate pool.
Eight base features and all 28 pairwise products: 36 candidates. (Squares of a two-valued term would duplicate term, so we skip squares.)
The notebook does this for all 25 features plus ratios and reaches 650 columns. Best-subset over 650 columns means \(2^{650}\) models — the reason the next step is a screen, not a search.
Compared with the plain 8-feature model, the 36-column model will have:
36 columns: train 0.195, test 0.079. 8 columns: train 0.178, test 0.141. The interactions bought 1.7 points on training data and cost 6 on test.
Best-subset selection fits every non-empty subset of \(k\) candidates. For \(k = 8\), how many fits?
255
For 36 candidates it would be \(2^{36} - 1 \approx 6.9 \times 10^{10}\). So we first screen the pool by absolute correlation with EAD and keep the top 8 — the notebook keeps the top 15 of 650.
Adjusted R² climbs 0.143 → 0.152 → 0.158 → 0.161 at \(k=4\) and then falls as more candidates are added. The winner: int_rate, term*fico, int_rate*creditdays, term*revol_util.
Every grey dot is one of the 255 fits. The red envelope peaks at \(k = 4\): beyond that, every extra column is charged more than it earns.
Best-4 on test: 0.130 — far above the 36-column model (0.079), with a tenth of the columns.
But the plain 8 base features score 0.141. The correlation screen kept five variants of int_rate and term and dropped install and balance — weak marginally, useful jointly (the lesson of the x1*x2 slide, in reverse). Screening is itself a modelling choice.
Prediction asks “how close?”. Inference asks “is this coefficient real?” — and that question needs assumptions the prediction never did.
Two hard diagonal edges: EAD lives in \([0, 1]\), so residual \(= y - \hat y\) cannot exceed \(1 - \hat y\) or fall below \(-\hat y\). The spread is not constant and the errors are not normal.
Assumptions matter for inference, not for prediction
R², adjusted R², RMSE and the predictions are computed from \(y\) and \(\hat y\) alone — no distributional assumption. t-tests, p-values and confidence intervals need independent, constant-variance errors (and normality or a large \(n\)): violate them and the numbers still print, but their coverage is wrong. Validate assumptions when you are about to call a coefficient significant; skip it when you only need a forecast.
A coefficient has |t| = 0.8. The right reading is:
19 of 25 coefficients have |t| < 2. term (7.5), balance (4.9) and int_rate (3.5) carry the model — exactly the variables the best-subset search kept reaching for. Small |t| or \(p > 0.05\) → redundant in this model; large |t| with a bad residual plot → trust the direction, not the exact p-value.
\[\text{VIF}_j = \frac{1}{1 - R^2_j}, \quad R^2_j = R^2 \text{ of regressing } x_j \text{ on the other predictors}\]
We kept all three of home_OWN, home_RENT, home_MORTGAGE. Their VIFs will be:
home_RENT 290.7, home_MORTGAGE 286.3, home_OWN 97.0 — the dummy trap from §4.1. Grades B–D sit at 19–31 because int_rate is set from the grade. Follow the notebook: drop the three home dummies and grades A–D.
Max VIF falls from 291 to 10.8 (dti and install share the income denominator). Seven fewer columns, and test adjusted R² rises from 0.107 to 0.121. Collinearity hurts inference badly and prediction a little.
Build keep = the features of m_ead with |t| ≥ 2, refit, and print how many survived and the adjusted R².
Six features (int_rate, term, revol_util, install, balance, creditdays), adjusted R² 0.178 — above the 25-feature model’s 0.174. Nineteen columns were pure cost.
x1*x2); interactions belong in the candidate pool.Next: §4.3 — let a tree find the interactions for you.
A regression tree finds thresholds and interactions on its own — and memorises noise just as readily. You will watch one split, one overfit, then tame it by averaging (random forest) and by correcting residuals (boosting), and turn boosting on ten years of the S&P 500 with a walk-forward split. Then the missing factor: PD by logistic regression, judged by ROC and a cutoff that reflects gain and loss.
Eight numeric features, the same seed and split as before: 1 146 / 458 defaulted loans.
Which pattern can a linear regression not capture without you engineering a feature?
A regression tree partitions the feature space into boxes and predicts the mean of \(y\) in each box — a piecewise-constant approximation with thresholds and interactions built in.
It tries every cut-point of every feature and keeps the one that most reduces the sum of squared errors: \(\text{SSE}_{\text{left}} + \text{SSE}_{\text{right}}\) versus the parent’s SSE.
On the 1 146 training loans, EAD has standard deviation ≈ 0.227. Roughly what is the parent SSE, \(\sum (y - \bar y)^2\)?
59.1
Cutting int_rate at 17.93 % drops SSE from 59.11 to 54.44. The tree does this for every feature and takes the best; then repeats inside each child.
The root splits on term (36 vs 60 months), then each side splits on balance. Four leaves, four means from 0.57 to 0.86 — the tree found the term×balance interaction that §4.2 had to construct by hand, and its test RMSE (0.191) already matches OLS.
DecisionTreeRegressor() with default settings on 1 146 rows. Training RMSE will be:
1 145 leaves for 1 146 loans, train 0.0, test 0.279 — 46 % worse than the two-level tree’s 0.191. This is the overfitting gap in its purest form.
| Parameter | Effect | Typical |
|---|---|---|
max_depth |
caps the number of successive splits | 2–7 |
min_samples_leaf |
no leaf smaller than this | 5–50 |
min_samples_split |
no split attempted on a smaller node | 10–20 |
max_leaf_nodes |
direct cap on complexity | 4–32 |
Test RMSE bottoms out around depth 2–3 and climbs from depth 5 as training RMSE keeps falling. Simple tree = high bias, low variance; deep tree = the reverse. Pick the depth on validation data, never on training.
Change min_samples_leaf so the fully grown tree’s test RMSE falls below 0.196.
A leaf size of 50 gives 0.195 with 19 leaves; 20 (0.197, 44 leaves) is not enough. The leaf size is a variance dial.
Fit 100 deep trees on 100 bootstrap samples and average them. What does that reduce?
Forest test RMSE 0.191 — identical to OLS on these 8 features, with a lower training error (0.185 vs 0.205). With 1 146 rows the nonlinearity buys little; on the 150 000-loan notebook the forest lifted test adjusted R² from 0.163 to 0.216.
term, balance, int_rate dominate — the same three the t-tests and the best-subset search flagged in §4.2.
Read it as a description of the forest, not of the world
Importance ranks the forest’s habits. For “does this variable matter?”, use permutation importance on test data, or a t-test in a model you can interpret.
In gradient boosting with squared-error loss, tree \(m\) is fitted to:
Five one-split stumps, learning rate 0.5: test RMSE 0.200 → 0.189 — already at the forest’s level. \(f_M(x) = \bar y + \nu \sum_m T_m(x)\); each stump corrects what the previous ones left behind.
GradientBoostingRegressor with 100 small treesBoosting test 0.192, forest 0.191, OLS 0.191 — three models within a thousandth on a weak signal. The knobs: n_estimators (more trees = more capacity), learning_rate (smaller steps need more trees, generalise better), max_depth 1–3 (weak learners), subsample < 1 (row sampling per tree, adds variance reduction). Tune them on validation data.
The notebook’s last example turns boosting on prices. The target is “does the S&P 500 rise tomorrow?”, the features are yesterday’s information — and the split must respect time.
Why is train_test_split(shuffle=True) wrong for a price-direction model?
2 495 trading days, 2015–2024; 53.8 % are up-days. Seven features, all lagged by at least one day. TimeSeriesSplit gives an expanding window: each fold trains on everything before its test block.
Train AUC vs test AUC across the four folds will look like:
Test AUC 0.52 on average (0.49–0.56) against train AUC of 0.71–0.83 — highest on the smallest training block, where memorising is easiest. The notebook’s 140-feature LightGBM on 5-minute bars lands at 0.538 ± 0.031 — same picture. Daily direction is nearly unforecastable from its own past.
On the 821 out-of-sample days the model was confident, the S&P averaged 0.089 % a day against 0.054 % overall; on its 296 bearish days, −0.027 %. The notebook’s version: 0.077 % vs 0.023 %. A tiny edge, before costs — and the importance ranking (lag4 first) is the unstable kind §4.3 warned about.
In Colab
Pyodide has no lightgbm; the notebook runs the walk-forward above on 125 MB of 5-minute bars (140 features, weekly folds) with it. Same gradient-boosting idea, engineered for speed and regularisation:
from lightgbm import LGBMClassifier, early_stopping
model = LGBMClassifier(n_estimators=5000, learning_rate=0.02, num_leaves=63,
subsample=0.8, colsample_bytree=0.8,
reg_alpha=1.0, reg_lambda=2.0, random_state=42, verbose=-1)
model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], eval_metric="auc",
callbacks=[early_stopping(200, verbose=False)])num_leaves = 63 ≈ depth 6 per tree; subsample / colsample_bytree = row and feature sampling (the forest’s tricks inside boosting); reg_alpha / reg_lambda = L1 / L2 penalties on leaf values; early_stopping picks n_estimators on the validation fold.TimeSeriesSplit — train on the past, test on the future, never shuffle.The third factor. Default is 0/1, so the model must output a probability — and a probability is not a decision until you choose a cutoff.
What goes wrong with OLS on a binary target?
\[\log\frac{p}{1-p} = \beta_0 + \beta^\top x \quad\Longleftrightarrow\quad p = \frac{1}{1 + e^{-(\beta_0 + \beta^\top x)}}\]
Fitted on all 5 595 training loans, not just the defaulted ones. int_rate (z = 9.8) and term (5.7) raise the log-odds; fico (−4.8) lowers them. Mean predicted PD equals the default rate, 0.205 — logistic regression is calibrated on average by construction.
Predicted PDs average 0.205 and rarely exceed 0.5. With cutoff 0.5, roughly what share of the 458 real defaulters in the test set are flagged (the TPR)?
about 0.07
Cutoff 0.5 catches 7 % of defaulters and wrongly flags 1.5 % of good loans. Cutoff 0.2 catches 63 % but flags 35 % of good loans. TPR = P(flag | default), FPR = P(flag | repaid): every cutoff is a trade between them.
AUC = 0.702 on test (0.692 on train — no overfitting in a 9-parameter logit): a random defaulter is ranked riskier than a random good loan 70 % of the time. The notebook’s 150 000-loan logit reaches a similar 0.70 — origination data simply do not separate the classes more than this.
Approve a loan when it is not flagged. Per applicant, expected profit is \[\Pi(c) = g\,(1 - \text{FPR}(c)) - \ell\,(1 - \text{TPR}(c)),\] with \(g\) = gain from a good loan × P(good), \(\ell\) = loss on a bad loan × P(bad). Setting \(d\Pi/dc = 0\): \[\frac{d\,\text{TPR}}{d\,\text{FPR}} = \frac{g}{\ell} \quad\text{— the ROC slope at the optimum equals gain over loss.}\]
If the loss per bad loan rises relative to the gain per good loan, the optimal cutoff:
Gain on a good loan ≈ 10 % interest; loss on a bad one ≈ EAD × LGD ≈ 0.62 of principal (§4.1). Weight by the base rates.
The slope target is 0.68, so the optimum sits where the ROC has flattened: cutoff ≈ 0.17, flagging 78 % of defaulters at the price of 50 % of good applicants. Approving everyone loses 3.8 cents per applicant; at the optimum the book earns 1.4 cents — and 0.5 was never a candidate.
Boosting: train 0.745, test 0.703; logit: 0.692 / 0.702. The flexible model gains 5 points on training data and one tenth of a point on test. When the signal is weak, a flexible model mostly finds more noise.
int_rate at 17.93 %), and finds interactions (term then balance) that OLS needs hand-built.TimeSeriesSplit, never a shuffle. Boosting on ten years of the S&P 500: train AUC 0.7, test 0.52 — memorised noise, a sliver of edge.Next: §4.4 — a model that predicts well can still be the wrong lever to pull.
Everything so far answered “given \(x\), what is \(y\)?”. A manager asks “if I change \(x\), what happens to \(y\)?”. You will build a case where the best predictor is a useless lever, fix it with a control, and then meet two designs for when the control is not observed.
“Borrowers who use our budgeting tool repay 20 % more.” Which action needs a causal answer?
Potential outcomes: each borrower has \(Y(1)\) with the tool and \(Y(0)\) without; we see one. The observed gap is \(\underbrace{E[Y(1)-Y(0)\mid T=1]}_{\text{causal}} + \underbrace{E[Y(0)\mid T=1] - E[Y(0)\mid T=0]}_{\text{selection bias}}\).
Simulate a hidden trait \(U\) (financial discipline) that drives both tool adoption \(T\) and repayment \(Y\). By construction the tool has zero effect.
Coefficient 2.17 with t = 24 and R² = 0.23. As a predictor of repayment, tool usage is excellent. As a policy, pushing the tool would move repayment by exactly 0.
Add \(U\) as a second regressor. The coefficient on \(T\) will:
Adjusted: 0.066 (truth 0). With a real effect of 0.5, naive says 2.67, adjusted says 0.45. This works only because \(U\) was measured. Draw the graph \(T \leftarrow U \rightarrow Y\) first; include confounders, never mediators or colliders.
U (unobserved)
/ \
v v
T -> YTreated group gets a policy between period 0 and 1; the control group does not. \[\hat\tau_{\text{DiD}} = (\bar Y^{\text{tr}}_{1} - \bar Y^{\text{tr}}_{0}) - (\bar Y^{\text{ctl}}_{1} - \bar Y^{\text{ctl}}_{0})\]
The identifying assumption of DiD is:
Treated group starts 1.0 higher, everyone drifts up 0.5, and the policy adds 0.8 to the treated group in period 1.
Both naive numbers are wrong: the post gap (1.85) includes the pre-existing level difference; the before-after (1.31) includes the common drift.
did currently holds the treated group’s before-after change. Subtract the control group’s change so that did is the difference-in-differences.
The regression \(y = \alpha + \beta\,\text{group} + \gamma\,\text{period} + \delta\,(\text{group} \times \text{period})\) gives \(\hat\delta\) = 0.861 (se 0.051) — identical to the hand calculation, now with a standard error. Parallel trends cannot be tested after treatment; with more pre-periods, check that the two groups moved in parallel before it.
An instrument \(Z\) moves \(T\) but touches \(Y\) only through \(T\). Three conditions: relevance (\(Z \to T\)), exclusion (no \(Z \to Y\) path except via \(T\)), independence (\(Z \perp U\)).
Which condition can never be verified from the data alone?
Classic instruments: distance to college for schooling (Card 1995), draft-lottery number for military service (Angrist 1990), rainfall for agricultural income. Two-stage least squares: regress \(T\) on \(Z\), then \(Y\) on \(\hat T\).
Hidden \(U\) drives \(T\) and \(Y\); true effect of \(T\) on \(Y\) is 1.0; \(Z\) is a clean shock to \(T\).
OLS says 2.013 — twice the truth, because \(U\) pushes \(T\) and \(Y\) the same way. 2SLS uses only the part of \(T\) that \(Z\) explains and lands on 1.027. The one-instrument case reduces to the ratio \(\text{cov}(Z,Y)/\text{cov}(Z,T)\).
Strong instrument: F = 637. Weak one: F = 7.0 and the 2SLS estimate drifts to 0.72 with a huge standard error. Rule of thumb: first-stage F above 10, or do not trust the second stage.
| Design | Replaces the missing control with | Fails when |
|---|---|---|
| Regression adjustment | the measured confounder | a confounder is unmeasured |
| Difference-in-differences | a control group’s time trend | trends are not parallel |
| Instrumental variables | an exogenous shock to \(T\) | \(Z\) is weak or has its own path to \(Y\) |
Every month a fund sees the same object: a matrix of stocks by characteristics, and next month’s returns as the target. You will standardise it without peeking at the future, replace the sector as “peer group” with one attention layer written in numpy, and put both through the walk-forward protocol of §4.3 — and discover that the protocol, not the architecture, is the lesson.
At month-end \(t\), \(X_t \in \mathbb{R}^{N \times K}\) holds \(N\) stocks × \(K\) characteristics and \(y_t \in \mathbb{R}^N\) holds next month’s returns. The prediction problem is cross-sectional: rank the \(N\) stocks, month by month.
25 335 rows = 119 months × 213 stocks in 11 sectors, 2015-01 … 2024-11. Five price-based characteristics: last-month return, 12-1 momentum, 21-day realised volatility, log dollar volume, price / 252-day high. ret_next at month \(t\) is exactly ret_1_0 at month \(t+1\) (0.100776 = 0.100776) — the target is the next row’s return.
The universe is today’s large caps, tracked back to 2015. Nothing that was large in 2015 and has since failed, shrunk or been delisted is in the file.
What does this survivorship bias do to a backtest run on the file?
Production rule: build the universe as of each month from a point-in-time index membership file. This file is fine for learning the mechanics — never quote its Sharpe ratio as evidence.
Why is (rvol_21d − rvol_21d.mean()) / rvol_21d.std() over the whole 2015–2024 file a look-ahead?
Median and std are computed within each month (and, for the _s columns, within each month × sector — the production invariant: a stock is judged against the peers it is compared with on the day). Sector medians are exactly 0 afterwards; values are clipped at ±3 so one meme month cannot dominate a tree split.
“Sector” is a label assigned by an index provider. In October 2022 NVDA’s behavioural peers — high volume, high volatility, far below the 52-week high — included AMD, but also Netflix, Alibaba and Tesla, none of them “Technology”.
Attention builds the peer group from the data. With queries \(Q = XW_q\), keys \(K = XW_k\), values \(V = XW_v\):
\[A = \text{softmax}_{\text{rows}}\!\left(\frac{QK^\top}{\sqrt{d}\,\tau}\right), \qquad C = AV\]
With \(W_q = W_k = I\), the weight stock \(i\) gives stock \(j\) is proportional to
NVDA’s attention row in October 2022: AMD 0.353, itself 0.263, NFLX 0.154, BABA 0.077, TSLA 0.055, AMZN 0.028 — six stocks carry more than 1 %, from three sectors. The context \(C\) is the peer-weighted characteristic vector: 12-1 momentum −1.49 for NVDA against −1.26 for its peers, dollar volume 3.0 against 2.63.
With identity weights and \(\tau \to \infty\) every score goes to 0. What does NVDA’s context vector collapse to?
the cross-sectional mean of X — every weight is 1/N
Temperature is the dial between two crude peer groups: \(\tau \to 0\) attends only to the most similar stock (at \(\tau = 0.05\) AMD carries 0.997 — NVDA is not even its own nearest point), \(\tau \to \infty\) gives the cross-sectional mean (0.02, 0.10, 0.19, 0.00, −0.14), which plain per-month standardisation already removes. \(\tau = 1\) sits in between: a handful of behavioural peers.
“Artificial Intelligence Asset Pricing Models” (KKMX, 2025) puts a transformer over the stock cross-section, not over words:
What we do instead
Weights stay fixed (\(W = I\); a seeded random \(W\) works the same way and is the random-features view). The attention output is used only as feature engineering: for every stock, the context \(C\) (peer-weighted characteristics) and the deviation \(X - C\) (“how I differ from my attention-weighted peers”). The learner on top is the boosting machine from §4.3.
Fifteen columns per stock-month: 5 market-relative characteristics, 5 peer contexts, 5 deviations. NVDA in October 2022: 12-1 momentum −1.49 against −1.38 for its peers, price / 52-week high −2.67 against −2.59 — deviations of −0.10 and −0.08. With a self-weight of 0.26 and AMD at 0.35, NVDA largely is its peer group that month. Only month-\(t\) rows enter month \(t\)’s attention: no look-ahead.
The row dated 2018-12-31 has ret_next = January 2019’s return. May it sit in the training set when we predict the 2019-01-31 row?
Expanding window from 2019-01: 71 test months, 6 refits (every 12 months, predicting the next 12 — a monthly refit would be 71 fits, same protocol, ten times the runtime). Two scores per month: rank IC (Spearman between prediction and realised return across the 213 stocks) and the top-20 minus bottom-20 equal-weight spread.
Which feature set will have the highest mean rank IC — and will the difference be significant?
Mean rank IC: −0.015 (t = −1.04) raw, −0.014 (t = −0.99) with sector features, −0.010 (t = −0.67) with attention. All three are zero within noise (standard error ≈ 0.015). The long-short spread is positive (+7.7 %, +9.3 %, +2.8 % a year; Sharpe 0.47, 0.62, 0.17, before costs) — the extremes rank slightly better than the middle, but 71 months cannot separate a Sharpe of 0.6 from 0. Attention features did not help here; the protocol that tells you so is the deliverable.
The t-statistic of a spread with Sharpe \(S\) over \(T\) months is \(S\sqrt{T/12}\): with \(T = 71\), the best set’s Sharpe of 0.62 gives t = 1.51; the raw set’s 0.47 gives 1.15; attention’s 0.17 gives 0.40. To reach t = 2 the sector set would need 124 months of the same performance (10 years), the raw set 18 years. Every gain in the plot comes with an equally plausible zero — and the file is survivorship-biased in the spread’s favour.
Change the temperature (tau_new = 3.0) or build the keys from a subset of characteristics (keys_new = ["ret_12_1_z", "rvol_21d_z"]) so that ic_new differs from the baseline attention IC. Does the peer-group definition move the IC by more than its standard error (about 0.015)?
\(\tau = 3\): IC −0.010 (unchanged); keys from momentum and volatility only: −0.025 (t = −1.64); \(\tau = 0.3\): −0.041 (t = −2.60); \(\tau = 10\): −0.008. The sign of the change flips with the setting, and the one “significant” number appears after four tries — that is multiple testing, not a signal. In KKMX the peer group is learned on a Sharpe objective with thousands of stocks and six decades; here it is a fixed kernel on 213 survivors.
Warning
Zillow Offers used the company’s Zestimate models to buy homes directly, renovate and resell them. The models were good predictors of sale prices on the market as observed. But once Zillow acted on them — bidding at the model’s price — the data-generating process changed: sellers accepted the offers the model had overpriced and walked away from the ones it had underpriced (adverse selection), and a 2021 shift in the housing market moved prices faster than the model updated.
In Q3 2021 Zillow wrote down its home inventory by $304 million, disclosed roughly 7 000 homes still to be sold, and on 2 November 2021 shut the business and cut about 25 % of its workforce.
Lesson for this chapter: a model tuned on test RMSE answers “how close is \(\hat y\) to \(y\) under the world that produced the training data?”. The moment your predictions become actions — offers, prices, approvals — you are asking a causal question, and the selection that follows your action is a confounder the test set never contained.
To: Head of Credit, consumer lending From: <Your name>, credit analytics Subject: Move the PD approval cutoff from 0.50 to 0.17 Date: 2026-09-15
Recommendation: approve applicants with predicted PD below 0.17; refer the rest for manual review.
Evidence: - Logistic PD model, 8 origination features, test AUC 0.70 (8 000-loan sample, 30 % held out). - Expected loss per bad loan ≈ EAD × LGD = 0.70 × 0.89 = 0.62 of principal; expected gain per good loan ≈ 0.10. - Profit-maximising cutoff (ROC slope = gain/loss) is 0.17: catches 78 % of defaulters at the cost of 50 % of good applicants, turning −3.8 cents per applicant into +1.4. At 0.50 the model flags only 7 % of defaulters.
Caveats: - LGD is effectively unpredictable (test adjusted R² < 0); the 0.89 mean is the model. - The cutoff is a prediction, not a causal estimate: tightening approvals changes the applicant pool (Zillow lesson). Pilot on a random 10 % of applications for one quarter.
Next step: A/B pilot; re-estimate AUC and the profit curve on the pilot’s realised outcomes.
| Concept | Tool |
|---|---|
| Default, EAD, LGD | map, ledger arithmetic on funded_amnt, total_rec_prncp, recoveries |
| Feature engineering | str.extract, pd.to_datetime(format=), fillna(360), pd.get_dummies(...).iloc[:, :-1] |
| Multiple regression | sm.OLS(y, sm.add_constant(X)).fit(), .rsquared_adj, test RMSE |
| Interactions, best subset | itertools.combinations, adjusted R² envelope over \(k\) |
| Inference | .tvalues ( |
| Trees and ensembles | DecisionTreeRegressor(max_depth, min_samples_leaf), RandomForestRegressor, GradientBoostingRegressor |
| Time-series prediction | lagged features, TimeSeriesSplit walk-forward, per-fold AUC |
| Classification | sm.Logit, confusion table, roc_curve, roc_auc_score, cutoff at ROC slope = gain/loss |
| Causal analysis | confounder adjustment, DiD interaction term, 2SLS via np.linalg.lstsq |
| Cross-sectional attention | per-month (sector) standardisation, Q @ K.T / sqrt(d) + row softmax in numpy, context \(C\) and \(X - C\) features, expanding-window rank IC and top − bottom spread |
Next: Chapter 5 — Rethinking Statistics with Bayesian Methods.
balance and install, and state what it would cost.Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python