Chapter 4 · Statistical Predictive Models
Section 4.2 · Chapter 4 · Learning Statistics with Python
Statistical Predictive Models
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
More variables always raise R². You will see why adjusted R² is the honest score, create interaction terms that reveal hidden structure, search all 255 subsets of 8 candidates, and then read t-tests and VIF to decide which coefficients mean anything.
Same seed 5650, same 1 146 / 458 defaulted rows, same adjusted R² 0.174 as §4.1.
We append 10 columns of pure noise (seed 1) to the 25 features and refit.
After adding 10 noise columns, R² and adjusted R² will:
R² 0.1920 → 0.1959; adjusted R² 0.1740 → 0.1706. Ten columns that cannot possibly matter “explained” 0.4 % more variance. Never select on raw R².
y = x1 * x2 + noise with independent standard-normal x1, x2. What are the three correlations corr(y,x1), corr(y,x2), corr(y,x1·x2)?
−0.027, −0.081, 0.887. A term-by-rate interaction in the loan data could hide the same way — so we add products to the candidate pool.
Eight base features and all 28 pairwise products: 36 candidates. (Squares of a two-valued term would duplicate term, so we skip squares.)
The notebook does this for all 25 features plus ratios and reaches 650 columns. Best-subset over 650 columns means \(2^{650}\) models — the reason the next step is a screen, not a search.
Compared with the plain 8-feature model, the 36-column model will have:
36 columns: train 0.195, test 0.079. 8 columns: train 0.178, test 0.141. The interactions bought 1.7 points on training data and cost 6 on test.
Best-subset selection fits every non-empty subset of \(k\) candidates. For \(k = 8\), how many fits?
255
For 36 candidates it would be \(2^{36} - 1 \approx 6.9 \times 10^{10}\). So we first screen the pool by absolute correlation with EAD and keep the top 8 — the notebook keeps the top 15 of 650.
Adjusted R² climbs 0.143 → 0.152 → 0.158 → 0.161 at \(k=4\) and then falls as more candidates are added. The winner: int_rate, term*fico, int_rate*creditdays, term*revol_util.
Every grey dot is one of the 255 fits. The red envelope peaks at \(k = 4\): beyond that, every extra column is charged more than it earns.
Best-4 on test: 0.130 — far above the 36-column model (0.079), with a tenth of the columns.
But the plain 8 base features score 0.141. The correlation screen kept five variants of int_rate and term and dropped install and balance — weak marginally, useful jointly (the lesson of the x1*x2 slide, in reverse). Screening is itself a modelling choice.
Prediction asks “how close?”. Inference asks “is this coefficient real?” — and that question needs assumptions the prediction never did.
Two hard diagonal edges: EAD lives in \([0, 1]\), so residual \(= y - \hat y\) cannot exceed \(1 - \hat y\) or fall below \(-\hat y\). The spread is not constant and the errors are not normal.
Assumptions matter for inference, not for prediction
R², adjusted R², RMSE and the predictions are computed from \(y\) and \(\hat y\) alone — no distributional assumption. t-tests, p-values and confidence intervals need independent, constant-variance errors (and normality or a large \(n\)): violate them and the numbers still print, but their coverage is wrong. Validate assumptions when you are about to call a coefficient significant; skip it when you only need a forecast.
A coefficient has |t| = 0.8. The right reading is:
19 of 25 coefficients have |t| < 2. term (7.5), balance (4.9) and int_rate (3.5) carry the model — exactly the variables the best-subset search kept reaching for. Small |t| or \(p > 0.05\) → redundant in this model; large |t| with a bad residual plot → trust the direction, not the exact p-value.
\[\text{VIF}_j = \frac{1}{1 - R^2_j}, \quad R^2_j = R^2 \text{ of regressing } x_j \text{ on the other predictors}\]
We kept all three of home_OWN, home_RENT, home_MORTGAGE. Their VIFs will be:
home_RENT 290.7, home_MORTGAGE 286.3, home_OWN 97.0 — the dummy trap from §4.1. Grades B–D sit at 19–31 because int_rate is set from the grade. Follow the notebook: drop the three home dummies and grades A–D.
Max VIF falls from 291 to 10.8 (dti and install share the income denominator). Seven fewer columns, and test adjusted R² rises from 0.107 to 0.121. Collinearity hurts inference badly and prediction a little.
Build keep = the features of m_ead with |t| ≥ 2, refit, and print how many survived and the adjusted R².
Six features (int_rate, term, revol_util, install, balance, creditdays), adjusted R² 0.178 — above the 25-feature model’s 0.174. Nineteen columns were pure cost.
x1*x2); interactions belong in the candidate pool.Next: §4.3 — let a tree find the interactions for you.
Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python