Chapter 4 · Statistical Predictive Models
Section 4.3 · Chapter 4 · Learning Statistics with Python
Statistical Predictive Models
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
A regression tree finds thresholds and interactions on its own — and memorises noise just as readily. You will watch one split, one overfit, then tame it by averaging (random forest) and by correcting residuals (boosting), and turn boosting on ten years of the S&P 500 with a walk-forward split. Then the missing factor: PD by logistic regression, judged by ROC and a cutoff that reflects gain and loss.
Eight numeric features, the same seed and split as before: 1 146 / 458 defaulted loans.
Which pattern can a linear regression not capture without you engineering a feature?
A regression tree partitions the feature space into boxes and predicts the mean of \(y\) in each box — a piecewise-constant approximation with thresholds and interactions built in.
It tries every cut-point of every feature and keeps the one that most reduces the sum of squared errors: \(\text{SSE}_{\text{left}} + \text{SSE}_{\text{right}}\) versus the parent’s SSE.
On the 1 146 training loans, EAD has standard deviation ≈ 0.227. Roughly what is the parent SSE, \(\sum (y - \bar y)^2\)?
59.1
Cutting int_rate at 17.93 % drops SSE from 59.11 to 54.44. The tree does this for every feature and takes the best; then repeats inside each child.
The root splits on term (36 vs 60 months), then each side splits on balance. Four leaves, four means from 0.57 to 0.86 — the tree found the term×balance interaction that §4.2 had to construct by hand, and its test RMSE (0.191) already matches OLS.
DecisionTreeRegressor() with default settings on 1 146 rows. Training RMSE will be:
1 145 leaves for 1 146 loans, train 0.0, test 0.279 — 46 % worse than the two-level tree’s 0.191. This is the overfitting gap in its purest form.
| Parameter | Effect | Typical |
|---|---|---|
max_depth |
caps the number of successive splits | 2–7 |
min_samples_leaf |
no leaf smaller than this | 5–50 |
min_samples_split |
no split attempted on a smaller node | 10–20 |
max_leaf_nodes |
direct cap on complexity | 4–32 |
Test RMSE bottoms out around depth 2–3 and climbs from depth 5 as training RMSE keeps falling. Simple tree = high bias, low variance; deep tree = the reverse. Pick the depth on validation data, never on training.
Change min_samples_leaf so the fully grown tree’s test RMSE falls below 0.196.
A leaf size of 50 gives 0.195 with 19 leaves; 20 (0.197, 44 leaves) is not enough. The leaf size is a variance dial.
Fit 100 deep trees on 100 bootstrap samples and average them. What does that reduce?
Forest test RMSE 0.191 — identical to OLS on these 8 features, with a lower training error (0.185 vs 0.205). With 1 146 rows the nonlinearity buys little; on the 150 000-loan notebook the forest lifted test adjusted R² from 0.163 to 0.216.
term, balance, int_rate dominate — the same three the t-tests and the best-subset search flagged in §4.2.
Read it as a description of the forest, not of the world
Importance ranks the forest’s habits. For “does this variable matter?”, use permutation importance on test data, or a t-test in a model you can interpret.
In gradient boosting with squared-error loss, tree \(m\) is fitted to:
Five one-split stumps, learning rate 0.5: test RMSE 0.200 → 0.189 — already at the forest’s level. \(f_M(x) = \bar y + \nu \sum_m T_m(x)\); each stump corrects what the previous ones left behind.
GradientBoostingRegressor with 100 small treesBoosting test 0.192, forest 0.191, OLS 0.191 — three models within a thousandth on a weak signal. The knobs: n_estimators (more trees = more capacity), learning_rate (smaller steps need more trees, generalise better), max_depth 1–3 (weak learners), subsample < 1 (row sampling per tree, adds variance reduction). Tune them on validation data.
The notebook’s last example turns boosting on prices. The target is “does the S&P 500 rise tomorrow?”, the features are yesterday’s information — and the split must respect time.
Why is train_test_split(shuffle=True) wrong for a price-direction model?
2 495 trading days, 2015–2024; 53.8 % are up-days. Seven features, all lagged by at least one day. TimeSeriesSplit gives an expanding window: each fold trains on everything before its test block.
Train AUC vs test AUC across the four folds will look like:
Test AUC 0.52 on average (0.49–0.56) against train AUC of 0.71–0.83 — highest on the smallest training block, where memorising is easiest. The notebook’s 140-feature LightGBM on 5-minute bars lands at 0.538 ± 0.031 — same picture. Daily direction is nearly unforecastable from its own past.
On the 821 out-of-sample days the model was confident, the S&P averaged 0.089 % a day against 0.054 % overall; on its 296 bearish days, −0.027 %. The notebook’s version: 0.077 % vs 0.023 %. A tiny edge, before costs — and the importance ranking (lag4 first) is the unstable kind §4.3 warned about.
In Colab
Pyodide has no lightgbm; the notebook runs the walk-forward above on 125 MB of 5-minute bars (140 features, weekly folds) with it. Same gradient-boosting idea, engineered for speed and regularisation:
from lightgbm import LGBMClassifier, early_stopping
model = LGBMClassifier(n_estimators=5000, learning_rate=0.02, num_leaves=63,
subsample=0.8, colsample_bytree=0.8,
reg_alpha=1.0, reg_lambda=2.0, random_state=42, verbose=-1)
model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], eval_metric="auc",
callbacks=[early_stopping(200, verbose=False)])num_leaves = 63 ≈ depth 6 per tree; subsample / colsample_bytree = row and feature sampling (the forest’s tricks inside boosting); reg_alpha / reg_lambda = L1 / L2 penalties on leaf values; early_stopping picks n_estimators on the validation fold.TimeSeriesSplit — train on the past, test on the future, never shuffle.The third factor. Default is 0/1, so the model must output a probability — and a probability is not a decision until you choose a cutoff.
What goes wrong with OLS on a binary target?
\[\log\frac{p}{1-p} = \beta_0 + \beta^\top x \quad\Longleftrightarrow\quad p = \frac{1}{1 + e^{-(\beta_0 + \beta^\top x)}}\]
Fitted on all 5 595 training loans, not just the defaulted ones. int_rate (z = 9.8) and term (5.7) raise the log-odds; fico (−4.8) lowers them. Mean predicted PD equals the default rate, 0.205 — logistic regression is calibrated on average by construction.
Predicted PDs average 0.205 and rarely exceed 0.5. With cutoff 0.5, roughly what share of the 458 real defaulters in the test set are flagged (the TPR)?
about 0.07
Cutoff 0.5 catches 7 % of defaulters and wrongly flags 1.5 % of good loans. Cutoff 0.2 catches 63 % but flags 35 % of good loans. TPR = P(flag | default), FPR = P(flag | repaid): every cutoff is a trade between them.
AUC = 0.702 on test (0.692 on train — no overfitting in a 9-parameter logit): a random defaulter is ranked riskier than a random good loan 70 % of the time. The notebook’s 150 000-loan logit reaches a similar 0.70 — origination data simply do not separate the classes more than this.
Approve a loan when it is not flagged. Per applicant, expected profit is \[\Pi(c) = g\,(1 - \text{FPR}(c)) - \ell\,(1 - \text{TPR}(c)),\] with \(g\) = gain from a good loan × P(good), \(\ell\) = loss on a bad loan × P(bad). Setting \(d\Pi/dc = 0\): \[\frac{d\,\text{TPR}}{d\,\text{FPR}} = \frac{g}{\ell} \quad\text{— the ROC slope at the optimum equals gain over loss.}\]
If the loss per bad loan rises relative to the gain per good loan, the optimal cutoff:
Gain on a good loan ≈ 10 % interest; loss on a bad one ≈ EAD × LGD ≈ 0.62 of principal (§4.1). Weight by the base rates.
The slope target is 0.68, so the optimum sits where the ROC has flattened: cutoff ≈ 0.17, flagging 78 % of defaulters at the price of 50 % of good applicants. Approving everyone loses 3.8 cents per applicant; at the optimum the book earns 1.4 cents — and 0.5 was never a candidate.
Boosting: train 0.745, test 0.703; logit: 0.692 / 0.702. The flexible model gains 5 points on training data and one tenth of a point on test. When the signal is weak, a flexible model mostly finds more noise.
int_rate at 17.93 %), and finds interactions (term then balance) that OLS needs hand-built.TimeSeriesSplit, never a shuffle. Boosting on ten years of the S&P 500: train AUC 0.7, test 0.52 — memorised noise, a sliver of edge.Next: §4.4 — a model that predicts well can still be the wrong lever to pull.
Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python