Predict business outcomes, choose losses and evaluate decisions across delivery, retail, customer retention, fraud and finance.
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
Read each section’s concept and formula before running Python. Predict the result, run an example, modify one input in that cell, then explain what changed and why. Each section deck includes complete cells.
The slides are the complete self-study route. The Book expands intuition, derivations and failure modes. Classroom time selects discussions and experiments from this material, rather than limiting its coverage.
Download the worked Chapter 4 notebook. Bring one coefficient interpretation, one model-comparison question and one data/risk limitation. The case demonstrates statistical methods and can transfer to forecasting and adverse-event decisions outside finance.
Follow the numbered parts in order within each section. Each part states a question and a result. Read the concept, predict the example, run it, and explain the output before attempting the check.
Classroom discussion selects important examples and experiments. The complete Slides and Book support self-study.
Worked examples run automatically when the page opens or refreshes. Each cell includes required earlier imports, data and definitions under Previously introduced setup. Run Code includes that setup; Copy complete code copies the whole cell. Practice and repair cells run when you choose. Make input changes in the current cell before the operation you are testing.
| Section | Statistical question | Evidence |
|---|---|---|
| §4.1 | What does a multiple regression estimate? | Coefficients, residuals and forecast errors |
| §4.2 | Which specification and uncertainty claims are defensible? | Inference, validation, regularisation and VIF |
| §4.3 | How do nonlinear models and event probabilities work? | Trees, ensembles, ROC, Brier and thresholds |
| §4.4 | What would change under an intervention? | Confounding, DiD and IV assumptions |
| §4.5 | How can cross-sectional context form features? | Fixed attention and walk-forward comparison |
| §4.6 | How do forecasts enter a risk decision? | Frozen rules, exposure, cash, costs and NAV |
| Step | Specify before fitting | Example |
|---|---|---|
| Objective | What outcome and costs matter? | Stockout versus surplus cost |
| Y | Unit, horizon, event and availability | Demand next week; no purchase for 30 days |
| Loss | What statistical quantity is estimated? | Squared: mean; absolute: median; pinball: quantile |
| Validation | What score chooses a model or rule? | RMSE, Brier, declared cost; select before test |
| Action | How does a forecast change a decision? | Order quantity, warning, investigation, offer |
| Evidence | Does the frozen process work later? | Untouched outcomes, costs, capacity and uncertainty |
Better alignment is a reason to test a model, not proof of improved performance. Compare the full policy on later data under the stated objective.
| Part | Question |
|---|---|
| 4.1.1 Targets, rows and information clocks | What can the model know and what must it predict? |
| 4.1.2 Fitting and interpreting OLS | What coefficient minimises squared residuals? |
| 4.1.3 Assumptions, baselines and forecast errors | Which claim is supported by this regression? |
Question: What can the model know and what must it predict?
Result: A sample that retains stock/date columns and separate decision, availability and target times.
Begin with the definitions and assumptions; then trace the worked example.
For the same four deliveries, does predicting minutes answer the same question as predicting a missed promise?
Invented teaching data; comparisons are illustrations, not measured deployment gains.
One row is a delivery. Minutes is a continuous Y; late means actual minutes strictly exceed the stated promise. Mean minutes, event probability and a high time quantile answer different questions. Use order-time features; actual delivery time is an outcome, not a predictor.
Regression: What is the next-month simple return \(Y_{i,t+1}\) conditional on today’s features?
Classification: What is the probability of \(Y_{i,t+1}\le-0.05\)? This event means a next-month closing-to-closing loss of at least 5%. It does not mean an intramonth 5% drawdown.
The public Chapter 2 handoff supplies nine stocks, 71 signal dates and 639 candidates. This small current-survivor sample teaches statistical reasoning, rather than validating a profitable strategy.
The same questions arise in demand forecasting, equipment failures and insurance: predict a continuous outcome or the probability of a defined adverse event.
%%{init: {"flowchart": {"nodeSpacing": 24, "rankSpacing": 20}, "themeVariables": {"fontSize": "20px"}}}%%
flowchart LR
A["aligned available<br/>features"] --> B["Time-ordered<br/>train / valid / test"]
B --> C["OLS<br/>next return"]
B --> D["Logit<br/>P(loss ≥ 5%)"]
C --> E["Evaluate fit<br/>and uncertainty"]
D --> E
E --> F["Freeze<br/>decision rule"]
F --> G["Walk forward<br/>reconcile accounts"]
Keep stock/date columns, feature availability and label maturity visible through every stage. A missing future label prevents fitting or evaluation, but does not remove the current prediction candidate.
The split prints 207 training, 99 validation, 306 test and 9 pending rows. Freeze fitting at 1 January 2021 and rule selection at 1 January 2022. December 2020 and December 2021 each contribute nine boundary rows whose labels arrive after the respective cutoff.
The table’s stored predictions are ignored. We fit new models using only the three named features and two sector dummies. Future returns, label status and prior predictions never enter X.
| Input | Meaning | Unit |
|---|---|---|
ret_12_1 |
Twelve-to-one momentum | Simple cumulative return |
rvol_21d |
Trailing realised volatility | Source volatility units; verify annualisation metadata |
momentum_sector_z |
Momentum relative to same-date sector peers | Standard deviations |
| Sector dummies | Financial Services and Technology versus Energy | 0/1 indicators |
An intercept plus every sector dummy is collinear. drop_first=True supplies a reference group. Static sectors and assumed month-end availability remain classroom limitations.
Predict: Why would fitting on every numeric column accidentally admit future information?
Question: What coefficient minimises squared residuals?
Result: Fit OLS and interpret a conditional association in its units.
Builds on Targets, rows and information clocks.
Demand is 2, 2, 2 or 10 units. Would you order the mean, median or a higher quantile when a shortage costs more?
Invented teaching data; comparisons are illustrations, not measured deployment gains.
Squared loss targets a conditional mean; absolute loss a median; pinball loss at tau a tau-quantile. Here shortage cost 4 and surplus cost 1 imply tau=4/(4+1)=0.8, and order 10 minimises this sample cost. These are illustrative historical outcomes, not a guaranteed future optimum. Sales can be capped by stock: observed sales need not equal unconstrained demand.
\[Y_i=\beta_0+\beta_1 x_{i1}+\cdots+\beta_p x_{ip}+\varepsilon_i,\qquad E[\varepsilon_i\mid X_i]=0.\]
\(\beta_j\) is the conditional change in expected Y for a one-unit increase in \(x_j\), holding the other regressors fixed within the model. Correlated features can have very different marginal and conditional associations.
The residual \(e_i=y_i-\widehat y_i\) is observed after fitting. The unobserved error \(\varepsilon_i\) belongs to the population model.
A predictive coefficient requires stability to generalise. A causal interpretation requires additional identification assumptions (§4.4).
\[\widehat\beta=\arg\min_b\sum_i(y_i-x_i^\top b)^2.\]
With full column rank, \(\widehat\beta=(X^\top X)^{-1}X^\top y\). Numerical software uses more stable linear algebra rather than requiring you to form the inverse.
Squared loss heavily penalises large errors. An extreme return can materially influence coefficients. Check leverage, residuals and sensitivity before deleting an observation.
Check: A low training sum of squares establishes fit to observed outcomes; it does not establish future prediction quality.
Training R² is 0.1766, adjusted R² 0.1561, while validation RMSE is 0.1014 in return units (10.14 percentage points).
The clustered fit has identical point estimates but different standard errors. It allows within-date correlation across stocks. Only 23 training dates supply clusters, so large-sample inference remains approximate.
The momentum coefficient is about −0.0484. A 0.10 increase in momentum corresponds to about −0.00484 predicted return, conditional on the other inputs in this fitted sample.
The Technology coefficient is about +0.0880 relative to Energy, conditional on the numeric features. This is a sample association, rather than a prescription to change a stock’s sector.
Question: Which claim is supported by this regression?
Result: Separate fit, inference and later-sample prediction against a baseline.
Builds on Fitting and interpreting OLS.
| Assumption | What it supports |
|---|---|
| Full rank | Identifiable coefficients |
| Correct conditional mean / exogeneity | Unbiased or consistent OLS under suitable sampling conditions |
| Constant error variance and uncorrelated errors | Classical OLS variance formula |
| Normal errors with fixed-X classical assumptions | Exact finite-sample t/F inference |
| Suitable temporal stability | Generalisation to future periods |
OLS can produce predictions without Normal errors. Robust covariance changes inference, rather than repairing a wrong target, leakage or an unstable conditional mean.
\[R^2=1-\frac{\sum_i e_i^2}{\sum_i(y_i-\bar y)^2},\qquad \bar R^2=1-(1-R^2)\frac{n-1}{n-p-1}.\]
These formulas assume a model with an intercept. Adding regressors cannot reduce training R². Adjusted R² penalises their number, but remains a training criterion.
\[\operatorname{RMSE}=\sqrt{n^{-1}\sum_i(y_i-\widehat y_i)^2},\qquad \operatorname{MAE}=n^{-1}\sum_i|y_i-\widehat y_i|.\]
Evaluate future predictions against a simple training-fitted baseline. Out-of-sample R² can be negative.
Compare both errors on exactly the same validation rows. A flexible model is useful only if the improvement serves the intended decision and survives future evaluation.
Modify: Restrict the numeric feature list, rerun fitting, and explain which quantities may improve mechanically and which need validation evidence.
Which statement is justified by the OLS example?
Select only the declared eligible training rows. The starter currently includes future validation/test dates and pending labels.
Which condition is needed to define an identifiable OLS coefficient vector?
Adding an irrelevant regressor to OLS with an intercept can do what?
| Part | Question |
|---|---|
| 4.2.1 Coefficient uncertainty and residual checks | What uncertainty and diagnostics accompany the fit? |
| 4.2.2 Specification, redundancy and regularisation | How do feature choices change bias and variance? |
| 4.2.3 Frozen selection and held-out evidence | When can the final test be opened? |
Question: What uncertainty and diagnostics accompany the fit?
Result: Interpret robust intervals, joint tests and residual patterns.
Begin with the definitions and assumptions; then trace the worked example.
Two forecasts make errors on different products. Which model wins if errors on high-margin items matter more?
Invented teaching data; comparisons are illustrations, not measured deployment gains.
Model B wins unit RMSE; A wins the declared margin-weighted absolute error. Margins must be available when decisions are made. This weighted score is a chosen error penalty, not automatically realised profit. Select the score and model on development data; estimate final performance on a later untouched sample.
This section runs independently. Preserve the same training and validation clocks. Model selection uses validation data; the 2022–2024 test remains unopened until the rule is fixed.
For a classical homoskedastic linear model,
\[\widehat{\operatorname{Var}}(\widehat\beta)=s^2(X^\top X)^{-1},\qquad s^2=\frac{\sum_i e_i^2}{n-p-1}.\]
A coefficient interval uses estimate ± critical value × standard error. A t statistic tests a specified coefficient value, often \(H_0:\beta_j=0\).
A confidence interval for the conditional mean differs from a prediction interval for one future return, which also includes outcome noise. Clustered standard errors address a different error structure and typically use approximate inference.
State the covariance estimator, clustering unit and number of dates. Robust standard errors do not make the coefficient causal.
Repeatedly testing coefficients after searching many subsets changes the inference problem. A nominal p-value from the selected model ignores selection unless the procedure accounts for it.
For q restrictions in nested classical OLS models,
\[F=\frac{(SSE_R-SSE_U)/q}{SSE_U/(n-p_U-1)}.\]
The null says all specified restrictions hold. A significant joint test need not imply every individual coefficient is significant, particularly with correlated regressors.
The code uses a cluster-robust Wald test with asymptotic calibration, rather than claiming the exact classical F distribution.
A curve suggests a missing nonlinear mean relationship. A widening fan suggests changing error variance. Clusters by date may reflect common shocks. None can be ruled out by a single global residual mean near zero, which OLS with an intercept enforces mechanically.
Modify: Colour the plot by year or sector. Explain which assumption the visible pattern questions.
Chapter 2 subtracted a sector mean with transform. Multiple regression can remove both sector averages and a numeric liquidity component from a supplied momentum score.
\(\text{momentum}=\text{intercept}+\text{sector terms}+\beta\,\text{log volume}+\text{residual}\).
The residual is observed momentum minus its fitted component. This cross-sectional transformation uses one date; it does not forecast a future return.
| Response | Controls |
|---|---|
| Momentum | Sector |
| Same stock rows | Log trading volume |
| Calculation | Output |
|---|---|
| Observed − fitted | Residual feature |
| Original row index | Retained |
Use nine stocks from the same monthly table. log_dvol_126d is the supplied log trading-volume measurement. These fixed sectors are classroom descriptions.
get_dummies turns sector categories into 0/1 columns. drop_first=True omits the reference sector; add_constant supplies the intercept. Here the response is the observed momentum feature.
OLS with an intercept and sector indicators gives approximately zero mean residual within each sector. It also makes residuals orthogonal to the included numeric control in this fitted sample.
These identities verify the fit. They do not demonstrate useful forecasts, causal effects or a sector-neutral portfolio. Repeating the transformation on another date requires a separate same-date fit.
Question: How do feature choices change bias and variance?
Result: Compare interactions, subsets, VIF and penalties within development data.
Builds on Coefficient uncertainty and residual checks.
\[E[Y\mid X]=\beta_0+\beta_1m+\beta_2v+\beta_3mv+\beta_4m^2.\]
The momentum slope becomes \(\beta_1+\beta_3v+2\beta_4m\). With an interaction, its main coefficient is the slope at the other interacting variable’s zero value.
Feature construction uses information available at each row. Select transformations using training/validation, then freeze them before test evaluation.
Five candidate regressors produce \(2^5-1=31\) non-empty subsets. Searching them can overfit the validation set even though each OLS fit is simple.
The helper below iterates over model specifications, rather than processing data rows. It records every attempted subset. Adjusted R² is a useful diagnostic, but we select by the declared validation RMSE.
Regress feature j on the remaining regressors to obtain \(R_j^2\):
\[VIF_j=\frac{1}{1-R_j^2}.\]
Near-collinearity makes separate coefficients unstable. Exact collinearity prevents unique identification. Prediction may remain stable within the observed feature region even while coefficients change sharply.
A VIF cutoff is a diagnostic convention, rather than a theorem that a feature should be deleted. Consider interpretation, redundancy, sample size and validation.
Ridge minimises \(SSE+\lambda\sum_j\beta_j^2\). Lasso uses \(SSE+\lambda\sum_j|\beta_j|\) and can set some coefficients to zero. The intercept is normally unpenalised.
Scaling matters because penalties act on coefficient magnitudes. Fit the scaler on training rows only. Choose penalty strength on validation data or time-aware folds.
Question: When can the final test be opened?
Result: A frozen specification and a clear distinction between interval types.
Builds on Specification, redundancy and regularisation.
For squared prediction loss, expected error can be decomposed into irreducible noise, squared bias and variance under a specified data-generating model.
More flexible fits often reduce training bias while increasing estimation variance. Regularisation and averaging can reduce variance. The trade-off depends on the sample and model, rather than a universal algorithm ranking.
%%{init: {"flowchart": {"nodeSpacing": 24, "rankSpacing": 20}, "themeVariables": {"fontSize": "20px"}}}%%
flowchart LR
A["Training<br/>fit parameters"] --> B["Validation<br/>compare rules"]
B --> C["Freeze features<br/>settings and cutoff"]
C --> D["Test once<br/>report limitations"]
D --> E["New ideas need<br/>new evaluation"]
Which is a defensible model-selection claim?
What does a coefficient confidence interval describe?
A large VIF most directly indicates what?
| Part | Question |
|---|---|
| 4.3.1 Trees, ensembles and feature use | How does a nonlinear learner make a prediction? |
| 4.3.2 Event probabilities and classification scores | How good are the event probabilities? |
| 4.3.3 Decision thresholds and held-out evaluation | Which probability should trigger a risk action? |
Question: How does a nonlinear learner make a prediction?
Result: Compare trees, forests and boosting on the same validation rows.
Begin with the definitions and assumptions; then trace the worked example.
Regression targets a continuous return. Classification targets the fixed 5% loss event. Keep the split and candidate spine unchanged when comparing models.
For a candidate split s, minimise the two child sums of squared errors:
\[\sum_{i\in L(s)}(y_i-\bar y_L)^2+\sum_{i\in R(s)}(y_i-\bar y_R)^2.\]
Each leaf predicts its training mean. Recursive splits capture thresholds and interactions automatically, but make piecewise-constant predictions.
flowchart TD
A["Is volatility below the fitted threshold?"] -->|Yes| B["Is momentum below a second threshold?"]
A -->|No| C["Leaf: mean training return"]
B -->|Yes| D["Leaf: mean training return"]
B -->|No| E["Leaf: mean training return"]
This diagram illustrates the mechanism. The fitted thresholds below come from training data.
For one insurance risk segment, should a tree predict the mean claim payout or the typical claim payout?
Invented teaching data; comparisons are illustrations, not measured deployment gains.
Payout is thousands of dollars per filed claim, in one fixed risk segment. No split is possible, so squared_error fits the training mean 3 and absolute_error the median 1. On the predeclared later fixture, the first wins MSE and the second MAE. Expected payout for budgeting and a typical claim are different targets. Both losses were declared before evaluation; these eight invented claims do not establish generalisation or total policy risk.
Read each path as a sequence of conditional comparisons. The leaf’s predicted return comes from outcomes of training rows reaching that leaf.
Modify: Increase depth or reduce minimum leaf size. Compare training and validation error; do not select settings using test returns.
For class proportions \(p_k\) in a node, Gini impurity is \(1-\sum_k p_k^2\) and entropy is \(-\sum_k p_k\log p_k\).
A split maximises the weighted reduction in impurity. A leaf estimates class probability from its training observations. A pure leaf can be a tiny, unreliable sample.
Classification impurity and regression SSE answer different loss functions. Neither automatically reflects the economic cost of a missed loss event.
Check: Why can a deep classifier attain perfect training accuracy yet give poor probability forecasts?
Bagging fits models on bootstrap samples and averages their predictions. A random forest also samples features at candidate splits, helping decorrelate trees. Averaging reduces variance most when errors are not perfectly correlated.
Gradient boosting adds a small learner in the direction that reduces the chosen loss. Under squared error, the negative gradient is the current residual.
\[F_m(x)=F_{m-1}(x)+\eta h_m(x).\]
Learning rate η, tree depth, leaf size and number of stages jointly control flexibility. Tune them on time-aware validation, rather than assuming one ensemble dominates.
These settings are a small fixed demonstration, not an exhaustive optimisation. Compare with the OLS and constant baselines from §4.1. More models tried means more selection uncertainty.
Impurity importance totals the training loss reductions credited to each feature. It can favour features with many possible split points.
Permutation importance measures how a declared evaluation score changes when a feature is shuffled. Correlated substitutes can obscure the contribution of either variable. Arbitrary shuffling can create implausible market observations or break time structure.
Neither measure is a causal effect. Report the model, sample, score and perturbation scheme. A stable predictive feature still needs an availability and implementation check.
Question: How good are the event probabilities?
Result: Explain log odds, confusion counts, discrimination and probability quality.
Builds on Trees, ensembles and feature use.
The next purchases occur after 7, 21, 45 and 80 days. Which customers count as inactive at each horizon?
Invented teaching data; comparisons are illustrations, not measured deployment gains.
Y30 means no purchase in the next 30 days; Y60 means no purchase in the next 60. Neither means permanent churn. Only use complete follow-up windows or explicitly handle censoring. The same p30 is scored against different events here to expose a target mismatch; the two Brier scores are not evidence that one horizon is inherently easier or better. Churn risk does not identify who benefits from a retention offer.
\[p_i=P(Y_i\le-0.05\mid X_i),\qquad \log\frac{p_i}{1-p_i}=\beta_0+X_i^\top\beta,\qquad p_i=\frac1{1+e^{-(\beta_0+X_i^\top\beta)}}.\]
A coefficient changes log odds. \(e^{\beta_j}\) is the conditional odds multiplier for a one-unit feature increase. Odds and probabilities differ.
Logit estimates coefficients by maximising Bernoulli likelihood, equivalently minimising log loss. Regularisation adds a penalty. A probability model requires calibration checks as well as ranking scores.
The event rates are about 21.7% in training and 16.2% in validation. Validation AUC is about 0.544.
StandardScaler inside the pipeline learns its means and scales only during training. A numeric coefficient here measures a one-training-standard-deviation change; scaling dummy variables also changes their coefficient units.
Check: AUC above 0.5 in one validation sample does not establish a calibrated probability or a net economic benefit.
Predict an adverse event when \(p\ge c\).
| Actual / prediction | Predict event | Predict no event |
|---|---|---|
| Event | True positive (TP) | False negative (FN) |
| No event | False positive (FP) | True negative (TN) |
Precision = TP/(TP+FP), recall = TP/(TP+FN), specificity = TN/(TN+FP). Accuracy can look high when events are rare.
Lowering c tends to increase both true positives and false positives. State how undefined ratios are handled when a denominator is zero.
ROC AUC measures ranking across event/non-event pairs. ROC traces recall against false-positive rate as the threshold changes. Precision–recall curves are useful when positives are rare, and depend strongly on prevalence.
Brier score is \(n^{-1}\sum_i(p_i-y_i)^2\). Log loss is \(-n^{-1}\sum_i[y_i\log p_i+(1-y_i)\log(1-p_i)]\).
Calibration compares predicted probabilities with observed event frequencies. Small bins and dependent observations produce uncertain calibration estimates.
Use a training-prevalence baseline. AUC ignores threshold costs and probability calibration.
Question: Which probability should trigger a risk action?
Result: A validation-chosen threshold evaluated once on held-out outcomes.
Builds on Event probabilities and classification scores.
Predict, then move: Which cases change at thresholds 0.20 and 0.80? Explain why the minimum toy cost does not identify the correct threshold for the real validation sample.
Suppose a false alarm costs 1 and a missed adverse event costs 4, in declared abstract decision-cost units. With calibrated probabilities and constant costs, the expected-cost threshold is \(1/(1+4)=0.20\).
The model may be miscalibrated. This exercise also evaluates a fixed threshold grid on validation only.
The grid selects 0.65, costing 56 units versus 64 for never alarming and 83 for always alarming. This is a validation result, with tuning uncertainty. These abstract costs are not realised trading P&L.
One transaction is fraudulent. Does the usual 0.5 cutoff minimise missed-fraud and investigation costs?
Invented teaching data; comparisons are illustrations, not measured deployment gains.
Y is fraud confirmed within a specified 30-day outcome window. Fit probabilities with log loss or Brier; assess ranking with AUC; choose actions with declared costs and capacity. Under calibrated probabilities and constant costs, alert at p >= C_FP/(C_FP+C_FN). Otherwise validate the rule. This tiny validation comparison is not final test evidence.
Test AUC is about 0.431, and Brier score about 0.2176. This weak result is useful evidence against assuming the fitted risk model generalises.
Inspecting this test to invent another feature or threshold would turn it into development data. A revised model needs a new evaluation protocol.
Which claim is supported by a validation AUC of 0.544?
The event is a next-month simple return of −5% or worse. Fix the comparison without inventing labels for pending rows.
A logit coefficient is log(2) per unit. What doubles for a one-unit increase, holding other inputs fixed?
Under squared-error gradient boosting, the next learner fits what?
| Part | Question |
|---|---|
| 4.4.1 Prediction, intervention and confounding | Would changing a predictor change the outcome? |
| 4.4.2 Difference-in-differences | Can a comparison group identify a change? |
| 4.4.3 Instrumental variables | Can external variation identify the effect? |
Question: Would changing a predictor change the outcome?
Result: Distinguish association from an intervention and identify confounding.
Begin with the definitions and assumptions; then trace the worked example.
A randomised offer raises average revenue by 3 per customer. Should it be used if each offer costs 4?
Invented teaching data; comparisons are illustrations, not measured deployment gains.
Invented randomised groups illustrate an estimated average revenue effect of 3, before uncertainty. With cost 4 the estimated net effect is -1; with cost 2 it is +1, assuming revenue is contribution before offer cost. High predicted spending alone does not identify incremental response. Observational adjustment, DiD and IV each need their own identification assumptions; evaluate incremental contribution and uncertainty.
“Traders who use a risk dashboard have higher risk-control scores.” Which action needs a causal answer?
Potential outcomes: each trader has \(Y(1)\) with the tool and \(Y(0)\) without; we see one. The observed gap is \(\underbrace{E[Y(1)-Y(0)\mid T=1]}_{\text{causal}} + \underbrace{E[Y(0)\mid T=1] - E[Y(0)\mid T=0]}_{\text{selection bias}}\).
Simulate a hidden trait \(U\) (risk discipline) that drives both tool adoption \(T\) and risk-control score \(Y\). By construction the tool has zero effect.
Coefficient 2.17 with t = 24 and R² = 0.23. As a predictor of risk-control score, tool usage is excellent. As a policy, pushing the tool would move risk-control score by exactly 0.
Add \(U\) as a second regressor. The coefficient on \(T\) will:
Adjusted: 0.066 (truth 0). With a real effect of 0.5, naive says 2.67, adjusted says 0.45. This works only because \(U\) was measured. Draw the graph \(T \leftarrow U \rightarrow Y\) first; choose controls from the causal graph. Conditioning on mediators or colliders can change or bias the target effect.
%%{init: {"flowchart": {"nodeSpacing": 24, "rankSpacing": 20}, "themeVariables": {"fontSize": "20px"}}}%%
flowchart LR
U["Unobserved risk discipline"] --> T["Dashboard adoption"]
U --> Y["Risk-control score"]
T --> Y
Question: Can a comparison group identify a change?
Result: Calculate DiD and state the parallel-trends assumption.
Builds on Prediction, intervention and confounding.
Treated group gets a policy between period 0 and 1; the control group does not. \[\hat\tau_{\text{DiD}} = (\bar Y^{\text{tr}}_{1} - \bar Y^{\text{tr}}_{0}) - (\bar Y^{\text{ctl}}_{1} - \bar Y^{\text{ctl}}_{0})\]
The identifying assumption of DiD is:
Treated group starts 1.0 higher, everyone drifts up 0.5, and the policy adds 0.8 to the treated group in period 1.
Both naive numbers are wrong: the post gap (1.85) includes the pre-existing level difference; the before-after (1.31) includes the common drift.
did currently holds the treated group’s before-after change. Subtract the control group’s change so that did is the difference-in-differences.
The regression \(y = \alpha + \beta\,\text{group} + \gamma\,\text{period} + \delta\,(\text{group} \times \text{period})\) gives \(\hat\delta\) = 0.861 (se 0.051) — identical to the hand calculation, now with a standard error. Counterfactual parallel trends cannot be verified directly. Pre-treatment trends can challenge the assumption but cannot prove it. This independently simulated cross-section uses an ordinary OLS standard error; real repeated-unit designs need design-appropriate inference.
Question: Can external variation identify the effect?
Result: Explain relevance, exclusion and the weakness of a weak instrument.
Builds on Difference-in-differences.
An instrument \(Z\) moves \(T\) but touches \(Y\) only through \(T\). Three conditions: relevance (\(Z \to T\)), exclusion (no \(Z \to Y\) path except via \(T\)), independence (\(Z \perp U\)).
Which condition can never be verified from the data alone?
Classic instruments: distance to college for schooling (Card 1995), draft-lottery number for military service (Angrist 1990), rainfall for agricultural income. Two-stage least squares: regress \(T\) on \(Z\), then \(Y\) on \(\hat T\).
Hidden \(U\) drives \(T\) and \(Y\); true effect of \(T\) on \(Y\) is 1.0; \(Z\) is a clean shock to \(T\).
OLS says 2.013 — twice the truth, because \(U\) pushes \(T\) and \(Y\) the same way. 2SLS uses only the part of \(T\) that \(Z\) explains and lands on 1.027. The one-instrument case reduces to the ratio \(\text{cov}(Z,Y)/\text{cov}(Z,T)\).
Strong instrument: F = 637. Weak one: F = 7.0 and the 2SLS estimate drifts to 0.72 and weak identification. The first-stage F > 10 rule is a context-dependent heuristic, not proof of valid identification. The hand calculation reports point estimates only; second-stage OLS standard errors are not valid IV standard errors.
| Design | Replaces the missing control with | Fails when |
|---|---|---|
| Regression adjustment | the measured confounder | a confounder is unmeasured |
| Difference-in-differences | a control group’s time trend | trends are not parallel |
| Instrumental variables | an exogenous shock to \(T\) | \(Z\) is weak or has its own path to \(Y\) |
| Part | Question |
|---|---|
| 4.5.1 Snapshots and comparable feature scales | Which stocks and features belong in the same snapshot? |
| 4.5.2 Attention weights and peer context | How are peer observations combined? |
| 4.5.3 Walk-forward feature comparison | Does contextual information improve a fixed baseline? |
Question: Which stocks and features belong in the same snapshot?
Result: A dated matrix with explicit universe and scaling limitations.
Begin with the definitions and assumptions; then trace the worked example.
Can a store use peer information without looking at those stores' future sales?
Invented teaching data; comparisons are illustrations, not measured deployment gains.
Rows and columns identify stores in the same snapshot. Affinity is exp(-distance/temperature), normalised across peers; these fixed weights are an attention-style illustration, not a trained Transformer. Values must be known before the forecast. A smoother context is a candidate feature; compare it with an unchanged baseline on later data under the chosen sales loss.
At month-end \(t\), \(X_t \in \mathbb{R}^{N \times K}\) holds \(N\) stocks × \(K\) characteristics and \(y_t \in \mathbb{R}^N\) holds next month’s returns. The prediction problem is cross-sectional: rank the \(N\) stocks, month by month.
25 335 rows = 119 months × 213 stocks in 11 sectors, 2015-01 … 2024-11. Five price-based characteristics: last-month return, 12-1 momentum, 21-day realised volatility, log dollar volume, price / 252-day high. ret_next at month \(t\) is exactly ret_1_0 at month \(t+1\) (0.100776 = 0.100776) — the target is the next row’s return.
The universe is today’s large caps, tracked back to 2015. Nothing that was large in 2015 and has since failed, shrunk or been delisted is in the file.
What does this survivorship bias do to a backtest run on the file?
Production rule: build the universe as of each month from a point-in-time index membership file. This file is fine for learning the mechanics — never quote its Sharpe ratio as evidence.
Why is (rvol_21d − rvol_21d.mean()) / rvol_21d.std() over the whole 2015–2024 file a look-ahead?
Median and std are computed within each month (and, for the _s columns, within each month × sector — the within-date transformation rule: a stock is judged against the peers it is compared with on the day). Sector medians are exactly 0 afterwards; values are clipped at ±3 to reduce the numerical influence of extremes; clipping does not guarantee robust estimation.
Question: How are peer observations combined?
Result: Trace similarity, temperature and context into a feature vector.
Builds on Snapshots and comparable feature scales.
“Sector” is a label assigned by an index provider. In October 2022 NVDA’s behavioural peers — high volume, high volatility, far below the 52-week high — included AMD, but also Netflix, Alibaba and Tesla, none of them “Technology”.
Attention builds the peer group from the data. Compare query vectors \(Q = XW_q\) with reference vectors \(K = XW_k\), then combine the value vectors \(V = XW_v\):
\[A = \text{softmax}_{\text{rows}}\!\left(\frac{QK^\top}{\sqrt{d}\,\tau}\right), \qquad C = AV\]
With \(W_q = W_k = I\), the weight stock \(i\) gives stock \(j\) is proportional to
NVDA’s attention row in October 2022: AMD 0.353, itself 0.263, NFLX 0.154, BABA 0.077, TSLA 0.055, AMZN 0.028 — six stocks carry more than 1 %, from three sectors. The context \(C\) is the peer-weighted characteristic vector: 12-1 momentum −1.49 for NVDA against −1.38 for its peers, dollar volume 3.0 against 2.52.
With identity weights and \(\tau \to \infty\) every score goes to 0. What does NVDA’s context vector collapse to?
the cross-sectional mean of X — every weight is 1/N
Temperature is the dial between two crude peer groups: \(\tau \to 0\) attends only to the most similar stock (at \(\tau = 0.05\) AMD carries 0.997 — NVDA is not even its own nearest point), \(\tau \to \infty\) gives the cross-sectional mean (0.02, 0.10, 0.19, 0.00, −0.14), while median-centred per-month features need not have exactly zero means. \(\tau = 1\) sits in between: a handful of behavioural peers.
“Artificial Intelligence Asset Pricing Models” (KKMX, 2025) puts a transformer over the stock cross-section, not over words:
What we do instead
Weights stay fixed (\(W = I\); a seeded random \(W\) works the same way and is the random-features view). The attention output is used only as feature engineering: for every stock, the context \(C\) (peer-weighted characteristics) and the deviation \(X - C\) (“how I differ from my attention-weighted peers”). The learner on top is the boosting machine from §4.3.
Fifteen columns per stock-month: 5 market-relative characteristics, 5 peer contexts, 5 deviations. NVDA in October 2022: 12-1 momentum −1.49 against −1.38 for its peers, price / 52-week high −2.67 against −2.59 — deviations of −0.10 and −0.08. With a self-weight of 0.26 and AMD at 0.35, NVDA largely is its peer group that month. Only month-\(t\) rows enter the transformation. This prevents future cross-sections entering this step, but does not establish historical source availability.
Question: Does contextual information improve a fixed baseline?
Result: A common-protocol comparison with uncertainty and sample limits.
Builds on Attention weights and peer context.
The row dated 2018-12-31 has ret_next = January 2019’s return. May it sit in the training set when we predict the 2019-01-31 row?
Expanding window from 2019-01: 71 test months, 6 refits (every 12 months, predicting the next 12 — a monthly refit would be 71 fits, same protocol, ten times the runtime). Two scores per month: rank IC (Spearman between prediction and realised return across the 213 stocks) and the top-20 minus bottom-20 equal-weight spread.
Which feature set will have the highest mean rank IC — and will the difference be significant?
Mean rank IC: −0.015 (t = −1.04) raw, −0.014 (t = −0.99) with sector features, −0.010 (t = −0.67) with attention. None of these approximate unadjusted tests rejects zero at the usual 5% level. Failure to reject does not establish zero effect or equivalence. The long-short spread is positive (+7.7 %, +9.3 %, +2.8 % a year; Sharpe 0.47, 0.62, 0.17, before costs) — the extremes rank slightly better than the middle, but 71 months cannot separate a Sharpe of 0.6 from 0. Attention features did not help here; the protocol that tells you so is the deliverable.
Under an i.i.d. standard-error approximation, the t-statistic of a spread with Sharpe \(S\) over \(T\) months is \(S\sqrt{T/12}\): with \(T = 71\), the best set’s Sharpe of 0.62 gives t = 1.51; the raw set’s 0.47 gives 1.15; attention’s 0.17 gives 0.40. To reach t = 2 the sector set would need 124 months of the same performance (10 years), the raw set 18 years. Every gain in the plot comes with an equally plausible zero — and the file is survivorship-biased in the spread’s favour.
Change the temperature (tau_new = 3.0) or build the reference vectors from a subset of characteristics (peer_columns_new = ["ret_12_1_z", "rvol_21d_z"]) so that ic_new differs from the baseline attention IC. Does the peer-group definition move the IC by more than its standard error (about 0.015)?
\(\tau = 3\): IC −0.010 (unchanged); reference vectors from momentum and volatility only: −0.025 (t = −1.64); \(\tau = 0.3\): −0.041 (t = −2.60); \(\tau = 10\): −0.008. The sign of the change flips with the setting, and the one “significant” number appears after four tries — that is multiple testing, not a signal. In KKMX the peer group is learned on a Sharpe objective with thousands of stocks and six decades; here it is a fixed kernel on 213 survivors.
| Part | Question |
|---|---|
| 4.6.1 Frozen forecasts and stateful refits | When is each forecast formed and which labels are mature? |
| 4.6.2 Risk rules, holdings and cash | What happens to uninvested capital and trading costs? |
| 4.6.3 Comparison and the decision memo | What evidence supports the proposed policy? |
Question: When is each forecast formed and which labels are mature?
Result: Forecasts with stock and date columns from an availability-aware refitting procedure.
Begin with the definitions and assumptions; then trace the worked example.
At 10 February, can a model train on a January customer whose full 30-day outcome is still unknown?
Invented teaching data; comparisons are illustrations, not measured deployment gains.
The displayed table is a later export. On 10 February only the first outcome was available; even the second row’s eventual zero was not yet known. Keep new predictions and pending outcomes. Predeclare the target, fitting loss, validation metric and action rule; then evaluate later realised contribution, service levels and costs. A corrected objective may improve alignment, but generalisation must be measured.
%%{init: {"flowchart": {"nodeSpacing": 24, "rankSpacing": 20}, "themeVariables": {"fontSize": "20px"}}}%%
flowchart LR
A["Return<br/>forecast"] --> C["Rank stocks<br/>select top three"]
B["Loss<br/>probability"] --> D["Apply frozen<br/>risk threshold"]
C --> D
D --> E["Target weights<br/>retain cash"]
E --> F["Trade from<br/>drifted holdings"]
F --> G["Costs, realised<br/>P&L and NAV"]
The return model ranks candidates. The classifier can reduce exposure under a declared policy. Evaluate each statistical model separately, then evaluate the complete net account. Weak classification evidence remains a limitation even if one portfolio path looks good.
For each calendar year, refit using labels available strictly before 1 January. Predict every month in that year using its available features. This conservative annual protocol differs from refitting at each month-end.
Ridge alpha 100 (selected by the fixed validation grid), logit C 1, the 5% event definition and risk threshold 0.65 are frozen settings. The threshold grid was studied using 2021 validation. Do not retune them using the subsequent portfolio outcomes.
There are 315 predictions, 306 evaluable outcomes and 9 pending predictions. Preprocessing is refitted within each training sample. Stock/date columns beside predictions survive concatenation.
The loop represents three successive fitting clocks. It does not mix future outcomes into an earlier fit. Exact execution, historical membership and vendor timestamps remain unverified classroom assumptions.
Question: What happens to uninvested capital and trading costs?
Result: A holdings, cash and fee ledger that reconciles through time.
Builds on Frozen forecasts and stateful refits.
At the frozen 0.65 threshold, this sample blocks none of the selected positions, so the baseline and filtered accounts are identical. This rule supplies no realised risk reduction here. An allowed selected stock keeps weight 0.33. A blocked stock receives zero, and its allocation stays in cash. Rescaling the survivors to 99% would define a different risk policy.
The row order provides a deterministic tie rule in this exercise. A production specification must state tie-breaking, capacity, borrow and concentration constraints.
This function carries holdings and cash across periods, so a short stateful loop is appropriate. Trades start from drifted holdings, rather than last month’s target weights. Cash earns zero in this exercise.
Both accounts contain 34 realised monthly holding periods. Initial purchases incur fees. NAV is measured after fees and realised simple-return P&L. The identity NAV change = gross P&L − fees must hold at every row.
Question: What evidence supports the proposed policy?
Result: A paired comparison that reports exposure, uncertainty and pending outcomes.
Builds on Risk rules, holdings and cash.
Report the actual values produced, exposure differences, turnover and fee assumptions. Apply Chapter 3’s paired, dependence-aware uncertainty to the monthly return difference.
Thirty-four months and nine surviving stocks cannot establish a reliable risk reduction. A net account can reconcile perfectly while its model generalises poorly.
One of three 0.33-weight stocks is blocked by the risk rule. What follows from this policy?
Report target and horizon; candidate/label counts; feature availability; training/validation/test clocks; fitted models; baseline metrics; frozen decision costs and threshold; exposure, net fees and drawdown; uncertainty and limitations.
The fixed classifier’s test AUC of 0.431 does not support claiming a dependable loss detector. The attention extension does not establish that a transformer learns a profitable strategy. Keep negative findings visible.
Your investigation: Change one declared assumption on a development sample, record the attempt, explain the changed predictions and account, and propose fresh evidence for evaluation. Do not rename an examined test “untouched”.
The last nine candidates have pending outcomes. What is appropriate?
If NAV change equals P&L minus fees at every date, what has been established?
| Concept | What to explain |
|---|---|
| Regression | Conditional mean, residuals, assumptions and forecast errors |
| Inference | Covariance estimator, confidence interval, tests and selection uncertainty |
| Classification | Event horizon, odds, calibration, ranking and threshold costs |
| Causal reasoning | Identification assumptions rather than predictive association |
| Feature context | Fixed attention as engineering, with availability and survival limits |
| Risk decisions | Exposure, pending labels, drifted holdings, fees and NAV reconciliation |
ISOM 5650 · Prof. Xuhu Wan · HKUST Business School