Chapter 3 · Reshaping Statistics
Section 3.3 · Chapter 3 · Learning Statistics with Python
Reshaping Statistics
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
So far the data arrived and you tested it. Now you design the experiment: how many units to catch a one-point lift, what peeking at the dashboard does to your false-positive rate, and when earning while learning (a bandit) beats a fixed test. These are simulations by necessity — you cannot rerun a market.
Two arms, fixed \(N\), a hypothesis fixed before the data. The design is a century old (Fisher’s agricultural plots) and still the gold standard when you can afford it. You will discover its single most-violated requirement.
Control outcome \(\sim N(0.10, 1)\), treatment \(\sim N(0.18, 1)\), 800 each. The \(z\)-statistic is \(\hat\tau / \widehat{\mathrm{SE}}\), reject if \(|z| > 1.96\). Predict whether this experiment detects the true 0.08 lift.
There is a real 0.08 lift. Does \(|z| > 1.96\) (significant)?
A genuine effect, declared “not significant.” The discovery: the failure is not the effect — it is the sample size. Significance is a property of power, and power is something you size before you run.
For a baseline rate \(p = 0.10\) and an absolute lift of \(0.01\) (10 % → 11 %), the formula \(n \approx 2\sigma^2(z_\alpha + z_\beta)^2/\tau^2\) at 80 % power. Predict the order of magnitude.
Required \(N\) per arm for \(p = 0.10\), lift \(= 0.01\), \(z_\alpha = 1.96\), \(z_\beta = 0.84\). Integer (use math.ceil).
14112
About 14 000 per arm to catch a one-point lift. Most “failed” experiments never had the sample size to succeed. The previous slide’s \(n = 800\) was doomed before it ran.
The required \(N\) scales with \(1/\tau^2\): doubling the detectable lift should cut \(N\) by four.
Set n2 to the required \(N\) per arm (integer, math.ceil) for the same baseline \(p = 0.10\) but a lift of 0.02. Check that it is one quarter of 14 112.
Real teams peek — they check the dashboard at lunch and stop when “it looks significant.” You will discover, by simulation, exactly how much damage that does to the false-positive rate.
We simulate under a true null (no effect). At each new observation we recompute \(z\) and stop the first time \(|z| > 1.96\). Predict the resulting false-positive rate over 1 000 observations.
True null, peek-and-stop at every observation up to \(N = 1000\). False-positive rate?
Over half the null experiments produce a “significant” result. The rule you discovered: every look spends Type-I error. Naive peeking turns a 5 % test into a coin flip.
If repeated looks spend error, the cure is to budget it across looks. Predict which spending rule lets you peek hardest early.
Which sequential boundary is stringent early, relaxed later — the clinical-trial default?
Note
The cost of a properly-designed sequential test: ~5–15 % more samples than a fixed-\(N\) test of equal power. The gain: a principled option to stop early when the effect is large. Lan–DeMets \(\alpha\)-spending generalises this to any look schedule.
When the goal flips from “estimate the effect precisely” to “earn the most reward while learning,” A/B is the wrong tool. Bandits trade exploration against exploitation. You will race three policies head-to-head.
Three arms with true conversion rates [0.08, 0.10, 0.12]; arm 2 is best. Over 4 000 pulls we race \(\varepsilon\)-greedy, UCB1, and Thompson sampling. Predict which earns the most.
Which policy collects the most total reward here?
A perfect oracle expects \(0.12 \times 4000 = 480\); the gap is regret. Thompson lands closest — its pull counts pile onto arm 2 fastest. \(\varepsilon\)-greedy keeps wasting 10 % of pulls on losers forever. That is the exploration–exploitation trade-off, made concrete.
The design is decided before the data, and it depends on what the business actually pays for. Predict the right tool for each scenario.
You are picking ad creatives; the revenue during the test counts. Best tool?
Fixed-\(N\) A/B when you need a defensible point estimate (board, regulator) and the cost of a wrong call is one-time and large. Sequential when stakeholders will peek or units are expensive. Bandit when learning is earning.
You now know peeking inflates Type-I error. Predict another silent killer of experiments.
You measure 12 KPIs and declare victory on whichever turns significant. The bug?
Note
Other traps: SUTVA violations (treatment leaks between units → cluster-randomise / switchbacks); novelty effects (a new UI shines for a week → pre-register a 30-day hold-out); CUPED variance reduction can cut required \(N\) by 30–60 % at zero cost. Decisions made after seeing the numbers are p-hacking by another name — the same lesson as §7.6’s 42 rules on trial.
Next: §3.4 — two variables at once: which correlation coefficient sees a parabola?
Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python