3.4 — Association: Linear, Monotonic, and Nonlinear

Chapter 3 · Reshaping Statistics

Prof. Xuhu Wan

Section 3.4 · Chapter 3 · Learning Statistics with Python

Association: Linear, Monotonic, and Nonlinear

Reshaping Statistics

Prof. Xuhu Wan

ISOM, HKUST Business School · 2026 Edition

Association: Linear, Monotonic, and Nonlinear

Pearson’s \(r\) measures only the linear part of a relationship. You will build data that is strongly related yet fools Pearson completely, find which coefficient survives a monotone bend, which survives any bend at all — and then ask whether the association between two stocks is the same on the worst days as on ordinary ones.

Predict: can Pearson see a U-shape?

We make \(y = (x-5)^2 + \text{noise}\) — a clean, strong, non-linear relationship. Predict what Pearson’s \(r\) reports.

Pearson \(r\) between \(x \sim U(0,10)\) and \(y = (x-5)^2 + \varepsilon\) (seed 61, \(n = 300\)). Near 0, near 0.5, or near 1?

"near 0"

0.04

\(r \approx 0.04\) on a relationship you can see with your eyes. Pearson is blind to anything non-monotone. Its requirements — finite variance, linearity, outlier sensitivity — are assumptions, not guarantees.

Association: Pearson sees straight lines only

\(y = \sin x + \varepsilon\) is perfectly determined by \(x\) up to noise. Predict Pearson’s \(r\).

For y = sin(x) + noise over \(x \in [0, 100]\), Pearson’s \(r\) is…

Close to +1
Close to −1
Close to 0
Undefined

In the index panel spy is tomorrow’s SPY change and the other columns are today’s moves: every correlation in the spy row is near zero. Today’s Hang Seng is not a linear predictor of tomorrow’s SPY. “No association” here means no linear association.

Will rank measures rescue you?

Spearman (\(\rho_S\), Pearson on ranks) and Kendall (\(\tau\), concordant minus discordant pairs) catch any monotone relationship. Predict whether they catch a parabola.

For a symmetric parabola, Spearman and Kendall will report…

Strong positive values (~0.8)
Strong negative values
Near zero — they only see monotone trends
Exactly Pearson’s value by coincidence

Spearman: ranks instead of values

Spearman’s \(\rho\) is Pearson’s \(r\) on the ranks. Predict it for \(y = \log x\).

\(\log\) is strictly increasing, so the rank of \(y\) equals the rank of \(x\) for every point. What is Spearman’s \(\rho\)?

round(stats.spearmanr(xs, np.log(xs))[0], 3)

1.0

Spearman rescues monotone nonlinearity (log: Pearson 0.92 → Spearman 1.00) and nothing else: the parabola (0.08 / 0.06) and the cosine (0.04 / 0.03) are invisible to both coefficients.

Kendall’s tau: the same verdict from pairs

Kendall counts pairs: \(\tau = (\text{concordant} - \text{discordant}) / \binom{n}{2}\). It is more robust to ties and outliers than Spearman, and just as blind to a bend.

Log: \(\tau = 1.000\); parabola: \(0.039\); cosine: near zero. The rule discovered: rank measures fix outliers and curvature-but-monotone, not non-monotonicity. For “any dependence at all” you need distance correlation (Székely, 2007), which is zero if and only if \(X \perp Y\).

Distance correlation sees any dependence

Székely’s distance correlation is zero if and only if the variables are independent. The notebook uses the dcor package; we build it from pairwise distances.

In Colab

!pip install dcor --quiet
import dcor
dcor.distance_correlation(parabo["x"], parabo["y"])      # ≈ 0.49, 12 ms on 10 000 points

Which measure will be clearly positive for the parabola?

Pearson only
Spearman only
Pearson and Spearman
Distance correlation only

Distance correlation: log 0.97, parabola 0.50, cosine 0.37 — the notebook’s dcor gives 0.97 / 0.49 / 0.35 on its own draws. Only this column sees the two non-monotone shapes.

Your turn: does today’s return predict tomorrow’s?

feat holds next-day return (target), today’s return (lag1) and the 5-day sum (ret5) for 2020–2021. Set dc_lag1 to the distance correlation between lag1 and target, and compare with Pearson.

Does association survive on the worst days?

returns.csv holds daily S&P 500 and Tesla returns, 2015–2024. One Pearson number describes ten years. Predict what happens conditionally on the S&P’s worst 5 % of days.

On the S&P’s worst 5 % of days, what share are also among Tesla’s worst 5 %?

About 5 % — as under independence
About 12 %
About a third — some six times the independence value
100 % — Tesla always crashes with the market

Pearson 0.465 over 2 515 days; on the 126 worst S&P days Tesla was in its own worst 5 % on 32.5 % of them — 6.5 times the independence rate. (Pearson within those days, 0.355, is lower — conditioning on a slice of \(x\) truncates its variance and mechanically shrinks \(r\).) The tail is where diversification is tested, and §3.5 is about the tail.

What you discovered

Pearson: linear only. The U-shape gave \(r = 0.04\); \(\sin x\) gave about 0. Zero correlation is not independence.

Spearman and Kendall see monotone bends (log: 1.00) and nothing else (parabola: 0.06 and 0.04).

Distance correlation is zero only under independence: parabola 0.50, cosine 0.37, built from pairwise distances with pdist.

In the index panel, no lagged index predicts tomorrow’s SPY linearly; in 2020–2021 the distance correlation of today’s Apple return with tomorrow’s is the practice you just did.

One correlation number hides the tail: Tesla joins the S&P’s worst 5 % of days a third of the time, 6.5 times the independence rate.

Next: §3.5 — the tails alone: what is the worst 20-day loss you should plan for?