Chapter 1 · Data Structures and Methods of Series
Section 1.2 · Chapter 1 · Learning Statistics with Python
Data Structures and Methods of Series
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
Every row-by-row loop you are tempted to write has a one-line vectorised replacement. You will time them, predict what where keeps, and discover why the 95th percentile is where outliers get clipped.
Three ways to flag prices above 50. Predict the fastest before timing.
Which is fastest on 20 000 prices?
price.apply(compare)for loop appending to a listprice.gt(50)All three agree (True True); only the last is essentially free — tens of microseconds against milliseconds. Rule: reach for apply only when no vectorised method exists.
where keep?price.where(price.ge(50), other=0) returns …
[0.0, 0.0, 0.0, 100.0, 121.95] — exactly the notebook output — versus [44.99, 24.81, 46.0, 0.0, 0.0] for mask. The "Big" version works but turns the whole Series into object dtype: no more arithmetic.
Labelling prices “Expensive” / “Cheap” is an if-else on every row. Three ways — one slow, two fast.
{'Expensive': 15146, 'Cheap': 4854} all three ways. np.where(cond, if_true, if_false) is the idiom you will reuse most: it reads like the sentence you meant.
The notebook practice downloads a stock with yfinance; here the same AAPL closes come from appl.csv.
ret holds AAPL daily returns 2015–2024. Compute sharpe = mean / std of ret (use .agg or the methods), and win_std = the std of returns on winning days only, using where (not a filter).
NaN is not a number — and not an error. It must be counted, dropped or filled before any model sees it. The notebook’s full inventory file has SoldFlag missing for every Active product (61.8 %). Our 20 000-row sample is all Historical, so we take the notebook’s other dataset, publicCompany, where dividendYield is blank for non-payers.
1 610 of 3 084 companies have no dividendYield. What does round(dy.isna().mean(), 3) print?
0.522
Same trick as before: the mean of isna() is the missing rate, 52.2 %. Aggregations skip NaN by default — convenient, and dangerous if you forgot to look.
dy has 1 474 numbers and 1 610 holes. Predict two things: does .sum() return a number, and what does dy + 1 leave in the holes?
dy.sum() and (dy + 1).isna().sum() give …
NaN and 0 — one hole poisons the sum, but adding 1 fills it48.28 and 1610 — the sum skips holes, the addition keeps themNaN and 1610 — holes poison everything48.28 and 0 — holes are treated as 0 everywhereThe split you found: aggregations skip NaN; element-wise operations propagate it. Same missing value, opposite treatment — and count() (non-missing) is not len().
Assign to dy_clean a version of dy with the missing dividend yields removed (.dropna()). Expected: 1 474 entries, zero NaN.
Strategy, not reflex: for prices, ffill() (“use the last known price”) is often right; for a dividend yield, dropping — or filling with 0 for a confirmed non-payer — depends on what the blank means. The tool follows the meaning of the data.
pd.Series([1, NaN, NaN, 4]).interpolate() fills the gap with …
1, 12, 30, 02.5, 2.5sold is a 0/1 flag with two gaps (the notebook’s SoldFlag for Active products). Fill the gaps with the mode (sold.mode().loc[0]) into filled. Expected: no NaN, and the total is still 1.
price.sort_values() with no arguments — which end is the 0.0 minimum at?
0.0
Ascending by default; the row labels travel with the values ({15511: 2800.0, 4071: 2083.09, 2218: 1580.0} at the top), so a later sort_index() restores label order and forgets the value sort. Sorting returns a new Series — price is untouched.
s.quantile(0.8) == 21 tells you …
Clipping = capping and flooring. The max falls from 2 800 to 271.67, the min rises from 0 to 12.50, and the standard deviation drops from 85.93 to 71.16 — one method call removes the influence of 5 % of the tails on each side.
Clip the 2024 AAPL closes close24 to lie between their 5th and 95th percentiles, into clipped. Expected min ≈ 169.30, max ≈ 247.86.
dlist = [40, 20, 30, 20, 10] has the value 20 at positions 1 and 3.
Which call returns [40, 30, 10] — no 20 at all?
dlist.drop_duplicates()dlist.drop_duplicates(keep="first")dlist.drop_duplicates(keep="last")dlist.drop_duplicates(keep=False)partPrice = price.loc[:10] — the first eleven list prices (.loc is inclusive, so 0 … 10).
The first four prices are 44.99, 24.81, 46.00, 100.00. What are their ranks (1 = smallest) within the eleven?
[3.0, 1.0, 4.0, 7.0]
rank() gives 1 to the smallest (ties get the average rank); rank(pct=True) rescales to \((0, 1]\) — a cross-sectional percentile in one call, the workhorse feature of Project 2. replace takes a list → one value, or a dict of old → new. The result is object dtype the moment a string enters.
rlist holds body weights. Build label: "strong" where weight > 50, else "slim". Use np.where (or .where twice, or .apply).
pd.cut(price, bins=4) on prices from 0 to 2 800 — how full is the first bin?
Equal width: 19 990 of 20 000 in (-2.8, 700]. The notebook’s own edges give [607, 19389, 4, 0] — 607 free items exposed. qcut produces four bins of ≈ 5 000. Binning turns a number into a category — the input to groupby in Chapter 2.
apply hides a loop; .gt, np.where, .where run in C — hundreds of times faster.where(cond, other) keeps where True and never changes length; mask is its mirror.isna().mean() is the missing rate; aggregations skip NaN, element-wise operations propagate it; choose dropna / fillna / ffill / interpolate by what the gap means.sort_values is ascending by default; clip at the 5th/95th percentiles caps and floors the tails.drop_duplicates(keep=…), rank, replace, pd.cut / pd.qcut — all return new Series.Next: §1.3 — naming and re-numbering the index, and the one rule about .loc that bites everyone.
Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python