Chapter 1 · Data Structures and Methods of Series
Section 1.4 · Chapter 1 · Learning Statistics with Python
Data Structures and Methods of Series
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
A price is a number; a return is a relation between today and yesterday. You will build that relation with shift, smooth it with rolling, track the worst loss with cummax, and change the clock with resample.
weather[“date”].iloc[0] right after pd.read_csv is a …
strTimestampdatetime64int (days since 1970)In Colab
weather = isom5650.data.weather() — the same Delhi daily-climate table (1 462 rows, 2013-01-01 … 2017-01-01).
The data run 2013-01-01 … 2017-01-01. How many rows does dates[dates.dt.is_month_end] have?
48
48 month-ends across 2013–2016 (the index touches 5 calendar years because it ends on 2017-01-01). strftime turns dates back into text in any layout; to_datetime reads them back. Why .dt? Series.year would be ambiguous — an attribute you added, or the year inside the values? The accessor makes it explicit.
Mondays are said to under-perform Fridays. Compare the mean daily AAPL return by weekday.
Compute mon_mean = mean of r_aapl on Mondays and fri_mean = mean on Fridays (use r_aapl.index.day_name() as the mask). Expected: Monday ≈ 0.0023, Friday ≈ 0.0001 — the opposite of the folklore.
shift(1) moves every value one row later. That single move is what turns a price series into a return series. The notebook downloads MSFT with yfinance; microsoft.csv holds the same ticker for 2015–2018.
msft.shift(1).iloc[0] is …
NaN\[r_t=\frac{P_t-P_{t-1}}{P_{t-1}}\] — today minus yesterday, over yesterday. With shift that is one line; pct_change is the same line pre-written.
True — identical. Mean 0.00091, sd 0.0142 per day. The forward return shift(-1) ends in NaN: tomorrow is not known yet, which is exactly why it is the target of a forecasting model, never a feature.
A rolling mean replaces each value by the average of the last \(k\) — a fast window reacts, a slow one filters. Predict how many rows the window costs.
msft.rolling(10).mean() needs ten values before it can average. How many leading NaN?
9
The 40-day line barely notices the dips the 10-day line follows. A fast average crossing above a slow one is the oldest trend signal in finance — you will test it in Project 1.
The 21-day realised volatility peaked at 43 % annualised on 2015-09-18 and averaged 14 % in 2017. rolling(...).std() is the risk measure GARCH will model in Chapter 7.
cumsum, cumprod, cummax carry a running total, product or record high. Predict what the product of gross returns recovers.
(1 + r).cumprod().iloc[-1] equals …
Compounded growth 1.882× — exactly the price ratio: (1 + r).cumprod() is the equity curve of a buy-and-hold backtest, two lines and no loop. The running sum says 0.711, a different (and wrong) number. cummax is the staircase of record highs; cummin never moves once the 40.29 minimum is in.
\[\text{drawdown}_t=\frac{\text{cummax}_t - P_t}{\text{cummax}_t}\] — how far below its record high the price sits. The maximum drawdown is the worst peak-to-trough loss. The notebook asks for ten years of a stock: appl.csv.
Compute the drawdown Series dd for aapl with cummax(). Expected maximum ≈ 0.387 on 2019-01-03.
Resampling changes the clock: daily → monthly is down-sampling (aggregate), minute → hourly likewise; monthly → daily is up-sampling (invent rows). Predict what label a month gets.
msft.resample(“ME”).mean() — what is the first index label?
2015-01-02, the first trading day2015-01-31, the calendar month end2015-01-15, the midpoint2015-01, a month period"W" means weeks ending Sunday. MSFT’s first 2015 row is Friday 2 January. Predict how many observations the first weekly bucket holds.
The first bucket (ending Sunday 2015-01-04) covers one trading day; full weeks hold five. resample is groupby for time — name the bucket ("W", "ME", "5min") and the aggregation (.last, .mean, .sum, .count) — and always check the first and last bucket for partial weeks.
resample("ME").agg("mean") returns one row per month. .transform("mean") returns one row per day, each carrying its month’s mean — so you can subtract it.
38 rows versus 779. The step line is the monthly mean broadcast back to every day — the same idea as groupby().transform in Chapter 2. The rolling mean is the smooth cousin of the same idea.
A US session has 390 minutes, 09:31 … 16:00. How many rows does m["Close"].resample("5min").ohlc() return?
79
Not 78 but 79: bins are anchored on the clock (09:30, 09:35 …), so the first bar holds only 09:31–09:34 and the 16:00 print sits alone in the last one. .ohlc() rebuilds candlesticks at any frequency; .sum() is right for volume, .last() for a close, .mean() for a level. The aggregation must match what the number means.
Three 5-minute closes become eleven 1-minute rows, eight of them empty. ffill holds the last known value (what a trader knew); interpolate draws straight lines (what a chart wants). Up-sampling never adds information — only rows.
Two ways to draw a Series: matplotlib directly, or the .plot family every Series carries. The notebook starts with 5 000 normal draws (mean 10, sd 5) — the one place it simulates.
In plt.hist(x, bins=50, density=True) the bars are scaled so that …
fig.add_subplot(rows, cols, n) places panel n on a grid; every Series.plot.* accepts ax= so pandas draws into the panel you chose. The KDE over the density histogram is the picture of “fat tails” you will test formally in Chapter 2. Try msft.plot.box().
value_counts().plot.bar() / .barh() / .pie() is the whole recipe for a categorical column. Prefer bars: a pie hides that Healthcare (500) and Technology (486) are almost tied.
pd.to_datetime turns text into Timestamps; .dt unlocks day_name, is_month_end, strftime.shift(1) builds yesterday; (P − P.shift(1)) / P.shift(1) is pct_change; shift(−1) is tomorrow — the forecast target.rolling(k) costs \(k-1\) rows; rolling(21).std()·√252 is realised volatility.(1+r).cumprod() compounds to \(P_T/P_0\); (cummax − P)/cummax is drawdown.resample("ME") labels month-ends, "W" Sundays (first bucket may be partial); agg collapses, transform broadcasts, ohlc rebuilds bars; density=True makes area = 1.Next: Chapter 2 — two-dimensional data: DataFrames, groupby, pivot, and the statistics you compute on them.
Warning
In Growth in a Time of Debt (2010), Carmen Reinhart and Kenneth Rogoff reported that countries with public debt above 90 % of GDP averaged −0.1 % real growth — a number cited by the UK Treasury, the European Commission and the US Congress to justify austerity after 2010.
In April 2013 Thomas Herndon, a UMass Amherst graduate student, obtained the spreadsheet. The averaging formula covered rows 30–44 instead of 30–49: Australia, Austria, Belgium, Canada and Denmark were silently excluded. Combined with an unusual country-weighting choice and some omitted years, the corrected figure for the > 90 % bucket was +2.2 % — no cliff at all.
Lesson for this chapter: a slice is a claim about which rows count. .loc[a:b] is right-inclusive, .iloc[a:b] is not; a NaN skipped by mean() is a row you did not average. Print len(), print isna().sum(), and never trust an aggregate whose denominator you have not seen.
Series methods produce numbers; the deliverable is a recommendation. Fill this in from your §1.4 output.
To: Head of Trading, Equity Desk From: <Your name>, quantitative analyst Subject: Proposal to avoid holding AAPL over weekends Date: 2026-09-08
Recommendation: Do not implement a Friday-sell / Monday-buy rule.
Evidence (AAPL 2015–2024, 2 515 daily returns,
appl.csv): - Mean Monday return 0.23 % vs Friday 0.01 % — the opposite sign to the folklore, and 0.2 % per day against a daily sd of 1.8 % is well inside noise. - 22 days lost more than 5 %; the worst, 2020-03-16 (−12.9 %), was a Monday — but Mondays also supplied 4 of the 21 best days. - Round-trip cost of the rule (≈ 0.02 % × 2 × 505 weekends) exceeds any plausible gain.Caveats: a single stock, one decade, no significance test yet (Chapter 2 supplies the t-test). Returns computed from
Close, notAdj Close.Next step: repeat on
sp500.csvandreturns.csv; run the two-sample test from §3.2 before revisiting.
An LLM writes pandas fluently — and confidently mixes versions and conventions. Three habits for this chapter:
resample("M") is deprecated in favour of "ME", fillna(method="ffill") is now .ffill(), and Series[0] on a labelled index no longer means position 0. Old answers still run — with warnings you will ignore until they become errors..loc["a":"c"] inclusive?” gets a correct answer nine times out of ten. The tenth silently drops a month from your backtest. A 5-element Series is a two-second check.apply without asking “is there a vectorised method?” Copilots default to apply(lambda …) because it always works. Ask for the np.where / .where / .clip version and for df.shape printed before and after every filter.| Concept | Tool |
|---|---|
| Build a Series; labels vs positions | pd.Series(data, index=, name=), .loc, .iloc |
| Vectorised arithmetic with alignment | + − × ÷, .add(fill_value=), & \| ~ |
| Aggregation | .mean() .median() .std(ddof=) .quantile() .agg([...]), mask.mean() |
| Conversion | .astype(), pd.to_datetime(), .dt |
| If-else without loops | np.where, .where, .mask |
| Missing data | .isna().mean(), .dropna(), .fillna(), .ffill(), .interpolate() |
| Sorting, clipping, duplicates, ranking, binning | .sort_values(), .clip(), .drop_duplicates(keep=), .rank(), .replace(), pd.cut / pd.qcut |
| Index surgery | .rename(), .rename_axis(), .reset_index(drop=), .reindex(), .sample() |
| Time series | .shift(), .pct_change(), .rolling(k), .cumprod(), .cummax(), .resample("ME") |
| Plotting | plt.hist(density=True), Series.plot.{line,hist,kde,bar,pie}, fig.add_subplot |
series1 + series2 returned NaN where the indexes did not match. Give one business situation where that NaN is exactly what you want, and one where fill_value=0 is the right call..loc["2020-01-01":"2020-12-31"] and .iloc[0:253] returned the same 253 rows for the S&P 500. Write down a case where the two differ by one row and explain which one is the bug..ohlc(); a monthly mean from daily closes with .mean(). Why would .mean() be wrong for volume, and .sum() wrong for a price?Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python