Learning Statistics with Python
Chapter 1 · Learning Statistics with Python
pandas Series end to end: operators, aggregation, apply/where, missing data, sorting, clipping, ranking, binning, indexing, dates, rolling windows, resampling, plotting.
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
for loops — and they align on the index.where, fillna, clip, rank, replace, pd.cut..loc by label (right-inclusive), .iloc by position (right-exclusive), sample, reindex.shift → returns, rolling → moving averages, cummax → drawdown, resample → any frequency.Why this matters
Chapters 3–7 fit models to columns. A model is only as clean as the Series you feed it; the 30 methods here are 90 % of the data-preparation code you will write in this course and in practice.
| Section | Concept | Tool |
|---|---|---|
| 2.1 | Series basics, operators, aggregation, conversion | pd.Series, .index, .add, .agg, .astype, pd.to_datetime |
| 2.2 | Manipulation: if-else, missing data, sorting, clipping, ranking, binning | .apply, .where, .fillna, .sort_values, .clip, .rank, .replace, pd.cut |
| 2.3 | Indexing: rename, reset, .loc vs .iloc, sampling, reindexing |
.rename, .reset_index, .loc, .iloc, .sample, .reindex |
| 2.4 | Dates and time: shifting, rolling, cumulative, resampling, plotting | .dt, .shift, .pct_change, .rolling, .cummax, .resample, .plot |
A Python list holds numbers but cannot summarise them. A Series can — and it carries a label for every value. You will predict what the index does to arithmetic, and discover why ddof changes your standard deviation.
alist = [300, 100, 0, 200, 1000] is five monthly sales figures. Predict what happens when you ask the list for its mean.
What does alist.mean() do?
320.0[300, 100, 0, 200, 1000]AttributeError — lists have no meanNoneSame five numbers, but now sales.mean() returns 320.0. The index is a RangeIndex(start=0, stop=5) — pandas invented positional labels because you gave none.
A Series accepts a list, a dict, or a NumPy array. Predict which constructor sets the index for you.
What does pd.Series({"Jan": 300, "Feb": 100}).index print?
Index(['Jan', 'Feb'], dtype='object')
Dict keys become labels. The NumPy array gives float64, the integer list int64 — the dtype is inferred from the values, and every element of a Series shares it.
sales is now labelled Jan … May. There are two doors into it.
Which pair returns the January and May figures?
sales.loc[0] and sales.loc[4]sales.loc["Jan"] and sales.iloc[-1]sales.iloc["Jan"] and sales.iloc["May"]sales[0] and sales["May"] — both raiseComparing a Series to a number gives a Series of booleans of the same length — a mask. Predict its dtype and what indexing with it returns.
sales.median() is 200. Which labels survive sales[sales > sales.median()]?
['Jan', 'May']
bool mask, {'Jan': 300, 'May': 1000} survive. Combine masks with & and | (each side in parentheses) — never with Python’s and/or, which cannot handle five answers at once.
scoresheet holds your subject scores. Build low = the scores strictly below the mean (72.5). Expected: only Stats 20.
The notebook’s Kaggle sales-analysis file: historical sales for products a retailer must decide to keep or drop. PriceReg is the list price, SoldFlag = 1 if sold in the past six months.
df["col"] (or df.SoldFlag) hands back a Series named after the column, with the frame’s RangeIndex — 44.99, 24.81, 46.0 ….
(ReleaseNumber + 2) / 100 applies to every element at once (broadcasting). No loop.
The first ReleaseNumber is 15. What is the first element of (ReleaseNumber + 2) / 100?
0.17
[0.17, 0.09, 0.02] and revenues [674.85, 173.67, 0.0]. Try alist + 2 on a plain list: TypeError. Broadcasting is a Series (NumPy) privilege.
pd.Series([1,2,3,4]) + pd.Series([100,200,300]) gives …
ValueError: lengths differNaN4[101.0, 202.0, 303.0, nan] then [101.0, 202.0, 303.0, 4.0]. The dtype silently became float64 — NaN is a float. The method form .add(...).div(...) chains: each method works on the outcome of the previous one.
appl.csv runs 2015–2024 (2 516 days); microsoft.csv runs 2014-12-31 … 2018-02-05 (780 days). Predict how many days of aapl + msft are not NaN.
(aapl + msft).notna().sum() is …
The rule you uncovered: arithmetic aligns by label, not position, then operates. 2 517 labels in the union, 779 matched. A hand-written loop would need a lookup dict, a membership test and a policy for misses — one + folds in all three.
>, <, == return masks; .gt, .lt, .eq, .between are their method twins. Combine with &, |, ~ or np.logical_and.
All three spellings give {'Jan': 300, 'April': 200}. ~s1 flips the mask: {'Feb': 100, 'March': 0}. .between is inclusive by default, so it counts 3.
Central tendency, dispersion, shape — every summary statistic is one method call. But one keyword changes the answer, and you should predict which way.
\[s=\sqrt{\frac{\sum(x_i-\bar x)^2}{n-1}} \qquad \sigma=\sqrt{\frac{\sum(x_i-\mu)^2}{n}}\]
For the same data, how does price.std() compare with price.std(ddof=0)?
ddof only affects .var()ddof must always be givenMean 109.85, median 89.95, mode 0.0 — a right tail up to 2 800 and a spike of free items. The two standard deviations (85.9252 vs 85.9230) differ only in the 4th digit because \(n = 20\,000\); with \(n = 5\) they differ by 12 %.
True counts as 1. If 15 146 of 20 000 prices exceed 50, what does round(price.gt(50).mean(), 3) print?
0.757
The trick you discovered: the mean of a boolean mask is a proportion. It is how you will compute default rates, hit rates and p-values all term. idxmax returns the label of the maximum — row 15511.
agg takes names, lists, and your own functionsSkewness 3.78 and kurtosis 70.1 — a normal distribution has 0 and 0. agg with a list returns a Series of statistics; with a function it returns one number. Three spellings — price.mean(), price.agg("mean"), price.agg(mymean) — same answer. Each method returns a scalar, so ratios come free: a Sharpe ratio is mean / std.
For a discrete or text Series the summary is value_counts() — the frequency table.
2010 leads with 1 796 products, then 2008, 2009, 2007. 61 distinct years — including a 0, which is a missing value in disguise. normalize=True turns counts into shares (9.0 %).
A Series has one dtype. Changing it is explicit — and the most important change, next slide, is text → dates.
After p32 = price.astype(“float32”), which statement is true?
price is now float32 toop32 uses half the memory and price is unchangedp32 holds exactly the same decimals as priceastype raises because prices have decimalshistory.index holds "10/09/2022" …. Before conversion, what is type(history.index[0]).__name__?
str
str → Timestamp. Only after pd.to_datetime can you ask for .year, compare against a date, or slice by month. A date stored as text sorts alphabetically — "10/09" before "2/01". For a Series of dates, the .dt accessor (Sunday, Monday) is the door to those properties.
Select the months of sales with units at least 100 and below 500 into between. Expected: Jan 300, Feb 100, April 200.
NaN, and NaN forces float64..loc uses labels, .iloc positions; a comparison returns a boolean mask you index with.mask.mean() is a proportion; std() divides by \(n-1\) unless you say ddof=0..agg accepts names, lists and your own functions; astype and pd.to_datetime return new Series.Next: §1.2 — if-else without loops, missing values, clipping, ranking and binning.
Every row-by-row loop you are tempted to write has a one-line vectorised replacement. You will time them, predict what where keeps, and discover why the 95th percentile is where outliers get clipped.
Three ways to flag prices above 50. Predict the fastest before timing.
Which is fastest on 20 000 prices?
price.apply(compare)for loop appending to a listprice.gt(50)All three agree (True True); only the last is essentially free — tens of microseconds against milliseconds. Rule: reach for apply only when no vectorised method exists.
where keep?price.where(price.ge(50), other=0) returns …
[0.0, 0.0, 0.0, 100.0, 121.95] — exactly the notebook output — versus [44.99, 24.81, 46.0, 0.0, 0.0] for mask. The "Big" version works but turns the whole Series into object dtype: no more arithmetic.
Labelling prices “Expensive” / “Cheap” is an if-else on every row. Three ways — one slow, two fast.
{'Expensive': 15146, 'Cheap': 4854} all three ways. np.where(cond, if_true, if_false) is the idiom you will reuse most: it reads like the sentence you meant.
The notebook practice downloads a stock with yfinance; here the same AAPL closes come from appl.csv.
ret holds AAPL daily returns 2015–2024. Compute sharpe = mean / std of ret (use .agg or the methods), and win_std = the std of returns on winning days only, using where (not a filter).
NaN is not a number — and not an error. It must be counted, dropped or filled before any model sees it. The notebook’s full inventory file has SoldFlag missing for every Active product (61.8 %). Our 20 000-row sample is all Historical, so we take the notebook’s other dataset, publicCompany, where dividendYield is blank for non-payers.
1 610 of 3 084 companies have no dividendYield. What does round(dy.isna().mean(), 3) print?
0.522
Same trick as before: the mean of isna() is the missing rate, 52.2 %. Aggregations skip NaN by default — convenient, and dangerous if you forgot to look.
dy has 1 474 numbers and 1 610 holes. Predict two things: does .sum() return a number, and what does dy + 1 leave in the holes?
dy.sum() and (dy + 1).isna().sum() give …
NaN and 0 — one hole poisons the sum, but adding 1 fills it48.28 and 1610 — the sum skips holes, the addition keeps themNaN and 1610 — holes poison everything48.28 and 0 — holes are treated as 0 everywhereThe split you found: aggregations skip NaN; element-wise operations propagate it. Same missing value, opposite treatment — and count() (non-missing) is not len().
Assign to dy_clean a version of dy with the missing dividend yields removed (.dropna()). Expected: 1 474 entries, zero NaN.
Strategy, not reflex: for prices, ffill() (“use the last known price”) is often right; for a dividend yield, dropping — or filling with 0 for a confirmed non-payer — depends on what the blank means. The tool follows the meaning of the data.
pd.Series([1, NaN, NaN, 4]).interpolate() fills the gap with …
1, 12, 30, 02.5, 2.5sold is a 0/1 flag with two gaps (the notebook’s SoldFlag for Active products). Fill the gaps with the mode (sold.mode().loc[0]) into filled. Expected: no NaN, and the total is still 1.
price.sort_values() with no arguments — which end is the 0.0 minimum at?
0.0
Ascending by default; the row labels travel with the values ({15511: 2800.0, 4071: 2083.09, 2218: 1580.0} at the top), so a later sort_index() restores label order and forgets the value sort. Sorting returns a new Series — price is untouched.
s.quantile(0.8) == 21 tells you …
Clipping = capping and flooring. The max falls from 2 800 to 271.67, the min rises from 0 to 12.50, and the standard deviation drops from 85.93 to 71.16 — one method call removes the influence of 5 % of the tails on each side.
Clip the 2024 AAPL closes close24 to lie between their 5th and 95th percentiles, into clipped. Expected min ≈ 169.30, max ≈ 247.86.
dlist = [40, 20, 30, 20, 10] has the value 20 at positions 1 and 3.
Which call returns [40, 30, 10] — no 20 at all?
dlist.drop_duplicates()dlist.drop_duplicates(keep="first")dlist.drop_duplicates(keep="last")dlist.drop_duplicates(keep=False)partPrice = price.loc[:10] — the first eleven list prices (.loc is inclusive, so 0 … 10).
The first four prices are 44.99, 24.81, 46.00, 100.00. What are their ranks (1 = smallest) within the eleven?
[3.0, 1.0, 4.0, 7.0]
rank() gives 1 to the smallest (ties get the average rank); rank(pct=True) rescales to \((0, 1]\) — a cross-sectional percentile in one call, the workhorse feature of Project 2. replace takes a list → one value, or a dict of old → new. The result is object dtype the moment a string enters.
rlist holds body weights. Build label: "strong" where weight > 50, else "slim". Use np.where (or .where twice, or .apply).
pd.cut(price, bins=4) on prices from 0 to 2 800 — how full is the first bin?
Equal width: 19 990 of 20 000 in (-2.8, 700]. The notebook’s own edges give [607, 19389, 4, 0] — 607 free items exposed. qcut produces four bins of ≈ 5 000. Binning turns a number into a category — the input to groupby in Chapter 2.
apply hides a loop; .gt, np.where, .where run in C — hundreds of times faster.where(cond, other) keeps where True and never changes length; mask is its mirror.isna().mean() is the missing rate; aggregations skip NaN, element-wise operations propagate it; choose dropna / fillna / ffill / interpolate by what the gap means.sort_values is ascending by default; clip at the 5th/95th percentiles caps and floors the tails.drop_duplicates(keep=…), rank, replace, pd.cut / pd.qcut — all return new Series.Next: §1.3 — naming and re-numbering the index, and the one rule about .loc that bites everyone.
The index is the address book of a Series. You will rename it, reset it, and predict how many rows a .loc slice returns versus an .iloc slice — the answers differ by one.
to_str = lambda v: "ID-" + v. What is aSeries.rename(to_str).index[0]?
'ID-ONE'
A function maps every label; a dict maps only the ones you list; assigning .index replaces the whole thing (and must have the right length).
aep.rename_axis(“FancyName”) changes …
"FancyName""FancyName".reset_index()reset_index() turns a Series into a two-column DataFrame whose first column is named after the index (FancyName, or Datetime). drop=True throws the timestamps away and leaves a plain RangeIndex — useful after filtering, dangerous for time series.
In Colab
energy = isom5650.data.energy() returns the same PJM hourly load table (2004–2018) with Datetime as the index; aep = energy["AEP"].
How many rows do aep.loc[“2017-01-01 01:00”:“2017-01-01 05:00”] and aep.iloc[0:5] return?
Label slice ends at 05:00 (5 rows); position slice ends at 04:00 (positions 0–4). .loc is right-inclusive because you name the end you want; .iloc is right-exclusive because it is Python. iloc[:trainsize] is how you will carve a time-ordered training set.
The notebook slices SPY with loc["2020-01-01":"2020-12-31"] and gets 253 rows. How many rows does sp500.loc["2020"] return?
253
"2020" means the whole year (253 trading days), "2020-03" the whole month (22 days). The March 2020 low was 2 237.40 on the 23rd.
Jan 2020 trading days sit at positions 0 (Jan 2), 1 (Jan 3), 2 (Jan 6), 3 (Jan 7), 4 (Jan 8) …
Do sp500.loc[“2020-01-06”:“2020-01-08”] and sp500.iloc[2:5] return the same rows?
.loc gives 2 rows, .iloc gives 3.iloc includes Jan 9Same three closes (3246.28, 3237.18, 3253.05) — then iloc[2:4] silently drops Jan 8. A backtest that mixes the two rules is off by one day at every boundary, and the tests still pass.
.loc also accepts a boolean mask — or a function that builds one.
Q3 = 147.4; 4 998 products above it, both ways. Passing a function (compare, or a lambda) is handy inside a method chain when the Series has no name yet.
s[0] is dangerousSort the prices descending: top = price.sort_values(ascending=False). Its index is now 15511, 4071, 2218, … — integer labels out of order. Predict top[0].
The discovery: top[0] matched the label 0, not the first element. Bare [ ] guesses, and its guess depends on the index type. Always say .loc or .iloc — one extra word, zero ambiguity.
The notebook asks for MSFT’s worst days; microsoft.csv covers 2015–2018.
From MSFT daily returns r_msft, select the days with a loss worse than 5 % into worst using .loc and a mask. Expected 3 days; the worst is 2015-01-27 at −9.25 %.
You want MSFT days with a return strictly between +2 % and +5 %. Predict what happens if you drop the parentheses.
r[r > 0.02 & r < 0.05] …
TypeError — & binds tighter than >41 days in the band, 83 beyond ±2 %. Two rules you will live by: &, |, ~ — never and, or, not — and parentheses around every comparison.
price.sample(500, replace=True, random_state=0) …
frac is givenfrac=0.002 of 20 000 is 40 rows. With replacement, only 492 of the 500 draws are distinct — 8 products came up twice.
aSeries has labels jan, feb, march. What does aSeries.reindex(["jan", "Tom"]).tolist() print?
[100.0, nan]
reindex conforms the Series to the labels you give — existing ones are selected, new ones get NaN, and the dtype turns float. .loc with a missing label raises KeyError instead. reindex is how you align two calendars.
Add 100 to every price above 50 in the first five products. There is a right way (one .loc) and a wrong way (chained [ ][ ]). Predict the right way’s result.
p = [44.99, 24.81, 46.0, 100.0, 121.95]. After p.loc[p > 50] = p.loc[p > 50] + 100, what is p.tolist()?
[44.99, 24.81, 46.0, 200.0, 221.95]
The rule: any time you assign to a slice, use one .loc. Two brackets in a row return a copy — your write is silently discarded (pandas may warn, may not). This is the most common pandas bug there is.
The notebook practice pairs every Friday close with the following Monday for NVDA; nvda_spy_daily_2023_2024.csv holds two years of NVDA.
nvda.index.day_name() gives the weekday of every trading day. Select the Friday closes into fridays. Expected 102 Fridays. (Bonus: pair each with the following Monday via nvda.shift(-1).)
rename maps labels (function or dict); rename_axis names the index itself; .index = … overwrites.reset_index() moves the index into a column; drop=True discards it..loc[a:b] is right-inclusive, .iloc[a:b] is right-exclusive; "2020" slices the whole year..loc accepts masks and functions — & | ~ with parentheses; bare s[0] guesses label-or-position; sample(replace=True) may repeat rows.reindex selects by label and invents NaN; .loc raises KeyError on unknown labels; one .loc = to write, never chained [ ][ ].Next: §1.4 — dates, shifts, rolling windows, drawdowns and resampling.
A price is a number; a return is a relation between today and yesterday. You will build that relation with shift, smooth it with rolling, track the worst loss with cummax, and change the clock with resample.
weather[“date”].iloc[0] right after pd.read_csv is a …
strTimestampdatetime64int (days since 1970)In Colab
weather = isom5650.data.weather() — the same Delhi daily-climate table (1 462 rows, 2013-01-01 … 2017-01-01).
The data run 2013-01-01 … 2017-01-01. How many rows does dates[dates.dt.is_month_end] have?
48
48 month-ends across 2013–2016 (the index touches 5 calendar years because it ends on 2017-01-01). strftime turns dates back into text in any layout; to_datetime reads them back. Why .dt? Series.year would be ambiguous — an attribute you added, or the year inside the values? The accessor makes it explicit.
Mondays are said to under-perform Fridays. Compare the mean daily AAPL return by weekday.
Compute mon_mean = mean of r_aapl on Mondays and fri_mean = mean on Fridays (use r_aapl.index.day_name() as the mask). Expected: Monday ≈ 0.0023, Friday ≈ 0.0001 — the opposite of the folklore.
shift(1) moves every value one row later. That single move is what turns a price series into a return series. The notebook downloads MSFT with yfinance; microsoft.csv holds the same ticker for 2015–2018.
msft.shift(1).iloc[0] is …
NaN\[r_t=\frac{P_t-P_{t-1}}{P_{t-1}}\] — today minus yesterday, over yesterday. With shift that is one line; pct_change is the same line pre-written.
True — identical. Mean 0.00091, sd 0.0142 per day. The forward return shift(-1) ends in NaN: tomorrow is not known yet, which is exactly why it is the target of a forecasting model, never a feature.
A rolling mean replaces each value by the average of the last \(k\) — a fast window reacts, a slow one filters. Predict how many rows the window costs.
msft.rolling(10).mean() needs ten values before it can average. How many leading NaN?
9
The 40-day line barely notices the dips the 10-day line follows. A fast average crossing above a slow one is the oldest trend signal in finance — you will test it in Project 1.
The 21-day realised volatility peaked at 43 % annualised on 2015-09-18 and averaged 14 % in 2017. rolling(...).std() is the risk measure GARCH will model in Chapter 7.
cumsum, cumprod, cummax carry a running total, product or record high. Predict what the product of gross returns recovers.
(1 + r).cumprod().iloc[-1] equals …
Compounded growth 1.882× — exactly the price ratio: (1 + r).cumprod() is the equity curve of a buy-and-hold backtest, two lines and no loop. The running sum says 0.711, a different (and wrong) number. cummax is the staircase of record highs; cummin never moves once the 40.29 minimum is in.
\[\text{drawdown}_t=\frac{\text{cummax}_t - P_t}{\text{cummax}_t}\] — how far below its record high the price sits. The maximum drawdown is the worst peak-to-trough loss. The notebook asks for ten years of a stock: appl.csv.
Compute the drawdown Series dd for aapl with cummax(). Expected maximum ≈ 0.387 on 2019-01-03.
Resampling changes the clock: daily → monthly is down-sampling (aggregate), minute → hourly likewise; monthly → daily is up-sampling (invent rows). Predict what label a month gets.
msft.resample(“ME”).mean() — what is the first index label?
2015-01-02, the first trading day2015-01-31, the calendar month end2015-01-15, the midpoint2015-01, a month period"W" means weeks ending Sunday. MSFT’s first 2015 row is Friday 2 January. Predict how many observations the first weekly bucket holds.
The first bucket (ending Sunday 2015-01-04) covers one trading day; full weeks hold five. resample is groupby for time — name the bucket ("W", "ME", "5min") and the aggregation (.last, .mean, .sum, .count) — and always check the first and last bucket for partial weeks.
resample("ME").agg("mean") returns one row per month. .transform("mean") returns one row per day, each carrying its month’s mean — so you can subtract it.
38 rows versus 779. The step line is the monthly mean broadcast back to every day — the same idea as groupby().transform in Chapter 2. The rolling mean is the smooth cousin of the same idea.
A US session has 390 minutes, 09:31 … 16:00. How many rows does m["Close"].resample("5min").ohlc() return?
79
Not 78 but 79: bins are anchored on the clock (09:30, 09:35 …), so the first bar holds only 09:31–09:34 and the 16:00 print sits alone in the last one. .ohlc() rebuilds candlesticks at any frequency; .sum() is right for volume, .last() for a close, .mean() for a level. The aggregation must match what the number means.
Three 5-minute closes become eleven 1-minute rows, eight of them empty. ffill holds the last known value (what a trader knew); interpolate draws straight lines (what a chart wants). Up-sampling never adds information — only rows.
Two ways to draw a Series: matplotlib directly, or the .plot family every Series carries. The notebook starts with 5 000 normal draws (mean 10, sd 5) — the one place it simulates.
In plt.hist(x, bins=50, density=True) the bars are scaled so that …
fig.add_subplot(rows, cols, n) places panel n on a grid; every Series.plot.* accepts ax= so pandas draws into the panel you chose. The KDE over the density histogram is the picture of “fat tails” you will test formally in Chapter 2. Try msft.plot.box().
value_counts().plot.bar() / .barh() / .pie() is the whole recipe for a categorical column. Prefer bars: a pie hides that Healthcare (500) and Technology (486) are almost tied.
pd.to_datetime turns text into Timestamps; .dt unlocks day_name, is_month_end, strftime.shift(1) builds yesterday; (P − P.shift(1)) / P.shift(1) is pct_change; shift(−1) is tomorrow — the forecast target.rolling(k) costs \(k-1\) rows; rolling(21).std()·√252 is realised volatility.(1+r).cumprod() compounds to \(P_T/P_0\); (cummax − P)/cummax is drawdown.resample("ME") labels month-ends, "W" Sundays (first bucket may be partial); agg collapses, transform broadcasts, ohlc rebuilds bars; density=True makes area = 1.Next: Chapter 2 — two-dimensional data: DataFrames, groupby, pivot, and the statistics you compute on them.
Warning
In Growth in a Time of Debt (2010), Carmen Reinhart and Kenneth Rogoff reported that countries with public debt above 90 % of GDP averaged −0.1 % real growth — a number cited by the UK Treasury, the European Commission and the US Congress to justify austerity after 2010.
In April 2013 Thomas Herndon, a UMass Amherst graduate student, obtained the spreadsheet. The averaging formula covered rows 30–44 instead of 30–49: Australia, Austria, Belgium, Canada and Denmark were silently excluded. Combined with an unusual country-weighting choice and some omitted years, the corrected figure for the > 90 % bucket was +2.2 % — no cliff at all.
Lesson for this chapter: a slice is a claim about which rows count. .loc[a:b] is right-inclusive, .iloc[a:b] is not; a NaN skipped by mean() is a row you did not average. Print len(), print isna().sum(), and never trust an aggregate whose denominator you have not seen.
Series methods produce numbers; the deliverable is a recommendation. Fill this in from your §1.4 output.
To: Head of Trading, Equity Desk From: <Your name>, quantitative analyst Subject: Proposal to avoid holding AAPL over weekends Date: 2026-09-08
Recommendation: Do not implement a Friday-sell / Monday-buy rule.
Evidence (AAPL 2015–2024, 2 515 daily returns,
appl.csv): - Mean Monday return 0.23 % vs Friday 0.01 % — the opposite sign to the folklore, and 0.2 % per day against a daily sd of 1.8 % is well inside noise. - 22 days lost more than 5 %; the worst, 2020-03-16 (−12.9 %), was a Monday — but Mondays also supplied 4 of the 21 best days. - Round-trip cost of the rule (≈ 0.02 % × 2 × 505 weekends) exceeds any plausible gain.Caveats: a single stock, one decade, no significance test yet (Chapter 2 supplies the t-test). Returns computed from
Close, notAdj Close.Next step: repeat on
sp500.csvandreturns.csv; run the two-sample test from §3.2 before revisiting.
An LLM writes pandas fluently — and confidently mixes versions and conventions. Three habits for this chapter:
resample("M") is deprecated in favour of "ME", fillna(method="ffill") is now .ffill(), and Series[0] on a labelled index no longer means position 0. Old answers still run — with warnings you will ignore until they become errors..loc["a":"c"] inclusive?” gets a correct answer nine times out of ten. The tenth silently drops a month from your backtest. A 5-element Series is a two-second check.apply without asking “is there a vectorised method?” Copilots default to apply(lambda …) because it always works. Ask for the np.where / .where / .clip version and for df.shape printed before and after every filter.| Concept | Tool |
|---|---|
| Build a Series; labels vs positions | pd.Series(data, index=, name=), .loc, .iloc |
| Vectorised arithmetic with alignment | + − × ÷, .add(fill_value=), & \| ~ |
| Aggregation | .mean() .median() .std(ddof=) .quantile() .agg([...]), mask.mean() |
| Conversion | .astype(), pd.to_datetime(), .dt |
| If-else without loops | np.where, .where, .mask |
| Missing data | .isna().mean(), .dropna(), .fillna(), .ffill(), .interpolate() |
| Sorting, clipping, duplicates, ranking, binning | .sort_values(), .clip(), .drop_duplicates(keep=), .rank(), .replace(), pd.cut / pd.qcut |
| Index surgery | .rename(), .rename_axis(), .reset_index(drop=), .reindex(), .sample() |
| Time series | .shift(), .pct_change(), .rolling(k), .cumprod(), .cummax(), .resample("ME") |
| Plotting | plt.hist(density=True), Series.plot.{line,hist,kde,bar,pie}, fig.add_subplot |
series1 + series2 returned NaN where the indexes did not match. Give one business situation where that NaN is exactly what you want, and one where fill_value=0 is the right call..loc["2020-01-01":"2020-12-31"] and .iloc[0:253] returned the same 253 rows for the S&P 500. Write down a case where the two differ by one row and explain which one is the bug..ohlc(); a monthly mean from daily closes with .mean(). Why would .mean() be wrong for volume, and .sum() wrong for a price?Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python