2.4 — Method Chaining and Pandas Idioms

Chapter 2 · DataFrames

Prof. Xuhu Wan

Section 2.4 · Chapter 2 · Learning Statistics with Python

Method Chaining and Pandas Idioms

DataFrames

Prof. Xuhu Wan

ISOM, HKUST Business School · 2026 Edition

Method Chaining and Pandas Idioms

Most pandas methods return a new DataFrame. That one fact lets you chain them with . — a pipeline, like a Unix pipe. You will check that a chain and a pile of temporary variables give the same answer on the five-stock panel, provoke the most common pandas bug, and replace a slow loop.

Spaghetti vs chain — same result?

Drop the index (GSPC), flag up-days, then per stock count days, average the daily change and count up-days. Once with temporary variables, once as a single chain.

The temporary-variable version and the chained version produce…

  • Identical output — chaining is the same steps without temp names
  • Different output — chaining drops a step
  • An error — you cannot chain groupby after assign
  • The same rows but different column order

assign does not mutate

assign adds columns and returns a new frame. Predict: after new = msft.assign(ret=…), does the original msft gain a ret column?

After new = msft.assign(ret=lambda d: d.Close.pct_change()), the original msft…

  • Now has a ret column too (assign mutates in place)
  • Is unchanged — assign returns a new frame
  • Is deleted
  • Raises a SettingWithCopyWarning

Worked example: daily prices → a monthly summary in one chain

Label each AAPL row by month, group, summarise (last price, trading days, volatility of the daily change), flag the volatile months. Predict how many months January 2007 – February 2016 contains.

Nine full years plus January and February 2016. How many rows does monthly have?

len(monthly)

110

Five verbs, one chain — filter, label, group, summarise, flag. Wrap it in a function and you have a reusable pipeline; pipe(f, …) lets you drop that function into the middle of another chain.

Why the chain wins

  • No temporary variables to misname or leave stale.
  • Linear, top-to-bottom — each line is one transformation.
  • Easy to comment out a step while exploring.
  • Composable — wrap it in a function; insert it anywhere with .pipe.

Tip

The opening ( lets each .method(...) sit on its own line, PEP 8-style. Always wrap a multi-line chain in (...). The chain-friendly verbs: query, assign, pipe, loc.

The SettingWithCopy bug

You want a regime label wherever Close > 80. Two ways to write it — predict which one lands the write on the original frame.

Which write actually changes px?

  • sub = px[px.Close > 80]; sub["regime"] = "high"
  • px.loc[px.Close > 80, "regime"] = "high"
  • Both
  • Neither — DataFrames are immutable

Write it with one .loc[rows, cols] =. The wrong way silently evaporates and you only get a warning — the most common pandas bug there is.

Fix the slow loop

iterrows walks the frame one Python row at a time. On 11 525 rows it is already sluggish; on a million it is minutes. Replace it with one vectorised line.

The loop was cut to the first 100 rows because it is slow. Replace it with the vectorised product long["value"] * long["change"] / 100 over all rows, into dollar_move (the daily dollar move per share).

Both produce the same column; the vectorised line runs in C, about 100× faster and half the code. iterrows is almost always a code smell.

What you discovered

  • A method chain (df.query().assign().groupby().agg()) gives the same result as temporary variables — with nothing to misname.
  • assign is non-mutating and later columns can reference earlier ones — that is why chains are safe.
  • query / assign / pipe / loc are the four chain-friendly verbs; pipe inserts your own function.
  • Write with one .loc[rows, cols] = — never df[mask][col] =, which writes to a copy.
  • Vectorise first; np.where / pd.cut next; apply only when no built-in works; iterrows almost never.

Next: Chapter 3 — the frames are clean; now the statistics: which distribution, which test, which measure of association, and how bad the worst day can be.

Working with an AI Copilot

Three prompts that make an LLM useful for this chapter — and the pitfall each one guards against.

  1. “Before you aggregate, print df.index.duplicated().sum() and df.isna().sum(); if either is non-zero, stop and show me the offending rows.” A copilot will happily groupby a frame with a doubled 02:00 hour and a 99 %-empty column and report a clean-looking table.
  2. “Show me the .shape after every load, reshape or merge (read_csv, pivot_table, melt, unstack, concat, pd.merge) and explain each change in row count.” The chapter’s discipline in one instruction: 2 305 × 5 → 11 525 × 3 is either a melt or a bug; 401 → 380 after a merge is either an inner join or lost data; a file that always has exactly 65 536 rows is a truncation.
  3. “Write every modification as one .loc[rows, cols] = or as an assign inside a chain — never df[mask][col] = … — and tell me if pandas would raise SettingWithCopyWarning.” The copy-versus-view bug runs without error and leaves the original frame unchanged; an LLM that reproduces the idiom from old tutorials will reproduce the bug.

Mistakes Library: Public Health England’s missing 15 841 cases (October 2020)

Warning

On 4 October 2020 Public Health England announced that 15 841 positive COVID-19 tests taken between 25 September and 2 October had been left out of the United Kingdom’s daily case counts. Commercial laboratories sent results as CSV files; an automated pipeline loaded each file into a legacy Excel .xls template before it entered the national dashboard. The .xls format caps a sheet at 65 536 rows, and the way each result was laid out meant a template filled after about 1 400 cases. Every row beyond the cap was silently dropped — no error, a plausible-looking total every day for a week.

The consequence was not a wrong chart. The contacts of those 15 841 people — an estimated 48 000 — were not traced for up to eight days during the autumn wave, and the reported seven-day incidence for England was understated by roughly a fifth. The fix was a .xlsx template and a split of the incoming files; the real fix would have been a single line comparing the row count after the load with the row count in the source file.

Lesson for this chapter: a load, a concat and a merge each have a row count you can predict. Print it, assert it, and treat a count that never changes as a symptom. A pipeline that runs without error has proved only that it ran.

Decision Memo — Which utilities enter the 2011–2012 load model, and how

To: Head of Forecasting, grid analytics From: <Your name>, data engineering Subject: Handling of missing and duplicated hours in AEP_hourly.csv / pjm_hourly.csv before modelling Date: 2026-09-15

Recommendation: Model AEP, COMED and FE only; drop NI. Forward-fill the two single-hour holes; drop the rows of FE’s five-month gap rather than fill them; keep both copies of each November 02:00 hour with a DST flag and average them only when an hourly profile is needed.

Evidence: - load.isna().sum() on 2011–2012 (17 540 hours): AEP and COMED each miss 1 hour, FE misses 3 624 (January–May 2011), NI is present in 1 hour of 17 540. A plain dropna() keeps 0 rows; dropna(subset=["FE"]) keeps 13,916. - Filling FE’s gap with its median keeps every row but cuts the column’s standard deviation from 1,422.5 to 1,267.7 MW: a fifth of the series would sit at one value, understating the variance any risk or forecast model is built on. - bfill on a time-ordered frame copies a future hour into the past — look-ahead bias; ffill is the only defensible one-hour fill. - pjm_hourly.csv 2014–2017 has no NaN but 4 duplicated labels, all 02:00 on the first Sunday of November: two real loads, one clock label. keep="first" or "last" discards a real hour; keep=False discards two.

Caveats: - Dropping FE’s gap removes January–May 2011 for all three utilities if the model needs aligned rows; a per-utility model keeps 17 539 rows for AEP and COMED. - The 02:00 decision changes a groupby("hour").mean() by one part in 1 460 per year — immaterial for the profile, material for a peak-hour audit.

Next step: Add an assert on the row count and on index.duplicated().sum() at the end of the loading chain, so that the next file with a hole or a doubled hour fails loudly instead of averaging quietly.

Chapter Summary

Concept Tool
Axes, sorting, alignment df.axes, sum(axis=), sort_values, sort_index, df1 + df2
Aggregation agg([...]), agg({col: [...]}), agg(name=(col, fn)), apply(f, axis=)
Missing and duplicates isna().sum(), fillna, ffill, dropna(subset=), index.duplicated, drop_duplicates(keep=)
Filtering and subsets (a) & (b), query, isin, loc[mask, cols], iloc
Reshaping and joining groupby().agg/transform/filter, pd.merge(how=), pivot_table, crosstab, melt, stack/unstack, pd.concat, pd.cut
Idioms (df.query().assign().groupby().agg()), pipe, one .loc[rows, cols] =, vectorise over iterrows

Next: Chapter 3 — Reshaping Statistics.

Discussion Questions

  1. fillna(median) kept every row of FE but cut its standard deviation. For which downstream tasks is that harmless (a plot? a regression? a VaR?) and for which is it a material error?
  2. The duplicated 02:00 hour is real data from a real clock change. Should you drop one copy, average the two, or keep both with a flag? What does each choice do to a groupby("hour").mean()?
  3. A left merge of the 2015 rankings onto 2016 left some research_16 cells NaN. In a model of “did the university rise?”, is that NaN missing data, or is it the answer?
  4. The Public Health England pipeline ran without error for a week and produced a plausible dashboard every day. Which single line from this chapter — a .shape print, an assert on a row count, a duplicated().sum() — would have caught it, and where in a method chain would you put it?

Prof. Xuhu Wan · HKUST ISOM · Learning Statistics with Python