1.1 — Series Basics, Operators, Aggregation and Conversion

Chapter 1 · Data Structures and Methods of Series

Prof. Xuhu Wan

Section 1.1 · Chapter 1 · Learning Statistics with Python

Series Basics, Operators, Aggregation and Conversion

Data Structures and Methods of Series

Prof. Xuhu Wan

ISOM, HKUST Business School · 2026 Edition

Series Basics, Operators, Aggregation and Conversion

A Python list holds numbers but cannot summarise them. A Series can — and it carries a label for every value. You will predict what the index does to arithmetic, and discover why ddof changes your standard deviation.

A list cannot take its own mean — what can?

alist = [300, 100, 0, 200, 1000] is five monthly sales figures. Predict what happens when you ask the list for its mean.

What does alist.mean() do?

• Prints 320.0
• Prints [300, 100, 0, 200, 1000]
• Raises AttributeError — lists have no mean
• Returns None

Wrap it in a Series

Same five numbers, but now sales.mean() returns 320.0. The index is a RangeIndex(start=0, stop=5) — pandas invented positional labels because you gave none.

Three ways to build one

A Series accepts a list, a dict, or a NumPy array. Predict which constructor sets the index for you.

What does pd.Series({"Jan": 300, "Feb": 100}).index print?

pd.Series({"Jan": 300, "Feb": 100}).index

Index(['Jan', 'Feb'], dtype='object')

Dict keys become labels. The NumPy array gives float64, the integer list int64 — the dtype is inferred from the values, and every element of a Series shares it.

Labels or positions?

sales is now labelled Jan … May. There are two doors into it.

Which pair returns the January and May figures?

• sales.loc[0] and sales.loc[4]
• sales.loc["Jan"] and sales.iloc[-1]
• sales.iloc["Jan"] and sales.iloc["May"]
• sales[0] and sales["May"] — both raise

Filter with a mask

Comparing a Series to a number gives a Series of booleans of the same length — a mask. Predict its dtype and what indexing with it returns.

sales.median() is 200. Which labels survive sales[sales > sales.median()]?

sales[sales > sales.median()].index.tolist()

['Jan', 'May']

bool mask, {'Jan': 300, 'May': 1000} survive. Combine masks with & and | (each side in parentheses) — never with Python’s and/or, which cannot handle five answers at once.

Your turn: scores below the mean

scoresheet holds your subject scores. Build low = the scores strictly below the mean (72.5). Expected: only Stats 20.

A real column: 20 000 products from the inventory file

The notebook’s Kaggle sales-analysis file: historical sales for products a retailer must decide to keep or drop. PriceReg is the list price, SoldFlag = 1 if sold in the past six months.

In Colab

url = "https://drive.google.com/uc?export=download&id=1rXXLrU6SRvcqFsASaVOYNT021OMbN30e"
df = pd.read_csv(url)          # 198 917 rows; the course server holds a 20 000-row sample

df["col"] (or df.SoldFlag) hands back a Series named after the column, with the frame’s RangeIndex — 44.99, 24.81, 46.0 ….

Operators broadcast — predict the first value

(ReleaseNumber + 2) / 100 applies to every element at once (broadcasting). No loop.

The first ReleaseNumber is 15. What is the first element of (ReleaseNumber + 2) / 100?

((ReleaseNumber + 2) / 100).iloc[0]

0.17

[0.17, 0.09, 0.02] and revenues [674.85, 173.67, 0.0]. Try alist + 2 on a plain list: TypeError. Broadcasting is a Series (NumPy) privilege.

Two Series of different length — what happens?

pd.Series([1,2,3,4]) + pd.Series([100,200,300]) gives …

• ValueError: lengths differ
• Three values — the extra one is dropped
• Four values — the last one is NaN
• Four values — the last one is 4

[101.0, 202.0, 303.0, nan] then [101.0, 202.0, 303.0, 4.0]. The dtype silently became float64 — NaN is a float. The method form .add(...).div(...) chains: each method works on the outcome of the previous one.

Alignment on real calendars: AAPL + MSFT

appl.csv runs 2015–2024 (2 516 days); microsoft.csv runs 2014-12-31 … 2018-02-05 (780 days). Predict how many days of aapl + msft are not NaN.

(aapl + msft).notna().sum() is …

• 2 516 — the longer calendar wins
• 779 — only the dates in both files
• 780 — the shorter calendar wins
• 0 — the calendars never line up exactly

The rule you uncovered: arithmetic aligns by label, not position, then operates. 2 517 labels in the union, 779 matched. A hand-written loop would need a lookup dict, a membership test and a policy for misses — one + folds in all three.

Logical operators

>, <, == return masks; .gt, .lt, .eq, .between are their method twins. Combine with &, |, ~ or np.logical_and.

All three spellings give {'Jan': 300, 'April': 200}. ~s1 flips the mask: {'Feb': 100, 'March': 0}. .between is inclusive by default, so it counts 3.

Sample or population standard deviation?

Central tendency, dispersion, shape — every summary statistic is one method call. But one keyword changes the answer, and you should predict which way.

\[s=\sqrt{\frac{\sum(x_i-\bar x)^2}{n-1}} \qquad \sigma=\sqrt{\frac{\sum(x_i-\mu)^2}{n}}\]

For the same data, how does price.std() compare with price.std(ddof=0)?

• Slightly larger — pandas divides by \(n-1\) by default
• Slightly smaller — pandas divides by \(n\) by default
• Identical — ddof only affects .var()
• Undefined — ddof must always be given

Mean 109.85, median 89.95, mode 0.0 — a right tail up to 2 800 and a spike of free items. The two standard deviations (85.9252 vs 85.9230) differ only in the 4th digit because \(n = 20\,000\); with \(n = 5\) they differ by 12 %.

Sum a mask, or take its mean?

True counts as 1. If 15 146 of 20 000 prices exceed 50, what does round(price.gt(50).mean(), 3) print?

round(price.gt(50).mean(), 3)

0.757

The trick you discovered: the mean of a boolean mask is a proportion. It is how you will compute default rates, hit rates and p-values all term. idxmax returns the label of the maximum — row 15511.

agg takes names, lists, and your own functions

Skewness 3.78 and kurtosis 70.1 — a normal distribution has 0 and 0. agg with a list returns a Series of statistics; with a function it returns one number. Three spellings — price.mean(), price.agg("mean"), price.agg(mymean) — same answer. Each method returns a scalar, so ratios come free: a Sharpe ratio is mean / std.

Counting categories

For a discrete or text Series the summary is value_counts() — the frequency table.

2010 leads with 1 796 products, then 2008, 2009, 2007. 61 distinct years — including a 0, which is a missing value in disguise. normalize=True turns counts into shares (9.0 %).

What does astype do to memory?

A Series has one dtype. Changing it is explicit — and the most important change, next slide, is text → dates.

After p32 = price.astype(“float32”), which statement is true?

• price is now float32 too
• p32 uses half the memory and price is unchanged
• p32 holds exactly the same decimals as price
• astype raises because prices have decimals

Text dates are just strings — until you convert

history.index holds "10/09/2022" …. Before conversion, what is type(history.index[0]).__name__?

type(history.index[0]).__name__

str

str → Timestamp. Only after pd.to_datetime can you ask for .year, compare against a date, or slice by month. A date stored as text sorts alphabetically — "10/09" before "2/01". For a Series of dates, the .dt accessor (Sunday, Monday) is the door to those properties.

Your turn: a logical operator

Select the months of sales with units at least 100 and below 500 into between. Expected: Jan 300, Feb 100, April 200.

What you discovered

• A Series = values + index + one dtype; lists have none of the three.

• Arithmetic aligns on the index first; unmatched labels become NaN, and NaN forces float64.

• .loc uses labels, .iloc positions; a comparison returns a boolean mask you index with.

• mask.mean() is a proportion; std() divides by \(n-1\) unless you say ddof=0.

• .agg accepts names, lists and your own functions; astype and pd.to_datetime return new Series.

Next: §1.2 — if-else without loops, missing values, clipping, ranking and binning.