Chapter 1 · Learning Statistics with Python
Understand columns with an index, calculate and clean values, preserve identity, then build and explain time-series features.
Prof. Xuhu Wan
ISOM, HKUST Business School · 2026 Edition
Each section ends with a concept recap. Chapter 2 combines the column foundations into complete tables.
| Section | Concept | Tool |
|---|---|---|
| 1.1 Series Foundations | Structures, then arithmetic, then summaries and conversion | pd.Series, .shape |
| 1.2 Value Transformations and Cleaning | Change values and diagnose missingness | .where, .fillna |
| 1.3 Selection and Index Alignment | Select observations and preserve index values | .loc, .align |
| 1.4 Time-series Features and Resampling | Order dates, calculate features and summarise calendar bins | .shift, .rolling |
| 1.5 Figures and Statistical Charts | Choose and explain a statistical chart | plt.subplots, .plot |
| Part | Question |
|---|---|
| 1.1.1 pandas structures and indexed observations | What does one observation represent? |
| 1.1.2 Arithmetic, broadcasting and alignment | Which values will pandas combine? |
| 1.1.3 Summaries, masks and type conversion | What does the summary count? |
Question: What does one observation represent?
Result: A Series with explicit index values, units and dtype.
Chapter 0 supplied Python objects. pandas adds indexed observations.
pandas is a Python library for working with indexed data: read a file, inspect its columns, clean values, calculate features and summarise results.
pd is the conventional short name for the library. pd.Series(...) and pd.DataFrame(...) construct its two main data structures.
| Structure | What it represents | Example |
|---|---|---|
| Series | One-dimensional values with index values | One product-price column |
| DataFrame | A two-dimensional table with row and column names | Product identifiers and prices together |
Chapter 1 develops columns with an index; Chapter 2 develops table operations.
df names the table. index = index values; columns = column names; shape = dimensions. head(n) previews its first n rows.
Each row is one shop. Select units; follow its index, name and shape.
pd.read_csv reads a CSV file; usecols chooses its columns. A column name in square brackets selects one Series.
DataFrame: three rows, two columns
Series: one column, same index
The two outputs have different structures. type(obj).__name__ reports the structure’s name; .shape reports its dimensions.
The table has shape (3, 2); the selected Series has shape (3,) and values 44.99, 24.81, 46.00. Selection keeps index values 0, 1, 2. The column name becomes the Series name, PriceReg.
df["PriceReg"] returns a Series. df[["PriceReg"]] passes a list of column names and retains a DataFrame.
With the three-row table above, what does intro_table[[“PriceReg”]] return?
(3,)(3, 1)A Series has a .name and a one-dimensional .shape. A single-column DataFrame still has rows and columns.
.index.equals(...) checks whether two indexes have the same values in the same order.
Series: name and shape
DataFrame: one column
The Series has name PriceReg and shape (3,). The DataFrame has shape (3, 1). Both retain index values 0, 1, 2.
A Series combines values, index values and a dtype. Its name describes the column.
| Position | Index value | Value |
|---|---|---|
| 0 | Jan | 300 |
| 1 | Feb | 100 |
| 2 | March | 0 |
| 3 | April | 200 |
| 4 | May | 1000 |
name="units", dtype=int64. Positions describe display order; index values identify observations. The recorded zero in March is a value, not a missing observation.
A DataFrame is a table of columns with an index. Selecting one column gives a Series; full table operations arrive in Chapter 2.
pd.Series(values, index=months, name="units") creates a column with an index.
alist = [300, 100, 0, 200, 1000] is five monthly sales figures. Predict what happens when you ask the list for its mean.
What does alist.mean() do?
320.0[300, 100, 0, 200, 1000]AttributeError — lists have no meanNoneSame five numbers, but now sales.mean() returns 320.0. The index is a RangeIndex(start=0, stop=5) — pandas invented positional index values because you gave none.
| Tool | Purpose and return value |
|---|---|
arange |
np.arange(5) constructs positions 0–4; its stop is excluded. |
A Series accepts a list, a dict, or a NumPy array. Predict which constructor sets the index for you.
Before running: Read the three constructors below. Which index and dtype will each Series have?
Dictionary entry names become index values. Values infer one dtype: float64 for the array here, int64 for the integer list.
An index value identifies an entry. A position describes its current order, counting from zero. pandas provides two explicit selectors:
| Selector | Supply | Example | Result |
|---|---|---|---|
.loc[...] |
An index value | sales.loc["Jan"] |
300 units |
.iloc[...] |
An integer position | sales.iloc[0] |
300 units |
.iloc[...] |
A position counted from the end | sales.iloc[-1] |
1000 units |
sales.loc[0] looks for index value 0; it does not mean “first entry”. This Series uses months as its index and has no index value 0.
For Series selection, always use .loc[...] for index values or .iloc[...] for positions. DataFrame column selection still uses df["column"]. Use square brackets after each selector.
For an ordered slice, .loc["Jan":"March"] includes both end index values. .iloc[0:3] keeps positions 0, 1 and 2, excluding stop position 3.
The outputs are 300, 300, 1000, 1000. Each scalar selection returns one value; selecting a list of index values, such as sales.loc[["Jan", "May"]], returns a Series with an index.
Both slices select the first three entries: .loc["Jan":"March"] includes March; .iloc[0:3] includes positions 0, 1, 2, excluding 3. §1.3 develops these rules.
sales has the index Jan … May. Select by index with .loc or by position with .iloc.
Which pair returns the January and May figures?
sales.loc[0] and sales.loc[4]sales.loc["Jan"] and sales.iloc[-1]sales.iloc["Jan"] and sales.iloc["May"]sales.iloc[1] and sales.iloc[-1]The observation unit tells you what each index value represents. Both datasets can be a Series.
| Time-series data | Cross-sectional data | |
|---|---|---|
| Hold fixed | One entity: AAPL | One date: 5 February 2018 |
| Vary | Date: 1 Feb → 2 Feb → 5 Feb | Entity: AAPL ↔︎ MSFT |
| Index | Trading date | Ticker |
| Ask | How did this stock change over time? | How do these stocks compare on this date? |
An ordered history supports changes and rolling windows. A dated cross-section supports peer comparisons and ranks. Observations in either dataset need not be independent.
index_col="Date" makes Date the index; parse_dates=True reads it as dates. sort_index() puts the dates in chronological order.
1. Read and preview the data
2. Use dates as the index
Date moves from a column to the index. The next example selects Close as a Series.
| Tool | Purpose and return value |
|---|---|
sort_index |
Return entries ordered by index values; dates must first be parsed. |
Read the supplied loading step, then inspect the resulting Series. These are historical catalogue values, not live quotes.
The three stored closes are approximately 41.945, 40.125 and 39.123 USD. Every row concerns AAPL; the date changes. The weekend adds no observed trading row. Sorting chronologically is necessary before computing changes (§1.4).
Each column is a stock; .loc[comparison_date] selects one date as a Series.
1. Prices from the comparison date
2. One date → one Series
Approximately AAPL 39.123; MSFT 88.000 USD on the same date. The MSFT file ends here; later NaN entries mean missing observations. Different share units make price levels unsuitable for declaring which stock is cheaper or better. A repeated set of dated cross-sections is panel data, developed in Chapter 2.
cross_section is ordered AAPL, MSFT on 5 February. If you divide MSFT’s value by the preceding row’s value, what are you comparing?
State entity, date, variable and units. Shifting across ticker index values compares entities, not earlier dates.
Question: Which values will pandas combine?
Result: A calculation aligned by index with checked matches and missing partners.
Once values and index values are clear, examine how arithmetic pairs observations.
The notebook’s Kaggle sales-analysis file: historical product records that can support an inventory review. PriceReg is the list price, SoldFlag = 1 if sold in the past six months.
df["col"] (or df.SoldFlag) hands back a Series named after the column, with the frame’s RangeIndex — 44.99, 24.81, 46.0 ….
A scalar is one number. Multiplying price by 0.9 applies the same 10% discount to every price.
1. Original prices
2. Apply the same rule to every value
The values change from 44.99, 24.81 and 46.00 to 40.491, 22.329 and 41.4. The index remains 0, 1, 2; the result is still a Series. This is an illustrative discount, not an observed sale.
With two Series, multiplication matches the index values. .sum() then adds the resulting line values to obtain the order total.
1. Unit price
2. Quantity
3. Line value and total
A: 10 × 2 = 20; B: 20 × 1 = 20; C: 30 × 3 = 90. The total is 130. Each result stays attached to its product index.
Match the index first, then add the values.
The result keeps all four index values. At index 3, the right-hand Series has no observation, so the sum is NaN.
The right input is deliberately in a different order. Match index values before adding.
.add(...) performs addition with an explicit missing-partner policy. Divide the resulting Series by 10 using / 10; the index values are retained.
pd.Series([1,2,3,4]) + pd.Series([100,200,300]) gives …
ValueError: lengths differNaN4The outputs are [101.0, 202.0, 303.0, nan] and [101.0, 202.0, 303.0, 4.0]. Here float64 represents missingness; nullable integers differ. .add chooses the missing-partner policy before division scales values.
| Tool | Purpose and return value |
|---|---|
notna |
Return a Boolean Series: True where a value is present. |
isna |
Return a Boolean Series: True where a value is missing. |
dropna |
Return entries with missing values removed; length can shrink. |
appl.csv runs 2015–2024 (2 516 days); microsoft.csv runs 2014-12-31 … 2018-02-05 (780 days). Only shared dates can have an observed sum.
Before running: Read the two date indexes. Which dates can produce a sum, and which must remain missing? Run the code to inspect and count them.
Arithmetic aligns index values before calculating. The union has 2 517 dates; 779 match. The unmatched dates require a missing-value policy.
s.index.is_unique is a Boolean attribute—no parentheses. Check uniqueness before alignment; matching duplicate index values can produce multiple pairs.
Alignment matches index values, including repeated ones. Predict how many pairs share index value A before adding.
Two left observations match three right observations: six results, [11, 21, 31, 12, 22, 32]. Check that index values are unique before interpreting aligned arithmetic.
For these unequal indexes, each left A matches all three right A entries.
Combine parenthesised masks with & (both), | (either) and ~ (reverse): (s > 100) & (s < 500). Python’s and/or require one truth value.
| Tool | Purpose and return value |
|---|---|
between |
Return a mask for a value interval: s.between(a, b, inclusive=“both”). |
>, <, == return Boolean Series. Combine with & (AND), | (OR) and ~ (NOT). Put each comparison in parentheses.
The AND mask gives {'Jan': 300, 'April': 200}. ~s1 flips the mask: {'Feb': 100, 'March': 0}. .between is inclusive by default, so it counts 3.
Question: What does the summary count?
Result: An interpreted statistic and a record of parsing failures.
After calculating indexed values, reduce or select them and interpret the denominator.
The mean is the observed sum divided by the observed count. The median is the middle sorted value, or the average of the two middle values.
For [300, 100, 0, 200, 1000], mean = 320, median = 200. The large 1000 raises the mean more.
sales.median() returns one number. sales > sales.median() returns a Boolean Series. sales.loc[mask] retains entries marked True. State whether your summary describes all observations or only the selected sample.
Read a line from the object outwards. Each operation receives the result of the previous one.
| Tool | Purpose and return value |
|---|---|
median |
Return the middle sorted observed value (average two middle values for even n). |
Comparing a Series to a number gives a Series of booleans of the same length — a mask. Predict its dtype and what indexing with it returns.
Before running: Read the condition below. Which months will remain in the filtered Series?
The bool mask retains Jan = 300 and May = 1000. Use & and | with parenthesised comparisons.
Compare the original scores with the Series of scores strictly below the mean.
1. Original scores
2. Select scores below the mean
The mean is 72.5. Only Stats 20 satisfies the condition; .loc keeps its index and the Series name Score.
| Method | Meaning |
|---|---|
.min(), .max() |
Smallest and largest observed values |
.mode() |
Most frequent value(s), returned as a Series |
.quantile(q) |
Cut point at fraction q of the sorted sample |
.var(ddof=1) |
Average squared deviation with divisor n−1 |
.std(ddof=1) |
Square root of variance, in the original units |
The range is max minus min; the IQR is the 75th minus 25th percentile. Quantiles can interpolate between observations; ties mean exact fractions need not lie below a cut point. Missing values are excluded by default.
For illustrative prices 10, 20, 30, the mean is 20. Deviations are −10, 0, 10; squared deviations sum to 200. Sample variance is 200 / (3 − 1) = 100, so sample SD is 10 in the original price units.
Dividing 200 by 3 gives variance 66.667 and SD 8.165. ddof changes the divisor; SD retains the measurement’s units.
Central tendency, dispersion, shape — every summary statistic is one method call. But one keyword changes the answer, and you should predict which way.
\[s=\sqrt{\frac{\sum(x_i-\bar x)^2}{n-1}} \qquad \sigma=\sqrt{\frac{\sum(x_i-\mu)^2}{n}}\]
For the same data, how does price.std() compare with price.std(ddof=0)?
ddof only affects .var()ddof must always be givenMean 109.85, median 89.95, mode 0.0 — a right tail up to 2 800 and 607 recorded zero prices; zero alone does not establish that a product is free. The two standard deviations (85.9252 vs 85.9230) differ only in the 4th digit because \(n = 20\,000\); with \(n = 5\) they differ by 12 %.
A Boolean mask’s .sum() counts True entries; .mean() gives their share. Float-NaN comparisons are normally False. For observed-only shares, select s.loc[s.notna()] first.
| Tool | Purpose and return value |
|---|---|
agg |
Apply named or custom summaries; one summary gives a scalar, a list gives named statistics. |
idxmax |
Return the index value of the first maximum, rather than its value. |
Before running: Read the Boolean comparison. What do its sum and mean measure?
A mask mean is a proportion, used for rates and probabilities. idxmax returns the maximum’s index value: row 15511.
For illustrative prices 10, missing, 100, there is one price above 50 and two observed prices. Decide whether the denominator is all records or observed prices.
The float comparison marks the missing price False, giving 1/3. Selecting observed prices gives 1/2. Report the unknown row separately.
Use price.mean() for one statistic and price.agg(["mean", "std"]) for a named collection of two summaries. describe() bundles count, mean, standard deviation, extrema and quartiles.
The IQR is the 75th minus 25th percentile. The coefficient of variation is standard deviation divided by mean; it becomes unstable near a zero mean.
skew() describes asymmetry; kurt() reports excess kurtosis relative to the normal reference of zero. The summaries describe this sample; Chapter 3 develops distributional reasoning.
Mean 109.852, SD 85.925, IQR 95.41 and coefficient of variation 0.7822 describe different aspects of the same prices. Skewness 3.777 and excess kurtosis 70.075 flag strong asymmetry and tail concentration.
s.agg(function) passes the whole Series to a named summary function. Use it when the same custom statistic belongs in several grouped summaries; Chapter 2 develops that use.
The answer remains 0.7822. A custom function is an alternative implementation of the statistic already taught.
| Tool | Purpose and return value |
|---|---|
value_counts |
Count observed categories; normalize=True gives shares among counted entries. |
nunique |
Count distinct observed values; missing values are excluded by default. |
For a discrete or text Series the summary is value_counts() — the frequency table.
2010 leads with 1 796 products, then 2008, 2009, 2007. 61 distinct years — including a 0, which is a missing value in disguise. normalize=True turns counts into shares (9.0 %).
Preserve raw records; invalid dates become NaT.
The converted values sum to 6. The original still contains text.
pd.to_datetime() parses date text. Here format="%m/%d/%Y" declares month/day/four-digit year.
1. Before conversion: text index
2. Convert the index to dates
The index changes from text (object) to dates (datetime64[ns]). The four values and their order stay the same. §1.3 uses the parsed dates for selection; §1.4 develops calendar features.
| Tool | Purpose and return value |
|---|---|
copy |
Make a separate data object before intentionally damaging or editing an example. |
to_numeric |
Parse numeric text; errors=“coerce” creates missing values for invalid text. |
count |
Count non-missing values; len(s) instead counts all entries. |
A conversion can create missing values. This deliberately damaged copy of four inventory prices makes the failure visible. Predict how many values remain numeric after parsing.
Before running: Read the damaged text values. Which entry will become missing after conversion?
errors="coerce" gives one missing value and three numbers. Count newly missing values separately from values that were already missing. Use errors="raise" when invalid input must stop the pipeline.
| Tool | Purpose and return value |
|---|---|
fillna |
Replace missing entries with a declared scalar or aligned values. |
SKU numbers are identifiers, not measurements. The capital-I dtype Int64 retains whole-number identifiers together with pd.NA. The missing ID below is introduced only to demonstrate the representation.
Int64 retains integer IDs with missing values. A comparison with pd.NA may remain unknown. Filling a mask with False excludes unknown IDs; it does not replace them with zero.
Select the months of sales with units at least 100 and below 500 into between. Expected: Jan 300, Feb 100, April 200.
.shape reports axis sizes..loc selects index values and .iloc counts positions. Arithmetic matches index values.Next: §1.2.
| Part | Question |
|---|---|
| 1.2.1 Conditional transformations and lookups | Should values change, or should rows disappear? |
| 1.2.2 Missing values and bounded repairs | What evidence supports a replacement? |
| 1.2.3 Ordering, ranks and bins | How should observations be compared or categorised? |
Question: Should values change, or should rows disappear?
Result: A transformed Series whose index values and length have been checked.
The summary described the sample. Now choose whether to transform values or select rows.
One comparison produces 20,000 Boolean answers; 15,146 are True.
A Boolean mask gives one True/False answer per index value.
| Expression | Result |
|---|---|
s.loc[condition] |
Keep only qualifying entries; length can shrink |
s.where(condition, other=0) |
Keep values where True; replace the rest with zero |
s.mask(condition, other=0) |
Replace values where True; keep the rest |
where and mask preserve the index and length. Without other=, replacement is missing. Choose between row selection and value replacement before writing the condition.
Before running: Which operations keep all three index values? Compare the printed Series.
Selection returns two rows; both replacement methods return three. Change the threshold to 70 and predict which index values and values change.
where keep?| Tool | Purpose and return value |
|---|---|
where |
Keep values where a condition is True; replace the others, with missing by default. |
mask |
Replace values where a condition is True; keep the others. |
price.where((price >= 50), other=0) returns …
where gives [0.0, 0.0, 0.0, 100.0, 121.95]; mask gives [44.99, 24.81, 46.0, 0.0, 0.0]. Replacing numbers with "Big" changes dtype to object.
np.where(condition, if_true, if_false) chooses one output per input position and returns an array. pd.Series(..., index=price.index) restores the original index values.
There are 15,146 Expensive and 4,854 Cheap index values. Series where keeps existing values where True; NumPy where chooses between the two supplied alternatives.
For prices in chronological order, the observed price return is
\[r_t = P_t/P_{t-1}-1.\]
pct_change(fill_method=None) compares with the previous row, retains index values and leaves gaps unfilled. It returns a fraction: 0.01 = 1%. The first return is missing.
The exercise’s daily return-to-risk ratio is mean / SD, with zero risk-free return. It is not annualised and excludes dividends and costs.
Illustrative history: one stock, four successive observed dates; prices in currency units are 100, 110, 99, 108.9. Returns are missing, 0.10, −0.10, 0.10: 110 / 100 − 1 = 0.10; 99 / 110 − 1 = −0.10.
where retains four index values; std() uses two observed winning returns. With fewer than two observations, sample SD is missing.
| Tool | Purpose and return value |
|---|---|
pct_change |
Return previous-row fractional changes, P_t/P_(t−1)−1; fill_method=None keeps gaps. 0.01 means 1%. |
Use the supplied historical AAPL closes from appl.csv so every team works with the same observations.
ret holds AAPL daily returns 2015–2024. Compute sharpe = mean / std of ret (use .mean() and .std()), and win_std = the std of returns on winning days only, using where (not a filter).
If the check fails: Check the selected observations and the sample-SD or mean calculation before comparing the two reported numbers.
map can also apply a function to each value.
Context: one company per ticker index value; sector names are categories, not numbers. The lookup adds information about each company’s sector.
| Tool | Purpose and return value |
|---|---|
map |
Look up each value in a dictionary, or transform it with a function; keep input index values. |
replace |
Recode specified values; keep uncovered values unchanged. |
This incomplete sector dictionary covers two sectors among four companies. Predict an uncovered value under lookup (map) and recoding (replace).
Before running: Read the lookup dictionary. What happens to a sector with no matching entry?
map produces ["FIN", NaN, NaN, "HLT"] and preserves the caller’s index. replace leaves uncovered sector names unchanged. Check unmatched lookup values before using the result as a feature.
| Tool | Purpose and return value |
|---|---|
isin |
Return a mask testing membership in a supplied set/list of values. |
upper |
With .str.upper(), return upper-case text for each value. |
strip |
With .str.strip(), remove leading and trailing whitespace from each text value. |
all |
Reduce checks to one Boolean: did every retained check pass? Handle unknown checks explicitly. |
Use isin to ask whether each value belongs to a permitted set. Clean whitespace and case before matching text; do not infer a permanent security identity by stripping parts of a ticker.
Selected index values: SI, CCCC. All four cleaned ticker strings match. .all() and .any() reduce checks to scalars; & and | combine them element-wise.
For the scalar function used here, price.apply(compare) calls compare(value) for each price and keeps one returned result per input index value.
price > 50 applies the comparison to every price and returns a Boolean Series with the original index. A built-in numerical operation usually avoids repeated Python function calls.
Timing measures this computer and run, not a universal speed ratio. First verify that the results agree; then compare runtime. Prefer the clearest suitable built-in method.
| Tool | Purpose and return value |
|---|---|
perf_counter |
time.perf_counter() reads a high-resolution clock; subtract two readings for elapsed seconds. |
apply |
For this scalar function, call it on each value and retain its returned result and input index value. |
All three agree (True True); the recorded times compare three equal-result implementations on this run. Rule: reach for apply only when no vectorised method exists.
Question: What evidence supports a replacement?
Result: Missingness counts, a stated repair policy and a valid denominator.
Conditional replacement raises the next question: what does an unknown value mean?
Before running: Inspect the Series preview. What do isna().sum() and isna().mean() measure?
Same trick as before: the mean of isna() is the missing rate, 52.2 %. Aggregations skip NaN by default — convenient, and dangerous if you forgot to look.
dy has 1 474 numbers and 1 610 holes. Predict two things: does .sum() return a number, and what does dy + 1 leave in the holes?
dy.sum() and (dy + 1).isna().sum() give …
NaN and 0 — one hole poisons the sum, but adding 1 fills it48.28 and 1610 — the sum skips holes, the addition keeps themNaN and 1610 — holes poison everything48.28 and 0 — holes are treated as 0 everywhereThe split you found: aggregations skip NaN; element-wise operations propagate it. Same missing value, opposite treatment — and count() (non-missing) is not len().
Assign to dy_clean a version of dy with the missing dividend yields removed (.dropna()). Expected: 1 474 entries, zero NaN.
Strategy, not reflex: for prices, ffill() requires an explicit last-known-price valuation policy; for a dividend yield, dropping — or filling with 0 for a confirmed non-payer — depends on what the blank means. The tool follows the meaning of the data.
| Method | Policy |
|---|---|
s.dropna() |
Remove unknown observations |
s.fillna(value) |
Replace them with a declared value |
s.ffill() |
Carry an earlier observed value forward |
Many remaining rows do not make dropping unknown values unbiased: if missingness depends on the value, the observed sample may differ systematically.
These operations answer different questions. Forward filling can create a stale value; use it only when carrying the earlier observation forward is meaningful for the variable and its information timing. Preserve a missingness flag when relevant.
| Tool | Purpose and return value |
|---|---|
ffill |
Carry a preceding observed value forward in the current row order. |
pd.Series([1, np.nan, np.nan, 4]).ffill() fills the two missing rows with …
1, 12, 30, 02.5, 2.5Choose a policy, then trace the replacement and the number of observed or retained values.
Mode means the most frequent recorded value. .mode() returns a Series because ties can give several modes; .iloc[0] chooses the first. This toy exercise practises the operation. Imputing real sales outcomes would require justification; a completed target must not be invented from the mode.
sold is a 0/1 flag with two gaps (the notebook’s SoldFlag for Active products). Fill the gaps with the mode (sold.mode().loc[0]) into filled. Expected: no NaN, and the total is still 1.
| Tool | Purpose and return value |
|---|---|
isfinite |
np.isfinite(values) marks finite numbers; infinity and missing values are not finite. |
any |
Reduce checks to one Boolean: did at least one retained check pass? |
Compare list price with the inventory file’s low user price. A zero denominator is not a very large valid ratio. Keep invalid results missing and count them.
Three of four ratios are valid. The second low user price is zero, so its ratio remains missing. notna() alone does not reject infinity; np.isfinite does. Filling that missing ratio with zero would change its meaning.
Question: How should observations be compared or categorised?
Result: An ordered or categorised Series with ties and cut points explained.
With a stated missingness policy, compare and categorise the retained observations.
s.sort_values() orders entries by their values; s.sort_index() orders them by their index values. Both return a new Series unless explicitly asked to modify in place. Index values travel with their values.
s.clip(lower=a, upper=b) replaces values below a by a and values above b by b; it retains the rows. A Boolean filter instead removes rows.
Clipping is a declared transformation, not proof that extremes were errors. Estimate thresholds on training data when they will be used for prediction.
| Tool | Purpose and return value |
|---|---|
sort_values |
Return entries ordered by value; ascending=False puts larger values first. |
Before running: Read the sorting calls. Will the first row contain the smallest value, largest value or largest index?
Ascending by default; the index values travel with the values ({15511: 2800.0, 4071: 2083.09, 2218: 1580.0} at the top), so a later sort_index() restores index value order and forgets the value sort. Sorting returns a new Series — price is untouched.
| Tool | Purpose and return value |
|---|---|
clip |
Cap/floor values at supplied bounds while retaining index values and length. |
s.quantile(0.8) == 21 tells you …
Clipping = capping and flooring. The max falls from 2 800 to 271.67, the min rises from 0 to 12.50, and the standard deviation drops from 85.93 to 71.16 — one method call limits tail magnitudes while retaining all rows.
Clip the 2024 AAPL closes close24 to lie between their 5th and 95th percentiles, into clipped. Expected min ≈ 169.30, max ≈ 247.86.
If the check fails: Check all 252 dates: retain their order, replace only prices outside the two bounds, and leave every interior price unchanged.
| Tool | Purpose and return value |
|---|---|
drop_duplicates |
Remove repeated values; keep=“first”, “last”, or False selects the survivor policy. |
dlist = pd.Series([40, 20, 30, 20, 10]) has the value 20 at positions 1 and 3.
Which call returns [40, 30, 10] — no 20 at all?
dlist.drop_duplicates()dlist.drop_duplicates(keep="first")dlist.drop_duplicates(keep="last")dlist.drop_duplicates(keep=False)Compare a common date and horizon; a price rank is not investment value.
partPrice = price.loc[:10] — index values 0 … 10, eleven products.
The eleven prices, in index value order, are:
| Index value | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Price | 44.99 | 24.81 | 46.00 | 100.00 | 121.95 | 132.00 | 95.95 | 207.80 | 119.81 | 49.95 | 36.95 |
Before running: Read the Series and its ranks. Which index values identify the lowest prices?
rank() gives 1 to the smallest (ties get the average rank); rank(pct=True) rescales to \((0, 1]\) — a cross-sectional percentile rank in one call, the workhorse feature of Project 2. replace takes a list → one value, or a dict of old → new. The result is object dtype the moment a string enters.
Calculate changes within each stock’s time series, then select the same date and rank the resulting cross-section. The observed returns are AAPL −2.498% and MSFT −4.119%. Predict which receives the larger ascending rank; then verify both calculations.
AAPL −2.498%, MSFT −4.119%; percentiles 1.0 and 0.5. AAPL ranks higher in this two-stock snapshot. This describes an observed session, excludes dividends and establishes no future performance. Repeating the rank separately on each date requires Chapter 2’s grouped pipeline.
rlist holds body weights. Build label: "strong" where weight > 50, else "slim". Use np.where and wrap its values in a Series with index=rlist.index.
Intervals are right-closed by default; inspect edges, counts and missing assignments before adopting a business rule.
For −4, −1, 0, 1, 4, equal-width bins have inner boundaries −2, 0, 2; sample quartiles are −1, 0, 1. The data are symmetric, but the boundaries differ.
retbins=True returns two objects: the assignments and the numerical boundaries. Unpack them to inspect how the groups were formed.
Now apply the same distinction to the inventory prices. Inspect counts as well as boundaries; ties can make quantile groups unequal.
pd.cut(price, bins=4) has a first upper boundary of 700, with 19,990 of 20,000 prices at or below it. How full is that bin?
Equal width: 19 990 of 20 000 in (-2.8, 700]. The notebook’s own edges give [607, 19389, 4, 0] — 607 recorded zero prices separated. qcut produces four bins of ≈ 5 000. Binning turns a number into a category — the input to groupby in Chapter 2.
where keeps True values; mask replaces them. map performs a lookup; replace recodes.Next: §1.3.
| Part | Question |
|---|---|
| 1.3.1 Index, positions and slices | Which entries does this selector address? |
| 1.3.2 Inspection, sampling and deliberate alignment | Which identities survive this operation? |
| 1.3.3 Selection and safe assignment | Which object receives the change? |
Question: Which entries does this selector address?
Result: A selection with the intended index values and endpoints.
Cleaning retains index values. Explicit selection now identifies the observations you need.
AEP examples use hourly timestamp index values and electric load in MW.
Before running: Read each rename operation. Which index values change, and which data values stay the same?
A function maps every index value; a dict maps only the ones you list; assigning .index replaces the whole thing (and must have the right length).
| Tool | Purpose and return value |
|---|---|
reset_index |
Move Series index values into a DataFrame column; drop=True discards them and retains a Series. |
rename_axis |
Set the index name, rather than its individual index values. |
Check your understanding
aep.rename_axis(“FancyName”) changes …
Answer: The index’s name — the header above the index values
rename_axis changes the name of the index (the header printed above the index values), not the index values themselves. It matters at reset_index time: the index name becomes the new column’s name.
reset_index() turns a Series into a two-column DataFrame whose first column is named after the index (FancyName, or Datetime). drop=True throws the timestamps away and leaves a plain RangeIndex — useful after filtering, dangerous for time series.
In Colab
energy = isom5650.data.energy() returns the same PJM hourly load table (2004–2018) with Datetime as the index; aep = energy["AEP"].
The table is ordered by its numeric index. The index values are not row positions.
How many rows do aep.loc[“2017-01-01 01:00”:“2017-01-01 05:00”] and aep.iloc[0:5] return?
The index value slice ends at 05:00; the position slice at 04:00 (positions 0–4). Both have five rows. A positional split uses iloc[:trainsize]; check its boundary dates.
Context: the S&P 500 file records an index level, in index points, for each observed trading date. A year or month string selects a period of that date index.
| Tool | Purpose and return value |
|---|---|
date |
Timestamp.date() returns its calendar date without the time for display. |
idxmin |
Return the index value of the first minimum, rather than its value. |
Before running: Read the two year slices. Should a whole-year selection and explicit year endpoints select the same observations?
"2020" means the whole year (253 trading days), "2020-03" the whole month (22 days). The March 2020 low was 2 237.40 on the 23rd.
Within jan = sp500.loc["2020"], trading days sit at positions 0 (Jan 2), 1 (Jan 3), 2 (Jan 6), 3 (Jan 7), 4 (Jan 8) …
For jan = sp500.loc[“2020”], do jan.loc[“2020-01-06”:“2020-01-08”] and jan.iloc[2:5] return the same rows?
.loc gives 2 rows, .iloc gives 3.iloc includes Jan 9The matching slices return 3246.28, 3237.18, 3253.05. Changing to iloc[2:4] excludes Jan 8. Verify boundary dates when mixing index value and position slices.
Build a Boolean condition with the original index, then pass it to .loc. Give intermediate results names so you can inspect them.
The cutoff is 147.4; 4,998 prices are strictly above it. The selected Series keeps its original product-index values.
Sort the prices descending: top = price.sort_values(ascending=False). Its index is now 15511, 4071, 2218, …. Compare the printed results of .loc[0] and .iloc[0].
Sorting changes positions; each price keeps its original index value.
top.loc[0] selects index value 0, not the first position. Use .loc or .iloc to make the rule explicit.
The notebook asks for MSFT’s worst days; microsoft.csv covers 2015–2018.
From MSFT daily returns r_msft, select the days with a loss worse than 5 % into worst using .loc and a mask. Expected 3 days; the worst is 2015-01-27 at −9.25 %.
You want MSFT days with a return strictly between +2 % and +5 %. Predict what happens if you drop the parentheses.
r.loc[r > 0.02 & r < 0.05] …
TypeError — & binds tighter than >41 days fall in the band; 83 lie beyond ±2%. Combine masks with &, |, ~ and parenthesised comparisons.
Question: Which identities survive this operation?
Result: A reproducible sample or aligned result with index values retained.
A selected value still needs an identity when you sample, align or export it.
s.head(n) takes the first n entries; s.tail(n) takes the last n. Neither is a random sample.
s.sample(n, random_state=0) draws entries reproducibly. Without replacement, each selected entry appears at most once; replace=True allows repeated draws.
Sampling retains index values. A fixed seed reproduces a draw, but does not establish representativeness or independence. Time-based training splits must preserve chronological order.
price.sample(500, replace=True, random_state=0) …
frac is givenfrac=0.002 of 20 000 is 40 rows. With replacement, only 492 of the 500 draws are distinct — 8 products came up twice.
| Tool | Purpose and return value |
|---|---|
reindex |
Conform to requested index values and their order; absent index values create missing entries. |
Before running: Read the requested index. What will happen to a new index value that has no observation?
reindex follows the supplied index values, adding NaN for absent ones; numeric dtype may change. .loc instead raises KeyError for a missing index value.
s.index.is_monotonic_increasing tests whether index values are in non-decreasing order. It does not establish uniqueness. s.duplicated() checks values; s.index.duplicated() checks observation index values.
| Tool | Purpose and return value |
|---|---|
duplicated |
Mark repeated values on a Series, or repeated index values on its index. |
Two products may legitimately share a price. Two observations sharing one supposed unique product index value need investigation. Predict which rows each check marks.
The value mask is [False, True, False]; the index value mask is [False, False, True]. Sorted index values can still be duplicated. Decide what each index value represents before dropping rows. Chapter 2 uses ticker and date together to identify rows in a stock panel.
Three of four products match; correlation is 1.0 under proportional repricing. corr uses aligned, observed pairs. Report their count; correlation does not establish equal prices.
Scalar comparison retains the index. Comparing two Series requires identical index values and order; arithmetic can align different indexes automatically.
The sum is A = 20, B = 40; after explicit alignment the comparison is False for both index values. join="inner" keeps common index values: inspect which observations were excluded before interpreting the result.
Sorting changes positions while the product index values stay attached to their prices. Apply a transparent 10% price reduction to the sorted Series, then compare index-based and position-based selection.
.loc still gives the first original SKU’s reduced price, 40.491. .iloc[0] gives a different, first-sorted product: 22.329. Use reindex to restore a declared order.
revised_price retains its product index values but follows the sorted price order. Use reindex to put it in the original order given by sku_price.index.
Return the revised prices in the original product order.
Check both the index values and the values. Matching lengths alone cannot prove that a revised price belongs to the correct product.
Question: Which object receives the change?
Result: One explicit selection or assignment, checked against its source.
Selection and assignment use similar syntax but produce different effects.
| Tool | Purpose and return value |
|---|---|
filter |
Select by index text: items lists index values, like matches a substring, regex matches a pattern. |
Use items for named index values, like for a substring, or regex for a pattern. These arguments inspect the index.
like="Ma" keeps March and May, including the zero. The value mask keeps January and May. Choose one of items, like, and regex per call.
Add 100 to every price above 50 in the first five products. Use one .loc assignment to update the original Series. Chaining two selections writes into a temporary Series.
Before running: Read the assignment. Which prices will change, and will their index change?
A means units > 2; B means units < 6. True retains the original observation index.
Context: one NVDA trading session per date index value; values are stored closes. Select dates by weekday while retaining their prices.
nvda.index.day_name() returns the weekday name for each date. Comparing these names with "Friday" forms the selection mask. After a Monday holiday, Friday’s next observed session may be Tuesday; verify timestamps before pairing dates.
The notebook practice starts from Friday closes; pair them with the next observed trading session for NVDA, which need not be Monday. nvda_spy_daily_2023_2024.csv holds two years of NVDA.
Select all Friday closes into fridays: 102 dates. For a future outcome using shift(-1), verify the next session is Monday before calling it a Friday–Monday pair.
If the check fails: Check that all 102 Friday dates are present, no other weekday is present, and the closes retain their original values.
.loc for index values (inclusive stop), .iloc for positions (exclusive stop). Sorting changes positions.sample draws observations; reindex conforms to index values and can introduce missing entries.reindex for a required row order, and assign through one .loc.Next: §1.4.
| Part | Question |
|---|---|
| 1.4.1 Dates, lags and observed returns | What was observed before this row? |
| 1.4.2 Rolling and cumulative features | Which observations enter the statistic? |
| 1.4.3 Calendar bins and resampling | What belongs in each time bin? |
Question: What was observed before this row?
Result: A chronological Series with amount changes and fractional changes distinguished.
A date index value adds chronological meaning. Establish order before calculating a change.
Row windows count observations; resampling outputs calendar-bin index values.
Context: one Delhi calendar day per row. The date column is recorded text; we inspect year, month and day before using dates to select observations.
weather[“date”].iloc[0] right after pd.read_csv is a …
strTimestampdatetime64int (days since 1970)In Colab
weather = isom5650.data.weather() — the same Delhi daily-climate table (1 462 rows, 2013-01-01 … 2017-01-01).
Use .dt for date values in a Series: .dt.year, .dt.dayofweek (Monday 0), .dt.is_month_end (Boolean) and .dt.day_name(). A DatetimeIndex exposes these properties directly, without .dt. Date text must first be parsed.
| Tool | Purpose and return value |
|---|---|
strftime |
Format datetimes into text; %d day, %m month, %y two-digit year, %a short weekday. |
Before running: Read the date operations. Which return a Series, and which return one count? Run to inspect the actual dates.
There are 48 month-ends in 2013–2016; the series ends on 2017-01-01, touching five calendar years. strftime formats dates as text. Series date properties use .dt.
| Tool | Purpose and return value |
|---|---|
DataFrame |
Construct a table; dictionary entry names become column names and Series values align by index values. |
shift |
Move values by row positions while retaining the index; positive periods uses earlier rows. |
shift(1) moves values one row later. The MSFT example uses the historical microsoft.csv file for 2015–2018.
msft.shift(1).iloc[0] is …
NaN| Tool | Purpose and return value |
|---|---|
allclose |
np.allclose(a, b) checks approximate numerical agreement in array order, not index value identity. |
\[r_t=\frac{P_t-P_{t-1}}{P_{t-1}}\] — today minus yesterday, over yesterday. With shift that is one line; pct_change is the same line pre-written.
Manual returns and pct_change agree (True). Daily mean = 0.00091, SD = 0.0142. shift(-1) puts a future outcome on today’s row; use it as a target, not a feature.
Prices are ordered by date. Positive shift brings past prices; negative shift brings future prices.
| Tool | Purpose and return value |
|---|---|
diff |
Return current minus previous-row values, in the original units. |
The first three MSFT closes show the difference. Both methods compare successive observations, which need not be successive calendar days.
Friday, Monday, Tuesday: differences are [NaN, -0.43, -0.68] dollars; fractional changes are [NaN, -0.0092, -0.0147]. 0.01 means 1%, not 0.01%. Position-value differences become P&L only after units and cash flows are handled.
Mondays are said to under-perform Fridays. Compare the mean daily AAPL return by weekday.
Compute mon_mean = mean of r_aapl on Mondays and fri_mean = mean on Fridays (use r_aapl.index.day_name() as the mask). Expected: Monday ≈ 0.0023, Friday ≈ 0.0001 — the opposite of the folklore.
If the check fails: Check the selected observations and the sample-SD or mean calculation before comparing the two reported numbers.
Damage the middle of the three observed prices to inspect the missing-data rule. Filling the price first answers a different question from leaving it unknown.
Unfilled prices give no observable adjacent return here. Forward filling creates a zero return and a later multi-observation move. It requires a valuation policy and does not recover a tradable price.
Question: Which observations enter the statistic?
Result: A dated feature with a valid window and decision-time cutoff.
Lags and returns provide inputs for a trailing or cumulative statistic.
s.rolling(k).mean() calculates a mean from the most recent k rows at each position. For an integer window, the default requires k observed values; min_periods changes that requirement.
s.resample("ME").mean() instead groups a DatetimeIndex into calendar-month bins and summarises each bin. The month-end index value identifies the bin boundary, which need not be an observed trading date.
Sort chronologically first. A row window counts observations; a calendar bin measures clock time. Missing readings and partial bins need coverage checks.
Before running: The window needs ten observations. Which early rows cannot yet have a ten-observation mean?
Advance the observation date and compare which values are included in the mean.
A three-row window can contain fewer than three valid numbers. This damaged four-price excerpt makes the validity choice visible.
Strict windows produce no valid mean; relaxed windows produce two. The second rule accepts two observed prices, not three. Document the minimum coverage and inspect it before treating a rolling statistic as a feature.
| Tool | Purpose and return value |
|---|---|
subplots |
fig, ax = plt.subplots(figsize=(width, height)) creates a Figure canvas and one Axes panel; sizes are inches. |
plot |
ax.plot(series, index value=…) draws on the named panel, using the Series index as its horizontal coordinate. |
legend |
Show index values assigned to chart elements. |
set |
ax.set(title=…, xlabel=…, ylabel=…) names the chart and its units. |
show |
Display the completed Matplotlib figure. |
plt is the short name for Matplotlib’s pyplot module, imported with import matplotlib.pyplot as plt. It creates and displays charts. §1.5 explains plotting in detail; here, compare the moving averages.
The 10-day mean follows dips more closely; the 40-day mean smooths and lags. A crossover defines a candidate rule, not evidence of prediction.
The first three closes average 46.247. For a before-close decision, that mean is first available on row four. An after-close rule has a different cutoff.
r.rolling(21).std() calculates sample standard deviation within each trailing 21-observation window. The current observation is included; shift when the decision occurs before it is known.
Multiplying by np.sqrt(252) uses a trading-day annualisation convention. It assumes variance scales with time; it is not a guaranteed forecast.
Realised volatility peaks at about 43% annualised on 2015-09-18 and averages about 14% in 2017. Chapter 6 develops conditional volatility models.
rolling(k).apply(function) passes each complete window to a function returning one scalar. Prefer a built-in statistic when it expresses the rule.
Both give the same Series result. A callback adds no benefit for a built-in median; keep it for a genuinely custom window rule.
Simple returns sum to zero, but wealth loses 1. Distinguish gross growth, cumulative return and a sum of returns.
Only the initial return is missing in this complete sample; filling it with zero starts growth at one. Investigate internal gaps. This price path excludes dividends, fees and cash flows.
| Tool | Purpose and return value |
|---|---|
cumprod |
Return running products; multiplying valid gross returns (1+r) compounds growth. |
cumsum |
Return running sums; summing simple returns does not compound them. |
cummax |
Return the maximum observed so far at each row. |
cummin |
Return the minimum observed so far at each row. |
cumsum, cumprod, cummax carry a running total, product or record high. Predict what the product of gross returns recovers.
(1 + r).cumprod().iloc[-1] equals …
Compounded growth is 1.882×, matching the price ratio. The sum of simple returns, 0.711, is a different quantity. cummax tracks record highs; cummin stays at 40.29. Full accounts also need dividends, costs and cash flows.
\[\text{drawdown}_t=\frac{\text{cummax}_t - P_t}{\text{cummax}_t}\] — how far below its record high the price sits. The maximum drawdown is the worst peak-to-trough loss. The notebook asks for ten years of a stock: appl.csv.
Compute the drawdown Series dd for aapl with cummax(). Expected maximum ≈ 0.387 on 2019-01-03.
If the check fails: Check every date against its running peak. The first drawdown is zero; matching only the largest drawdown is insufficient.
| Tool | Purpose and return value |
|---|---|
expanding |
Define the entire observed prefix at each row; min_periods requires a minimum valid count. |
A rolling window forgets old observations. An expanding window retains the entire observed prefix; min_periods still counts valid observations.
The means are [NaN, 3, 2.333..., 3]; the sum is [2, 6, 7, 12]. The current observation is included. Shift the completed statistic when the decision precedes that observation.
Question: What belongs in each time bin?
Result: A calendar summary with meaningful aggregation and coverage checks.
A row window and a calendar bin answer different aggregation questions.
resample("ME").mean() groups by calendar month and index values the result at month end, even on a non-trading day. Early and late bins can be partial. Check .count() alongside the statistic.
| Tool | Purpose and return value |
|---|---|
resample |
Group a DatetimeIndex into clock bins, then aggregate: ME month-end, W week ending Sunday. |
msft.resample(“ME”).mean() — what is the first index value?
2015-01-02, the first trading day2015-01-31, the calendar month end2015-01-15, the midpoint2015-01, a month period| Tool | Purpose and return value |
|---|---|
last |
Return the last observed value in each resampling bin. |
"W" means weeks ending Sunday. MSFT’s first 2015 row is Friday 2 January. Predict how many observations the first weekly bucket holds.
Before running: Read the first dates and weekly counts. Why might the first week contain fewer observations?
The week ending 2015-01-04 contains one trading day; full weeks contain five. Specify a bin (W, ME, 5min) and aggregation. Check first and last bins for partial coverage.
| Tool | Purpose and return value |
|---|---|
transform |
Broadcast a group statistic back to each input row; a complete-month mean uses later days within that month. |
resample("ME").agg("mean") returns one row per month. .transform("mean") returns one row per day, each carrying its month’s mean — so you can subtract it.
ax.step(...) draws the repeated monthly mean as a staircase.
Aggregation gives 38 monthly rows; transform broadcasts to 779 daily rows. The staircase repeats each month’s mean. Chapter 2 applies this rule with groupby().transform.
Daily units are grouped by calendar month. agg reduces each group; transform retains every date.
Context: one AAPL minute timestamp per row, on one recorded session. The example aggregates the Close column; it cannot recover highs and lows that occurred inside each minute. | Tool | Purpose and return value | |—|—| | ohlc | Return first/max/min/last within each bin of the selected Series; close-only inputs cannot recover full intraminute highs/lows. |
Before running: Compare the first and last timestamps with the five-minute output. Are the first and last bins complete?
There are 79, not 78, five-minute bins: 09:30 holds 09:31–09:34, and 16:00 is alone. .ohlc() summarises Close observations; full candles need original OHLC columns. Use .sum() for volume, .last() for a close and .mean() for a level.
.asfreq() places observations on a one-minute grid and introduces gaps. .ffill() carries prior values under an explicit policy; it adds no measurements.
| Tool | Purpose and return value |
|---|---|
asfreq |
Return values on the new frequency grid, leaving new timestamps missing. |
Three five-minute closes become eleven one-minute rows, with eight gaps. Forward-filled values are assumptions, not new observations.
.dt accesses date values while a DatetimeIndex has date properties directly.shift is a row lag; diff measures an amount; pct_change measures a fractional change.Next: §1.5.
| Part | Question |
|---|---|
| 1.5.1 Figure and Axes foundations | Where will pandas draw the result? |
| 1.5.2 Distributions and paired observations | What does this chart represent? |
Question: Where will pandas draw the result?
Result: A labelled chart on the intended Axes.
The earlier sections produced Series results. Choose the canvas and axes that will display them.
| Tool | Purpose and return value |
|---|---|
subplots |
Create a Figure and named Axes together: fig, ax = plt.subplots(…). |
plot |
Draw values against index values; ax= selects the destination panel. |
set |
ax.set(title=…, xlabel=…, ylabel=…) names the chart and its units. |
tight_layout |
Adjust subplot spacing to reduce index value overlap. |
Load a sorted Series, calculate observed returns and draw into an explicitly named Axes.
fig controls the whole canvas; ax controls this chart. plot_close.plot(ax=ax) uses the pandas index as the horizontal axis.
Question: What does this chart represent?
Result: A distribution or paired comparison with units and denominator stated.
With the destination fixed, choose a chart whose geometry matches the question.
Which object owns the title and y-axis label of one subplot?
Six observations; unequal bin widths. Density height × bin width gives the probability in a bin.
| Tool | Purpose and return value |
|---|---|
default_rng |
Create a NumPy random generator with a fixed seed for reproducible teaching draws. |
normal |
rng.normal(mean, sd, size) draws a simulated sample with the declared parameters. |
hist |
Draw a histogram; density=True makes total bar area one, not the sum of heights. |
Create Figure/Axes, draw on ax and index value units. This example simulates 5 000 normal draws: mean 10, SD 5.
In ax.hist(x, bins=50, density=True) the bars are scaled so that …
value_counts() returns category counts indexed by category. .plot.barh(ax=ax) draws horizontal bars on the same Figure/Axes pattern used for a line chart.
Bars make close counts easy to compare. Missing sector index values remain in the denominator as Unknown.
| Tool | Purpose and return value |
|---|---|
linspace |
np.linspace(start, stop, n) returns n equally spaced points, including both endpoints by default. |
abs |
Return absolute magnitudes; use to check both negative and positive extremes. |
The outside-bin count is 0. Density scales the displayed histogram to area one; its bar heights are not probabilities.
| Tool | Purpose and return value |
|---|---|
scatter |
ax.scatter(x, y) plots paired observations; s sets marker area, alpha transparency. |
axhline |
Draw a horizontal reference line at the supplied y value. |
axvline |
Draw a vertical reference line at the supplied x value. |
Plot yesterday’s return against today’s return after aligning the two Series and removing incomplete pairs.
There are 778 paired rows. A cloud of points is descriptive evidence; it does not establish a trading rule or causal effect.
Which quantity represents probability in a density histogram?
How should missing x and y observations be removed?
| Tool | Purpose and return value |
|---|---|
set_ylabel |
Name the vertical axis and state its units. |
box |
Draw median, quartiles and whiskers; beyond-whisker points are flags, not proven errors. |
Use one panel for prices and one for return dispersion. Route each pandas plot through its own ax.
A boxplot summarises median, quartiles and whiskers. Points beyond the whiskers are flagged observations, not proven data errors.
The histogram depends on bins; the KDE depends on bandwidth. Neither establishes normality.
| Tool | Purpose and return value |
|---|---|
set_xlabel |
Name the horizontal axis and state its units. |
Count all sector index values. Collapse the remaining sectors into Other so the pie accounts for the complete sample.
The categories account for all 3,084 companies. Bars make close counts easier to compare; a top-five-only pie would show shares of those five sectors.
Use the Chapter 1 data to create one chart for a business question. State the sample, units, missingness rule and denominator. Change a bin width or date range and explain what changes.
Submit the labelled figure and two sentences of interpretation; an attractive chart alone does not answer the question.
Warning
Reinhart and Rogoff’s Growth in a Time of Debt (2010) reported −0.1% average growth above 90% debt/GDP. The later replication found excluded rows and other accounting choices.
In April 2013, Thomas Herndon found rows 30–44 averaged instead of 30–49, excluding Australia, Austria, Belgium, Canada and Denmark. Correcting exclusions, weighting and omitted years gave +2.2% for the >90% group.
Lesson: a slice determines which observations count. .loc includes its end index value; .iloc excludes its end position. Report len(), missing counts and the aggregate’s denominator.
Use the matching L1 or L2 notebook for the company project. This retail example practises the same memo structure; it is not a second assignment.
| Part | What the reader needs |
|---|---|
| Decision | Which products should receive a limited number of reviews? |
| Scope | Which observations and period are represented? Does the file contain the current active catalogue? |
| Rule and evidence | State the price threshold, sale indicator, candidate count and review capacity; compare an alternative threshold |
| Interpretation | Explain what a stable or changing shortlist demonstrates |
| Limitation and next step | Name a material assumption and the evidence needed to investigate it |
A pricing-review shortlist supports investigation. A decision to remove products or change prices needs further business evidence.
Next: Chapter 2 — validated tables and grouped feature pipelines.
An LLM writes pandas fluently — and confidently mixes versions and conventions. Three habits for this chapter:
.ffill(), resample("ME"), .loc and .iloc.| Concept | Tool |
|---|---|
| pandas structures and shape | pd.Series, pd.DataFrame, df["column"], df[["column"]], .shape |
| Time series vs cross-section | Dates within one entity vs entities on one date; panel index values combine both |
| Build a Series; index vs positions | pd.Series(data, index=, name=), .loc, .iloc |
| Vectorised arithmetic with alignment | + − × ÷, .add(fill_value=), & \| ~ |
| Aggregation | .mean() .median() .std(ddof=) .quantile() .agg([...]), mask.mean() |
| Conversion and lookups | .astype(), pd.to_numeric(), Int64, .map(), .isin(), .str, .dt |
| If-else without loops | np.where, .where, .mask |
| Missing data | .isna().mean(), .dropna(), .fillna(), .ffill() |
| Sorting, clipping, duplicates, ranking, binning | .sort_values(), .clip(), .drop_duplicates(keep=), .rank(), .replace(), pd.cut / pd.qcut |
| Concept | Tool |
|---|---|
| Observation identity and validation | .align(), .corr(), .reindex(), .index.is_unique, .all(), .any() |
| Selection and identity | .loc, .iloc, .reindex(), .align(), .sample(); rename, rename_axis, reset_index and index-text filters |
| Time series | .diff(), .shift(), .pct_change(fill_method=None), .rolling(k, min_periods=), .cumprod(), .cummax(), .resample("ME") |
| Plotting | fig, ax = plt.subplots(), s.plot(ax=ax), ax.hist, ax.scatter, s.plot.barh, ax.set |
NaN, and when is .add(fill_value=0) justified?ISOM 5650 · Prof. Xuhu Wan · HKUST Business School