Pandas & NumPy
The data-wrangling operations every ML notebook runs on — selection, groupby, joins, reshaping, vectorization.
NumPy provides the ndarray: a typed block of numbers on which every operation runs in compiled code over the whole array at once. pandas builds the DataFrame on top of it: labeled columns, each with its own dtype, plus an index that aligns rows in every operation.
The mental model is the same for both. Say what to do to a whole column, never loop over rows in Python, and let alignment and broadcasting line the data up. The rows cover pandas 2 and 3; pandas 3 turns on Copy-on-Write and a dedicated string dtype by default, and the gotchas that changes are marked.
groupby is split-apply-combine. .agg collapses each group to one row; .transform computes the same group statistic but writes it back onto every original row, so it can sit beside the data, as in v / v.groupby(k).transform("sum").Load & inspect
- pd.read_csv(path)
- set
usecols=,dtype=andparse_dates=at load time instead of fixing columns later - chunksize=100_000
- with
read_csv: stream a file too big for memory as an iterator of frames - pd.read_parquet(path)
- columnar, typed and compressed — far faster than CSV; needs
pyarrow - df.info() · df.describe()
- dtypes, non-null counts and memory · summary statistics per numeric column
- df.head() · df.sample(5)
- first rows · random rows — a sample shows more than the top of a sorted file
- s.value_counts()
- frequency of each value;
normalize=Truefor shares,dropna=Falseto count NaN - df.nunique()
- distinct values per column — spot IDs, constants and category candidates
- to_csv(p, index=False)
- write without the index, which otherwise comes back as an
Unnamed: 0column
Selection (the part everyone fumbles)
- df.loc[rows, cols]
- by LABEL / boolean mask —
df.loc[df.x > 5, "y"] - df.iloc[i, j]
- by POSITION —
iloc[0]first row,iloc[:, -1]last col - Boolean masks
- combine with
& | ~and parentheses — neverand/or - df.query("x > 5")
- readable chained filters, e.g.
"x > 5 and y == 'a'";@varreads a Python variable - SettingWithCopyWarning
- assign with one
.loc[mask, col] = val; chaineddf[mask][col] = valnever writes in pandas 3 - df[["a", "b"]]
- double brackets return a DataFrame; single
df["a"]returns a Series - s.isin(["a", "b"])
- membership mask, the vectorized
in;~negates it - s.between(1, 10)
- range mask, inclusive at both ends;
inclusive="left"makes it half-open
Clean & transform
- df.isna().sum()
- the null audit; then
fillna/dropna(subset=…) - drop_duplicates(subset)
- dedupe on key columns;
keep="last"keeps the newest instead of the first - df.assign(c=lambda d: …)
- add columns inside a method chain; the lambda sees the frame built so far
- df.rename(columns={…})
- rename columns;
df.columns.str.lower()normalizes them all at once - s.str.contains("x")
- vectorized string methods; pass
case=False, na=Falseso missing values don’t break the mask - s.str.extract(r"(\d+)")
- regex groups into columns;
str.split(",", expand=True)splits into columns - pd.to_numeric(s, errors=)
errors="coerce"turns unparseable strings into NaN instead of raising- s.map({"a": 1})
- look each value up in a dict; values without a match become NaN
- pd.cut · pd.qcut
- bin by fixed edges, e.g. ages · into equal-count quantiles:
pd.qcut(s, 4) - df.explode("tags")
- one row per element of a list column
- astype("category")
- large memory win on low-cardinality strings (each value stored once, rows hold small integer codes)
Groupby & joins
- groupby("k")["v"].mean()
- split-apply-combine, the core move
- .agg(["mean","count"])
- multiple stats at once; dict for per-column stats
- .agg(total=("v", "sum"))
- named aggregation: flat output columns, one
name=(column, func)pair each - .transform("mean")
- group stat broadcast back to the original rows — e.g. subtract each group’s mean in one line
- as_index=False
- with
groupby: keep group keys as columns, so noreset_index()afterwards - groupby(dropna=False)
- by default rows whose key is NaN silently vanish from the result
- pd.merge(a, b, on="k")
- SQL joins with
how="left"or"inner";indicator=Truedebugs match rates - validate="m:1"
- make a merge raise when the right key is not unique — catches silent row explosions
- pd.concat([a, b])
- stack frames;
ignore_index=Truerenumbers,axis=1goes side by side
Reshape & time series
- pivot_table / melt
- long→wide with aggregation / wide→long (tidy)
- pd.to_datetime + .dt
- parse once, then
.dt.date/.dt.dayofweek; resample by.resample("W") - resample("W", on="ts")
- time buckets from a datetime column; month-end is
"ME"— pandas 3 rejects"M" - s.rolling(7).mean()
- 7-row moving average;
rolling("7D")uses a time window on a DatetimeIndex - s.shift(1) · s.diff()
- previous row’s value · change from it;
pct_change()gives growth - set_index · reset_index
- move a column into the index and back; many bugs are a forgotten
reset_index()
Speed & memory
- df.apply(f, axis=1)
- a Python loop in disguise — slow; vectorize with column math or
np.where - memory_usage(deep=True)
- true memory per column, counting string contents
- df.to_numpy()
- the underlying array for NumPy or scikit-learn; prefer it to
.values
NumPy core
- Vectorize, never loop
a * b + con whole arrays — 100× over Python loops- Broadcasting
- shapes align right-to-left; (n,1) × (1,m) → (n,m)
- np.where(cond, x, y)
- vectorized if/else; chain with
np.select - argmax / argsort
- indices, not values — top-k =
argsort()[-k:][::-1] - np.argpartition(a, -k)
- top-k in O(n) without a full sort; the k results come back unordered
- @ / np.dot
- matrix multiply — cosine sim =
a@b / (norm(a)*norm(b)) - axis rule
axis=0collapses rows (per-column),axis=1collapses columns (per-row)- a.reshape(-1, 1)
-1infers that dimension — here, a 1-D array becomes a column- np.random.default_rng(0)
- the modern seeded generator:
rng.normal(size=5),rng.integers(0, 10, 5) - Views vs copies
- a slice is a view, so editing it changes the original;
.copy()breaks the link - np.nanmean(a)
- statistics that skip NaN; plain
meanreturns NaN if any value is NaN - np.linalg.norm(a, axis=1)
- row-wise vector lengths;
keepdims=Truekeeps a column you can divide by - np.unique(a)
- sorted distinct values;
return_counts=Trueadds how often each appears - a.astype(np.float32)
- half the memory of float64; integer types overflow silently (int8 wraps past 127)
Worked example: Clean a messy export, then pivot it
Six order rows as a sales tool exported them: prices as text with $ and commas, one duplicated row, a blank and a lowercase region, and a price of n/a.
order_id,date,region,product,price
1001,2026-10-01,North,Widget,"$1,200.00"
1002,2026-10-01,South,Gadget,$85.50
1002,2026-10-01,South,Gadget,$85.50
1003,2026-10-02,,Widget,"$1,150.00"
1004,2026-10-02,North,Gadget,n/a
1005,2026-10-03,south,Widget,$990.00df = pd.read_csv("orders.csv", parse_dates=["date"])
df = (
df.drop_duplicates()
.assign(
price=lambda d: pd.to_numeric(
d["price"].str.replace(r"[$,]", "", regex=True), errors="coerce"),
region=lambda d: d["region"].str.title().fillna("Unknown"),
)
)
df.pivot_table(index="region", columns="product", values="price",
aggfunc="sum", fill_value=0)product Gadget Widget
region
North 0.0 1200.0
South 85.5 990.0
Unknown 0.0 1150.0drop_duplicates()removes the second 1002, which would have doubled South’s gadget sales.- The regex strips
$and the thousands comma, andto_numericturns the rest into floats.n/awas already NaN:read_csvtreats it as missing by default. str.title()folds “south” into “South”; without it the pivot would have two South rows.- North’s gadget shows 0.0, yet there was an order. Its price is missing,
sumskips NaN andfill_value=0hides the gap. Add"count"toaggfuncwhenever a zero could really mean “unknown”.
Recipes
Aggregate per group and keep the top 3 in each
import pandas as pd
df = pd.read_csv("sales.csv", parse_dates=["date"])
summary = (
df.groupby(["region", "product"], as_index=False)
.agg(revenue=("amount", "sum"), orders=("order_id", "nunique"))
.sort_values(["region", "revenue"], ascending=[True, False])
.groupby("region")
.head(3)
)Named aggregation gives flat column names, and groupby(...).head(3) after sorting keeps the first three rows per group: the pandas version of SQL’s top-N per group.
Join two tables and check the match rate
merged = orders.merge(customers, on="customer_id", how="left",
indicator=True, validate="m:1")
print(merged["_merge"].value_counts())
unmatched = merged[merged["_merge"] == "left_only"]validate="m:1" raises if a customer appears twice, which would duplicate orders, and the _merge column counts the orders that found no customer.
Weekly totals and a 4-week moving average
weekly = (
df.set_index("date")["amount"]
.resample("W")
.sum()
)
trend = weekly.rolling(4).mean()resample needs a datetime index (or on="date"), and the first three values of the moving average are NaN because the window is not yet full.
Cosine similarity of one vector against many
import numpy as np
docs = np.random.default_rng(0).normal(size=(1000, 384))
query = docs[0]
docs_n = docs / np.linalg.norm(docs, axis=1, keepdims=True)
scores = docs_n @ (query / np.linalg.norm(query))
top5 = np.argsort(scores)[-5:][::-1]Normalizing the rows once turns every similarity into a single matrix-vector product; keepdims=True keeps the norms as a column so broadcasting divides each row.
Shrink a DataFrame’s memory
before = df.memory_usage(deep=True).sum()
for col in df.columns:
if pd.api.types.is_string_dtype(df[col]) and df[col].nunique() < 0.5 * len(df):
df[col] = df[col].astype("category")
df = df.astype({c: "float32" for c in df.select_dtypes("float64").columns})
print(before, df.memory_usage(deep=True).sum())Repetitive strings become categories and floats drop to 32 bits; is_string_dtype works on pandas 2 and 3, where string columns have different dtypes.
When to use it, and when not
pandas is the right tool for labeled, tabular data that fits in memory: exploring a dataset, cleaning an export, building features for a model, producing a summary table. NumPy is the layer underneath, for homogeneous numeric arrays such as embeddings, images, matrices and simulations.
- Bigger than memory. Read in
chunksizepieces if the job is a simple running aggregate; otherwise use an out-of-core or lazy engine such as Polars, DuckDB or Spark. - Already in a database. Push the filter and the
GROUP BYinto SQL and pull back the small result, not the whole table. - Row-by-row state. When each row depends on the previous result, the work does not vectorize. A plain loop over
to_numpy()arrays, or a compiled function, beatsapply. - Tiny data inside a service. Looking up a few records per request is lighter with dicts; building a DataFrame per request adds overhead the work does not need.
Trade-offs and limits
Both libraries buy their speed by fixing one dtype per column, and most surprises trace back to that choice.
| Choice | Gives you | Gives up |
|---|---|---|
| float32 instead of float64 | Half the memory | About 7 significant digits instead of about 16; integers above 16,777,216 (2²⁴) are no longer exact |
| Default int64 | Fast, compact integers | One missing value turns the column into float64; the nullable "Int64" dtype keeps integers |
category | Small integer codes plus one copy of each label; faster groupby | A value outside the categories raises TypeError until you add it |
| CSV | Readable by anything | No stored types: every load re-parses text and re-guesses dtypes, so dates and IDs need dtype= and parse_dates= each time. Parquet keeps them |
Common mistakes
- Losing real values to the NA list.
read_csvtreats strings such asNA,n/aandnullas missing, so Namibia’s country codeNAbecomes NaN. Passkeep_default_na=False, plusna_values=for the markers you do mean. - Comparing with NaN.
s == np.nanis False for every row, NaN included, because NaN equals nothing. Uses.isna(). - Assigning a Series that aligns on the index. After filtering, a frame’s index might be [1, 2].
f["x"] = pd.Series([100, 200])matches labels, not positions: row 1 gets 200 and row 2 gets NaN. Assign.to_numpy()for positional values. df = df.dropna(inplace=True). Withinplace=Truethe method returns None, sodfis now None. Dropinplaceand reassign; pandas 3.1 deprecates it ondropnaanyway.- IDs read as numbers.
read_csvturns ZIP code 02139 into the integer 2139, and a merge between an int64 key and a string key raisesValueError. Read identifiers withdtype=str.
Check yourself
What shape is np.zeros((3, 1)) + np.zeros(4)? And np.zeros(3) + np.zeros(4)?
(3, 4): the (4,) is treated as (1, 4) and both size-1 axes stretch. The second raises ValueError: operands could not be broadcast together with shapes (3,) (4,).
For s = pd.Series([1, None, 3]), what are s.dtype, s.sum() and s.mean()?
float64, 4.0 and 2.0. The None forces floats, and both reductions skip the NaN.
What does np.arange(6).reshape(2, 3).sum(axis=0) return?
array([3, 5, 7]): axis 0 collapses the rows, leaving one total per column.
A left frame has 3 rows with key x, a right frame 2 rows with key x. How many rows does an inner merge on the key return?
6: every left row pairs with every matching right row. validate="m:1" would raise instead, because the right key is not unique.
Questions about Pandas & NumPy Cheat Sheet
What is the difference between loc and iloc in pandas?
loc selects by label: index values and column names, plus boolean masks, and its slices include the end label, so df.loc[2:5] can return four rows. iloc selects by integer position, like a Python list, so df.iloc[2:5] returns the third to fifth rows and excludes position 5. The difference only matters when the index is not a plain 0 to n-1 range, which is exactly when bugs appear, for example after filtering or sorting. Use loc whenever you mean a named row or column, and iloc for first, last or nth.
How do I fix SettingWithCopyWarning in pandas?
The warning appears when you assign through chained indexing, such as df[df.status == "late"]["fee"] = 10, because the first step may return a copy and the assignment would then be lost. The fix is a single indexing call that selects rows and column together: df.loc[df.status == "late", "fee"] = 10. When you really want a separate DataFrame, call .copy() when you create it. In pandas 3, Copy-on-Write is the default, the old warning is gone, and chained assignment never modifies the original, so code that relied on it must switch to loc.
Why is my pandas code slow?
Usually because Python is running once per row. apply with axis=1, iterrows and explicit loops all execute Python code for every row, while column arithmetic, comparisons, np.where and the built-in string and datetime accessors run in compiled code over the whole column at once, often a hundred times faster. Next, check dtypes: numbers stored as strings and object columns are slow and large, so convert them and use category for repetitive text. For big inputs, read only the columns you need and prefer Parquet over CSV.
What is broadcasting in NumPy?
Broadcasting is how NumPy combines arrays of different shapes without copying data. Shapes are compared from the last dimension backwards, and two dimensions are compatible when they are equal or one of them is 1; a dimension of 1 is stretched to match. So an array of shape (1000, 3) minus one of shape (3,) subtracts the same row from all 1000 rows, and shapes (n, 1) and (1, m) combine into (n, m). When shapes do not line up, add an axis with reshape or keepdims rather than writing a loop.