Pandas DataFrame Internals

A DataFrame looks like a spreadsheet — rows and columns of mixed types side by side — but it isn't stored that way. Internally, pandas groups same-dtype columns into shared NumPy arrays called blocks, so a DataFrame with 5 float columns and 2 int columns typically has just two underlying blocks, not seven separate ones. That single design decision explains a lot of pandas' performance behavior, both good and bad.

Why column-oriented storage

Grouping columns by dtype lets pandas apply the same fast, vectorized NumPy operations it uses on plain arrays: summing a float column, filtering an int column, or computing correlations between numeric columns all stay within contiguous, homogeneous memory. This is also why df.dtypes matters more than it looks — a DataFrame with a single object-dtype column mixed among numeric ones can quietly force operations that touch both to fall back to slower, Python-level loops instead of vectorized NumPy calls, because object columns hold arbitrary Python objects rather than a packed numeric array.

Worked example

import pandas as pd, numpy as np
df = pd.DataFrame({
    "price": np.random.rand(1_000_000) * 100,
    "size":  np.random.rand(1_000_000) * 1000,
    "side":  np.random.choice(["buy", "sell"], 1_000_000),
})

%timeit df["price"].sum()                          # ~1 ms — stays in the float block, vectorized
%timeit sum(row["price"] for _, row in df.iterrows())  # ~10+ seconds — thousands of times slower

iterrows() reconstructs a new Python Series object for every single row, crossing block boundaries and losing every advantage of columnar, vectorized storage — for 1 million rows this can be a five-order-of-magnitude slowdown compared to df["price"].sum(), which never leaves the underlying NumPy array.

What this means in practice

The practical rule this internals picture leads to is: reach for column-wise, vectorized operations (df["col"].method(), boolean masking, .apply() on a Series) before ever reaching for .iterrows(), .itertuples(), or a manual Python loop over rows — the latter are sometimes unavoidable for genuinely row-dependent logic, but they should be a last resort in a hot path. It's also why converting an object-dtype column of mixed types to a proper numeric or categorical dtype early in a pipeline (pd.to_numeric, astype("category")) often produces large, free speedups: it lets that column join a fast, contiguous block instead of sitting as an array of Python object pointers.