GroupBy: Split-Apply-Combine, Explained

You have a table of trades across 500 symbols and a full year of dates, and you want each symbol's average daily volume. Writing a loop that filters to one symbol, computes the mean, and stores the result — repeated 500 times — works but is slow and doesn't generalize. df.groupby("symbol")["volume"].mean() does the same thing in one line, using a pattern general enough to cover almost any "compute something per group" task: split-apply-combine.

Split-apply-combine breaks any per-group computation into three stages: split the data into groups sharing a key, apply a function independently to each group, then combine the per-group results back into one output structure. .groupby() handles the split and combine mechanics; you supply only the apply step.

The three stages

Split: df.groupby("symbol") partitions the DataFrame's rows into one sub-DataFrame per distinct symbol value, without computing anything yet — this step is lazy, just organizing row indices by group.

Apply: a function runs independently on each group's data. This can be a built-in aggregation (.mean(), .sum(), .std()), a custom aggregation via .agg(), or an arbitrary per-group transformation via .apply() or .transform().

Combine: the per-group results are stitched back into a single Series or DataFrame, indexed by the group keys (for an aggregation, one row per group) or aligned back to the original row index (for a transform, same shape as the input).

Split partitions rows by group key, apply runs independently per group, combine reassembles one indexed result — all three stages hidden behind a single chained call.

Worked example

df = pd.DataFrame({
    "symbol": ["AAPL", "AAPL", "MSFT", "MSFT"],
    "volume": [1_000_000, 3_200_000, 1_500_000, 2_100_000],
    "price":  [190.0, 192.0, 410.0, 415.0],
})
df.groupby("symbol")["volume"].mean()
# symbol
# AAPL    2100000.0
# MSFT    1800000.0

df.groupby("symbol").agg(avg_vol=("volume", "mean"), avg_price=("price", "mean"))
#         avg_vol  avg_price
# AAPL   2100000.0    191.0
# MSFT   1800000.0    412.5

.agg() with named tuples applies a different function to different columns within the same groups in one pass — here, the mean of both volume and price, computed per symbol, combined into one two-column result indexed by symbol.

What this means in practice

Split-apply-combine underlies most cross-sectional quant computations: per-symbol volatility, per-sector average return, per-day rank of a signal across the universe. .transform() is the variant worth knowing distinctly from .agg(): it applies a per-group function but returns a result the same shape as the input (broadcasting the group's summary value back to every row in that group), which is exactly what's needed to, say, subtract each stock's own sector mean from every row for that stock, without a separate merge step.

.apply() on a GroupBy object is the most flexible option but also the slowest — it falls back to a Python-level loop over groups when the operation isn't one pandas can vectorize internally. Prefer a named aggregation (.agg()) or built-in method (.mean(), .sum()) whenever the computation fits one, and reserve .apply() for genuinely custom per-group logic that has no vectorized equivalent.

GroupBy: Split-Apply-Combine

The three stages

Worked example

What this means in practice

Related concepts

Practice in interviews

Further reading