Columnar Storage and Parquet

A research query against a year of tick data almost never touches every column. "Give me the timestamp and price for AAPL trades, ignore size and exchange flag" is typical. A row-oriented file (like CSV, or a plain database table) stores record 1's timestamp, price, size, flag, then record 2's timestamp, price, size, flag, and so on — so answering that query still means reading every byte of every column, off disk, only to throw most of it away. Columnar storage flips the layout: all timestamps together, then all prices together, then all sizes together. A query that needs two of four columns reads roughly half the file, not all of it. Parquet is the columnar file format that has become the default for this in data engineering.

The idea: same values, grouped, compress far better

Storing a column contiguously has a second effect beyond skipping unused columns: values of the same type and similar magnitude sit next to each other, which compression algorithms exploit heavily. A column of prices that mostly move in small increments compresses far better as one contiguous block than when it's interleaved with unrelated timestamp and size bytes.

Parquet also stores per-column, per-chunk statistics (min, max, null count) in its metadata. A query filtering price > 500 can skip an entire chunk without reading it at all if that chunk's stored maximum is 480 — this is called predicate pushdown, and it only works because the format knows where each column's data lives without scanning it.

Worked example: row-store vs column-store on disk

Three trades, four fields each:

Row-oriented layout (CSV-style):
09:30:01,100.10,200,NYSE | 09:30:02,100.12,150,NYSE | 09:30:03,100.09,300,NASDAQ

Column-oriented layout (Parquet-style):
timestamps: [09:30:01, 09:30:02, 09:30:03]
prices:     [100.10, 100.12, 100.09]
sizes:      [200, 150, 300]
exchanges:  [NYSE, NYSE, NASDAQ]

A query for "average price" against the row layout must read every field of every record — timestamps, sizes and exchange flags included — just to discard them. Against the column layout it reads only the prices block: one contiguous run of bytes, nothing else touched. At three rows the difference is trivial; at a billion rows with twenty columns, reading one column instead of twenty is roughly a 20x reduction in bytes pulled off disk for that query, before compression is even counted.

import pandas as pd

df = pd.DataFrame({
    "timestamp": ["09:30:01", "09:30:02", "09:30:03"],
    "price":     [100.10, 100.12, 100.09],
    "size":      [200, 150, 300],
    "exchange":  ["NYSE", "NYSE", "NASDAQ"],
})
df.to_parquet("trades.parquet")                       # columnar on disk
avg = pd.read_parquet("trades.parquet", columns=["price"])["price"].mean()
# only the price column is read off disk to answer this

Where it shows up

Interview questions probe whether you understand why column stores are fast, not just that they are — expect "why does Parquet read less data than CSV for a filtered aggregation query" as a follow-up to any data-engineering question, and expect predicate pushdown and partitioning (see Partitioning and File Layout) to come up together, since they compound.

In practice, tick and bar archives, feature stores, and factor libraries are almost universally stored as Parquet (or the closely related columnar formats built on Arrow and In-Memory Formats) precisely because research workflows are column-selective and read-heavy — a quant pulling five years of one factor across a universe of 3,000 names reads a sliver of the total data on disk, not all of it.