Factorization Machines

Modeling trade execution cost from categorical features like ticker, venue, and time bucket, each with hundreds of values, a linear model captures each feature's own effect but misses interactions — a ticker that's expensive specifically on one venue. Adding an explicit parameter for every pair means thousands of new parameters, most seen in only a handful of rows — too noisy to estimate directly. Factorization machines (FMs) capture pairwise interactions without that cost.

The analogy: describing people by traits, not every possible pairing

Instead of memorizing a compatibility score for every pair of people at a party — hopeless for pairs who've never met — describe each person by a short list of traits. Two people's compatibility can then be estimated from how their trait profiles line up, without ever needing data on that exact pair. FMs do this with features: a short embedding vector per feature value, and interactions estimated from how embeddings line up.

The mechanics: an embedding per feature, interactions from dot products

A linear model is $\hat y = w_0 + \sum_j w_j x_j$. An FM adds pairwise terms, but factorizes each interaction weight into a dot product of learned embeddings $v_i, v_j \in \mathbb{R}^k$:

$$\hat y = w_0 + \sum_j w_j x_j + \sum_{i<j} \langle v_i, v_j\rangle \, x_i x_j .$$

In plain English: every feature value gets its own short vector, and the interaction strength between any two values is the dot product of their vectors — high if they point in similar directions. A pair never needs to have co-occurred often for its interaction to be estimated: it's estimated indirectly, through each embedding fit across all rows that value appears in.

Worked example: estimating an unseen ticker-venue combination

"Ticker = XOM" has learned embedding $(0.8, -0.2)$ from its many rows across various venues; "venue = IEX" has embedding $(0.6, 0.1)$ from its rows across various tickers — but XOM has never been observed on IEX. The interaction is estimated anyway: $\langle(0.8,-0.2),(0.6,0.1)\rangle = 0.48-0.02=0.46$, borrowed entirely from how XOM behaves elsewhere and how IEX behaves with other tickers. A model with one free parameter per pair has no way to estimate this combination at all.

Each feature's embedding is a point in the space above; interaction strength is how aligned two such points are — the same dot-product geometry the transform's axes illustrate.

What this means in practice

Factorization machines fit exactly the high-cardinality categorical data common in market microstructure — many tickers, many venues — where explicit pairwise terms would be too numerous and sparse, but embeddings can share statistical strength across related rows. Reach for them once you suspect specific feature combinations, not just individual features, matter.