FT-Transformer for Tabular Data

An ordinary neural network layer squashes an entire tabular row into one vector and processes it all at once, which makes it hard for the model to represent a specific interaction — like "cheap valuation combined with this particular sector" — without a human first engineering that cross-term by hand. Transformers, originally built to let one word in a sentence attend to every other word, offer a different route: what if every column got to explicitly attend to every other column?

The analogy: a sentence made of columns

Treat each column of a row — P/E, momentum, sector, size — as if it were a word in a short sentence. Self-attention lets each "word" ask every other "word" how relevant it is to itself, and blend information accordingly, before anything gets combined into a final answer. FT-Transformer (Feature Tokenizer + Transformer) does exactly this: it turns each feature into its own token embedding, adds a special summary token, and runs the whole thing through an ordinary transformer encoder.

The mechanics

Each feature value $x_j$ is mapped to its own embedding vector (numeric features via a learned linear map, categorical features via a lookup table), plus one extra learned [CLS] token that will hold the final summary. All tokens attend to each other via scaled dot-product attention:

$$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V$$

In words: for every pair of feature-tokens, compute how relevant one is to the other (the $QK^\top$ term), turn those relevance scores into weights that sum to one (softmax), and blend the tokens' values ($V$) by those weights. After several such layers, the [CLS] token's final vector — now informed by every column, weighted by how relevant each was — feeds a small prediction head.

Worked example 1: one attention score by hand

Toy 2-D embeddings for two feature tokens, P/E: $q=(1,0)$ and sector: $k=(0,1)$, scaling by $\sqrt{d}=\sqrt{2}\approx1.41$. Raw score $q\cdot k = 1(0)+0(1)=0$; after scaling still $0$. Now compare P/E to momentum, $k'=(1,1)$: raw score $q\cdot k' = 1$, scaled $1/1.41\approx0.71$. Passed through softmax alongside the other scores, momentum ends up receiving noticeably more attention weight from P/E's token than sector does — the model has learned P/E and momentum are more relevant to each other than P/E and sector are.

Worked example 2: what attention buys over a plain MLP

A plain MLP given P/E and sector as two separate input dimensions can only combine them if training happens to discover a useful weighted sum or if a human manually adds a "P/E × sector-dummy" interaction feature beforehand. FT-Transformer's attention layer computes, for every row, a data-dependent weighting between the two tokens on the fly — the equivalent interaction term is learned per-attention-head rather than hand-engineered once for the whole dataset.

What this means in practice

FT-Transformer is a genuine option when feature interactions matter and the dataset is large enough to train attention weights reliably; it also composes naturally with other modalities (text, images) in the same model, since everything is already a token. It typically needs more data and compute than a tree ensemble to reach the same accuracy on a small-to-medium tabular dataset.