TabNet: Attentive Tabular Learning

Neural networks are excellent at images and text, but a plain spreadsheet row — a mix of valuation ratios, sector codes, and momentum scores — has no spatial or sequential structure for a convolution or a recurrence to exploit, and a single dense layer that looks at every column at once tends to lose track of which columns actually mattered. Gradient-boosted trees have long dominated this setting partly because they naturally pick a small number of relevant columns at each split. TabNet tries to give a neural network that same habit.

The analogy: a careful analyst, several passes

Instead of skimming a spreadsheet once and forming a snap judgment, imagine an analyst who deliberately re-reads it several times, each pass choosing to focus on a different, small subset of columns based on what the previous passes already revealed — first checking valuation, then, having noted it's cheap, checking whether momentum confirms it. TabNet formalizes this as sequential attention: a fixed number of decision steps, each producing a sparse "which columns matter right now" mask over the input features.

The mechanics

At decision step $k$, a learned mask $M_k$ selects a subset of the $D$ input features:

$$M_k = \text{sparsemax}\big(P_{k-1} \cdot h_k(a_{k-1})\big)$$

In words: a small network $h_k$ looks at the previous step's leftover attention $a_{k-1}$ and proposes new feature scores; sparsemax (a relative of softmax that can output exact zeros) turns those scores into a mask where most columns get exactly zero weight and a few get nonzero weight; $P_{k-1}$, a running "prior scale," discourages reusing the same columns every step. The selected features $M_k \odot x$ feed a small transformer-like block whose output both contributes to the final prediction and informs the next step's mask.

Worked example 1: two decision steps over four features

Four features: P/E, momentum, sector, size. Step 1 mask (sparsemax output): $M_1 = (0.6, 0, 0.4, 0)$ — the model attends mostly to P/E, partly to sector, ignoring momentum and size entirely this step. Step 2, having already used P/E and sector, mask: $M_2 = (0, 0.7, 0, 0.3)$ — now attends to momentum and size instead, because the prior-scale term down-weights repeats.

Worked example 2: combining steps into a decision

Suppose each step's small block outputs a scalar contribution: step 1 contributes $0.8$, step 2 contributes $-0.3$. TabNet sums contributions across all $K$ steps before a final activation: $0.8 + (-0.3) = 0.5$. Because each contribution is tagged with the mask that produced it, you can report exactly which features drove each part of that $0.5$ — P/E and sector pushed it up in step 1, momentum and size pulled it down in step 2.

What this means in practice

Because the feature masks are computed per row, TabNet gives per-instance interpretability that gradient-boosted trees only offer in aggregate — useful when a compliance or risk team needs to know why this specific prediction fired, not just which features matter on average across the whole model. It also handles mixed categorical and continuous tabular features (sector codes alongside valuation ratios) natively.