Transformers for Cross-Sectional Return Prediction

A standard cross-sectional model predicts each stock's return from its own features — momentum, valuation, earnings surprise — independently, as if each stock existed in a vacuum with no relationship to the others on the same day. In reality a stock's expected return often depends on its peers: what its sector is doing, how correlated names have been moving, whether it looks like an outlier relative to similar companies right now.

Applying the transformer's attention mechanism to this problem means letting the model compute, for each stock, a weighted combination of information from every other stock in that day's cross-section, with the weights (the "attention") learned from the data rather than fixed in advance by a sector classification. A stock's prediction can end up implicitly conditioned on how its five most-similar peers (by learned attention weight, not necessarily an analyst's sector label) are behaving that day, capturing relationships a per-stock independent model structurally cannot see.

This comes at real cost: attention scales with the square of the number of stocks being compared at once, is more data-hungry than tree-based cross-sectional models, and its added benefit over simpler cross-sectional normalization or explicit sector-neutral features is empirically mixed — several studies find transformer architectures don't reliably beat well-tuned gradient boosting or even linear baselines on this exact task, making the added complexity a genuine tradeoff rather than an automatic upgrade.