Two-Tower Retrieval Architectures

Given one query — say, a target factor exposure — and a universe of a million candidate stocks, how do you rank all million by relevance without running an expensive joint model on all million query-candidate pairs, every single time a query arrives? Comparing every pair directly is far too slow to do live. You need to do the expensive work once, in advance, and leave only a cheap operation for query time.

The analogy: two catalogs, one shelf coordinate system

A library keeps a subject catalog and a title catalog — built differently, indexed differently — but both ultimately point to the same physical shelf coordinates. Find a book's shelf number from either catalog and matching becomes trivial: just compare coordinates. A two-tower architecture builds exactly this: one tower encodes the query, a separately weighted tower encodes the item, and both are trained so their outputs land in the same shared vector space, even though the towers themselves can differ in architecture.

The mechanics

Query tower $f_q(\text{query}) = e_q$ and item tower $f_i(\text{item}) = e_i$ are trained jointly so that matching pairs score high on a dot product:

$$\text{score}(q, i) = e_q \cdot e_i$$

In words: the relevance of an item to a query is just how aligned their two vectors are — no joint computation between them at inference time. Crucially, because the item tower doesn't depend on the query, every item's embedding can be precomputed once and stored; only the query tower needs to run live, followed by a fast nearest-neighbor lookup.

Worked example 1: ranking three candidates

Query embedding $e_q = (1, 0, 2)$. Three precomputed item embeddings: $e_A=(0,1,1)$, $e_B=(2,0,1)$, $e_C=(1,1,1)$. Dot products: $e_q\cdot e_A = 0+0+2=2$; $e_q\cdot e_B=2+0+2=4$; $e_q\cdot e_C=1+0+2=3$. Ranking: B (4) > C (3) > A (2) — pure arithmetic, no rerun of any network on the pair.

Worked example 2: why precomputation matters at scale

Suppose scoring one query-item pair with a full joint model costs 1 unit of compute, and the universe has $n=1{,}000{,}000$ candidates. A joint (cross-encoder) approach costs $n$ units per query. A two-tower approach precomputes all $n$ item embeddings once (a one-time $n$-unit cost, amortized across all future queries), then each query costs only 1 unit to encode plus a cheap approximate-nearest-neighbor search — turning a per-query cost of a million units into a handful.

What this means in practice

In a factor-screening pipeline, the item tower can encode the entire investable universe overnight into stored vectors; at query time, only the query tower (e.g., encoding "target exposure profile") runs live, followed by an approximate-nearest-neighbor search over the precomputed vectors — the same pattern used by search engines and recommender systems to serve results in milliseconds.