Siamese and Twin-Tower Networks

Suppose you want to tell whether two return-pattern charts, two signatures, or two chunks of research text describe "the same thing," but the set of possible pairs is effectively infinite — you can never train an ordinary classifier with one output per pair. What you actually need is a notion of distance: a network that turns any single input into a point in space, such that similar inputs land near each other and dissimilar ones land far apart, regardless of whether it has ever seen this exact pair before.

The analogy: one grader, two exams

Imagine grading two students' essays with the same rubric, by the same grader, so the two scores are directly comparable even though the grader never compared the essays to each other. A siamese network is exactly that: one network architecture, with tied (identical, shared) weights, applied independently to each of two inputs. It is not two separately trained networks that happen to agree — it is literally the same function evaluated twice.

The mechanics

Each input $x_1, x_2$ passes through the shared network $f_\theta$ to produce embeddings $e_1 = f_\theta(x_1)$, $e_2 = f_\theta(x_2)$. A distance, usually Euclidean, $d = \|e_1 - e_2\|$, then feeds a contrastive loss:

$$L = y \, d^2 + (1-y) \, \max(m - d, 0)^2$$

Here $y=1$ means the pair is a genuine match and $y=0$ means it is not, and $m$ is a margin — the minimum distance we insist dissimilar pairs stay apart. In words: for matching pairs, the loss directly penalizes any distance (push them together); for non-matching pairs, the loss only penalizes distances smaller than the margin (push apart only until "far enough," then stop caring).

Worked example 1: a matching pair

Two 2-D toy embeddings for similar return signatures: $e_1 = (1.0, 2.0)$, $e_2 = (1.4, 2.3)$, $y=1$. Distance: $d = \sqrt{(1.4-1.0)^2+(2.3-2.0)^2} = \sqrt{0.16+0.09} = \sqrt{0.25}=0.5$. Loss: $L = 1 \times 0.5^2 = 0.25$. The gradient pushes $e_1, e_2$ closer together.

Worked example 2: a non-matching pair

Now $e_1=(1.0,2.0)$, $e_2=(4.0,2.0)$, $y=0$, margin $m=2$. Distance $d=3.0$. Since $d>m$: $\max(2-3,0)=0$, so $L=0$ — the pair is already far enough apart and contributes nothing to training. If instead $d=1.5$: $\max(2-1.5,0)^2=0.25$, and the network is pushed to separate them further.

What this means in practice

This is how face and signature verification, near-duplicate document detection, and pairs-trading candidate screening all work: instead of training a classifier per pair, you train one embedding function once, then compare any two new items by distance at inference time — including pairs never seen during training. The margin $m$ is a real hyperparameter: too small and dissimilar pairs are barely separated; too large and training wastes effort over-separating pairs that are already distinguishable enough.