Graph Attention Networks (GAT)

A graph convolutional network weights a neighbor's contribution by a rule fixed in advance — one over the square root of the two nodes' degrees, computed purely from graph structure, with no regard for what those nodes actually contain. But two suppliers of a company might have identical degree and still matter very differently: one might be a critical single-source supplier of a key component, the other a replaceable commodity vendor. Degree alone can't distinguish them. A graph attention network instead learns how much weight each neighbor deserves, computed from the content of the two nodes' feature vectors, exactly the way self-attention lets one word learn how much to attend to another based on what they actually mean, not just their position in the sentence.

The analogy: a manager weighing advisors by relevance, not headcount

A portfolio manager consults several analysts before making a call. A GCN-style rule would weight every analyst's opinion by some fixed organizational fact, like how many other people report to them — irrelevant to whether their specific opinion is useful right now. A GAT-style manager instead judges, case by case, how relevant each analyst's specific input is to the specific decision at hand: the credit analyst's view matters enormously when the question is about default risk and much less when the question is about a rate-sensitivity trade. The weight given to each advisor is computed fresh for each decision, based on the content of what's being asked and what each advisor actually knows — not a fixed number set by the org chart.

The mechanism: a learned compatibility score per edge

For an edge between node $v$ and neighbor $u$, a GAT first computes a raw, unnormalized attention score:

$$e_{vu} = \text{LeakyReLU}\Big(a^{\top} \big[W h_v \,\|\, W h_u\big]\Big)$$

Here $h_v$ and $h_u$ are the current feature vectors of nodes $v$ and $u$, $W$ is a shared learned weight matrix that transforms both into a common space, $\|$ means concatenation — stitching the two transformed vectors together end to end into one longer vector, $a$ is a learned vector of weights that turns that concatenated vector into a single number, and LeakyReLU is a mild nonlinearity. In words: transform both nodes' features, glue the two transformed vectors together, and pass the result through one small learned scoring function to get a single number saying "how relevant is $u$ to $v$ right now." This score depends on the actual content of both nodes, unlike a GCN's purely structural degree weight.

These raw scores are then normalized with softmax across all of $v$'s neighbors, so they sum to one and behave like proper weights:

$$\alpha_{vu} = \text{softmax}_u(e_{vu}) = \frac{\exp(e_{vu})}{\sum_{w \in \mathcal{N}(v)} \exp(e_{vw})}$$

In words: exponentiate every neighbor's raw score and divide by the sum across all neighbors, exactly the same softmax normalization used in ordinary self-attention. Finally, the node's update is the attention-weighted sum of its (transformed) neighbors:

$$h_v^{(k+1)} = \sigma\left(\sum_{u \in \mathcal{N}(v)} \alpha_{vu}\, W h_u\right)$$

which looks structurally identical to a GCN's update, with one crucial difference: $\alpha_{vu}$ is learned and content-dependent, not a fixed function of degree.

Worked example 1: computing one attention weight by hand

Simplify to a 1-dimensional feature per node, with $W=1$ (identity), so $Wh_v = h_v$ directly. Node $v$ has $h_v = 2.0$, neighbor $u_1$ has $h_{u_1}=3.0$, neighbor $u_2$ has $h_{u_2}=0.5$. Suppose the learned scoring function collapses (after concatenation and $a$) to $e_{vu} = h_v \times h_u$ for simplicity of the arithmetic. Then $e_{vu_1} = 2.0 \times 3.0 = 6.0$ and $e_{vu_2} = 2.0 \times 0.5 = 1.0$.

Softmax: $e^{6.0}=403.4$, $e^{1.0}=2.72$, sum $=406.2$. So $\alpha_{vu_1} = 403.4/406.2 = 0.993$ and $\alpha_{vu_2} = 2.72/406.2 = 0.007$. Despite both being direct neighbors with the same graph-structural role, node $u_1$ receives essentially all the attention weight because its content is far more compatible with $v$'s — this couldn't happen in a GCN, where two neighbors of equal degree always get identical structural weight regardless of content.

Worked example 2: the update after weighting

Continuing the example, suppose $h_{u_1}=3.0$ and $h_{u_2}=0.5$ are the values being aggregated (same as above, $W=1$). The node's update is

$$h_v^{(k+1)} = 0.993(3.0) + 0.007(0.5) = 2.979 + 0.0035 = 2.983$$

The result, $2.983$, sits almost exactly at $u_1$'s value — node $v$'s new representation has essentially adopted its most relevant neighbor's information and nearly ignored the other, a sharp, content-driven decision that a fixed degree-based rule could never make, since a GCN would have blended both neighbors according to structure alone, regardless of how relevant either one actually was.

What this means in practice

GATs are the right tool when a graph's edges are structurally identical but not equally important — every supplier link looks the same in the raw graph, but a single-source critical supplier should influence a company's risk score far more than a commodity vendor with ten alternatives, and only the actual features of the two connected nodes can reveal that difference. Multi-head attention (running several independent attention computations per edge and combining them, exactly as in transformer self-attention) is standard in GATs too, letting the model learn several different notions of "relevance" simultaneously — say, financial exposure and operational dependence as two separate attention heads.

Practice

1. Two neighbors of node $v$ have identical degree but very different feature vectors. Explain, step by step, why a GCN would weight them identically while a GAT would not.
2. If the learned scoring function $a^\top[Wh_v \| Wh_u]$ were replaced with a constant (say, always outputting 1 regardless of $h_v, h_u$), what would the resulting attention weights $\alpha_{vu}$ reduce to, and how would the GAT then compare to an unweighted average?
3. Why does GAT normalize attention scores per node (softmax over just that node's neighbors) rather than across the entire graph at once?