Clustering for Portfolios

Classical portfolio construction feeds a covariance matrix into a mean-variance optimizer and gets back weights. The trouble is that the optimizer treats every asset as competing with every other one at once, so tiny errors in the estimated correlations produce wildly unstable, concentrated weights, the notorious instability of Markowitz. Clustering offers a gentler path. Instead of solving one giant optimization, you first group assets that move alike, then allocate across and within groups. The structure you discover is intuitive, robust to noise, and needs no matrix inversion.

The core insight: assets are not a flat list, they have a hierarchy. Tech stocks resemble each other more than they resemble banks; banks cluster together; commodities sit apart. Clustering makes that tree explicit, and once you can see the groups, sensible diversification follows almost automatically.

From correlation to distance

To cluster, you need a notion of "how far apart" two assets are. Correlation runs from $-1$ to $+1$, so convert it to a proper distance that is zero for perfectly correlated assets and large for uncorrelated or opposite ones:

$$d_{ij} = \sqrt{\tfrac{1}{2}\,(1 - \rho_{ij})}.$$

Here $\rho_{ij}$ is the correlation between assets $i$ and $j$, and $d_{ij}$ is the distance between them. Two assets with $\rho = 1$ have distance 0 (identical), $\rho = 0$ gives $d = \sqrt{0.5} \approx 0.71$, and $\rho = -1$ gives $d = 1$ (maximally far). With distances in hand, hierarchical clustering repeatedly merges the two closest items, then the next closest, building a tree, a dendrogram, from individual assets up to one big cluster.

Worked example: four assets

Take two tech names (A, B) and two banks (C, D). Suppose the within-group correlation is 0.8 and the cross-group correlation is 0.2. Convert:

• Within a group ($\rho = 0.8$): $d = \sqrt{0.5(1 - 0.8)} = \sqrt{0.1} \approx 0.32$.
• Across groups ($\rho = 0.2$): $d = \sqrt{0.5(1 - 0.2)} = \sqrt{0.4} \approx 0.63$.

Hierarchical clustering merges A with B first (distance 0.32) and C with D first (also 0.32), then joins the two clusters only at distance ~0.63. Cutting the tree at any level between 0.32 and 0.63 gives exactly two clusters: {A, B} and {C, D}, which matches the economics.

Now allocate. A naive equal-weight scheme puts 25% in each name, but because A and B are nearly the same bet, that quietly overweights tech. A cluster-aware scheme splits the risk across the two clusters first (50% tech, 50% banks), then within each (25% A, 25% B), so the tech and bank bets are balanced. If instead there were three tech names and one bank, equal-weighting would put 75% of the portfolio into a single correlated bet, exactly the concentration clustering is designed to prevent. This across-then-within logic is the heart of Hierarchical Risk Parity (López de Prado).

Why practitioners like it

• No matrix inversion. Mean-variance optimization inverts the covariance matrix, which amplifies estimation error; clustering only uses the correlations to build a tree, so it is far more stable out of sample.
• Interpretable. You can look at the dendrogram and see why assets grouped, unlike the opaque weights an optimizer spits out.
• Robust to noisy estimates. Small correlation errors rarely change the tree's coarse structure, so the allocation moves smoothly rather than lurching.

Pitfalls

• Linkage choice matters. How you define the distance between two clusters (single, complete, average, or Ward linkage) changes the tree. Single linkage chains assets into stringy clusters; Ward or average linkage usually gives more balanced, sensible groups.
• Correlations are unstable and regime-dependent. In a crash, everything correlates toward 1 and the clusters collapse, precisely when you needed diversification most. Estimate correlations robustly and refresh them.
• How many clusters? Cutting the tree at different heights gives different group counts; there is no single right answer, use the gap in merge distances (a big jump signals a natural cut) rather than a magic number.
• Look-ahead. Building the tree on the full sample and then backtesting leaks future correlations. Refit the clustering inside each rebalance window.

Clustering sits alongside dimensionality reduction as the two main ways to tame a wide, correlated universe: one groups the assets, the other rotates them into components. Both start from the same admission, markets have far fewer independent bets than they have tickers.