HDBSCAN

DBSCAN finds clusters by density — a point belongs to a cluster if enough other points sit within some radius $\varepsilon$ of it — and unlike k-means it never assumes clusters are round or forces every point into one. But it needs a single $\varepsilon$ for the entire dataset, and financial data rarely obliges: a group of large-cap tech stocks might move together tightly (dense) while a group of small-cap energy names moves together loosely (sparse). One $\varepsilon$ that fits the tight cluster will split the loose one into noise; one that fits the loose cluster will merge the tight one into everything nearby.

Building and pruning the hierarchy

HDBSCAN starts by re-scaling distances using mutual reachability distance, which inflates the distance between two points if either sits in a sparse region — so an isolated point is pushed even further from everything else before clustering starts. It then builds a minimum spanning tree on these adjusted distances and progressively cuts the longest edges, producing a hierarchy: at a loose threshold everything is one cluster, and as the threshold tightens, that cluster splits into smaller ones, some of which keep splitting further and some of which just shrink and vanish into noise.

The last step is cluster stability selection: for every candidate cluster in that hierarchy, HDBSCAN measures how large a range of thresholds it survives across before either splitting or disappearing. Clusters that persist across a wide range are kept as the final answer; clusters that only ever existed briefly, at one narrow threshold, are absorbed back into their parent or discarded as noise. This is what lets a tight cluster and a loose cluster both survive in the same final result — each is evaluated on its own persistence, not against a shared, dataset-wide radius.

Worked example

Clustering 800 global equities by co-movement, plain DBSCAN with $\varepsilon = 0.15$ finds a clean 40-stock cluster of mega-cap semiconductor names (tightly correlated) but lumps 300 unrelated mid-caps into one giant blob, because $\varepsilon = 0.15$ is loose enough to bridge them. Tightening to $\varepsilon = 0.08$ breaks up the blob sensibly but now shatters the semiconductor cluster into six fragments, since 0.08 is too strict for how those 40 stocks actually co-move. Running HDBSCAN on the same data (with min_cluster_size = 15) recovers the 40-stock semiconductor cluster intact — because it's dense and stable across a narrow $\varepsilon$ range — while also splitting the mid-cap blob into several sensible, looser sector groupings, because each is separately stable at its own, wider $\varepsilon$ range.

What this means in practice

HDBSCAN is the more forgiving default whenever cluster density varies across a dataset, which is the normal case in cross-sectional finance — large caps, small caps, and different sectors simply don't co-move with the same tightness. Its main cost is a less interpretable single knob: min_cluster_size replaces $\varepsilon$, and it controls the smallest group HDBSCAN is willing to call a cluster rather than noise, which takes some experimentation to set well.