Choosing the Number of Clusters

K-means clustering needs the number of clusters, $k$, handed to it before it starts — it has no built-in way to discover the "right" number on its own. And a naive fix, "pick the $k$ that minimizes within-cluster distance," fails immediately: within-cluster distance keeps shrinking every time $k$ goes up, hitting zero once $k$ equals the number of data points, where every point is its own one-member cluster. That's a perfect fit and a useless answer, so choosing $k$ needs a rule that penalizes complexity, not one that just rewards fit.

Two ways to find k

The elbow method plots within-cluster sum of squared distances (inertia) against $k$. Inertia always decreases as $k$ grows, but early on each new cluster explains a lot of remaining structure, and past some point it explains very little — the curve bends from a steep decline to a shallow one, and that bend (the "elbow") is read off as the reasonable $k$. It's intuitive but the elbow is often ambiguous — real data rarely bends as cleanly as a textbook picture.

The silhouette score is more precise: for each point $i$, compute $a(i)$, its average distance to other points in its own cluster, and $b(i)$, its average distance to points in the nearest other cluster. The silhouette for that point is $s(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}$. In words: it's the gap between "how far am I from my neighbors" and "how far am I from the nearest rival cluster," scaled to sit between -1 and +1. A point near +1 sits comfortably inside a tight, well-separated cluster; a point near 0 sits right on a boundary between two clusters; a negative value means the point is actually closer to a different cluster than its own — a sign it was probably misassigned. Averaging $s(i)$ across all points for a given $k$ gives a single comparable score, and the $k$ with the highest average silhouette is the pick.

Worked example

Clustering 500 stocks by their trailing 60-day return correlations, inertia drops sharply from $k=2$ to $k=4$, then only gradually from $k=4$ onward — a visible but not perfectly crisp elbow around $k=4$. Computing the average silhouette score across $k = 2$ through $8$ gives: 0.31, 0.38, 0.44, 0.40, 0.35, 0.33, 0.30 — a clear peak at $k=4$, confirming the elbow's rough read with an actual number. Choosing $k=4$ groups the 500 stocks into four correlation-based clusters that, on inspection, roughly line up with cyclicals, defensives, high-beta growth, and rate-sensitive financials — clusters a portfolio manager can sanity-check against sector intuition.

What this means in practice

In finance, "the right number of clusters" is often as much a business decision as a statistical one — a risk model built for position sizing might want 6-10 clusters to be actionable, even if the silhouette score technically favors 4. Treat the silhouette score and the elbow as evidence to weigh, not an automatic answer, and always look at what the resulting clusters actually contain before committing to a $k$.