t-SNE

PCA compresses a high-dimensional dataset into 2D by finding the directions of greatest variance, and it is honest about global structure — distances and angles in the 2D picture still mean roughly what they meant in the original space. t-SNE throws that honesty away on purpose. It is built for one job only: take a dataset with hundreds of dimensions and produce a 2D scatter plot where points that were close neighbors originally stay close together in the picture, so a human can eyeball clusters that would otherwise be invisible. Everything else — absolute distances, cluster sizes, the meaning of the axes — is not preserved and should not be trusted.

The core idea: match neighbor probabilities, not distances

For every pair of points in the original high-dimensional space, t-SNE computes a probability that one would pick the other as a neighbor, using a Gaussian centered on each point — nearby points get high neighbor-probability, far points get near-zero. It then places points in 2D and computes an analogous neighbor-probability there too, but using a heavier-tailed distribution (a Student's t-distribution, which is where the "t" in t-SNE comes from) so that points which are moderately far apart in 2D are not forced unnaturally close just to satisfy the high-dimensional probabilities. t-SNE then nudges the 2D points, iteratively, to make the 2D neighbor-probabilities match the original high-dimensional neighbor-probabilities as closely as possible. Points that were neighbors in 200 dimensions get pulled together in 2D; points that were not stay apart. Nothing in this process asks the 2D distances to correspond to actual high-dimensional distances — only the ranking of who is whose neighbor is being preserved.

A worked sketch: why clusters can still lie about size

Suppose 300 stocks are described by 50 fundamental and price features each, and there really are two natural groups in that space: one tight cluster of 250 similar large-cap names, and one loose, spread-out cluster of 50 heterogeneous small-cap names. Feed this into t-SNE and it will typically render two visually separated blobs — the neighbor structure genuinely says these are two different groups. But the size and spacing of the two blobs is not reliable evidence about the size and spacing of the groups in the original 50 dimensions: t-SNE tends to inflate the visual size of tight clusters and compress loose ones, because it is only ever trying to match neighbor-probabilities, not preserve scale. Two blobs of similar visual diameter can correspond to a dense original cluster and a sparse one — the plot alone cannot tell you which.

The Gaussian and t-distribution kernels t-SNE uses to convert distances into neighbor-probabilities are shaped like fast-decaying curves such as this one: a small distance gets a large probability weight, and probability collapses toward zero for anything beyond a short range. That collapse is exactly why only local neighbor structure survives the compression — the far tail simply doesn't register.

What this means in practice

t-SNE is a diagnostic, not a feature-engineering step: run it on a large basket of stocks' feature vectors and it will often reveal groupings a covariance matrix would miss (nonlinear clusters that PCA's linear rotation cannot separate). It is far too unstable and slow to embed as an actual model input — the mapping is not linear, changes every time you rerun it with a different random seed, and provides no formula for placing a brand-new stock into the same map without recomputing everything from scratch.