Truncated SVD Features
Truncated SVD compresses a large, often sparse feature matrix into a small number of dense components that capture most of its structure, commonly used to shrink text or one-hot encoded data before modeling.
Singular value decomposition (SVD) breaks a matrix down into components ranked by how much of the data's structure each one captures. Truncated SVD keeps only the top handful of those components — say, the top 50 out of thousands — discarding the rest, which typically hold mostly noise. The result is a small set of dense numeric features that summarize most of the useful variation in a much larger, often sparse matrix.
It's especially common on text data (a term-frequency matrix with tens of thousands of word columns) or heavily one-hot encoded categorical data, both of which produce huge, sparse matrices that are impractical to feed directly into many models. Truncated SVD compresses these into, say, 50 or 100 dense components that a downstream model can use far more efficiently, at the cost of losing some interpretability — each component is a blend of many original columns, not a single recognizable feature.
Truncated SVD trades a huge sparse matrix for a small dense one that keeps most of the signal, making it a standard compression step before modeling text or high-cardinality categorical data — at the cost of individual components no longer being directly interpretable.
Worked example
A dataset of product reviews is turned into a term-frequency matrix with 30,000 word columns, almost all zero for any given review. Applying truncated SVD to keep the top 100 components compresses each review down to just 100 dense numbers that capture most of the meaningful variation in word usage, letting a downstream classifier train far faster than it could on the original 30,000-column sparse matrix.
Related concepts
Practice in interviews
Further reading
- Hastie, Tibshirani & Friedman, The Elements of Statistical Learning (ch. on dimension reduction)