Probabilistic PCA
Probabilistic PCA reframes ordinary PCA as a statistical model with explicit noise, which lets it handle missing data, give confidence estimates, and be embedded inside larger probabilistic models.
Ordinary PCA finds the directions of greatest variance in data through pure linear algebra — there's no notion of noise or uncertainty built in. Probabilistic PCA rewrites the same idea as a generative statistical model: each observed data point is assumed to be a few latent (hidden) factors, mapped linearly into the observed space, plus Gaussian noise. Fitting the model with maximum likelihood recovers essentially the same directions ordinary PCA finds, but now wrapped in a full probability distribution.
Probabilistic PCA treats each data point as latent factors passed through a linear map plus random noise, so unlike ordinary PCA it comes with an explicit noise model — which is what lets it handle missing values, generate new synthetic samples, and report how confident it is in each reconstruction.
What the probability model buys you
Because it is a proper generative model, probabilistic PCA can be fit with the EM algorithm even when some data entries are missing — the missing entries just get treated as extra latent variables to infer alongside the true latent factors, something ordinary PCA has no natural way to do. It also gives a likelihood, which lets you compare different numbers of latent dimensions using a proper statistical criterion rather than eyeballing a scree plot.
Worked example
A dataset of 500 customers has 20 features, but 5% of entries are missing due to survey non-response. Fitting probabilistic PCA with 3 latent factors via EM alternates between estimating the missing entries given the current model and re-estimating the model given the filled-in data, converging to both a complete dataset and 3 latent factors — a straightforward extension that plain PCA's eigenvector approach can't offer without first imputing the missing values by some separate, ad hoc method.
As the assumed noise variance shrinks toward zero, probabilistic PCA's solution converges to ordinary PCA exactly — it isn't a wholly different technique, just PCA with an explicit probabilistic scaffold added around it, so don't expect dramatically different components on clean, complete data.
Related concepts
Practice in interviews
Further reading
- Tipping and Bishop, 'Probabilistic Principal Component Analysis'