Gaussian Process Classification
A classifier that outputs a full probability distribution over its predictions, giving a built-in sense of how confident it is, rather than just a single decision boundary.
Most classifiers output a single number that looks like a probability but isn't really calibrated — a logistic regression's 0.9 doesn't necessarily mean "90% chance correct" in any precise sense. Gaussian process classification takes a different approach: it treats the underlying decision function itself as uncertain, described by a full probability distribution over possible functions, and then squashes that distribution through a link function to produce class probabilities that reflect genuine uncertainty about the function's shape, not just about noise.
The practical benefit shows up exactly where training data is sparse: in regions far from any observed examples, a Gaussian process classifier's predicted probability naturally moves toward uncertain (close to 50/50 for two classes) rather than confidently extrapolating, because the underlying function distribution is wide wherever there's no data to pin it down. The tradeoff is cost — inference scales poorly with the number of training points, making it impractical for very large datasets without approximations.
Worked example. Near a region with dense historical data, a Gaussian process classifier predicts a stock-direction probability of 0.85 with a narrow uncertainty band; far from any observed data (an unusual combination of features never seen before), it predicts something close to 0.50 with a wide band, correctly flagging that it's really just guessing.
Gaussian process classification treats the decision function itself as uncertain, so its predicted probabilities widen out toward uncertain in data-sparse regions instead of confidently extrapolating — at the cost of scaling poorly to large training sets.
Related concepts
Further reading
- Rasmussen & Williams, Gaussian Processes for Machine Learning, ch. 3