Gaussian Process Classification

Most classifiers output a single number that looks like a probability but isn't really calibrated, a logistic regression's 0.9 doesn't necessarily mean "90% chance correct" in any precise sense. Gaussian process classification takes a different approach: it treats the underlying decision function itself as uncertain, described by a full probability distribution over possible functions, and then squashes that distribution through a link function to produce class probabilities that reflect genuine uncertainty about the function's shape, not just about noise.

The practical benefit shows up exactly where training data is sparse: in regions far from any observed examples, a Gaussian process classifier's predicted probability naturally moves toward uncertain (close to 50/50 for two classes) rather than confidently extrapolating, because the underlying function distribution is wide wherever there's no data to pin it down. The tradeoff is cost, inference scales poorly with the number of training points, making it impractical for very large datasets without approximations.

Worked example. Near a region with dense historical data, a Gaussian process classifier predicts a stock-direction probability of 0.85 with a narrow uncertainty band; far from any observed data (an unusual combination of features never seen before), it predicts something close to 0.50 with a wide band, correctly flagging that it's really just guessing.