Semi-Supervised Learning
Labels are often the scarce resource, not data — a firm might have millions of unlabeled ticks and only a few thousand confirmed trade outcomes, and semi-supervised learning uses the unlabeled bulk to shape a model that only a sliver of labels directly supervises.
Prerequisites: Logistic Regression, Overfitting
Labeling a trade as "should have been executed differently" or a news article as "moved the stock" takes an analyst's time, which is expensive, while raw ticks, order-book snapshots, and news text arrive by the terabyte for free. Most real datasets look like this: a huge pool of unlabeled examples and a small, hard-won pool of labeled ones. Semi-supervised learning is the family of methods that use the unlabeled majority to help, instead of throwing it away and training only on the small labeled slice.
The unlabeled data doesn't supply answers directly — it supplies structure: where the data actually clusters, what the feature space's natural shape looks like. A model that respects that structure, and is anchored by even a small number of labels, often generalizes better than one trained on the labels alone.
The core idea: self-training
The simplest semi-supervised method is self-training (also called pseudo-labeling): train a model on the small labeled set, use it to predict labels for the unlabeled examples it's most confident about, add those predicted labels to the training set as if they were real, and retrain. Repeated over several rounds, the labeled set grows using the model's own confident guesses.
This only works because of an assumption about the data, not a mathematical guarantee: it assumes points that are close together in feature space usually share a label (the cluster assumption). If that assumption holds, a model trained on a handful of true labels can identify dense clusters using the unlabeled data, and confidently propagate the nearest true label to the rest of each cluster. If it doesn't hold — if the true label boundary cuts straight through a dense region — self-training can just as easily amplify the model's early mistakes, since a wrong pseudo-label gets treated exactly like a real one in the next round.
Worked example
A desk has 4,000 confirmed labels for "was this a genuine fat-finger trade" out of 2 million total historical trades, most unlabeled. A classifier trained on just the 4,000 labeled trades achieves 71% accuracy on a held-out labeled set. Applying self-training — predicting on the 2 million unlabeled trades, keeping only the 60,000 predictions the model is over 95% confident in, adding those as pseudo-labels, and retraining — raises held-out accuracy to 79%, because the newly-added 60,000 examples fill in regions of feature space (specific combinations of size, venue, and time-of-day) that the original 4,000 labels barely covered. A second self-training round adds diminishing returns and a third actually lowers accuracy slightly, because by then the pseudo-labels are reinforcing the model's own blind spots rather than adding new information.
What this means in practice
Semi-supervised learning is worth trying whenever labels are the bottleneck and there's reason to believe the cluster assumption roughly holds — trade classification, news-sentiment tagging, order-type detection. It is not a substitute for getting more real labels when the true decision boundary is genuinely ambiguous or cuts through dense regions, since self-training will confidently reinforce whatever mistake the small labeled set already encoded.
Always validate a self-trained model against a labeled holdout that was never touched by pseudo-labeling — measuring accuracy on pseudo-labeled data is circular, since the model is partly being checked against its own earlier predictions.
Related concepts
Practice in interviews
Further reading
- Chapelle, Schölkopf & Zien (eds.), Semi-Supervised Learning (2006)
- Zhu, 'Semi-Supervised Learning Literature Survey' (2005)