Label Noise and Learning with Noisy Targets

Most supervised learning assumes the training labels are ground truth — the "correct" answer the model should try to reproduce. In finance that assumption is often false in a way it rarely is for, say, labelling cats in photos. A label like "this trade would have been profitable" or "this was a momentum regime" is frequently a noisy proxy for something unobservable, corrupted by execution slippage that wasn't modelled, a labelling rule that's right on average but wrong on individual cases, or randomness in the market outcome itself that has nothing to do with the input features. A model trained to match those labels exactly is, in part, training itself to reproduce noise.

The analogy: grading essays from an inconsistent teacher

Imagine training a student to predict essay grades by studying a large stack of essays graded by a teacher who is usually fair but occasionally distracted, grading identical-quality essays a full letter grade apart depending on the day. A student who studies hard enough to predict every single grade in the stack perfectly, including the teacher's off days, hasn't learned what makes a good essay — they've learned to reproduce the teacher's inconsistency. A student who instead learns the general pattern, tolerating the fact that a few grades in the stack don't quite fit the pattern, ends up with the better and more transferable understanding, even though they "get more wrong" on the training stack itself.

How noise gets in, and what it costs

Label noise in financial ML commonly comes from three sources: outcome noise (the true target has irreducible randomness — a coin-flip-like outcome baked into the label itself), rule-based mislabeling (an automated labelling heuristic, like "up more than X% in Y days," misclassifies edge cases), and delayed or corrupted ground truth (the label depends on a future realisation — an earnings surprise, a fill price — that's itself measured imperfectly). For a binary label flipped independently with probability $\eta$ (the noise rate), a model fit by minimising the ordinary training loss on the noisy labels doesn't converge to the true decision boundary; it converges to a boundary systematically biased toward the noise. A model with high enough capacity to reach zero training error on noisy labels has, by construction, memorised every flipped label along with the real pattern.

Worked example: noise rate versus achievable accuracy

Suppose the true relationship between features and outcome is genuinely learnable to 85% accuracy at best — some irreducible randomness remains even with a perfect model. Now corrupt the training labels by randomly flipping 10% of them ($\eta = 0.10$).

A model that fits the noisy training labels perfectly (100% training accuracy) has, by definition, matched every flipped label too — it has memorised 10% pure noise as if it were signal. Evaluated against the true, uncorrupted labels on new data, that same model's accuracy typically falls well below the 85% ceiling, often into the 60–70% range for a high-capacity model, because a meaningful share of its learned decision boundary was shaped by noise that doesn't generalise. A more heavily regularised model that only reaches 88% accuracy on the noisy training labels — worse-looking on paper — can still land close to the achievable 85% on true out-of-sample labels, because it never had the capacity to fit the flipped 10% precisely in the first place. The model that "did worse" on the label you gave it did better on the label you actually cared about.

What this means in practice

The practical defences mirror the defences against overfitting generally, but applied with the explicit understanding that some fraction of "errors" on the training set are actually the model being right and the label being wrong: stronger regularization and early stopping so the model doesn't have the freedom to chase every training point, loss functions less sensitive to individual bad labels (e.g. reweighting or clipping the contribution of anomalously high-loss training examples, which are disproportionately likely to be mislabeled), and cross-validation that measures performance against held-out labels rather than assuming the training labels are perfect ground truth. Where possible, going back to the label-construction rule itself and tightening it — using a longer holding period to reduce outcome noise, or excluding edge cases the labelling heuristic handles badly — fixes the problem at the source rather than asking the model to work around it.

It's worth distinguishing label noise from the closely related idea of feature noise, since the fixes differ. Feature noise corrupts the inputs the model sees; label noise corrupts the answer key it's graded against. A model can, in principle, learn to ignore noisy features by discovering they're uninformative, but it has no way to distinguish "this label is wrong" from "this label reveals a genuine pattern I haven't found yet" — from the model's point of view, a mislabeled point and a hard-but-correct edge case look identical, which is exactly why guarding against label noise has to happen through regularization and loss design rather than through anything the model can figure out on its own.