Bias Initialization for Imbalanced Outputs

When a neural network's target is rare — say, predicting whether a trade will be flagged as a fat-finger error, which happens in a fraction of a percent of trades — a network initialized in the standard way starts by outputting predictions near 50% for everything. Since the true rate is under 1%, the very first several rounds of training are spent doing nothing more useful than pushing every output down toward the base rate, before the network can even start learning which examples are actually different from the average.

The fix is to initialize the final layer's bias term directly to the value that reproduces the base rate, rather than leaving it at zero. If a fraction $p$ of examples are positive, the bias is set so the network's output before seeing any input already equals $p$ — for a sigmoid output layer, that means setting the bias to $\log(p / (1-p))$. Training then starts from a sensible baseline and its early gradient steps go toward learning actual patterns instead of correcting a badly wrong starting guess.

Worked example

A model predicting a rare fill-quality flag has a true positive rate of 0.4%. Instead of leaving the output bias at zero (which gives a starting prediction of 50%), it's initialized to $\log(0.004/0.996) \approx -5.5$, so the sigmoid output starts at roughly 0.4% for every example. Loss on the very first training batch is dramatically lower than with zero initialization, and training converges faster because it isn't burning early epochs just discovering the base rate.