Seed Averaging in Production Models

Train the same neural network architecture on the same data twice, changing only the random seed that initializes its weights and shuffles the training batches, and you'll get two noticeably different models — different weights, and different predictions on new data, purely from randomness in the optimization path rather than anything meaningful about the data. Seed averaging deals with this by training several copies (often 5-20) with different seeds and averaging their output predictions, rather than shipping any single run.

The reasoning is the same as averaging repeated noisy measurements: each seed's model has learned a genuine signal plus some seed-specific noise, and if those noise components are roughly independent across seeds, averaging cancels much of it out while the shared signal survives. A single-seed model predicting next-month stock returns might have meaningfully different top-decile picks than the same architecture trained again from scratch — averaging several seeds' predicted scores before ranking stocks produces a more stable ranking that isn't hostage to one unlucky initialization.

The cost is purely computational: training $k$ seeds costs roughly $k\times$ as long (though runs can go in parallel), for a variance reduction in the ensemble's predictions of roughly $1/k$ under the independent-noise assumption. In production, seed averaging is often cheaper than more elaborate ensembling because every seed shares the same architecture, features, and hyperparameters — only the random state differs.