Randomness, Seeds and Run-to-Run Variance

A neural network's weights usually start at random values, its training batches are usually shuffled in a random order, and techniques like dropout randomly silence neurons during training — all of that randomness is generated by a pseudo-random number generator seeded with a single starting number, the seed. Fix the seed and, on the same hardware, you get the exact same sequence of "random" choices and therefore the exact same trained model. Change only the seed and, with everything else identical, you can get a meaningfully different model — different final accuracy, different learned weights, sometimes a different qualitative behavior.

This matters because a single training run's result is really one sample from a distribution of possible outcomes, not the model's "true" performance. If retraining the same architecture on the same data with five different seeds produces test accuracies of 91.2%, 89.8%, 92.1%, 90.5%, and 91.9%, the honest summary is a mean of roughly 91.1% with a spread of about 2 points — not "91.2%," which is just whichever seed happened to run first. Comparing two architectures by a single run each is comparing noise, if that 2-point run-to-run spread is larger than the difference between the architectures.

The practical fix is to always train with multiple seeds when a claim of improvement is being made, and to report a mean and spread rather than one number, exactly as you would treat any other noisy measurement.