Nondeterminism in GPU Training

A researcher sets a fixed random seed, trains a model, and gets a validation Sharpe ratio of 1.15. They rerun the exact same script, same seed, same data, same code — and get 1.13. Nothing was changed on purpose, and the difference is small, but it raises an uncomfortable question: if the "same" run doesn't produce the same result, how do you know whether a real improvement from a code change is genuine or just this same background noise?

The idea: not everything on a GPU runs in a fixed order

Setting a random seed fixes the sequence of "random" numbers a program generates — the same seed always produces the same shuffle order, weight initialization, and dropout mask. That much is fully controllable. What a seed does not control is the low-level order in which a GPU's many parallel cores finish their arithmetic. GPU operations run fast by letting thousands of cores work simultaneously and combine results in whatever order they finish — and because floating-point addition isn't perfectly associative, the result of a large matrix operation can vary slightly run to run even with an identical seed, code, and data.

That tiny numerical noise, injected into a training loop with thousands of steps, compounds: an early step nudges a weight by a fraction of a decimal place, that nudge changes the next gradient, and small differences snowball into a training curve that ends up meaningfully different — even though every controllable input was identical.

Worked example: telling noise from a real signal

Suppose five repeated runs of the same configuration produce validation Sharpe ratios of 1.10, 1.13, 1.11, 1.15, 1.12 — a spread of about 0.05 from GPU-level nondeterminism alone, nothing else changed. If an actual code change then moves the Sharpe ratio from 1.12 to 1.14, that 0.02 improvement is smaller than the 0.05 of noise already observed between identical runs — it's not safe to conclude the change helped; it could easily sit within the same noise band.

The correct comparison isn't one run against one run; it's the distribution of several repeated baseline runs against several repeated runs of the new configuration, checking whether the two distributions are distinguishable or just overlapping noise. Some teams sidestep the ambiguity for critical comparisons by enabling deterministic GPU operations, forcing a fixed computation order, accepting a real slowdown for bit-for-bit reproducibility — typically reserved for final model comparisons, not every exploratory run.

Drag the parameters here to see how a spread this wide can easily swallow a "0.02 improvement" — the point isn't the exact numbers, it's that a single before/after comparison can't distinguish a real effect from noise unless you know how wide the noise itself is.

What this means in practice

Nondeterminism is a fact of GPU life, not a bug to hunt down in every case — but it means single-run comparisons between hyperparameter configurations are unreliable exactly when the true difference is small, which is often the situation a late-stage sweep finds itself in. The practical response is running each serious candidate configuration multiple times and comparing distributions, not single numbers, and reserving fully deterministic (slower) training for the final handful of models under real consideration before a go-live decision.