Dataloader Throughput and GPU Utilisation

A team rents a top-of-the-line GPU to train a large panel-data model, checks utilization mid-run, and finds the GPU sitting at 20% busy — expensive silicon spending most of its time waiting. The model isn't the bottleneck; the pipeline feeding it batches of data is. This is one of the most common and most fixable inefficiencies in training infrastructure, and understanding it changes how you diagnose "training is slow" from the very first symptom.

The idea: training is a relay race, and the GPU can't run ahead

Training a neural network alternates between two very different jobs: the dataloader reads raw files from disk, decodes them, applies transformations (normalizing features, building a batch), and hands a ready-to-use tensor to the GPU; the GPU then does the actual matrix multiplication for that batch. These two jobs happen on different hardware — dataloading is usually CPU and disk work, training is GPU work — and if the CPU/disk side can't prepare batches as fast as the GPU can consume them, the GPU simply waits with nothing to do.

It's like a chef (the GPU) who can plate a dish in ten seconds, fed by a prep cook (the dataloader) who takes thirty seconds per ingredient. The kitchen's overall speed is capped by the prep cook — buying the chef a faster knife does nothing if the bottleneck is upstream.

Worked example: measuring and fixing a bottleneck

A training run processes one batch every 2.0 seconds. Profiling shows the GPU's actual forward-and-backward computation for a batch takes 0.4 seconds — the remaining 1.6 seconds is the CPU decoding files and building the next batch while the GPU sits idle. GPU utilization in this run will read roughly $0.4 / 2.0 = 20\%$, matching what monitoring tools would show.

The standard fixes attack the CPU-side bottleneck directly: increase the dataloader's worker-process count so multiple cores decode batches in parallel; prefetch batches — have workers prepare batch $n+1$ while the GPU is still busy on batch $n$; and move expensive preprocessing to a one-time offline step rather than repeating it every epoch. If increasing workers from 2 to 8 brings per-batch time to 0.5 seconds, utilization rises to $0.4 / 0.5 = 80\%$ — four times the throughput, entirely from fixing the feeder.

Read that curve as "GPU utilization" against "dataloader worker count": utilization climbs steeply as workers go from too few to enough, then flattens out once the CPU side is no longer the constraint — adding workers past that point buys nothing, because the GPU itself is now the bottleneck.

What this means in practice

Before spending more on a faster GPU, check utilization first — a cheap dataloader fix can outperform an expensive hardware upgrade if the GPU was never the constraint. This is especially relevant for panel-data models, where a single "batch" involves joining and reshaping large historical windows across thousands of instruments — exactly the CPU-heavy work that stalls a GPU if it isn't parallelized or precomputed.