Training Cost and Compute Budgeting

A quant research team can burn through a cloud budget in a week without a single line of misbehaving code — someone just launched ten hyperparameter sweeps on a dataset ten times larger than the last one, each on a rented GPU, and nobody added up what that costs before hitting "run." Training cost and compute budgeting is the discipline of estimating that bill in advance: how many GPU-hours a training run needs, what that costs in dollars, and whether the expected improvement in the model is worth it. Without this, research decisions get made by whoever's job finishes first, not by whoever's idea was actually worth the spend.

The basic unit: GPU-hours, not epochs

The number researchers actually pay for is GPU-hours (or GPU-seconds), not "epochs" or "iterations" — a training run's cost is roughly proportional to the number of floating-point operations it performs, which scales with model size, dataset size, and number of passes over the data. A useful back-of-envelope form is:

$$\text{FLOPs} \approx 6 \times N \times D$$

where $N$ is the number of model parameters and $D$ is the number of training tokens or samples processed. In plain English: total compute is roughly six times the parameter count times the amount of data seen, because each parameter needs a handful of multiply-add operations for both the forward pass and the backward pass. Divide the FLOPs figure by a GPU's sustained throughput (a realistic, not peak, number) to get GPU-hours, and multiply by the hourly rental rate to get a dollar estimate before a single job is launched.

Worked example: pricing a sweep before running it

Suppose a single training run takes 4 GPU-hours on a cloud instance billed at $3.00 per GPU-hour, so one run costs $4 \times 3 = 12$, i.e. $12. A hyperparameter sweep testing 5 learning rates × 4 regularization strengths × 3 seeds is $5 \times 4 \times 3 = 60$ runs, costing $60 \times 12 = 720$, i.e. $720 — worth knowing before launch, not after the invoice arrives. If the team's monthly compute budget is $5,000, this one sweep consumes roughly 14% of it, which is the kind of comparison that turns "let's just try everything" into a prioritized shortlist: cut the seed count to 2 and the learning-rate grid to 3 values, dropping the sweep to 24 runs and $288, and spend the savings on a more promising architecture instead.

Drag the exponent in the plot above to see why compute cost curves are rarely straight lines — doubling dataset or model size doesn't just double the bill, since larger runs often need proportionally more tuning passes too, so extrapolating linearly from a small pilot tends to underestimate the real cost of scaling up.

What this means in practice

Before a large run, estimate its GPU-hours and dollar cost from a small pilot at a fraction of the scale, then extrapolate. Track spend against a monthly ceiling per project, not just per run, since it's the accumulation of many "small" sweeps that actually blows a budget. And always weigh a run's expected cost against its expected payoff: a $50 experiment that might improve a live signal's Sharpe ratio is an easy yes, but a $5,000 architecture search chasing a marginal accuracy gain rarely pays for itself.