Model Complexity vs Inference Cost

A more complex model, deeper trees, more ensemble members, a bigger neural network, usually fits training data better and often generalizes better too, up to a point. But every added layer of complexity also costs something at prediction time: more floating-point operations per prediction, more memory to hold the model, and more wall-clock latency between receiving new data and producing a signal. In research, complexity is nearly free, a backtest can run overnight. In live trading, complexity has a real-time price tag, and if a signal takes too long to compute, the market has already moved past the price the model was reacting to.

For example, a 500-tree gradient boosting ensemble might improve out-of-sample accuracy by half a percentage point over a 50-tree version, but if it takes 8 milliseconds to score instead of 1 millisecond, and the strategy is trading on a signal that decays within a few milliseconds, the "better" model can produce worse live P&L than the simpler one, the accuracy gain never gets captured because the trade is placed too late.

This is why production quant teams typically report a model's inference latency and memory footprint alongside its accuracy metrics from day one, rather than optimizing purely for backtest performance and discovering the latency problem only at deployment.