C++ Inference for Tree Ensembles

A gradient-boosted ensemble of, say, 500 trees is trained comfortably in Python using a library like XGBoost or LightGBM. Training happens once, offline, and can take minutes without anyone caring. Serving that same model live in a trading system is a different problem entirely: every incoming tick needs a prediction in a handful of microseconds, and Python's interpreter overhead alone — object creation, dynamic type dispatch, the global interpreter lock — can burn through that entire budget before a single tree has even been walked. C++ inference for tree ensembles re-expresses the trained model as compiled, allocation-free C++ code that does nothing but the arithmetic the trees actually require.

The analogy: a recipe read aloud versus a recipe memorized

Following a recipe by reading each instruction aloud, looking up unfamiliar terms, and re-checking the ingredient list every step is fine for a home cook with no time pressure. A short-order cook who has to plate a dish in fifteen seconds instead has the recipe memorized as a fixed sequence of hand motions, no reading, no re-checking, nothing but the motions themselves. Python evaluating a tree ensemble is the recipe read aloud — flexible, but paying an interpretation cost on every single step. Compiled C++ inference is the memorized sequence: the exact same logic, with all the interpretation cost removed ahead of time.

Worked example 1: what a single tree becomes in C++

A trained decision tree with a handful of splits, expressed at inference time as nested comparisons, becomes almost literally this in C++:

double eval_tree_1(const Features& f) {
  if (f.spread < 0.015) {
    if (f.imbalance < -0.3) return -0.42;
    else return 0.11;
  } else {
    if (f.momentum_5s < 0.0) return -0.05;
    else return 0.38;
  }
}

No dynamic dispatch, no memory allocation, no object wrapping the feature vector — f is a plain, fixed-layout struct of doubles, and the whole function compiles down to a handful of comparison and load instructions. Summing 500 such functions (one per tree in the ensemble, each contributing a small leaf value) and adding a base score gives the ensemble's raw prediction, typically in well under a microsecond for a few hundred shallow trees.

Worked example 2: the actual speed gap

Benchmarking the same 500-tree ensemble: evaluating it in Python via the library's own .predict() call on a single row takes roughly 150 microseconds, dominated by Python-object overhead and the library's generality (it's built to score millions of rows at once efficiently, not one row instantly). The same ensemble, re-expressed as generated C++ code compiled ahead of time, evaluates a single row in about 0.8 microseconds — nearly 200x faster. That gap is not about C++ being "faster at math" in some abstract sense; it's that Python's per-call overhead is a roughly fixed cost regardless of how simple the underlying computation is, while compiled C++ pays only for the arithmetic actually needed.

What this means in practice

The common workflow trains the model in Python, then exports its structure (splits, thresholds, leaf values) into a format a code generator turns into C++ — either fully unrolled functions like the example above, or a compact array-based tree-walking loop for larger ensembles where generating one function per tree would bloat compile times. The generated code is unit-tested against the original Python model's predictions on a held-out sample to confirm bit-for-bit (or acceptably close, given floating-point differences) agreement before it's trusted in production.