Custom Objectives and Gradients in Boosting

Squared error and log-loss are the default objectives in every gradient boosting library, but they encode a specific assumption about what kind of mistake is bad — squared error treats overprediction and underprediction identically, and treats a miss of $4 as sixteen times worse than a miss of $1. A trading application rarely wants that; it more often wants to penalize missing a big down-move far more than missing a small one, or to penalize being on the wrong side of a trade regardless of the exact magnitude. Custom objectives let you say precisely, in code, what "bad" means for your problem, and boosting will optimize exactly that.

The analogy: telling a tutor exactly what mistakes cost you

An ordinary tutoring service scores every wrong answer the same way — one point off, regardless of the question. A custom-objective tutor lets you tell it in advance, "getting question 7 wrong costs me ten times as much as getting question 3 wrong, and answering too high is twice as bad as answering too low." The tutor doesn't change how it studies, it changes what it treats as worth correcting. That is exactly what a custom objective does to a boosted ensemble: same tree-building machinery, different definition of "wrong."

What you actually have to supply

A gradient boosting library builds each new tree using two pieces of information about the current model's errors, calculated per training row: the gradient $g_i = \partial L/\partial \hat{y}_i$ and the Hessian $h_i = \partial^2 L/\partial \hat{y}_i^2$ (see Newton Boosting and Second-Order Approximations). Supplying a custom objective means writing a function that, given the current predictions and true labels, returns $g_i$ and $h_i$ for every row — the library never needs the loss value itself during training, only its slope and curvature.

$$g_i = \frac{\partial L(y_i, \hat{y}_i)}{\partial \hat{y}_i}, \qquad h_i = \frac{\partial^2 L(y_i, \hat{y}_i)}{\partial \hat{y}_i^2}$$

In plain English: at every point, tell the ensemble which direction reduces your custom cost (the gradient) and how sharply that cost curves (the Hessian) — the tree-building itself is unchanged, only the numbers it fits to are different.

Worked example 1: an asymmetric loss for downside risk

Define a loss that penalizes underprediction of a return three times as heavily as overprediction: $L = a(y - \hat{y})^2$ if $\hat{y} < y$ (model underpredicted), else $L = (y-\hat{y})^2$, with $a=3$. At a point where $y = 2\%$ and $\hat{y} = 0.5\%$ (an underprediction), the gradient is $g = -2a(y-\hat{y}) = -2(3)(1.5) = -9$ and the Hessian is $h = 2a = 6$. At a point where $y=2\%$, $\hat{y}=3.5\%$ (an overprediction), $g = -2(1)(-1.5) = 3$ and $h = 2$. The underprediction case gets a gradient three times larger in magnitude and a Hessian three times larger, so the ensemble is pulled far more aggressively to fix underpredictions than overpredictions of the same size — exactly the asymmetric behavior a symmetric squared-error loss cannot produce.

Worked example 2: verifying a custom gradient numerically

Suppose you implement $L(y, \hat{y}) = |y - \hat{y}|^{1.5}$ and derive $g_i$ by hand. Before trusting it, check with a numerical derivative at, say, $y=1, \hat{y}=0.8$: compute $L$ at $\hat{y}=0.801$ and $\hat{y}=0.799$, and estimate $g \approx \frac{L(0.801) - L(0.799)}{0.002}$. If your hand-derived formula gives $g = -1.5(y-\hat{y})^{0.5} \cdot \text{sign}(\ldots)$, plugging in numbers should match the numerical estimate to a few decimal places. Doing this check once, on a couple of points, catches sign errors and algebra mistakes before they silently corrupt an entire training run.

The explorer above shows steps following a loss curve's slope, exactly as boosting does — the only difference with a custom objective is that the shape of the curve being descended is one you designed, not squared error.

What this means in practice

Quant teams use custom objectives to align a model directly with a trading P&L structure — for example, penalizing a missed short signal before a large drawdown far more than a missed small move, or building an objective around a Sharpe-like ratio rather than raw prediction error. This closes the gap between "the model minimizes squared error" and "the model minimizes what I actually lose money on."