Measuring the Economic Value of a Forecast

Two models forecast the same stock's return. Model A has a lower mean-squared error. Model B has a higher mean-squared error but is right at exactly the moments when the sign of the forecast matters most for trading. A researcher who picks based on mean-squared error alone would choose Model A and leave money on the table, because statistical accuracy and economic value are related but genuinely different questions — and only the second one pays the bills.

Why statistical accuracy can mislead

Mean-squared error weighs every prediction error the same regardless of whether that period was one where getting it right would have mattered for P&L. A model can minimize average error by being very accurate on the many small, unimportant moves and mediocre on the rare large moves — exactly the moves that drive most of a trading strategy's actual profit. Statistical loss functions are symmetric and blind to this; a trading account is not.

A direct economic-value test

The standard fix is to skip the intermediate statistical metric and simulate the trading decision the forecast would actually drive:

$$\text{Economic Value} = \sum_{t} \left[ \text{sign}(\hat{r}_t) \times r_t \right] - \text{Costs}$$

In words: for every period, take the position the forecast would have told you to take (long if it predicted positive, short if negative), multiply by what actually happened, sum it up across the whole test period, and subtract what it would have cost to trade that way. This turns a forecast into a strategy and evaluates the strategy, not the forecast's error statistics.

Worked example

Model A has a mean-squared prediction error of 0.015 (in daily-return-squared units) across a test period. Model B has a higher error of 0.021. Running the economic-value test above: Model A correctly calls the sign of the next-day return 51% of the time, generating a simulated gross return of 4.2% over the test period before costs, and about 1.8% after costs. Model B calls the sign correctly only 49% of the time overall — worse hit rate — but its errors are concentrated on quiet, low-volatility days, while its rare large predictions, which happen to coincide with large actual moves, are directionally right more often; its simulated strategy earns 7.9% gross and 5.4% after costs. Model B has higher statistical error and higher economic value, and by the metric that actually matters for a trading desk, it is the better model.

What this means in practice

Because economic-value testing simulates a full strategy, it also naturally exposes issues that pure statistical metrics hide — like a model whose edge lives entirely in illiquid names where the simulated trade could never actually be executed at the assumed cost. It is standard practice to run both: statistical metrics for quick iteration during model development, and an economic-value backtest as the final gate before anything trades real capital.