Short-Horizon Price Prediction From the Order Book

At the scale of individual ticks, the limit order book contains real, exploitable structure: when there's much more resting size on the bid than the ask, the next price move is, on average, slightly more likely to be up than down. This is one of the few places in finance where "predict the next move from current state" has a genuinely strong, well-documented statistical relationship — order flow imbalance is a real predictor of short-horizon price direction. It's also one of the easiest problems to accidentally solve by peeking, because the book's own state changes within microseconds of a trade, and a label built even slightly carelessly can let the model see the outcome baked into its own input.

What makes this problem different from ordinary supervised learning

The features are snapshots of the book: best bid/ask sizes, imbalance ratios, recent trade signs, spread. The label is typically the sign of the mid-price move some short horizon later — a few hundred milliseconds to a few seconds. On the surface this is an ordinary classification problem (see The Supervised Learning Framework), and models from simple logistic regression up to deep sequence models on the raw book have been applied to it.

The trap is timing precision at a scale that doesn't exist in daily-frequency alpha research. If the feature snapshot is taken even slightly after the label horizon starts — say, the "current" book state used as a feature actually includes size changes that occurred because of the very trade whose aftermath defines the label — the model isn't predicting the future, it's reading off a cause from its own effect. At daily frequency a one-day lag error is usually forgivable in relative terms; at sub-second frequency, a timestamp error of the same relative size is on the order of milliseconds, and getting it wrong produces a model that looks almost perfectly accurate and is completely useless, because the "prediction" already happened by the time it would need to be acted on.

A worked example

A researcher builds a feature: order flow imbalance, defined as (bid size − ask size) / (bid size + ask size), sampled at the top of book, used to predict the sign of the mid-price move over the next 500 milliseconds.

Careless version: the imbalance feature is computed from a book snapshot taken at the same database write as the label's starting mid-price, but the snapshot pipeline batches updates and, under load, sometimes includes a few extra milliseconds of order flow that technically occurred just after the labelled starting point. In backtest, the resulting model achieves an implausibly high accuracy — well above what published order-flow-imbalance research would suggest is achievable.

Corrected version: the feature snapshot is taken strictly using the last book update with a timestamp less than or equal to the label's start time, with an explicit buffer subtracted to account for any known latency in the data pipeline itself. Accuracy drops substantially, landing much closer to the modest but real edge order flow imbalance is known to carry — clearly better than a coin flip, far from the near-certainty the leaky version implied.

A correlation this modest — noticeably positive, far from one — is roughly the honest signature of order flow imbalance against short-horizon direction once timing is done correctly. A backtest that looks more like a near-perfect line than a loose scatter at this horizon is a signal to re-check timestamps before celebrating.

Where it goes wrong in practice

Beyond timestamp leakage, tick-level labels are also severely imbalanced by nature — most short intervals see no meaningful price change at all — so a naive classifier can achieve high accuracy by predicting "no move" and never actually catching the informative cases, unless the evaluation explicitly weights the class of interest (see Sampling and Class Balance in Tick-Level Models).