State Design for Trading Agents

An RL agent can only ever act on what its state representation hands it — if the current spread, the agent's own remaining inventory, and the time left in the trading window aren't in the state, the agent has no way to condition its behavior on them, no matter how sophisticated its learning algorithm is. Give it too little and it's flying blind, unable to distinguish situations that genuinely call for different actions. Give it everything imaginable — the full order book, every recent trade, every macro indicator — and the state space balloons so large that the agent can no longer see the same situation twice, making learning painfully slow or impossible without heavy function approximation.

State design is the choice of exactly which variables get fed to the agent at each timestep, and it is arguably the single highest-leverage design decision in building a trading RL system — more consequential, in practice, than the choice of learning algorithm itself.

What belongs in the state

A well-designed trading state typically balances three kinds of information:

• Market state — recent price moves, spread, order-book imbalance, volatility — the observable environment the agent is reacting to.
• Agent-internal state — current inventory or position, remaining time in the execution window, cash used so far — information the market doesn't have but the agent's own future decisions depend on critically (an agent near the end of its execution window with unfilled inventory faces a fundamentally different problem than one at the start).
• Derived / normalized features — raw prices and volumes vary wildly in scale across time and instruments, so state variables are usually transformed into scale-invariant quantities (e.g., inventory as a fraction of the total order size, time as a fraction of the window remaining) so the same learned policy generalizes across different order sizes and market regimes.

Since markets are only ever partially observable — the true intentions of other participants are never directly visible — trading state design also has to decide how much recent history to include as a proxy for hidden information the agent can't see directly (see Partially Observable MDPs).

Worked example

An execution-agent designer initially builds a state with just two variables: current price and shares remaining to execute. Backtesting shows the agent learns a mediocre policy that ignores how much time is left, front-loading trades early in calm markets and then panicking with a large residual near the close. Adding time remaining as a fraction of the window lets the agent learn a materially better policy: it now conditions aggressiveness on urgency, trading passively early and more aggressively as the deadline approaches — cutting average shortfall by roughly 18% from that one added variable, with no change to the reward function or algorithm.

What this means in practice

Poor state design is a common, quiet reason a trading RL agent looks fine in backtest but never converges to a sensible policy, or converges to one that only works for the exact order size and time-of-day it was trained on. Before touching the learning algorithm, it's worth auditing whether the state contains everything the agent genuinely needs to distinguish meaningfully different situations, and whether it's been normalized so those distinctions generalize rather than overfitting to one specific scale.