Action Spaces for Order Placement

An RL agent's decisions are only ever as rich as the set of actions it's allowed to choose from. An agent restricted to "buy 100 shares at market" or "do nothing" can never learn subtler behavior like placing a limit order slightly inside the spread — the action simply doesn't exist in its universe of choices, no matter how much training data it sees. But an action space that lets the agent pick any price to arbitrary precision and any size up to the full remaining order is often too large and too continuous to explore efficiently, especially early in training when the agent's policy is close to random.

The action space is the full menu of moves available to the agent at each decision point, and — much like state design — the choice of granularity here determines what kinds of execution strategies the agent can possibly discover, independent of how good the learning algorithm is.

Discrete versus continuous action spaces

Two broad designs dominate execution RL:

• Discrete action spaces. The agent picks from a small fixed menu: place a limit order at the best bid, one tick inside, or at the mid; or place a marketable order for a fixed chunk of remaining size. Discrete spaces are easier to learn with simpler algorithms (tabular Q-learning, DQN) and easier to interpret, at the cost of only approximating what a continuous strategy could do.
• Continuous action spaces. The agent outputs a real-valued price offset and/or a real-valued fraction of remaining size to trade, requiring policy-gradient methods (see Proximal Policy Optimization or Actor-Critic Methods) capable of handling continuous outputs, but able in principle to learn much finer-grained behavior.

Many practical systems compromise: a coarse discrete grid over price levels (say, five to ten price offsets) combined with a small number of size buckets, giving enough expressiveness to matter while keeping the action space small enough to explore in a reasonable amount of training time.

Worked example

An execution agent is initially given a 3-action discrete space: market order for all remaining size, limit order at the best bid, or do nothing. Backtested on a 50,000-share order over a trading day, this coarse agent achieves an average shortfall of 12bp versus a TWAP baseline. Expanding to 15 actions — five price levels crossed with three size fractions (25%, 50%, 100% of remaining) — lets the agent learn to place smaller, passive orders early and reserve larger, aggressive ones for when urgency rises, cutting shortfall to 7bp. Expanding further to a fully continuous action space with a policy-gradient algorithm produces only a marginal further improvement to 6.5bp, at several times the training compute — a well-chosen discrete grid often captures most of the achievable benefit without the complexity of continuous control.

What this means in practice

Choosing an action space is a tradeoff between expressiveness and how fast the agent can learn to use it well — an overly coarse space caps performance regardless of training time, while an overly fine one wastes training time exploring distinctions that barely matter. A practical starting point is a moderate discrete grid, benchmarked against a simple execution baseline like TWAP (see Benchmarking RL Execution Agents Against TWAP), with continuous control reserved for cases where the discrete grid has demonstrably plateaued.