Hierarchical RL for Multi-Timescale Trading

A single reinforcement-learning agent asked to trade a portfolio has to make decisions that live on wildly different clocks at once: "should I be long or short semiconductors this month" and "should I lift this offer in the next 200 milliseconds" are both trading decisions, but they don't belong in the same policy. Force one flat policy to handle both and it either reacts to microstructure noise as if it were a real macro signal, or moves its month-long book position one tick-sized trade at a time with no sense of urgency. Hierarchical reinforcement learning fixes this by splitting the agent into layers, each owning its own timescale.

The analogy: a portfolio manager and a trading desk

Think of how a real trading operation is organized. A portfolio manager decides, maybe once a day, "get long $2m of this name, flat that one" — a slow, high-level call based on views that don't change minute to minute. A trader on the desk then spends the rest of the day figuring out how to actually get to that $2m position: which venue, what size clips, when to be passive versus aggressive. The PM never micromanages individual child orders, and the trader never second-guesses the daily target. Hierarchical RL formalizes exactly this split: a high-level policy picks a slow-changing goal (a target position, a risk budget for the day), and a low-level policy is rewarded purely for achieving that goal efficiently within its own fast decision loop.

Technically, the high-level policy operates over "options" — temporally extended actions that, once chosen, run for many low-level steps before the high-level policy checks in again. The high-level policy sees a coarser state (say, hourly features) and picks an option like "reduce inventory by 30% over the next hour." The low-level policy then receives that target as part of its own state and reward, and solves a much simpler, shorter-horizon problem: which micro-actions minimize slippage while hitting the target.

Worked example: a two-layer inventory manager

Suppose the high-level policy re-plans once per hour and decides the desk should end the hour with inventory reduced from 10,000 to 7,000 shares — a target of −3,000 shares. That target, plus current market state, becomes the input to the low-level policy, which acts every second. The low-level policy's reward each second is a simple execution-quality signal: negative slippage versus mid-price, minus a penalty for finishing the hour away from the −3,000 target. Over the hour it might place small passive orders early, then switch to more aggressive marketable orders in the final ten minutes if it's falling behind pace — decisions the high-level policy never has to think about, because it's already moved on to setting next hour's target based on updated market views.

What this means in practice

Hierarchical RL matters wherever a trading problem genuinely has two clocks: portfolio-level allocation decisions that should react to slow-moving views, and execution decisions that must react to the order book in real time. Training a flat agent on tick data to also learn portfolio strategy tends to need enormous amounts of data and produces noisy, hard-to-interpret policies. Splitting the problem lets each layer be trained (and audited) on the timescale where its signal actually lives, and lets a firm reuse a good low-level execution policy across many different high-level strategies.