Partially Observable MDPs

A standard Markov decision process (MDP) assumes the agent always sees the true state exactly — it knows precisely which situation it is in before choosing an action. This is a convenient fiction that rarely holds in markets: you never directly observe "the true underlying regime" or "the true fair value" of an asset, only noisy, partial signals about them — a price, a volume figure, an order-book snapshot — that are consistent with several different possible true states. A partially observable MDP (POMDP) is the model built for exactly this situation: the agent receives only an observation, a noisy or incomplete function of the hidden true state, and must act while carrying genuine uncertainty about what that true state actually is.

The added piece: a belief instead of a state

In an ordinary MDP, the agent's input at each step is simply the current state. In a POMDP, the agent instead maintains a belief state: a full probability distribution over which true (hidden) state it is most likely in, given everything it has observed so far. Every time a new observation arrives, that belief gets updated using Bayes' rule — exactly the same update mechanism used in a hidden Markov model, since a POMDP is, underneath, a hidden Markov model with actions and rewards layered on top. The agent then chooses its action based on this belief distribution, not on a single known state, because a single known state simply is not available.

Worked sketch: is the market trending or mean-reverting?

Suppose the true (hidden) market regime is one of two states, "trending" or "mean-reverting," and it is never directly observable — only a noisy signal, today's realized short-term autocorrelation, is available, and that signal overlaps somewhat between the two regimes. Before any data, the agent's belief might be 50/50. Today's observed autocorrelation comes in mildly positive — a signal seen three times more often under trending than under mean-reverting. Bayes' rule updates the belief from 50/50 to roughly 75% trending, 25% mean-reverting. The agent does not simply declare "it's trending" and act as if certain — it acts on the full 75/25 belief, perhaps sizing a momentum position at 75% conviction rather than betting everything, and if tomorrow's observation reinforces the signal, the belief sharpens further; if it flips, the belief swings back.

The belief-updating step at the heart of a POMDP is precisely the mechanic shown above: a prior distribution over which hidden state is true, updated into a posterior as each new noisy observation arrives. A POMDP-solving agent repeats exactly this update at every timestep and chooses its action based on the current posterior, not on any assumed-known ground truth.

What this means in practice

Almost every real trading decision is better modeled as a POMDP than as a plain MDP, because the "state" that actually matters — the true regime, the true fair value, other participants' true intentions — is never directly visible, only inferable from noisy prices, volumes, and order flow. The honest complication is that solving POMDPs exactly is far harder computationally than solving plain MDPs, because the belief itself is a continuous, high-dimensional object (a full probability distribution, not a single number), which is why most practical approaches approximate the belief rather than tracking it exactly.