Generalized Advantage Estimation

Policy-gradient methods need to know whether an action was better or worse than average — its advantage — but estimating that advantage from a single next step is noisy while waiting for the full episode's return is slow and high-variance. Generalized advantage estimation (GAE) is a way of averaging over every possible horizon at once, controlled by a single decay parameter, lambda, that plays the same bias-variance role it plays in TD(lambda).

At lambda = 0, GAE reduces to the plain one-step TD-error advantage — fast to compute but biased by an imperfect value function. At lambda = 1, it reduces to the full Monte Carlo advantage — unbiased but noisy, since it depends on an entire trajectory's random rewards. Values in between blend the two, weighting near-term TD errors more heavily and letting the contribution of far-future errors decay geometrically.

Worked example. With lambda = 0.9 and discount gamma = 0.99, the GAE advantage at time t is the sum of TD errors delta_t + (gamma*lambda) delta_{t+1} + (gamma*lambda)^2 delta_{t+2} + .... Since gamma*lambda = 0.891, each successive TD error contributes about 89% as much as the one before it, so errors more than roughly 20 steps out barely matter.

GAE is now the default advantage estimator inside most modern policy-gradient algorithms, including PPO, precisely because it lets practitioners tune bias against variance without changing the underlying algorithm.