Soft Actor-Critic

An agent that locks onto one action too early can get stuck in a mediocre strategy and never discover a better one nearby. Soft actor-critic (SAC) guards against this by adding an explicit bonus for keeping the policy's action distribution spread out — its entropy — so the agent is rewarded for maximizing expected return and for staying appropriately random, rather than collapsing to a single deterministic choice too soon.

The entropy weight, often called alpha, is the knob that decides how much randomness is worth trading off against pure reward — too high and the agent never commits to a good action, too low and it collapses back to ordinary actor-critic behavior with no exploration bonus at all. Modern implementations of SAC tune alpha automatically during training rather than fixing it by hand.

Worked example. Suppose two actions have nearly identical expected value, 10.0 and 10.2. An ordinary actor-critic agent quickly puts almost all its probability on the 10.2 action. SAC's entropy bonus penalizes that collapse, keeping meaningful probability on both actions — which matters if the 10.2 estimate later turns out to be noisy and the true best action is actually the other one.

SAC is also off-policy, meaning it can reuse old experience from a replay buffer rather than requiring fresh data after every policy update, making it considerably more sample-efficient than on-policy alternatives.