RLHF and Preference Alignment

A language model trained only to predict the next word in internet text will produce fluent, plausible-sounding continuations — but "plausible continuation of internet text" is not the same objective as "helpful, honest answer to a user's question." Left alone, a base model is just as likely to continue a question with more rambling internet-style text as with a genuinely useful answer. RLHF (reinforcement learning from human feedback) is the fine-tuning stage that closes that gap, by directly optimizing for what humans judge to be a better response.

The analogy: training a new employee by ranking their memos, not correcting each sentence

Imagine training a new analyst not by rewriting every sentence they draft, but by handing them two draft memos on the same topic and saying "this one is better." Repeated across many topics, the analyst gradually internalizes what makes a memo good, without anyone ever specifying an exact rule for "good." RLHF trains a model the same way: rather than hand-labeling the single "correct" answer to every prompt, humans just rank a handful of candidate outputs, and that comparison signal is used to steer the model.

The three-stage pipeline

1. Supervised fine-tuning (SFT). Start from a pretrained base model and fine-tune it on a smaller set of high-quality human-written demonstrations of the desired behavior, giving the model a reasonable starting policy.
2. Train a reward model. Generate several candidate responses to the same prompt from the SFT model, have human labelers rank them, and train a separate reward model $r_\phi(x, y)$ to predict a scalar score matching those human rankings — higher for responses humans preferred.
3. Fine-tune with reinforcement learning. Use the reward model's score as the reward signal in a reinforcement-learning algorithm (commonly a policy-gradient method like PPO) to further adjust the language model's parameters so it generates responses that score higher under the reward model, typically with a penalty that keeps it from drifting too far from the SFT model's fluent behavior.

In plain English: instead of an expert writing down a formula for "good response," humans supply comparative judgments, a reward model distills those judgments into a scorable function, and reinforcement learning pushes the language model to actually produce outputs that score well.

Worked example: from a ranking to a reward score

Suppose three candidate answers to "explain this filing's risk factors" are ranked by a labeler as $y_1 \succ y_2 \succ y_3$ (best to worst). The reward model is trained so that its predicted scores respect this ordering — for instance ending up with $r_\phi(x,y_1)=2.3$, $r_\phi(x,y_2)=1.1$, $r_\phi(x,y_3)=-0.4$. The exact numeric gaps are learned from many such comparisons across many prompts, not from this single ranking, but the reward model's job is precisely to have learned a scoring function consistent with thousands of these pairwise human judgments.

Worked example: the reward-versus-drift tradeoff

During the RL fine-tuning stage, the objective typically maximizes $\mathbb{E}[r_\phi(x,y)] - \beta \, \mathrm{KL}(\pi_\theta \| \pi_{\text{SFT}})$ — the expected reward, minus a penalty $\beta$ times how far the new policy $\pi_\theta$ has drifted from the original SFT model $\pi_{\text{SFT}}$. If $\beta$ is too small, the model can learn to exploit quirks in the reward model — producing text that scores artificially high on the reward model's own scale without actually being a better answer (a failure mode called reward hacking). If $\beta$ is too large, the model barely updates at all. Choosing $\beta$ is a genuine tradeoff between how much the model improves and how much it risks gaming its own training signal.

What this means in practice

RLHF is why a chat-tuned model responds helpfully to instructions instead of just continuing text in whatever style the prompt resembles, and why two models built on the same pretrained base can behave very differently depending on what their human feedback rewarded. For quant applications, it matters because an RLHF-aligned model is optimized to sound confident and helpful — not necessarily to admit uncertainty about a financial fact it doesn't know, a common source of overconfident, ungrounded answers.