RLHF takes a pretrained model and aligns it with what humans actually prefer: collect comparisons of outputs, train a reward model on those preferences, then optimize the LLM against that reward with reinforcement learning (classically PPO). It is the step that turned autocomplete engines into assistants. Simpler successors like DPO reach similar results without the RL machinery.
Worked example: humans rank pairs of responses, a reward model learns those preferences, then RL (PPO) nudges the policy toward higher reward while a KL penalty keeps it near the original model. Gotcha: the policy can ‘reward-hack’ — exploiting quirks of the reward model to score high while getting worse for humans — which is why the KL leash and careful reward design matter.