Reinforcement learning,
and learning by trying.

Reinforcement learning is learning to make decisions by trial and error. An agent interacts with an environment in a loop: it observes the current state, chooses an action, and the environment returns a reward (a number) and the next state. Repeat. The agent's goal is to learn a policy — a mapping from states to actions — that earns the most reward over time.

The defining difference from supervised learning: there are no labels. Nobody tells the agent the right action; it only gets a reward signal, often delayed (a move in chess isn't rewarded until you win or lose much later). Learning which actions lead to reward, through experience alone, is the whole challenge.

The agent isn't maximizing the immediate reward — it's maximizing the return: the total reward accumulated over the whole future. That's what makes RL about strategy, not greed: sometimes a low-reward action now (sacrificing a piece) leads to far more reward later (winning the game).

Future rewards are usually discounted by a factor γ (gamma, between 0 and 1): a reward k steps away counts as γ^k times its value. This makes the infinite future sum finite and expresses "sooner is better," and γ tunes how far-sighted the agent is — near 1 is patient and long-term, near 0 is myopic. Formally, the agent maximizes the expected discounted return.

Return = r₀ + γr₁ + γ²r₂ + γ³r₃ + …

Here's RL's defining dilemma. To earn reward the agent should exploit — do what it already knows works. But to find better strategies it must explore — try unfamiliar actions whose outcomes it doesn't yet know. These conflict: every step spent exploring is a step not exploiting, and vice versa.

Lean too far toward exploitation and the agent gets stuck repeating a mediocre habit, never discovering the better path (a local optimum). Explore too much and it wastes time on bad actions and never cashes in on what it learned. Good RL balances the two — a common simple scheme is ε-greedy (mostly exploit, occasionally act randomly), often exploring more early and exploiting more as it learns. This tradeoff has no free lunch, and managing it well is much of what separates an agent that learns from one that stalls.

One family of methods learns value functions: how much future reward to expect from a situation. A state value estimates the return from a state; an action value (the Q-value) estimates the return from taking a specific action in a state and acting well thereafter. If you knew the Q-values, acting optimally is trivial: in each state, pick the action with the highest Q-value.

Q-learning learns these values from experience using the Bellman idea: the value of an action is its immediate reward plus the discounted value of the best next action. The agent bootstraps — updating each estimate toward "reward now + best estimated future" — and the estimates converge toward the true values. Deep Q-Networks (DQN) use a neural network to approximate Q-values for huge state spaces (this is how RL first beat Atari games from raw pixels). Value methods shine when actions are discrete.

The update itself is one line, and it's enough to learn an optimal policy from reward alone:

On a chain where you walk right to a goal, the learned values come out as γ^distance — 1.0 one step away, 0.9, 0.81, 0.73 — so the greedy policy is "go right" everywhere. The runnable version below trains it and rolls out to the goal.

The other family skips value estimates and optimizes the policy directly. A policy gradient method parameterizes the policy (usually a neural network mapping states to a distribution over actions) and adjusts its parameters to make high-reward actions more likely — nudging up the probability of actions that led to good returns, down for bad ones. REINFORCE is the basic version; PPO (Proximal Policy Optimization) is the workhorse, adding a constraint that stops each update from changing the policy too drastically, which keeps training stable.

Policy methods handle continuous actions (robot joint torques) and naturally produce stochastic policies, where value methods struggle. In practice, actor-critic methods combine both worlds: an "actor" (policy) chooses actions while a "critic" (value function) evaluates them to reduce the noise in the policy's updates. PPO — a stable actor-critic-style method — is exactly what's used to fine-tune LLMs with human feedback (section 08).

RL is powerful but notoriously finicky, and interviewers probe whether you know why. Three issues dominate. Instability: combining function approximation (neural nets), bootstrapping (updating estimates from other estimates), and off-policy learning — the "deadly triad" — can cause training to diverge. Sample inefficiency: RL often needs an enormous number of environment interactions to learn, because it learns only from sparse, delayed reward signals rather than dense labels.

And reward design is treacherous: the agent optimizes exactly what you reward, not what you meant. A poorly-specified reward invites reward hacking — the agent finds a degenerate way to score high that violates your intent (the classic: a boat-race agent that spins in circles hitting bonus targets instead of finishing). Specifying rewards that truly capture the goal is one of RL's deepest practical problems.

A final axis. Model-free RL (Q-learning, PPO) learns purely from experience without any model of how the environment works — simple and general, but sample-hungry because it can only learn from real interactions. Model-based RL additionally learns (or is given) a model of the environment — how states and rewards follow from actions — and can then plan or generate imagined experience, making it far more sample-efficient.

The tradeoff: a learned model can be wrong, and errors compound when you plan against it, so model-based methods add complexity and a source of bias. Landmark systems blend the ideas — AlphaGo/AlphaZero combine a learned value/policy network with explicit search (planning) over the game tree. Choose model-free for simplicity and when interactions are cheap; reach for model-based when real experience is expensive and a decent model is learnable.

RL's headline wins are agents that learn superhuman play (Atari, Go, StarCraft) and control problems (robotics, resource scheduling). But its most consequential recent use is aligning large language models. RLHF (reinforcement learning from human feedback) is how a raw, capable base model becomes a helpful assistant: humans compare model outputs, a reward model is trained to predict which output people prefer, and the LLM is optimized with RL (usually PPO) to produce higher-reward responses.

Everything above shows up here. The reward model is an imperfect proxy, so unconstrained optimization leads to reward hacking — the model games the reward with degenerate outputs. The fix is a KL penalty keeping the policy close to the original model, exactly the "don't drift too far" idea PPO embodies. So RL isn't just for games: it's the mechanism that turned language models into the assistants we actually use. For the full pipeline, see Design an RLHF Pipeline.