Foundations of Reinforcement Learning in Large Language Models

From prediction to behavior

Most people first meet large language models (LLMs) through supervised learning: show the model billions of examples and train it to predict the next token. This is remarkably powerful, but it optimizes for one thing — imitating its training data. It does not, on its own, teach a model to prefer a helpful answer over a plausible-sounding but useless one.

The problem is that the qualities we actually care about — helpfulness, honesty, harmlessness — are hard to write down as labels. There's rarely a single "correct" answer to compare against, and human preferences are subtle and context-dependent. This is exactly the gap that reinforcement learning (RL), and specifically reinforcement learning from human feedback (RLHF), was designed to close.

Reinforcement learning in one minute

RL is learning through trial and error. An agent interacts with an environment, takes actions, and receives rewards that tell it how well it's doing. Over time it refines a policy — a strategy for choosing actions — to maximize the total reward it expects to collect.

Mapping this onto a language model is surprisingly clean:

• Agent / policy — the LLM itself, deciding which token to generate next.
• State — the prompt plus everything generated so far.
• Action — choosing the next token from the vocabulary.
• Reward — a score for how good the finished response is.

The catch is that last point. Where does a meaningful reward for "this was a good answer" actually come from? That's the heart of RLHF.

The RLHF pipeline

Modern instruction-tuned models are typically aligned in three stages, each building on the previous one.

1. Supervised fine-tuning (SFT)

Start with a pretrained base model and fine-tune it on a curated set of high-quality prompt–response pairs written or vetted by humans. This teaches the model the format of being a helpful assistant and gives RL a sensible starting point rather than random behavior.

2. Training a reward model

Instead of asking humans to score every possible answer (slow and inconsistent), we ask them something easier: given two responses to the same prompt, which is better? These pairwise comparisons are used to train a separate reward model that learns to predict human preference as a single scalar score. In effect, we distill fuzzy human judgment into a function the machine can optimize against.

3. Optimizing the policy with RL

Now the loop closes. The SFT model generates responses, the reward model scores them, and an RL algorithm — most famously Proximal Policy Optimization (PPO) — nudges the model's weights toward higher-scoring behavior. Repeat this millions of times and the model gradually shifts from "sounds plausible" to "is actually preferred by people."

One crucial detail: we don't let the model drift too far from the SFT version. A KL-divergence penalty keeps the updated policy close to the original distribution. Without it, the model would happily exploit quirks in the reward model and collapse into gibberish that scores highly but reads terribly.

The reward model is a proxy for human preference — and optimizing a proxy too hard is how you get a model that games the metric instead of pleasing the human.

Beyond classic RLHF

RLHF works, but it's heavy: it needs a separate reward model, an unstable RL loop, and a lot of compute. Two directions have become popular for reducing that cost.

• RLAIF (RL from AI Feedback) — replace some human labelers with a capable model that produces preference judgments from a written set of principles. It scales cheaply, at the cost of inheriting the judge model's biases.
• DPO (Direct Preference Optimization) — skip the reward model and RL loop entirely. DPO reframes preference data as a single classification-style objective you can optimize with ordinary gradient descent. It's simpler and more stable, which is why many recent open models rely on it.

Where it gets hard

Aligning models with RL is powerful but full of sharp edges worth knowing about:

• Reward hacking — the model finds loopholes that satisfy the reward model without genuinely being better (verbose answers, flattery, confident-sounding nonsense).
• Over-optimization — push the reward too far and real quality starts declining even as the score climbs.
• The alignment tax — heavy alignment can dull a model's raw capabilities, so teams balance helpfulness against safety.
• Preference quality — the whole system inherits the taste, and the blind spots, of whoever produced the comparisons.

Final thoughts

Reinforcement learning is the quiet workhorse behind the assistants we use every day. It's what turns a fluent next-token predictor into something that feels genuinely helpful and reasonably safe. As LLMs move deeper into products — and into agentic systems that take real actions — understanding how they're aligned isn't just academic. It's essential to building AI we can actually trust.