RLHF and preference optimization.
A model that reads a lot learns to predict the next word. That makes it fluent. It does not make it helpful. This is the field that teaches the difference.
Reinforcement learning from human feedback, RLHF, and the broader family of preference optimization are how a language model is shaped by judgments of what a good answer looks like, not just by raw text. We walk it the way the field is built: what it is, why it is hard, how it works, what it unlocks, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
Teaching a model what a good answer looks like.
A pretrained model optimizes one thing: predict the next token. That yields fluency and broad knowledge. It does not, on its own, yield the answer a person would prefer.
The gap between "likely text" and "the answer a person wanted" is the problem this field addresses. A model trained only to continue text will happily continue it in a way no one asked for: a confident wrong answer, a wall of filler, a reply that sidesteps the actual question.
The core insight (Christiano et al., 2017) is that you do not need a human to write down a reward function. A good answer is hard to define as a rule but easy to compare. People struggle to specify the perfect reply, yet they can reliably look at two replies and say which is better. Ranking is cheaper and steadier than scoring, and those rankings can be turned into a training signal.
A little vocabulary, introduced gently. The model being tuned is the policy. The learned scorer trained from preferences is the reward model. The original model the policy is tuned from, held fixed as a yardstick, is the reference. The whole family is preference optimization, and RLHF is one method inside it.
“Absolutely. Friday is guaranteed.”
“I’ll confirm with operations before we promise Friday.”
Preferences are easier than rules, but they are not free.
Learning from comparisons sidesteps the hard job of defining "good" by hand. It brings its own difficulties, and an honest explainer names them.
Good is easier to compare than to write down.
Writing a metric for helpful, honest, and harmless by hand is brittle. Preferences sidestep that. The trade is that you now depend on whose comparisons you collected and how consistent they are.
Push too hard and the model games the scorer.
Optimize a policy too aggressively toward a reward model and it finds answers that score high but are not actually good: sycophancy, padding, exploiting the scorer's blind spots. This is why a KL penalty keeps the policy near the reference. A leash, not a finish line.
Human comparisons are slow, costly, and noisy.
Raters disagree, and the data encodes whose preferences were gathered. Annotation guidelines and rater selection are part of the model's behavior, not an afterthought bolted on at the end.
The scorer is only trustworthy near what it has seen.
A reward model judges well around the answers it was trained on. As the policy changes and explores new replies, the reward model can go stale and start scoring unfamiliar answers badly.
Alignment, not size, drove perceived quality.
As an attributed academic figure, InstructGPT (Ouyang et al., 2022) reported that a 1.3B-parameter tuned model was preferred by human raters over the 175B GPT-3 base model on their prompts. That result belongs to the paper that published it, never to anyone reading this page.
Rank answers, learn a scorer, then tune toward it.
Start from demonstrations, collect preferences, fit a reward model, then optimize the policy against it with a guardrail. Newer methods fold steps together.
- 01Demonstrate
Train on written responses.
- 02Compare
Collect response preferences.
- 03Score
Learn a reward model.
- 04Optimize
Tune policy with a reference constraint.
The pipeline InstructGPT (Ouyang et al., 2022) lays out: four jobs in order, each doing one thing.
1Supervised fine-tuning
Start from demonstrations.
Before any preferences, people write good answers to prompts and the model imitates them. This supervised fine-tuning, or SFT, gets the model into the right neighborhood: answering instructions rather than just continuing text. It is the warm start that the preference stages then refine.
Draft a short follow-up with a clear next step.
Thanks for today’s discussion. I’ll send the revised proposal on Friday.
Train the model to produce responses like the demonstrations.
2Collect preferences
Show two answers, pick the better one.
For many prompts, sample several answers from the model and have raters rank them, often just choosing the better of two. This produces a dataset of triples: a prompt, a preferred answer, and a dispreferred one. No one has to define what good means in the abstract. They only have to compare.
What is the price for 50 seats?
Response A
“About forty dollars a seat, I think.”
Response B
“I can’t confirm that price. I can help find the current quote.”
The preference rewards acknowledging missing evidence.
3Train the reward model
Turn the rankings into a scorer.
Fit a model that takes a prompt and an answer and returns a single number, trained so that preferred answers score higher than dispreferred ones. The objective is a logistic, Bradley-Terry style loss over the pairs. The bridge from preference games to real language tasks is Stiennon et al. (2020), which showed this reward-model recipe beating supervised baselines on summarization.
Illustrative reward scores; they are not probabilities or universal quality ratings.
4Optimize the policy
Tune toward the scorer, on a short leash.
Now tune the policy to produce answers the reward model scores highly, while a KL-divergence penalty keeps it close to the frozen reference model so it does not drift off the manifold of fluent text. The classic optimizer here is PPO, proximal policy optimization. The penalty is the answer to reward hacking: optimize too hard and the model finds high-scoring answers that are not actually good, so the leash holds it near trustworthy language.
Reward objective
- Encourage
- Responses the learned scorer ranks highly.
KL penalty
- Constrain
- Large departures from the reference policy.
The regularization strength changes the trade-off.
5The shortcut: DPO
Skip the reward model and the RL loop.
Direct Preference Optimization (Rafailov et al., 2023) shows the optimal RLHF policy can be written in closed form in terms of the reference model. So you can drop the separate reward model and the reinforcement-learning loop and train the policy directly on the preference pairs with a single classification-style loss. Fewer moving parts, easier to keep stable. DPO is the reference point for the broader "preference optimization" framing this page uses.
Reward-model route
- Preferences
Collect response comparisons.
- Reward model
Fit a scorer.
- Policy optimization
Optimize a policy using that scorer.
DPO route
- Preferences
Collect response comparisons.
- Direct objective
Optimize the policy against chosen and rejected responses.
6Feedback from a model
Let a model do some of the judging.
Constitutional AI (Bai et al., 2022) replaces or supplements the human labeler with a model that judges answers against a written set of principles, a "constitution." This is RLAIF, reinforcement learning from AI feedback. It scales the feedback step and, just as usefully, makes the judging criteria explicit and auditable instead of locked inside a rater's head.
Do not invent a price that the supplied evidence does not confirm.
| Response | Assessment |
|---|---|
| It is around forty a seat. | Unsupported price |
| I need the current quote to confirm. | Acknowledges missing evidence |
This stage is what turns a text model into an assistant.
Instruction following, declining unsafe requests, staying on the question asked, and a steady tone are largely a product of preference optimization. The examples below illustrate the technique. They are not claims about any one product.
Follows instructions and stays on task.
The behaviors people associate with a helpful assistant, doing what was asked, declining what should be declined, holding a consistent tone, come largely from this training stage rather than from raw scale.
Adapt a base model to a way of working.
Examples of preferred answers can encode a domain's conventions: how a team summarizes a call, what a clean CRM note looks like, which framing is acceptable. As an illustration of preference optimization in enterprise SaaS, this is how a model can be shaped to a company's way of selling without retraining from scratch. Used here as an example of the technique, not a claim about any system.
A knob you can point at safety or format.
The same machinery can be aimed at helpfulness, safety, format adherence, or brevity, depending on what the preference data rewards. The neighboring jobs of answering only from given data and acting through tools have their own pages, linked below, and are not restated here as product claims.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Direct losses beyond DPO.
Methods such as IPO, KTO, and ORPO trade the reinforcement-learning loop for a direct training loss and study how to keep it stable and resistant to over-optimization. The trend is fewer moving parts.
More model-in-the-loop judging.
RLAIF and process-level or step-level rewards, scoring the reasoning along the way rather than only the final answer, aim to gather more signal without a matching rise in human labeling cost.
Keeping reward models trustworthy.
Open work on how to stop reward models from being gamed, and how to keep them accurate as the policy explores answers they were never trained to judge.
Many viewpoints, not one average.
How to represent diverse and conflicting preferences rather than collapsing them into a single average, so a model is not tuned to one narrow notion of good.
Beyond "raters preferred it."
Better ways to measure alignment than win rate alone, so progress is judged on more than whether one set of raters liked one set of answers.
The papers behind this page.
- Christiano et al. (2017). Deep Reinforcement Learning from Human Preferences. arXiv:1706.03741.
- Stiennon et al. (2020). Learning to Summarize from Human Feedback. arXiv:2009.01325.
- Ouyang et al. (2022). Training Language Models to Follow Instructions with Human Feedback (InstructGPT). arXiv:2203.02155.
- Bai et al. (2022). Constitutional AI: Harmlessness from AI Feedback (RLAIF). arXiv:2212.08073.
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model (DPO). arXiv:2305.18290.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
The field's whole aim is an answer you would have preferred. See it on a real call.
No app required. Just your phone number.