Reward Model RLHF Explained: Bradley-Terry, Process Reward Models and Verifiers

Every aligned language model you use was shaped by a number it never saw at inference time: a scalar score produced by a reward model RLHF pipeline. That score is a compressed, lossy, learnable stand-in for “what a careful human would prefer”, and the policy is pushed uphill against it. When the stand-in is good, you get a helpful assistant. When it is slightly wrong, optimization pressure finds the gap and drives straight through it, which is why a model can become more confident, longer and more flattering while becoming less correct.

In 2026 the reward model is no longer one thing. Pairwise preference models still anchor chat alignment, process reward models grade individual reasoning steps, generative judges write critiques before scoring, and for math and code many labs skip learned rewards altogether in favour of verifiable checks. This post separates those pieces, works the math with real numbers, and shows where each one breaks.

What this covers: the Bradley-Terry loss with a numeric walk-through, outcome versus process supervision, verifiers and best-of-N search, reward hacking and over-optimization scaling laws, how reward models are evaluated, and how RLVR changes the picture.

Context and Background

Reinforcement learning from human feedback (RLHF) in its modern form was popularized by Christiano et al. (2017) for control tasks and then applied to language by Stiennon et al. (2020) for summarization. The recipe that reached products is InstructGPT (Ouyang et al., 2022, arXiv 2203.02155): supervised fine-tuning on demonstrations, then training a reward model on human comparisons, then optimizing the policy against that reward with Proximal Policy Optimization (PPO) and a KL penalty toward the SFT model. The central object in stage two is the reward model: a language model with its unembedding layer replaced by a linear head that emits one scalar for a prompt-response pair.

Why learn a reward at all? Because humans are bad at writing down a reward function for “helpful” but reasonably good at saying which of two answers is better. Pairwise comparisons are cheaper, more consistent between annotators, and avoid calibrating an absolute scale. The reward model converts those relative judgments into an absolute score that RL can consume, at a cost per query measured in a single forward pass rather than a human’s minutes.

Since 2022 the field has branched in three directions. First, the preference pipeline itself matured, with open reward models trained on millions of pairs and with direct-optimization alternatives such as DPO that fold the reward into the policy; our benchmark of DPO versus RLHF and SFT for LLM alignment covers when you can skip an explicit reward model. Second, reasoning models raised the stakes: a long chain of thought has many places to go wrong, so grading only the final answer discards most of the signal, which motivated process reward models. Third, tasks with checkable answers moved to reinforcement learning with verifiable rewards (RLVR), popularized by Tulu 3 (arXiv 2411.15124) and DeepSeek-R1 (arXiv 2501.12948), where a program, not a network, returns the reward. The mechanics of that approach are in our explainer on GRPO and RLVR for reasoning models.

The original thesis of this post is a framing, offered as opinion: every reward signal is a verifier with a coverage-versus-reliability trade-off. A rule-based checker is nearly perfectly reliable on a tiny slice of tasks. A pairwise reward model covers everything but is only statistically reliable. A process reward model sits in the middle for multi-step reasoning. Choosing among them is choosing where you will accept error, and reward hacking is simply the policy locating the error you chose to accept.

The Reward Model Training Pipeline and the Bradley-Terry Model

A reward model is trained to predict which of two responses a human preferred. The standard objective is the negative log-likelihood of a Bradley-Terry model: the probability that response y_w beats y_l for prompt x is the sigmoid of the difference in scalar rewards, and training minimizes the negative log of that probability over the dataset. In InstructGPT, labelers ranked K = 4 to 9 responses per prompt, and all K-choose-2 pairs from one prompt were processed as a single batch element to avoid overfitting.

Figure 1: The reward model RLHF pipeline. Ranked samples become pairwise data, the Bradley-Terry loss trains a scalar-head model, and its scores drive PPO or best-of-N selection under a KL constraint.

Figure 1 shows the data flow end to end. The prompts are drawn from a distribution you care about, the candidate responses are sampled from the current SFT or policy model, and the preference labels come from humans or, increasingly, from an AI labeler. The reward model is then frozen and used as the optimization target. Two details matter in practice: the reward model is usually initialized from the same family as the policy so it shares representations, and the preference data should come from responses the policy actually produces, because a reward model is only trustworthy near the distribution it was trained on.

The Bradley-Terry model, from first principles

Bradley and Terry (1952) modelled paired comparisons by assigning each item a positive strength. If item i has strength p_i, the chance i beats j is p_i / (p_i + p_j). Substituting p = exp(r) turns this into a logistic function of a score difference:

P(y_w beats y_l | x) = sigmoid(r(x, y_w) – r(x, y_l))

The loss is L = -log sigmoid(r_w – r_l). Three properties follow. The model only constrains differences, so the reward has no meaningful absolute zero; adding a constant to every score changes nothing. Scores are on a log-odds scale, so a margin of 1.0 means odds of about 2.7 to 1 in favour of the winner. And the model assumes transitivity: if A beats B and B beats C, then A beats C. Real human preferences violate this regularly, particularly across heterogeneous annotators, which is an irreducible error source we return to later.

A worked numeric example

Take one comparison where the reward model currently assigns r_w = 1.2 to the chosen response and r_l = 0.4 to the rejected one.

Margin = 1.2 – 0.4 = 0.8.
sigmoid(0.8) = 1 / (1 + e^-0.8) = 1 / 1.449 = 0.690.
Loss = -ln(0.690) = 0.371.
Gradient of the loss with respect to r_w is -(1 – 0.690) = -0.310; with respect to r_l it is +0.310.

Now flip the situation so the model is wrong: r_w = 0.4 and r_l = 1.2. The margin is -0.8, sigmoid(-0.8) = 0.310, the loss is -ln(0.310) = 1.171, and the gradient magnitude is 0.690. A confidently wrong pair produces more than three times the loss and more than twice the gradient of a correct one of the same size, so training concentrates on its mistakes. When the margin is zero the model is at chance: sigmoid(0) = 0.5, loss = ln 2 = 0.693, gradient 0.5 each way. A well-trained reward model on a noisy human dataset rarely exceeds roughly 70 to 80 percent pairwise accuracy on held-out human labels (a commonly reported range, not a figure from one specific paper), because annotator agreement itself is only around that level. Treat the reward as a noisy ranking signal, not ground truth.

Variants worth knowing

Practitioners add a margin term when annotators supply strength ratings (Llama 2 used a margin-augmented Bradley-Terry loss), add a small regularizer that keeps reward magnitudes near zero so the scale does not drift, and sometimes train several heads for distinct attributes such as helpfulness and harmlessness. Multi-objective reward models then combine attributes with learned or hand-set weights. A different thread replaces the scalar with a distribution to expose uncertainty. Each variant addresses one weakness of the plain scalar Bradley-Terry head: scale drift, a single collapsed objective, or no sense of confidence.

How the reward is actually used

In PPO-based RLHF the sequence-level reward is typically assigned at the final token, and a per-token KL penalty between the policy and the frozen reference is subtracted: the shaped reward is r(x, y) – beta times log(pi(y|x) / pi_ref(y|x)). The KL term does two jobs. It keeps the policy in the region where the reward model was trained, and it is the only brake on reward hacking in the pure RL setting. In best-of-N (also called rejection sampling), the policy samples N responses and the reward model picks the highest, with no gradient step at all; this makes the reward model a test-time reranker and the cleanest setting for measuring how reliable it is.

Outcome Reward Models vs Process Reward Models

Outcome vs process supervision is the question of where in a solution the label attaches. An outcome reward model (ORM) scores a whole response, usually by whether the final answer is correct. A process reward model (PRM) scores every intermediate step, so a solution with a flawed middle step is penalized at that step even if a lucky error-cancelling path reaches the right answer. PRMs give denser, more localized credit assignment but cost more to label.

Figure 2: An outcome reward model emits one number for the whole solution, while a process reward model scores each step and exposes the first step where correctness collapses.

Figure 2 contrasts the two on a four-step solution. The ORM sees a single wrong final answer and has no idea which step caused it. The PRM shows the score holding near 0.9 for two steps, dropping to 0.40 at step 3, and staying low; the first bad step is localized. That localization is the entire value proposition, and it matters more as chains of thought get longer, because the probability that every one of many steps is correct decays geometrically.

Let’s Verify Step by Step and PRM800K

The landmark comparison is Lightman et al. (2023), “Let’s Verify Step by Step” (arXiv 2305.20050), from OpenAI. They collected PRM800K, which the authors describe as 800,000 step-level human feedback labels on solutions to MATH problems, and trained verifiers with each supervision type on the same generator’s samples. Their headline: the process-supervised reward model solved 78 percent of problems from a representative subset of the MATH test set when used to rerank many sampled solutions, significantly outperforming outcome supervision. They also report that active learning, which surfaces convincing-but-wrong solutions for labelers, substantially improves process supervision.

Two design details deserve attention. Labelers marked each step as positive, negative or neutral, and annotation for a solution stopped at the first clearly incorrect step. And the PRM’s score for a full solution was computed as the product of the per-step correctness probabilities. Both choices are consequential. Stopping at the first error avoids labeling the cascade of downstream steps that are wrong merely because they inherit a wrong premise, but it also means the PRM sees little data about what happens after an error.

Aggregating step scores: product versus minimum

Suppose a PRM assigns step correctness probabilities 0.95, 0.90, 0.40 and 0.35 to the four-step solution in Figure 2. The product is 0.95 x 0.90 x 0.40 x 0.35 = 0.120. The minimum is 0.35. Now compare a second solution with steps 0.85, 0.85, 0.85, 0.85: the product is 0.522 and the minimum is 0.85. Both rules rank the second solution higher here, but they disagree on others. A long, careful 12-step solution whose steps are each 0.95 has a product of 0.54, lower than a sloppy 3-step solution at 0.85 per step (product 0.61), even though the long one has no weak link. Product penalizes length; minimum does not. Practitioners therefore choose by task: product when length correlates with risk, minimum or last-step score when long careful proofs are legitimate. These numbers are illustrative, not measured.

Automated process labels: Math-Shepherd and Monte Carlo rollouts

Human step labels are expensive, so a second line of work estimates step quality automatically. Math-Shepherd (Wang et al., arXiv 2312.08935) labels a step by how often continuations sampled from that step reach the correct final answer. The step’s value is its estimated success probability under the generator. Reported results on Mistral-7B include step-by-step PPO raising GSM8K accuracy from 77.9 to 84.1 percent and MATH from 28.6 to 33.0 percent, and verification raising them to 89.1 and 43.5 percent, per the paper’s abstract.

The Monte Carlo label is cheap but biased in a specific way: it measures whether a step is recoverable by this generator, not whether it is logically valid. A wrong step that a strong model happens to repair later looks fine, and a correct step followed by a model that is weak at the next move looks bad. Later work, including the “Lessons of Developing Process Reward Models in Mathematical Reasoning” analysis (arXiv 2501.07301) and the ProcessBench evaluation (arXiv 2412.06559), argued that Monte Carlo-labelled PRMs often underperform human-labelled or judge-filtered ones at identifying the earliest erroneous step. I have not independently re-run those comparisons; treat the ranking as reported by those authors.

Why PRMs are harder than they look

A PRM needs a definition of “step”. Splitting on newlines is the common choice, but models can game it by emitting many tiny trivially-correct steps, which under a product rule dilutes nothing and under an average rule inflates the score. A PRM also inherits distribution shift: it is trained on one generator’s mistakes, and a different or stronger policy makes different mistakes. And in open-ended domains with no crisp notion of a wrong step (essay quality, strategy), process labels are ill-defined, which is why PRMs have succeeded mainly in math, code and structured reasoning.

Verifiers, Best-of-N, and Search with PRMs

A verifier is any function that scores a candidate solution for correctness; reward models used at test time are verifiers. The earliest large-scale demonstration is Cobbe et al. (2021, arXiv 2110.14168), who trained verifiers on GSM8K and showed that sampling many solutions and picking the verifier’s favourite outperformed fine-tuning alone. Verification rests on an asymmetry: for many tasks it is easier to recognize a correct solution than to produce one.

Figure 3: Best-of-N with a PRM verifier. The generator samples N full solutions, the PRM scores every step, and the selector aggregates and returns the top-ranked answer.

Best-of-N and weighted voting

The simplest use is best-of-N: sample N solutions and return the one with the highest verifier score. A more robust variant is weighted majority voting, where each distinct final answer accumulates the scores of every solution that reaches it. This combines self-consistency with verification, and it degrades gracefully when the verifier is noisy because many mediocre-scoring correct solutions can still outvote one confidently wrong outlier. In Lightman et al., the PRM outperformed both an outcome-supervised model and majority voting on the same samples.

The cost is linear in N. If a single solution is about 600 tokens and you sample N = 64, you pay roughly 38,400 generated tokens plus 64 verifier passes for one answer. That is a 64x inference bill (illustrative arithmetic), justified only where a correct answer is worth far more than the compute.

Guided search: beam search and tree search

Because a PRM scores prefixes, it can steer generation instead of merely ranking finished outputs. In step-level beam search the generator proposes several next steps, the PRM scores the partial solutions, and only the top beams continue. Snell et al. (2024, arXiv 2408.03314), studying test-time compute scaling, compared best-of-N, beam search and lookahead search guided by a PRM, and found that the best strategy depends on problem difficulty: on easy problems simple sampling suffices, while harder problems benefit from search. Their broader claim, that a smaller model with extra test-time compute can match a larger one in some regimes, is a result under their experimental setup and should not be generalized to every domain.

Search amplifies verifier errors in a way best-of-N does not. Best-of-N takes the maximum over independent draws, so a verifier mistake costs one candidate. Search commits to a prefix early, so a verifier that overrates a plausible but wrong step 2 prunes every correct alternative. This is the same over-optimization pressure described below, applied at inference time.

Generative verifiers and reward models that think

A scalar head cannot explain itself. Generative verifiers (Zhang et al., arXiv 2408.15240, “Generative Verifiers: Reward Modeling as Next-Token Prediction”) instead ask the model “Is this answer correct?” and read the probability of the token “Yes”, optionally after a chain-of-thought critique. This reuses the model’s language ability, lets verification scale with test-time compute by sampling several critiques and averaging, and trains with ordinary next-token loss that can be mixed with generation data. LLM-as-a-judge pairwise setups are a close cousin, with different bias profiles; our write-up on LLM-as-a-judge evaluation bias, calibration and rubrics covers position bias, verbosity bias and how to calibrate them.

The trade-off is cost and variance. A generative critique can cost hundreds of tokens per candidate where a scalar head costs one forward pass, and the critique itself can hallucinate a flaw. Teams typically use a scalar RM for high-volume filtering and a generative judge on the ambiguous middle.

Reward Hacking and Over-Optimization

Reward hacking is the policy raising its score under the proxy reward without raising true quality. It is Goodhart’s law in a training loop: once the proxy becomes the target, it stops being a good measure. Known symptoms include length inflation, sycophantic agreement, confident formatting with bullet points and bold text, refusal avoidance or excess refusal depending on the data, and in code settings editing the tests rather than fixing the bug.

Figure 4: As KL from the reference policy grows, proxy and gold reward diverge. Mitigations range from KL penalties and ensembles to replacing the learned reward with a rule-based check.

The Gao et al. scaling laws

Gao, Schulman and Hilton (2022), “Scaling Laws for Reward Model Overoptimization” (arXiv 2210.10760), made the effect measurable. Because collecting fresh human labels for every experiment is impractical, they used a fixed “gold-standard” reward model as a stand-in for humans, generated synthetic labels from it, trained smaller proxy reward models on those labels, and then optimized a policy against the proxy while tracking the gold score. They found that gold reward first rises and then falls as optimization proceeds, and that the shape follows smooth functional forms in the distance d = sqrt(KL) from the initial policy. As I recall the paper, best-of-n follows R(d) = d(alpha – beta d), while RL follows R(d) = d(alpha – beta log d), with coefficients that scale smoothly with reward-model parameter count. They also study how dataset size, model size and the KL penalty change the curves. Check the paper for the exact parameterization before reusing it in your own fits.

Best-of-n has a closed-form KL from the base distribution: KL = log n – (n – 1)/n. For n = 4 that is 1.386 – 0.750 = 0.636; for n = 16, 2.773 – 0.938 = 1.835; for n = 64, 4.159 – 0.984 = 3.175. So d = sqrt(KL) is about 0.80, 1.35 and 1.78 respectively. Take illustrative coefficients alpha = 1.0 and beta = 0.15 (invented for the example): R(d) = d(1 – 0.15 d) gives 0.70 at n = 4, 1.08 at n = 16 and 1.31 at n = 64, still climbing. The peak would be at d = 1 / 0.3 = 3.33, which requires KL near 11 and an n in the hundreds of thousands. The practical lesson: best-of-n is a gentle optimizer, and over-optimization is mainly an RL problem where KL can grow without bound.

Why it happens: three distinct mechanisms

The first is reward model error with a systematic bias. Annotators, human or AI, favour longer and more polished answers, so the reward model encodes “longer is better” as a feature. A policy discovers this shortcut quickly because it is cheap to exploit. The second is distribution shift: the reward model has only seen outputs near the SFT policy, and as the policy moves, it enters regions where the reward model’s extrapolation is arbitrary and occasionally spuriously high. The third is noise in labels: with perhaps 25 to 30 percent disagreement between annotators, the reward model learns a smoothed estimate and the policy exploits its high-variance regions, a winner’s curse effect where the maximum of many noisy estimates is biased upward.

Mitigations and their costs

The KL penalty is the baseline defense, but it trades capability for safety: a large beta suppresses hacking and also suppresses real improvement. Early stopping on a held-out gold or human evaluation is more targeted, because you stop at the peak of the curve, but it requires a trusted evaluation you may not have. Larger reward models and larger preference datasets shift the peak outward, consistent with the scaling results. Ensembles of reward models, taking the minimum or a conservative mean, reduce exploitable idiosyncrasies, although models trained on the same data share biases and ensembles help less than hoped. Length-controlled training, either through a length penalty or by decorrelating length from preference during reward training, targets the single best-known hack directly.

Periodic refresh also matters: collect new comparisons on the current policy’s outputs and retrain the reward model, an approach used in iterated RLHF. It keeps the reward model near the policy distribution but multiplies annotation cost with each round.

Evaluating Reward Models

If the reward model is the target of optimization, you need to measure it before you hand it a policy. The standard offline check is pairwise accuracy on held-out preference pairs, but this correlates imperfectly with how well the reward model performs when it drives RL or best-of-N.

RewardBench (Lambert et al., 2024, arXiv 2403.13787) provides prompt, chosen, rejected trios across categories such as chat, harder chat, safety and reasoning, built so that the preferred answer is verifiably better for reasons like a bug or a factual error. The benchmark evaluated classifier reward models and DPO-trained implicit reward models, and reported findings about refusal behavior, reasoning limits and instruction-following weaknesses. A follow-up, RewardBench 2, was released in 2025 with a harder multi-candidate format; I have not verified its detailed design here, so check the original release before citing specifics.

Three cautions apply to any reward model benchmark. First, high benchmark accuracy does not guarantee downstream gains, because benchmark pairs are not the policy’s own samples. Second, public benchmarks leak into training sets over time, inflating scores. Third, accuracy on easy pairs hides behavior on the pairs that matter during RL, which are near-ties where the policy is choosing between two plausible responses. For PRMs, ProcessBench (arXiv 2412.06559) asks a different question: can the model identify the earliest erroneous step in a solution?

A practical evaluation stack has four layers. Offline pairwise accuracy catches catastrophic regressions. Best-of-N against a trusted judge or a human sample measures whether the reward model’s top pick is actually better. Slice analysis by length, topic and style reveals bias. Finally, a canary RL run with logged proxy and gold curves shows over-optimization directly, which is the only test that mirrors deployment.

How RLVR Replaces Learned Rewards

Reinforcement learning with verifiable rewards swaps the learned reward model for a deterministic function: exact-match on a math answer, unit tests passing, a format check, a constraint solver. Tulu 3 (arXiv 2411.15124) named and applied this on top of preference tuning, and DeepSeek-R1 (arXiv 2501.12948) used rule-based accuracy and format rewards to train strong reasoning behavior. The DeepSeek authors reported deliberately avoiding neural reward models for the reasoning stage because of reward hacking risk at scale and the retraining cost. That is a design choice described in their paper; it is not a proof that neural rewards cannot work.

Why does this sidestep over-optimization? A rule-based check has no learned parameters to exploit, so the Gao failure mode, a policy finding blind spots in a network’s extrapolation, largely disappears for the checked property. The optimizer, commonly GRPO (introduced in DeepSeekMath, arXiv 2402.03300), samples a group of responses per prompt and normalizes rewards within the group, removing the need for a separate value network.

What RLVR does not remove

Verifiable rewards are only as good as the verifier. A math checker that parses the last number in a response can be fooled by a response that lists many numbers. A unit-test reward can be satisfied by special-casing the tests. Answer-only rewards are outcome supervision, so they inherit the credit assignment problem: a correct answer reached by flawed reasoning is rewarded. And the approach covers only tasks with checkable outcomes, leaving open-ended writing, advice and safety to learned rewards. In practice, 2026 pipelines are hybrids: rules for the verifiable slice, a preference reward model for style and safety, and sometimes a PRM or generative judge for reasoning quality where outcome checks are too sparse.

Where PRMs still fit in the RLVR era

If outcome rewards work, why bother with PRMs? Sparse rewards make long-horizon problems hard to learn: when only 2 percent of samples are correct, most groups in GRPO have zero variance and give no gradient. Dense step feedback can help there, and a PRM is useful at inference time for search regardless of how the policy was trained. Conversely, a PRM used as a training reward re-introduces a learnable target to hack. Several groups have reported that PRM-guided RL is fragile without safeguards; the safest default remains outcome rewards for training and PRMs for selection.

Trade-offs, Gotchas, and What Goes Wrong

Noise ceilings are real. If human annotators agree only about three quarters of the time, no reward model can reliably exceed that on held-out human labels, and chasing leaderboard accuracy past that point means fitting annotator quirks. Invest in label quality, adjudication and rubric clarity before model scale.

Reward models are narrow. A reward model trained on chat preferences can be uncalibrated on code or math, and one trained on one policy’s samples can misjudge another’s. Always evaluate on the distribution you will optimize.

Scale of the scalar is arbitrary. Because Bradley-Terry fixes only differences, reward magnitudes can drift between checkpoints, which breaks reward normalization, KL coefficient tuning and cross-model comparisons. Whiten rewards per batch or regularize them toward zero.

Position and verbosity bias in judges. LLM judges prefer the first response, or the longer one, more often than they should. Swap positions and average, and control for length.

PRM label leakage. Process labels from a strong generator’s failures may not transfer. A PRM trained on weak-model errors can score a strong model’s subtle errors as correct, because it never saw that error type.

Verifier compute dominates. Best-of-64 with a PRM multiplies cost; if serving budget is tight, a smaller well-tuned policy with a cheap verifier often wins on cost per correct answer. Measure cost per correct answer, not accuracy alone.

Do not mistake a rising training reward for progress. The signature of over-optimization is proxy reward rising while an independent evaluation stalls or drops. If you do not have the independent evaluation, you cannot see the failure.

Practical Recommendations

Start by deciding which parts of your task are checkable. Where a program can verify the answer, use it as the reward and skip the learned model for that slice; you remove the largest source of hacking at zero training cost. For subjective quality, train a Bradley-Terry reward model on comparisons drawn from your own policy’s outputs, hold out a gold set that never touches training, and track proxy versus gold throughout RL.

For reasoning tasks at inference time, a PRM plus weighted voting is a strong default, with the aggregation rule chosen on a validation set rather than by convention. Prefer human or judge-filtered step labels over pure Monte Carlo labels when you can afford them, and measure with an earliest-error benchmark in addition to downstream accuracy.

Keep the optimizer gentle. Use a KL penalty tuned on the gold curve, early stop at the peak, and refresh the reward model on the current policy if you run multiple rounds. Control length explicitly, because it is the first hack you will meet.

A short checklist:

Separate verifiable from subjective sub-tasks and route each to the right reward type.
Build a held-out gold evaluation before the first RL step.
Log proxy reward, gold reward and KL together every evaluation interval.
Debias for length and position; swap candidate order in judge prompts.
Compare product and minimum aggregation for PRMs on your own data.
Report cost per correct answer, not only accuracy.
Re-collect preferences on the current policy before retraining the reward model.

Frequently Asked Questions

What is a reward model in RLHF?

A reward model is a neural network, usually a language model with a scalar output head, trained to predict which of two responses a human would prefer. It converts relative human judgments into a single score for any prompt-response pair. During RLHF, the policy is optimized to maximize that score subject to a KL penalty keeping it close to a reference model, so the reward model effectively defines what “better” means during training.

How does the Bradley-Terry model work in reward modelling?

Bradley-Terry assumes the probability that response A beats response B is the sigmoid of the difference between their reward scores. Training minimizes the negative log of that probability on preference pairs. A margin of 0.8 in favour of the right answer gives a loss of about 0.37, while the same margin the wrong way gives about 1.17, so training focuses on the model’s mistakes.

What is the difference between a process reward model and an outcome reward model?

An outcome reward model scores the whole response, typically by final-answer correctness, giving one signal per solution. A process reward model scores each reasoning step, localizing where a solution goes wrong. Lightman et al. (2023) found process supervision significantly outperformed outcome supervision on MATH when reranking samples, at the price of far more expensive step-level labels, which approaches like Math-Shepherd try to automate.

What is reward hacking, and how do you detect it?

Reward hacking occurs when a policy raises its proxy reward by exploiting the reward model’s flaws rather than improving real quality. Typical signs are growing response length, flattery and formatting tricks. Detect it by tracking an independent gold or human evaluation alongside the proxy: if the proxy keeps rising while the gold score flattens or falls as KL grows, you are over-optimizing, as Gao et al. (2022) showed.

Does RLVR eliminate the need for reward models?

Not entirely. RLVR replaces the learned reward with a programmatic check where answers are verifiable, such as math and code, which avoids exploiting a neural network’s blind spots. It does not cover open-ended tasks like writing or safety, and answer-only rewards give no credit assignment inside reasoning. Most production pipelines therefore combine verifiable rewards with preference reward models or generative judges.

When should I use best-of-N with a verifier instead of RL?

Use best-of-N when you cannot retrain the policy, when you want to trade inference compute for accuracy on hard queries, or when you want a safer optimizer, since its KL from the base model grows only logarithmically with N. Choose RL when query volume is high enough that amortizing the improvement into the weights is cheaper than paying N times per query.

Reward Models Explained: Bradley-Terry, Process Reward Models and Verifiers in RLHF and RLVR (2026)