RLVR and its cracks: what verifiable rewards actually teach

RLVR is the best training signal we have for reasoning models, and it is also the one most likely to lie to us. The pitch is seductive: drop the learned reward model, check answers with a program, and the reward can no longer be gamed. Three papers — from April 2025, June 2025, and October 2025 — each found a crack in that story, from opposite ends. Random rewards produce nearly all of the gain on a popular family of math models. A single training example produces most of the gain too. And when the verifier itself mislabels, the loop quietly trains on noise.

One of those failure modes I have verified in my own training runs, with a control arm that pays the model in random rewards. A second I engineered against before it could bite: my verifier records enumerable per-sample failure classes — parse_failure, execution_error, executable_not_full_pass — so a verifier miss shows up as a labeled class instead of silent noise. The third I know from a reproduction, not a scar. They converge on the same conclusion: the RL loop is no longer the bottleneck. The verifier is. This post reads the three papers, then walks through the control arm I ran on VerifierForge that turned the first paper's warning into a concrete number.

The field map

RLVR replaces the reward model with a deterministic check. The pipeline is small:

policy samples -> programmatic verifier -> binary reward -> GRPO update

GRPO dominates the family because it drops the critic: rewards are normalized against a group of samples for the same prompt, and the policy is updated with a clipped surrogate objective. It is cheap, it scales, and it is what most open post-training runs use today. I covered the family tree in my GRPO notes, so here I only need the parts that matter for the cracks: the clip term, the group normalization, and the fact that the verifier is the only source of truth in the loop.

Where RLVR is strong, it is very strong. Math and code have exact, cheaply checkable answers, and the reward signal is dense enough that models like the Qwen2.5-Math line climb fast. Where it is stretched, it is stretched exactly where verifiers get fuzzy: agentic tasks, long-horizon trajectories, open-ended writing. That is not a coincidence. The instrument that makes RLVR work in math is the same one that breaks elsewhere, and agentic RL inherits every crack in this post.

The three cracks I care about:

1. Spurious rewards: the reward carries no information, and the gradient still moves.
2. Noisy verifiers: the reward is wrong a fraction of the time, silently.
3. Extreme data scarcity: one example is enough, which says something deep about what the gradient is actually doing.

Close read: spurious rewards

Spurious Rewards: Rethinking Training Signals in RLVR (Rulin Shao et al., 2025) is the paper I wish every RLVR practitioner had read before their first training run. The headline result: on Qwen2.5-Math-7B, GRPO training with randomly assigned rewards improves MATH-500 by 21.4 percentage points, nearly matching the 29.1-point gain from ground-truth rewards.

Random rewards. The model gets better at a math benchmark while being paid in noise.

The mechanism: GRPO's clipped surrogate objective carries a bias that can amplify behaviors the model already prefers from pretraining, even when the reward has zero information. The clip term is supposed to stabilize training by capping the importance ratio, but the cap is asymmetric: it limits how hard the update can push against samples the policy already favors. With an informative reward that asymmetry is harmless, even useful. With a random reward it is the only thing left in the update, so the policy drifts toward whatever it already does often. The paper's case study is a behavior they call code reasoning: producing a solution as code without executing it. Under spurious rewards, its frequency in Qwen2.5-Math models climbs from 65% to over 90%. The model is not learning math. It is learning to lean harder into a habit the base model already had.

Two consequences matter for anyone training with verifiers. First, the effect is model-dependent: the same spurious-reward setup does not produce gains on Llama3 or OLMo2. A positive RLVR result on one model family is evidence about that family alone, which is exactly the paper's closing point: validate RL methods across diverse models rather than a single de facto choice. Second, benchmark gains are not capability gains. A lift on MATH-500 under a corrupted reward is a red flag, not a win.

This is why I treat control arms as non-negotiable, and why the M3/M4 comparison in my own work below exists at all.

Close read: one-shot RLVR

Reinforcement Learning for Reasoning in Large Language Models with One Training Example (Yiping Wang et al., NeurIPS 2025) attacks the question from the other end: how little signal do you need? Answer: one example.

Training Qwen2.5-Math-1.5B on a single carefully chosen math problem lifts MATH-500 from 36.0% to 73.6%, 8.6 points beyond what pure format correction buys, and the average over six math benchmarks from 17.6% to 35.7%. One example matches the 1.2k-example DeepScaleR subset at 73.6% MATH-500 and 35.9% average. Two examples edge past it: 74.8% and 36.6%. The result holds across model families, including Qwen2.5-Math-7B, Llama3.2-3B-Instruct, and DeepSeek-R1-Distill-Qwen-1.5B, and across both GRPO and PPO.

Three phenomena make the paper worth a close read. Post-saturation generalization: test performance keeps improving after training accuracy saturates, the signature of something other than memorizing the training answer. Cross-category generalization: the single example improves categories it does not belong to. And the ablation showing the gain comes from the policy gradient loss rather than from grokking-style representation dynamics. The paper also reports observations on label robustness and prompt modification, which are worth reading with the noisy-verifier problem in mind: even oracle-style labels are not as clean as they look.

The exploration angle is the one I want to emphasize. The authors show that promoting exploration, for example an entropy loss with the right coefficient, is critical for one-shot training to work. With one example the gradient is a single noisy direction; if the policy collapses onto it, you learn one behavior. If the group stays diverse, you get a search process that keeps paying off. That distinction, search versus imitation, is the whole game, and it is the same distinction that decides whether an RLVR run generalizes at any data scale.

I reproduced this result in my paper-reproductions repo, which lists it as a NeurIPS 2025 paper and the official program confirms. Why spend a run on a one-example result? Because it is the cheapest possible probe of whether a verifier carries information. If a model cannot improve from a single example under your verifier, no amount of data will save the loop. The crack and the opportunity are the same object.

Close read: noisy verifiers

RL with Verifiable yet Noisy Rewards under Imperfect Verifiers (2025) attacks the assumption the other two papers leave standing: that the verifier is right. It is not, and the paper formalizes what that does to training.

The setup models the verifier as a stochastic reward channel with two asymmetric noise rates: the false-positive rate ρ0, accepting a wrong answer, and the false-negative rate ρ1, rejecting a correct one. Both are common in practice. Exact-match graders on formatted outputs produce false negatives constantly. Programmatic checks that are narrower than the task produce false positives. Either way, the binary reward is wrong some fraction of the time, and the policy trains on the error as if it were truth.

The paper derives two lightweight corrections, implemented as hooks into a GRPO pipeline. The backward correction yields an unbiased surrogate reward, and therefore an unbiased policy-gradient estimator in expectation: you can train as if the verifier were clean. The forward correction reweights the score-function terms so the expected update aligns with the clean gradient direction, and it needs only the false-negative rate. Both improve math reasoning under synthetic and real verifier noise, and the forward variant is more stable when the noise is heavy.

The most useful piece for me is the appeals mechanism: a lightweight LLM verifier re-checks rejected samples to estimate the false-negative rate online, and that estimate feeds the correction. It turns "how wrong is my verifier" from a vibe into a measured, continuously updated number, and it is cheap because only the rejected minority gets re-checked.

Takeaway: a noisy verifier is not a reason to avoid RLVR. It is a reason to instrument the verifier and correct the reward, the same way you would instrument any measurement device before trusting its readings.

My position: what I would build

Put the three papers next to each other and a consistent picture emerges. The reward is a measurement, not a ground truth. It can be empty, nearly empty, or wrong, and in every case the loop dutifully turns it into gradients. The loop does what it is told. What you get out is bounded by the verifier, and by whether you can tell a real gain from a clipped-prior artifact.

So what I would build, and what I did build, is a verifier-first training stack with three parts.

First, verifier auditing as a standard step. Before any run, estimate the false-positive and false-negative rates on a labeled held-out set, and re-estimate them during training with an appeals-style sampler. Put ρ0 and ρ1 on the wall next to the loss curve. A drifting verifier is the earliest warning of a drifting reward.

Second, corrected rewards by default. The forward correction from the noisy-verifier paper is almost free as a hook, and it removes a whole class of silent training bugs. If your verifier rejects correct answers, and it does, you are currently training with a biased estimator.

Third, control arms inside the training loop. The spurious-rewards paper showed the danger; the fix is cheap. Keep a random-reward arm and a verifier-reward arm of the same run and compare them. If they converge, your verifier is not doing the work, and your benchmark numbers are suspect no matter how good they look.

One more design point. The noisy-verifier paper notes that many RLVR systems binarize rewards precisely to reduce verifier hacking, and it formalizes what that costs: the channel collapses to accept or reject, and only the two noise rates remain. When the verifier can say more than yes or no, for example scoring tier by tier like VerifierForge does, keeping that intermediate structure is usually worth it. Information at the margin is what separates a reward that teaches from one that only sorts.

The one-shot result adds the efficiency consequence: with a good verifier, you need very little data, so the loop worth optimizing is the verifier feedback cycle; the data pipeline can stay small. A fast, correct verifier you trust is worth more than another ten thousand examples. That trade gets sharper in agentic settings, where ground-truth checks barely exist and the verifier problem is worse. I wrote about that in my agentic RL notes.

What I actually did: VerifierForge

That position is not hypothetical. VerifierForge is a training harness I built solo in one week, with a live web demo at verifierforge-web.vercel.app. You hand it a programmatic verifier; it trains a small model with GRPO against it. The flagship eval is SQL completion, where the verifier scores responses tier by tier:

extract -> parse-guard -> execute -> compare

The tiered shape is the right instinct for the noise problem: each stage is a cheap, precise gate, so the channel stays clean instead of trusting one black-box judge. And every sample lands in a recorded failure class — parse failure, execution error, executable but not a full pass — so what the verifier did is auditable per sample; nothing is left to vibes. The results on a frozen 60-row held-out exam: independent pass@1 went from 0.583 to 0.783, a +20pp gain, and pass@8 from 0.767 to 0.900.

Given everything above, the check I care about most is the falsification arm. I ran a spurious-reward control on the same task and frozen training pool: a random-reward run (M4), paid with a deterministic Bernoulli(0.5) reward derived from immutable input hashes, alongside the verifier-reward run (M3). The asymmetry deserves to be said out loud, because the repo itself says it: M3 is Qwen2.5-1.5B-Instruct trained for 400 steps, M4 is Qwen2.5-0.5B-Instruct trained for 200, so reward was not the only changed variable. The repo's own framing is the right one: a useful falsification check, with no pretense of a strict causal experiment, and neither training-pool curve is the quality claim. What the arm showed: M4's training-pool monitor wobbled 0.4 to 0.5 and back, ending at 0.40, while M3's ended at 0.80. One reconciliation matters here: the 0.80 is the per-step monitor on the 50-row training pool, a different instrument from the frozen 60-question held-out exam that produced the 0.783 above. And M4 never got a held-out exam at all, by design. What the control rules out is narrow and worth having: the verifier-rewarded curve is not what a reward-free run does. The quality claim rests on the held-out exam, not on the comparison.

Two scars from that build are worth more than the headline number. First, a hidden total_epochs=10 default killed my first 400-step run at step 120: the outer loop ended at 12 × 10 = 120 steps, nothing crashed, and the run simply looked finished. The repair is a config guard (trainer/grpo_config.py:185-194) that raises before training starts if total_epochs would cap the run below its approved step target — the class of bug it kills is the silent kind, the config equivalent of a spurious reward. Second, checkpoint selection. All eight checkpoints sat the same frozen 60-question exam; step 350 won at 0.783 pass@1 while the final step 400 fell to 71.7%. So the shipped checkpoint is the best verified one. I don't ship the last step.

I also reproduced One-Shot RLVR, the single-example result, in my paper-reproductions repo. The point was not the benchmark number. It was the probe: with a verifier that carries real information, a single example should move the model. If it does not, the problem is upstream, in the reward, and adding data will not fix it.

The crack to engineer for

None of this makes RLVR less attractive. It makes it precise. The reward channel is the one part of the stack you can actually improve: make the verifier strict where it matters, measure its noise rates, correct its bias, and run the control arm that proves the signal is real. Do that, and the loop below it mostly takes care of itself.

Sources linked in this post were fetched and verified.