Verifier engineering as a discipline

Reward design is where RLVR actually lives. The policy model gets the credit, but the verifier decides what counts as correct, and the optimizer will find every gap in that decision. Get the verifier wrong and you do not get a weak model. You get a model that is confidently wrong in exactly the way the verifier cannot see.

That is why I treat verifier engineering as its own discipline: reward design, hacking, and robustness. Not a footnote to training, but the part that decides whether training is worth anything.

Field map

Two families, one axis.

Rule-based verifiers parse the output and compare it against a reference. Deterministic, cheap, no model in the loop. Their failure mode is brittleness: the same answer written differently scores as wrong.

Model-based verifiers score outputs or reasoning steps with another model. Flexible, good for open-ended tasks, but they are models themselves: gameable, drifty, and expensive to run at scale.

The axis is where the signal sits. Outcome rewards judge the final answer. Process rewards judge the steps. The same verifier family behaves completely differently on each side. For the RL mechanics around this, see grpo-family-tree.

Close-read: three papers

Rule-based verifiers are brittle in a specific way

From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning (Huang et al., 2025) runs the experiment that needed running: what happens when verifiers are wrong in real RL training? Open-source rule-based verifiers often fail to recognize equivalent answers presented in different formats across common math datasets. The paper puts a number on it: in static evaluation on responses from a strong long-CoT model, the rule-based verifiers held precision above 99% but averaged only 86% recall, meaning 14% of correct responses were scored as incorrect, and recall dropped to 0.78 on the hardest dataset. That false negative rate is not a constant. It bites harder as the policy model gets stronger, because a stronger policy produces more varied-but-correct output, which the verifier then rejects.

The static numbers are the trap. Model-based verifiers achieve higher verification accuracy in static evaluation, but under RL they get hacked: they misclassify certain response patterns as correct, the policy exploits that, and rewards inflate without real gains. Static verifier accuracy does not transfer to robustness under optimization. That is the paper's core lesson, and the reason I distrust verifier accuracy numbers presented without the training curve.

Process rewards need credit assignment, not just density

PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization (Ding et al., 2026) attacks the other side of the same problem. Even a good verifier yields one scalar per trajectory in critic-free setups like GRPO, leaving intermediate reasoning unsupervised. Process reward models fix that, but used alone they risk premature collapse, as early low-reward tokens push the policy toward truncated outputs.

PRPO stays critic-free. It segments reasoning sequences on semantic clues, normalizes PRM scores into token-level advantages, and aligns them with outcome advantages through location-parameter shift. On MATH500 it raises Qwen2.5-Math-1.5B from 61.2% to 64.4% over GRPO with only eight rollouts and no value network. The detail I care about: verifier design and credit assignment are now inseparable. Dense process signal is only as safe as the mechanism that distributes it.

The grader as a product, and the product winding down

OpenAI's reinforcement fine-tuning docs make the grader the whole product. You define a programmable grader, the platform samples responses, scores them, and applies policy-gradient updates. The docs are unusually honest about failure modes: the task should be guess-proof, because a model that can score reward on a lucky guess gets a noisy signal, and the model can learn to hack the grader, producing high scores without being correct.

Then the platform started winding down. The same official docs state that the fine-tuning platform is no longer accessible to new users, and the deprecation timeline pins the concrete deadline: active existing customers can create new training jobs only until January 6, 2027, and fine-tuned models remain available for inference until their base models are deprecated. RFT as a hosted service is closing. That pushes verifier work in-house: teams now own their graders end to end. Owning it end to end is also what the hosted loop could never give you: VerifierForge runs a random-reward falsification arm and a hash-pinned held-out exam alongside training, and no hosted service was going to run a control arm to check whether your grader was the thing that worked.

OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning (Zhang et al., 2024) showed the open-source version of the loop: fine-tune a generalist reasoning model for domain tasks under RFT-like settings, using question augmentation, synthesized reasoning-process data, and few-shot in-context learning, with notable gains on SciKnowEval from only 100 domain-specific samples per task. With that few samples, every reward decision carries weight. That is the regime where verifier engineering matters most.

My position

Two rules fall out of the close-reads. The third is what I had to invent that the papers didn't give me.

Prefer programmatic verifiers wherever a program can define correctness. A verifier you can execute beats a verifier you can argue with.

Structure reward as tiers, not a scalar. Each tier boundary should be a real check, and reaching a higher tier must score strictly higher. Scalar scores hide which capability the policy actually gained. None of the three papers hands you this; they evaluate verifiers, they do not design rewards. The tier ladder was the piece I had to work out in practice, building VerifierForge, and it is what makes failures enumerable instead of a single mysterious number.

Run a falsification arm. A control with random reward tells you whether your curve rose because of the verifier or despite it. If the control rises too, your reward signal is leaking.

What I would build next is the discipline itself: verifier benchmarks, standard falsification arms, and a public taxonomy of verifier failure modes. Verifier engineering deserves the same rigor as training.

VerifierForge

VerifierForge is my attempt to apply those rules end to end (github.com/xesws/verifierforge). A programmatic verifier scores SQL completions tier by tier: extract a read-only statement, parse-guard it, execute it against the frozen schema, compare the result set. The highest tier reached is the reward, with partial credit for valid and executable SQL and full reward only on an exact result-set match.

The falsification arm is the part I am proudest of. The random-reward control's training-pool monitor ended at 0.40, after wobbling 0.4 up to 0.5 mid-run and back to 0.4 at the final step, while the verifier-rewarded run's monitor ended at 0.80. One clarification that matters: those are training-pool monitors, a different metric from the held-out exam, and I do not use either monitor as the quality claim. The arm is a falsification reference rather than strict causal proof, and I treat it as such. On the frozen 60-question held-out exam, pass@1 improved from the base model's 58.3% to 78.3% at the selected checkpoint, step 350 of 400; the final step fell to 71.7%, so the shipped model is the best verified checkpoint, not the last one.

The thesis is that verifier quality, not model size, is the leverage. Everything else in the loop, from training to serving, is logistics around that.

Sources linked in this post were fetched and verified.