← logs/

The GRPO family tree: what each descendant actually fixes

I have implemented GRPO from the DeepSeekMath paper, and I have run it against a programmatic verifier until the run produced a checkpoint worth shipping. So this post is not a survey written from abstract pages. It is a family tree. GRPO is the ancestor, and every serious descendant exists because it patches one specific pathology in the ancestor. Once you know which pathology each patch targets, the acronym zoo turns into a change log.

The tree has two branches. The reasoning branch fixes problems inside a single rollout: DAPO fixes clipping, sampling, and credit normalization; Dr. GRPO fixes a length bias hidden in the advantage estimator. The agents branch fixes the unit of credit: GiGPO and ARPO both move from trajectory-level to step-level credit, in two very different ways, because a multi-step agent trajectory cannot be scored like a long math answer.

GRPO (DeepSeekMath, 2024) — group-relative advantage, no critic
├── DAPO (2025)          — clip collapse, zero-reward groups, token credit, length hacking
├── Dr. GRPO (2025)      — length-normalization and reward-standardization bias
├── GiGPO (NeurIPS 2025) — trajectory-level credit → step-level credit for agents
└── ARPO (ICLR 2026)     — trajectory-level sampling → step-level exploration after tool calls

The ancestor: critic-free by construction

GRPO is PPO with the value network removed. DeepSeekMath samples G responses per prompt, scores them, and uses the group itself as the baseline. No critic, no bootstrap, no learned advantage. The whole estimator is:

A_i = (r_i - mean(r)) / (std(r) + epsilon)

and the update is the clipped policy ratio with a KL penalty to a frozen reference policy, straight out of the PPO playbook. What this buys you is concrete: a critic costs roughly as much memory as the policy itself, and a bad critic destabilizes training in ways that are hard to diagnose. GRPO trades the critic for a noisier group baseline and calls it a day. DeepSeek-R1 then showed the same recipe at scale, with plain verifiable rewards, producing emergent self-correction and verification behavior.

The dangerous part is what GRPO leaves open. Every line of that estimator is a hidden inductive bias. Reward standardization makes each advantage depend on the other samples in its group. Loss normalization makes per-token credit depend on response length. The group size G is an implicit exploration budget. Most descendants are papers that discovered one of these hidden choices was lying to the optimizer, then patched exactly that line.

One more thing is worth saying about the ancestor before the tree grows. GRPO assumes the reward is comparable across the group. That holds for a math answer, where the verifier is a correctness check. It weakens the moment the reward is a proxy, an LLM judge, a heuristic, or a partial-credit formula, because then the group baseline is normalizing noise as much as signal. This is the fault line the agents branch keeps running into.

DAPO: four patches, four failure modes

DAPO is ByteDance Seed and Tsinghua AIR's open-source patch release for GRPO, and the closest thing the family has to a canonical one. It ships a full verl-based RL system and reports 50 points on AIME 2024 with a Qwen2.5-32B base model. The four techniques map one-to-one onto failure modes:

  • Clip-Higher. Vanilla GRPO clips the policy ratio symmetrically around 1. When a token with positive advantage keeps rising in probability, the upper clip kills its gradient, and entropy on winning responses collapses. Clip-Higher raises the upper bound so positive-advantage tokens stay learnable. It patches premature entropy collapse.
  • Dynamic Sampling. In math RLVR, a large share of prompts produce groups where every sample scores zero. Those groups teach nothing and still cost a full rollout pass. Dynamic sampling drops the groups whose samples all score the same, all wrong or all right, and oversamples the mixed groups that still carry gradient signal.
  • Token-level loss. GRPO normalizes the loss per response, so a 2000-token correct chain and a 40-token correct chain contribute the same gradient mass. DAPO normalizes by total tokens in the group, so long chains get credit proportional to the work they did. It patches gradient dilution on long reasoning.
  • Overlong reward shaping. A raw correctness reward leaves a length hack open: write more tokens, raise the odds that the right token appears. DAPO adds a soft penalty for overlong responses. It patches the most common RLVR failure at scale.

Read together, DAPO is the argument that GRPO was not wrong in kind, only under-engineered. That is mostly true, and the reproducibility of the open system is a genuine contribution. But the four patches share a boundary: DAPO shapes the length incentive, it does not remove it. The clip bounds are tuned constants; nobody derived them. And none of the four touches the unit of credit: a trajectory is still scored as one blob. That boundary is exactly where the next two branches start.

Dr. GRPO: the two-line fix

Understanding R1-Zero-Like Training, by Liu et al., is the paper that deflated the "aha moment" narrative and then introduced a two-line algorithm called Dr. GRPO. The finding: GRPO has an optimization bias that artificially increases response length, especially for incorrect outputs. The mechanism lives in the interaction of two normalizations. Length normalization divides each response's loss by its token count, and reward standardization divides the group's rewards by their standard deviation. The per-token advantage a long, wrong response receives is not the signal the reward intended. The practical effect is a policy that learns to write longer rather than better.

Dr. GRPO removes response-length normalization and reward standardization from the estimator. That is the whole patch. The paper reports it improves token efficiency while keeping reasoning performance, and its minimalist recipe reaches 43.3% on AIME 2024 with a 7B base model.

This is my favorite patch in the family because it is the most honest. DAPO fights the length hack with a reward shape. Dr. GRPO deletes the bias that created the hack. But it also leaves a hole: remove the length incentive and you still need some incentive for the extra thinking that hard problems require. The pathology is fixed; the problem is not. Which is fine, as long as you know that when you pick the patch.

GiGPO: when the group is a trajectory

GiGPO, by Feng et al., accepted at NeurIPS 2025, is the first serious break with the single-turn frame. An agent rollout is a trajectory of environment interactions, the reward arrives at the end, sparse and delayed. Trajectory-level GRPO assigns one advantage per trajectory, which is grading an essay with a single number: the good steps and the bad steps share one credit.

GiGPO keeps the episode-level group and adds a step-level group. The mechanism is anchor state grouping: when two trajectories pass through the same environment state, the actions taken at that state form a group, and each action's micro advantage is computed relative to its peers at that same state. Repeated states become the anchor points where step credit can be assigned without a learned value function, without extra rollouts, and without additional memory. The paper reports gains over GRPO of more than 12 points on ALFWorld and more than 9 points on WebShop, with the same GPU footprint.

What remains unfixed is structural. Anchor states only exist when the environment exposes observable, repeatable states. That holds in a simulated household or a web shop. It mostly fails in open-ended coding or research tasks, where trajectories rarely revisit the same state and the step-level groups come up empty. GiGPO also inherits GRPO's group sensitivity: the episode-level baseline is only as good as the diversity inside the group. Anchor state grouping is also the first place where the tree learns from the environment rather than from the rollout: the step groups are discovered from observed states rather than manufactured by the algorithm. That is a different kind of patch than DAPO's, and it points at where the next generation of variants will live.

ARPO: entropy spikes after the tool call

ARPO (Dong et al., ICLR 2026; arXiv version) starts from a different observation. After a tool call, the model's token entropy spikes. The model genuinely does not know what the tool returned, and that uncertainty is a signal about where exploration is needed. Trajectory-level sampling ignores it: every step of every trajectory gets the same rollout budget, whether the policy is confident or lost.

ARPO's entropy-based adaptive rollout spends the group's budget where entropy is high, branching step-level samples at the tool-call rounds that need them, and an advantage attribution mechanism pushes step-level credit back through the tool-use interactions. Across 13 benchmarks spanning computational reasoning, knowledge reasoning, and deep search, ARPO reports consistent gains over trajectory-level RL algorithms, at roughly half the tool-use budget. The half-budget result is the part I care about most: it is evidence that rollout budget is being spent where the policy is confused, which is a much stronger claim than a benchmark delta.

The open question is the same one that runs through the whole agents branch: the entropy signal says where to look, not what is correct. ARPO fixes the sampling pathology of trajectory-level RLVR. The reward sparsity that makes agent credit hard in the first place is still the verifier's problem; the algorithm cannot fix it.

My position: keep a pathology ledger, not a variant zoo

The tree only looks like a zoo from a distance. From inside a training run, each descendant is one fix to one measurable failure:

| Pathology | Where it lives | Patch | Variant | | --- | --- | --- | --- | | Entropy collapse on winning responses | symmetric clip | asymmetric upper bound | DAPO | | Zero-reward groups waste compute | sampling | drop and oversample | DAPO | | Long-chain gradients diluted | loss normalization | token-level loss | DAPO | | Length hacking | reward | overlong shaping | DAPO | | Length and std bias in the advantage | estimator | drop both normalizations | Dr. GRPO | | Trajectory-level credit | advantage unit | anchor-state grouping | GiGPO | | Post-tool uncertainty ignored | rollout policy | entropy-adaptive branching | ARPO |

On top of this, the useful artifact is a diagnostic harness; the objective itself can stay put. Instrument a GRPO run for the seven pathologies before choosing a variant: per-prompt zero-reward fraction, correct-versus-incorrect length divergence, entropy trajectories at tool boundaries, state-recurrence rate in agent environments, group std collapse over steps. Each measured signal maps to a patch in the ledger, so variant selection stops being vibes and starts being readings. I suspect most real runs want two or three patches at once, and the ledger is the only honest way to pick them.

Where this bites: my own training loop

I have written separately about the reward side of RLVR in RLVR and its cracks and about verifier quality in verifier engineering. This post is the algorithm side of the same story.

VerifierForge trains with GRPO against a programmatic verifier. The verifier is executable, not a model: extract one read-only SQL statement, parse it, run it against a frozen schema, and compare the result set with the expected rows. Valid and executable SQL earns partial credit; only an exact result-set match earns full reward. The loop samples G=8 completions per prompt, normalizes each reward against its group, and applies the clipped update with a KL penalty. That estimator is the one from the DeepSeekMath paper. I implemented GRPO myself from that paper in my HoReN-paper-reproduction repo, and that implementation drives VerifierForge's training loop.

Three details from that run belong in this post. First, the length tension is deliberate. The estimator keeps GRPO's vanilla per-response loss normalization and group reward standardization — the exact two lines Dr. GRPO flags as bias sources — while the verifier subtracts 0.05 from any completion past 400 characters, which is DAPO-style overlong shaping bolted onto the reward itself. I run that combination on purpose: the textbook estimator is the one my reproduction implements and the one I have debugged, and the shaping term lives in verifier code I can audit, so I pay for a known estimator bias with a visible reward patch. Which side of that trade dominates on this workload, I have not measured; the pathology ledger is real even when you do not use the papers' names for it, and the ledger says which instruments would tell me. Second, the config arms an entropy brake — a tripwire against the entropy collapse DAPO's Clip-Higher exists to fight — and across the full 400 steps it never triggered. An untriggered guard is still a reading: on this workload, collapse never started. Third, the run ships a deliberately imperfect random-reward control next to the verifier run, and the shipped checkpoint is chosen on a frozen held-out exam, not by taking the last step. The family tree is full of clever objective surgery, but the boring selection and falsification discipline is what keeps the surgery honest.

The pattern

GRPO won because it removed the critic. Its descendants are winning because they remove the remaining hidden biases, one estimator line at a time. When I read a new GRPO variant now, I ask one question: which pathology does it name? A variant that names one is an engineering fix you can evaluate. A variant that names none is probably trading one bias for another. The tree makes the field legible: same ancestor, same group baseline, one specific lie removed per branch.

Sources linked in this post were fetched and verified.