Agentic RL: training the agent, not prompting it

For most of the past year, the default answer to "my agent is not good enough" was a better prompt: longer system instructions, sharper tool descriptions, a more elaborate reasoning scaffold. That path has diminishing returns, because prompting can only expose capability the policy already has. The alternative with real evidence behind it is to train the agent: treat the policy as the object, the environment as the training ground, and RL as the gradient on agentic behavior.

Search-R1 taught a model when to call a search engine. ReTool taught one when to open a code interpreter. iStar taught step-level credit assignment when rewards are unverifiable. And an ICML 2025 result supplies the framing: SFT memorizes, RL generalizes.

The field map: agents as policies in a POMDP

The survey by Zhang et al. (The Landscape of Agentic Reinforcement Learning for LLMs, published in TMLR in January 2026, synthesizing 500+ works) draws the line cleanly: classic LLM RL is a degenerate single-step MDP, prompt to response to reward. Agentic RL is a POMDP: multi-step interaction, partial observability, and an action space that interleaves free-form text with tool calls and environment commands. Their taxonomy organizes the field by capability: planning, tool use, memory, reasoning, self-improvement, and perception.

My own map of the design space has three axes:

1. Reward source: verifiable (test pass, task completion, search hit) versus learned or judged.
2. Credit assignment: outcome-level versus step-level.
3. Algorithm: the policy-gradient family (PPO, GRPO, see grpo-family-tree) versus the preference family (DPO and friends).

The question that splits the field in practice: can you afford rollouts, and can you write down a reward? This post is the training-side complement to my agent taxonomy (agent-taxonomy-four-types).

Close read: Search-R1

Search-R1 (Jin et al., March 2025) is RL for search-augmented reasoning. The model learns to emit search queries mid-reasoning, interacts with a real search engine over multiple turns, and gets a simple outcome-based reward. Two engineering choices stand out: retrieved tokens are masked during training for stable RL, and the reward stays coarse.

Results: 41% improvement for Qwen2.5-7B and 20% for Qwen2.5-3B over RAG baselines on seven QA datasets. The point is not the benchmark numbers. The point is that when to search, what query to write, and when to stop searching became learned behavior; nobody prompted it in.

Close read: ReTool

ReTool (Feng et al., April 2025) does the same trick for code interpreters. The model interleaves real code execution into its reasoning, and RL with outcome feedback trains the tool-use strategy. The recipe matters as much as the algorithm: synthetic cold-start traces, SFT to bootstrap, then RL.

Numbers: a 32B model reaches 67% on AIME in 400 training steps, versus 40% for a text-only RL baseline at 1080 steps, and 72.5% in an extended setting, 27.9 points above o1-preview. The model also shows emergent code self-correction, the tool-use version of the "aha moment".

Two lessons. Tool invocation is a strategy, and strategies are best learned from outcome feedback. And the SFT cold start is not a compromise, it is the format bootstrap that makes RL work, exactly the complementarity the memorization paper predicts.

Close read: iStar, or credit assignment without checkable rewards

The hard case is where task success is not a checkable boolean: open-ended interaction, social tasks, long-horizon delegation. Agentic RL with Implicit Step Rewards (iStar, ICLR 2026) tackles credit assignment there with implicit step rewards. It jointly trains an implicit process reward model with the policy, using a multi-turn DPO objective over trajectory preferences, then combines step-level advantages with episode-level ones. No extra rollouts, no step labels, though the step rewards inherit whoever produced the trajectory preferences, so the verifier question returns one level up: preferences of which judge, against what rubric?

Results on WebShop, VisualSokoban, and SOTOPIA: state of the art versus frontier LLMs and strong RL baselines, with better sample efficiency and training stability. My read: this is the direction that matters. Real agent tasks mostly do not come with verifiers, so the field will live or die on learned, step-level credit assignment.

Why train instead of prompt

SFT Memorizes, RL Generalizes (Chu et al., ICML 2025) built two testbeds, an arithmetic reasoning card game and a navigation environment with textual and visual variants. RL with outcome rewards generalized across unseen rule variants and unseen visuals; SFT memorized the training distribution. And SFT remains essential: it stabilizes output format so RL can then generalize.

The agentic version of this is simple. Prompting tunes the behavior you engineered; RL changes the policy, and the policy is what shows up in a novel environment. This is also where the cracks in RLVR-style training show up (rlvr-and-its-cracks): sparse terminal rewards, high variance, sensitivity to the reward channel, which is exactly why step-level credit and environment design are the real bottleneck.

My position

I would use agentic RL where three conditions hold: a reward can be defined, rollouts are affordable, and the gap is in strategy rather than base knowledge. Everything else stays prompt-and-harness, because training on a bad reward is worse than not training at all. My shipped SafeRoutes is the prompt-and-harness case: the model produces judgments about the world, a deterministic gate enforces the invariants, and no training is involved.

There is also an economics gate, and it is a decision rule rather than a slogan: RL pays only if the trained behavior outlives the next base-model generation. If the behavior you bought with rollouts ships free in the next base model, the training spend was a rental. So train the behaviors you expect to survive at least one model generation, and prompt the ones the base models are visibly converging on.

This position is my current setup, not a plan. I implemented GRPO myself from the DeepSeekMath paper (grpo-family-tree), and VerifierForge trains with it against a programmatic verifier (verifier-engineering). The run carries a deliberately imperfect random-reward falsification reference: the verifier-rewarded run's training-pool monitor ended at 0.80 while the random-reward run's monitor ended at 0.40. That is a reference, not strict causal proof, since the control is a smaller model trained for fewer steps, but if the random arm had risen the same way, the verifier story would be dead.

The harness version of that position: an agentic RL harness with pluggable environments and reward functions, ordered by trust in the reward: verifiable first (tool success, execution outcomes, task completion), learned rewards second, iStar-style implicit step rewards for open-ended tasks. The harness details: masked observations for retrieved content, GRPO-style group advantages (grpo-family-tree) to skip the critic, and an evaluation protocol that tests held-out variants, not just the training distribution, because that is the difference between memorizing and generalizing.

The honest warning: RL agents will hack real tools to game rewards, rollouts cost real money, and long-horizon credit assignment is unsolved. None of the three is a reason to stay on the prompt side; each is a spec for the next harness.

Sources linked in this post were fetched and verified.