Inference systems for AGENT workloads: why agent loops != chat serving

The default mental model for LLM serving is still chat: a user sends a prompt, the server streams a reply, the connection closes. Chat serving systems are optimized for that shape. Token throughput, time to first token, time per output token, all measured in the steady state of many concurrent conversations.

An agent run does not look like that at all. One task produces dozens of generations: a tool call, a JSON blob, a short analysis, then one long planning pass over the whole trajectory. Every generation shares a prefix with the ones before it. Some steps are tiny. Some carry a hundred thousand tokens of accumulated context. The system that serves a chat app well can serve an agent badly, because the optimization target is the wrong one.

This post is a field map of inference systems, read through the lens of agent workloads. I close-read three papers that shaped how I think about this, add three more from the wider serving literature, and end with the design I would build, mapped onto the serving layer of PiPlan.ai, the product I work on.

What an agent loop looks like at the request level

An agent loop is never one generation; it is a sequence of them. The shape:

turn 1:  system prompt + task         -> model -> plan (tool calls)
turn 2:  prefix + tool results        -> model -> next action
turn 3:  prefix + more tool results   -> model -> next action
...
turn N:  full trajectory              -> model -> final answer

Three properties separate this from chat serving.

1. Prefix reuse is the dominant cost driver. The system prompt, the task, and the accumulating trajectory are re-read by every turn. A run of twenty steps reads the same growing prefix twenty times. If the KV cache is not reused across turns, you pay the prefill cost for the entire history on every single step, and the run's cost grows quadratically in the number of steps.

2. Work arrives in bursts, not streams. Tool execution pauses generation for seconds or minutes, then a large blob lands and must be processed promptly. The server alternates between idle and saturated, and nothing about that rhythm is captured by a steady-state throughput number. The final planning turn arrives with a much longer context than anything the mid-loop steps produced, which makes it a qualitatively different request.

3. The tail is long and latency-bound. Most steps are short: a function call, a classifier output, a one-line summary. Per-step latency is what the user feels; aggregate throughput is invisible to them. A forty-step loop at 100ms per step is done in seconds; the same loop at two seconds per step feels broken, and throughput can look great the whole time.

Chat serving optimizes the steady state. Agent serving has to optimize the sequence: the cache hit rate across turns, the p95 of a short step, and time-to-done for the whole run.

DistServe: split prefill from decode

DistServe (Zhong et al., 2024) argues that prefill and decode should not share a GPU. Prefill is compute-bound and bursty. Decode is memory-bound and steady. Colocated, they interfere: a long prefill delays every decoder in the batch, and with separate latency requirements for time to first token and time per output token, the system ends up sacrificing one or over-provisioning to keep both.

DistServe assigns the two phases to different GPUs and co-optimizes resource allocation and parallelism for each phase separately. The evaluation reports up to 7.4x more requests served, or 12.6x tighter SLOs, within latency constraints for over 90% of requests, compared to state-of-the-art systems.

Why this matters for agents: a single agent run produces both extremes of the phase spectrum. The tool-call turns are decode-heavy and latency-sensitive. The planning turn is prefill-heavy, with a context that has grown over the whole run. In a colocated server, the planner and the executor fight each other, and the fight gets worse as runs get longer. Disaggregation lets you right-size each side: many small GPUs for the execution steps, dedicated prefill capacity for the planning turn.

The cost is communication. The KV cache must cross the network between the prefill and decode workers, and DistServe places the phases with cluster bandwidth in mind. That is the honest caveat: disaggregation only pays off when the interconnect can keep up with the cache traffic, and the longer the context, the bigger the transfer.

Mooncake: the KV cache as the scheduler's currency

Mooncake (Qin et al., 2024) is the serving platform behind Moonshot AI's Kimi, and it pushes disaggregation further. Prefill and decode clusters are separated, and on top of that the KV cache is treated as a first-class storage tier: underutilized CPU, DRAM, and SSD across the GPU cluster hold cache that would otherwise be dropped.

The core is a KVCache-centric scheduler that balances effective throughput against latency SLOs, plus a prediction-based early rejection policy for overload. The paper reports up to a 525% increase in throughput in certain simulated long-context scenarios while staying within SLOs, and a 75% increase in requests handled on the real Kimi workload.

Two ideas transfer directly to agent serving. First, the cache is the asset, not the GPU. In a multi-turn loop the shared prefix is worth more than any single generation, so the system should be organized around keeping it warm, including on cheaper storage tiers when a tool call is running and the GPU would otherwise be idle. Mooncake's tiering is the right instinct: cache has a home outside the GPU, and the scheduler knows where.

Second, admission control is legitimate. Mooncake rejects requests early rather than degrading everyone. An agent system at capacity should fail fast on a new run instead of stalling the steps of every agent already running. A delayed step in a running loop is worse than a rejected start, because the loop is blocked on it.

Mooncake also gets the priority order right. Most serving systems start from batching and bolt caching on afterward. Mooncake starts from the cache and schedules everything else around it. For agent workloads that inversion is not a detail, it is the architecture.

EAGLE: speculate on short, structured steps

EAGLE (Li et al., 2024) is a speculative decoding method. A small draft head predicts the next tokens at the feature level, the target model verifies the draft in parallel, and accepted tokens cost almost nothing. For LLaMA2-Chat 70B the paper reports a 2.7x to 3.5x latency speedup with roughly doubled throughput.

Speculative decoding is usually sold as a general latency win, but it is especially valuable in agent loops. Most agent steps are structured outputs: JSON, function arguments, short labels. Those are highly predictable token sequences, which is exactly when drafts get accepted. A 3x per-step speedup on a forty-step loop is not a nicer demo. It is the difference between a loop that feels instant and one that feels stuck, and what the user experiences is the whole run, every generation of it.

The second angle is that a draft head is cheap enough to keep warm. In a serving setup where mid-loop steps are short and frequent, the draft model's state can persist across turns. A cold draft head gives up most of the benefit on a three-token output, so warmth across turns has to be designed in from the start.

Speculative decoding does not touch the prefill problem. The long planning turn still has to process the whole trajectory. But it changes the per-step cost of the common case, and in agent workloads the common case dominates wall-clock time.

Prefix caching and KV compression: the wider toolkit

SGLang (Zheng et al., 2023) introduced RadixAttention, which stores KV cache in a radix tree so prefixes are shared across and within requests, and reports up to 6.4x higher throughput than state-of-the-art inference systems on tasks that include agent control. For agent loops this is the natural data structure: retries, parallel branches, and repeated tool-call formats all share prefixes, and a radix tree stores each prefix once. If I were designing an agent-serving system today, prefix reuse across turns would be a hard requirement, not an optimization. The trajectory is a tree; cache it like a tree.

KV compression attacks the memory side. H2O (Zhang et al., 2023, NeurIPS 2023) shows that a small set of "heavy hitter" tokens dominates attention, and an eviction policy that keeps those plus recent tokens can shrink the cache dramatically, reporting up to 29x throughput gains over some baselines on OPT models. KIVI (Liu et al., 2024, ICML 2024) quantizes the cache to 2 bits, per-channel for keys and per-token for values, and reports about 2.6x less peak memory and 2.35x to 3.47x throughput on real workloads, tuning-free.

Compression matters for agents because the cache grows monotonically with the run, and a long run on a fixed GPU budget becomes a memory problem before it becomes a compute problem. But there is a real interaction with prefix caching: evicting tokens to save memory can destroy the shared prefix the next turn was about to reuse, silently converting a cache hit into a full prefill. The lesson is that eviction and reuse must be coordinated by the same policy. Quantize first, evict only at turn boundaries, and tell the scheduler which prefixes died.

Three decisions for an agent serving stack

If I were building inference for agents from scratch today, three decisions carry the design, and I would want each tradeoff on the table rather than in a footnote.

First, prefix caching: yes, unconditionally. A radix-tree cache queried by the scheduler before anything else is the central data structure; SGLang proved the shape, and agent trajectories are trees, so retries and branches share prefixes by construction. The tradeoff is that the cache becomes a consistency problem. Eviction and reuse must be coordinated by one policy: quantize KV rather than evict mid-run, evict only at turn boundaries, and tell the scheduler which prefixes died. A cache that silently converts hits into full prefills is worse than none, because the scheduler plans around hits it will not get.

Second, disaggregated prefill and decode: not at my scale. DistServe and Mooncake are datacenter patterns; they pay when there are separate prefill and decode fleets to right-size and an interconnect that absorbs the KV traffic. On a single multi-GPU node, which is the scale PiPlan.ai and VerifierForge actually run at, vLLM's chunked prefill absorbs much of the same interference by slicing a long prefill into pieces that share steps with ongoing decodes, with no KV transfer across a network at all. Disaggregation starts to pay when prefill is long and bursty enough to justify a dedicated worker class and the cache traffic is cheaper than idle decode capacity. I have not measured a workload of mine that crosses that line, so the split I would build happens at the routing layer, escalating the long planning turn to frontier capacity; a GPU-fleet split can wait.

Third, speculative decoding on the execution path: plausible, untested. The theory is right; JSON blobs, tool calls, and short labels are predictable token sequences, and per-step latency is what a loop feels. But acceptance rate is workload-specific, and I have not measured draft acceptance on my own tool-call steps. Until I do, EAGLE-style drafting stays on the experiment list, not in the design.

Admission control survives from the longer version of this list: refuse a new run early rather than let it steal SLO from running loops, Mooncake-style. The unit of optimization is the run, not the request. Benchmarks for agent serving should report end-to-end run completion, cache hit rate across turns, and p95 per-step latency, because those are the numbers a loop actually feels.

PiPlan.ai: the two-tier serving layer

PiPlan.ai's serving layer is a self-hosted multi-GPU vLLM node with a dynamic routing layer: routine agent steps run locally, and complex planning escalates to frontier APIs. Read through the lens above, that is a product-level version of the phase split.

agent run
   |
   v
dynamic routing layer -----------------------+
   | routine steps                           | complex planning
   v                                         v
local multi-GPU vLLM node          frontier API, cold start
(cache stays warm across turns)    (no local prefix)

The local node is the execution path: short, frequent, latency-bound steps run on hardware we control, with no per-step network round trip to an external API. The frontier escalation is the planning path: the long-context, prefill-heavy turn goes to capacity that is not worth owning locally. The routing layer is the scheduler, and the interesting design work is in its policy; the GPUs are the substrate.

Two tradeoffs stand out. First, escalation breaks cache continuity. When a turn escalates, the local node loses the shared prefix, and the frontier API starts cold on a long context. That means routing decisions should be stable across a run: per-step flapping between local and frontier would fragment the trajectory and pay the cold-start cost repeatedly. The cost of a wrong routing call is measured in lost cache hits, so the policy should be conservative.

Second, the routing layer decides before generation starts, so it cannot react to observed load mid-step. Capacity planning has to happen ahead of time, which pushes the interesting questions into workload prediction: which runs will escalate, and when.

The same logic shows up in VerifierForge's serving, which scales to zero. Its two live wake cycles reached ready in 282.14 and 266.68 seconds (about 4.4 to 4.7 minutes) on an RTX 4000 Ada pod billed at $0.20 an hour, served 200 real requests split 111 default, 89 tuned, zero fallback, and were then reaped by the 30-minute idle reaper all the way to provider-inventory zero. That wake time is too slow for interactive first tokens, so scale to zero only works for workloads that tolerate a cold start and arrive in bursts short enough to fit inside the reclaim window. For an agent's mid-loop steps that would be the wrong trade. For a verifier that scores batches of completions, it is the right one. Zero-cost idle is the honest alternative to pretending you can serve cold starts locally at all times.

The through-line: serving for agents is not a bigger chat server. It is a system that treats the trajectory as the unit, keeps the cache warm across turns, splits the phases, and makes admission and routing decisions explicit.

Sources linked in this post were fetched and verified.