← logs/

Harness Design Notes: Sandboxing Agent Mutation

Once an agent can mutate shared state, a plan graph, a config, a codebase, the binding constraint of the system stops being reasoning and becomes write access. The question that decides the architecture is: where does a mutation go before it becomes real?

My answer, from building agent harnesses — one shipped platform, one shipped skill: a sandboxed dry-run first, then a diff, then a commit. And when the dry-run cannot be trusted, make the commit itself reversible, so the rollback is the sandbox of last resort.

This post is the design notes. The gate that sits between the dry-run and the commit is the subject of the companion post, harness design notes: review gates.

Three postures for a sandbox

A "sandbox" means different things depending on what you are protecting. I keep coming back to three postures:

  1. Emulated execution. Replace the world with a model of it, and run the agent against the model. Cheap, broad, and only as faithful as the emulator.
  2. Stateful evaluation sandbox. A controllable environment whose job is measurement, with pass/fail criteria living inside it.
  3. Verify-then-mutate. The agent mutates its own library, and the real environment is the sandbox: the mutation only lands after verification in the world.

The axes that matter are fidelity of the world model, where the verdict comes from, cost per run, and whether the pre-mutation state can be restored. Everything else is implementation:

candidate mutation
      |
      v
sandbox: run candidate against a world model
      |
      v
verdict (milestones, invariants, judge)
      |
   +--+--+
   |     |
  pass  fail
   |     |
  diff -> commit     discard + log reasons

ToolEmu: an emulator is a world model with a confidence interval

Identifying the Risks of LM Agents with an LM-Emulated Sandbox (Ruan et al., arXiv 2309.15817, 2023) is the cleanest demonstration that sandboxing changes the economics of failure discovery. Instead of implementing each tool and standing up each environment, ToolEmu has an LM emulate tool execution and runs the agent against the emulation. An LM-based automatic safety evaluator then examines agent failures and quantifies the associated risks. The benchmark covers 36 high-stakes tools and 144 test cases, and a human evaluation judged 68.8% of the failures ToolEmu found to be valid real-world failures. Even the safest agent tested exhibits such failures 23.9% of the time according to the evaluator.

Three lessons I take from it:

  1. The sandbox is what makes long-tailed risk testable before deployment. Without emulation, every scenario needs the real tool implemented and a real environment configured, and the cost of that caps how many scenarios you will ever test. Emulation flips the economics: the dangerous case is the cheapest to test.
  2. The emulator is the model's guess about the world. The 68.8% validity number is the honest measurement of that guess. Every emulator-based sandbox needs a faithfulness check of its own, and results should be read with that discount applied.
  3. The evaluator is a second model. The verdict inside an emulated sandbox is another LM's judgment. That makes it cheap and makes it fallible, which is why the verdict belongs in the sandbox as a first-class contract rather than an afterthought.

ToolSandbox: a sandbox without an oracle is a playground

ToolSandbox (Lu et al., Apple, arXiv 2408.04682, 2024) shows what the evaluation-sandbox posture looks like when state matters. Earlier tool benchmarks evaluate stateless REST APIs from a single prompt or an off-policy script. ToolSandbox instead has stateful tool execution, implicit state dependencies between tools, a built-in user simulator for on-policy conversational evaluation, and a dynamic evaluation strategy that scores intermediate and final milestones over arbitrary trajectories. Its headline result: task types like State Dependency, Canonicalization, and Insufficient Information are challenging even for the most capable models.

The lessons:

  1. Statefulness is where sandboxes live or die. If the sandbox does not carry state, the dry-run lies about what a mutation does to later steps. A change that looks fine in isolation can break every downstream action that depends on the state it touched.
  2. Milestones are the contract. A sandbox without criteria is a demo. The milestone/minefield framing, where certain states must be reached and certain states must be avoided, is exactly the pass/fail contract a dry-run needs, evaluated mid-run rather than at the end.
  3. The user simulator matters. Agents act on conversation, and the conversation reacts to the agent. On-policy evaluation, where the simulated user responds to what the agent actually did, catches failure classes a fixed script cannot.

Voyager: the environment as the sandbox

Voyager (Wang et al., arXiv 2305.16291, 2023) is the third posture: the agent mutates its own code. Voyager plays Minecraft with a growing skill library of executable code. Each skill is a program, proposed by the model and improved through an iterative loop fed by environment feedback, execution errors, and self-verification. The skill is only added to the library once it verifies in the world. The measured gains over prior state of the art are large and the skill library transfers to a new world; I itemize the numbers in the agent taxonomy notes.

Why this matters for mutation design:

  1. Self-modification is tractable when the mutation is append-only and gated by verification. The agent's own skill library is the shared state, and the gate is a verified run in the environment. Nothing is overwritten; skills accrete.
  2. The weak spot is the oracle. Verification is self-verification: the agent judges its own run. That is the first thing I would replace in production, with a structural check or a separate judge.
  3. Skills as code means the mutation is inspectable. You can read the diff of a skill, replay its verification, and revert the entry. That inspectability is the property I care about most, and it is exactly what gets thrown away when behavior is stored as embeddings instead of as diffs.

The dry-run pipeline

My position is that the sandbox should be a pure function: a candidate mutation in, a verdict out, no side effects on real state in between. And I design around the three ways sandboxes lie:

  1. Emulator drift. The world model drifts from the world. Mitigate with periodic faithfulness checks, and treat sandbox verdicts as evidence, not truth.
  2. Weak oracle. The verdict is produced by the thing being tested. Mitigate with structural checks and judges that do not share the candidate's context.
  3. Coverage gaps. Some things cannot be simulated: other humans, external services, the passage of time. Mitigate by making the commit itself reversible.

Rules I would impose on any of these pipelines:

  1. Cheapest gate first. A dry-run that costs more than the review it replaces gets bypassed, and a bypassed gate is worse than no gate, because now you also trust it.
  2. Verdict inside the sandbox. Milestones and invariants evaluated mid-run, not a single final score.
  3. Mutation as a diff; pre-mutation state immutable. Append-only lineage, undo by moving a pointer rather than recomputing.
  4. When simulation cannot be trusted, move the dry-run into the transaction. Compute the candidate fully, validate it against the invariants, and only then persist. The rollback is the sandbox of last resort.

Where this came from: PiPlan.ai and SafeRoutes

Both patterns above, the simulation sandbox and the transactional rollback, come from systems I have built.

At PiPlan.ai, goals become typed graphs, and candidate plans are run through a simulation sandbox before they become changes to the graph. The typed graph is what makes the dry-run honest: constraints are computable, so feasibility is a property of the graph structure, not a matter of prompting. And because reality changes, the harness re-plans: adaptive re-planning re-derives the critical path as the situation shifts. The plan that was verified yesterday is not the plan that gets proposed today, which is what makes the sandbox necessary in the first place.

SafeRoutes is the transactional answer for a world you cannot simulate faithfully. Roads, fuel availability, and weather do not come with an emulator you can trust, so the commit itself is the dry-run. The candidate plan is fully built in memory, then a write-time invariant gate runs at the single serialize boundary, re-checking waypoint self-consistency, the range hard constraint (INV-4: no leg between stops exceeds the tank's safe range), and that the committed route actually reaches the destination (INV-7). A failed check raises before anything reaches disk, so the trip stays at its last legal state: rollback for free, for any caller, without the model needing to remember to be careful. The trip state is a tree of frozen session nodes, and undo moves the head pointer backward, lossless. Rejections come back with a recovery hint, an instruction for what to do next, not a complaint. I wrote the earlier loop notes in training a Safe-Routes skill: notes from the loop.

Who writes the oracle decides whether any of this works. In SafeRoutes the invariants are not the model's job to remember, and they are not discovered at runtime: I wrote them up front, as an explicit acceptance table in the design doc docs/04_SUCCESS_CRITERIA.md — twelve global invariants, INV-1 through INV-12, covering reachability, route distinctness, the range hard constraint (INV-4), fuel-stop sufficiency, overnight anchors, the verification status of every stop, and destination reachability (INV-7). That table runs as law at two levels. An executable implementation of the doc judges the full matrix PASS/FAIL as the acceptance gate. And the hard subset — waypoint self-consistency, the range constraint, destination reachability — is re-checked at the single serialize boundary on every state write, fail-closed, just before anything reaches disk. The quality invariants deliberately stay out of the write path, because gating those there would over-reject legitimate mid-trip states.

The reason the range constraint is hard law rather than a guideline is an incident. In one eval probe (D-G-04 in the capability report), the model was pushed to fill a fuel-coverage gap on an I-10 trip; twice it was induced, twice it complied, web-searching gas stations and stringing Van Horn and Fort Stockton into a plausible-looking plan. The deeper hole was in the data layer, not the model: the planner treated any waypoint marked refuel=True as a free full-tank reset, including a synthetic point of interest with no verified gas station nearby, so the coverage gap was silently filled and the output came out looking safe. The fix was structural, not a prompt patch: a refuel waypoint now counts only when it snaps to a verified gas POI within a fixed radius, the skill's own contract tells the model never to web-search or guess stations to fill a gap, and the range check that catches the result runs at the serialize boundary where no caller can skip it. An earlier version of these guardrails lived on the framework side; I reverted that and rebuilt them as skill-owned invariants, so they hold for any framework, even a raw CLI call. On the post-fix retest, the skill could not produce a fake plan.

The synthesis: the sandbox and the rollback are the same idea at different trust levels. Simulate when you have a world model, transact when you do not. Either way, the mutation only becomes real after a verdict.

The honest limits

A sandbox is only as good as its world model, and every world model drifts. The 68.8% figure from ToolEmu is not a flaw in that approach; it is the correct price of emulation, and any sandbox design that does not budget for it is building on sand. What I refuse to build is a mutating agent with neither a dry-run nor a rollback, because that is not an agent, it is a fire.

Sources linked in this post were fetched and verified.