← logs/

Harness Design Notes: Review Gates

The least appreciated layer of an agent harness is the commit flow. Not the model, not the tools, not the memory: what does a change have to survive before it becomes real? My default is proposal-first. Nothing mutates shared state until a proposal has cleared a gate: draft, diff, human commit. Approving an action is reactive, frame by frame. Approving a diff is one decision about a whole mutation, made before anything lands.

This is the companion to my sandboxing notes. The sandbox decides whether a candidate plan is viable. The review gate decides whether it is wanted.

The gating spectrum

A gate is a place where a change can be stopped. The design space is a spectrum:

  1. No gate. The agent mutates on every tool call. Fastest path to value, largest blast radius. Fine for toys, expensive to learn from.
  2. Structural gate. Invariants, types, schemas, constraint checks. Microseconds to run, catches exactly what you can encode, never tired.
  3. Eval gate. Tests, replay, sandbox dry-runs, judge models. Catches behavior you cannot encode, costs more, and is only as good as its oracle.
  4. Human gate. Review and approval. The most general judge there is, and the slowest. It does not scale, which is why it must be reserved.

The craft is stacking them in cost order: structural, eval, human, with human time reserved for the high-blast-radius surface. The goal is not the most gates; it is the cheapest gate that catches the failure class you cannot afford. Two properties of gates are worth internalizing. First, every gate is a place where a failure can be caught before it becomes a mutation. Second, every gate can be bypassed under pressure, and a bypassed gate is worse than no gate, because you still trust it afterwards.

The failure taxonomy as a gate checklist: MAST

Why Do Multi-Agent LLM Systems Fail? (Cemri et al., arXiv 2503.13657, 2025) builds MAST, the Multi-Agent System Failure Taxonomy: system design issues, inter-agent misalignment, and task verification, distilled from a large annotated trace corpus. The corpus numbers and the full taxonomy get a close read in why multi-agent systems fail; here I want the gating read.

Category three, task verification, is literally the category of missing gates: inadequate output validation, missing quality checks, errors propagating because nobody re-checks intermediate results. And the paper's central conclusion, that base-model improvements will not cover the taxonomy, is the empirical case for harness work: the failures are structural, so the fixes are structural. A review gate is the smallest structural fix that catches all three categories at once — at commit time. It catches specification drift when the proposal does not match intent, misalignment when the diff touches the wrong thing, and verification failures when the diff fails the checks. But catching is the whole claim: the gate prevents none of these at design time, and a team that reads the gate as prevention will keep authoring the same specification bugs and keep paying review cost to stop them. One mechanism, three failure classes caught, none designed out.

The harness is the data pipeline

Both of these harness notes orbit one paper: The Interplay of Harness Design and Post-Training in LLM Agents (Kim, Choi, Lee, Jun, Kim, Park, arXiv 2606.25447, June 2026). It defines the harness as the scaffolding that decides which tools are exposed, how they are described, and what auxiliary information accompanies each observation, and observes that this scaffolding is treated as a fixed engineering detail while post-training assumes a static environment. The authors extend ALFWorld to make the harness a controllable dimension and evaluate under task and tool environment shifts. Their results: harness-aware post-training improves in-distribution performance and makes agents robust out-of-distribution, while minimal harness design effort leads to a drastic performance drop when tool environments shift.

I agree with the direction, and I want to sharpen the mechanism. The paper treats the harness as a context around post-training. My experience says the harness is the post-training data pipeline. The event log doubles as training-data trajectories: the gate determines which trajectories exist at all. A review gate that blocks bad mutations is also a filter on the training distribution, and every rejection reason is a supervision signal. Two consequences follow.

First, harness-aware post-training is not only about exposing better tools. It is about generating the right trajectories in the first place, and about labeling the wrong ones at the moment they are rejected, when the reason is still cheap to record.

Second, their out-of-distribution finding matches a failure mode I worry about in production. A cheap harness produces degenerate trajectories that look fine in-distribution and fall apart when the environment shifts, because nothing ever pushed back on the shortcuts. The gate is what keeps the trajectory distribution honest. That is my read of their result; the authors do not claim it, but it is the reason the review gate is not a safety feature. It is a data quality feature.

The human as a pre-execution gate: InferAct

InferAct (Fang et al., arXiv 2407.11843, 2024) makes the case for putting the human gate before the action instead of after it. InferAct uses belief reasoning grounded in Theory of Mind to detect actions that are misaligned with user intent before execution, and alerts the user for timely correction, preventing outcomes like an accidental buy-now in web shopping. The authors report up to 20% improvement in misaligned-action detection over baselines.

Two lessons. Pre-execution detection beats post-hoc repair: once an action executes in the real world, fixing it later may not be an option. And the human is part of the gate, but alert-per-action does not scale. Proposal-first review is the compression that makes human oversight viable: the agent proposes, and the human reviews one diff instead of a transcript of two hundred tool calls.

Gates need judges: Agent-as-a-Judge

Eval-driven iteration only works if the eval is affordable. Agent-as-a-Judge (Zhuge et al., arXiv 2410.10934, 2024) is the strongest version of that argument: an organic extension of LLM-as-a-Judge that uses agentic systems to evaluate agentic systems, with agentic features enabling intermediate feedback over the whole task-solving process rather than a final grade. On their DevAI benchmark, 55 realistic AI development tasks with 365 hierarchical requirements, the judge is dramatically more reliable than LLM-as-a-Judge and comparable to a human baseline, which the authors position as the reward signal needed for dynamic and scalable self-improvement.

The lesson for gates: a gate you cannot afford to run is a gate that does not exist. Judge agents make per-proposal evaluation cheap enough to run on every proposal, which is the precondition for eval-driven iteration.

The proposal pipeline

The shape I keep building:

draft    agent proposes a mutation (plan node, config, code)
diff     machine-readable delta against current state
gate 1   structural: invariants, types, constraints
gate 2   eval: dry-run in the sandbox + judge verdict
gate 3   human: approve or reject, with one-click rollback
commit   append-only event-log entry (doubles as training data)

Design rules:

  1. Every mutation is a proposal. No side channels, no background writes. If a change does not exist as a diff, it cannot be reviewed, and if it cannot be reviewed, it should not happen.
  2. Gates in cost order. Structural on every proposal, eval on every proposal that passes, human only when the blast radius is high or the eval's confidence is low. Where to draw the human line is a design call, and I want to frame it as one rather than as a result: I have no production misroute data showing the threshold is calibrated. The failure modes of a wrong threshold run in both directions — too loose and bad mutations land unreviewed, too tight and the reviewer becomes a rubber stamp, which is a bypassed gate with extra steps — so the honest move is to start conservative and adjust on what the event log shows; asserting a threshold up front would be guessing.
  3. The human reviews a diff, not a transcript. People are good at judging "does this match intent" and bad at reading logs. A typed plan graph makes this honest: the diff of a plan is a real object.
  4. The event log records every verdict and its reason. That log is your failure taxonomy in the making. MAST gives the starting vocabulary, spec, misalignment, verification; the taxonomy you harvest from your own rejections beats the taxonomy you guess up front.
  5. Re-planning is a proposal too. Reality changes, the critical path moves, and the new plan goes through the same gates as the first one. The gate is not a one-time ceremony; it is the steady state.
  6. Negative verdicts are first-class. A rejected proposal is a labeled example and a free supervision signal; nothing about the turn is wasted.

Where this came from: PiPlan.ai and SafeRoutes

Both rules above come from systems I have built.

At PiPlan.ai, goals become typed graphs, and the harness governs every change: proposal-first review, with each mutation arriving as a diff that a person signs; a simulation sandbox for candidate plans; and a full event log that doubles as training-data trajectories. Adaptive re-planning re-derives the critical path as reality changes, so the gate fires on every revision. The conversational planner-executor agent is in progress; the harness that would govern it is the pattern this post describes.

SafeRoutes is the gate taken to its logical extreme: correctness as a property of the data plane, not the prompt. Every state write is gated by an invariant check: waypoint self-consistency, path continuity, range windows. A failed check rolls the state back and returns an error written as an instruction for how to proceed. Waypoint mutation is a first-class operation: the agent can propose a waypoint change, and the gate decides whether it lands. The model can say whatever it wants; the data plane disagrees at its own speed. I wrote the loop notes in training a Safe-Routes skill: notes from the loop.

Rule 4's failure-taxonomy-in-the-making is not hypothetical there; the rejections are the artifact. Every coverage-gap error ships a caller_guidance string built to reach the model through the tool's own return value: "Coverage gap: do NOT substitute web-searched or guessed gas stations (unverified = fabricated fuel safety), and do NOT lower or inflate --safe-range to force it. Load corridor data (run doctor, then ingest), or have the user name a real stop the skill can verify." Invariant violations come back as structured code, detail, and recovery: a truncated route returns INV-7, "committed route does not reach the destination", with the recovery "re-plan; a truncated route must not be persisted", and an over-range leg returns the recovery "load corridor fuel data (doctor → ingest) or have the user name a verified stop; never lower/raise --safe-range to force it". Each rejection is already a labeled example: which invariant, at what point in the trip, and what to do instead.

The review gate is how you turn an agent from something that talks into something you let write: the model drafts, the gates filter, and a person signs the commit.

Sources linked in this post were fetched and verified.