2026-08-31
To Forge or Not to Forge: a pre-registered study of ten-example pilots
Every fine-tuning project starts with a decision nobody measures: is this task worth fine-tuning at all? The usual answers are intuition, or paying for the full run to find out. I wanted the upstream decision itself on the table, so I ran the experiment: under one frozen protocol — Qwen2.5-1.5B-Instruct, a fixed LoRA recipe and budgets, greedy decoding, execution-based verifiers — pair a ten-example pilot run with a full fine-tune on 61 tasks, and take the measured gain as ground truth.
The result is a technical report, embedded in full at the bottom of this page. This note is the short version, plus the process story that didn't fit in the paper's margins.
The question, pre-registered
The hypothesis had two endpoints, committed before the remaining GPU slots ran: a binary go/no-go label (does full fine-tuning help at all?) and a secondary ranking endpoint (does the pilot's gain rank the full gain?). The pre-registration carried an honesty clause: if the bootstrap CI of the go/no-go AUC covers 0.5, that negative gets published, not negotiated away.
That clause turned out to be the most important line in the study.
What came back
Forging is the default in this regime. Fine-tuning helps on 53 of 61 tasks (87%). But 23 tasks start at pass@1 = 0, and most of those gains are format unlock — the training teaches the answer format the verifier can parse, not new reasoning.
The pilot ranks, and the ranking has budget value. Spearman 0.755 (CI [0.60, 0.86]) between pilot gain and full gain. The operational framing: if you can only afford K full fine-tunes, spending them in pilot order at K = 20 captures 0.72 of all positive-gain mass, against 0.33 for random ordering and 0.74 for an oracle — within three points of perfect.
The binary gate is a registered negative. The pre-registered classifier's leave-one-task-out AUC was 0.755 with a CI touching 0.50 — a negative under the honesty clause, printed as one. A declared revision then located the mechanism: the registered model was over-parameterised (12 features against 8 negative tasks), not signal-free. A parsimonious three-feature pre-decision model — invariant to both a leaked feature and a declared bug fix — is significant at 0.767 (CI [0.65, 0.88], permutation p = 0.004).
Binary framing was the wrong target anyway. With no-go tasks this scarce, a perfect gate is worth 2.4% of positive-gain mass over just always fine-tuning. The gate question mostly doesn't matter; the ranking question does.
You still have to run the probe. Instruction embeddings predict the label at chance (AUC 0.512). No amount of reading the task description substitutes for ten examples through the actual training loop — which costs a median 0.17 of the full training-plus-eval run.
The process story
Every measurement in the report was produced by an autonomous discover → pilot → decide → train → validate system: an LLM coding agent packed task bundles, drove a serial GPU queue, diagnosed failures, and executed the sealed analysis, with a human operator holding a small set of STOP points. Without that harness the 61-task ground-truth table would not exist — running 61 paired pilot-plus-full fine-tunes by hand was never going to happen.
Three design choices did the real work:
- Fail closed. A checker that hit a missing hash key aborted the task rather than packing a substitute; a CUDA OOM produced no label rather than a guessed one. A rail that invents a number to keep a queue green is not a rail.
- The contract forbids the tempting moves. After the seal, adding a feature, dropping the worst task, or moving the go threshold were exactly the moves the pre-registration ruled out. The negative stayed primary because it contractually had to.
- Split the diagnosis. The agent owns hash checks, resume logic, and writing PARTIAL instead of a guessed delta; the human owns mechanism rulings and exclusions — and exclusions are only legal before any label exists.
One declared bug fix deserves its own sentence, because it is the kind of thing that usually gets buried: a format-compliance feature silently returned 1.0 for two tasks, the fix landed after the revision contract was written, and it moves one headline AUC by +0.18 on its own. The report prints every affected number under both definitions rather than folding the jump into a cleaner story.
The paper
The full report — protocol, the 61-task table, the decomposed AUC ledger, and the broader-impact section on fail-closed harnesses — is here:
This is independent research, done as a self-contained study: the pilot signal is real but it is a ranker, not a stoplight, and the honest framing of that result matters more to me than a rounder number would have.