The optimizer revival: Muon and friends

For a decade, "which optimizer?" was the one pretraining question nobody asked twice. AdamW worked, everyone used it, the topic closed. Then Keller Jordan's Muon broke the nanoGPT and CIFAR-10 speedrun records, Moonshot scaled it to a 1-trillion-parameter production model, and the optimizer became a training-strategy decision again.

Why it matters: at a fixed data budget and a fixed compute budget, the optimizer is the lever on how much capability you get per FLOP. Token efficiency is model quality now. The Muon line of work deserves a close read, not a headline.

The field map: from per-element adaptivity to update geometry

The classic frame treats optimization as a per-weight problem. AdamW keeps two momentum buffers per parameter and adapts the step size element by element. It is robust, it is memory-heavy, and for years it was good enough that nobody cared about the constant factor.

The new frame treats the update as a matrix with geometry. Muon (MomentUm Orthogonalized by Newton-Schulz) applies SGD-momentum to a 2D weight matrix, then runs a Newton-Schulz iteration that maps the update to the nearest semi-orthogonal matrix, the UV^T of its SVD. The original writeup's motivation is empirical: transformer 2D updates have very high condition number and are almost low-rank, and orthogonalization rescales the rare directions that matter for learning.

AdamW  : two buffers (m, v), per-element adaptive steps
Muon   : one momentum buffer, then orthogonalize the update matrix
         W -= lr * NewtonSchulz(M)      (2D params only)

Everything else follows: one momentum buffer instead of two, Newton-Schulz stable in bfloat16, and the auxiliary rule that scalars, vectors, embeddings, and the output head stay on AdamW.

The memory story is real economics. Adam's two buffers cost 8 bytes per parameter, Muon's single momentum buffer costs 4, and roughly 90% of transformer parameters are 2D hidden weights, so optimizer-state memory drops by about 45% (end-to-end savings are smaller; the PyTorch blog measured 9% peak-memory savings on a 3B fine-tune). By mid-2026, Muon support was shipping in DeepSpeed, which says something about how far it has escaped the speedrunning corner (Using Muon Optimizer with DeepSpeed).

Close read: the original writeup

Muon: An optimizer for hidden layers in neural networks (Keller Jordan et al., December 2024) is the primary source. Its headline results, all speedrun-style with reproducible logs: CIFAR-10 at 94% accuracy in 2.6 A100-seconds, down from 3.3; a 1.35x improvement on the nanoGPT speedrun target of 3.28 validation loss on FineWeb; scaling to 774M and 1.5B parameters; and a 1.5B transformer reaching GPT-2 XL-level HellaSwag in 10 8xH100-hours where AdamW needed 13.3.

Two things make the writeup worth reading as literature. It connects Muon to the Shampoo lineage through Bernstein and Newhouse's analysis, so the trick is framed as update-geometry research rather than a hack. And it is explicit about standards of evidence: the case is compute-matched, wall-clock, reproducible, and it says plainly that non-matrix parameters should stay on a standard optimizer. That split is the recipe, not an implementation detail.

Close read: Moonlight, or Muon at 3B/16B

Muon is Scalable for LLM Training (Moonshot AI, February 2025) is the paper that moved Muon from speedrun curiosity to pretraining recipe. Two techniques made it scale: adding weight decay, and carefully adjusting the per-parameter update scale. With those, Muon achieves roughly 2x computational efficiency over AdamW under compute-optimal training. The team trained Moonlight, a 3B/16B Mixture-of-Experts model, on 5.7T tokens, and open-sourced a memory-optimal, communication-efficient distributed Muon implementation.

The interesting part is what had to be fixed. Weight decay and update RMS are exactly the details a speedrun can ignore and a 5.7T-token run cannot. The paper named the scaling failure modes, which is what made the next step possible.

Close read: Kimi K2, or Muon at 1T

The Kimi K2 technical report (Kimi Team, July 2025) is the production argument. Moonshot proposed MuonClip: Muon with weight decay, consistent RMS matching, and QK-Clip. The problem QK-Clip solves: at scale, attention logits can explode past 1000 and training diverges. The fix is surgical: per attention head, if the max logit exceeds a threshold, rescale that head's query and key projection weights.

The result: a 32B-active, 1T-total-parameter MoE pretrained on 15.5T tokens with zero loss spikes, state of the art among open-source non-thinking models at release, with its strongest scores in agentic and software-engineering tasks.

My read: this is the cascade that matters. An optimizer change forced an attention-adjacent stability fix, which is why "which optimizer" is now a training-recipe decision with architecture consequences, not a hyperparameter.

My position

I am on the side of the revival, with guardrails.

Compare optimizers under compute-matched budgets, not step-matched ones. A harness that reports FLOPs and wall-clock to a fixed target is the right instrument; a fixed-step loss curve will mislead you.

Adopt the split recipe as the default: Muon on 2D hidden matrices, AdamW on embeddings, heads, norms, and everything 1D. That is what every serious Muon result does, from speedruns to K2.

Monitor two diagnostics, and be honest about their provenance. Max attention logit is proven telemetry: K2 watched it climb past 1000 until QK-Clip was needed. The condition number of the update is different. It is the motivation in the original Muon writeup, but nobody publishes it as training telemetry, which is exactly why my harness would log it rather than borrow it as field practice: if orthogonalization pays when the update is ill-conditioned, that number should say when Muon is earning its keep before the loss curve does.

Where I would hold back: fine-tuning and RL-stage runs. The strong evidence is pretraining-centric. The PyTorch blog's fine-tune of Moonlight-16B-A3B shows Muon edging AdamW on three of four benchmarks (MBPP+, MMLU, GSM8K) while trailing slightly on MBPP base, so the case is plausible but thin. I would measure before switching, not switch on faith. My own VerifierForge runs are 400 GRPO steps on a 1.5B model; at that horizon optimizer choice is second-order to verifier quality, so I have not switched. What would change my mind is a FLOP-matched fine-tune loss curve at my scale, not another pretraining result at someone else's.

The concrete build is a small optimizer harness that logs those two diagnostics across Muon and AdamW runs, FLOP-matched, with the Muon-plus-AdamW split as the starting configuration, so the Muon-versus-AdamW call at my scale stops being folklore. The Muon story is a reminder that the boring parts of the training stack are where the 2x lives, and the broader recipe map is where I keep score on the rest (foundation-models-field-map).

Sources linked in this post were fetched and verified.