Architectures 2026: What Actually Changed Since the Vanilla Transformer

If your mental model of a language model is stacked transformer blocks, softmax attention, and a dense FFN, you are describing 2019. The open-weight frontier in mid-2026 is a composition of four modular changes: KV-compressed attention (MLA), trainable sparse attention (NSA), hybrid linear attention (Gated DeltaNet in Qwen3.6), and expert-routed FFNs with learned load balancing (DeepSeekMoE). They ship in models you can download today, and they compose.

2023 block:  [softmax attention] -> [dense FFN]          every layer the same
2026 stack:  [MLA / NSA / Gated DeltaNet] -> [MoE FFN]   per-block mix, routed

This is a close read of the three sources that define the current state: the DeepSeek-V3 technical report, DeepSeek's Native Sparse Attention paper, and Qwen3.6's official model card. Part of my MODELS field-map series.

MLA: compress the cache, not the heads

The DeepSeek-V3 technical report is the anchor: 671B total parameters, 37B activated per token, Multi-head Latent Attention (MLA), DeepSeekMoE. MLA is the KV cache trick that makes a model this size servable. Keys and values are jointly compressed into a low-rank latent vector, so the cache stores one small latent per token instead of per-head keys and values; the MHA2MLA line of work on porting MLA into existing models describes it exactly that way: significantly compressing the KV cache into a latent vector.

The contrast with grouped-query attention is instructive. GQA shrinks the cache by making heads share keys and values. MLA shrinks it by compressing what is stored. Same goal, different mechanism. One precision worth keeping: MLA is a constant-factor reduction in per-token cache footprint, not a different scaling law. The absolute savings grow linearly with context length, the same scaling GQA gets but with a better constant, and context length is where agent workloads actually live.

Two more numbers from the report are worth calibrating against: 14.8 trillion training tokens, and 2.788M H800 GPU hours for the full run with no rollbacks. And two design choices became 2026 defaults: an auxiliary-loss-free strategy for load balancing, so routing learns to balance without a loss term that distorts it, and a multi-token prediction training objective.

NSA: sparse attention you can train end to end

Sparse attention earned a bad reputation when the 2023 generation bolted sparsity on after training and measured speedups in one phase only. Native Sparse Attention (NSA) (Yuan, Gao, et al.) is the rebuttal: a dynamic hierarchical sparse strategy that combines coarse-grained token compression with fine-grained token selection, implemented with hardware-aligned kernels and balanced arithmetic intensity, and, critically, trained end to end. Models pre-trained with NSA maintain or exceed full-attention baselines across general benchmarks, long-context tasks, and instruction-based reasoning, while delivering substantial speedups over full attention at 64k sequence length across decoding, forward, and backward passes.

What changed: sparse attention stopped being an inference-time trick and became a trainable inductive bias. The model learns what to compress and what to select, and the speedup is real on all three phases of the model lifecycle.

Operationally this means the architecture decision happens at pre-training time. NSA is not something you retrofit onto a dense checkpoint; the compression and selection heads are trained, not tuned, and every downstream phase either pays for the choice or gets paid by it.

Qwen3.6: hybrid linear attention in production

Qwen3.6-35B-A3B shipped in April 2026 (model card; QwenLM/Qwen3.6 repository), and it is the latest production hybrid-linear-attention model from the Qwen line.

The card reads like a road map of 2026 architecture. 35B total parameters, 3B activated. 40 layers arranged as ten blocks of three Gated DeltaNet layers followed by one Gated Attention layer, each feeding an MoE FFN with 256 experts, 8 routed plus 1 shared active. 262,144 tokens of native context, extensible toward 1,010,000. MTP, the multi-token prediction head, trained with multi-steps and used as the speculative-drafting path in vLLM and SGLang. SWE-bench Verified at 73.4 on Qwen's internal agent scaffold.

One detail an architecture-focused read can miss: the model is multimodal. The card lists it as a causal language model with a vision encoder and reports a full vision-language benchmark suite. The hybrid attention layout described above is a property of the language model side.

The linear layers are Gated DeltaNet (Yang et al.), which pairs two mechanisms the paper shows to be complementary: gating for rapid memory erasure and the delta rule for targeted memory updates, trained with a parallel algorithm that outperforms Mamba2 and DeltaNet baselines. In serving terms, those layers keep a compact recurrent state instead of a growing KV cache; the periodic full-attention layer provides precise retrieval across the whole context. The 3:1 mix is the architectural lesson: linear layers carry long-context state cheaply, full attention handles the retrieval that pure recurrent state fumbles, and you want both.

MoE routing: the FFN became a router

The least glamorous change is maybe the most important: the dense FFN became an expert router. DeepSeekMoE's 671B/37B split is the canonical number: 37B activated per token because routing decides which experts compute each token. Pair that with the auxiliary-loss-free load balancing from the same report, and you get the 2026 equation: compress what you cache (MLA), activate only what you compute (MoE), attend selectively (NSA), and replace some attention with recurrence (Gated DeltaNet). Each change attacks one cost, but the honest caveat is that MLA, NSA, and linear attention all attack the same KV bottleneck, so they partly substitute for each other rather than stack cleanly. The right mix is workload-dependent, not universal.

My position

What actually changed is that the block stopped being monolithic. Attention is now a menu: compressed (MLA), sparse (NSA), recurrent (Gated DeltaNet), or plain full attention, chosen per layer. The FFN is a router. The engineering question is no longer which single architecture wins, but how a mix behaves on your workload.

For my own serving work the priorities are clear. KV cache cost dominates agent workloads: long tool outputs, long reasoning traces, huge contexts. MLA-style compression and Gated DeltaNet-style linear layers are the practical levers, and the serving-side consequences are in my inference-systems-for-agents notes. NSA is the interesting option for retrieval-heavy long-context jobs. The operational risk is the kernel surface: hybrids only pay off where flash linear attention kernels and serving engines have caught up. A linear-attention layer without a fused implementation runs slower than the full attention it replaced, so the architecture bet is really a kernel bet. That support is real for Qwen3.6 in vLLM and SGLang today, but it is young.

If I were standing up a serving stack right now: DeepSeek-V3-style MLA plus MoE for the workhorse, hybrid linear layers for long-context cost, and sparse attention as the first lever to pull when the context budget outgrows what the KV cache can carry.

Sources linked in this post were fetched and verified.