← logs/

Weights vs context: when knowledge belongs in the model

A personal agent has two places to keep what it learns. The first is the context window: retrieve text and paste it into the prompt, every time. The second is the weights: edit the model so the knowledge is part of what it knows, with no retrieval step at all. Most agent memory systems never make this choice deliberately. They default to context, call it memory, and move on.

I think that default is wrong for one specific class of knowledge: durable beliefs and preferences. This post maps the field, reads the two literatures that matter, and argues for a router that decides where each memory lives.

The field map: three places knowledge can live

The survey Memory in the Age of AI Agents (Hu et al., December 2025, updated January 2026) organizes agent memory along three axes: forms, functions, and dynamics. Under functions it separates factual, experiential, and working memory. Under forms it distinguishes three realizations:

  • token-level memory: raw text stored in the input,
  • parametric memory: knowledge learned into model weights,
  • latent memory: compressed vector representations.

The forms axis is the weights-versus-context axis. RAG and context engineering are token-level memory. Model editing is parametric memory. The survey's most useful move is a negative one: it carefully delineates agent memory from RAG and from context engineering, because systems writing uses those terms interchangeably and they solve different problems. RAG is a lookup layer over external storage. Context engineering is prompt assembly. Parametric memory is the only one of the three where the answer does not depend on what happens to be retrieved or how much room is left in the window. Latent memory, the third form, does not get a vote in the routing rule: a compressed vector is a representation the other two forms pass through, not a store you can write to, audit, or roll back on its own, so the live choice stays two-way.

That distinction is not academic. Token-level memory is cheap, reversible, and attributable: you can point at the retrieved chunk that produced an answer. Its weakness is that it is prompt-scoped. Retrieval ranking, chunk boundaries, and context budget all sit between the memory and the answer. Parametric memory has the opposite profile: once written it survives retrieval being off and takes no prompt space, but it is sticky, and a bad edit can contaminate unrelated queries. So the engineering question is not which one is better. It is who decides, and on what evidence.

Close read: the editing line, from ROME to HoReN

The parametric side of this choice used to be fragile. The classic line, ROME, MEMIT, and GRACE, established two families of editors: locate-and-edit methods that write into MLP weights, and parameter-preserving codebooks that route queries through an external key-value memory. Both families share a scaling problem: as edits accumulate, locality and generalization degrade, and most editors collapse before ten thousand sequential edits. I covered that arc in sequential knowledge editing. I have also reproduced GRACE hands-on, and the lesson that stuck is the one HoReN cashes in: once values are frozen at write time, the key decides everything, and how the key is matched is where paraphrase generalization lives or dies.

That matters here because a personal agent's memory is inherently sequential. Beliefs accumulate one at a time, forever. An editor that works for ten edits but drifts by the hundredth is not a memory system; it is a demo.

HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing (arXiv 2605.08143, May 2026) is the strongest published result on sequential-edit scaling out to 50K edits. HoReN wraps a single MLP layer with a codebook in which each entry is both a knowledge key and a Hopfield stored pattern. Retrieval happens by angular similarity on the unit hypersphere, which supplies locality, followed by damped Hopfield dynamics that pull paraphrases into the correct memory basin. On a controlled ZsRE stress test with LLaMA-3.1-8B, HoReN holds overall editing performance above 0.93 at the 50,000-edit mark, where prior editors collapse or degrade severely before reaching 10,000. It unifies ROME's key-value view with GRACE's codebook architecture, and normalization is the load-bearing piece: magnitude-sensitive routing is what breaks paraphrase generalization in older editors.

Why does this change the weights-versus-context decision? Because HoReN lowers the risk of the parametric path. Writing a durable belief into weights stops being a fragile operation that might corrupt unrelated knowledge; it becomes a bounded, attributable update with a measurable locality guarantee. That is the precondition for treating weight edits as a routine memory operation rather than a research artifact.

My position: build a router, not a better store

If the parametric path is now solid enough, the scarce resource is the routing decision. I would build the system around one rule:

user turn -> extract memory candidates -> classify

  belief -> consolidation -> HoReN weight edit
  fact   -> reversible RAG store
  noise  -> dropped

The router is the actual system. The two write paths do not cost the same, and that asymmetry is what prices the rule. A RAG write is a database insert: cheap, instant, reversible. A weight edit is a consolidation pass: GPU time training the next adapter against an 8B base model, then a hot-swap. Engram runs it as a background job, serving the previous adapter until the new edit module is ready. So "beliefs go to weights" really means "beliefs are the writes worth paying a consolidation pass for"; anything you could cheaply re-fetch or would rather be able to delete stays in the store.

Extraction has to classify a turn into these buckets with a confidence threshold, because misrouting is worse than dropping: a fact written into weights becomes sticky and hard to unwind, and a belief parked in RAG vanishes the first time retrieval is turned off or the chunk is evicted.

The second design problem is keying. Personal belief queries collapse into a handful of near-identical forms: "What do I believe?", "What is my preference?", "What should I choose?" If the edit is keyed on the query alone, one belief's memory is unreachable from a differently phrased question, or worse, the key itself encodes the answer. The fix is to derive extra keys from the subject, the relation, and the belief statement, so similar query forms still reach the right edited memory without leaking the answer into the lookup key.

Engram: the fact-versus-belief split in practice

This is exactly the shape of Engram, a continuous-learning memory system I built around the same router philosophy. Engram separates three kinds of memory: ordinary facts, documents, and schedules stay in a reversible RAG store; durable beliefs and preferences go through a consolidation pass and are written into model weights via HoReN as the editing backend; transient chat noise is dropped.

The demo is built around attribution rather than vibes. Teach a durable belief, consolidate it, then ask the probe with retrieval disabled. If retrieved == [] and the answer still recalls the taught belief, the knowledge is living in the edited module rather than in the prompt. The edit module can also be toggled off and on: off, the belief disappears; on, it comes back. That toggle is the cleanest demonstration I know of the difference between prompt stuffing and an actual weight-level memory.

Engram also implements the keying strategy above. It adds extra canonical keys derived from the user query, the belief statement, and subject or domain words, all answer-free, so "What do I believe?" and "What is the best X?" retrieve the right edited memory without the key leaking the target answer.

Full build notes are in Engram: writing user beliefs into model weights. The lesson that generalizes: a memory system should not only remember; it should be able to prove where the answer came from.

Sources linked in this post were fetched and verified.