A 2B model learned to say "I don't know". It still remembers less than a fifth.

A personal assistant that forgets is annoying. One that invents a memory is worse. The study asked:

How much of a ~2B model's long-horizon memory gap can external retrieval and LoRA post-training close, without adding parameters, and what does each do to the model's willingness to say it does not know?

That second half is why the benchmark includes unanswerable probes: questions about things the user never said. A model that answers them confidently is fabricating memory.

Student: google/gemma-4-E2B-it (~2B effective parameters), pinned. It is a vision-language base, but all training and evaluation here are text-only.

Teacher for distillation: google/gemma-4-E4B-it.

Memory: a per-persona SQLite store with short-term, episodic and semantic tiers. Retrieval fuses FTS5 BM25 with 384-dim multilingual-e5-small embeddings by reciprocal-rank fusion, re-scores on recency, importance, confidence and entity match, then applies MMR diversification at top-k=8.

Post-training (all LoRA r=16, alpha 32, TRL): SFT on 2,480 examples, DPO on 2,277 preference pairs, then on-policy distillation.

Benchmark: 688 probes over 8 personas: 608 answerable, 80 unanswerable. Accuracy is a single-response judge score (correct at 3 of 5 or above); abstention is measured by a rule-based detector.

Compute: one rented RTX PRO 6000 (~96 GB).

Ablate one component at a time. Each system adds one thing to the last: raw model, then memory, then SFT, then DPO, then distillation. Every system is scored on the same 688 probes.

Memory extraction is checked against the source. An LLM proposes memory claims from the conversation, and each must cite a verbatim span of the turn it came from. 503 of 505 proposed claims from 459 turns passed.

Pairwise comparisons control for position bias. Where two systems are compared head to head, each pair is judged in both orders, so a judge that prefers whichever answer comes first cannot manufacture a win.

Claims are tracked. A 112-entry claims registry (97 verified) links published numbers to 76 hash-pinned artifacts, and 34 errata record the corrections made so far.

Full benchmark, 688 probes, one seed (results/*/metrics.json):

Retrieval does most of the recall work. Adding memory alone moved accuracy from 1 correct answer in 608 to 15.13%. Post-training added about 3.5 points on top.

Post-training does most of the abstention work. Memory alone did not help the model decline, and post-training raised abstention on unanswerable probes above 70%.

Head-to-head (105 probes, judged in both orders): DPO + memory beat SFT + memory 45.7% to 21.0%. Distillation + memory beat DPO + memory 38.1% to 30.5%, a +7.6 point gap attributable to the distillation stage alone.

Scaling the data 10x first looked like it made abstention worse. Two bugs together explained it: a deduplication step had collapsed about 227 abstention examples into 1, and the abstention detector did not recognize the phrasing the model had just been trained to use. Fixing both exposed a real third problem, over-hedging on answerable questions, which needed its own rebalance. That rebalance cut false abstention to 32.1% but also cut abstention on unanswerable probes from 96.25% to 70.0%.

Retrieval alone slightly reduced abstention (13.75% to 8.75%), though the confidence intervals overlap, so this is not shown to be a real effect.

The k-sweep peak is not a free win. Accuracy peaked at k=8 (17.1% on a 120-probe subset), but abstention at that setting was 0.0.

The "+3.3 points from distillation" figure spans two stages. It compared a checkpoint before DPO with one after distillation, so the errata re-attribute it and the distillation-only effect is the pairwise +7.6.

ORPO was not run: the pinned TRL version did not support it.

Absolute accuracy is under 20%. This is a study of what small-model memory can and cannot do, not a capable memory system.

Everything is one seed and one run. Training data was synthetic, generated and judged by the same model family, and not human-reviewed. There is no full-benchmark score for the DPO checkpoint on its own.

The study's central comparison, a 2B model with memory against an 8B model without it, has not been run. Quantization sizes and speeds are not backed by committed artifacts and are not claimed here.

Run the 2B-with-memory vs 8B-without crossover, add seeds so the abstention and pairwise gaps get intervals, and fill the benchmark's alternative-answer field so strict accuracy becomes measurable. Eight checkpoints are already on Hugging Face under arjhinety.