urd-bench

What survives a session boundary, inside the architecture?

Every lab measures memory within a context. What persists between contexts is only ever measured at the system level (LLM + external store + retrieval: LongMemEval, LoCoMo, BEAM, EverMemBench, the Mem0/Zep evals) or as classical continual learning (labelled gradient updates, backward-transfer metrics). Nobody measures it at the architectural level: what persists in the state or weights themselves, under an explicit budget, for knowledge acquired in context.

urd-bench measures exactly that. Learn something in session A, cross a hard boundary, answer in session B from a fresh context. The only things allowed across the boundary are:

• a persistent artifact of bounded size (the fiche — weighed by the harness, never read by it), and
• a bounded consolidation budget (self-declared tokens processed, checked against a cap of 10× one forward pass over the transcript).

A system that exceeds either budget is disqualified loudly (BudgetViolation), not silently truncated. Systems that declare themselves out of contract (diagnostics) are measured anyway and labelled out_of_contract all the way up to the summary — never comparable, never lost.

Failure by capacity, not by decree. Nothing forbids carrying the serialized recurrent state across the boundary — the session is simply sized beyond the information capacity of the budget. What never fit on the slate is not on the photo. When the state-carry baseline scores zero, that is a theorem about bits, not a rule we wrote.

(Urd — the Norn of what has come to pass. Her well is where the gods come to sit in judgement, daily.)

---

Tasks (rung 1)

Two tasks, chosen because they fail differently — no single trick wins both:

T1 — delayed associative recall (lookups). 64 random key→value pairs (12-char values, exact-entropy payload: 480 bytes), exposed over 96 lines with a zipfian skew (8 hot keys seen 5×, carrying 75 % of the query mass — selective retention is part of the task, not a separate one). 20 % of queries ask for keys never seen: the confabulation rate is a published metric. Session ≈ 1.8 K tokens.

T2 — rule persistence (dispositions). An arbitrary convention is taught by 8 examples in session A and must be applied in session B — without being restated. Scored as discrimination: applied-on-trigger minus applied-on-non-trigger (12/12 queries). The base-rate correction is not cosmetic — see the 0.4B result below.

Budgets (profile c1, model class 125M–400M): the session load is fixed; splits vary only the fiche budget.

split	fiche budget	payload/budget	oracle ceiling
over	120 B	4× over capacity	78.6 %
sanity	960 B	0.5× (fits)	100 %

Headline metric: retention B/A — score in session B divided by the same model's in-session control (score_A, measured once per episode, no boundary). Retention can exceed 100 %: a small clean context can beat a large dirty state (measured — see below).

Baselines (the honest starting line)

system	the trick
vanilla	nothing crosses; the floor
gzip_tail	last bytes of the transcript, gzip-squeezed (recency)
retrieval	dedup index of exposure lines ranked by salience, truncated to budget
summary	the model writes its own fiche
state_carry	serialized post-session state, top-k by magnitude
replay_tune	state-tuning (RWKV-PEFT) on the session transcript, steps capped to fit the 10× consolidation contract; the learned time_state packed to budget
replay_banc	same, at a realistic nightly consolidation recipe (64 steps @ctx6144) — 19.2× the cap, declared out-of-contract, kept as the diagnostic that gives the contract's number its meaning

Backbones: RWKV-7 g1d 0.1B and 0.4B (pinned revision + sha256, see Reproducibility). Greedy decoding throughout. Everything below runs on a laptop CPU.

---

Results

Retention B/A, 6 episodes × 25 queries per task, both scales. * = out of contract.

0.1B

	T1 over	T1 sanity	T2 over	T2 sanity
retrieval	30.0 %	125.7 %	0	112.5 %
gzip_tail	8.6 %	87.1 %	18.8 %	100 %
summary	7.1 %	4.3 %	0	0
state_carry / replay_tune / replay_banc* / vanilla	0	0	0	0

0.4B

	T1 over	T1 sanity	T2 over	T2 sanity
retrieval	42.9 %	141.7 %	0	85.7 %
gzip_tail	16.7 %	77.4 %	32.7 %	100 %
summary	11.9 %	33.3 %	0	6.1 %
state_carry / replay_tune / replay_banc* / vanilla	0	0	0	0

What holds at both scales

1. No single trick wins both tasks. Salience (retrieval) wins T1; recency (gzip) wins T2-over — a rule statement doesn't fit in 120 bytes, the last examples do. This two-column structure is the point of having two tasks.
2. The state channel scores zero under contract. At both scales, both budgets, both tasks.
3. Retention > 100 % is real: in sanity, retrieval recalls 55 % of cold keys where the in-session control manages 3 % — and confabulates 0 % where the control confabulates 75 %. The control is a reference point, not a ceiling.
4. Scale signal: summary ×8 at 0.4B (T1 sanity) — a bigger model starts to know how to write its own fiche. The most scale-sensitive baseline.

Control 1 — the ceiling ("you starved it")

The obvious objection to the replay zeros: the 120-byte fiche strangles it. So we hand it the entire tuned state (~38 000× the budget), same harness, same judges:

retention, full state	T1	T2
replay_tune (in-contract dose), 0.1B	0 %	−3 %
replay_banc (19.2× dose), 0.1B	20 %	37.5 %
replay_tune, 0.4B	0 %	0 %
replay_banc, 0.4B	4.8 %	0 %

At the contract dose, the state carries nothing measurable even with unlimited bits — bits and compute kill independently. At 19.2× the dose, the 0.1B shows an asymmetry (habits retained ~2× better than facts) that collapses at 0.4B, and the detail is the finding: the 0.4B applies the rule at 38.9 % on triggers and 38.9 % on non-triggers (control: 81.9 / 13.9). The disposition is installed — 2.8× the base rate — but fully unconditioned. A naive "application rate" metric would have declared this baseline the winner; base-rate-corrected discrimination scores it zero, correctly. Overdosed consolidation imprints a disposition without its condition, and the effect worsens with scale at equal nominal dose.

Control 2 — the bits curve ("your compressor is naive")

The fiche of the state channel is a top-k by magnitude — a blunt encoding. Instead of arguing, we sweep the budget (T2, replay_banc, 0.1B, most favourable point for the channel):

fiche budget	120 B	960 B	4 KB	16 KB	64 KB	256 KB	1 MB	full (4.7 MB)
retention	0	0	0	0	0	0	28.1 %	37.5 %

The signal appears between 256 KB and 1 MB — ~8 700× the contract budget. Honest scope: top-k minimizes L2 error at a given sparsity, not effect-on-generation; a smarter encoding could do better. What the curve establishes is the order of magnitude a better encoder would have to recover (10³–10⁴), not an impossibility. A tuned state appears to have no sparse subset that carries its function — the information is diffuse.

Known limits (read before quoting)

• 6 episodes × 25 queries per configuration. No confidence intervals. These are signals with a clear structure, not tight estimates.
• T2 sanity is degenerate for compression baselines: a T2 session (~490 B) fits whole in the 960 B fiche — gzip reproduces the control to the digit there. That split only checks the floor.
• The banc dose was calibrated at 0.1B; at 0.4B the same nominal recipe imprints harder. The clean control is a dose–response curve per scale — deliberately out of rung-1 scope.
• Absolute scores across different backbones are not comparable (training data/token budgets differ); retention B/A normalizes against each model's own control, compare that.

---

Reproducibility

• Byte-identical data regeneration. One master seed (20260807); python -m bench.calibrate regenerates every episode byte-for-byte (sha256 manifest in calibration/). No data files need to travel.
• Pinned everything. Checkpoints are pulled by pinned revision with sha256 verification (upstream deletes old checkpoints from main on every release — pulling main dies silently). RWKV-PEFT and json2binidx are pinned to exact commits in the bootstrap.
• Provenance committed. artifacts/replay*/costs.json records, for every tuning artifact: steps, trained tokens (counted with the harness tokenizer — the same unit the contract is checked in), GPU, wall time, artifact sha256.
• Raw responses kept. Every run writes responses.jsonl (re-judgeable offline) next to summary.json. Two protocol bugs were found because of this; keep it.

Running it

python3.12 -m venv .venv && .venv/bin/pip install torch rwkv pytest
# checkpoints (pinned revision, sha256-verified): see infra/pod_bootstrap_tuning.sh
#   -> models/rwkv7-g1d-0.1b-20260129-ctx8192.pth  (and/or 0.4b)

.venv/bin/python -m pytest -q                 # 59 tests, <10 s, no GPU
.venv/bin/python -m bench.calibrate           # regenerate data/ (byte-identical)

# the five text/state baselines, full table, laptop CPU (~35 min at 0.1B):
.venv/bin/python -m bench.run \
  --checkpoint models/rwkv7-g1d-0.1b-20260129-ctx8192.pth \
  --t1 data/c1/t1.jsonl --t2 data/t2.jsonl \
  --systems vanilla,gzip_tail,retrieval,summary,state_carry \
  --episodes 6 --queries 25 --out results/mine

The two replay baselines need tuning artifacts (one GPU-hour on a rented pod, ~$0.60): scripts/pod.py + infra/pod_bootstrap_tuning.sh + scripts/tuning_pod.py, then pass --systems ...,replay_tune,replay_banc --replay-artifacts artifacts/replay. Artifact directories are per model (artifact names are transcript hashes, identical across models).

Controls: scripts/diag_replay_plafond.py (ceiling), scripts/diag_replay_courbe.py (bits curve).

Roadmap

• Rung 1 (this): fixed budgets, two tasks, seven baselines, two scales.
• Rung 2: interference / multi-session chains (A₁…Aₙ, backward transfer), the Pareto frontier recall × bits × consolidation-FLOPs, and dose–response curves per scale.
• Rung 3: naturalistic tasks (user preferences, code conventions).

New systems welcome: implement MemorySystem.consolidate(transcript, budget_bytes) -> bytes and .answer(fiche, query, header) -> str (bench/harness/interface.py, ~30 lines). If your system beats replay-tuning at iso-budget, we want to see it.

License

Apache-2.0.