rwkv-toolcaller-bench

A 2.9B RWKV (an RNN, not a transformer) matches Qwen3-30B-A3B on real-world tool calling after $0.53 of state-tuning. This repository contains everything needed to verify or attack that claim.

Write-ups, in order: part 1, reaching parity for $0.53 · part 2, the ceiling and the variable that breaks it · part 3, what a bench really measures · part 4, the wall was a door: the ceiling was readout (FR, EN mirrors linked there).

Why you should be able to check any of this: our method, and everything that failed. Lab: ScarletWolf AI.

⚠️ Base checkpoints: pin the revision, not main. On 2026-08-05, with the G1i release, the four g1g checkpoints were deleted from BlinkDL/rwkv7-g1 (the g1h ones followed on 08-08), and the g1i/g1h preview files were deleted from BlinkDL/temp-latest-training-models. Every base this bench was built on is gone from main — a run that pulled it there now dies on Entry Not Found, which would silently take the reproducibility of these results with it. Hugging Face keeps history, so all scripts here now pin a revision: 6d5762253b34 for g1g/g1h, 159cf82ed0f1 for the g1i release, 1ad45a754e49 for the previews. The SHA-256 assertions already present in the scripts verify that the file fetched by revision is byte-identical to the one the published numbers were produced with — we checked, it is.

Headline numbers (frozen 82-case judge, same scorer for every model)

Category (n)	G1g 2.9B base	+ state-tuning ($0.53)	Qwen3-30B-A3B
Abstention (17)	0	12	17
Argument filling (22)	9	19	17
Hard selection (23)	12	16	14
Simple selection (18)	7	11	10
Multi-step (2)	0	0	0
Total (82)	28 (34%)	58 (70%)	58 (70%)

• Paired McNemar vs base: p = 6.9e-8 (32 discordant cases won, 2 lost).
• vs Qwen: exact tie, perfectly symmetric disagreement (16-16).
• Zero false refusals on the 65 call cases; 1 unparsable output in 82, grammar-free.
• Fully replicated: a second end-to-end run (retrain + re-eval, different machine) produced the same 58/82 and byte-identical generations (164/164). Both trained states share the same SHA-256. The chain is deterministic.

Rung 3 update (2026-07-22)

Targeted data v3 (+162 purpose-built examples) and a composition probe:

	state v3	LoRA v3	combo (state v3 on LoRA v3)
Abstention (17)	16	17	17
Arguments (22)	18	15	17
Hard selection (23)	15	15	17
Simple selection (18)	12	12	8
Total (82)	61 (74%)	59	59

Campaign champion: state v3, 61/82, byte-reproducible (states/rwkv-state-v3.pth). LoRA v3 reaches perfect abstention (17/17, Qwen's score) but confirms the capacity-vs-arguments trade-off. The composition probe (a state trained on the base injected onto merged LoRA weights) does not collapse: campaign record on hard selection (17/23) with perfect abstention, at the price of simple selection and format degradation. Raw generations and scored results for all three are in results/. Native composition was run as rung 4 (below).

Rung 4 update (2026-07-26) — native composition

Question: does training the state directly on the merged LoRA v3 weights (instead of injecting a base-trained state, rung 3's out-of-distribution probe) repair the mismatch damage while keeping the records?

	combo (OOD, rung 3)	native (rung 4)	state v3 (champion)
Abstention (17)	17	17	16
Arguments (22)	17	15	18
Hard selection (23)	17	17	15
Simple selection (18)	8	10	12
Total (82)	59	59	61

Half-confirmed. What the native training repairs, as hypothesized: output format (4 → 1 unparsable), simple selection partially (8 → 10), while holding perfect abstention and the hard-selection campaign record. The surprise is the other half: arguments drop to exactly the LoRA v3 solo score (15/22) where state v3 alone scores 18. A state trained natively on a LoRA substrate aligns with the substrate, weaknesses included — it does not correct it. The OOD combo kept arguments at 17 precisely because the mismatch preserved state-v3-like behavior. Composition does not sum strengths: the substrate dominates. State v3 (61/82) remains the campaign champion.

Artifacts: states/rwkv-state-native.pth (SHA-256 502ff660f6f3772fced955d89c0e100ac2b1e29caa5aa9ee4a55cdbcb843b1dd — only meaningful injected on the LoRA v3 merged weights, SHA-256 5492d92afa06f2726a2e5a7059f404e3e843128a007700631d39800e27802f58, 5.6 GB, not committed), raw generations + scored results in results/, recipe in training/rung4_native.sh.

Rung 5 update (2026-07-26) — targeted data hits the capacity wall

Rung 3 showed that 113 purpose-built examples conquered abstention. Rung 5 asked whether the same surgery works on the state's weakest category, simple selection (12/18): error analysis of the 6 misses showed namespace collisions (linear_search_issues → github_search_issues), invented tool names (browser_switch_theme, absent from the 231), and one argument hallucination. Data v4 = v3 + 135 examples aimed exactly at those failures (bidirectional namespace disambiguation, exact-name anchoring, argument minimalism), all schema-validated, Jaccard-filtered against the judge. Same champion recipe (state-tuning on the base model, one variable: the data).

Result: 58/82 — and 0 of the 7 targeted cases were fixed. The wrong answers shifted instead of correcting (model_get_info → model_manage, browser_switch_theme → browser_set_theme); on the selection metric that produces the headline scores, 7 previously-passing cases regressed elsewhere while 4 recovered (61 − 7 + 4 = 58). Meanwhile abstention — a disposition, not a lookup — reached 17/17, the state's first perfect score on it.

Recompute either count from the committed JSON:

import json
def arm(f, k=None):
    d = json.load(open(f)); return {s["id"]: s for s in (d[k] if k else d["tuned_direct"])}
v3, v4 = arm("results/results_rung3_state.json"), arm("results/results_rung5_state_v4.json")
reg = [i for i in v3 if v3[i]["selection_ok"] and not v4[i]["selection_ok"]]   # 7
rec = [i for i in v3 if not v3[i]["selection_ok"] and v4[i]["selection_ok"]]   # 4

Under the stricter pass rule (correct tool and a perfect argument score) the same diff reads 8 regressions / 3 gains, but the totals become 61 → 56 rather than 61 → 58. The numbers quoted throughout this README are the selection metric, so the 7/4 figures are the ones consistent with them.

Reading: an 11 MB initial state moves dispositions (when to abstain) but cannot pin fine-grained name mappings — consistent with the state-as-index collapse we measured earlier (a fixed state cannot retain 231 tools). Four rungs land at 61/59/59/58: the state family is at its capacity ceiling around ~60/82 on this judge, and further data curation reshuffles the profile instead of raising it. Next escalations are reward-based training (GRPO) or more adapter capacity, not more examples. Artifacts: states/rwkv-state-v4.pth (SHA-256 487dd65283a9dcb86a5cdbd84706828718f9a9fd322db310332dd719567e4bb3), training/additions_v4.jsonl + training/gen_data_v4.py + training/rung5_state.sh, raw generations + scored results in results/.

Rung 6 update (2026-07-27) — GRPO: RL slides along the frontier

We implemented GRPO (group-normalized advantages, clipped surrogate, mechanical reward = our scorer) on top of the champion state v3, in two arms run on separate pods: state (keep training the initial state, LR 5e-4) and lora (freeze state v3, train a fresh LoRA r=16 on top, LR 2e-5 — 1e-4 destroys the policy in one round: abstention 8/8 → 1/8). Four rounds × 250 prompts × K=8, on-policy resampling with weight resync every round. Code in training/grpo/.

Judge 82	SFT state v3	GRPO state	GRPO lora
Abstention	16	16	16
Arguments	18	13	13
Hard selection	15	16	16
Simple selection	12	15	14
Total	61	60	59

Findings: (1) RL moves what targeted data could not — simple selection 12→15 where rung 5 fixed 0/7 — but pays for it in argument precision, exactly as the reward shaping prices it (+1 selection vs +0.5 arguments). (2) Both capacity vehicles converge to nearly the same profile (args 13, hard 16, simple 14-15): the policy optimum is set by the reward, not by where the trainable parameters live. (3) Early stopping does not help: probe peaks (0.896/0.927 at rounds 1-2) score 58/59/60 on the judge. Nine measurements land in 58-61: the band is a Pareto frontier of the 2.9B backbone — training chooses the operating point on it, never the level.

Rung 7 update (2026-07-27) — one variable resolves the wall: 69/82 at 7.2B

Same G1g generation (7.2B checkpoint dated three days before our 2.9B), same v4 data, same state-tuning recipe, same judge — the only change is backbone size. Cost: ~$3 of L40S. And the control that makes the claim honest: the raw 7.2B scores 29/82 (35%) — statistically indistinguishable from the raw 2.9B (28/82), with the same signature (abstention 0/17, 9 unparsable).

Category (n)	7.2B base	2.9B ceiling (9 runs)	7.2B G1g state-tuned	Qwen3-30B (231 raw)	Qwen3-30B (+ top-40)
Abstention (17)	0	16-17	17	17	17
Arguments (22)	9	≤18	21	17	20
Hard selection (23)	12	≤17	18	14	17
Simple selection (18)	8	≤15	13	10	16
Multi-step (2)	0	0	0	0	1
Total (82)	29 (35%)	58-61	69 (84%)	58	71

The wall was the backbone — and scale is only worth what tuning activates. Raw, the two checkpoints are twins (28 vs 29/82). State-tuned with the same $1 recipe, they are 61 vs 69: the tuning delta grows with size (+33 → +40). After six methods failed to buy a single point above 61, scaling 2.9B → 7.2B buys eight — and the categories that move are precisely the ones diagnosed as backbone-bound (arguments 18→21, beating even Qwen+top-40's 20; hard selection to a campaign-record 18). The 7.2B RNN now beats Qwen3-30B in its production configuration by 11 points (69 vs 58) and sits 2 points from the 30B with equal scaffolding — at 4× fewer parameters. Simple selection (13/18) still lags: the namespace-collision weakness survives scaling, a real thread for future work. Multi-step stays at 0 (one-turn harness, unchanged). Artifacts: states/rwkv-state-72b.pth (SHA-256 a2e4f11266277fcd16dbf3cc538ec982794d94da8a9e521380dc367cfc25864d, loads on rwkv7-g1g-7.2b-20260523-ctx8192), recipe training/rung7_72b.sh, raw generations + scored results for tuned and base control in results/.

Rung 8 update (2026-07-27) — the architecture control cuts both ways

The open question from rung 7: is the wall a small-model thing or an RWKV thing? Control: Qwen2.5-7B-Instruct (transformer, size-matched to our 7.2B champion), on the exact protocol — raw on the judge, then LoRA-tuned (r=32, 2 epochs) on the same 2,920 examples, same raw-text format, and re-judged.

	raw	after our recipe, applied as-is
RWKV G1g 2.9B	28	61 (+33)
RWKV G1g 7.2B	29	69 (+40)
Qwen2.5-7B-Instruct	69	51 (−18, p = 9.1e-4)

Two findings, one per direction. Raw, the transformer already knows the task: 69/82 without any tuning (hard selection 20/23 — above every model we measured, its bigger sibling included). The gap between families is pretraining, not architecture: Qwen2.5's corpus is saturated with function calls, RWKV G1's is not. Our $3 state-tune closes exactly that gap.

And the recipe is not portable. Transplanted naively onto the already-instruction-tuned transformer (LoRA substituted for state-tuning, no chat template, hyperparameters untouched), the same data degrades it: 69 → 51, driven by 25 false refusals — the abstention curriculum that RWKV needed (it never abstained) teaches an already-calibrated model to shut up on real calls. A Qwen-optimized SFT would likely behave otherwise; that is precisely the point. Recipes are model-fit, not commodities: state-tuning is RWKV-native, and that fit is what +40 for $3 buys.

Artifacts: results/generations_qwen7b_{base,tuned}.jsonl + scored results; recipe training/grpo/rung8_transformer.py.

Checkpoint survey (2026-07-28) — the protocol was hiding the progress

Peng Bo (BlinkDL) asked for the raw, untuned numbers of the newer checkpoints, then pointed out that our prompt template might be penalising them. Both turned out to matter. Everything below is in results/checkpoint-survey/, scripts in training/checkpoint-survey/.

Round 1 — raw, our published protocol (zero state, greedy, 150 tokens, ```json forced right after Assistant:):

Checkpoint	Raw score	Unparsable
g1g 2.9B	28/82	—
g1g 7.2B	29/82	9
g1h 2.9B	20/82	16
g1h 7.2B	29/82	9
g1i-preview 7.2B	28/82	26
g1i-preview 13.3B	44/82	3

The 13.3B is a different animal: best raw number we have ever measured on this judge (arguments 16/22, hard selection 17/23 — the score our state-tuned 2.9B reached after training), and it holds format discipline where the 7.2B does not.

Update, 29 July — the newest 2.9B, measured at BlinkDL's request. He asked us to check g1i_preview5031-2.9b-20260729, published that morning. Raw, in the official G1x catalog format, plain greedy without penalties: 35/82, the best 2.9B this benchmark has seen.

2.9B raw, each in its own format	Score
g1g (checkpoint 20260526)	25
g1h (checkpoint 20260710)	26
g1i preview5031 (20260729)	35

The gain is spread rather than concentrated — arguments 12/22, hard selection 12/23, simple 11/18, all above both predecessors — so the training run is improving tool selection itself, not just format compliance. What has not moved is abstention: 0/17, as with every raw checkpoint we have ever run. No untrained model has scored on that category here. It remains the thing only training installs. (Comparability note: the g1g and g1h rows carry repetition penalties, dropped for this run on BlinkDL's advice; the effect measured elsewhere is about one point.)

Round 2 — each checkpoint on its own declared format. Following BlinkDL's suggestion, we probed each model with free completions from token0 + "System: {", "tool_call" and "System: You", and read back the templates it had been trained on. g1h reconstructs <tool_call>…</tool_call> with ChatML roles; g1i emits OpenAI-style calls and a plural <tool_call_begin> path. The formats are also declared upstream in RWKV-Vibe/rwkv-Primitive-Bench (COMPLETION_TOOL_FORMATS). Re-evaluated with each model's own format and the official decode defaults (greedy, 1024 tokens, presence 1.0 / frequency 0.1 / decay 0.99):

Checkpoint	Our format	Native format	Δ
g1g 2.9B (control)	28	25	−3
g1h 2.9B	20 (16 unparsable)	26 (1)	+6
g1i-preview 7.2B	28 (26 unparsable)	37 (2)	+9

The control is what makes this readable: g1g gains nothing from the new setup — it loses 3 — so the +6 and +9 are not the longer budget or the penalties, they are the match between checkpoint and training format. g1i is not "equal to g1g" as our raw table said; it is clearly ahead once asked properly. Our protocol was measuring our harness as much as the models.

What the format does not fix: abstention is 0/17 and multi-step 0/2 for all three. Syntax was the artificial ceiling; judgement is the real one — and that is what the $1-3 state-tune buys (+33 to +40 points, rungs 1-7 above).

Prompt length (2026-07-28) — "rwkv likes short prompts" is worth up to 20 points

BlinkDL's third hint. Our system block was ~34k characters of full JSON schemas for the retriever's top-40 tools. We replaced it with one line per tool — name(arg*, arg) : short description — same 40 tools, same 82 cases, same native format. Artifacts in results/prompt-length/, scripts in training/prompt-length/.

Raw g1i-preview 7.2B, one change at a time:

Setup	Score	Arguments
our original protocol	28/82	8/22
+ native format	37/82	12/22
+ compact prompt	48/82	20/22

Our own state-tuned 7.2B (the 69/82 of rung 7), same state, same tools, same training-time assistant prefix, only the system block changes. Note the control: re-running the long prompt in this harness scores 70, not the published 69 — the +1 comes from the decode defaults adopted here (1024 tokens, repetition penalties). The compact gain is measured against that 70:

	full schemas	compact	full descriptions, one-line args
Abstention	17/17	17/17	17/17
Arguments	21/22	22/22	22/22
Hard selection	19/23	19/23	19/23
Simple selection	13/18	13/18	13/18
Total	70/82	71/82	71/82
Eval wall-clock	1467 s	401 s	689 s

71/82 is exactly Qwen3-30B's score with our retrieval scaffold — reached by a 7.2B RNN trained for $3, with a prompt ten times shorter and a prefill that collapsed from ~5.3k tokens to ~600. The score gain is small (+1); the wall-clock gain is not (3.7x).

Format sweep — four renderings of the same 40 tools, same state, same prefix:

System block	Total	Args	Hard	Simple	Multi	Wall-clock
full JSON schemas	70	21/22	19	13	0/2	1467 s
compact (desc truncated to 90 chars)	71	22/22	19	13	0/2	401 s
full descriptions, one-line args	71	22/22	19	13	0/2	689 s
typed args, short desc	71	22/22	19	12	1/2	416 s
names + args only, no descriptions	62	21/22	18	11	1/2	245 s

Three readings. The plateau is 71 and three different renderings reach it, so the win is not about truncating descriptions — dropping them entirely costs 9 points, most of it abstention (17/17 → 11/17): the model needs to know what a tool does in order to decide not to call it. What the verbose JSON was costing is structure, not semantics. And restoring argument types in one line (card_id:integer*) is the only rendering that ever scored on multi-step (1/2, on a three-argument valkyrie_move_card) — the first point on that category by any model in this benchmark, Qwen3-30B included.

Two controls worth reading before quoting any of this. Swapping the assistant prefix to **Tool Call:** — the format declared upstream for g1g — costs our own state-tuned model 9 points and half its abstention (69 → 60, 17/17 → 8/17): a state trained on one priming does not transfer to another. And a top-20 shortlist scores 46/82, below the top-40 compact (48), but with the campaign's best simple selection (15/18): fewer tools means less confusion and more misses at once. The compact rendering also truncates descriptions to 90 characters and drops argument types, so it removes information, not just tokens — a format sweep is running to find where the optimum sits.

Rung 9 (2026-07-29) — training on the compact format makes it worse, and one-way

The obvious follow-up: if serving on a compact tool list is worth +1 and a 3.7x speedup, does training on that format push the ceiling further? We rebuilt the 2,920 examples with the typed one-line rendering (17.5M → 4.9M tokens), trained the same 7.2B state on them (3 epochs, ctx 2048, same LR schedule), and evaluated both ways.

7.2B state-tuned	eval in typed	eval on full schemas
trained on full schemas (rung 7)	71	70
trained on typed compact (rung 9)	65	39

Two findings, the second more useful than the first.

Training on the compact format costs 6 points (71 → 65), and the loss is concentrated in abstention (17/17 → 13/17) with a small gain in simple selection (13 → 14/18). The likely cause is corpus volume rather than format: the compact rendering shrinks the training set 3.6x, and abstention is the behaviour that needed the most examples to install in the first place.

More interesting: the state trained on long prompts transfers down to the compact format (71), while the state trained on compact prompts collapses on long ones (39/82, abstention 3/17). Generalisation is one-way. If you only get to train once, train on the richer format — you can always serve the short one. Artifacts: states/rwkv-state-72b-typed.pth (SHA-256 13b2702ad0b71d49b15b866e18a28e062975fe20c474c99db963ec25804a968f), recipe training/rung9_typed.sh, generations in results/prompt-length/.

Official G1x format (2026-07-29) — same score, a third of the prompt

BlinkDL pointed out that the G1 series has its own function-call templates and that some completions show a <think> block for harder calls. The upstream implementation is in rwkv-Primitive-Bench (g1i_tool_catalog_entry): the tool catalog is {name, description, arguments:{schemas}} — no OpenAI parameters/properties wrapper — with descriptions cut to the first sentence (140 chars max), per-argument descriptions dropped, type/enum/items kept, plus a short rules block. We reimplemented it verbatim and ran our state-tuned 7.2B on it, greedy, no penalties (his other advice).

System block	Total	Abstention	Args	Hard	Simple	Chars
our original (full JSON schemas)	70	17/17	21/22	19/23	13/18	34 469
our compact (hand-rolled)	71	17/17	22/22	19/23	13/18	~3 000
official G1x catalog	71	17/17	22/22	18/23	14/18	11 052

The official format matches our best score with a third of the original prompt, without the arbitrary truncation we had invented — and it keeps enum values, which our hand-rolled version dropped. It is the format we now use.

The <think> priming does not transfer to a state trained without it. The same catalog with Assistant: <think> (the upstream default for g1g/g1h) scores 58/82. The reasoning itself is fine — the model correctly identifies the tool in French prose — but it talks itself into over-filling: abstention collapses to 9/17 and arguments to 19/22, with invented status_key, due_date and tags appearing during the reasoning. Abstention is a disposition installed by training, and free reasoning routes around it.

Caveat, and it matters: our state was trained on fence-JSON priming with no <think> examples at all. This measures our state under think priming, not whether think priming works for G1 in general. A state trained with think blocks might well invert the result.

Update, 30 July — fake think is neutral here. RWKV7-G1x-templates.txt (in BlinkDL/RWKV-LM/RWKV-v7/, not findable by search, you have to list the directory) documents a third option we had missed: an empty think block, Assistant: <think>\n</think>, described upstream as "fake thinking, nice results and fast". Both spellings of it score 71/82, category for category identical to asking directly:

Priming	Total	Abstention	Args	Hard	Simple
direct JSON fence	71	17/17	22/22	18/23	14/18
<think>\n</think>	71	17/17	22/22	18/23	14/18
<think></think>	71	17/17	22/22	18/23	14/18
<think> (real reasoning)	58	9/17	19/22	17/23	13/18

The empty block is not inert — 20 of the 82 generations differ textually from the direct arm — but not a single verdict flips: zero cases won, zero lost, in either spelling. The differences never touch correctness. What destroys the score is the reasoning itself, not the marker. That is consistent with the upstream advice rather than against it: fake think is recommended for open-ended answers, where priming a careful register helps. A single short function call has nothing to make careful, the format constrains it all.

The same file also confirms two things we had measured the hard way ("the key is to keep it concise") and documents the multi-turn ReAct path, User: Function output:\n{...}\n\nAssistant: ```json, which is the route to the multi-step category that sits at 0/2 for every model here.

(A first attempt scored 0/82 — our stop condition cut generation at the opening fence, before a single character of the call. The bug is ours, the fixed run is the one reported. Both are in results/g1x-official/.)

Distilled hybrid (2026-07-29) — tool calling does not survive kldiv distillation

OpenMOSE builds RWKV-Qwen hybrids: a transformer converted to an RWKV hxa07D + NoPE-Attention architecture (45 of 48 layers linear, 93.75% KV cache reduction), distilled with RADLADS. One of them has a teacher we had already measured — Qwen3-30B-A3B-Instruct, our reference model. So the before/after is exact, on the identical judge.

He stated his own prediction up front: his distillation set contains few tool calls, everything is trained with kldiv only, and in his view tool-call distillation needs on-policy training. The measurement agrees with him.

	full schemas	G1x compact catalog
Teacher (Qwen3-30B-A3B)	58/82	71/82
Hybrid (RWKV-Qwen3-30B-A3B-hxa07d, IQ4_XS)	38/82	49/82
	−20	−22

The category breakdown says what is lost. Abstention goes from the teacher's 17/17 to 0/17 on full schemas and 4/17 on the compact catalog: asked a question no tool can answer, the hybrid calls text_join and stuffs an explanation of TCP versus UDP into its arguments. Content capability survives much better — arguments 17/22, simple selection 14/18, both respectable.

That maps onto what the rest of this benchmark keeps finding: dispositions are the fragile part. Knowing when to stay silent is not knowledge, it is a reflex, and a kldiv objective on non-tool data does not carry reflexes across. Same shape as our state-tuning result, from the opposite direction — training installs dispositions cheaply, distillation drops them.

Two caveats, both real: the hybrid is quantized to IQ4_XS while the teacher was measured unquantized through an API, so part of the gap may be compression; and the distillation never targeted tool calling in the first place. A hybrid distilled on-policy with tool-call data is the experiment that would settle it, and we would run it. Artifacts in results/hybrid-distill/, harness in training/eval_hybrid_api.py (llama-server from the hxa07d branch of OpenMOSE's llama.cpp fork; note nvcc must be on PATH for the CUDA build).

Think training (2026-07-29) — reasoning before the call costs more than it returns

We had said we would run this one. Since think priming collapsed a state that had never seen a think block, the obvious question is whether a state trained with them inverts the result. We built 2,920 training examples in the G1x catalog format, each with a short reasoning block before the call, generated from the query and the gold call so it justifies the right answer. Crucially, the 425 abstention cases got reasoning that explicitly concludes no tool applies — aimed straight at the observed failure, where free reasoning talks the model into acting.

7.2B state	total	abstention	args	hard	simple
no think training, think priming	58	9/17	19/22	17/23	13/18
think-trained, think priming	61	11/17	21/22	18/23	11/18
same state, no think at all	53	0/17	20/22	19/23	14/18
no think anywhere (our best)	71	17/17	22/22	19/23	13/18

Training on think recovers part of what think priming broke (58 → 61) but stays 10 points below simply asking directly. Abstention tops out at 11/17 despite the 425 refusal traces. On this task, reasoning before the call costs more than it returns.

Two side findings from the same run, both new. The one-epoch checkpoint already scored 61 — epochs 2 and 3 changed nothing, which means our three-epoch runs have been buying nothing. And the think-trained state scores 53 with abstention 0/17 when evaluated without think: the same one-way generalisation as rung 9, a state does not survive leaving the format it was trained on.

Artifacts: states/rwkv-state-72b-think.pth (SHA-256 bce70838d84dad33d86ec05446ed5b02629765731a293c75b0d55a10b72ec33e), trace generator training/gen_think_traces.py, recipe training/rung10_think.sh, generations in results/think-training/.

G1i release survey (2026-08-07) — raw flatness broken, and a new 2.9B champion

Measured at BlinkDL's request, the day after the release. Raw = zero tuning, greedy, each checkpoint in its official G1x format (the paper-3 rule), same frozen judge and scorer as every number in this file.

G1i raw (2026-08-05 release)	score
1.5B	29/82
2.9B	36/82 (29 in our old format)
7.2B	43/82
13.3B	52/82

The campaign's oldest constant is gone: g1g raw scored 28-29 at every size ("size alone buys nothing"), g1i raw scales — its pretraining now carries tool calling that scale can express. The 13.3B raw sits 6 points from our tuned 2.9B champions with args at 19/22. What has NOT changed: abstention is 0/17 at every size, fourth generation in a row — knowing when to stay silent still has to be installed, never inherited.

State-tuning the g1i 2.9B with the exact champion recipe: 63/82 — new 2.9B best (the g1g champion held 61 since July 22). Abstention 16/17, hard selection 18/23 (ties the tuned 7.2B category record), +34 over its raw base in training format. Fully replicated 2026-08-08: a second end-to-end run (retrain + re-eval, different pod, next day) produced the same state SHA-256 and byte-identical generations (82/82) — the chain is deterministic, as it was for the g1g champion. Remaining caveat: training data regenerated by the seeded pipeline (+6% tokens vs the original g1g run's log, likely catalog drift since July). Artifacts: states/rwkv-state-g1i29.pth (SHA-256 starts 4ec13cc7d8925a61, loads on rwkv7-g1i-2.9b-20260805-ctx16384), recipe training/rung_g1i_state.sh, train log and raw generations in results/checkpoint-survey/.

G1i 1.5B, per BlinkDL's ask (2026-08-11): raw 29/82, state-tuned 55/82 (exact champion data, byte-identical training file). Abstention hits 16/17 at the smallest size while everything else degrades down-scale: simple selection 9/18, args 15/22, and 4 unparsable outputs where every bigger size has zero. The disposition is the first thing that installs and the last that breaks; form and mappings go first.

And the readout equalizer holds one floor down. The paper-4 corrector (231 centroids from training data, tool-to-tool policy, decision layer L16/24 picked by corpus-only CV, the same depth fraction as L21/32), with the 2.9B's pre-registered margin applied as-is: free 57 -> 64/82 in the corrector harness (curve peaks at 65, exploratory). That puts 1.5B tuned + readout at the level of the bare tuned 2.9B (63). Corrector gains by size: +7 (1.5B), +6 (2.9B), +2 (7.2B) — the smaller the model, the more readout pays. Artifacts: states/rwkv-state-g1i15.pth, generations and reranker outputs in results/checkpoint-survey/.

Transformer control, per BlinkDL's ask: Qwen3.5-4B raw scores 59/82 on the raw-prompt protocol and 60/82 with its own chat template (format effect ~nil for Qwen vs +7 for g1i). It posts the first perfect simple selection ever measured on this bench (18/18 in both arms) and 20/23 hard selection, but abstains at 4/17. A 4B transformer now matches Qwen3-30B's production config (58).

Champion recipe at 13.3B (2026-08-09) — 70/82, a nominal record that scale did not earn

Same recipe as every state in this file (--peft state --op fla, x070, lr 1e-2→1e-3, one epoch on the v3 data, dims auto-detected from the checkpoint: n_layer 61, n_embd 4096), one variable changed: the base is the g1i 13.3B. 2,719 steps over 16,709,798 tokens, loss 0.688 → 0.046 (50-step means), ~6.5 h train + 70 min eval on one A40.

arm	total	simple	hard	args	abstention	multi
g1i 13.3B raw (official format)	52/82	15/18	18/23	19/22	0/17	0/2
g1i 13.3B state-tuned	70/82	14/18	19/23	20/22	17/17	0/2
g1x 7.2B state-tuned (previous best)	69/82	13/18	18/23	21/22	17/17	0/2
g1i 2.9B state-tuned (champion)	63/82	12/18	18/23	17/22	16/17	0/2

70/82 is the bench's highest standard-harness score, and it means less than it looks. Against the tuned 7.2B it is 4 won / 3 lost, exact McNemar p = 1.0 — indistinguishable. Against the tuned 2.9B champion, 11 won / 4 lost, p = 0.119 — still not significant on 82 cases. Going from 2.9B to 13.3B, a 4.6× parameter budget, buys 7 nominal points and no statistically detectable improvement. The state-tuned arm of this harness saturates around 69-70.

What the tuning actually bought at 13.3B is abstention: 0/17 → 17/17, which is 17 of the 18 points gained over its own raw base. Selection barely moves (simple 15/18 → 14/18, hard 18/23 → 19/23) and args are already at 19/22 raw. Fifth generation in a row with the same shape: the raw model carries the knowledge, the state installs the disposition. Argument filling is at ceiling everywhere (mean args score 0.994 over the 53 cases where the tool is right, vs 0.986 at 7.2B), and multi_step stays 0/2 for every model ever measured here, tuned or not, RWKV or transformer.

Artifacts: states/rwkv-state-g1i133.pth (SHA-256 starts 9b30081738b015d1, loads on rwkv7-g1i-13.3b-20260805-ctx16384), recipe training/rung_g1i13_state.sh, train log and raw generations in results/checkpoint-survey/. Single run, not replicated (the 2.9B chain was bit-replicated on 2026-08-08, so determinism is established for the recipe, not for this base).

Rung 6 (2026-08-11) — the curriculum hole: the law predicts, then the data confirms, at both scales

The residue of the gate work had a measured cause: in sft_raw_v3, queries carrying memory/promise/decision vocabulary include 145 examples that call a tool versus 6 that abstain — and all six are lexical false friends ("how does RAM work"). The model obeys its curriculum.

First pre-registered prediction of this campaign (frozen by public commit before a single training example was written): per the dispositions-vs-mappings law of parts 2-3, 160 new examples (120 negatives across three families, each with paired positives so a boundary gets installed rather than a silence reflex) should (P1) improve abstention on those families, (P2) not cost legitimate same-domain calls, and (P3) leave name confusions unchanged — the falsifiable control.

Fresh private 100-case judge, halves drawn at construction, power lock before any verdict scoring, single scoring, champion recipe, one variable (the data):

	2.9B (primary)	7.2B (replication)
abstention missed, champion → treated	18 → 7 (p = 0.013)	17 → 7, 0 lost (p = 0.002)
legitimate calls lost (bound 3)	0	3 — at the exact bound
name confusions	0 won / 4 lost	2 / 2 (p = 1)
verdict	SUCCESS	SUCCESS

The law has now been tested in both directions: 135 examples aimed at fine mappings fixed zero of them (rung 5); 160 examples aimed at a disposition installed it — and the same training left the mappings control flat.

Two findings the replication adds. The hardest family (hypothetical/retraction) is inert at 2.9B (4→4) and moves at 7.2B (7→4): whether a disposition responds to data is a capacity question, not a separate mechanism. And the boundary's cost mirrors it: zero legitimate calls lost at 2.9B, three at 7.2B — the model that best learns to keep quiet is the one that starts keeping quiet slightly too much.

Artifacts: training/gen_data_v6.py + additions_v6.jsonl (public), training/rung6_curriculum.sh, training/rung6_72b.sh. The judge stays private; its construction protocol, pre-registration and both scorings are timestamped by commits.

Knowing when to shut up (2026-08-10) — the abstention gate, pre-registered and confirmed

Abstention is the one behaviour four generations of raw checkpoints never had: 0/17 for every untuned model we ever measured, g1g through g1i, 1.5B through 13.3B, plus Qwen3.5-4B at 4/17. State-tuning installs it — but only inside its curriculum. On fresh judges the tuned models hold 7/17 and 9/17.

Same diagnosis as part 4, one layer down: the state knows, the generation doesn't say it. A binary direction — c(__null__) against the mean of the 231 tool centroids, taken from the SFT corpus, judge-disjoint — separates "no tool needed" at AUC 0.95-0.99, including on the very cases where the behaviour fails half the time.

The gate reads that direction at prefill and forces silence when it fires. It never invents a call, never changes a chosen tool, and cannot break an abstention the model already holds.

Pre-registered confirmation (protocol, rule, threshold source, criterion and a power lock all frozen by a public commit before generation; a fresh 100-case judge; a single scoring):

	7.2B (primary)	2.9B (replication)
abstentions held unaided	10/30	11/30
recoverable misses	20	19
recovered by the gate	16 (80%)	18 (95%)
exact McNemar	p = 3.1e-5	p = 7.6e-6
correct calls destroyed (of 20)	1	2
abstention with the gate	26/30	29/30

Two things this cost us, both published rather than buried:

A threshold relative to the batch does not survive prevalence. The first version scored each query in standard deviations of the evaluated batch. It worked (+5) until we made the judge denser in abstentions: Δ +3 at 21% abstention, +1 at 35%, 0 at 50%. A batch-relative threshold measures where a query sits among its neighbours, not what it is. Replaced by corpus centering, which keeps the AUC and makes the decision per-query. You transport a direction, not a value.

The first confirmation failed, by one case. Judge v4 left only 6 recoverable errors where the power calculation assumed 12 — because the power had been bootstrapped from older judges, and the new cases turned out easier (24% abstention failure vs 47-59% before). The gate recovered 5 of 6; the bar needed 6 of 6. We published it as a failure, then fixed the instrument rather than the criterion: dropping the 23 cases out of 50 that both models already held (greetings, small talk, vague asks) and concentrating on the families that actually trip them took recoverable errors from 6 to 20 at identical judge size. A judge is not scaled up, it is densified. And the lesson is now mechanical: a power lock counts harvestable errors on the visible calibration half before the verdict half is ever scored — under threshold, nothing is scored and no cartridge is burned.

Is the hole an artifact of our long prompt? Fair question, since our judges run on the original full-JSON system block while this bench adopted the official G1x catalog in July. Replaying the verdict half in another format would be the sin the protocol forbids, so we replayed the calibration half, which is visible by right, with one variable changed (system block 20,600 to 8,500 characters): abstention 13/30 in both formats, 17 harvestable errors in both, two discordant cases each way, exact McNemar p = 1.0. Selection gains a point (16 to 17 of 20), consistent with July's measurement. The hole is a fact about the model, not about the harness.

What still resists, on both models: "don't keep this in memory, I'm thinking out loud." The request talks about memorising, and neither the generation nor the state tells talking-about an action apart from asking for it. Not a threshold problem — a semantic boundary.

The wall was readout (2026-08-04) — state probe, training-free corrector, pre-registered confirmation

The campaign's open question (135 targeted examples fixing 0/7 name confusions; collisions surviving scale) has a mechanical answer: the information was in the model's state all along, and generation fails to read it. Full story in part 4.

The probe. Protocol frozen before measurement (probes/state-separability/PROTOCOLE.md), 8 really-confused pairs, 240 fresh leak-filtered queries, positive and negative controls, two context regimes. Verdict: 36/36 cells significant (1,000 permutations, Bonferroni 44/pair), separability 0.96-1.00, mostly by bare nearest-centroid. The decisive contrast, in the realistic top-40 context (raw model): behavior 147/270 with runtime_info vs model_get_info at 2/30, while state separability holds at 0.99. Same prompts feed both sides, and zero unparsable outputs in 540 generations: judgment, not syntax.

The corrector. One centroid per tool from existing training data (231 × 8, sft_raw_v3, judge-disjoint), decision by dot product against the shortlist at prefill time. Cross-context centroid transfer needs per-distribution mean-centering (76% raw → 95% centered; corpus mean alone: 73%). Policy: never override an abstention, correct tool → tool only, above a cosine margin — the pure selector destroys abstention (16/17 → 1/17).

2.9B champion + state v3 (public judge)	score
free generation (exact replication of the published 61/82)	61/82
pure selector	50/82
tool → tool corrector	65-67/82 across the whole margin curve

Pre-registered confirmation. Policy, margin (0.02), layer (L21) and centroid checksums frozen by a commit before the run, on a private never-seen 82-case judge (same distribution, mechanically validated against the schemas, retriever recall 65/65), single scoring: 48 → 56/82, Δ = +8, 10 won / 2 lost, exact McNemar p = 0.039 — criterion met. The judge stays private to remain an instrument; its construction protocol is public and the next round's pre-registration can happen under scrutiny.

Scale (exploratory). Baseline replicated exactly (69/82). The 2.9B-calibrated policy does not transfer (67/82 at τ=0.02, net-negative low-margin swaps on a model that errs less); τ=0.05 reaches 71/82, post-hoc — and it stays that way: the pre-registered v3 confirmation (private fresh judge, τ=0.05 and L27 frozen by prior commit, single scoring) returned non-conclusive (Δ = +1, 2 won / 1 lost, below the registered Δ ≥ +2 bar). Diagnostic: the v3 judge left the corrector little to harvest (free selection 34/41 vs 31/41 on v1, ~7 harvestable errors). Lesson recorded: confirming a corrector requires a judge dense in harvestable errors, not merely a fresh one. The 2.9B confirmation (+8, p = 0.039) is a separate judge and stands. Meanwhile the internal signal grows with scale — 231-way centroid CV, same corpus and procedure:

layer	2.9B	7.2B
15	0.895	0.916
21	0.899	0.933
27	—	0.949

Probe error halved, and the information migrates deeper (2.9B peaks around L21; 7.2B still rising at L27). Corrector gains shrink as the ceiling nears (+6 at 2.9B, +2 at 7.2B): the readout corrector is a small-model equalizer — corrected 2.9B (67) lands 2 points from the raw tuned 7.2B (69) at 2.5× fewer parameters, in a 150 MB artifact.

Side findings: valkyrie_delete_subtask does not exist in the 231 schemas (the remove→delete "confusion" is a name hallucination, a distinct phenomenon); the tuned state shown a 2-tool context continues the catalogue with plausible same-namespace entries — the state carries namespace→name associations.

Artifacts: probes/state-separability/ (frozen protocol, seeded corpora with SHA-256, raw probe results and generations for both context regimes), probes/state-reranker/ (centroid corpus, Kaggle/A40 kernels, free and forced generations at both scales, full margin curves including the ones that do not flatter us, results72/cv_scale.json), figures regenerated from committed JSONs by figs/make_figs_porte.py. Probe + 2.9B corrector: free Kaggle. 7.2B run: ~$1 of A40. Confirmation pre-registration and single scoring: timestamped commits (judge kept private).

What's in here

judge/heldout_v1.jsonl      82 frozen cases (mechanical labels, held out from training)
schemas/tool_schemas.json   the 231 real tool schemas of the Gungnir assistant
eval/                       evaluation scripts + precomputed prompts (verbatim)
results/                    raw generations (both runs) + scored results, all models
training/                   data generators, training data (2,623 examples), pod recipes, REPRODUCE.md
states/rwkv-0.pth           the trained initial state (11 MB) — the whole artifact
figs/make_figs.py           figures generated from the committed raw results

Verify the claim in 10 minutes (no GPU)

The raw generations of both runs are committed. Re-score them yourself:

python3 eval/score_generations.py --gen results/generations_rung1.jsonl --gold judge/heldout_v1.jsonl

Reproduce everything (~2 h, < $1)

See training/REPRODUCE.md: exact environment (RWKV-PEFT commit, pinned package versions, Docker image), SHA-256 of the base model and trained states, and the pod scripts used verbatim. Requires an Ampere-or-newer GPU (RWKV-7 training kernels are bf16/fp32 only; free Kaggle T4s cannot run them).

Note: GGUF cannot carry a trained initial state. Evaluation and deployment of state-tuned RWKV go through the native rwkv runtime (eval/eval_state_ab.py shows the state injection).

Provenance & honest limits

• The judge was frozen before any training measurement. 17 cases are hand-written; 65 were generated by a different model family (Llama 3.3) than the training data generator (Qwen3), human-reviewed, with a Jaccard anti-leak filter against the training set.
• Scaffolds differ by design: Qwen receives all 231 raw schemas (its normal serving mode); the RWKV receives a top-40 from a retriever measured at 100% recall on this judge. This is a system-vs-system comparison.
• Single task, single (real-world) benchmark. A replication on public tool sets is planned. Specialization cost on other capabilities is unmeasured.

Found a hole in the protocol? Open an issue or write to contact@scarletwolf.ai — that is what publishing this is for. The four questions we think anyone should ask a vendor announcing a number, applied to ourselves, are on the method page.

License

Apache-2.0. The tool schemas describe the open-source Gungnir assistant.