- Python 80.6%
- Shell 19.4%
| eval | ||
| figs | ||
| judge | ||
| results | ||
| schemas | ||
| states | ||
| training | ||
| LICENSE | ||
| README.md | ||
rwkv-toolcaller-bench
A 2.9B RWKV (an RNN, not a transformer) matches Qwen3-30B-A3B on real-world tool calling after $0.53 of state-tuning. This repository contains everything needed to verify or attack that claim.
Full write-up: https://scarletwolf.ai/blog/rwkv-state-tuning-parite-qwen.html (FR, EN mirror linked there). Lab: ScarletWolf AI.
Headline numbers (frozen 82-case judge, same scorer for every model)
| Category (n) | G1g 2.9B base | + state-tuning ($0.53) | Qwen3-30B-A3B |
|---|---|---|---|
| Abstention (17) | 0 | 12 | 17 |
| Argument filling (22) | 9 | 19 | 17 |
| Hard selection (23) | 12 | 16 | 14 |
| Simple selection (18) | 7 | 11 | 10 |
| Multi-step (2) | 0 | 0 | 0 |
| Total (82) | 28 (34%) | 58 (70%) | 58 (70%) |
- Paired McNemar vs base: p = 6.9e-8 (32 discordant cases won, 2 lost).
- vs Qwen: exact tie, perfectly symmetric disagreement (16-16).
- Zero false refusals on the 65 call cases; 1 unparsable output in 82, grammar-free.
- Fully replicated: a second end-to-end run (retrain + re-eval, different machine) produced the same 58/82 and byte-identical generations (164/164). Both trained states share the same SHA-256. The chain is deterministic.
Rung 3 update (2026-07-22)
Targeted data v3 (+162 purpose-built examples) and a composition probe:
| state v3 | LoRA v3 | combo (state v3 on LoRA v3) | |
|---|---|---|---|
| Abstention (17) | 16 | 17 | 17 |
| Arguments (22) | 18 | 15 | 17 |
| Hard selection (23) | 15 | 15 | 17 |
| Simple selection (18) | 12 | 12 | 8 |
| Total (82) | 61 (74%) | 59 | 59 |
Campaign champion: state v3, 61/82, byte-reproducible (states/rwkv-state-v3.pth).
LoRA v3 reaches perfect abstention (17/17, Qwen's score) but confirms the
capacity-vs-arguments trade-off. The composition probe (a state trained on the
base injected onto merged LoRA weights) does not collapse: campaign record on
hard selection (17/23) with perfect abstention, at the price of simple selection
and format degradation. Native composition is the next rung. Raw generations
and scored results for all three are in results/.
What's in here
judge/heldout_v1.jsonl 82 frozen cases (mechanical labels, held out from training)
schemas/tool_schemas.json the 231 real tool schemas of the Gungnir assistant
eval/ evaluation scripts + precomputed prompts (verbatim)
results/ raw generations (both runs) + scored results, all models
training/ data generators, training data (2,623 examples), pod recipes, REPRODUCE.md
states/rwkv-0.pth the trained initial state (11 MB) — the whole artifact
figs/make_figs.py figures generated from the committed raw results
Verify the claim in 10 minutes (no GPU)
The raw generations of both runs are committed. Re-score them yourself:
python3 eval/score_generations.py --gen results/generations_rung1.jsonl --gold judge/heldout_v1.jsonl
Reproduce everything (~2 h, < $1)
See training/REPRODUCE.md: exact environment (RWKV-PEFT commit, pinned
package versions, Docker image), SHA-256 of the base model and trained states,
and the pod scripts used verbatim. Requires an Ampere-or-newer GPU (RWKV-7
training kernels are bf16/fp32 only; free Kaggle T4s cannot run them).
Note: GGUF cannot carry a trained initial state. Evaluation and deployment of
state-tuned RWKV go through the native rwkv runtime (eval/eval_state_ab.py
shows the state injection).
Provenance & honest limits
- The judge was frozen before any training measurement. 17 cases are hand-written; 65 were generated by a different model family (Llama 3.3) than the training data generator (Qwen3), human-reviewed, with a Jaccard anti-leak filter against the training set.
- Scaffolds differ by design: Qwen receives all 231 raw schemas (its normal serving mode); the RWKV receives a top-40 from a retriever measured at 100% recall on this judge. This is a system-vs-system comparison.
- Single task, single (real-world) benchmark. A replication on public tool sets is planned. Specialization cost on other capabilities is unmeasured.
Found a hole in the protocol? Open an issue or write to contact@scarletwolf.ai — that is what publishing this is for.
License
Apache-2.0. The tool schemas describe the open-source Gungnir assistant.