Tool-use post-training worked. It also broke zero-shot math.

Tool-use fine-tuning is usually scored on whether the model can emit a well-formed call. The harder policy question is whether it knows when to: call a tool, answer from what it already knows, ask for clarification, or decline because no tool fits.

So the study asked two things:

1. Can post-training improve a 2B model's tool-call policy, not just its call formatting, on a partition fixed before any result was seen?
2. What does that training do to everything else? General reasoning, instruction following, and knowledge are not in the tool-use eval, which is exactly why they need measuring.

Base model: Qwen/Qwen3.5-2B, pinned to a single revision.

Training data: the Canonical-v2 corpus, 173,237 records (161,966 trainable) normalized from Glaive, ToolACE, When2Call and xLAM.

Stages:

Tool-policy eval: a 1,277-example confirmatory partition drawn from When2Call, frozen and committed before training and scored once. It is an internal partition, not an untouched external benchmark: the wider population had been used in earlier checkpoint selection.

Capability eval: GSM8K (1,319 questions, zero-shot and 8-shot), IFEval (541 prompts), MMLU-Pro (12,032 items), run over the Base / M0 / M1-v2 ladder on H200 for $11.39 total.

Sources: configs/experiments/, reports/M0_CANONICAL_V2_FINAL_EXECUTION_REPORT.md, results/final_campaign_verdict.json.

Pre-registration. The confirmatory partition and its metrics were frozen before training. Call F1 is reported as the headline, with the caveat that it flatters a model that calls tools indiscriminately. The promotion rule was not held to the same standard; see what did not work.

Evaluation controls that separate a regression from an artifact. A capability number can fall because the model got worse, because it ran out of tokens, or because it refused. The re-scoring harness is GPU-free and works from 48,840 stored per-example records across 15 benchmark x checkpoint combinations, so each of those is counted separately:

• Refusal vs attempt accounting. A refusal is not a wrong answer; both are reported.
• Truncation bounds. Where truncation differs between checkpoints, the effect is reported as a range, not a point.
• Verbosity checks. One 768-token run turned out to measure verbosity rather than capability, and was reclassified.

Claim audit. About 990 published claims were checked against committed artifacts. 93 did not hold in the main audit and 7 more in a later site recheck; all 100 were resolved, with corrections to hash-pinned reports recorded in an errata log rather than edited in place.

Tool-call policy (1,277-example pre-registered partition, docs/EXPERIMENT_RESULTS.md):

The base model's 0.97 recall is the problem, not the achievement: it calls a tool on 62% of items that need none. Post-training traded some recall for a large drop in over-calling. Almost all of that came from SFT; DPO added +0.0078 call F1 (351 vs 344 of 453 calls), which the repository itself classes as within noise.

General capability (results/final_campaign_verdict.json):

Both tuned checkpoints refused all 1,319 zero-shot GSM8K questions: zero attempts, zero truncations, so this is refusal, not a formatting or length artifact. With eight examples in the prompt the same models answer every question again, but only 55.5% correctly against the base model's 70.4%. The campaign classes that 8-shot gap as confirmed capability loss, not refusal (refusal is 0% in that arm). So there are two effects: a learned refusal on bare questions, and a real drop in ability underneath it.

Quantization. Nine GGUF levels were built from a BF16 export. Q6_K (1.45 GiB) was selected under a rule frozen in advance, with 98.0% decision agreement against llama.cpp BF16 (25 flips in 1,277).

The promotion gate could not see the regression. It scored tool-call policy on a partition with no plain-answer examples, so a model that refuses every zero-shot GSM8K question passed it. The gate was doing its job; its job was too narrow.

M1-v2 was promoted under a gate revised after M0 was scored. It fails the earlier gate on all four development checkpoints, and its measured difference from M0 is within noise. The study records the promotion and its reason rather than presenting DPO as a measured improvement.

Applying DPO directly to the base model (M1-v1) failed on tool policy. Call F1 fell to 0.1715, and it still refused 70.7% (933 of 1,319) of zero-shot GSM8K.

The only repeat run did not reproduce. A restore of the M1-v1 run is logged as NON_REPRODUCIBLE_REPEAT instead of being counted as a second seed.

The xLAM ablations are confounded. Removing xLAM dropped call F1 to 0.6030 (fixed compute) and 0.5557 (matched exposure), but each ablation removed a data source and its whole call-prediction signal at once, and the "matched" run saw 1.19x the tokens.

One base model, one training lineage, one seed per arm, and no confidence intervals. Every gain above is a point estimate.

MMLU-Pro is the least clean comparison: the base model truncated 21.8% of items against about 6% for the tuned checkpoints, so the size of that drop is bounded rather than measured exactly.

Only 3 of 17 planned external benchmarks have run. Q6_K passed the selection rule but failed the stricter release bar, and that pass sits inside rerun noise.

Study 002 is designed and has not used any GPU time: six research questions, an 11-arm matrix, and three seeds per arm with a paired cluster bootstrap. Its data work has run: a 199,260-row normalization over four pinned sources (181,433 accepted, byte-identical across two builds) and a 581-item policy annotation set labelled by hand (single annotator, still a snapshot rather than frozen gold).