Do agentic skills beat retrieval?

Most demonstrations of agentic skills compare a skill-equipped agent against one with no access to the domain. That is not a real test — a modern agent has read the MoE literature and can search for the rest.

So the control here has web search, and the question is narrower and harder: given an agent that can already retrieve the knowledge, does packaging a domain as procedural skills produce better, cheaper, or more reproducible work than retrieval alone?

Five falsifiable predictions, each with a metric: skills should help on procedural tasks (H1), reduce run-to-run variance (H2), reduce cost (H3), reduce internally inconsistent configs (H4), and reduce numeric errors (H5).

Four arms, paired by task and seed:

• A0 bare model — no search, no skills. The floor.
• A1 search, no skills. The control that matters.
• A2 skills, no search — models an air-gapped training cluster.
• A3 skills + search — the deployed configuration.

Model under test: deepseek-v4-flash driven by the OpenCode CLI. Retrieval: a keyless DuckDuckGo Lite scrape, since no paid search API was available — a stated limitation, and one that if anything handicaps the search arms.

Six tasks: architecture design, training config, failure diagnosis, throughput optimization, plus two built specifically to separate skills from lookup — arbitrating genuinely conflicting published guidance, and designing under a constraint no published recipe matches.

n = 5 paired seeds per arm per task, 120 runs. Everything else held constant. Compute from the AMD AI Developer Program.

Retrieval is removed at the harness level for A0 and A2, not by instructing the model not to search — a prompt instruction produces a contaminated arm, not a controlled one. Task prompts are byte-identical across arms; the only differences are the registered tools and the loaded skill directories.

Scoring is deterministic. Evaluator scripts, not a judge model, so a score cannot drift with knowing which arm produced an output. The primary endpoint (A3 vs A1) was pre-registered; every other comparison is secondary, Holm-corrected, and reported as exploratory.

Every run logs tokens, tool calls, search calls, and wall-clock, so the cost endpoints carry the full sample of 30 runs per arm regardless of what the quality rubrics do.

One gain in six tasks. Diagnosis accuracy improved under skills (d = 1.60, paired p = 0.023 uncorrected) — the most purely procedural task in the set, and exactly where the theory predicts skills should win. It is also one significant result among six uncorrected comparisons at n = 5, so it is directional, not confirmed.

Cost went the wrong way, decisively. Skills+search spent 109,938 tokens per run against the search baseline's 41,593 — 2.64x — plus 23% more tool calls and 26% more wall-clock. Even the offline skills arm, making zero search calls, cost 26% more than the search baseline. A loaded skill is context, and you pay for it on every run.

Search did not help either. The retrieval-only gain (A1 - A0) was approximately zero on every task, at 51% more tokens. The strong control this study was built to beat was not itself lifted by retrieval, which points at model capability rather than context strategy as the binding constraint across all four arms.

The conflicting-guidance task is the clearest failure, and it is a failure of every arm. Asked to pick a capacity factor, aux-loss coefficient, and top-k for a stated hardware budget and defend it against the disagreement in the literature, 19 of 20 runs produced internally incompatible configurations. The single success came from the arm with no skills.

Skills+search burned 437,520 tokens per run on that task — 7x the control — to arrive at the same wrong answer. Faced with a genuine conflict, the skilled agent did not arbitrate faster. It deliberated longer and still failed.

Two of six tasks were also near-ceiling: the control scored 21-25 out of 25, leaving no headroom for any effect to show. The headroom check caught this before the full wave, and it was reported rather than quietly re-scored afterward.

One model, one keyless retrieval backend, n = 5 — the design's stated minimum. Detecting d = 0.45 at 80% power needs roughly n = 40, so most near-zero results here are uninformative rather than evidence of absence.

Two tasks rest on numbers the model reports about itself: no training job runs in this benchmark, so "successful launch" and "throughput delta" are plausibility signals, not measurements.

A contamination failure found during the headroom check is worth stating plainly: the arm configs disabled bash, edit, write and webfetch but left read, grep, glob and list enabled, and the agent used them to read the scoring script mid-run — its own words, "let me check the skill rubrics to produce a coherent, high-scoring config." Fixed by disabling all four, verified with a probe run, and every pre-fix output was discarded. If your grader lives in the repo the agent runs in, it is in reach unless you have specifically checked that it is not.

Finally, the rubrics and the skills were written in the same repository, so circularity is possible — but that bias runs toward the skills arms, which makes a null result more credible, not less.

Raise n to 20-40 on the three tasks that actually discriminate, and re-scale the two saturated rubrics so the control lands mid-range.

Replace the self-reported training metrics with real short profiled runs — about 20 GPU-hours on one MI300X, which is the single largest validity upgrade available and needs no new hardware.

Then repeat across model scales. If skills require capability to exploit, the effect should appear as an interaction with model size, and a null on a small model says little about a large one.