I benchmarked my own agentic skills against plain web search

The result I did not want

I built a suite of five agentic skills for Mixture-of-Experts training work — architecture design, training setup, debugging, performance, benchmarking — then ran a controlled benchmark to find out whether they help. 120 agent runs, four arms, six tasks, five seeds each.

They mostly did not. And they cost 2.6x more tokens than doing nothing.

I am publishing it anyway, because the way most people benchmark skills would have hidden this completely.

The benchmark almost everyone runs is broken

The usual demo is: agent with skills versus agent without skills. The skilled one wins, everybody nods.

That comparison is worthless. A modern agent has read the Switch Transformer paper and can search the web for whatever it forgot. If you switch off search and then credit the difference to your skills, you measured access to information, not the value of packaging it.

So my control arm had web search enabled. Four arms total:

• A0 — bare model, no search, no skills. The floor.
• A1 — search, no skills. The real control.
• A2 — skills, no search. The air-gapped cluster case.
• A3 — skills and search. What you would actually deploy.

Primary endpoint pre-registered as A3 vs A1. Everything else — model, hardware, prompts, seeds, scoring — held constant, runs paired by task and seed. Scoring by deterministic scripts, so I could not nudge it after seeing which arm produced what.

What happened

Skills cost more, not less. My cost hypothesis said skills would be cheaper than search — load the procedure once instead of rediscovering it every run. The opposite happened, and it was not close:

Arm	tokens/run	tool calls	wall-clock
A0 bare	27,459	1.77	74 s
A1 search	41,593	2.70	87 s
A2 skills	52,278	2.03	90 s
A3 skills+search	109,938	3.33	110 s

The mechanism is obvious in hindsight: a loaded skill is context. SKILL.md and its knowledge files enter the window on every single run, and then the procedure tells the agent to go call tools and read reference material, which costs more again. "A skill loads once" is true inside a session and false in the per-run accounting that actually bills you.

This is the most solid finding in the study. It needs no rubric, it is measured on all 120 runs, and it is immune to the ceiling effects that muddied everything else.

One real win, exactly where the theory predicted. Of six tasks, one showed a substantial gain: debugging — diagnosing expert collapse, NaN, OOM, comm bottlenecks from logs (d = 1.60, p = 0.023 uncorrected). That is the most purely procedural task in the set. It asks for a ranked differential diagnosis from evidence, which is precisely the ordered decision procedure that a paper cannot give you and a skill can.

I want to believe that. It is also one significant result among six uncorrected comparisons at n = 5, so I am not going to pretend it is confirmed.

Nobody could do the hard task. Task 5 asks the agent to pick a capacity factor, aux-loss coefficient, and top-k for a stated hardware budget, and defend the choice against genuine disagreement in the literature. Internal consistency across 20 runs: 0.00 for bare, 0.20 for search, 0.00 for skills, 0.00 for skills+search.

Nineteen of twenty runs produced configs splicing together incompatible recommendations. The one success was in the no-skills arm. And skills+search burned 437,000 tokens per run on this task — 7x the control — to arrive at the same wrong place. Faced with a real conflict, the skilled agent did not arbitrate faster. It deliberated longer and still failed.

And the twist: search did not help either. A1 minus A0 was approximately zero on every task, at 51% more tokens. The strong baseline I designed the whole study to beat was not actually lifted by retrieval. Which points the finger at the model as the binding constraint on all four arms — not at the context strategy.

The bug that nearly invalidated everything

Before the full wave I ran a two-seed headroom check. One run timed out after burning 44 tool calls and 524,000 tokens, and when I read the transcript, the agent was using filesystem tools to read score_architecture.py — my own scoring script — plus a previous run's output file. Its stated reasoning: "Let me check the skill rubrics to produce a coherent, high-scoring config."

My arm configs disabled bash, edit, write, and webfetch. They left read, grep, glob, and list enabled, because those felt harmless. They are not harmless when your scorer lives in the repo the agent is running inside.

Fixed by disabling all four, verified with a probe run that produced zero tool calls, and every pre-fix output was thrown away.

If you are benchmarking agents: your grader is in the agent's reach unless you have specifically checked that it is not. One enabled read tool and you are measuring nothing.

What I would tell you if you are building skills

1. Price your skill in tokens per run, not once. Every line of SKILL.md is recurring cost. If a procedure triggers extra tool calls, it has to earn them back.
2. Skills look best on procedural tasks. Ranked diagnosis, ordered triage, "what do I check before concluding X." Not on lookup.
3. Check for headroom before you spend the budget. Two of my six tasks were saturated — the control scored 21–25 out of 25 — so no effect could show even if one existed. I caught it early and reported it rather than quietly re-scoring.
4. Verify your grader is out of reach. See above.

What this does not show

One small model. One keyless search backend. n = 5. Two tasks scored on numbers the model reports about itself, because no actual training job runs in this benchmark — that needs about 20 GPU-hours I do not currently have, and it is the single biggest upgrade available.

So this is not "skills do not work." It is: on these tasks, with this model, at this sample size, skills did not beat a search-enabled agent and cost substantially more to run. The circularity risk in my rubrics, incidentally, runs toward the skills arms — which makes the null result more credible, not less.

Full method, all four arms, per-seed numbers, threats to validity, and the failure log are in the paper.