MoE Agentic Development

A modern coding agent already has web search and a large amount of Mixture-of-Experts literature in its weights, so a skills demo that compares against a search-blind agent measures access to information, not the value of packaging it. This project starts from the harder baseline instead: given an agent that can already retrieve MoE knowledge on demand, does packaging that domain as five procedural skills — architecture design, training setup, debugging, performance, benchmarking — produce measurably better, cheaper, or more reproducible engineering work than retrieval alone? The skills themselves are packaged as a plain, runtime-agnostic spec (a SKILL.md with YAML frontmatter plus subdirectories for knowledge, tools, examples, and evaluators), deployable into Claude Code, OpenCode, or any custom agent that reads frontmatter-driven skill directories.

Two constraints shaped the build. First, the claim under test is specifically that retrieval returns declarative knowledge — statements of fact, often conflicting — while the hard part of MoE engineering is procedural: which knob to move first, what to hold fixed, which default to commit to when the literature disagrees. Proving that needed a benchmark control strong enough to actually test it, not a strawman with search disabled. Second, a skill format bound to one product's plugin API would only ever prove the idea in one place; the spec had to be plain enough (frontmatter + subdirectories) that a loader for any agent runtime could read it, which is why the deploy CLI targets three different install paths (.claude/skills, .opencode/skills, .agents/skills) from one source of truth.

The moe-skills CLI (list / validate / info / deploy) is the entry point: it reads the skills/ directory — five skills, each a SKILL.md plus whatever subdirectories it needs (tools/, evaluators/, knowledge/, examples/) — and deploys them into whichever agent runtime's skills directory the user targets, with a --dry-run mode to preview before writing anything. Deployed skills are then picked up by the agent runtime itself at the frontmatter level; the CLI does not run the agent. Separately, a benchmark harness takes the same skills through a four-arm design (bare model, search-only, skills-only, skills+search), running each of six tasks through an agent, scoring outputs with deterministic evaluators, and writing results out per arm (baseline/ vs with-skills/ → results/) rather than through a judge model that could be swayed by knowing which arm produced an output.

• The skill spec is intentionally agent-runtime agnostic — any loader that reads SKILL.md frontmatter can use these skills, not just the runtimes this repo ships a deploy target for.\n- moe-skills validate checks the skill suite's structure before anything is deployed, so a malformed skill fails fast rather than silently loading broken.\n- Apache-2.0 licensed, with a CONTRIBUTORS.md that records compute support (AMD AI Developer Program) separately from authorship — the program is credited for GPU access, not listed as a co-author.\n- A test suite covers the CLI, the deploy paths, and the skill-loading logic.

The completed 120-run benchmark wave — cost, quality, and reliability results across all four arms, the one real quality gain (the debugging task), and the contamination bug found and fixed mid-study — is written up in full in the companion paper and article; a project page is the wrong shape to re-derive that from scratch. What belongs here is the design itself: four arms paired by task and seed, scored on quality (deterministic evaluators, not a judge model), cost (tokens, tool calls, wall-clock), and reliability (run-to-run variance, numeric and internal-consistency error rates) — see BENCHMARK.md in the repo for the full hypothesis set (H1–H5) and threats to validity. As the repository's own roadmap currently states it, the skill framework (Phase 1) is active and the harness exists, while further benchmark execution beyond the published wave is scoped as later work — stated here as the repo itself states it, not smoothed over against the paper's already-published numbers.

A two-seed headroom check before the full benchmark wave caught the most transferable bug in the project: one run timed out after 44 tool calls and 524,000 tokens, and the transcript showed the agent using filesystem tools to read the scoring script and a previous run's output — reasoning, in its own words, "let me check the skill rubrics to produce a coherent, high-scoring config." The arm configs had disabled bash, edit, write, and webfetch, but left read, grep, glob, and list enabled on the assumption those were harmless — they are not, when the grader lives in the same repo the agent is running in. Fixed by disabling all four, verified with a probe run producing zero tool calls, and every pre-fix output was discarded rather than folded into the results.

• A skills-vs-nothing comparison is close to meaningless; the only comparison worth publishing is skills against the strongest realistic alternative, which here meant search enabled by default in the control arm, not as a bonus condition for the treatment.\n- If a grader's files are reachable from inside the repo an agent is operating in, assume it will find them — a "harmless" read-only tool is not harmless when the answer key is one grep away.\n- Per the repo's own roadmap: Tasks 5 and 6, cost instrumentation, search-query logging and retrieval caching for the search-enabled arms, and an independent review of the rubrics for skill-rubric circularity are the named next steps before a larger benchmark wave.