Articles / Viewpoints and methods
12 minFor tool users

Addy Osmani's agent-skills, Tested: Installed Skills Sat Unused Until We Named Them

We ran addyosmani/agent-skills through 18 OpenCode runs on three free models. Installed but unnamed, no skill was ever called. Named in the prompt, agents wrote tests first.

Aaron HuangSenior Brand Manager

Installing addyosmani/agent-skills did not make OpenCode's free models use it. Our team ran two coding tasks on three free models, 18 runs in total. In the six runs where the skills were installed but not named, the agent called a skill zero times. In the six runs where the prompt named a skill, every agent followed it: it wrote tests first or reproduced the bug first, and the resulting money-rounding fixes were correct on all 20,000 prices we checked afterwards.

Tested on: Debian 12 (node:22-bookworm Docker container), OpenCode 1.19.5, Node.js 22.23.4, agent-skills commit 301b02b (Nov 1, 2026), Nov 12, 2026

I would install it, but I would name the skill I want in the prompt every time. For a new feature I would write "use test-driven-development"; for a bug in existing code I would name debugging-and-error-recovery. Leaving all 25 skills installed and hoping the agent picks one never happened in our runs, and it added about 3,000 tokens to every step. For a small change I can describe in one sentence, I would skip it rather than pay two to three times the tokens.

The short answer: naming the skill is what changed behavior

agent-skills is Addy Osmani's MIT-licensed collection of engineering workflow skills for coding agents. The version we tested has 25 skills, from specs and planning through testing, debugging, review, security and shipping. Each one is a SKILL.md file; the README describes them as the workflows and quality gates senior engineers use, packaged so agents follow them.

We tested whether installing the pack changes what an agent does. We set up three conditions and gave each one the same starting project and the same prompt:

Setup What we did Skill calls across both tasks
No skills Nothing installed n/a
Installed, not named npx skills add installed all 25, prompt unchanged 0 of 6 runs
Named Same install, plus one sentence at the start of the prompt: "Use the ○○ skill for this." 6 of 6 runs

Comparison diagram of the three setups: skill calls, tests written, bug-fix sweep errors and token cost for no skills, installed but not named, and named in the prompt

Every run also faced a hidden acceptance test suite that was copied into the container only after the agent finished. All 18 runs passed it. The setups differed in whether the agent left tests behind, and in whether its fix held up when we checked far more inputs than the hidden tests did.

How we ran the test

Each run started a fresh node:22-bookworm container that was deleted when the run ended, with no accounts or paid API keys. OpenCode's built-in free models did all the work:

  • opencode/deepseek-v4.1-flash-free
  • opencode/big-pickle
  • opencode/beam-free

The OpenCode config allowed file edits and shell commands inside the container, and denied web fetches and access outside the project folder. Each run used opencode run --format json, which logs every tool call, and had a 300-second timeout. Nothing timed out. Two tasks, three models and three setups made 18 runs, one per cell. We ran the three models in parallel containers, so the timings include free-tier queueing and are only a rough guide.

What the agent actually sees after you install 25 skills

In OpenCode's run log, skills show up as a tool called skill; an agent applies a skill by calling it. You can check what OpenCode found with opencode debug skill:

skills from .agents/skills: 25
built-in: 1

With no skills, the first step of each run carried 6,615 to 7,640 tokens of context. With the 25 skills installed and none called, it carried 9,671 to 10,715 tokens. The 25 SKILL.md files total 7,335 lines, so roughly 3,000 extra tokens looks like the list of names and descriptions rather than the full text. The free models report a cost of zero, so it did not matter here; on a paid API it would show up on the bill.

The pack includes a router skill, using-agent-skills, whose description says: "Use when starting a session, or when you need to decide which skill or workflow applies to the piece of work at hand." In the six installed-but-not-named runs, no agent called it.

Installing agent-skills in OpenCode and checking it worked

From the project root:

npx -y skills add addyosmani/agent-skills --agent opencode -y

All 13 installs we ran printed Installed 25 skills. The files land in .agents/skills/, along with a skills-lock.json. Each entry in it lists source, sourceType, skillPath and a computedHash, so you can see which file from which repository each skill came from. The installer ends with this line: Review skills before use; they run with full agent permissions.

Terminal output from npx skills add installing 25 skills, followed by opencode debug skill counting 25 project skills and 1 built-in skill

Our first count, opencode debug skill | grep -c ".agents/skills", returned 11. Writing the output to a file first and then counting gave 25. The piped output was cut off before the process exited.

If you only want one or two skills, the installer takes --skill. We ran --skill test-driven-development; it printed Installed 1 skill and opencode debug skill found exactly one project skill. How much context a single skill adds was outside the scope of this run.

Task 1: a new function in an empty project

The prompt asked for parseDateRange(input) in src/dateRange.js. It takes YYYY-MM-DD..YYYY-MM-DD, returns the two original strings and the number of days counting both ends, and throws on a bad format, a date that does not exist (such as 2026-02-30) or an end date before the start. The project had no test files. The hidden suite had 10 cases, including a leap-day range, a year boundary, 2026-02-30, 2027-02-29 and slash-separated dates.

Model Setup Seconds Hidden tests Skill called Tests left in the project
beam-free No skills 18 10/10 — 0
beam-free Installed, not named 42 10/10 — 0
beam-free Named 37 10/10 test-driven-development 5
deepseek-v4.1-flash-free No skills 67 10/10 — 0
deepseek-v4.1-flash-free Installed, not named 15 10/10 — 0
deepseek-v4.1-flash-free Named 114 10/10 test-driven-development 22
big-pickle No skills 11 10/10 — 0
big-pickle Installed, not named 18 10/10 — 0
big-pickle Named 19 10/10 test-driven-development 16

Without a named skill, every model got the function right and none left a test. All six checked a few inputs with throwaway node -e commands; two of them said in their reply that npm test had zero tests.

Run log for big-pickle with no skills: after writing the function, npm test reports tests 0, while the hidden suite passes 10 of 10

With test-driven-development named, the order flipped: write a test, watch it fail, then write the code. big-pickle's first implementation failed 3 of its own 16 tests, on single-day ranges and on returning the original strings. It fixed the code and reran to 16 of 16. Its closing message said the tests had caught and fixed a slice offset bug. deepseek's 22 tests, by its own summary, covered single days, month and year boundaries, leap years, bad formats, invalid dates and reversed ranges. beam-free's first file had a syntax error that npm test surfaced as SyntaxError: Invalid or unexpected token; it read the file, fixed it and passed its 5 tests.

Run log for big-pickle with test-driven-development named: the skill is called, a test file is written, the first implementation fails 3 tests, and the fix passes 16 of 16

On this task, correctness did not change. What changed is that the project ended up with 5 to 22 tests that the next person can rerun. The cost was tokens: the named runs processed 2.5 to 3.7 times as many tokens as the no-skill runs.

Task 2: two bugs in an existing checkout module

The second task is closer to everyday work. The project already had src/cart.js and two passing tests. The prompt relayed two support tickets. First, one item priced at 40 with the MINUS50 coupon came out at -10. Second, one item priced at 1.15 with SAVE10 (10 percent off) showed 1.03 instead of 1.04. The second bug comes from floating point: Node.js prints 1.15 * 0.9 as 1.035, but multiplied by 100 it is 103.49999999999999, so rounding lands on 103.

All nine runs fixed both reported cases and passed all 8 hidden tests. Without a named skill, deepseek and big-pickle still added two regression tests on their own, something none of the models did in task 1, where the project started without tests. beam-free added none in the two runs where the skill was not named.

In all three runs that named debugging-and-error-recovery, the first tool call was the skill, and the bug was reproduced before any code changed. deepseek printed:

MINUS50 40: -10
SAVE10 1.15: 1.03
raw: 1.035 103.49999999999999

big-pickle went further and ran five rounding approaches side by side to see which ones turned 1.035 into 1.04, then picked one.

The other two named runs compared fewer options. deepseek printed a naive result, its candidate fix and a string conversion side by side for five amounts, including 1.035, 1.005 and -10. beam-free tried four ways to round 1.035: Math.round, toFixed, floor plus 0.5, and rounding after scaling by 1e9. Only the last gave 1.04, and a fixed 1e-9 offset is the version it shipped.

A wider check caught bugs the hidden tests missed

The hidden tests only covered the one price from the ticket. So we applied each of the nine fixes to the original code and, in a container with networking turned off, ran a 10 percent discount on every price from 0.01 to 200.00. That is 20,000 prices, each compared with an exact calculation done in whole cents.

Sweep results: the original code was wrong on 117 prices; deepseek with no skills and with skills installed but not named was wrong on 89, beam-free with no skills on 115, and every other fix on 0

Fix Wrong prices First failure
Original code 117 1.15 became 1.03
deepseek, no skills and installed-not-named 89, 89 4.35 became 3.91, should be 3.92
beam-free, no skills 115 2.65 became 2.38, should be 2.39
All three named-skill runs 0, 0, 0 —
big-pickle's two unnamed runs, beam-free installed-not-named 0, 0, 0 —

Without a named skill, deepseek used a popular string-shift trick, Number(Math.round(Number(amount + 'e2')) + 'e-2'). It fixes 1.035, but 4.35 times 0.9 is already 3.9149999… in memory, so it still rounds the wrong way. beam-free, also unnamed, added Number.EPSILON to the amount; that value is too small to cover the error once the amount is above 1. Among the named runs, deepseek and big-pickle scaled the correction with the amount. beam-free added a fixed 1e-9, which was right on every price up to 200; larger amounts were outside the sweep.

Why naming the skill matters: how skills get triggered

In task 1, the six unnamed runs all opened with ls, cat package.json or find, then wrote the code. None called skill. In the named runs, the skill call came first or second.

These results apply to OpenCode 1.19.5. Claude Code installs the pack as a plugin and may trigger skills differently, so it needs its own check.

When agent-skills is worth installing

Situation What we would do
New feature, and you want tests left behind Install it and name test-driven-development
Bug in existing code, especially money or date math Install it and name debugging-and-error-recovery
Project already has tests and the change is small You can skip naming; two of three models added regression tests anyway
Installing everything and waiting for the agent to choose It never happened in our runs, and it adds about 3,000 tokens per step
Paid API and token-sensitive Name a skill only for tasks that need it; named runs used 1.5 to 3.7 times the tokens

What this test does not show

Each model ran each setup once. Free models give different results on the same prompt, so the tables are a record of these runs, not averages. Timings were taken with three containers running at once and include queueing.

The scope was three free OpenCode models and the two skills we named. Paid models such as Claude or GPT, the Claude Code plugin install, the Codex integration and the other 23 skills, including spec, code review and security, were all outside it. Larger projects and multi-turn tasks may behave differently.

Reproducing this test yourself

You can rerun the whole comparison without an account or a paid key. Our 18 runs, the sweep and the screenshots took about 40 minutes with Docker.

  1. Start a throwaway container with docker run --rm -it node:22-bookworm bash, then install OpenCode with npm i -g opencode-ai@1.19.5.
  2. Create a project folder, run git init, and add a package.json whose test script is node --test. For task 2, add the buggy src/cart.js and its two tests before you commit.
  3. Add an opencode.json that allows edits and shell commands and denies web fetches and external folders, so the agent can run tests but cannot reach outside the project.
  4. For the installed setups, run the npx skills add command above, then confirm the count with opencode debug skill, writing the output to a file before counting.
  5. Run opencode run --format json --model opencode/<model> "<prompt>" and save the JSON. Every tool call, including any skill call, appears in that log.
  6. Only after the run ends, copy your own acceptance tests into the container and run them. We kept ours hidden so every setup faced the same unseen check.
  7. For anything involving money or dates, sweep a wide range of inputs against an exact calculation. A single-case test is what let three of our unnamed fixes look finished.

We started a new container for every run, so no files or session state carried over between setups.

What the two named skills actually ask for

test-driven-development tells the agent to check package.json, the README and existing tests before writing any, then go red, then green, then refactor. For bug fixes it describes a "Prove-It Pattern": reproduce the bug with a test before attempting a fix. It says not to assume npm test is the right command in every project. In the three named task 1 runs, the agents read README.md before writing anything and then used npm test, the command the project declared.

debugging-and-error-recovery lists six steps: stop, preserve evidence, diagnose, fix the root cause, guard against recurrence, and resume only after verification. That is close to what the named runs did in task 2: each one printed the failing values before editing, and each one added tests for both tickets. The pair was the same in all three runs, under slightly different names: MINUS50 on one item priced at 40 must total 0, and SAVE10 on one item priced at 1.15 must total 1.04. beam-free called the first one "MINUS50 no negative"; big-pickle called it "MINUS50 never yields a negative total".

The test-driven-development description says to use the skill when fixing a bug "even when the code is pasted inline and the request does not mention tests." Our task 2 prompt asked for a bug fix without mentioning tests, and with the skill installed but not named, no agent picked it up.

FAQ

How can I tell whether the agent used a skill? Run it with opencode run --format json. Every tool call in the log has a "tool" field; a skill call shows "tool":"skill", and state.input.name holds the skill name.

Does installing all 25 slow the agent down? Run times with the skills installed went up in some runs and down in others, with no clear pattern. The roughly 3,000 extra tokens of context on every step was consistent.

Do skills get more permissions than the agent? No. The installer says skills run with the agent's existing permissions. If the agent can run shell commands, a skill can tell it to, so read each SKILL.md before installing.

Related on Gwarket: What an AI skill is, OpenCode free models tested, Skills we recommend.

Sources: addyosmani/agent-skills (commit 301b02b, Nov 1, 2026), Addy Osmani's write-up. Last checked Nov 12, 2026.

Subscribe to AI tool picks and practical advice

Get worthwhile tool picks, test findings and important updates in your inbox.

Get a monthly-updated list of AI tools and Skills right away. We email when there are worthwhile tool picks, test findings or important updates. Unsubscribe anytime. Privacy

Almost done: check your inboxClick the button in the confirmation email and you will get the download link for the tool list. If you do not see it, check your spam or Promotions folder.

Or follow us on: