1. What is measured
Two separate things, reported separately. The static health check (level 1) reads a skill's files and reports facts and problems that can be determined without running a model. The benchmark (level 2) runs real tasks in Claude Code twice — once with the skill installed and once without — and reports whether the skill changed the outcome. A clean health check says nothing about whether a skill helps; only the benchmark addresses that.
2. Which version is tested
Every skill is pinned to a full 40-character Git commit SHA. The files are downloaded as the GitHub archive of that commit, so the result always describes those exact bytes. When the upstream repository changes the skill later, the page says so (the upstream-changed flag); a new commit is re-pinned and re-checked deliberately, and earlier results remain available.
3. Level 1: static health check
The checker (version 1.0.0) runs the checks below on the skill directory. Results are pass, warning, fail or information. Flags describe what a skill contains — bundled scripts, network calls, destructive commands — and are not verdicts: a skill that calls the network may be doing exactly what it advertises. There is no weighted overall score, because any weighting would be arbitrary.
| Check id | Meaning |
|---|---|
skillmd.present | A SKILL.md file exists at the skill root. |
frontmatter.parse | SKILL.md starts with YAML frontmatter that parses as a mapping. |
frontmatter.name | name is present, at most 64 characters, lowercase letters, digits and hyphens. |
frontmatter.nameMatchesDir | name matches the directory that contains SKILL.md. |
frontmatter.description | description is present and at most 1024 characters; under 50 is a warning. |
frontmatter.descriptionTrigger | description says when to use the skill ("use when…", "when the user…"). |
frontmatter.descriptionVoice | description is written in the third person and contains no XML tags. |
frontmatter.unknownKeys | Frontmatter keys outside the documented set (information only). |
body.tokensApprox | Approximate token count of the SKILL.md body; above 5000 is a warning. |
body.lines | Line count of the SKILL.md body; above 500 is a warning. |
refs.brokenLinks | Relative links in SKILL.md point to files that exist. |
files.inventory | Number and total size of files in the skill directory (information only). |
files.binary | No executables or archives (.exe, .zip, .jar, …) are bundled. |
scripts.present | Bundled scripts the agent may run (information only). |
scripts.network | Scripts that call the network (information only). |
scripts.destructive | Scripts with destructive commands such as rm -rf, sudo or chmod 777. |
secrets.scan | No strings that look like private keys or API tokens. |
instructions.risk | No instructions to override prior instructions, exfiltrate secrets or bypass safety controls. |
license.detected | A license is declared in frontmatter, the skill directory or the repository. |
repo.activity | The skill path changed within 365 days of the check, and the repository is not archived. |
pinned.drift | Whether upstream changed the skill after the pinned commit (information only). |
Token counts are approximate: one token per four non-CJK characters plus one per CJK character; the warning threshold is 5000. Risk patterns were tuned to need an explicit instruction (for example "send the credentials to https://…") because broader patterns flagged ordinary technical prose. Placeholder credentials such as "ghp_your_token" are ignored. The check is re-run weekly; a record's date changes only when its result changes.
4. Level 2: benchmark design
- Host: Claude Code in headless mode (claude -p with stream-json output). The Claude Code version, the requested model and the model ids actually used are recorded for every run.
- Two arms per task. They are identical — same prompt, fixtures, tools, turn limit and settings — except that the with-skill arm has the skill directory in .claude/skills/. Each trial starts from a fresh temporary project with an empty Claude configuration directory, so no user skills, plugins, memory or CLAUDE.md can leak in. Trials run inside a container.
- Isolation is verified, not assumed: the start-up event of each trial must list the skill in the with-skill arm and must not list it in the other. A trial that fails this check is marked invalid and reported, never silently dropped.
- Tasks are realistic requests that do not name the skill, so the model has to decide to use it. Prompts are published on each report.
- Grading is a deterministic script that inspects the files the task produced. It never sees which arm a trial came from. Before a task is used, its grader must pass a reference solution and fail an empty workspace. Model-judged grading is allowed only when labelled and is never part of the headline pass rate.
- Negative-control tasks are unrelated to the skill and run only with it installed; they measure how often the skill is invoked when it should not be.
- A skill counts as invoked when the transcript contains a Skill tool call for it. Each task and arm is repeated N times (N is stated on every report), in an order shuffled with a recorded seed so the arms are interleaved over time.
- Timeouts, invalid trials and infrastructure errors (rate limits, server errors) are excluded from n and reported in separate counts. Infrastructure errors are retried at most twice.
5. Statistics
- Pass rates and invocation rates are reported as k/n with a Wilson score 95% interval.
- The difference in pass rate between arms uses the Newcombe hybrid score interval (Newcombe 1998, method 10).
- A report says a skill improves or hurts results only when that 95% interval excludes zero; otherwise it says there was no detectable difference at that sample size. Small samples will usually show no detectable difference — that is a statement about the evidence, not a claim that the skill is useless.
- Duration, cost and tokens are reported as medians with interquartile ranges. The headline result pools a skill's positive tasks; every task is also shown on its own.
- The site's build recomputes every published summary from the raw trials and fails if any number disagrees.
6. Publication and corrections
A published run is immutable and has a stable URL. A correction is a new run that names the run it supersedes; the earlier run stays listed with its status. Measurements are released under CC BY 4.0. That licence covers SkillFlux's measurements only, not the third-party skills; where a skill's licence does not allow redistribution, only metrics are published. Raw transcripts, with home paths redacted, are attached to the run as a release asset.
7. What the results do not show
- Results hold for the tested commit, host, model and tasks. Other models, hosts, versions or tasks may differ.
- Tasks are chosen because a script can grade them. Skills whose value is judgment, tone or taste are under-represented in level 2.
- Skill authors are not consulted before publication. Disputes come in by email to hello@skillflux.app; corrections are published as a new run or noted on the page concerned.
- SkillFlux takes no payment for listing, ordering or testing.