SkillFlux

    ↑ ↓ select↵ openSearch all of SkillFlux
    Browse directory

    Versions

    A published version is never edited. A change in method becomes a new version, and every result links to the version it was produced under.

    1. What is measured

    Two separate things, reported separately. The static health check (level 1) reads a skill's files and reports facts and problems that can be determined without running a model. The benchmark (level 2) runs real tasks in Claude Code twice — once with the skill installed and once without — and reports whether the skill changed the outcome. A clean health check says nothing about whether a skill helps; only the benchmark addresses that.

    2. Which version is tested

    Every skill is pinned to a full 40-character Git commit SHA. The files are downloaded as the GitHub archive of that commit, so the result always describes those exact bytes. When the upstream repository changes the skill later, the page says so (the upstream-changed flag); a new commit is re-pinned and re-checked deliberately, and earlier results remain available.

    3. Level 1: static health check

    The checker (version 1.0.0) runs the checks below on the skill directory. Results are pass, warning, fail or information. Flags describe what a skill contains — bundled scripts, network calls, destructive commands — and are not verdicts: a skill that calls the network may be doing exactly what it advertises. There is no weighted overall score, because any weighting would be arbitrary.

    Check idMeaning
    skillmd.presentA SKILL.md file exists at the skill root.
    frontmatter.parseSKILL.md starts with YAML frontmatter that parses as a mapping.
    frontmatter.namename is present, at most 64 characters, lowercase letters, digits and hyphens.
    frontmatter.nameMatchesDirname matches the directory that contains SKILL.md.
    frontmatter.descriptiondescription is present and at most 1024 characters; under 50 is a warning.
    frontmatter.descriptionTriggerdescription says when to use the skill ("use when…", "when the user…").
    frontmatter.descriptionVoicedescription is written in the third person and contains no XML tags.
    frontmatter.unknownKeysFrontmatter keys outside the documented set (information only).
    body.tokensApproxApproximate token count of the SKILL.md body; above 5000 is a warning.
    body.linesLine count of the SKILL.md body; above 500 is a warning.
    refs.brokenLinksRelative links in SKILL.md point to files that exist.
    files.inventoryNumber and total size of files in the skill directory (information only).
    files.binaryNo executables or archives (.exe, .zip, .jar, …) are bundled.
    scripts.presentBundled scripts the agent may run (information only).
    scripts.networkScripts that call the network (information only).
    scripts.destructiveScripts with destructive commands such as rm -rf, sudo or chmod 777.
    secrets.scanNo strings that look like private keys or API tokens.
    instructions.riskNo instructions to override prior instructions, exfiltrate secrets or bypass safety controls.
    license.detectedA license is declared in frontmatter, the skill directory or the repository.
    repo.activityThe skill path changed within 365 days of the check, and the repository is not archived.
    pinned.driftWhether upstream changed the skill after the pinned commit (information only).

    Token counts are approximate: one token per four non-CJK characters plus one per CJK character; the warning threshold is 5000. Risk patterns were tuned to need an explicit instruction (for example "send the credentials to https://…") because broader patterns flagged ordinary technical prose. Placeholder credentials such as "ghp_your_token" are ignored. The check is re-run weekly; a record's date changes only when its result changes.

    4. Level 2: benchmark design

    • Host: Claude Code in headless mode (claude -p with stream-json output). The Claude Code version, the requested model and the model ids actually used are recorded for every run.
    • Two arms per task. They are identical — same prompt, fixtures, tools, turn limit and settings — except that the with-skill arm has the skill directory in .claude/skills/. Each trial starts from a fresh temporary project with an empty Claude configuration directory, so no user skills, plugins, memory or CLAUDE.md can leak in. Trials run inside a container.
    • Isolation is verified, not assumed: the start-up event of each trial must list the skill in the with-skill arm and must not list it in the other. A trial that fails this check is marked invalid and reported, never silently dropped.
    • Tasks are realistic requests that do not name the skill, so the model has to decide to use it. Prompts are published on each report.
    • Grading is a deterministic script that inspects the files the task produced. It never sees which arm a trial came from. Before a task is used, its grader must pass a reference solution and fail an empty workspace. Model-judged grading is allowed only when labelled and is never part of the headline pass rate.
    • Negative-control tasks are unrelated to the skill and run only with it installed; they measure how often the skill is invoked when it should not be.
    • A skill counts as invoked when the transcript contains a Skill tool call for it. Each task and arm is repeated N times (N is stated on every report), in an order shuffled with a recorded seed so the arms are interleaved over time.
    • Timeouts, invalid trials and infrastructure errors (rate limits, server errors) are excluded from n and reported in separate counts. Infrastructure errors are retried at most twice.

    5. Statistics

    • Pass rates and invocation rates are reported as k/n with a Wilson score 95% interval.
    • The difference in pass rate between arms uses the Newcombe hybrid score interval (Newcombe 1998, method 10).
    • A report says a skill improves or hurts results only when that 95% interval excludes zero; otherwise it says there was no detectable difference at that sample size. Small samples will usually show no detectable difference — that is a statement about the evidence, not a claim that the skill is useless.
    • Duration, cost and tokens are reported as medians with interquartile ranges. The headline result pools a skill's positive tasks; every task is also shown on its own.
    • The site's build recomputes every published summary from the raw trials and fails if any number disagrees.

    6. Publication and corrections

    A published run is immutable and has a stable URL. A correction is a new run that names the run it supersedes; the earlier run stays listed with its status. Measurements are released under CC BY 4.0. That licence covers SkillFlux's measurements only, not the third-party skills; where a skill's licence does not allow redistribution, only metrics are published. Raw transcripts, with home paths redacted, are attached to the run as a release asset.

    7. What the results do not show

    • Results hold for the tested commit, host, model and tasks. Other models, hosts, versions or tasks may differ.
    • Tasks are chosen because a script can grade them. Skills whose value is judgment, tone or taste are under-represented in level 2.
    • Skill authors are not consulted before publication. Disputes come in by email to hello@skillflux.app; corrections are published as a new run or noted on the page concerned.
    • SkillFlux takes no payment for listing, ordering or testing.