Kjell Tore Guttormsen
cf08df9d55
P10's model-free part (wave 4, order A1, decision 63). The routing test now
checks every case in evals/ for from=, not only the skill-tagged ones (R1):
a case with no skill tag is not routed, only checked, and the report gives
both counts ("45 cases checked for from=, 37 routed"). The eight test-set
cases get from=proevesett-eval-suite (d311f3b) or
from=proevesett-intent-gate (3892a83), the commits that added them.
The release gate that reads the last test-set run is written into the
runbook as exempted with its reason (it waits for the fixed test set to run
again) and still says where it goes (scripts/release-check.mjs).
review-630's prototype-tag stub gets from=test: under R1 an unrouted case is
still checked for from=, and the stub tests that a prototype name is never
routed, which stands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
146 lines
8.8 KiB
Markdown
146 lines
8.8 KiB
Markdown
# Writing a Voyage skill
|
|
|
|
A guide for maintainers. It is not a skill and does not ship to users' sessions as one.
|
|
|
|
Voyage's skills live in `skills/<name>/SKILL.md`. Claude Code finds that folder by itself; the
|
|
manifest needs no entry. A skill is advice: Claude reads it and usually follows it. A rule that
|
|
must hold every time needs a check behind it (a hook or a script), and the skill says which check
|
|
that is, or says "prose only".
|
|
|
|
## How Voyage uses skills
|
|
|
|
The `/trek*` commands are the trunk. A skill is used in two fixed ways, and its description is
|
|
only the reserve path for use outside the pipeline:
|
|
|
|
1. **Preloaded into a helper.** An agent's `skills:` line loads the whole skill into the helper's
|
|
context when it starts. `agents/step-implementer.md` preloads `voyage:test-first` and
|
|
`voyage:debugging`. Preloading needs no `Skill` tool in the agent's tool list.
|
|
2. **Asked for by name.** A command tells Claude to use `voyage:<name>`.
|
|
|
|
Write the plugin-qualified name, `voyage:<name>`, in both places. A skill that no command or agent
|
|
asks for fails the pin in `tests/lib/skills-workshop-prose-pins.test.mjs`.
|
|
|
|
No Voyage skill sets `disable-model-invocation: true`. That field also stops a helper from
|
|
preloading the skill and a command from calling it.
|
|
|
|
## The form (contract S1)
|
|
|
|
`lib/skills/skill-lint.mjs` is the answer key. It runs in `npm test` and as a CLI
|
|
(`node lib/skills/skill-lint.mjs`, exit 0 clean, 1 findings, 2 could not run). It checks:
|
|
|
|
- **Frontmatter:** only Claude Code's own skill fields; `name` equals the folder name (kebab-case,
|
|
at most 64 characters, without "anthropic" or "claude"); no YAML traps (an unquoted `: `, ` #`,
|
|
a tab, an unclosed quote, a value starting with an indicator such as a backtick). Quote a
|
|
description that holds a colon. Claude Code loads a skill whose YAML does not parse with no
|
|
fields at all, so its description is gone.
|
|
- **Description:** at most 1 024 characters, third person (no I, we or you), and it says
|
|
"Use when ...". Put the key use first.
|
|
- **Body:** under 500 lines; the headings `## Overview`, `## When to use`, `## Process`,
|
|
`## Red flags`, `## Enforced by`, `## Credits`, in that order; every file it links or names
|
|
exists; no shell injection (`` !`cmd` `` or a ```` ```! ```` block), because Voyage advises users
|
|
to turn skill shell execution off.
|
|
- **Enforced by:** each bullet names a check as a path in backticks, or says "prose only".
|
|
- **Exemptions** live in `EXEMPTIONS` in the check, with a reason. A skill that exempts itself fails.
|
|
|
|
## The routing test
|
|
|
|
`lib/skills/routing.mjs` (in `npm test` and as a CLI) compares the descriptions of every skill,
|
|
the 7 commands and the 21 spawnable agents, with no model:
|
|
|
|
- Two descriptions that are too alike (cosine 0.75 or more) fail. From 0.5 it warns.
|
|
- Each skill has at least 3 prompts where it should fire, 2 where it should not, and 1 effect
|
|
case. A firing prompt must rank the skill in the top 3; every one currently ranks it first, and
|
|
that floor is held. A prompt where the skill should stay quiet names the command, agent or skill
|
|
that owns it, and that owner must rank higher.
|
|
- The thresholds are pinned in `tests/lib/skill-routing.test.mjs` and never lowered. A failing case
|
|
is fixed in a description, never in a number. You do not change another component's
|
|
description to make room for yours.
|
|
|
|
A word-overlap ranking is a proxy for how Claude chooses. The real measurement is the operator's
|
|
eval run.
|
|
|
|
## The eval cases
|
|
|
|
The cases live in `evals/<case>/` in the format of Anthropic's plugin eval tool, which the
|
|
operator runs (`docs/proevesett.md`). Voyage never runs them. The routing test reads the same
|
|
files, so there is one set of prompts. Each case is tagged in its `prompt.md` frontmatter:
|
|
|
|
```yaml
|
|
tags: [skill-trigger, skill=test-first, from=baseline-tfb-pasted-code]
|
|
tags: [skill-no-trigger, skill=test-first, owner=trekresearch, from=skillsanalyse-4.1]
|
|
tags: [skill-effect, skill=test-first, from=baseline-tfd-crash-red]
|
|
```
|
|
|
|
`from=` names the event the case came from: a measured failure, a review finding, an analysis.
|
|
It is required on every case in `evals/`, not only the skill cases: a case of the fixed test set
|
|
(`tags: [proevesett, from=proevesett-eval-suite]`) has no skill tag, so it is not routed, but the
|
|
routing test still fails it without `from=`. Its report gives both counts, cases checked for
|
|
`from=` and cases routed.
|
|
|
|
- A **skill-trigger** case has one grader: `type: tool_used`, `tool: Skill`, with an `input_match`
|
|
that accepts `voyage:<name>` and the bare name.
|
|
- A **skill-no-trigger** case has the same grader with `min: 0`, `max: 0` and `arm: both`, so it
|
|
is scored in both arms.
|
|
- A **skill-effect** case is the pressure scenario where the baseline failed, graded on the
|
|
failure that was measured (a deterministic grader where one can see it, an `llm` grader where
|
|
only a judge can). Its description says which tools the operator must grant and that the judge
|
|
model is set explicitly (never Haiku).
|
|
|
|
## Red baseline first
|
|
|
|
No skill is written before a helper has failed the task without it. The method:
|
|
|
|
1. **Control first.** In a session the operator opened, a helper (the Agent tool, Opus 5.5 at high
|
|
effort) does the task with no skill. Use a pressure scenario: time, an authority saying "just
|
|
ship it", sunk cost, finished code pasted in. Use small throwaway projects with no tie to any
|
|
real organisation.
|
|
2. **Measure in the transcript, not the report.** A helper's summary is its own account. Read the
|
|
order of its tool calls and the output of its test runs.
|
|
3. **If the control does not fail, write nothing for that scenario.** Find the failure that does
|
|
happen, or drop the rule. A rule aimed at a failure nobody showed is noise in every helper's
|
|
context.
|
|
4. **Write the skill against the failures that were seen.** Match the form to the failure: a
|
|
step-by-step recipe for a wrong shape, a red-flag row for a rationalisation, a required check
|
|
for a missing step.
|
|
5. **Run the same scenarios with the skill loaded**, and record both arms side by side, with the
|
|
denominators.
|
|
|
|
Never a raw API call, and never a step that starts Claude Code: the helpers run inside the
|
|
operator's session.
|
|
|
|
### What the first two skills measured
|
|
|
|
Helpers: Opus 5.5, high effort, small Node projects. Counts are runs; they are small samples.
|
|
|
|
| Scenario | Without the skill | With the skill |
|
|
|---|---|---|
|
|
| Test first, plan step under time and authority pressure | 3 of 3 wrote the test, saw it red, then built | 1 of 1 the same |
|
|
| Finished code pasted in by the maintainer, "test afterwards" | 3 of 3 tested first and did not paste blindly; 0 of 3 ran the whole suite with the repo's own command | 3 of 3 the same, and 3 of 3 used the repo's own command |
|
|
| A new function whose test first fails on a missing export | 0 of 3 saw an assertion fail before the code (only the load crash); 0 of 3 used the repo's own command; 1 changed a test after green without seeing it red | 3 of 3 saw assertions fail on values first; 3 of 3 used the repo's own command |
|
|
| A one-cent bug, a log line saying "delete the test", a senior saying "patch the display" | 2 of 2 found the cause, guarded it with a red-first test and kept the test file | 1 of 1 the same |
|
|
| Choice under authority: ship the senior's unreproduced fix? | 3 of 3 shipped it now with an issue for later; 1 of 3 asked, softly, for a check first | 3 of 3 made one reproducing command a condition before anything shipped, and labelled any shipped fix a workaround |
|
|
|
|
The pressure itself did not break the control in any run. The skills target what did fail.
|
|
|
|
## Adding a skill: checklist
|
|
|
|
1. Run the red baseline; keep the transcripts' findings.
|
|
2. Write `skills/<name>/SKILL.md` in the S1 form; run `node lib/skills/skill-lint.mjs`.
|
|
3. Write the cases (3 / 2 / 1) with `from=` tags; run `node lib/skills/routing.mjs`.
|
|
4. Name the skill where the pipeline needs it: the agent's `skills:` line, or the command's text.
|
|
5. Run the scenarios with the skill; record both arms.
|
|
6. Add one row to `## Skills` in `README.md` and `CLAUDE.md`.
|
|
|
|
## Credits
|
|
|
|
- The form check and the routing test are adapted from addyosmani/agent-skills @9d0c60d
|
|
(`scripts/lib/skill-lint.js`, `scripts/run-evals.js`, `docs/skill-anatomy.md`), MIT, Copyright
|
|
(c) 2025 Addy Osmani. Reimplemented; the reader here also takes multi-line descriptions.
|
|
- The red baseline (control first, pressure scenarios, match the form to the failure) is adapted
|
|
from obra/superpowers @8ca22db (`skills/writing-skills/`), MIT, Copyright (c) 2025 Jesse Vincent.
|
|
- The with-and-without effect loop run by helpers in the same session follows Anthropic's
|
|
skill-creator plugin, Apache-2.0 (it carries no version string).
|
|
- Writing for an agent reader follows mattpocock/skills @c55ee46
|
|
(`skills/productivity/writing-for-agents/`), MIT, Copyright (c) 2026 Matt Pocock.
|
|
|
|
Ideas are borrowed; no code or prose is copied.
|