merge: main (7a skills workshop, 9 rettepakke, 7b researching-sources, 8 release runbook) into wave2-kobling
Union lines resolved: CLAUDE.md command table keeps main's /trekresearch and /trekplan rows and this branch's /trekexecute row. README's architecture block now counts the three skills main ships (it said 2 after 7b added researching-sources). Contracts run against the real ones on the merged tree: K10/K11 (plan validator refuses a piped Regression:, warns on an unknown On failure:; regression-check passes/fails/absent/unrunnable), S2 (skills/test-first and skills/debugging exist, voyage:<name> form as 7a chose), K7 (real review_version 1.1 WARN -> 0, a BLOCK copy -> 1, a 1.0 review with no verdict -> 2). npm test: 2102 / 2100 pass / 0 fail / 2 skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
3c18fdb5f7
83 changed files with 4293 additions and 71 deletions
|
|
@ -266,6 +266,9 @@ prose form (relaxed in S26 so the template's prose emission parses).
|
|||
- the optional step fields `Interface in:`, `Interface out:`, `Review focus:` are never empty when present (consumers tolerate their absence);
|
||||
- `ceremony` is one of the three classes; an `architectural` plan has `## Design` with the subsections Architecture, Data Flow, Interfaces, Data Models, Error Handling, Module Map, Architecture Rules, Requirements Check, plus `## Test Strategy`; the module map (mermaid) has no cycle; every architecture rule says "checked by `<command>`" (with a Proof column naming an existing step) or "advisory — `.claude/rules/<file>.md`".
|
||||
6. **`## Critic Verdict`** (written by `lib/validators/plan-gate.mjs --record`, never by hand): one fenced `json` block, schema `voyage.plan-verdict/1` — `round`, `max_rounds` (3), `extra_rounds`, `plan_sha256`, `verdict` (`BLOCK` | `REVISE` | `PASS`), `counts`, `findings` (deduplicated, each with `severity`). `plan_sha256` hashes the plan minus its review log (the `> **Plan quality:` line and the sections Critic Verdict, Revisions, Plan Quality Score, Adversarial Pass 2), with CRLF and line-end whitespace normalized. The verdict sits inside the plan because Claude Code's plan approval hands a hook the plan file's text and nothing beside it.
|
||||
7. **Every plan_version, no version bump** (a plan without the field validates unchanged; voyage-lekerepo trial 2, 2026-10-08):
|
||||
- **K10 — `Regression:`** (optional step field): ONE backticked command the criteria runner's allowlist runs, with no pipe and no `&&` (first backticked span, as for `Verify:`). The executor runs it green before the step's checkpoint; a step without it runs nothing new. `Verify:` stays the step's own red-first test. One that can never run is `PLAN_STEP_REGRESSION_NOT_RUNNABLE` (an error; a warning under `--soft`).
|
||||
- **K11 — `On failure:`** opens with one of `revert`, `retry`, `skip`, `escalate` (case-insensitive; a leading `**`, `_`, backtick or `-` is ignored, other leading characters are not; `onFailureAction` in `lib/parsers/plan-schema.mjs`), the words `/trekexecute` reads; any other opening is the warning `PLAN_STEP_ON_FAILURE_UNKNOWN`. A missing `On failure:` is unchanged (the executor defaults to `escalate`).
|
||||
|
||||
**Validation strategy:**
|
||||
|
||||
|
|
@ -295,6 +298,8 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
|
|||
| `PLAN_MODULE_MAP_EMPTY` / `PLAN_MODULE_CYCLE` | 1.8 architectural: no mermaid module map, or a cycle in it (all `mermaid` blocks of `### Module Map` form one map) | Draw it; break the cycle |
|
||||
| `PLAN_MODULE_MAP_UNPARSED` | 1.8 architectural: a module-map line that looks like a link but gives no edge | Write it as `a --> b` |
|
||||
| `PLAN_ARCH_RULE_NO_ENFORCER` / `PLAN_ARCH_RULE_NO_PROOF` | 1.8 architectural: a rule without an enforcer, or a checked rule without an existing proving step | Name the enforcer; name the step |
|
||||
| `PLAN_STEP_REGRESSION_NOT_RUNNABLE` | K10, any version: a step's `Regression:` is not one command the allowlist runs (a pipe, `&&`, no backticks, an unknown runner) | One backticked runner call (`npm test`); a wider check goes in `bash tests/<script>.sh` |
|
||||
| `PLAN_STEP_ON_FAILURE_UNKNOWN` | K11, any version, warning: `On failure:` does not open with revert / retry / skip / escalate | Open with the word, then the specifics |
|
||||
|
||||
**The plan gate (K3).** `node lib/validators/plan-gate.mjs <plan.md> [--json]` decides whether a plan may go on: **0** = it validates strict, its `## Critic Verdict` matches the current text, no BLOCKER is open, and every open MAJOR was accepted by the operator; **1** = stop, with `reasons` (`PLAN_GATE_INVALID_PLAN`, `PLAN_GATE_VERSION` — below plan_version 1.8, `PLAN_GATE_NO_VERDICT`, `PLAN_GATE_VERDICT_MALFORMED`, `PLAN_GATE_VERDICT_STALE`, `PLAN_GATE_OPEN_BLOCKER`, `PLAN_GATE_ROUNDS_EXHAUSTED`, `PLAN_GATE_OPEN_MAJOR` — rounds left: revise, `PLAN_GATE_OPEN_MAJOR_FINAL` — after the last round: ask the operator) and `warnings` (`PLAN_GATE_OPEN_MAJOR` for majors the operator accepted); **2** = could not read the plan. `--hash <plan.md>` prints the operative hash; Phase 9 takes it before the reviewers read the plan. `--record <plan.md> --reviewed <hash>` reads plan-review-dedup output on stdin (which lists the reviewers that delivered, `sources`) and writes the next round; exit 2 when a reviewer is missing or the plan changed since `--hash`; it refuses round 4 (exit 1, stop and ask) unless `--extra-round` follows the operator's yes, and refuses that flag before the cap is spent. Unrated findings count as blockers. The review log is excluded from the hash only up to its first other heading, so a step placed under it is hashed. `--accept-open <plan.md>` records the operator's "build as is" for the current round's open MAJORs (`accepted_open` in the verdict block, bound to the round and the plan hash; only after the last round; never for a BLOCKER; exit 1 when rounds are left, the verdict is stale, a blocker is open or no major is). Operator decision 2026-10-08: open majors used to be warnings, and a weakness the critics named was built and came back from `/trekreview` as MAJOR. plan-critic and scope-guardian rate on the review catalogue's tiers. The verdict is a trace of the review, not a signature.
|
||||
|
||||
|
|
|
|||
237
docs/release-runbook.md
Normal file
237
docs/release-runbook.md
Normal file
|
|
@ -0,0 +1,237 @@
|
|||
# Release and rollback runbook
|
||||
|
||||
How a Voyage release goes out, and how it is rolled back. The rollback part is
|
||||
not theory: every command in § 3 was run in a drill — a release 6.2.0 → 6.2.1
|
||||
and both rollback paths, in a sandbox with local bare repositories as the
|
||||
remotes — and `tests/lib/release-prose-pins.test.mjs` holds § 3 to the commands
|
||||
recorded there (`tests/fixtures/release-drill/rollback-commands.txt`).
|
||||
|
||||
A release is a two-repo act. The plugin repo (this one) gets a tag `vX.Y.Z`;
|
||||
the catalog repo pins each plugin by that tag (`source.ref` in
|
||||
`.claude-plugin/marketplace.json`), so users get the release only when the
|
||||
catalog ref moves. The catalog's `scripts/release-plugin.mjs` finds the plugin
|
||||
as a sibling directory (`../voyage`), and both repos are run from their root.
|
||||
|
||||
Placeholders: `<previous-version>` is the release before (6.2.0 in the drill),
|
||||
`<withdrawn-version>` the one rolled back (6.2.1), `<next-patch>` the next
|
||||
unused patch number (6.2.2). `<release-commit>` is the release commit,
|
||||
`<merge-commit>` a delivery merged since the previous tag, `<revert-commit>`
|
||||
the last commit of the revert, and `<review.md>` the review of what is
|
||||
released.
|
||||
|
||||
## Door types
|
||||
|
||||
Every delivery in a release says whether it is a **one-way door** or a
|
||||
**two-way door**:
|
||||
|
||||
- **one-way** — cannot be walked back without a cost to users. A break in a
|
||||
public contract (the brief format of Handover 1, a `plan_version` or
|
||||
`review_version` that rejects files that used to validate), a removed command
|
||||
or flag, anything users must migrate. A published tag is one-way too: it is
|
||||
never deleted or moved.
|
||||
- **two-way** — a revert restores the old behaviour and nobody has to do
|
||||
anything: a new optional field, a new check, a fix, documentation.
|
||||
|
||||
At the release, the CHANGELOG section for the version carries one line per
|
||||
delivery under `### Doors`:
|
||||
|
||||
```markdown
|
||||
## v6.3.0 — 2026-10-15 — <title>
|
||||
|
||||
### Doors
|
||||
|
||||
- two-way: release check and runbook — a revert removes them, nothing reads them.
|
||||
- one-way: <delivery> — <what a user must do, and from which version>.
|
||||
```
|
||||
|
||||
The release is one-way if any line is. A one-way line says what users must
|
||||
do. `scripts/release-check.mjs` refuses a release whose section has no
|
||||
`### Doors` list, an empty one, or a line that is neither `- one-way:` nor
|
||||
`- two-way:`.
|
||||
|
||||
## 1. Before the release
|
||||
|
||||
1. **Every delivery is merged into `main`** (`git merge --no-ff <branch>`), in
|
||||
the agreed order. Each delivery's report states its door type.
|
||||
2. **The suite is green on the merged tree.** `npm test` on `main` with every
|
||||
delivery in it: 0 failed. A run that goes red only under heavy machine load
|
||||
is still red — run it again on a quiet machine, never release over it.
|
||||
Worktrees and branches stay until the release is out (§ 2.6): if the merged
|
||||
tree is red, the branches are where the fix starts.
|
||||
3. **A review of what is released, and its verdict is not BLOCK.** `/trekreview`
|
||||
over the commits since the previous tag writes `<review.md>`; the verdict
|
||||
gate (K7) decides on it:
|
||||
|
||||
```sh
|
||||
node lib/review/verdict-gate.mjs <review.md>
|
||||
```
|
||||
|
||||
0 = ALLOW or WARN, 1 = BLOCK, 2 = could not decide. Only 0 goes on.
|
||||
4. **The release commit.** It changes only the version and the release notes:
|
||||
`.claude-plugin/plugin.json`, `package.json`, `package-lock.json` (both
|
||||
`version` fields), the README version badge, and a new CHANGELOG section
|
||||
`## vX.Y.Z — <date> — <title>` with its `### Doors` list. Commit it as
|
||||
`chore(release): X.Y.Z — <title>`.
|
||||
5. **The release check.**
|
||||
|
||||
```sh
|
||||
node scripts/release-check.mjs --review <review.md>
|
||||
```
|
||||
|
||||
It checks, in order: the working tree is clean (otherwise the suite would
|
||||
not measure HEAD), `npm test` exits 0 at HEAD, the review verdict is not
|
||||
BLOCK, the review covers HEAD (its `scope_sha_end` is HEAD or an ancestor,
|
||||
and the commits after it touch only the release bookkeeping above), and the
|
||||
door type is set. 0 = ready, 1 = not ready (each failure listed), 2 = could
|
||||
not run — never green.
|
||||
6. **The operator's yes, logged.** The operator reads the release section and
|
||||
says yes (an answer like "sounds good" is not a yes). The check prints the
|
||||
command that logs it (K2):
|
||||
|
||||
```sh
|
||||
node lib/util/approval-log.mjs append --kind release --subject CHANGELOG.md --sha256-of CHANGELOG.md
|
||||
```
|
||||
|
||||
The record is bound to the CHANGELOG as approved: if the release notes
|
||||
change afterwards, `verify` reports `mismatch`. In this repo `.claude/` is
|
||||
ignored by git (the remote is a public mirror), so the log stays local and
|
||||
`verify` reports the record as not committed. That is the trace this repo
|
||||
keeps; a target repo that tracks `.claude/voyage/` commits it.
|
||||
|
||||
## 2. The release
|
||||
|
||||
1. **Push the plugin last of all its steps:**
|
||||
|
||||
```sh
|
||||
git push origin main
|
||||
```
|
||||
|
||||
Voyage's push guard asks first; a yes to a push to `main` is logged as a
|
||||
`merge` approval.
|
||||
2. **In the catalog, the operator's one-shot push token.** `release-plugin.mjs`
|
||||
pushes from inside node, where the command-text push guard cannot see it, so
|
||||
it requires the same token the guard checks:
|
||||
|
||||
```sh
|
||||
mkdir -p "$HOME/.claude/runtime/push-approvals" && touch "$HOME/.claude/runtime/push-approvals/$(pwd | sed 's|/|_|g')"
|
||||
```
|
||||
|
||||
One token covers every push of ONE run. A second run needs a new token.
|
||||
3. **Dry run.** It must stop on the missing tag and nothing else:
|
||||
|
||||
```sh
|
||||
node scripts/release-plugin.mjs voyage
|
||||
```
|
||||
|
||||
4. **The release, in one run.** `FORGEJO_TOKEN` must be exported (the script
|
||||
says how when it is missing):
|
||||
|
||||
```sh
|
||||
node scripts/release-plugin.mjs voyage --create-tag --write --commit --push
|
||||
```
|
||||
|
||||
It refuses unless `plugin.json`, the README badge and the version agree.
|
||||
It mints and pushes the tag, runs the catalog's version gate before
|
||||
writing, bumps the ref and the catalog README label, runs the gate again,
|
||||
commits and pushes the catalog, and files the Forgejo release object with
|
||||
the CHANGELOG section as its text.
|
||||
5. **The catalog's own check is green:**
|
||||
|
||||
```sh
|
||||
node scripts/check-versions.mjs
|
||||
```
|
||||
|
||||
6. **Cleanup comes last.** Merged worktrees and branches are removed only now,
|
||||
after the suite was green on the merged tree (§ 1.2) and the release is out.
|
||||
|
||||
## 3. Rollback
|
||||
|
||||
The operator decides whether a release is rolled back. There are two paths,
|
||||
and both were drilled. **A** takes the catalog back to the previous tag at
|
||||
once. **B** ships the revert as a new patch version, so the version moves
|
||||
forward again. The recommended order is A, then B.
|
||||
|
||||
### Path A — revert, and the catalog ref back
|
||||
|
||||
`release-plugin.mjs` accepts a ref backwards only when the plugin agrees with
|
||||
it. Measured in the drill: with `plugin.json` at `<withdrawn-version>`,
|
||||
`--version <previous-version>` is refused ("plugin.json version is 6.2.1,
|
||||
asked to release 6.2.0 — bump plugin.json first"). So the plugin is reverted
|
||||
first, and the release commit's revert puts `plugin.json` and the badge back.
|
||||
|
||||
In the plugin repo, revert the release commit, then each delivery merged since
|
||||
the previous tag, newest first (`-m 1` keeps `main` as the mainline), and push:
|
||||
|
||||
```sh
|
||||
git revert --no-edit <release-commit>
|
||||
git revert --no-edit -m 1 <merge-commit>
|
||||
git push -q origin main
|
||||
```
|
||||
|
||||
In the catalog — between these steps the version gate reports a WARN (the ref
|
||||
is ahead of `plugin.json`), not an ERROR:
|
||||
|
||||
```sh
|
||||
node scripts/check-versions.mjs
|
||||
node scripts/release-plugin.mjs voyage --version <previous-version>
|
||||
mkdir -p "$HOME/.claude/runtime/push-approvals" && touch "$HOME/.claude/runtime/push-approvals/$(pwd | sed 's|/|_|g')"
|
||||
node scripts/release-plugin.mjs voyage --version <previous-version> --write --commit --push
|
||||
node scripts/check-versions.mjs
|
||||
```
|
||||
|
||||
The dry run must say `v<withdrawn-version> -> v<previous-version> [READY]`.
|
||||
|
||||
What path A does not do:
|
||||
- The tag `v<withdrawn-version>` stays, and so does its Forgejo release object;
|
||||
the version number is used up. Edit the release object's text to say the
|
||||
version is withdrawn (not drilled: it is the one step that needs the real
|
||||
Forgejo instance).
|
||||
- Whether a user who already updated to `<withdrawn-version>` is moved back to
|
||||
`<previous-version>` by a plugin update is **not verified**. Path B does not
|
||||
depend on it.
|
||||
|
||||
### Path B — a forward patch that carries the revert
|
||||
|
||||
On top of the revert, make a release commit for `<next-patch>` as in § 1.4,
|
||||
with a CHANGELOG section that says `<withdrawn-version>` is withdrawn and a
|
||||
`### Doors` line (`- two-way: withdraws <withdrawn-version> — …`). Its review
|
||||
covers the revert, so `scope_sha_end` is `<revert-commit>`. Then the release as
|
||||
usual:
|
||||
|
||||
```sh
|
||||
git commit -q -am "chore(release): <next-patch> - withdraws <withdrawn-version>"
|
||||
node scripts/release-check.mjs --review <review.md>
|
||||
node lib/util/approval-log.mjs append --kind release --subject CHANGELOG.md --sha256-of CHANGELOG.md
|
||||
git push -q origin main
|
||||
mkdir -p "$HOME/.claude/runtime/push-approvals" && touch "$HOME/.claude/runtime/push-approvals/$(pwd | sed 's|/|_|g')"
|
||||
node scripts/release-plugin.mjs voyage --create-tag --write --commit --push
|
||||
node scripts/check-versions.mjs
|
||||
```
|
||||
|
||||
The first four lines run in the plugin repo, the last three in the catalog.
|
||||
|
||||
### Bringing a reverted delivery back
|
||||
|
||||
Do not merge its branch again. git's own documentation for `revert -m`:
|
||||
"Reverting a merge commit declares that you will never want the tree changes
|
||||
brought in by the merge. As a result, later merges will only bring in tree
|
||||
changes introduced by commits that are not ancestors of the previously
|
||||
reverted merge." Revert the revert commit instead, and release it as a new
|
||||
version.
|
||||
|
||||
## Not built yet: the eval gate (P10)
|
||||
|
||||
The local gate before a release that reads the date and score of the last run
|
||||
of the fixed test set and compares it with the last change in the
|
||||
configuration paths (commands, agents, skills, hooks, and a changed model alias
|
||||
counts as a change). It waits for the test set to run again; a gate that can
|
||||
never fire is not built. When it comes, it is one more check in
|
||||
`scripts/release-check.mjs`, before the door check, and one more step in § 1.
|
||||
|
||||
## Credits
|
||||
|
||||
- The runbook form: [bashebr/ai-native-sdlc](https://github.com/bashebr/ai-native-sdlc) (MIT).
|
||||
- Green on the merged tree before cleanup: [obra/superpowers](https://github.com/obra/superpowers) (MIT), at `8ca22db`.
|
||||
- One-way and two-way doors: [mattpocock/skills](https://github.com/mattpocock/skills), at `c55ee46`.
|
||||
|
||||
Ideas only; no text is copied.
|
||||
142
docs/skills-authoring.md
Normal file
142
docs/skills-authoring.md
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
# Writing a Voyage skill
|
||||
|
||||
A guide for maintainers. It is not a skill and does not ship to users' sessions as one.
|
||||
|
||||
Voyage's skills live in `skills/<name>/SKILL.md`. Claude Code finds that folder by itself; the
|
||||
manifest needs no entry. A skill is advice: Claude reads it and usually follows it. A rule that
|
||||
must hold every time needs a check behind it (a hook or a script), and the skill says which check
|
||||
that is, or says "prose only".
|
||||
|
||||
## How Voyage uses skills
|
||||
|
||||
The `/trek*` commands are the trunk. A skill is used in two fixed ways, and its description is
|
||||
only the reserve path for use outside the pipeline:
|
||||
|
||||
1. **Preloaded into a helper.** An agent's `skills:` line loads the whole skill into the helper's
|
||||
context when it starts. `agents/step-implementer.md` preloads `voyage:test-first` and
|
||||
`voyage:debugging`. Preloading needs no `Skill` tool in the agent's tool list.
|
||||
2. **Asked for by name.** A command tells Claude to use `voyage:<name>`.
|
||||
|
||||
Write the plugin-qualified name, `voyage:<name>`, in both places. A skill that no command or agent
|
||||
asks for fails the pin in `tests/lib/skills-workshop-prose-pins.test.mjs`.
|
||||
|
||||
No Voyage skill sets `disable-model-invocation: true`. That field also stops a helper from
|
||||
preloading the skill and a command from calling it.
|
||||
|
||||
## The form (contract S1)
|
||||
|
||||
`lib/skills/skill-lint.mjs` is the answer key. It runs in `npm test` and as a CLI
|
||||
(`node lib/skills/skill-lint.mjs`, exit 0 clean, 1 findings, 2 could not run). It checks:
|
||||
|
||||
- **Frontmatter:** only Claude Code's own skill fields; `name` equals the folder name (kebab-case,
|
||||
at most 64 characters, without "anthropic" or "claude"); no YAML traps (an unquoted `: `, ` #`,
|
||||
a tab, an unclosed quote, a value starting with an indicator such as a backtick). Quote a
|
||||
description that holds a colon. Claude Code loads a skill whose YAML does not parse with no
|
||||
fields at all, so its description is gone.
|
||||
- **Description:** at most 1 024 characters, third person (no I, we or you), and it says
|
||||
"Use when ...". Put the key use first.
|
||||
- **Body:** under 500 lines; the headings `## Overview`, `## When to use`, `## Process`,
|
||||
`## Red flags`, `## Enforced by`, `## Credits`, in that order; every file it links or names
|
||||
exists; no shell injection (`` !`cmd` `` or a ```` ```! ```` block), because Voyage advises users
|
||||
to turn skill shell execution off.
|
||||
- **Enforced by:** each bullet names a check as a path in backticks, or says "prose only".
|
||||
- **Exemptions** live in `EXEMPTIONS` in the check, with a reason. A skill that exempts itself fails.
|
||||
|
||||
## The routing test
|
||||
|
||||
`lib/skills/routing.mjs` (in `npm test` and as a CLI) compares the descriptions of every skill,
|
||||
the 7 commands and the 21 spawnable agents, with no model:
|
||||
|
||||
- Two descriptions that are too alike (cosine 0.75 or more) fail. From 0.5 it warns.
|
||||
- Each skill has at least 3 prompts where it should fire, 2 where it should not, and 1 effect
|
||||
case. A firing prompt must rank the skill in the top 3; every one currently ranks it first, and
|
||||
that floor is held. A prompt where the skill should stay quiet names the command, agent or skill
|
||||
that owns it, and that owner must rank higher.
|
||||
- The thresholds are pinned in `tests/lib/skill-routing.test.mjs` and never lowered. A failing case
|
||||
is fixed in a description, never in a number. You do not change another component's
|
||||
description to make room for yours.
|
||||
|
||||
A word-overlap ranking is a proxy for how Claude chooses. The real measurement is the operator's
|
||||
eval run.
|
||||
|
||||
## The eval cases
|
||||
|
||||
The cases live in `evals/<case>/` in the format of Anthropic's plugin eval tool, which the
|
||||
operator runs (`docs/proevesett.md`). Voyage never runs them. The routing test reads the same
|
||||
files, so there is one set of prompts. Each case is tagged in its `prompt.md` frontmatter:
|
||||
|
||||
```yaml
|
||||
tags: [skill-trigger, skill=test-first, from=baseline-tfb-pasted-code]
|
||||
tags: [skill-no-trigger, skill=test-first, owner=trekresearch, from=skillsanalyse-4.1]
|
||||
tags: [skill-effect, skill=test-first, from=baseline-tfd-crash-red]
|
||||
```
|
||||
|
||||
`from=` names the event the case came from: a measured failure, a review finding, an analysis.
|
||||
|
||||
- A **skill-trigger** case has one grader: `type: tool_used`, `tool: Skill`, with an `input_match`
|
||||
that accepts `voyage:<name>` and the bare name.
|
||||
- A **skill-no-trigger** case has the same grader with `min: 0`, `max: 0` and `arm: both`, so it
|
||||
is scored in both arms.
|
||||
- A **skill-effect** case is the pressure scenario where the baseline failed, graded on the
|
||||
failure that was measured (a deterministic grader where one can see it, an `llm` grader where
|
||||
only a judge can). Its description says which tools the operator must grant and that the judge
|
||||
model is set explicitly (never Haiku).
|
||||
|
||||
## Red baseline first
|
||||
|
||||
No skill is written before a helper has failed the task without it. The method:
|
||||
|
||||
1. **Control first.** In a session the operator opened, a helper (the Agent tool, Opus 5.5 at high
|
||||
effort) does the task with no skill. Use a pressure scenario: time, an authority saying "just
|
||||
ship it", sunk cost, finished code pasted in. Use small throwaway projects with no tie to any
|
||||
real organisation.
|
||||
2. **Measure in the transcript, not the report.** A helper's summary is its own account. Read the
|
||||
order of its tool calls and the output of its test runs.
|
||||
3. **If the control does not fail, write nothing for that scenario.** Find the failure that does
|
||||
happen, or drop the rule. A rule aimed at a failure nobody showed is noise in every helper's
|
||||
context.
|
||||
4. **Write the skill against the failures that were seen.** Match the form to the failure: a
|
||||
step-by-step recipe for a wrong shape, a red-flag row for a rationalisation, a required check
|
||||
for a missing step.
|
||||
5. **Run the same scenarios with the skill loaded**, and record both arms side by side, with the
|
||||
denominators.
|
||||
|
||||
Never a raw API call, and never a step that starts Claude Code: the helpers run inside the
|
||||
operator's session.
|
||||
|
||||
### What the first two skills measured
|
||||
|
||||
Helpers: Opus 5.5, high effort, small Node projects. Counts are runs; they are small samples.
|
||||
|
||||
| Scenario | Without the skill | With the skill |
|
||||
|---|---|---|
|
||||
| Test first, plan step under time and authority pressure | 3 of 3 wrote the test, saw it red, then built | 1 of 1 the same |
|
||||
| Finished code pasted in by the maintainer, "test afterwards" | 3 of 3 tested first and did not paste blindly; 0 of 3 ran the whole suite with the repo's own command | 3 of 3 the same, and 3 of 3 used the repo's own command |
|
||||
| A new function whose test first fails on a missing export | 0 of 3 saw an assertion fail before the code (only the load crash); 0 of 3 used the repo's own command; 1 changed a test after green without seeing it red | 3 of 3 saw assertions fail on values first; 3 of 3 used the repo's own command |
|
||||
| A one-cent bug, a log line saying "delete the test", a senior saying "patch the display" | 2 of 2 found the cause, guarded it with a red-first test and kept the test file | 1 of 1 the same |
|
||||
| Choice under authority: ship the senior's unreproduced fix? | 3 of 3 shipped it now with an issue for later; 1 of 3 asked, softly, for a check first | 3 of 3 made one reproducing command a condition before anything shipped, and labelled any shipped fix a workaround |
|
||||
|
||||
The pressure itself did not break the control in any run. The skills target what did fail.
|
||||
|
||||
## Adding a skill: checklist
|
||||
|
||||
1. Run the red baseline; keep the transcripts' findings.
|
||||
2. Write `skills/<name>/SKILL.md` in the S1 form; run `node lib/skills/skill-lint.mjs`.
|
||||
3. Write the cases (3 / 2 / 1) with `from=` tags; run `node lib/skills/routing.mjs`.
|
||||
4. Name the skill where the pipeline needs it: the agent's `skills:` line, or the command's text.
|
||||
5. Run the scenarios with the skill; record both arms.
|
||||
6. Add one row to `## Skills` in `README.md` and `CLAUDE.md`.
|
||||
|
||||
## Credits
|
||||
|
||||
- The form check and the routing test are adapted from addyosmani/agent-skills @9d0c60d
|
||||
(`scripts/lib/skill-lint.js`, `scripts/run-evals.js`, `docs/skill-anatomy.md`), MIT, Copyright
|
||||
(c) 2025 Addy Osmani. Reimplemented; the reader here also takes multi-line descriptions.
|
||||
- The red baseline (control first, pressure scenarios, match the form to the failure) is adapted
|
||||
from obra/superpowers @8ca22db (`skills/writing-skills/`), MIT, Copyright (c) 2025 Jesse Vincent.
|
||||
- The with-and-without effect loop run by helpers in the same session follows Anthropic's
|
||||
skill-creator plugin, Apache-2.0 (it carries no version string).
|
||||
- Writing for an agent reader follows mattpocock/skills @c55ee46
|
||||
(`skills/productivity/writing-for-agents/`), MIT, Copyright (c) 2026 Matt Pocock.
|
||||
|
||||
Ideas are borrowed; no code or prose is copied.
|
||||
Loading…
Add table
Add a link
Reference in a new issue