open/voyage
1
0
Fork 0
Code Issues Releases 13 Activity

feat(skills): writing-criteria, one source for runnable criteria (order 7c)

Browse source 
A brief success criterion and a plan's Verify, Regression, End-to-end and
Verification lines are each one command the criteria runner accepts, written
as the line's first backticked span; anything more is bash tests/<name>.sh,
and each must be able to go red (the runner reads the exit code only).

- skills/writing-criteria/SKILL.md (S1 form), with Runs:/Refused:/Never red:
  examples that a pin runs through the criteria runner
- brief-reviewer and plan-critic preload writing-criteria and
  reviewing-adversarially (S3 + S4); the copied runner rules leave them, the
  two templates and /trekplan, which names the skill where the plan is written
  and where its three reviewers start
- eval cases 3 fire / 2 quiet / 1 effect; tests/lib/skills-criteria-prose-pins
- review-630 @~1 pin follows the text into the skill; trekplan-stats line_ref
  re-measured

Baseline without the skill: 24 of 27 written checks refused by the runner,
and 0 of 3 reviewers saw that git status/git log can never go red. With it:
0 of 27 refused, 3 of 3 reviewers flagged both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-10-17 07:39:04 +02:00
commit 837e92545d
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
26 changed files with 605 additions and 53 deletions
Download patch file Download diff file Expand all files Collapse all files

119
skills/writing-criteria/SKILL.md Normal file
View file

@ -0,0 +1,119 @@
---
name: writing-criteria
description: "Criteria discipline for the checks Voyage runs itself: each brief success criterion, each plan Verify, Regression and End-to-end line and each Verification bullet is one command the criteria runner accepts, written as its first backticked span (a known test runner, plain words, no pipe, no &&, no shell), and a check that needs more becomes a script declared as bash tests/<name>.sh. Each criterion must be able to go red: a command that exits 0 whatever the code does is no criterion. Use when writing or reviewing success criteria, a Verify, Regression or End-to-end line or a Verification section, or when a validator reports a criterion as not runnable (BRIEF_SC_NOT_RUNNABLE, PLAN_STEP_VERIFY_NOT_RUNNABLE)."
---
# Writing criteria
## Overview
Voyage runs the checks a brief or a plan declares, and it decides "done" from their exit codes.
`lib/verification/criteria-runner.mjs` runs each one with no shell, as its words, and only when it is a
known test-runner invocation. Anything else is NOT RUN: never executed, never green. In `/trekexecute`
Phase 7 an unrunnable plan criterion stops the run; in `/trekreview` an unrunnable success criterion is
an absent measurement. A criterion that runs but cannot fail measures nothing either. This skill is the
one place that says how to write both right.
## When to use
- Writing a brief's `## Success Criteria`.
- Writing a plan step's `Verify:`, `Regression:` or `End-to-end:` line, or the plan's `## Verification`.
- Reviewing them: `brief-reviewer` and `plan-critic` preload this skill.
- A validator reports `BRIEF_SC_NOT_RUNNABLE`, `PLAN_VERIFICATION_NOT_RUNNABLE`,
`PLAN_STEP_VERIFY_NOT_RUNNABLE`, `PLAN_STEP_REGRESSION_NOT_RUNNABLE` or `PLAN_END_TO_END_NOT_RUNNABLE`.
- Not for: writing the test itself and seeing it fail (that is `voyage:test-first`), or starting the app
to look at a change by hand (Claude Code's own `run`).
## Process
1. **One criterion, one command.** No `&&`, `;`, `|`, `>`, `<`, `$(` or backtick inside it. Two things to
check are two criteria, or one script (step 5). A line that chains the suite, a setup and three
checks is refused whole, and then none of it is measured.
2. **The command is the first backticked span on the line.** The runner reads the first span as the
command. Name the feature in plain words, not in backticks before the command:
``- shelf count prints the number of books: `bash tests/count.sh` exits 0``. A line that opens with
the feature in backticks (`` `shelf count` prints ... ``) declares `shelf count` as its command.
3. **Use a runner the runner knows.** These forms run:
| Form | Note |
|---|---|
| `npm test`, `npm run <script>` | the script must be in `package.json` when the check runs; a plan may add it |
| `node --test <files>` | a glob only on Node 21 and later; elsewhere name the files |
| `vitest`, `jest`, `pytest`, `python -m pytest`, `uv run pytest` | with a short list of read-only options |
| `cargo test`, `go test`, `make test` | nothing may follow `make test` |
| `bash tests/<name>.sh`, `/bin/bash tests/<name>.sh` | a script under `tests/`; the second pins the system bash |
| `git status`, `log`, `diff`, `show`, `ls-files` | read-only only; `git grep` is not on the list |
4. **Plain words.** No shell reads the line, so a word that means something in a shell is refused: a
quote, a backslash, `$`, a brace, a leading `~`, a word starting with `@` (a git revision such as
`@~1` is the one exception), a path that climbs out with `..` or starts at `/`, an environment prefix
(`CI=1 npm test`), and any option not on the runner's read-only list. A trailing `# comment` is
dropped, unless it holds a quote, `$`, backtick or backslash; then the line is refused.
5. **A check that needs more is a script.** The product's own binary, a temporary folder, an
environment variable, comparing output, `jq`, `grep`, `curl`: put it in a script under `tests/` that
exits non-zero with a message when the check fails (`set -eu`, an explicit `exit 1`), and declare
`bash tests/<name>.sh`. The script goes in the `Files:` of the step that builds the behaviour. This is
how an `End-to-end:` line runs the product the way a user does.
6. **It must be able to go red.** Before writing a criterion, name the wrong result that would make it
fail. The runner decides on the exit code alone: the `→ expected:` text after the command is for the
reader and is never compared. So a command that exits 0 whatever the code does is no criterion:
`git status --porcelain` "expected: no output" exits 0 on a dirty tree too, `git log` and
`git diff --stat` report and exit 0, and `npm test` alone proves no new behaviour unless a test for
that behaviour is in it. A check on output goes in a script that compares it and exits 1. A step's `Verify:` is ONE command, the step's own new test, seen red before the
step; a `Verify:` that runs a test file which already passes can never be red. The suite the step must
not break goes in `Regression:`.
7. **Check it before handing it on.** `node ${CLAUDE_PLUGIN_ROOT}/lib/validators/plan-validator.mjs
--strict --json <plan>` and `lib/validators/brief-validator.mjs` name every unrunnable criterion.
A refused criterion is rewritten (steps 1 to 5), never kept with a note.
### Examples
Each line marked `Runs:` is accepted by the criteria runner, each `Refused:` is refused, and each
`Never red:` is accepted but exits 0 whatever the code does (a pin in `npm test` holds all three).
- Runs: `npm test`
- Runs: `npm run typecheck`
- Runs: `node --test tests/count.test.mjs`
- Runs: `bash tests/count.sh`
- Runs: `/bin/bash tests/e2e.sh`
- Runs: `pytest -q tests/test_count.py`
- Runs: `cargo test --workspace`
- Refused: `npm test && npm run lint` (two commands: write two criteria)
- Refused: `node bin/shelf.mjs count` (the product's binary is no test runner: `bash tests/count.sh`)
- Refused: `node bin/shelf.mjs list --json | jq length` (a pipe: move the check into a script)
- Refused: `SHELF_FILE=tmp/s.json node --test tests/cli.test.mjs` (an environment prefix: set it in the test)
- Refused: `node -e "process.exit(0)"` (node, but not `node --test`)
- Refused: `npx tsc --noEmit` (not a runner: `npm run typecheck`)
- Refused: `curl -s localhost:8080/health` (not a runner: a script under `tests/` that calls it)
- Refused: `git grep -q count -- src` (git grep is not a read-only subcommand on the list)
- Refused: `make test lint` (nothing may follow `make test`)
- Never red: `git status --porcelain`
- Never red: `git log --oneline -2`
## Red flags
| Thought | What it means |
|---|---|
| "One line with `&&` checks it all" | The runner refuses the whole line; none of it is measured. Two criteria, or one script. |
| "Name the feature in backticks, then the command" | The first backticked span is the command. The feature goes in plain words. |
| "A user runs `node bin/app.mjs`, so that is the check" | It is still no test runner. A script under `tests/` runs it. |
| "A quick `node -e` one-liner checks the JSON" | Refused. The same code in a test file runs as `node --test`. |
| "`npm test` exits 0, so the feature is done" | It exited 0 before the change too, unless a test for the feature is in it. |
| "`git status --porcelain` → expected: no output fails on a dirty tree" | The runner reads the exit code, never the expected text. It exits 0 either way: it can never go red. |
| "`Verify:` runs the step's test plus a CLI check" | `Verify:` is one command, the step's new test. The rest is a script or `Regression:`. |
| "NOT RUN is not a failure, so it can stay" | NOT RUN is never green: an unrunnable plan criterion stops `/trekexecute` Phase 7. |
## Enforced by
- `lib/verification/criteria-runner.mjs`: deterministic. It runs only what its allowlist accepts, with no
shell, and reports everything else NOT RUN, never green.
- `lib/validators/plan-validator.mjs` and `lib/validators/brief-validator.mjs`: the same allowlist when
the plan or brief is written (`PLAN_*_NOT_RUNNABLE` are errors in a plan, `BRIEF_SC_NOT_RUNNABLE` a
warning in a brief).
- prose only: that a criterion can go red and that it tests what the brief asks for. `brief-reviewer`
and `plan-critic` judge that by reading; no check runs it.
## Credits
- Voyage's own content; nothing is borrowed. The rules restate `lib/verification/criteria-runner.mjs`.
The red flags come from Voyage's own baseline runs without this skill (`docs/skills-authoring.md`).