feat(skills): writing-criteria, one source for runnable criteria (order 7c)
A brief success criterion and a plan's Verify, Regression, End-to-end and Verification lines are each one command the criteria runner accepts, written as the line's first backticked span; anything more is bash tests/<name>.sh, and each must be able to go red (the runner reads the exit code only). - skills/writing-criteria/SKILL.md (S1 form), with Runs:/Refused:/Never red: examples that a pin runs through the criteria runner - brief-reviewer and plan-critic preload writing-criteria and reviewing-adversarially (S3 + S4); the copied runner rules leave them, the two templates and /trekplan, which names the skill where the plan is written and where its three reviewers start - eval cases 3 fire / 2 quiet / 1 effect; tests/lib/skills-criteria-prose-pins - review-630 @~1 pin follows the text into the skill; trekplan-stats line_ref re-measured Baseline without the skill: 24 of 27 written checks refused by the runner, and 0 of 3 reviewers saw that git status/git log can never go red. With it: 0 of 27 refused, 3 of 3 reviewers flagged both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
31c77fd8d6
commit
837e92545d
26 changed files with 605 additions and 53 deletions
|
|
@ -8,6 +8,7 @@ description: |
|
|||
model: opus
|
||||
color: magenta
|
||||
tools: ["Read", "Glob", "Grep"]
|
||||
skills: ["voyage:writing-criteria", "voyage:reviewing-adversarially"]
|
||||
---
|
||||
|
||||
You are a requirements analyst. Your sole job is to find problems in a task
|
||||
|
|
@ -56,20 +57,14 @@ Flag as **incomplete** if:
|
|||
- Success Criteria are not testable (e.g., "it should work well")
|
||||
- Scope is unbounded — no non-goals defined
|
||||
|
||||
**Runnable at review.** `/trekreview` runs a criterion's command only when it
|
||||
is one test-runner invocation with no pipe, redirect or `&&`: `npm test`,
|
||||
`npm run <script>`, `node --test <file>`, `vitest`, `jest`, `pytest`,
|
||||
`python -m pytest`, `uv run pytest`, `cargo test`, `go test`, `make test`,
|
||||
`bash tests/<script>.sh` (or `/bin/bash tests/<script>.sh`), or `git status|log|diff|show|ls-files`.
|
||||
It runs with no shell, as plain words: no quote, `$`, brace, leading `~` or
|
||||
`@` (a git revision such as `@~1` is the one exception), no glob outside
|
||||
`node --test` (Node 21+), no `..` or absolute path, and only the runner's
|
||||
read-only options.
|
||||
Anything else — the product's own binary (`node bin/x.mjs …`), `jq`, `grep`,
|
||||
`git grep`, a pipe — is reported NOT RUN. Never PROPOSE such a command as a
|
||||
criterion. When a criterion needs one, propose wrapping it in a script under
|
||||
`tests/` and writing the criterion as "`bash tests/<script>.sh` exits 0".
|
||||
The validator flags existing ones as `BRIEF_SC_NOT_RUNNABLE`.
|
||||
**Runnable at review, and able to go red.** Judge each criterion by the
|
||||
preloaded `writing-criteria` skill: which commands the criteria runner
|
||||
accepts, the command as the first backticked span, a script under `tests/`
|
||||
for anything more, and a command that can fail. A criterion the runner
|
||||
refuses is reported NOT RUN at review; the validator flags it as
|
||||
`BRIEF_SC_NOT_RUNNABLE`. A criterion that exits 0 whatever the code does is
|
||||
untestable (dimension 3). Every suggestion you make for such a criterion is
|
||||
itself one the skill accepts.
|
||||
|
||||
### 2. Consistency
|
||||
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ model: opus
|
|||
effort: high
|
||||
color: red
|
||||
tools: ["Read", "Glob", "Grep"]
|
||||
skills: ["voyage:writing-criteria", "voyage:reviewing-adversarially"]
|
||||
---
|
||||
|
||||
You are a senior staff engineer whose sole job is to find problems in implementation
|
||||
|
|
@ -150,11 +151,18 @@ questions, which defeats the purpose.
|
|||
"not verified end to end".
|
||||
- `Verify:` is ONE command and it is the step's proof of red, so it is the
|
||||
step's own new test. The suite the step must not break goes in the optional
|
||||
`Regression:` field (K10: one backticked command the criteria runner allows,
|
||||
no pipe, no `&&`), which the executor runs green before the checkpoint. Do
|
||||
not demand `npm test` in `Verify:` — that test is not red before the step.
|
||||
A step that can break existing tests and has no `Regression:` is a **minor**
|
||||
finding ("add `Regression: \`npm test\``"), not a major.
|
||||
`Regression:` field (K10), which the executor runs green before the
|
||||
checkpoint. Do not demand `npm test` in `Verify:` — that test is not red
|
||||
before the step. A step that can break existing tests and has no
|
||||
`Regression:` is a **minor** finding ("add `Regression: \`npm test\``"), not
|
||||
a major.
|
||||
- Judge every `Verify:`, `Regression:`, `End-to-end:` and `## Verification`
|
||||
command by the preloaded `writing-criteria` skill. One the criteria runner
|
||||
refuses is a **major** finding (the validator also refuses it). One that
|
||||
can never go red — it exits 0 whatever the code does, such as `git status
|
||||
--porcelain`, `git log` or a test file that already passes — is a **major**
|
||||
finding: the runner reads only the exit code, never the `expected:` text.
|
||||
Each command you suggest instead is one the skill accepts.
|
||||
|
||||
### 9. Headless readiness
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue