feat(skills): writing-criteria, one source for runnable criteria (order 7c)
A brief success criterion and a plan's Verify, Regression, End-to-end and Verification lines are each one command the criteria runner accepts, written as the line's first backticked span; anything more is bash tests/<name>.sh, and each must be able to go red (the runner reads the exit code only). - skills/writing-criteria/SKILL.md (S1 form), with Runs:/Refused:/Never red: examples that a pin runs through the criteria runner - brief-reviewer and plan-critic preload writing-criteria and reviewing-adversarially (S3 + S4); the copied runner rules leave them, the two templates and /trekplan, which names the skill where the plan is written and where its three reviewers start - eval cases 3 fire / 2 quiet / 1 effect; tests/lib/skills-criteria-prose-pins - review-630 @~1 pin follows the text into the skill; trekplan-stats line_ref re-measured Baseline without the skill: 24 of 27 written checks refused by the runner, and 0 of 3 reviewers saw that git status/git log can never go red. With it: 0 of 27 refused, 3 of 3 reviewers flagged both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
31c77fd8d6
commit
837e92545d
26 changed files with 605 additions and 53 deletions
|
|
@ -82,10 +82,12 @@ Optional per-step fields: `Interface in:` (what the step consumes, and which
|
|||
earlier step or existing code produces it), `Interface out:` (what it produces
|
||||
for later steps), `Review focus:` (inputs the brief implies but no test
|
||||
exercises yet, each pinned to a test; and any Global Constraint the step could
|
||||
break, quoted verbatim), and `Regression:` (ONE backticked command the criteria
|
||||
runner allows, no pipe and no `&&` — usually `npm test` — that the executor
|
||||
runs green before the checkpoint, on a step that can break existing tests;
|
||||
`Verify:` stays the step's own red-first test). A field that is present is
|
||||
break, quoted verbatim), and `Regression:` (ONE backticked command — usually
|
||||
`npm test` — that the executor runs green before the checkpoint, on a step that
|
||||
can break existing tests; `Verify:` stays the step's own red-first test).
|
||||
Every `Verify:`, `Regression:`, `End-to-end:` and `## Verification` command is
|
||||
written by the `voyage:writing-criteria` skill: one command the criteria runner
|
||||
accepts, a script under `tests/` for anything more, and able to go red. A field that is present is
|
||||
never empty. `On failure:` opens with `revert`, `retry`, `skip` or `escalate`,
|
||||
then the specifics.
|
||||
|
||||
|
|
@ -118,7 +120,7 @@ Several test files share one `File:` line the same way:
|
|||
- Verifies: {what the test checks}
|
||||
- Pattern: `path/to/existing-test.ts` *(follow this style)*
|
||||
- **Verify:** `{exact command}` → expected: `{output}`
|
||||
- **Regression:** `{npm test}` → expected: exit 0 *(optional)*
|
||||
- **Regression:** `npm test` → expected: exit 0 *(optional)*
|
||||
- **On failure:** {revert | retry | skip | escalate} — {specific instructions}
|
||||
- **Checkpoint:** `git add {the step's Files and its test files} && git commit -m "{conventional commit message}"`
|
||||
- **Manifest:**
|
||||
|
|
@ -143,7 +145,7 @@ Several test files share one `File:` line the same way:
|
|||
- File: `path/to/test.ts` *(existing | new)*
|
||||
- Verifies: {what the test checks}
|
||||
- Pattern: `path/to/existing-test.ts`
|
||||
- **End-to-end:** `{a command the criteria runner allows that runs the product the way a user does — the app, the CLI, the simulator; usually bash tests/e2e.sh}` → expected: {what the user sees} *(the same command also stands under ## Verification)*
|
||||
- **End-to-end:** `{a script that runs the product the way a user does — the app, the CLI, the simulator; usually bash tests/e2e.sh}` → expected: {what the user sees} *(the same command also stands under ## Verification)*
|
||||
- **Verify:** `{exact command}` → expected: `{output}`
|
||||
- **On failure:** {revert | retry | skip | escalate} — {specific instructions}
|
||||
- **Checkpoint:** `git add {the step's Files and its test files} && git commit -m "{conventional commit message}"`
|
||||
|
|
|
|||
|
|
@ -112,10 +112,9 @@ verifiable by running a specific command or observing a specific system behavior
|
|||
- {criterion — e.g., "New endpoint answers ok: `bash tests/health.sh` exits 0 (the script calls the endpoint and checks the status field)"}
|
||||
- {criterion — e.g., "No TypeScript errors: `npm run typecheck` exits 0"}
|
||||
|
||||
One command per criterion, from the runner's allowlist (`npm test`, `npm run <script>`,
|
||||
`node --test`, `pytest`, `bash tests/<script>.sh`, …), with no pipe or `&&`,
|
||||
in plain words (it runs with no shell: no quote, `$` or glob outside `node --test`).
|
||||
Anything else is wrapped in a script under `tests/`; otherwise the review
|
||||
Each command is written by the `voyage:writing-criteria` skill: the first
|
||||
backticked span on the line, one command the criteria runner accepts, a script
|
||||
under `tests/` for anything more, and able to go red. Otherwise the review
|
||||
reports the criterion as NOT RUN.
|
||||
|
||||
Do NOT write vague criteria:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue