fix(verification): a section the runner cannot read SAYS so, and trekplan is pinned to the format it reads
Measured 2026-09-18: nine ordinary shapes of a plan's `## Verification` section parsed to zero criteria - an untagged fence, a ```text fence, a markdown table, `## Verification (acceptance)`, `## Verification:`, `### Verification`, an unclosed fence earlier in the document. Every one came out as `0 of 0`, NOT OK, exit 1, and Phase 7 then forbade `result: completed` without anyone being told that the FORMAT, not the code, was the problem. "The section is empty" and "I cannot read this format" are different facts. Three changes, one hole: - The runner reports `NO_CRITERIA` with a source line (`plan.md:NN`) when the section is there and nothing in it parsed, and names the two forms it does read. Same for a brief's `## Success Criteria`, so the evidence block the conformance reviewer gets says which of the two it is looking at rather than showing an empty table. - Phase 7 says it out loud instead of failing silently: report the source line and the two forms, and say that the plan is what failed there, not the run. - `/trekplan` now pins what it produces to what the runner reads: the heading is exactly `## Verification`, the criteria are a bullet whose first backticked span is the command or a shell-tagged fence, and the command must be one the allowlist runs. A doc-consistency test holds the writer and the reader together, so a runner that learns a new form must update the source. Honest about the round trip: the two round-trip tests were GREEN on arrival - the template already writes the bullet form the runner reads. What was missing was not the format but the PIN: `/trekplan` mandated neither the heading string nor the format, so a plan could satisfy the command's own instructions and still parse to nothing. The tests now hold that. Red first: 5 of the 7 new tests failed before the change (3 NO_CRITERIA, 2 doc-consistency); the 2 round-trip tests are guards, and said so above. Suite 1161 (1159/0/2). Gate unchanged: defects 0 of 7, intact, exit 1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
83d82f121d
commit
f90e1cf02d
5 changed files with 195 additions and 9 deletions
|
|
@ -1310,6 +1310,18 @@ A plan with no `## Verification` section exits 1 with
|
|||
`error.code = NO_VERIFICATION_SECTION`. That is deliberate: a plan that
|
||||
promises no end-to-end check cannot be reported as verified.
|
||||
|
||||
A plan whose `## Verification` section IS there but declares nothing the runner
|
||||
can read exits 1 with `error.code = NO_CRITERIA` and a **source line**
|
||||
(`{plan_path}:NN`). Say that in the report in plain words — "the plan's
|
||||
`## Verification` at {plan_path}:NN declares no criterion the runner reads; the
|
||||
plan is what failed here, not the run" — and name the two forms it does read
|
||||
(a bullet whose first backticked span is the command; a shell-tagged fence).
|
||||
Measured 2026-09-18: nine ordinary shapes of that section — an untagged fence,
|
||||
a ```text fence, a table, `## Verification (acceptance)`, `### Verification` —
|
||||
all parsed to zero criteria, and Phase 7 then forbade `result: completed`
|
||||
without anyone being told that the format, not the code, was the problem.
|
||||
Silence is what made an unreadable section indistinguishable from a broken run.
|
||||
|
||||
Record in the progress file (additive-optional; unknown keys are tolerated by
|
||||
`progress-validator.mjs`, so a legacy progress file still validates):
|
||||
|
||||
|
|
|
|||
|
|
@ -616,6 +616,28 @@ Write the plan following the template structure. The plan MUST include:
|
|||
What tests to write and which patterns to follow.
|
||||
8. **Verification** — Reuse the brief's **Success Criteria** as the baseline.
|
||||
Each criterion must be an executable command or observable condition.
|
||||
The heading is **exactly `## Verification`** and the criteria are written in
|
||||
one of two forms. This is not style: `lib/verification/criteria-runner.mjs`
|
||||
is what runs this section in trekexecute Phase 7, and it reads those two
|
||||
forms and nothing else — `## Verification (acceptance)`, `### Verification`,
|
||||
an untagged fence or a table all parse to zero criteria, which fells the
|
||||
run without verifying anything.
|
||||
- a bullet whose FIRST backticked span is the command:
|
||||
```markdown
|
||||
- [ ] `npm test` -> expected: exit 0
|
||||
```
|
||||
- or a shell-tagged fence, one command per line:
|
||||
```bash
|
||||
npm test
|
||||
node --test tests/lib/x.test.mjs
|
||||
```
|
||||
The command must also be one the criteria runner is allowed to run: a test
|
||||
runner (`npm test`, `npm run <script>`, `node --test <file>`, `vitest`,
|
||||
`jest`, `pytest`, `python -m pytest`, `uv run pytest`, `cargo test`,
|
||||
`go test`, `make test`), `bash <script under tests/>`, or a read-only `git`
|
||||
subcommand — with no pipe, redirect or `&&`. Anything else is reported NOT
|
||||
RUN, never executed and never failed. A check that needs more than that is
|
||||
wrapped in a script under `tests/` and declared as `bash tests/<script>.sh`.
|
||||
9. **Estimated Scope** — File counts and complexity rating.
|
||||
|
||||
### Quality standards
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue