fix(validate): a check that can never run is caught when the brief or plan is written

voyage-lekerepo end-to-end trial (2026-10-06), report points 2, 3, 4, 9:
11 of 13 brief criteria were NOT RUN at review with no warning in /trekbrief;
a plan `git grep` criterion passed validator, gate and three critic rounds,
then felled the run in Phase 7; the End-to-end field and Phase 7's text
search never met; a prose Verify left the test command undefined.

- criteria-runner: runnableCheck() screens a declared check without running
  it (same allowlist; an npm run script is not looked up at write time);
  /bin/bash tests/<script> allowed; the plan report carries end_to_end,
  verified when a passed criterion runs a step's End-to-end command (the
  older text form still counts).
- plan-validator 1.8: PLAN_VERIFICATION_NOT_RUNNABLE,
  PLAN_STEP_VERIFY_NOT_RUNNABLE, PLAN_END_TO_END_NOT_RUNNABLE,
  PLAN_END_TO_END_NOT_VERIFIED (the gate inherits them).
- brief-validator: BRIEF_SC_NOT_RUNNABLE warning; /trekbrief offers a
  tests/ script; brief-reviewer no longer proposes unrunnable commands.
- trekexecute: Phase 7 copies end_to_end from the runner; Verify's test
  command is its first backticked span.

Measured on the trial's own brief and plan: 11 warnings (the 11 NOT RUN),
and the plan stops on V5 git grep and the unlisted End-to-end command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-10-08 06:42:33 +02:00
commit c56f852ab6
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
14 changed files with 422 additions and 38 deletions

View file

@ -99,6 +99,7 @@ Optional but standard sections: `## Non-Goals`, `## Constraints`, `## Preference
| **v5.1 sequencing gate** | every read | `brief_version ≥ 2.1` requires `phase_signals` (list) OR `phase_signals_partial: true` — error `BRIEF_V51_MISSING_SIGNALS` on miss. Validator-only enforcement; commands surface, don't re-enforce. |
| **v5.5 framing gate** | every read | `framing` enum-checked on any version when present (`BRIEF_INVALID_FRAMING`); at `brief_version ≥ 2.2` a missing `framing` → `BRIEF_MISSING_FRAMING` and a missing `## TL;DR` → `BRIEF_MISSING_SECTION` (strict) / warning (soft). `trekreview` briefs are exempt. |
| Body sections | strict only | All `BRIEF_BODY_SECTIONS` present (`## TL;DR` added at ≥ 2.2) |
| Runnable criteria | every read | A `## Success Criteria` bullet whose first backticked span is a command the review's criteria runner never runs (the product's own binary, `jq`, `grep`, a pipe) → warning `BRIEF_SC_NOT_RUNNABLE`, never an error: a criterion a human judges stays allowed. Prose-only criteria are not flagged. `trekreview` briefs are exempt. No schema change. |
**State machine** detail: a brief that says it has research topics but skipped them must explicitly admit it (via `brief_quality: partial`). This is the most common failure mode the validator catches.
@ -128,6 +129,7 @@ Optional but standard sections: `## Non-Goals`, `## Constraints`, `## Preference
- `BRIEF_INVALID_EFFORT` → strict halt; effort ∉ `[low, standard, high]`.
- `BRIEF_INVALID_MODEL` → strict halt; model ∉ `BASE_ALLOWED_MODELS` (currently `[sonnet, opus, fable]`).
- `BRIEF_SIGNALS_MUTUALLY_EXCLUSIVE` → strict halt; cannot set both `phase_signals` and `phase_signals_partial: true`.
- `BRIEF_SC_NOT_RUNNABLE` → warning in both modes; `/trekbrief` names each one and offers a script under `tests/` the criterion runs (`bash tests/<script>.sh`).
**Compatibility direction of the v5.9 allowlist widening (`fable`):** enum
widening is safe for new readers of old data, not old readers of new data.
@ -281,6 +283,10 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
| `PLAN_STEP_NO_TEST_FIRST` | 1.8: a step lacks `Test first:` with File + Verifies | Add the test the step makes pass |
| `PLAN_NO_END_TO_END` | 1.8: no step carries `End-to-end:` with a command | Add the "run the thing" step |
| `PLAN_STEP_FIELD_EMPTY` | 1.8: `Interface in/out:` or `Review focus:` present but empty | Fill or remove it |
| `PLAN_VERIFICATION_NOT_RUNNABLE` | 1.8: a `## Verification` criterion has no command, or one the criteria runner's allowlist never runs | Name a test runner, or wrap the check as `bash tests/<script>.sh` |
| `PLAN_STEP_VERIFY_NOT_RUNNABLE` | 1.8: the first backticked span of a step's `Verify:` (its test command) is not one runnable test-runner call | Put the test command first; move other checks to a script under `tests/` |
| `PLAN_END_TO_END_NOT_RUNNABLE` | 1.8: an `End-to-end:` command the allowlist never runs (e.g. the product's own binary) | Run the product from `bash tests/e2e.sh` |
| `PLAN_END_TO_END_NOT_VERIFIED` | 1.8: an `End-to-end:` command does not also stand under `## Verification` | List the same command there; Phase 7 runs that section |
| `PLAN_CEREMONY_INVALID` | 1.8: `ceremony` not small / bounded / architectural | Copy the brief's value |
| `PLAN_DESIGN_MISSING` / `PLAN_DESIGN_SECTION_MISSING` | 1.8 architectural: no `## Design`, or a required subsection missing | Write it from `templates/design-template.md` |
| `PLAN_MODULE_MAP_EMPTY` / `PLAN_MODULE_CYCLE` | 1.8 architectural: no mermaid module map, or a cycle in it (all `mermaid` blocks of `### Module Map` form one map) | Draw it; break the cycle |