fix(validate): a check that can never run is caught when the brief or plan is written
voyage-lekerepo end-to-end trial (2026-10-06), report points 2, 3, 4, 9: 11 of 13 brief criteria were NOT RUN at review with no warning in /trekbrief; a plan `git grep` criterion passed validator, gate and three critic rounds, then felled the run in Phase 7; the End-to-end field and Phase 7's text search never met; a prose Verify left the test command undefined. - criteria-runner: runnableCheck() screens a declared check without running it (same allowlist; an npm run script is not looked up at write time); /bin/bash tests/<script> allowed; the plan report carries end_to_end, verified when a passed criterion runs a step's End-to-end command (the older text form still counts). - plan-validator 1.8: PLAN_VERIFICATION_NOT_RUNNABLE, PLAN_STEP_VERIFY_NOT_RUNNABLE, PLAN_END_TO_END_NOT_RUNNABLE, PLAN_END_TO_END_NOT_VERIFIED (the gate inherits them). - brief-validator: BRIEF_SC_NOT_RUNNABLE warning; /trekbrief offers a tests/ script; brief-reviewer no longer proposes unrunnable commands. - trekexecute: Phase 7 copies end_to_end from the runner; Verify's test command is its first backticked span. Measured on the trial's own brief and plan: 11 warnings (the 11 NOT RUN), and the plan stops on V5 git grep and the unlisted End-to-end command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
ca6ee8779a
commit
c56f852ab6
14 changed files with 422 additions and 38 deletions
|
|
@ -99,6 +99,7 @@ Optional but standard sections: `## Non-Goals`, `## Constraints`, `## Preference
|
|||
| **v5.1 sequencing gate** | every read | `brief_version ≥ 2.1` requires `phase_signals` (list) OR `phase_signals_partial: true` — error `BRIEF_V51_MISSING_SIGNALS` on miss. Validator-only enforcement; commands surface, don't re-enforce. |
|
||||
| **v5.5 framing gate** | every read | `framing` enum-checked on any version when present (`BRIEF_INVALID_FRAMING`); at `brief_version ≥ 2.2` a missing `framing` → `BRIEF_MISSING_FRAMING` and a missing `## TL;DR` → `BRIEF_MISSING_SECTION` (strict) / warning (soft). `trekreview` briefs are exempt. |
|
||||
| Body sections | strict only | All `BRIEF_BODY_SECTIONS` present (`## TL;DR` added at ≥ 2.2) |
|
||||
| Runnable criteria | every read | A `## Success Criteria` bullet whose first backticked span is a command the review's criteria runner never runs (the product's own binary, `jq`, `grep`, a pipe) → warning `BRIEF_SC_NOT_RUNNABLE`, never an error: a criterion a human judges stays allowed. Prose-only criteria are not flagged. `trekreview` briefs are exempt. No schema change. |
|
||||
|
||||
**State machine** detail: a brief that says it has research topics but skipped them must explicitly admit it (via `brief_quality: partial`). This is the most common failure mode the validator catches.
|
||||
|
||||
|
|
@ -128,6 +129,7 @@ Optional but standard sections: `## Non-Goals`, `## Constraints`, `## Preference
|
|||
- `BRIEF_INVALID_EFFORT` → strict halt; effort ∉ `[low, standard, high]`.
|
||||
- `BRIEF_INVALID_MODEL` → strict halt; model ∉ `BASE_ALLOWED_MODELS` (currently `[sonnet, opus, fable]`).
|
||||
- `BRIEF_SIGNALS_MUTUALLY_EXCLUSIVE` → strict halt; cannot set both `phase_signals` and `phase_signals_partial: true`.
|
||||
- `BRIEF_SC_NOT_RUNNABLE` → warning in both modes; `/trekbrief` names each one and offers a script under `tests/` the criterion runs (`bash tests/<script>.sh`).
|
||||
|
||||
**Compatibility direction of the v5.9 allowlist widening (`fable`):** enum
|
||||
widening is safe for new readers of old data, not old readers of new data.
|
||||
|
|
@ -281,6 +283,10 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
|
|||
| `PLAN_STEP_NO_TEST_FIRST` | 1.8: a step lacks `Test first:` with File + Verifies | Add the test the step makes pass |
|
||||
| `PLAN_NO_END_TO_END` | 1.8: no step carries `End-to-end:` with a command | Add the "run the thing" step |
|
||||
| `PLAN_STEP_FIELD_EMPTY` | 1.8: `Interface in/out:` or `Review focus:` present but empty | Fill or remove it |
|
||||
| `PLAN_VERIFICATION_NOT_RUNNABLE` | 1.8: a `## Verification` criterion has no command, or one the criteria runner's allowlist never runs | Name a test runner, or wrap the check as `bash tests/<script>.sh` |
|
||||
| `PLAN_STEP_VERIFY_NOT_RUNNABLE` | 1.8: the first backticked span of a step's `Verify:` (its test command) is not one runnable test-runner call | Put the test command first; move other checks to a script under `tests/` |
|
||||
| `PLAN_END_TO_END_NOT_RUNNABLE` | 1.8: an `End-to-end:` command the allowlist never runs (e.g. the product's own binary) | Run the product from `bash tests/e2e.sh` |
|
||||
| `PLAN_END_TO_END_NOT_VERIFIED` | 1.8: an `End-to-end:` command does not also stand under `## Verification` | List the same command there; Phase 7 runs that section |
|
||||
| `PLAN_CEREMONY_INVALID` | 1.8: `ceremony` not small / bounded / architectural | Copy the brief's value |
|
||||
| `PLAN_DESIGN_MISSING` / `PLAN_DESIGN_SECTION_MISSING` | 1.8 architectural: no `## Design`, or a required subsection missing | Write it from `templates/design-template.md` |
|
||||
| `PLAN_MODULE_MAP_EMPTY` / `PLAN_MODULE_CYCLE` | 1.8 architectural: no mermaid module map, or a cycle in it (all `mermaid` blocks of `### Module Map` form one map) | Draw it; break the cycle |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue