fix(validate): a check that can never run is caught when the brief or plan is written

voyage-lekerepo end-to-end trial (2026-10-06), report points 2, 3, 4, 9: 11 of 13 brief criteria were NOT RUN at review with no warning in /trekbrief; a plan `git grep` criterion passed validator, gate and three critic rounds, then felled the run in Phase 7; the End-to-end field and Phase 7's text search never met; a prose Verify left the test command undefined. - criteria-runner: runnableCheck() screens a declared check without running it (same allowlist; an npm run script is not looked up at write time); /bin/bash tests/<script> allowed; the plan report carries end_to_end, verified when a passed criterion runs a step's End-to-end command (the older text form still counts). - plan-validator 1.8: PLAN_VERIFICATION_NOT_RUNNABLE, PLAN_STEP_VERIFY_NOT_RUNNABLE, PLAN_END_TO_END_NOT_RUNNABLE, PLAN_END_TO_END_NOT_VERIFIED (the gate inherits them). - brief-validator: BRIEF_SC_NOT_RUNNABLE warning; /trekbrief offers a tests/ script; brief-reviewer no longer proposes unrunnable commands. - trekexecute: Phase 7 copies end_to_end from the runner; Verify's test command is its first backticked span. Measured on the trial's own brief and plan: 11 warnings (the 11 NOT RUN), and the plan stops on V5 git grep and the unlisted End-to-end command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 06:42:33 +02:00 · 2026-10-08 06:42:33 +02:00 · c56f852ab6
commit c56f852ab6
parent ca6ee8779a
14 changed files with 422 additions and 38 deletions
--- a/agents/brief-reviewer.md
+++ b/agents/brief-reviewer.md
@ -56,6 +56,17 @@ Flag as **incomplete** if:
 - Success Criteria are not testable (e.g., "it should work well")
 - Scope is unbounded — no non-goals defined

+**Runnable at review.** `/trekreview` runs a criterion's command only when it
+is one test-runner invocation with no pipe, redirect or `&&`: `npm test`,
+`npm run <script>`, `node --test <file>`, `vitest`, `jest`, `pytest`,
+`python -m pytest`, `uv run pytest`, `cargo test`, `go test`, `make test`,
+`bash tests/<script>.sh` (or `/bin/bash tests/<script>.sh`), or `git status|log|diff|show|ls-files`.
+Anything else — the product's own binary (`node bin/x.mjs …`), `jq`, `grep`,
+`git grep`, a pipe — is reported NOT RUN. Never PROPOSE such a command as a
+criterion. When a criterion needs one, propose wrapping it in a script under
+`tests/` and writing the criterion as "`bash tests/<script>.sh` exits 0".
+The validator flags existing ones as `BRIEF_SC_NOT_RUNNABLE`.
+
 ### 2. Consistency

 Check for internal contradictions:
--- a/commands/trekbrief.md
+++ b/commands/trekbrief.md
@ -330,6 +330,9 @@ sentence with your recommended answer first.
 **Success Criteria** (required):
 - _Anchor_: "How do we verify it is actually done? List 2–4 concrete,
  testable conditions — commands to run, observations, or metrics."
+  A command the review can run is one test-runner call (`npm test`,
+  `node --test <file>`, `bash tests/<script>.sh`); propose a script under
+  `tests/` for anything else (the product's own binary, a pipe, `jq`).
 - _Sharpen_ (criterion is vague): "'{quoted criterion}' is subjective.
  Which command, observation, or metric would prove this is met?"
 - _Quantify_ (performance/quality claim): "You mentioned it should be
@ -816,7 +819,12 @@ node ${CLAUDE_PLUGIN_ROOT}/lib/validators/brief-validator.mjs --json "{PROJECT_D

 If the validator returns errors, report them to the user and offer to
 re-enter Phase 4 with the validator's hints in scope. If only warnings,
-note them in the final report.
+note them in the final report — except `BRIEF_SC_NOT_RUNNABLE`: a success
+criterion whose command `/trekreview` will never run (the product's own
+binary, `jq`, `grep`, a pipe). Name each one to the user in plain words and
+offer to rewrite it as a script under `tests/` that the criterion runs
+(`bash tests/<script>.sh` exits 0) before the brief is approved; if the user
+keeps it, it stays a criterion a human judges, and the report says so.

 **Build the operator-annotation HTML, then print the report.** After the
 brief is validated, run `scripts/annotate.mjs` to produce a self-contained
--- a/commands/trekexecute.md
+++ b/commands/trekexecute.md
@ -131,9 +131,10 @@ Extract every `### Step N: {description}` heading (in order). For each step, ext
 - **Test first** — the test the step writes before it builds: its `File:`
  (existing | new), what it `Verifies:`, an optional `Pattern:` (required from
  plan_version ≥ 1.8, see below)
- **Verify** — command to run after implementation. It is also the step's
-  **test command**: proof of red (Sub-step B) and the done line (Sub-step G)
-  run it, so it must be one test-runner invocation
+- **Verify** — command to run after implementation: the first backticked
+  span of the field. It is also the step's **test command**: proof of red
+  (Sub-step B) and the done line (Sub-step G) run it, so it must be one
+  test-runner invocation
 - **On failure** — recovery action (revert/retry/skip/escalate)
 - **Checkpoint** — git commit command after success
 - **Manifest** — YAML block following Checkpoint (v1.7+)
@ -1333,10 +1334,12 @@ plan template lists them apart from `Files:`, so everywhere below **the step's
 scope = its `Files:` + its test files**: both are built, both are committed at
 the checkpoint, and neither is a scope finding.

-**The test command** is the step's `Verify:` command. It must be ONE
-test-runner invocation without shell operators (the allowlist of
-`lib/verification/criteria-runner.mjs`); the tools below refuse anything else
-with exit 2.
+**The test command** is the step's `Verify:` command: the FIRST backticked
+span of the field. Anything after it (expected output, notes) is description
+and is not run. It must be ONE test-runner invocation without shell operators
+(the allowlist of `lib/verification/criteria-runner.mjs`); the tools below
+refuse anything else with exit 2, and from plan_version 1.8 the plan validator
+already rejects such a step (`PLAN_STEP_VERIFY_NOT_RUNNABLE`).

 1. **The helper proves red before it builds** (Sub-step C): it writes the test
   from `Test first:`, runs the gate
@ -1849,20 +1852,22 @@ verification failure must not hide a manifest drift.

 ### End to end — "run the thing" (P9 (6))

-Tests passing is not the product working. From the runner's JSON, set
-`end_to_end`:
+Tests passing is not the product working. Copy `end_to_end.status` from the
+runner's JSON into `end_to_end` — the runner decides it, not this prose:

- `"verified"` — at least one criterion that **passed** says in its text that
-  it runs the thing end to end (its text contains `end to end`, `end-to-end`
-  or `run the thing`, any case);
+- `"verified"` — the verification passed and at least one passed criterion
+  runs a step's `End-to-end:` command (`by: "end-to-end-field"`), or, the
+  older form, says in its text that it runs the thing end to end (`end to
+  end`, `end-to-end` or `run the thing`, any case; `by: "criterion-text"`);
 - `"not-verified"` — otherwise, including a session spec and a plan whose
  verification did not pass.

 `"not-verified"` is never green: the report says **not verified end to end**
 in plain words, and the completion gate (Phase 7.5, clause 4) turns
 `completed` into `partial`. The fix is in the plan — a criterion that runs the
-built thing — not in the run. (This reads the existing `## Verification`
-criteria; it adds no plan field.)
+built thing — not in the run. (A ≥ 1.8 plan cannot reach here with its
+`End-to-end:` command missing from `## Verification`: the plan validator
+rejects that as `PLAN_END_TO_END_NOT_VERIFIED`.)

 ## Phase 7.5 — Manifest audit (independent)

--- a/commands/trekplan.md
+++ b/commands/trekplan.md
@ -640,7 +640,13 @@ Do NOT write this synthesis to disk. It is internal working context only.
 > empty when present); at least one step with `End-to-end:` and a backticked
 > command — the "run the thing" step; the header line carries
 > `` `plan_version: 1.8` · `ceremony: <class>` ``; an architectural plan carries
-> `## Design` (see "Design part" below).
+> `## Design` (see "Design part" below); and every check the run executes can
+> run — each `## Verification` criterion, the first backticked span of each
+> `Verify:` (the step's test command) and each `End-to-end:` command pass the
+> criteria runner's allowlist, and each `End-to-end:` command also stands under
+> `## Verification` (`PLAN_VERIFICATION_NOT_RUNNABLE`,
+> `PLAN_STEP_VERIFY_NOT_RUNNABLE`, `PLAN_END_TO_END_NOT_RUNNABLE`,
+> `PLAN_END_TO_END_NOT_VERIFIED`).
 >
 > **Validator self-check (mandatory after writing `plan.md`):** run
 > `node ${CLAUDE_PLUGIN_ROOT}/lib/validators/plan-validator.mjs --strict --json {plan_path}`
@ -696,7 +702,12 @@ Write the plan following the template structure. The plan MUST include:
   `End-to-end: \`<command>\` → expected: <what the user sees>`: the product run
   the way a user runs it (the CLI, the server and a request, the app; for iOS,
   the simulator). Without it the result is "not verified end to end", never
-   green.
+   green. The command passes the criteria runner's allowlist — the product's
+   own binary does not, so the run is wrapped in a script:
+   `End-to-end: \`bash tests/e2e.sh\``. The same command is also listed under
+   `## Verification`: Phase 7 of `/trekexecute` runs that section, and a passed
+   criterion that runs the step's `End-to-end:` command is what verifies the
+   run end to end.
 10. **Execution Strategy** — For plans with > 5 steps: group steps into sessions
    (3–5 steps each), organize sessions into waves (parallel where independent),
    specify scope fences per session. Omit for plans with ≤ 5 steps.
@ -726,8 +737,10 @@ Write the plan following the template structure. The plan MUST include:
   The command must also be one the criteria runner is allowed to run: a test
   runner (`npm test`, `npm run <script>`, `node --test <file>`, `vitest`,
   `jest`, `pytest`, `python -m pytest`, `uv run pytest`, `cargo test`,
-   `go test`, `make test`), `bash <script under tests/>`, or a read-only `git`
-   subcommand — with no pipe, redirect or `&&`. Anything else is reported NOT
+   `go test`, `make test`), `bash <script under tests/>` (also
+   `/bin/bash <script under tests/>`, to pin the system shell), or a read-only
+   `git` subcommand (`status`, `log`, `diff`, `show`, `ls-files` — not `git
+   grep`) — with no pipe, redirect or `&&`. Anything else is reported NOT
   RUN, never executed and never failed. A check that needs more than that is
   wrapped in a script under `tests/` and declared as `bash tests/<script>.sh`.
 9. **Estimated Scope** — File counts and complexity rating.
--- a/docs/HANDOVER-CONTRACTS.md
+++ b/docs/HANDOVER-CONTRACTS.md
@ -99,6 +99,7 @@ Optional but standard sections: `## Non-Goals`, `## Constraints`, `## Preference
 | **v5.1 sequencing gate** | every read | `brief_version ≥ 2.1` requires `phase_signals` (list) OR `phase_signals_partial: true` — error `BRIEF_V51_MISSING_SIGNALS` on miss. Validator-only enforcement; commands surface, don't re-enforce. |
 | **v5.5 framing gate** | every read | `framing` enum-checked on any version when present (`BRIEF_INVALID_FRAMING`); at `brief_version ≥ 2.2` a missing `framing` → `BRIEF_MISSING_FRAMING` and a missing `## TL;DR` → `BRIEF_MISSING_SECTION` (strict) / warning (soft). `trekreview` briefs are exempt. |
 | Body sections | strict only | All `BRIEF_BODY_SECTIONS` present (`## TL;DR` added at ≥ 2.2) |
+| Runnable criteria | every read | A `## Success Criteria` bullet whose first backticked span is a command the review's criteria runner never runs (the product's own binary, `jq`, `grep`, a pipe) → warning `BRIEF_SC_NOT_RUNNABLE`, never an error: a criterion a human judges stays allowed. Prose-only criteria are not flagged. `trekreview` briefs are exempt. No schema change. |

 **State machine** detail: a brief that says it has research topics but skipped them must explicitly admit it (via `brief_quality: partial`). This is the most common failure mode the validator catches.

@ -128,6 +129,7 @@ Optional but standard sections: `## Non-Goals`, `## Constraints`, `## Preference
 - `BRIEF_INVALID_EFFORT` → strict halt; effort ∉ `[low, standard, high]`.
 - `BRIEF_INVALID_MODEL` → strict halt; model ∉ `BASE_ALLOWED_MODELS` (currently `[sonnet, opus, fable]`).
 - `BRIEF_SIGNALS_MUTUALLY_EXCLUSIVE` → strict halt; cannot set both `phase_signals` and `phase_signals_partial: true`.
+- `BRIEF_SC_NOT_RUNNABLE` → warning in both modes; `/trekbrief` names each one and offers a script under `tests/` the criterion runs (`bash tests/<script>.sh`).

 **Compatibility direction of the v5.9 allowlist widening (`fable`):** enum
 widening is safe for new readers of old data, not old readers of new data.
@ -281,6 +283,10 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
 | `PLAN_STEP_NO_TEST_FIRST` | 1.8: a step lacks `Test first:` with File + Verifies | Add the test the step makes pass |
 | `PLAN_NO_END_TO_END` | 1.8: no step carries `End-to-end:` with a command | Add the "run the thing" step |
 | `PLAN_STEP_FIELD_EMPTY` | 1.8: `Interface in/out:` or `Review focus:` present but empty | Fill or remove it |
+| `PLAN_VERIFICATION_NOT_RUNNABLE` | 1.8: a `## Verification` criterion has no command, or one the criteria runner's allowlist never runs | Name a test runner, or wrap the check as `bash tests/<script>.sh` |
+| `PLAN_STEP_VERIFY_NOT_RUNNABLE` | 1.8: the first backticked span of a step's `Verify:` (its test command) is not one runnable test-runner call | Put the test command first; move other checks to a script under `tests/` |
+| `PLAN_END_TO_END_NOT_RUNNABLE` | 1.8: an `End-to-end:` command the allowlist never runs (e.g. the product's own binary) | Run the product from `bash tests/e2e.sh` |
+| `PLAN_END_TO_END_NOT_VERIFIED` | 1.8: an `End-to-end:` command does not also stand under `## Verification` | List the same command there; Phase 7 runs that section |
 | `PLAN_CEREMONY_INVALID` | 1.8: `ceremony` not small / bounded / architectural | Copy the brief's value |
 | `PLAN_DESIGN_MISSING` / `PLAN_DESIGN_SECTION_MISSING` | 1.8 architectural: no `## Design`, or a required subsection missing | Write it from `templates/design-template.md` |
 | `PLAN_MODULE_MAP_EMPTY` / `PLAN_MODULE_CYCLE` | 1.8 architectural: no mermaid module map, or a cycle in it (all `mermaid` blocks of `### Module Map` form one map) | Draw it; break the cycle |
--- a/lib/validators/brief-validator.mjs
+++ b/lib/validators/brief-validator.mjs
@ -11,6 +11,7 @@ import { fileURLToPath } from 'node:url';
 import { parseDocument } from '../util/frontmatter.mjs';
 import { issue, ok, fail } from '../util/result.mjs';
 import { BASE_ALLOWED_MODELS } from './profile-validator.mjs';
+import { parseSuccessCriteria, runnableCheck } from '../verification/criteria-runner.mjs';

 export const BRIEF_REQUIRED_FRONTMATTER = ['type', 'brief_version', 'task', 'slug', 'research_topics', 'research_status'];
 export const REVIEW_AS_BRIEF_REQUIRED_FRONTMATTER = ['type', 'task', 'slug', 'project_dir', 'findings'];
@ -262,6 +263,26 @@ export function validateBriefContent(text, opts = {}) {
    }
  }

+  // A criterion that opens with a command the review's criteria runner can
+  // never run is reported NOT RUN at /trekreview, long after anyone could fix
+  // it: 11 of 13 in the voyage-lekerepo trial (2026-10-06), and nothing in
+  // /trekbrief warned. A warning, not an error: a criterion may be one a human
+  // judges by reading, and that stays allowed. Prose-only criteria (no
+  // command) are not flagged.
+  if (fm.type !== 'trekreview') {
+    for (const c of parseSuccessCriteria(body)) {
+      if (!c.command) continue;
+      const r = runnableCheck(c, { cwd: opts.cwd });
+      if (r.runnable) continue;
+      warnings.push(issue(
+        'BRIEF_SC_NOT_RUNNABLE',
+        `Success criterion ${c.label} opens with \`${c.command}\`, which the review's criteria runner will never run (${r.reason})`,
+        'Wrap the check in a script under tests/ and write the criterion as `bash tests/<script>.sh` exits 0, ' +
+          'or name a test runner (`npm test`, `node --test <file>`); otherwise it is reported NOT RUN at review.',
+      ));
+    }
+  }
+
  if (typeof fm.brief_version === 'string' || typeof fm.brief_version === 'number') {
    const m = String(fm.brief_version).match(/^(\d+)\.(\d+)$/);
    if (!m) {
--- a/lib/validators/plan-validator.mjs
+++ b/lib/validators/plan-validator.mjs
@ -15,7 +15,12 @@
 //   - the optional step fields `Interface in:`, `Interface out:`,
 //     `Review focus:` are never empty when present;
 //   - `ceremony:` is small | bounded | architectural (absent = architectural);
-//     an architectural plan carries `## Design` (see checkDesign).
+//     an architectural plan carries `## Design` (see checkDesign);
+//   - every check the run will execute can run (checkRunnable): each
+//     `## Verification` criterion, each step's `Verify:` (its first backticked
+//     span is the step's test command) and each `End-to-end:` command pass the
+//     criteria runner's allowlist, and each `End-to-end:` command also stands
+//     under `## Verification`, where Phase 7 runs it.
 // Strict mode reports these as errors; --soft downgrades them to warnings.

 import { readFileSync, existsSync, realpathSync } from 'node:fs';
@ -28,6 +33,7 @@ import {
 import { validateAllManifests } from '../parsers/manifest-yaml.mjs';
 import { parseDocument } from '../util/frontmatter.mjs';
 import { issue, fail } from '../util/result.mjs';
+import { parsePlanVerification, firstCommand, runnableCheck } from '../verification/criteria-runner.mjs';

 export const OPTIONAL_STEP_FIELDS = Object.freeze(['Interface in', 'Interface out', 'Review focus']);
 export const DESIGN_SUBSECTIONS = Object.freeze([
@ -69,6 +75,56 @@ function checkSteps(sections) {
  return out;
 }

+const WAY_IN = 'A check the allowlist has no name for (a pipe, `jq`, `grep`, the product\'s own binary) is ' +
+  'wrapped in a script under tests/ and declared as `bash tests/<script>.sh`.';
+
+/**
+ * Every check the run will execute can run. Without this a plan passed the
+ * validator, the plan gate and three critic rounds with a `git grep` criterion
+ * the runner refuses, and Phase 7 felled a run whose steps had all passed
+ * (voyage-lekerepo trial, 2026-10-06).
+ */
+function checkRunnable(text, sections, opts) {
+  const out = [];
+  const criteria = parsePlanVerification(text);
+  for (const c of criteria) {
+    const r = runnableCheck(c, opts);
+    if (!r.runnable) {
+      out.push(issue('PLAN_VERIFICATION_NOT_RUNNABLE',
+        `## Verification ${c.label} can never run (${r.reason}): ${c.text}`,
+        `Phase 7 runs every criterion and fells the run on one it cannot run. ${WAY_IN}`));
+    }
+  }
+  const verified = new Set(criteria.filter(c => c.command).map(c => c.command.trim().replace(/\s+/g, ' ')));
+  for (const s of sections) {
+    const f = parseStepFields(s.body);
+    if (f.has('Verify')) {
+      const v = firstCommand(f.get('Verify').text);
+      const r = runnableCheck(v, opts);
+      if (!r.runnable) {
+        out.push(issue('PLAN_STEP_VERIFY_NOT_RUNNABLE',
+          `Step ${s.n}: \`Verify:\` does not open with one runnable test command (${r.reason})`,
+          'The first backticked span of `Verify:` is the step\'s test command: proof of red and the done line run it, ' +
+            `and nothing after it is run. ${WAY_IN}`));
+      }
+    }
+    if (f.has('End-to-end') && BACKTICKED.test(f.get('End-to-end').text)) {
+      const e = firstCommand(f.get('End-to-end').text);
+      const r = runnableCheck(e, opts);
+      if (!r.runnable) {
+        out.push(issue('PLAN_END_TO_END_NOT_RUNNABLE',
+          `Step ${s.n}: \`End-to-end:\` command can never run (${r.reason})`,
+          `Run the product from a script: \`bash tests/e2e.sh\`. ${WAY_IN}`));
+      } else if (!verified.has(e.command.trim().replace(/\s+/g, ' '))) {
+        out.push(issue('PLAN_END_TO_END_NOT_VERIFIED',
+          `Step ${s.n}: \`End-to-end:\` command \`${e.command}\` does not stand under ## Verification`,
+          'Phase 7 runs `## Verification`; list the same command there, or the result is "not verified end to end".'));
+      }
+    }
+  }
+  return out;
+}
+
 /** Table body rows (skips the header and the |---| separator). */
 function tableRows(text) {
  const rows = text.split('\n').filter(l => l.trim().startsWith('|'));
@ -176,7 +232,7 @@ export function validatePlanContent(rawText, opts = {}) {

  const ceremony = extractPlanCeremony(text);
  if (versionAtLeast(planVersion, '1.8')) {
-    const v18 = [...checkSteps(sections)];
+    const v18 = [...checkSteps(sections), ...checkRunnable(text, sections, { cwd: opts.cwd })];
    if (!ceremony.valid) {
      v18.push(issue('PLAN_CEREMONY_INVALID', `ceremony=${ceremony.raw}`,
        `Allowed: ${CEREMONY_VALUES.join(' | ')} (absent = architectural).`));
--- a/lib/verification/criteria-runner.mjs
+++ b/lib/verification/criteria-runner.mjs
@ -38,6 +38,7 @@ import { readFileSync } from 'node:fs';
 import { resolve, dirname, join } from 'node:path';
 import { fileURLToPath } from 'node:url';
 import { spawnSync } from 'node:child_process';
+import { sliceSteps, parseStepFields } from '../parsers/plan-schema.mjs';

 const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..', '..');

@ -135,7 +136,8 @@ export function looksLikeCommand(span) {

 // The first backtick-delimited span on the line is the command by convention —
 // both templates put it first and a second span holds the expected output.
-function firstCommand(text) {
+// A plan step's `Verify:` and `End-to-end:` fields follow the same convention.
+export function firstCommand(text) {
  const m = text.match(/`([^`]+)`/);
  if (!m) return { command: null, reason: 'no-command' };
  const raw = m[1].trim();
@ -273,10 +275,12 @@ function npmScripts(cwd) {
 // One entry per runner: given the command's tokens (and the working tree the
 // criteria run in), either allow it or say why not.
 const RUNNERS = {
-  npm: (t, cwd) => {
+  npm: (t, cwd, writeTime) => {
    if (t[1] === 'test') return permit();
    if (t[1] !== 'run') return deny(`npm, but not \`npm test\` or \`npm run\` (\`npm ${t[1] ?? ''}\`)`);
    if (t[2] === undefined) return deny('npm run names no script');
+    // When a brief or plan is written, the script may be one the plan adds.
+    if (writeTime) return permit();
    return npmScripts(cwd).has(t[2])
      ? permit()
      : deny(`npm run names a script package.json does not declare (${t[2]})`);
@ -292,6 +296,8 @@ const RUNNERS = {
  go: (t) => (t[1] === 'test' ? permit() : deny('go, but not `go test`')),
  make: (t) => (t[1] === 'test' ? permit() : deny('make, but not `make test`')),
  bash: (t) => (isTestScript(t[1]) ? permit() : deny('bash, but not a script under tests/')),
+  // The system bash by its path (macOS: 3.2), so a check can pin the shell.
+  '/bin/bash': (t) => (isTestScript(t[1]) ? permit() : deny('/bin/bash, but not a script under tests/')),
  git: (t) =>
    READONLY_GIT.has(t[1])
      ? permit()
@ -304,6 +310,25 @@ const RUNNERS = {
 * the package.json found there.
 */
 export function allowedCommand(command, opts = {}) {
+  return screenShape(command, opts, false);
+}
+
+/**
+ * Whether a declared check could ever run, decided WITHOUT running it: the
+ * same allowlist, used when a brief or a plan is written instead of only when
+ * the run is judged. Measured in the voyage-lekerepo trial (2026-10-06): 11 of
+ * 13 brief criteria were NOT RUN at review, and a plan criterion (`git grep`)
+ * passed the validator, the plan gate and three critic rounds, then felled the
+ * run in Phase 7. An `npm run` script is not looked up here: the plan may be
+ * what adds it. Returns `{runnable, reason}`.
+ */
+export function runnableCheck(criterion, opts = {}) {
+  if (!criterion?.command) return { runnable: false, reason: criterion?.reason || 'no command' };
+  const r = screenShape(criterion.command, opts, true);
+  return { runnable: r.allowed, reason: r.reason };
+}
+
+function screenShape(command, opts, writeTime) {
  const cwd = resolve(opts.cwd ?? process.cwd());
  const text = String(command ?? '');
  if (text.trim() === '') return deny('the command is empty');
@ -312,9 +337,8 @@ export function allowedCommand(command, opts = {}) {
  if (operator) return deny(`a shell operator (${operator[0]}) — one criterion declares one command, unpiped`);

  const tokens = text.trim().split(/\s+/);
-  const runner = RUNNERS[tokens[0]];
-  if (!runner) return deny(`\`${tokens[0]}\` is not a known test runner`);
-  return runner(tokens, cwd);
+  if (!Object.hasOwn(RUNNERS, tokens[0])) return deny(`\`${tokens[0]}\` is not a known test runner`);
+  return RUNNERS[tokens[0]](tokens, cwd, writeTime);
 }

 // --- running ----------------------------------------------------------------
@ -398,17 +422,52 @@ export function summarize(results, opts = {}) {
  return summary;
 }

-function report(kind, source, criteria, opts, error) {
+/** The commands a plan's steps declare in `End-to-end:` (first backticked span each). */
+export function endToEndCommands(markdown) {
+  return sliceSteps(String(markdown ?? '').replace(/\r\n?/g, '\n'))
+    .map((s) => parseStepFields(s.body).get('End-to-end'))
+    .filter(Boolean)
+    .map((f) => firstCommand(f.text).command)
+    .filter(Boolean);
+}
+
+const normalizeCommand = (c) => String(c).trim().replace(/\s+/g, ' ');
+const SAYS_END_TO_END = /end to end|end-to-end|run the thing/i;
+
+/**
+ * Whether the run was verified end to end (P9 (6)). A passed criterion
+ * verifies it when it runs a step's `End-to-end:` command — the field the plan
+ * validator requires — or, the older form, when its text says it runs the
+ * thing end to end. Searching the text alone missed a plan that had the field
+ * and ran it green (voyage-lekerepo trial, 2026-10-06). A verification that
+ * did not pass is never verified end to end.
+ */
+export function endToEndStatus(results, summary, e2eCommands = []) {
+  const no = { status: 'not-verified', by: null, criterion: null };
+  if (!summary.ok) return no;
+  const passed = results.filter((r) => r.status === 'passed');
+  const want = new Set(e2eCommands.map(normalizeCommand));
+  const byField = passed.find((r) => r.command && want.has(normalizeCommand(r.command)));
+  if (byField) return { status: 'verified', by: 'end-to-end-field', criterion: byField.label };
+  const byText = passed.find((r) => SAYS_END_TO_END.test(r.text));
+  if (byText) return { status: 'verified', by: 'criterion-text', criterion: byText.label };
+  return no;
+}
+
+function report(kind, source, criteria, opts, error, e2eCommands = []) {
  const requireCommand = kind === 'plan';
  const results = error ? [] : runCriteria(criteria, opts);
-  return {
+  const summary = summarize(results, { requireCommand });
+  const rep = {
    kind,
    source,
    heading: kind === 'plan' ? PLAN_HEADING : BRIEF_HEADING,
    results,
-    summary: summarize(results, { requireCommand }),
+    summary,
    error: error ?? null,
  };
+  if (kind === 'plan') rep.end_to_end = endToEndStatus(results, summary, e2eCommands);
+  return rep;
 }

 /**
@ -460,7 +519,7 @@ export function runPlanVerification(planPath, opts = {}) {
  if (criteria.length === 0) {
    return report('plan', planPath, [], opts, noCriteria(planPath, PLAN_HEADING, section.line));
  }
-  return report('plan', planPath, criteria, opts);
+  return report('plan', planPath, criteria, opts, null, endToEndCommands(md));
 }

 /**
@ -502,6 +561,12 @@ export function render(rep) {
      for (const line of r.output.split('\n').slice(0, 20)) out.push(`      ${line}`);
    }
  }
+  if (rep.end_to_end) {
+    const e = rep.end_to_end;
+    out.push(e.status === 'verified'
+      ? `  end to end: verified (${e.criterion}, ${e.by === 'end-to-end-field' ? "runs a step's End-to-end command" : 'its text says it runs the thing'})`
+      : '  end to end: not verified end to end');
+  }
  return out.join('\n');
 }

--- a/templates/plan-template.md
+++ b/templates/plan-template.md
@ -116,7 +116,7 @@ break, quoted verbatim). A field that is present is never empty.
  - File: `path/to/test.ts` *(existing | new)*
  - Verifies: {what the test checks}
  - Pattern: `path/to/existing-test.ts`
- **End-to-end:** `{the command that runs the product the way a user does — the app, the CLI, the simulator}` → expected: {what the user sees}
+- **End-to-end:** `{a command the criteria runner allows that runs the product the way a user does — the app, the CLI, the simulator; usually bash tests/e2e.sh}` → expected: {what the user sees} *(the same command also stands under ## Verification)*
 - **Verify:** `{exact command}` → expected: `{output}`
 - **On failure:** {revert | retry | skip | escalate} — {specific instructions}
 - **Checkpoint:** `git commit -m "{conventional commit message}"`
--- a/tests/fixtures/jsonl-schemas.md
+++ b/tests/fixtures/jsonl-schemas.md
@ -22,7 +22,7 @@
 | trekbrief-stats | ts, task, slug, mode, interview_turns, review_iterations, brief_quality, research_topics, auto_research, auto_result, intent_approved, project_dir, order_id, order_claimed_at, order_reason | commands/trekbrief.md (orchestrator-emit Phase 7; order_* added by lib/stats/stats-append.mjs) | trekbrief.md:1124-1139 | profile, phase_models, profile_source | none |
 | trekresearch-stats | ts, question, mode, scope, engine, slug, project_dir, brief_path, dimensions, dimensions_baseline, dimensions_baseline_preserved, effort, conv_turns, empty_turns, unique_sources, agents_local, agents_external, gemini_used, confidence, contradictions, open_questions | commands/trekresearch.md (orchestrator-emit Stats tracking) | trekresearch.md:634-676 | profile, phase_models, parallel_agents, external_research_enabled, profile_source | none |
 | trekplan-stats | ts, task, mode, slug, brief_path, project_dir, codebase_size, codebase_files, agents_deployed, deep_dives, research_briefs_used, research_scout_used, critic_verdict, guardian_verdict, outcome | commands/trekplan.md (orchestrator-emit Phase 12) | trekplan.md:805-826 | profile, phase_models, parallel_agents, profile_source | none |
-| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2325-2347 | profile, phase_models, profile_source | none |
+| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2330-2352 | profile, phase_models, profile_source | none |
 | trekexecute-stats (autonomy events) | ts, event, known_event, payload | lib/stats/event-emit.mjs `emit()` | event-emit.mjs:78-99 | payload.profile, payload.phase_models, payload.profile_source | none |
 | trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
 | trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | commands/trekreview.md (orchestrator-emit Phase 8) | trekreview.md:255 | profile, phase_models, profile_source | none |
--- a/tests/helpers/plan-v18-builder.mjs
+++ b/tests/helpers/plan-v18-builder.mjs
@ -99,7 +99,7 @@ export function buildPlan(o = {}) {
  const header = ceremony === null
    ? `> Generated by trekplan v6.1.1 on 2026-10-06 — \`plan_version: ${version}\``
    : `> Generated by trekplan v6.1.1 on 2026-10-06 — \`plan_version: ${version}\` · \`ceremony: ${ceremony}\``;
-  const stepOpts = o.steps ?? [{}, { endToEnd: 'node src/cli.mjs --demo' }];
+  const stepOpts = o.steps ?? [{}, { endToEnd: 'bash tests/e2e.sh' }];
  const parts = [
    '# Test plan',
    '',
@ -123,6 +123,8 @@ export function buildPlan(o = {}) {
    '## Verification',
    '',
    '- [ ] `npm test` → expected: exit 0',
+    // Each End-to-end command also stands here: Phase 7 runs this section.
+    ...[...new Set(stepOpts.filter(s => s.endToEnd).map(s => s.endToEnd))].map(c => `- [ ] \`${c}\` → expected: exit 0`),
    '',
  );
  return parts.join('\n');
--- a/tests/validators/plan-gate.test.mjs
+++ b/tests/validators/plan-gate.test.mjs
@ -73,7 +73,7 @@ test('K3: no critic verdict → 1 (the plan has not been through the critic)', (
 });

 test('K3: an invalid plan → 1 even with a clean verdict', () => {
-  const bad = buildPlan({ steps: [{ testFirst: false }, { endToEnd: 'node x' }] });
+  const bad = buildPlan({ steps: [{ testFirst: false }, { endToEnd: 'bash tests/e2e.sh' }] });
  const g = gatePlan(rec(bad, dedupOut()).text);
  assert.equal(g.code, 1);
  assert.ok(g.reasons.some(x => x.code === 'PLAN_GATE_INVALID_PLAN'));
--- a/tests/validators/plan-validator-v18.test.mjs
+++ b/tests/validators/plan-validator-v18.test.mjs
@ -31,7 +31,7 @@ test('a complete 1.8 architectural plan validates strict', () => {
 });

 test('1.8: a step without `Test first:` is rejected', () => {
-  const r = validatePlanContent(buildPlan({ steps: [{ testFirst: false }, { endToEnd: 'node src/cli.mjs' }] }));
+  const r = validatePlanContent(buildPlan({ steps: [{ testFirst: false }, { endToEnd: 'bash tests/e2e.sh' }] }));
  assert.equal(r.valid, false);
  assert.ok(codes(r).includes('PLAN_STEP_NO_TEST_FIRST'), codes(r).join(','));
  assert.match(r.errors.find(e => e.code === 'PLAN_STEP_NO_TEST_FIRST').message, /Step 1/);
@ -57,10 +57,10 @@ test('1.8: `End-to-end:` without a backticked command does not count', () => {
 test('1.8: K4 fields are optional, but an empty one is rejected', () => {
  const withFields = buildPlan({ steps: [
    { interfaceOut: '`run1(): number` read by Step 2', reviewFocus: 'empty input — pinned by `tests/mod1.test.mjs`' },
-    { interfaceIn: '`run1()` from Step 1', endToEnd: 'node src/cli.mjs' },
+    { interfaceIn: '`run1()` from Step 1', endToEnd: 'bash tests/e2e.sh' },
  ] });
  assert.deepEqual(validatePlanContent(withFields).errors, []);
-  const empty = buildPlan({ steps: [{ interfaceIn: '' }, { endToEnd: 'node src/cli.mjs' }] });
+  const empty = buildPlan({ steps: [{ interfaceIn: '' }, { endToEnd: 'bash tests/e2e.sh' }] });
  assert.ok(codes(validatePlanContent(empty)).includes('PLAN_STEP_FIELD_EMPTY'));
 });

@ -236,7 +236,7 @@ test('CLI --strict exits 1 on a 1.8 plan missing Test first; exits 0 through a s
  const dir = mkdtempSync(join(tmpdir(), 'plan v18 '));
  try {
    const bad = join(dir, 'bad.md');
-    writeFileSync(bad, buildPlan({ steps: [{ testFirst: false }, { endToEnd: 'node x' }] }));
+    writeFileSync(bad, buildPlan({ steps: [{ testFirst: false }, { endToEnd: 'bash tests/e2e.sh' }] }));
    let code = 0;
    try { execFileSync('node', [VALIDATOR, '--strict', '--json', bad], { stdio: 'pipe' }); } catch (e) { code = e.status; }
    assert.equal(code, 1);
--- a/tests/validators/runnable-checks.test.mjs
+++ b/tests/validators/runnable-checks.test.mjs
@ -0,0 +1,197 @@
+// tests/validators/runnable-checks.test.mjs
+// A check that can never run is caught when the brief or the plan is written,
+// not when the run is judged (voyage-lekerepo end-to-end trial, 2026-10-06,
+// report points 2, 3, 4 and 9):
+//   - 11 of 13 brief success criteria were NOT RUN at review, and nothing in
+//     /trekbrief warned;
+//   - a plan whose `## Verification` held `git grep ...` passed the validator,
+//     the plan gate and three critic rounds, then felled the run in Phase 7;
+//   - the plan's `End-to-end:` field and Phase 7's end-to-end check did not
+//     meet: Phase 7 only searched the criterion TEXT for "end to end";
+//   - a step's `Verify:` was prose with several commands, and nothing said
+//     which one the executor runs.
+// The runner's own allowlist decides "can run" — one list, used early and late.
+
+import { test } from 'node:test';
+import { strict as assert } from 'node:assert';
+import { mkdtempSync, writeFileSync, rmSync } from 'node:fs';
+import { join } from 'node:path';
+import { tmpdir } from 'node:os';
+import { validatePlanContent } from '../../lib/validators/plan-validator.mjs';
+import { validateBriefContent } from '../../lib/validators/brief-validator.mjs';
+import {
+  allowedCommand, runnableCheck, endToEndCommands, runPlanVerification,
+} from '../../lib/verification/criteria-runner.mjs';
+import { buildPlan } from '../helpers/plan-v18-builder.mjs';
+
+const codes = (list) => list.map((e) => e.code);
+
+// --- the shared screen -------------------------------------------------------
+
+test('runner: `/bin/bash tests/<script>.sh` is the same declared check as `bash tests/<script>.sh`', () => {
+  assert.equal(allowedCommand('/bin/bash tests/e2e.sh').allowed, true);
+  assert.equal(allowedCommand('/bin/bash scripts/deploy.sh').allowed, false);
+});
+
+test('runner: at write time an `npm run` script may still be added by the plan itself', () => {
+  const empty = mkdtempSync(join(tmpdir(), 'voyage-runnable-'));
+  try {
+    assert.equal(allowedCommand('npm run e2e', { cwd: empty }).allowed, false);
+    assert.equal(runnableCheck({ command: 'npm run e2e' }, { cwd: empty }).runnable, true);
+  } finally {
+    rmSync(empty, { recursive: true, force: true });
+  }
+});
+
+test('runner: runnableCheck says why a check can never run', () => {
+  const r = runnableCheck({ command: 'git grep -q -e --max -- README.md' });
+  assert.equal(r.runnable, false);
+  assert.match(r.reason, /read-only/);
+  assert.equal(runnableCheck({ command: null, reason: 'no-command' }).runnable, false);
+  assert.equal(runnableCheck({ command: 'npm test' }).runnable, true);
+});
+
+// --- the plan (point 3, 4, 9) ------------------------------------------------
+
+test('plan 1.8: a `## Verification` criterion the runner can never run is an error, named', () => {
+  const plan = buildPlan().replace(
+    '- [ ] `npm test` → expected: exit 0',
+    '- [ ] `npm test` → expected: exit 0\n- [ ] `git grep -q -e --max -- README.md` → expected: exit 0',
+  );
+  const r = validatePlanContent(plan);
+  assert.equal(r.valid, false);
+  const e = r.errors.find((x) => x.code === 'PLAN_VERIFICATION_NOT_RUNNABLE');
+  assert.ok(e, codes(r.errors).join(','));
+  assert.match(e.message, /git grep/);
+  assert.match(e.hint, /bash tests\//);
+});
+
+test('plan 1.8: a `## Verification` bullet with no command is an error (a plan promises commands)', () => {
+  const plan = buildPlan().replace(
+    '- [ ] `npm test` → expected: exit 0',
+    '- [ ] `npm test` → expected: exit 0\n- [ ] the README mentions the new flag',
+  );
+  assert.ok(codes(validatePlanContent(plan).errors).includes('PLAN_VERIFICATION_NOT_RUNNABLE'));
+});
+
+test('plan 1.8: an `End-to-end:` command the runner can never run is an error', () => {
+  const r = validatePlanContent(buildPlan({ steps: [{}, { endToEnd: 'node src/cli.mjs --demo' }] }));
+  assert.ok(codes(r.errors).includes('PLAN_END_TO_END_NOT_RUNNABLE'), codes(r.errors).join(','));
+});
+
+test('plan 1.8: an `End-to-end:` command that does not also stand under `## Verification` is an error', () => {
+  const plan = buildPlan().replace('\n- [ ] `bash tests/e2e.sh` → expected: exit 0', '');
+  assert.ok(!plan.includes('- [ ] `bash tests/e2e.sh`'), 'fixture: the builder lists the end-to-end command');
+  const r = validatePlanContent(plan);
+  assert.ok(codes(r.errors).includes('PLAN_END_TO_END_NOT_VERIFIED'), codes(r.errors).join(','));
+});
+
+test('plan 1.8: a step whose `Verify:` does not open with one runnable test command is an error', () => {
+  const plan = buildPlan().replace(
+    '- **Verify:** `node --test tests/mod1.test.mjs` → expected: pass',
+    '- **Verify:** `node bin/slug.mjs 2>&1 | grep -q -- --max` → expected: exit 0',
+  );
+  const r = validatePlanContent(plan);
+  const e = r.errors.find((x) => x.code === 'PLAN_STEP_VERIFY_NOT_RUNNABLE');
+  assert.ok(e, codes(r.errors).join(','));
+  assert.match(e.message, /Step 1/);
+});
+
+test('plan 1.8: a step whose `Verify:` opens with a test command and then describes output is fine', () => {
+  const plan = buildPlan().replace(
+    '- **Verify:** `node --test tests/mod1.test.mjs` → expected: pass',
+    '- **Verify:** `node --test tests/mod1.test.mjs` → expected: `fail 0`',
+  );
+  assert.deepEqual(validatePlanContent(plan).errors, []);
+});
+
+test('plan 1.8, soft: the runnability findings are warnings', () => {
+  const r = validatePlanContent(buildPlan({ steps: [{}, { endToEnd: 'node src/cli.mjs' }] }), { strict: false });
+  assert.ok(codes(r.warnings).includes('PLAN_END_TO_END_NOT_RUNNABLE'));
+  assert.ok(!codes(r.errors).includes('PLAN_END_TO_END_NOT_RUNNABLE'));
+});
+
+test('plan 1.7: none of the runnability rules apply', () => {
+  const plan = buildPlan({ version: '1.7', steps: [{}, { endToEnd: 'node src/cli.mjs' }] })
+    .replace('- [ ] `npm test` → expected: exit 0', '- [ ] `git grep x` → expected: exit 0');
+  const r = validatePlanContent(plan);
+  for (const c of ['PLAN_VERIFICATION_NOT_RUNNABLE', 'PLAN_END_TO_END_NOT_RUNNABLE', 'PLAN_END_TO_END_NOT_VERIFIED', 'PLAN_STEP_VERIFY_NOT_RUNNABLE']) {
+    assert.ok(!codes(r.errors).includes(c) && !codes(r.warnings).includes(c), c);
+  }
+});
+
+// --- Phase 7 reads the End-to-end field (point 4) ----------------------------
+
+function withPlan(text, fn) {
+  const dir = mkdtempSync(join(tmpdir(), 'voyage-e2e-'));
+  try {
+    const p = join(dir, 'plan.md');
+    writeFileSync(p, text);
+    return fn(p);
+  } finally {
+    rmSync(dir, { recursive: true, force: true });
+  }
+}
+
+const passAll = { exec: () => ({ status: 0, stdout: '', stderr: '' }), screen: () => ({ allowed: true, rule: '' }) };
+
+test('runner: the plan\'s `End-to-end:` commands are read from its steps', () => {
+  assert.deepEqual(endToEndCommands(buildPlan()), ['bash tests/e2e.sh']);
+});
+
+test('runner: a passed criterion that runs the step\'s End-to-end command verifies end to end — no magic words needed', () => {
+  withPlan(buildPlan(), (p) => {
+    const rep = runPlanVerification(p, passAll);
+    assert.equal(rep.summary.ok, true);
+    assert.equal(rep.end_to_end.status, 'verified');
+    assert.equal(rep.end_to_end.by, 'end-to-end-field');
+  });
+});
+
+test('runner: the older text form still verifies (session specs, plans below 1.8)', () => {
+  const plan = buildPlan({ steps: [{}, {}] }).replace(
+    '- [ ] `npm test` → expected: exit 0',
+    '- [ ] `npm test` → expected: exit 0 — runs the CLI end to end',
+  );
+  withPlan(plan, (p) => {
+    assert.equal(runPlanVerification(p, passAll).end_to_end.status, 'verified');
+  });
+});
+
+test('runner: a failing verification is never verified end to end', () => {
+  withPlan(buildPlan(), (p) => {
+    const rep = runPlanVerification(p, { ...passAll, exec: (cmd) => ({ status: cmd === 'npm test' ? 1 : 0, stdout: '', stderr: '' }) });
+    assert.equal(rep.summary.ok, false);
+    assert.equal(rep.end_to_end.status, 'not-verified');
+  });
+});
+
+// --- the brief (point 2) -----------------------------------------------------
+
+function brief(criteria) {
+  return [
+    '---', 'type: trekbrief', 'brief_version: "2.2"', 'task: t', 'slug: s',
+    'research_topics: 0', 'research_status: skipped', 'framing: new-direction', 'phase_signals_partial: true', '---', '',
+    '## TL;DR', '', 'A toy.', '', '## Intent', '', 'Why.', '', '## Goal', '', 'What.', '',
+    '## Success Criteria', '', ...criteria, '',
+  ].join('\n');
+}
+
+test('brief: a success criterion whose command the runner can never run is a warning that names it and the way in', () => {
+  const r = validateBriefContent(brief([
+    '- `npm test` exits 0.',
+    '- `node bin/slug.mjs --max 9 "Hello Big World"` prints `hello-big`.',
+    '- The README explains the flag.',
+  ]));
+  assert.equal(r.valid, true, JSON.stringify(r.errors));
+  const w = r.warnings.filter((x) => x.code === 'BRIEF_SC_NOT_RUNNABLE');
+  assert.equal(w.length, 1, JSON.stringify(r.warnings));
+  assert.match(w[0].message, /SC2/);
+  assert.match(w[0].message, /node bin\/slug\.mjs/);
+  assert.match(w[0].hint, /bash tests\//);
+});
+
+test('brief: runnable and prose-only criteria raise no runnability warning', () => {
+  const r = validateBriefContent(brief(['- `npm test` exits 0.', '- `bash tests/e2e.sh` exits 0.', '- The output reads well.']));
+  assert.ok(!codes(r.warnings).includes('BRIEF_SC_NOT_RUNNABLE'), JSON.stringify(r.warnings));
+});