fix(review): run the success-criteria commands and hand the reviewer the result (D-04)

The rubric required `brief-conformance-reviewer` to classify a Success
Criterion as Full only when "its verification command/test exists and passes".
Its tools are `Read`, `Glob`, `Grep`. It cannot run anything, so "passes" was
either guessed from the command's mere existence or quietly downgraded to
"exists" — a BLOCKER-tier rule key resting on an impression.

The reviewer stays read-only — a reviewer that executes the code it reviews is
not an independent reviewer. The command does the running instead:

- `/trekreview` Phase 4.5 runs the brief's `## Success Criteria` commands
  through `lib/verification/criteria-runner.mjs --brief --evidence` and captures
  the block as `sc_evidence_block`, pasted verbatim into the reviewer prompt in
  Phase 5. The exit code does not stop the review — a failing criterion is
  exactly what the review exists to find.
- `formatCriteriaEvidence` builds that block in code: one row per criterion with
  the command, the exit code and the first output line. Chose a code-built block
  over an orchestrator-written summary so the orchestrator cannot narrate a pass
  that never happened.
- The rubric now judges the supplied result: `PASS` supports Full, `FAILED` /
  `BLOCKED` is `Broken` with the exit code cited, and `NOT RUN` is the absence
  of a measurement — never evidence in either direction.
- Phase 4.5 is skipped in `quick` mode: that mode does not launch the
  conformance reviewer, so there is nobody to hand the result to.

Red first: seven tests in `tests/lib/criteria-runner.test.mjs` against a
committed brief fixture whose three criteria pass, fail, and are prose-only.
The two doc pins were verified red against the pre-fix files (rubric asked
"exists and passes"; no Phase 4.5; the block reached nobody).

Suite: 1117 (1115/0/2), up 9.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

agents/brief-conformance-reviewer.md

@@ -50,6 +50,18 @@ You will receive a prompt containing:
 - **Rule catalogue** — the 12-key catalogue in
   `lib/review/rule-catalogue.mjs`. You may only emit findings whose
   `rule_key` is in this set.
+- **Success-criteria check results** — a table produced by /trekreview
+  (Phase 4.5) holding, per criterion, the command it declared, the exit
+  code it produced, and the first line of its output. /trekreview ran
+  them; you did not, and you cannot. This block is your ONLY evidence
+  about whether a criterion's verification passes.
+
+**You never run a command.** Your tools are `Read`, `Glob` and `Grep`, and
+that is deliberate — a reviewer that executes the code it reviews is not an
+independent reviewer. Where the check results say `NOT RUN`, no result
+exists: judge that criterion on delivered code alone and say so in the
+Evidence column. Never infer, assume or reconstruct an outcome, and never
+report a criterion as verified because its command merely EXISTS.
 
 ## Your process
 
@@ -73,10 +85,14 @@ context is needed) and classify coverage:
 
 | Coverage | Meaning | Finding emitted |
 |----------|---------|-----------------|
-| **Full** | Code change visibly implements the criterion AND its verification command/test exists and passes | none |
+| **Full** | Code change visibly implements the criterion AND its check result is `PASS` | none |
-| **Partial** | Some pieces present but the verification path is incomplete (e.g., the command exists but tests are missing) | `MISSING_TEST` (MAJOR) or step-specific finding |
+| **Partial** | Some pieces present but the verification path is incomplete (e.g., the check result is `NOT RUN` because the criterion declares no command, or the command exists but tests are missing) | `MISSING_TEST` (MAJOR) or step-specific finding |
 | **Missing** | No delivered code maps to this criterion | `UNIMPLEMENTED_CRITERION` (BLOCKER) |
-| **Broken** | Code claims to implement the criterion but the verification fails or is structurally wrong | `BROKEN_SUCCESS_CRITERION` (BLOCKER) |
+| **Broken** | The check result is `FAILED` or `BLOCKED`, or the code is structurally wrong for the criterion | `BROKEN_SUCCESS_CRITERION` (BLOCKER) |
+
+A `FAILED` result is decisive: cite its exit code and output line in the
+finding's `detail`. A `NOT RUN` result is NOT decisive in either direction —
+it is the absence of a measurement, so it can never support **Full**.
 
 Cite the criterion text in `brief_ref` (e.g., `SC3 — "review.md is
 parseable as input to /trekplan"`).
@@ -163,8 +179,8 @@ parsers find it by reading the last `json` code fence.
 
 | Criterion | Coverage | Evidence |
 |-----------|----------|----------|
-| SC1 — "..." | Full | lib/foo.mjs:23 implements; tests/foo.test.mjs covers |
+| SC1 — "..." | Full | lib/foo.mjs:23 implements; check `npm test` PASS (exit 0) |
-| SC2 — "..." | Missing | no implementation found in diff |
+| SC2 — "..." | Missing | no implementation found in diff; check NOT RUN (no command) |
 | NG1 — "..." | Honored | no diff matches forbidden pattern |
 | NG2 — "..." | Violated | lib/bar.mjs:88 implements forbidden behavior |
 
@@ -231,6 +247,10 @@ parsers find it by reading the last `json` code fence.
 - **Cite, don't speculate.** Every finding includes a `file:line`
   citation taken from the diff. No "this might break" without quoted
   evidence.
+- **A check you did not get is not a check that passed.** Quote the exit
+  code from the supplied results when you call a criterion verified or
+  broken. If the results block is absent or says no criterion was checked,
+  state that in the Coverage matrix rather than judging the verification.
 - **Respect the triage map.** Files marked `skip` are out of scope.
   Cross-file inference is the coordinator's job, not yours.
 - **No praise.** "Looks good", "well done", "no issues" do not appear in

commands/trekreview.md

@@ -188,6 +188,56 @@ drops are `COVERAGE_SILENT_SKIP` (MAJOR) per the rule catalogue.
 
 If `mode == dry-run`: print the triage map and exit.
 
+## Phase 4.5 — Run the brief's success-criteria checks
+
+**Skipped in `quick` mode** (that mode does not launch
+`brief-conformance-reviewer`, so there is nobody to hand the result to) and in
+`dry-run`.
+
+`brief-conformance-reviewer` is asked to decide whether each Success Criterion's
+verification command passes. Its tools are `Read`, `Glob`, `Grep` — it cannot
+run anything, and it stays that way: a reviewer that executes the code it
+reviews is not an independent reviewer. So THIS command runs the commands, and
+the reviewer judges the RESULT.
+
+```bash
+# Resolve the plugin root ONCE. ${CLAUDE_PLUGIN_ROOT} is substituted in this
+# command's text but is EMPTY in the Bash tool's process env, and a bare
+# `node ${CLAUDE_PLUGIN_ROOT}/lib/…` then runs `node /lib/…`, which exits 1 —
+# indistinguishable from a criterion that failed.
+VOYAGE_ROOT="${CLAUDE_PLUGIN_ROOT:-}"
+case "$VOYAGE_ROOT" in
+  /*) ;;
+  *) VOYAGE_ROOT="$(ls -d "$HOME"/.claude/plugins/cache/*/voyage 2>/dev/null | head -1)" ;;
+esac
+if [ ! -f "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" ]; then
+  echo "[voyage] success-criteria checks could not run - plugin root unresolved."
+  echo "         NOT a pass: hand the reviewer NO results and say so."
+  exit 2
+fi
+
+# Every command is screened through the executor denylist before it reaches a
+# shell; a blocked command is reported BLOCKED, never run. Foreground only.
+node "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" \
+  --brief "{brief_path}" --evidence --cwd "$(git rev-parse --show-toplevel)"
+```
+
+Exit 0 means every criterion passed; exit 1 means at least one failed, was
+blocked, or had no command; exit 2 means the runner could not run. **The exit
+code does not stop the review** — a failing criterion is exactly what the review
+exists to find. Capture stdout as `sc_evidence_block`.
+
+If the runner exits 2 (or the root could not be resolved), set
+`sc_evidence_block` to a single line naming the failure and stating that no
+criterion was checked. An unrun check is never a pass, and the reviewer must be
+told which of the two it is looking at.
+
+`sc_evidence_block` is pasted verbatim into the `brief-conformance-reviewer`
+prompt in Phase 5, and its summary line goes into the Coverage section of
+`review.md`. Do NOT summarise, re-word or re-judge it on the way — the block is
+built by `formatCriteriaEvidence` precisely so the orchestrator cannot narrate a
+pass that never happened.
+
 ## Phase 5 — Launch parallel reviewers
 
 **Substrate routing (opt-in `--workflow`).** When `workflow_substrate == true`, run
@@ -226,6 +276,11 @@ Each reviewer prompt includes:
 - **Brief path** — `{brief_path}` (read on demand; do not inline).
 - **Rule catalogue** — reference to `lib/review/rule-catalogue.mjs`.
 
+`brief-conformance-reviewer` additionally receives `sc_evidence_block` from
+Phase 4.5, pasted verbatim — the command, exit code and first line of output for
+every Success Criterion. It is the ONLY evidence that agent has about whether a
+criterion's verification passes, because it cannot run one.
+
 Collect each reviewer's trailing JSON block and **validate it against the
 reviewer-output schema** rather than merely parsing it. Run:
 

lib/verification/criteria-runner.mjs

@@ -24,7 +24,9 @@
 //
 // Usage:
 //   node lib/verification/criteria-runner.mjs --plan  <plan.md>  [--json] [--cwd D] [--timeout MS]
-//   node lib/verification/criteria-runner.mjs --brief <brief.md> [--json] [--cwd D] [--timeout MS]
+//   node lib/verification/criteria-runner.mjs --brief <brief.md> [--json|--evidence] [--cwd D] [--timeout MS]
+// --evidence emits the markdown block /trekreview hands to the conformance
+// reviewer; --json emits the whole report. They are two shapes of one run.
 // Exit: 0 = ok, 1 = a criterion failed / was blocked / could not run, 2 = usage
 // or read error (deliberately distinct from "a criterion failed").
 
@@ -265,15 +267,69 @@ export function render(rep) {
   return out.join('\n');
 }
 
+/**
+ * The evidence block /trekreview hands to `brief-conformance-reviewer`.
+ *
+ * The reviewer's tools are Read/Glob/Grep, so it cannot run a success
+ * criterion's verification command. Building this block in code (rather than
+ * letting the orchestrator narrate it) is what makes "passes" an exit code the
+ * reviewer READS instead of a judgement it cannot make.
+ */
+export function formatCriteriaEvidence(rep) {
+  const out = [];
+  out.push('### Success-criteria check results (run for you — you did not run these)');
+  out.push('');
+  out.push(`Source: \`${rep.source}\` ${rep.heading}`);
+  out.push('');
+  if (rep.error) {
+    out.push(`**${rep.error.code}** — ${rep.error.message}.`);
+    out.push('');
+    out.push('No criteria were run. Treat every criterion as NOT RUN.');
+  } else {
+    out.push('| Criterion | Command | Result | Exit | Evidence |');
+    out.push('|---|---|---|---|---|');
+    for (const r of rep.results) {
+      const cmd = r.command ? `\`${cell(r.command)}\`` : '(none)';
+      const exit = r.exitCode === null ? '—' : `exit ${r.exitCode}`;
+      const evidence = r.status === 'passed' ? '—' : cell(firstLine(r.output)) || '—';
+      out.push(`| ${r.label} | ${cmd} | ${MARK[r.status]} | ${exit} | ${evidence} |`);
+    }
+    const s = rep.summary;
+    out.push('');
+    out.push(
+      `${s.passed} passed · ${s.failed} failed · ${s.blocked} blocked · ${s.unrunnable} not run (of ${s.total}).`
+    );
+  }
+  out.push('');
+  out.push(
+    '**NOT RUN is not a pass.** You cannot run these commands yourself — never infer, assume or ' +
+    'reconstruct a result for a criterion that has none. A criterion whose result is FAILED or ' +
+    'BLOCKED is `Broken`; a criterion with no result is judged on delivered code alone, and you ' +
+    'say so in the Evidence column.'
+  );
+  return out.join('\n');
+}
+
+function firstLine(text) {
+  return String(text ?? '').split('\n').map((l) => l.trim()).find((l) => l !== '') ?? '';
+}
+
+// Markdown table cells: no pipes, no newlines, bounded length.
+function cell(text) {
+  const t = String(text ?? '').replace(/\r?\n/g, ' ').replace(/\|/g, '\\|').trim();
+  return t.length <= 120 ? t : `${t.slice(0, 120)}…`;
+}
+
 // --- CLI --------------------------------------------------------------------
 
 const USAGE =
-  'usage: criteria-runner.mjs (--plan <plan.md> | --brief <brief.md>) [--json] [--cwd <dir>] [--timeout <ms>]';
+  'usage: criteria-runner.mjs (--plan <plan.md> | --brief <brief.md>) [--json | --evidence] [--cwd <dir>] [--timeout <ms>]';
 
 export function main(argv) {
   let mode = null;
   let path = null;
   let json = false;
+  let evidence = false;
   const opts = {};
   for (let i = 0; i < argv.length; i++) {
     const a = argv[i];
@@ -282,11 +338,16 @@ export function main(argv) {
       mode = a.slice(2);
       path = argv[++i];
     } else if (a === '--json') json = true;
+    else if (a === '--evidence') evidence = true;
     else if (a === '--cwd' && argv[i + 1]) opts.cwd = resolve(argv[++i]);
     else if (a === '--timeout' && argv[i + 1]) opts.timeoutMs = Number(argv[++i]);
     else { process.stderr.write(`criteria-runner: unknown argument ${a}\n${USAGE}\n`); return 2; }
   }
   if (!mode) { process.stderr.write(`criteria-runner: no artifact given\n${USAGE}\n`); return 2; }
+  if (json && evidence) {
+    process.stderr.write(`criteria-runner: --json and --evidence are two output shapes; pick one\n${USAGE}\n`);
+    return 2;
+  }
   if (opts.timeoutMs !== undefined && !Number.isFinite(opts.timeoutMs)) {
     process.stderr.write(`criteria-runner: --timeout must be a number\n${USAGE}\n`);
     return 2;
@@ -299,7 +360,8 @@ export function main(argv) {
     process.stderr.write(`criteria-runner: ${err.message}\n`);
     return 2;
   }
-  process.stdout.write((json ? JSON.stringify(rep, null, 2) : render(rep)) + '\n');
+  const text = json ? JSON.stringify(rep, null, 2) : evidence ? formatCriteriaEvidence(rep) : render(rep);
+  process.stdout.write(text + '\n');
   return rep.summary.ok ? 0 : 1;
 }
 

tests/fixtures/brief-success-criteria.md

@@ -0,0 +1,30 @@
+---
+task: criteria-runner fixture — a brief whose success criteria are mixed
+slug: criteria-runner-fixture-brief
+brief_version: "2.2"
+framing: new-direction
+---
+
+# Task Brief: criteria-runner fixture
+
+Fixture only. Consumed by `tests/lib/criteria-runner.test.mjs`. It is the
+falsifying case for D-04: a read-only reviewer cannot run these commands, so
+/trekreview runs them and hands over the result.
+
+## TL;DR
+
+Three success criteria: one passes, one fails on purpose, one is prose only.
+
+## Goal
+
+Exercise the brief side of the criteria runner.
+
+## Success Criteria
+
+- The runner reaches the shell: `true` exits 0
+- The failing case is visible: `false` exits 0 (FAILS on purpose — exit 1)
+- No new runtime dependencies are introduced
+
+## Non-Goals
+
+- Being executed by the pipeline. This file is a test fixture.

tests/lib/criteria-runner.test.mjs

@@ -22,6 +22,8 @@ import {
   runCriteria,
   summarize,
   runPlanVerification,
+  runSuccessCriteriaChecks,
+  formatCriteriaEvidence,
   render,
 } from '../../lib/verification/criteria-runner.mjs';
 
@@ -278,3 +280,73 @@ test('CLI: no mode flag exits 2 — it never guesses which artifact it was given
   assert.equal(r.status, 2);
   assert.match(r.stderr, /usage/);
 });
+
+// --- the brief path: evidence handed to a read-only reviewer ----------------
+//
+// D-04: brief-conformance-reviewer is asked to judge whether a Success
+// Criterion's verification command "exists and passes", but its tools are
+// Read/Glob/Grep — it cannot run anything. /trekreview runs the commands and
+// hands over the RESULT, and the block it hands over is built by code so the
+// orchestrator cannot narrate a pass that never happened.
+
+test('runSuccessCriteriaChecks: each criterion gets a real result, prose ones are NOT RUN', () => {
+  const report = runSuccessCriteriaChecks(join(FIX, 'brief-success-criteria.md'));
+  assert.equal(report.kind, 'brief');
+  assert.deepEqual(report.results.map((r) => r.label), ['SC1', 'SC2', 'SC3']);
+  assert.equal(report.results[0].status, 'passed');
+  assert.equal(report.results[1].status, 'failed');
+  assert.equal(report.results[1].exitCode, 1);
+  assert.equal(report.results[2].status, 'unrunnable');
+  assert.equal(report.summary.ok, false, 'a failing criterion is never ok, in either mode');
+});
+
+test('runSuccessCriteriaChecks: a brief with no Success Criteria section reports the code', () => {
+  const dir = mkdtempSync(join(tmpdir(), 'criteria-runner-'));
+  const p = join(dir, 'brief.md');
+  writeFileSync(p, '# Brief\n\n## Goal\n\nSomething.\n');
+  const report = runSuccessCriteriaChecks(p);
+  assert.equal(report.error.code, 'NO_SUCCESS_CRITERIA_SECTION');
+  assert.deepEqual(report.results, []);
+});
+
+test('formatCriteriaEvidence: one row per criterion, with command and exit code', () => {
+  const report = runSuccessCriteriaChecks(join(FIX, 'brief-success-criteria.md'));
+  const block = formatCriteriaEvidence(report);
+  for (const label of ['SC1', 'SC2', 'SC3']) assert.match(block, new RegExp(`\\| ${label} \\|`));
+  assert.match(block, /PASS/);
+  assert.match(block, /FAILED/);
+  assert.match(block, /NOT RUN/);
+  assert.match(block, /exit 1/);
+});
+
+test('formatCriteriaEvidence: the block forbids inferring a pass for a criterion with no result', () => {
+  const report = runSuccessCriteriaChecks(join(FIX, 'brief-success-criteria.md'));
+  const block = formatCriteriaEvidence(report);
+  assert.match(block, /NOT RUN/);
+  assert.match(
+    block, /never.*(infer|assume)/i,
+    'the evidence block must state in-band that NOT RUN is not a pass — the reviewer cannot re-run it',
+  );
+});
+
+test('formatCriteriaEvidence: an unreadable section still yields a block that says so', () => {
+  const dir = mkdtempSync(join(tmpdir(), 'criteria-runner-'));
+  const p = join(dir, 'brief.md');
+  writeFileSync(p, '# Brief\n\n## Goal\n\nSomething.\n');
+  const block = formatCriteriaEvidence(runSuccessCriteriaChecks(p));
+  assert.match(block, /NO_SUCCESS_CRITERIA_SECTION/);
+  assert.match(block, /no criteria were run/i);
+});
+
+test('CLI: --evidence emits the reviewer block, and --brief still exits 1 on a failure', () => {
+  const r = cli(['--brief', join(FIX, 'brief-success-criteria.md'), '--evidence']);
+  assert.equal(r.status, 1, r.stderr);
+  assert.match(r.stdout, /\| SC1 \|/);
+  assert.match(r.stdout, /NOT RUN/);
+});
+
+test('CLI: --evidence and --json together exit 2 — one output shape at a time', () => {
+  const r = cli(['--brief', join(FIX, 'brief-success-criteria.md'), '--evidence', '--json']);
+  assert.equal(r.status, 2);
+  assert.match(r.stderr, /usage/);
+});

tests/lib/doc-consistency.test.mjs

@@ -1852,3 +1852,30 @@ test('D-03: trekexecute Phase 7 runs the plan Verification on the single-session
     'a failing criterion must fell the run, not be noted in the final report',
   );
 });
+
+// End-state defect D-04: the rubric required brief-conformance-reviewer to decide whether a
+// Success Criterion's verification command "exists and passes" — with Read/Glob/Grep. The
+// command now runs the commands and hands over the result; the reviewer stays read-only.
+// Fix the SOURCE.
+test('D-04: the conformance reviewer judges a supplied result, and never runs a command', () => {
+  const a = read('agents/brief-conformance-reviewer.md');
+  assert.ok(
+    !/exists and passes/.test(a),
+    'the rubric may no longer ask a Read/Glob/Grep agent whether a command passes — that ask IS defect D-04',
+  );
+  assert.match(a, /^tools: \["Read", "Glob", "Grep"\]$/m, 'the reviewer stays read-only: no Bash');
+  assert.match(a, /You never run a command/, 'the reviewer must be told in-band that it does not execute');
+  assert.match(a, /Success-criteria check results/, 'the reviewer must document the supplied evidence as an input');
+});
+
+test('D-04: trekreview runs the success-criteria commands and hands the reviewer the result', () => {
+  const t = read('commands/trekreview.md');
+  const phase = (t.split("\n## Phase 4.5 — ")[1] || '').split('\n## ')[0];
+  assert.ok(phase.length > 0, 'trekreview.md must carry the phase that runs the success-criteria checks');
+  assert.match(phase, /criteria-runner\.mjs" \\\n  --brief .* --evidence/, 'the phase must invoke the runner in --evidence mode');
+  assert.match(phase, /sc_evidence_block/, 'the phase must name the captured block the reviewer prompt carries');
+  assert.match(
+    t.split('\n## Phase 5 — ')[1] || '', /sc_evidence_block/,
+    'the reviewer-launch phase must hand the block over — a block nobody passes is not evidence',
+  );
+});