fix(review): run the success-criteria commands and hand the reviewer the result (D-04)

The rubric required `brief-conformance-reviewer` to classify a Success
Criterion as Full only when "its verification command/test exists and passes".
Its tools are `Read`, `Glob`, `Grep`. It cannot run anything, so "passes" was
either guessed from the command's mere existence or quietly downgraded to
"exists" — a BLOCKER-tier rule key resting on an impression.

The reviewer stays read-only — a reviewer that executes the code it reviews is
not an independent reviewer. The command does the running instead:

- `/trekreview` Phase 4.5 runs the brief's `## Success Criteria` commands
  through `lib/verification/criteria-runner.mjs --brief --evidence` and captures
  the block as `sc_evidence_block`, pasted verbatim into the reviewer prompt in
  Phase 5. The exit code does not stop the review — a failing criterion is
  exactly what the review exists to find.
- `formatCriteriaEvidence` builds that block in code: one row per criterion with
  the command, the exit code and the first output line. Chose a code-built block
  over an orchestrator-written summary so the orchestrator cannot narrate a pass
  that never happened.
- The rubric now judges the supplied result: `PASS` supports Full, `FAILED` /
  `BLOCKED` is `Broken` with the exit code cited, and `NOT RUN` is the absence
  of a measurement — never evidence in either direction.
- Phase 4.5 is skipped in `quick` mode: that mode does not launch the
  conformance reviewer, so there is nobody to hand the result to.

Red first: seven tests in `tests/lib/criteria-runner.test.mjs` against a
committed brief fixture whose three criteria pass, fail, and are prose-only.
The two doc pins were verified red against the pre-fix files (rubric asked
"exists and passes"; no Phase 4.5; the block reached nobody).

Suite: 1117 (1115/0/2), up 9.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 01:34:18 +02:00
commit c23b009738
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
6 changed files with 274 additions and 8 deletions

View file

@ -1852,3 +1852,30 @@ test('D-03: trekexecute Phase 7 runs the plan Verification on the single-session
'a failing criterion must fell the run, not be noted in the final report',
);
});
// End-state defect D-04: the rubric required brief-conformance-reviewer to decide whether a
// Success Criterion's verification command "exists and passes" — with Read/Glob/Grep. The
// command now runs the commands and hands over the result; the reviewer stays read-only.
// Fix the SOURCE.
test('D-04: the conformance reviewer judges a supplied result, and never runs a command', () => {
const a = read('agents/brief-conformance-reviewer.md');
assert.ok(
!/exists and passes/.test(a),
'the rubric may no longer ask a Read/Glob/Grep agent whether a command passes — that ask IS defect D-04',
);
assert.match(a, /^tools: \["Read", "Glob", "Grep"\]$/m, 'the reviewer stays read-only: no Bash');
assert.match(a, /You never run a command/, 'the reviewer must be told in-band that it does not execute');
assert.match(a, /Success-criteria check results/, 'the reviewer must document the supplied evidence as an input');
});
test('D-04: trekreview runs the success-criteria commands and hands the reviewer the result', () => {
const t = read('commands/trekreview.md');
const phase = (t.split("\n## Phase 4.5 — ")[1] || '').split('\n## ')[0];
assert.ok(phase.length > 0, 'trekreview.md must carry the phase that runs the success-criteria checks');
assert.match(phase, /criteria-runner\.mjs" \\\n --brief .* --evidence/, 'the phase must invoke the runner in --evidence mode');
assert.match(phase, /sc_evidence_block/, 'the phase must name the captured block the reviewer prompt carries');
assert.match(
t.split('\n## Phase 5 — ')[1] || '', /sc_evidence_block/,
'the reviewer-launch phase must hand the block over — a block nobody passes is not evidence',
);
});