fix(review): run the success-criteria commands and hand the reviewer the result (D-04)

The rubric required `brief-conformance-reviewer` to classify a Success
Criterion as Full only when "its verification command/test exists and passes".
Its tools are `Read`, `Glob`, `Grep`. It cannot run anything, so "passes" was
either guessed from the command's mere existence or quietly downgraded to
"exists" — a BLOCKER-tier rule key resting on an impression.

The reviewer stays read-only — a reviewer that executes the code it reviews is
not an independent reviewer. The command does the running instead:

- `/trekreview` Phase 4.5 runs the brief's `## Success Criteria` commands
  through `lib/verification/criteria-runner.mjs --brief --evidence` and captures
  the block as `sc_evidence_block`, pasted verbatim into the reviewer prompt in
  Phase 5. The exit code does not stop the review — a failing criterion is
  exactly what the review exists to find.
- `formatCriteriaEvidence` builds that block in code: one row per criterion with
  the command, the exit code and the first output line. Chose a code-built block
  over an orchestrator-written summary so the orchestrator cannot narrate a pass
  that never happened.
- The rubric now judges the supplied result: `PASS` supports Full, `FAILED` /
  `BLOCKED` is `Broken` with the exit code cited, and `NOT RUN` is the absence
  of a measurement — never evidence in either direction.
- Phase 4.5 is skipped in `quick` mode: that mode does not launch the
  conformance reviewer, so there is nobody to hand the result to.

Red first: seven tests in `tests/lib/criteria-runner.test.mjs` against a
committed brief fixture whose three criteria pass, fail, and are prose-only.
The two doc pins were verified red against the pre-fix files (rubric asked
"exists and passes"; no Phase 4.5; the block reached nobody).

Suite: 1117 (1115/0/2), up 9.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 01:34:18 +02:00
commit c23b009738
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
6 changed files with 274 additions and 8 deletions

View file

@ -188,6 +188,56 @@ drops are `COVERAGE_SILENT_SKIP` (MAJOR) per the rule catalogue.
If `mode == dry-run`: print the triage map and exit.
## Phase 4.5 — Run the brief's success-criteria checks
**Skipped in `quick` mode** (that mode does not launch
`brief-conformance-reviewer`, so there is nobody to hand the result to) and in
`dry-run`.
`brief-conformance-reviewer` is asked to decide whether each Success Criterion's
verification command passes. Its tools are `Read`, `Glob`, `Grep` — it cannot
run anything, and it stays that way: a reviewer that executes the code it
reviews is not an independent reviewer. So THIS command runs the commands, and
the reviewer judges the RESULT.
```bash
# Resolve the plugin root ONCE. ${CLAUDE_PLUGIN_ROOT} is substituted in this
# command's text but is EMPTY in the Bash tool's process env, and a bare
# `node ${CLAUDE_PLUGIN_ROOT}/lib/…` then runs `node /lib/…`, which exits 1 —
# indistinguishable from a criterion that failed.
VOYAGE_ROOT="${CLAUDE_PLUGIN_ROOT:-}"
case "$VOYAGE_ROOT" in
/*) ;;
*) VOYAGE_ROOT="$(ls -d "$HOME"/.claude/plugins/cache/*/voyage 2>/dev/null | head -1)" ;;
esac
if [ ! -f "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" ]; then
echo "[voyage] success-criteria checks could not run - plugin root unresolved."
echo " NOT a pass: hand the reviewer NO results and say so."
exit 2
fi
# Every command is screened through the executor denylist before it reaches a
# shell; a blocked command is reported BLOCKED, never run. Foreground only.
node "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" \
--brief "{brief_path}" --evidence --cwd "$(git rev-parse --show-toplevel)"
```
Exit 0 means every criterion passed; exit 1 means at least one failed, was
blocked, or had no command; exit 2 means the runner could not run. **The exit
code does not stop the review** — a failing criterion is exactly what the review
exists to find. Capture stdout as `sc_evidence_block`.
If the runner exits 2 (or the root could not be resolved), set
`sc_evidence_block` to a single line naming the failure and stating that no
criterion was checked. An unrun check is never a pass, and the reviewer must be
told which of the two it is looking at.
`sc_evidence_block` is pasted verbatim into the `brief-conformance-reviewer`
prompt in Phase 5, and its summary line goes into the Coverage section of
`review.md`. Do NOT summarise, re-word or re-judge it on the way — the block is
built by `formatCriteriaEvidence` precisely so the orchestrator cannot narrate a
pass that never happened.
## Phase 5 — Launch parallel reviewers
**Substrate routing (opt-in `--workflow`).** When `workflow_substrate == true`, run
@ -226,6 +276,11 @@ Each reviewer prompt includes:
- **Brief path** — `{brief_path}` (read on demand; do not inline).
- **Rule catalogue** — reference to `lib/review/rule-catalogue.mjs`.
`brief-conformance-reviewer` additionally receives `sc_evidence_block` from
Phase 4.5, pasted verbatim — the command, exit code and first line of output for
every Success Criterion. It is the ONLY evidence that agent has about whether a
criterion's verification passes, because it cannot run one.
Collect each reviewer's trailing JSON block and **validate it against the
reviewer-output schema** rather than merely parsing it. Run: