fix(review): run the success-criteria commands and hand the reviewer the result (D-04)
The rubric required `brief-conformance-reviewer` to classify a Success Criterion as Full only when "its verification command/test exists and passes". Its tools are `Read`, `Glob`, `Grep`. It cannot run anything, so "passes" was either guessed from the command's mere existence or quietly downgraded to "exists" — a BLOCKER-tier rule key resting on an impression. The reviewer stays read-only — a reviewer that executes the code it reviews is not an independent reviewer. The command does the running instead: - `/trekreview` Phase 4.5 runs the brief's `## Success Criteria` commands through `lib/verification/criteria-runner.mjs --brief --evidence` and captures the block as `sc_evidence_block`, pasted verbatim into the reviewer prompt in Phase 5. The exit code does not stop the review — a failing criterion is exactly what the review exists to find. - `formatCriteriaEvidence` builds that block in code: one row per criterion with the command, the exit code and the first output line. Chose a code-built block over an orchestrator-written summary so the orchestrator cannot narrate a pass that never happened. - The rubric now judges the supplied result: `PASS` supports Full, `FAILED` / `BLOCKED` is `Broken` with the exit code cited, and `NOT RUN` is the absence of a measurement — never evidence in either direction. - Phase 4.5 is skipped in `quick` mode: that mode does not launch the conformance reviewer, so there is nobody to hand the result to. Red first: seven tests in `tests/lib/criteria-runner.test.mjs` against a committed brief fixture whose three criteria pass, fail, and are prose-only. The two doc pins were verified red against the pre-fix files (rubric asked "exists and passes"; no Phase 4.5; the block reached nobody). Suite: 1117 (1115/0/2), up 9. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
e55ca9fc89
commit
c23b009738
6 changed files with 274 additions and 8 deletions
|
|
@ -188,6 +188,56 @@ drops are `COVERAGE_SILENT_SKIP` (MAJOR) per the rule catalogue.
|
|||
|
||||
If `mode == dry-run`: print the triage map and exit.
|
||||
|
||||
## Phase 4.5 — Run the brief's success-criteria checks
|
||||
|
||||
**Skipped in `quick` mode** (that mode does not launch
|
||||
`brief-conformance-reviewer`, so there is nobody to hand the result to) and in
|
||||
`dry-run`.
|
||||
|
||||
`brief-conformance-reviewer` is asked to decide whether each Success Criterion's
|
||||
verification command passes. Its tools are `Read`, `Glob`, `Grep` — it cannot
|
||||
run anything, and it stays that way: a reviewer that executes the code it
|
||||
reviews is not an independent reviewer. So THIS command runs the commands, and
|
||||
the reviewer judges the RESULT.
|
||||
|
||||
```bash
|
||||
# Resolve the plugin root ONCE. ${CLAUDE_PLUGIN_ROOT} is substituted in this
|
||||
# command's text but is EMPTY in the Bash tool's process env, and a bare
|
||||
# `node ${CLAUDE_PLUGIN_ROOT}/lib/…` then runs `node /lib/…`, which exits 1 —
|
||||
# indistinguishable from a criterion that failed.
|
||||
VOYAGE_ROOT="${CLAUDE_PLUGIN_ROOT:-}"
|
||||
case "$VOYAGE_ROOT" in
|
||||
/*) ;;
|
||||
*) VOYAGE_ROOT="$(ls -d "$HOME"/.claude/plugins/cache/*/voyage 2>/dev/null | head -1)" ;;
|
||||
esac
|
||||
if [ ! -f "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" ]; then
|
||||
echo "[voyage] success-criteria checks could not run - plugin root unresolved."
|
||||
echo " NOT a pass: hand the reviewer NO results and say so."
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Every command is screened through the executor denylist before it reaches a
|
||||
# shell; a blocked command is reported BLOCKED, never run. Foreground only.
|
||||
node "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" \
|
||||
--brief "{brief_path}" --evidence --cwd "$(git rev-parse --show-toplevel)"
|
||||
```
|
||||
|
||||
Exit 0 means every criterion passed; exit 1 means at least one failed, was
|
||||
blocked, or had no command; exit 2 means the runner could not run. **The exit
|
||||
code does not stop the review** — a failing criterion is exactly what the review
|
||||
exists to find. Capture stdout as `sc_evidence_block`.
|
||||
|
||||
If the runner exits 2 (or the root could not be resolved), set
|
||||
`sc_evidence_block` to a single line naming the failure and stating that no
|
||||
criterion was checked. An unrun check is never a pass, and the reviewer must be
|
||||
told which of the two it is looking at.
|
||||
|
||||
`sc_evidence_block` is pasted verbatim into the `brief-conformance-reviewer`
|
||||
prompt in Phase 5, and its summary line goes into the Coverage section of
|
||||
`review.md`. Do NOT summarise, re-word or re-judge it on the way — the block is
|
||||
built by `formatCriteriaEvidence` precisely so the orchestrator cannot narrate a
|
||||
pass that never happened.
|
||||
|
||||
## Phase 5 — Launch parallel reviewers
|
||||
|
||||
**Substrate routing (opt-in `--workflow`).** When `workflow_substrate == true`, run
|
||||
|
|
@ -226,6 +276,11 @@ Each reviewer prompt includes:
|
|||
- **Brief path** — `{brief_path}` (read on demand; do not inline).
|
||||
- **Rule catalogue** — reference to `lib/review/rule-catalogue.mjs`.
|
||||
|
||||
`brief-conformance-reviewer` additionally receives `sc_evidence_block` from
|
||||
Phase 4.5, pasted verbatim — the command, exit code and first line of output for
|
||||
every Success Criterion. It is the ONLY evidence that agent has about whether a
|
||||
criterion's verification passes, because it cannot run one.
|
||||
|
||||
Collect each reviewer's trailing JSON block and **validate it against the
|
||||
reviewer-output schema** rather than merely parsing it. Run:
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue