fix(review): run the success-criteria commands and hand the reviewer the result (D-04)

The rubric required `brief-conformance-reviewer` to classify a Success
Criterion as Full only when "its verification command/test exists and passes".
Its tools are `Read`, `Glob`, `Grep`. It cannot run anything, so "passes" was
either guessed from the command's mere existence or quietly downgraded to
"exists" — a BLOCKER-tier rule key resting on an impression.

The reviewer stays read-only — a reviewer that executes the code it reviews is
not an independent reviewer. The command does the running instead:

- `/trekreview` Phase 4.5 runs the brief's `## Success Criteria` commands
  through `lib/verification/criteria-runner.mjs --brief --evidence` and captures
  the block as `sc_evidence_block`, pasted verbatim into the reviewer prompt in
  Phase 5. The exit code does not stop the review — a failing criterion is
  exactly what the review exists to find.
- `formatCriteriaEvidence` builds that block in code: one row per criterion with
  the command, the exit code and the first output line. Chose a code-built block
  over an orchestrator-written summary so the orchestrator cannot narrate a pass
  that never happened.
- The rubric now judges the supplied result: `PASS` supports Full, `FAILED` /
  `BLOCKED` is `Broken` with the exit code cited, and `NOT RUN` is the absence
  of a measurement — never evidence in either direction.
- Phase 4.5 is skipped in `quick` mode: that mode does not launch the
  conformance reviewer, so there is nobody to hand the result to.

Red first: seven tests in `tests/lib/criteria-runner.test.mjs` against a
committed brief fixture whose three criteria pass, fail, and are prose-only.
The two doc pins were verified red against the pre-fix files (rubric asked
"exists and passes"; no Phase 4.5; the block reached nobody).

Suite: 1117 (1115/0/2), up 9.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-09-18 01:34:18 +02:00

818 B

Raw Blame History

task	slug	brief_version	framing
criteria-runner fixture — a brief whose success criteria are mixed	criteria-runner-fixture-brief	2.2	new-direction

Task Brief: criteria-runner fixture

Fixture only. Consumed by tests/lib/criteria-runner.test.mjs. It is the falsifying case for D-04: a read-only reviewer cannot run these commands, so /trekreview runs them and hands over the result.

TL;DR

Three success criteria: one passes, one fails on purpose, one is prose only.

Goal

Exercise the brief side of the criteria runner.

Success Criteria

The runner reaches the shell: true exits 0
The failing case is visible: false exits 0 (FAILS on purpose — exit 1)
No new runtime dependencies are introduced

Non-Goals

Being executed by the pipeline. This file is a test fixture.

818 B Raw Blame History