fix(review): run the success-criteria commands and hand the reviewer the result (D-04)
The rubric required `brief-conformance-reviewer` to classify a Success Criterion as Full only when "its verification command/test exists and passes". Its tools are `Read`, `Glob`, `Grep`. It cannot run anything, so "passes" was either guessed from the command's mere existence or quietly downgraded to "exists" — a BLOCKER-tier rule key resting on an impression. The reviewer stays read-only — a reviewer that executes the code it reviews is not an independent reviewer. The command does the running instead: - `/trekreview` Phase 4.5 runs the brief's `## Success Criteria` commands through `lib/verification/criteria-runner.mjs --brief --evidence` and captures the block as `sc_evidence_block`, pasted verbatim into the reviewer prompt in Phase 5. The exit code does not stop the review — a failing criterion is exactly what the review exists to find. - `formatCriteriaEvidence` builds that block in code: one row per criterion with the command, the exit code and the first output line. Chose a code-built block over an orchestrator-written summary so the orchestrator cannot narrate a pass that never happened. - The rubric now judges the supplied result: `PASS` supports Full, `FAILED` / `BLOCKED` is `Broken` with the exit code cited, and `NOT RUN` is the absence of a measurement — never evidence in either direction. - Phase 4.5 is skipped in `quick` mode: that mode does not launch the conformance reviewer, so there is nobody to hand the result to. Red first: seven tests in `tests/lib/criteria-runner.test.mjs` against a committed brief fixture whose three criteria pass, fail, and are prose-only. The two doc pins were verified red against the pre-fix files (rubric asked "exists and passes"; no Phase 4.5; the block reached nobody). Suite: 1117 (1115/0/2), up 9. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
e55ca9fc89
commit
c23b009738
6 changed files with 274 additions and 8 deletions
30
tests/fixtures/brief-success-criteria.md
vendored
Normal file
30
tests/fixtures/brief-success-criteria.md
vendored
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
---
|
||||
task: criteria-runner fixture — a brief whose success criteria are mixed
|
||||
slug: criteria-runner-fixture-brief
|
||||
brief_version: "2.2"
|
||||
framing: new-direction
|
||||
---
|
||||
|
||||
# Task Brief: criteria-runner fixture
|
||||
|
||||
Fixture only. Consumed by `tests/lib/criteria-runner.test.mjs`. It is the
|
||||
falsifying case for D-04: a read-only reviewer cannot run these commands, so
|
||||
/trekreview runs them and hands over the result.
|
||||
|
||||
## TL;DR
|
||||
|
||||
Three success criteria: one passes, one fails on purpose, one is prose only.
|
||||
|
||||
## Goal
|
||||
|
||||
Exercise the brief side of the criteria runner.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- The runner reaches the shell: `true` exits 0
|
||||
- The failing case is visible: `false` exits 0 (FAILS on purpose — exit 1)
|
||||
- No new runtime dependencies are introduced
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Being executed by the pipeline. This file is a test fixture.
|
||||
Loading…
Add table
Add a link
Reference in a new issue