Kjell Tore Guttormsen
c23b009738
The rubric required `brief-conformance-reviewer` to classify a Success Criterion as Full only when "its verification command/test exists and passes". Its tools are `Read`, `Glob`, `Grep`. It cannot run anything, so "passes" was either guessed from the command's mere existence or quietly downgraded to "exists" — a BLOCKER-tier rule key resting on an impression. The reviewer stays read-only — a reviewer that executes the code it reviews is not an independent reviewer. The command does the running instead: - `/trekreview` Phase 4.5 runs the brief's `## Success Criteria` commands through `lib/verification/criteria-runner.mjs --brief --evidence` and captures the block as `sc_evidence_block`, pasted verbatim into the reviewer prompt in Phase 5. The exit code does not stop the review — a failing criterion is exactly what the review exists to find. - `formatCriteriaEvidence` builds that block in code: one row per criterion with the command, the exit code and the first output line. Chose a code-built block over an orchestrator-written summary so the orchestrator cannot narrate a pass that never happened. - The rubric now judges the supplied result: `PASS` supports Full, `FAILED` / `BLOCKED` is `Broken` with the exit code cited, and `NOT RUN` is the absence of a measurement — never evidence in either direction. - Phase 4.5 is skipped in `quick` mode: that mode does not launch the conformance reviewer, so there is nobody to hand the result to. Red first: seven tests in `tests/lib/criteria-runner.test.mjs` against a committed brief fixture whose three criteria pass, fail, and are prose-only. The two doc pins were verified red against the pre-fix files (rubric asked "exists and passes"; no Phase 4.5; the block reached nobody). Suite: 1117 (1115/0/2), up 9. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
30 lines
818 B
Markdown
30 lines
818 B
Markdown
---
|
|
task: criteria-runner fixture — a brief whose success criteria are mixed
|
|
slug: criteria-runner-fixture-brief
|
|
brief_version: "2.2"
|
|
framing: new-direction
|
|
---
|
|
|
|
# Task Brief: criteria-runner fixture
|
|
|
|
Fixture only. Consumed by `tests/lib/criteria-runner.test.mjs`. It is the
|
|
falsifying case for D-04: a read-only reviewer cannot run these commands, so
|
|
/trekreview runs them and hands over the result.
|
|
|
|
## TL;DR
|
|
|
|
Three success criteria: one passes, one fails on purpose, one is prose only.
|
|
|
|
## Goal
|
|
|
|
Exercise the brief side of the criteria runner.
|
|
|
|
## Success Criteria
|
|
|
|
- The runner reaches the shell: `true` exits 0
|
|
- The failing case is visible: `false` exits 0 (FAILS on purpose — exit 1)
|
|
- No new runtime dependencies are introduced
|
|
|
|
## Non-Goals
|
|
|
|
- Being executed by the pipeline. This file is a test fixture.
|