fix(review): run the success-criteria commands and hand the reviewer the result (D-04)
The rubric required `brief-conformance-reviewer` to classify a Success Criterion as Full only when "its verification command/test exists and passes". Its tools are `Read`, `Glob`, `Grep`. It cannot run anything, so "passes" was either guessed from the command's mere existence or quietly downgraded to "exists" — a BLOCKER-tier rule key resting on an impression. The reviewer stays read-only — a reviewer that executes the code it reviews is not an independent reviewer. The command does the running instead: - `/trekreview` Phase 4.5 runs the brief's `## Success Criteria` commands through `lib/verification/criteria-runner.mjs --brief --evidence` and captures the block as `sc_evidence_block`, pasted verbatim into the reviewer prompt in Phase 5. The exit code does not stop the review — a failing criterion is exactly what the review exists to find. - `formatCriteriaEvidence` builds that block in code: one row per criterion with the command, the exit code and the first output line. Chose a code-built block over an orchestrator-written summary so the orchestrator cannot narrate a pass that never happened. - The rubric now judges the supplied result: `PASS` supports Full, `FAILED` / `BLOCKED` is `Broken` with the exit code cited, and `NOT RUN` is the absence of a measurement — never evidence in either direction. - Phase 4.5 is skipped in `quick` mode: that mode does not launch the conformance reviewer, so there is nobody to hand the result to. Red first: seven tests in `tests/lib/criteria-runner.test.mjs` against a committed brief fixture whose three criteria pass, fail, and are prose-only. The two doc pins were verified red against the pre-fix files (rubric asked "exists and passes"; no Phase 4.5; the block reached nobody). Suite: 1117 (1115/0/2), up 9. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
e55ca9fc89
commit
c23b009738
6 changed files with 274 additions and 8 deletions
|
|
@ -1852,3 +1852,30 @@ test('D-03: trekexecute Phase 7 runs the plan Verification on the single-session
|
|||
'a failing criterion must fell the run, not be noted in the final report',
|
||||
);
|
||||
});
|
||||
|
||||
// End-state defect D-04: the rubric required brief-conformance-reviewer to decide whether a
|
||||
// Success Criterion's verification command "exists and passes" — with Read/Glob/Grep. The
|
||||
// command now runs the commands and hands over the result; the reviewer stays read-only.
|
||||
// Fix the SOURCE.
|
||||
test('D-04: the conformance reviewer judges a supplied result, and never runs a command', () => {
|
||||
const a = read('agents/brief-conformance-reviewer.md');
|
||||
assert.ok(
|
||||
!/exists and passes/.test(a),
|
||||
'the rubric may no longer ask a Read/Glob/Grep agent whether a command passes — that ask IS defect D-04',
|
||||
);
|
||||
assert.match(a, /^tools: \["Read", "Glob", "Grep"\]$/m, 'the reviewer stays read-only: no Bash');
|
||||
assert.match(a, /You never run a command/, 'the reviewer must be told in-band that it does not execute');
|
||||
assert.match(a, /Success-criteria check results/, 'the reviewer must document the supplied evidence as an input');
|
||||
});
|
||||
|
||||
test('D-04: trekreview runs the success-criteria commands and hands the reviewer the result', () => {
|
||||
const t = read('commands/trekreview.md');
|
||||
const phase = (t.split("\n## Phase 4.5 — ")[1] || '').split('\n## ')[0];
|
||||
assert.ok(phase.length > 0, 'trekreview.md must carry the phase that runs the success-criteria checks');
|
||||
assert.match(phase, /criteria-runner\.mjs" \\\n --brief .* --evidence/, 'the phase must invoke the runner in --evidence mode');
|
||||
assert.match(phase, /sc_evidence_block/, 'the phase must name the captured block the reviewer prompt carries');
|
||||
assert.match(
|
||||
t.split('\n## Phase 5 — ')[1] || '', /sc_evidence_block/,
|
||||
'the reviewer-launch phase must hand the block over — a block nobody passes is not evidence',
|
||||
);
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue