fix(verification): a backticked span is only run when it IS a command
"The first backtick span is the command" is right for a plan, whose template
puts the command first, and wrong for a brief, whose criterion usually opens
by NAMING the thing under discussion. Measured 2026-09-18 on the repo's own
example brief: 5 of 6 criteria FAILED, 3 of them parse artifacts - `--verbose`
run as a command gave exit 2 ("invalid option"), `tests/` gave exit 126 ("is a
directory"). The rubric reads a FAILED result as decisive, so each one became
a BROKEN_SUCCESS_CRITERION BLOCKER about prose.
looksLikeCommand() screens the span by SHAPE only - no filesystem lookup, so a
span parses the same everywhere. Refused: a leading flag, a directory, a token
carrying quotes/braces/prose, and a lone relative path with a slash (an
explicit ./, ../, / or ~/ still runs, as do env-var prefixes). A refused span
is `unrunnable` with reason `not-a-command` - its own outcome, never FAILED,
and it never reaches a shell.
It deliberately does NOT scan on to a later span. "The first span that LOOKS
like a command" invents commands out of prose: in that same example brief it
would have run `whoami` and `login`, two real binaries a sentence happens to
name. An absent measurement is honest; a guessed one is not.
The shape check applies to prose spans only. Inside a shell-tagged fence the
author has already declared shell, so `[ -f x ] || exit 1` still runs.
The rubric follows: a NOT RUN result is never on its own a finding. The
Partial row now describes half-built DELIVERED CODE, and the reviewer gets a
table of the three reason strings - no-command, placeholder, not-a-command -
with what each says about the sentence rather than about the code.
Not covered, stated for the record: a multi-token span whose first token is a
non-executable file (`tests/golden/login.stdout --check`) still runs, and a
criterion whose command is real but whose binary is absent still reports the
shell's exit 127 - that is a true measurement of a missing binary, not a
parse artifact.
Red first: 4 runner tests + 1 doc-consistency pin failed before this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
0013d292d6
commit
106dcb0091
4 changed files with 152 additions and 1 deletions
|
|
@ -1868,6 +1868,26 @@ test('D-04: the conformance reviewer judges a supplied result, and never runs a
|
|||
assert.match(a, /Success-criteria check results/, 'the reviewer must document the supplied evidence as an input');
|
||||
});
|
||||
|
||||
// Follow-up to D-04 (PM checkpoint 2026-09-18): the runner now reports a criterion
|
||||
// whose sentence named a flag or a path as NOT RUN with reason `not-a-command`,
|
||||
// instead of running it and reporting exit 2 / exit 126. The rubric must not turn
|
||||
// that absent measurement into a finding of its own - it judges delivered code.
|
||||
test('D-04b: a NOT RUN check result is never, on its own, a defect in the rubric', () => {
|
||||
const a = read('agents/brief-conformance-reviewer.md');
|
||||
assert.ok(
|
||||
!/NOT RUN` because the criterion declares no command/.test(a),
|
||||
'a criterion that declares no runnable command may no longer read as an incomplete verification path',
|
||||
);
|
||||
assert.match(
|
||||
a, /A `NOT RUN` result is never on its own a finding/,
|
||||
'the rubric must say in-band that an absent measurement is not a defect',
|
||||
);
|
||||
assert.match(
|
||||
a, /not-a-command/,
|
||||
'the reviewer must be told the reason string the runner emits for a sentence that named a flag or a path',
|
||||
);
|
||||
});
|
||||
|
||||
test('D-04: trekreview runs the success-criteria commands and hands the reviewer the result', () => {
|
||||
const t = read('commands/trekreview.md');
|
||||
const phase = (t.split("\n## Phase 4.5 — ")[1] || '').split('\n## ')[0];
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue