fix(verification): a backticked span is only run when it IS a command
"The first backtick span is the command" is right for a plan, whose template
puts the command first, and wrong for a brief, whose criterion usually opens
by NAMING the thing under discussion. Measured 2026-09-18 on the repo's own
example brief: 5 of 6 criteria FAILED, 3 of them parse artifacts - `--verbose`
run as a command gave exit 2 ("invalid option"), `tests/` gave exit 126 ("is a
directory"). The rubric reads a FAILED result as decisive, so each one became
a BROKEN_SUCCESS_CRITERION BLOCKER about prose.
looksLikeCommand() screens the span by SHAPE only - no filesystem lookup, so a
span parses the same everywhere. Refused: a leading flag, a directory, a token
carrying quotes/braces/prose, and a lone relative path with a slash (an
explicit ./, ../, / or ~/ still runs, as do env-var prefixes). A refused span
is `unrunnable` with reason `not-a-command` - its own outcome, never FAILED,
and it never reaches a shell.
It deliberately does NOT scan on to a later span. "The first span that LOOKS
like a command" invents commands out of prose: in that same example brief it
would have run `whoami` and `login`, two real binaries a sentence happens to
name. An absent measurement is honest; a guessed one is not.
The shape check applies to prose spans only. Inside a shell-tagged fence the
author has already declared shell, so `[ -f x ] || exit 1` still runs.
The rubric follows: a NOT RUN result is never on its own a finding. The
Partial row now describes half-built DELIVERED CODE, and the reviewer gets a
table of the three reason strings - no-command, placeholder, not-a-command -
with what each says about the sentence rather than about the code.
Not covered, stated for the record: a multi-token span whose first token is a
non-executable file (`tests/golden/login.stdout --check`) still runs, and a
criterion whose command is real but whose binary is absent still reports the
shell's exit 127 - that is a true measurement of a missing binary, not a
parse artifact.
Red first: 4 runner tests + 1 doc-consistency pin failed before this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
0013d292d6
commit
106dcb0091
4 changed files with 152 additions and 1 deletions
|
|
@ -95,6 +95,38 @@ function sectionLines(markdown, heading) {
|
|||
// `- [ ] text`, `- [x] text`, `- text`, `* text`, `1. text` -> text
|
||||
const BULLET = /^\s*(?:[-*]|\d+\.)\s+(?:\[[ xX]\]\s+)?(.*)$/;
|
||||
|
||||
const ENV_ASSIGN = /^[A-Za-z_][A-Za-z0-9_]*=\S*\s+/;
|
||||
|
||||
/**
|
||||
* Whether a backticked span is a command to RUN or something the sentence
|
||||
* merely NAMES. A plan's template puts the command first, but a brief's
|
||||
* criterion usually opens with the thing under discussion — a flag
|
||||
* (`--verbose`), a path (`tests/`) — and running those produced exit 2
|
||||
* ("invalid option") and exit 126 ("is a directory"), which the review rubric
|
||||
* reads as decisive: a BLOCKER invented out of prose. Measured 2026-09-18:
|
||||
* 3 of the 6 criteria in the repo's own example brief.
|
||||
*
|
||||
* Shape only, no filesystem lookup, so a span parses the same way everywhere.
|
||||
* And deliberately NO scanning on to a later span: "the first span that looks
|
||||
* like a command" invents commands out of prose — in that same example brief
|
||||
* it would have run `whoami` and `login`, two real binaries a sentence happens
|
||||
* to name. An absent measurement is the honest answer; a guessed one is not.
|
||||
*/
|
||||
export function looksLikeCommand(span) {
|
||||
let rest = String(span ?? '').trim();
|
||||
while (ENV_ASSIGN.test(rest)) rest = rest.replace(ENV_ASSIGN, '');
|
||||
const token = rest.split(/\s+/)[0] ?? '';
|
||||
if (token === '') return false;
|
||||
if (token.startsWith('-')) return false; // a flag
|
||||
if (token.endsWith('/')) return false; // a directory
|
||||
if (!/^[A-Za-z0-9_.@+~/-]+$/.test(token)) return false; // quotes, braces, prose
|
||||
if (!/[A-Za-z0-9]/.test(token)) return false;
|
||||
// A lone relative path with a slash is a file the sentence names. An explicit
|
||||
// `./`, `../`, `/` or `~/` is an invocation and still runs.
|
||||
if (rest === token && token.includes('/') && !/^(\.{1,2}\/|\/|~\/)/.test(token)) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
// The first backtick-delimited span on the line is the command by convention —
|
||||
// both templates put it first and a second span holds the expected output.
|
||||
function firstCommand(text) {
|
||||
|
|
@ -103,6 +135,7 @@ function firstCommand(text) {
|
|||
const raw = m[1].trim();
|
||||
// Template placeholders (`{exact command}`) are not commands.
|
||||
if (raw === '' || /^\{.*\}$/.test(raw)) return { command: null, reason: 'placeholder' };
|
||||
if (!looksLikeCommand(raw)) return { command: null, reason: 'not-a-command' };
|
||||
return { command: raw, reason: '' };
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue