fix(verification): a backticked span is only run when it IS a command

"The first backtick span is the command" is right for a plan, whose template
puts the command first, and wrong for a brief, whose criterion usually opens
by NAMING the thing under discussion. Measured 2026-09-18 on the repo's own
example brief: 5 of 6 criteria FAILED, 3 of them parse artifacts - `--verbose`
run as a command gave exit 2 ("invalid option"), `tests/` gave exit 126 ("is a
directory"). The rubric reads a FAILED result as decisive, so each one became
a BROKEN_SUCCESS_CRITERION BLOCKER about prose.

looksLikeCommand() screens the span by SHAPE only - no filesystem lookup, so a
span parses the same everywhere. Refused: a leading flag, a directory, a token
carrying quotes/braces/prose, and a lone relative path with a slash (an
explicit ./, ../, / or ~/ still runs, as do env-var prefixes). A refused span
is `unrunnable` with reason `not-a-command` - its own outcome, never FAILED,
and it never reaches a shell.

It deliberately does NOT scan on to a later span. "The first span that LOOKS
like a command" invents commands out of prose: in that same example brief it
would have run `whoami` and `login`, two real binaries a sentence happens to
name. An absent measurement is honest; a guessed one is not.

The shape check applies to prose spans only. Inside a shell-tagged fence the
author has already declared shell, so `[ -f x ] || exit 1` still runs.

The rubric follows: a NOT RUN result is never on its own a finding. The
Partial row now describes half-built DELIVERED CODE, and the reviewer gets a
table of the three reason strings - no-command, placeholder, not-a-command -
with what each says about the sentence rather than about the code.

Not covered, stated for the record: a multi-token span whose first token is a
non-executable file (`tests/golden/login.stdout --check`) still runs, and a
criterion whose command is real but whose binary is absent still reports the
shell's exit 127 - that is a true measurement of a missing binary, not a
parse artifact.

Red first: 4 runner tests + 1 doc-consistency pin failed before this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 02:04:46 +02:00
commit 106dcb0091
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 152 additions and 1 deletions

View file

@ -95,6 +95,38 @@ function sectionLines(markdown, heading) {
// `- [ ] text`, `- [x] text`, `- text`, `* text`, `1. text` -> text
const BULLET = /^\s*(?:[-*]|\d+\.)\s+(?:\[[ xX]\]\s+)?(.*)$/;
const ENV_ASSIGN = /^[A-Za-z_][A-Za-z0-9_]*=\S*\s+/;
/**
* Whether a backticked span is a command to RUN or something the sentence
* merely NAMES. A plan's template puts the command first, but a brief's
* criterion usually opens with the thing under discussion — a flag
* (`--verbose`), a path (`tests/`) — and running those produced exit 2
* ("invalid option") and exit 126 ("is a directory"), which the review rubric
* reads as decisive: a BLOCKER invented out of prose. Measured 2026-09-18:
* 3 of the 6 criteria in the repo's own example brief.
*
* Shape only, no filesystem lookup, so a span parses the same way everywhere.
* And deliberately NO scanning on to a later span: "the first span that looks
* like a command" invents commands out of prose — in that same example brief
* it would have run `whoami` and `login`, two real binaries a sentence happens
* to name. An absent measurement is the honest answer; a guessed one is not.
*/
export function looksLikeCommand(span) {
let rest = String(span ?? '').trim();
while (ENV_ASSIGN.test(rest)) rest = rest.replace(ENV_ASSIGN, '');
const token = rest.split(/\s+/)[0] ?? '';
if (token === '') return false;
if (token.startsWith('-')) return false; // a flag
if (token.endsWith('/')) return false; // a directory
if (!/^[A-Za-z0-9_.@+~/-]+$/.test(token)) return false; // quotes, braces, prose
if (!/[A-Za-z0-9]/.test(token)) return false;
// A lone relative path with a slash is a file the sentence names. An explicit
// `./`, `../`, `/` or `~/` is an invocation and still runs.
if (rest === token && token.includes('/') && !/^(\.{1,2}\/|\/|~\/)/.test(token)) return false;
return true;
}
// The first backtick-delimited span on the line is the command by convention —
// both templates put it first and a second span holds the expected output.
function firstCommand(text) {
@ -103,6 +135,7 @@ function firstCommand(text) {
const raw = m[1].trim();
// Template placeholders (`{exact command}`) are not commands.
if (raw === '' || /^\{.*\}$/.test(raw)) return { command: null, reason: 'placeholder' };
if (!looksLikeCommand(raw)) return { command: null, reason: 'not-a-command' };
return { command: raw, reason: '' };
}