fix(verification): a backticked span is only run when it IS a command

"The first backtick span is the command" is right for a plan, whose template
puts the command first, and wrong for a brief, whose criterion usually opens
by NAMING the thing under discussion. Measured 2026-09-18 on the repo's own
example brief: 5 of 6 criteria FAILED, 3 of them parse artifacts - `--verbose`
run as a command gave exit 2 ("invalid option"), `tests/` gave exit 126 ("is a
directory"). The rubric reads a FAILED result as decisive, so each one became
a BROKEN_SUCCESS_CRITERION BLOCKER about prose.

looksLikeCommand() screens the span by SHAPE only - no filesystem lookup, so a
span parses the same everywhere. Refused: a leading flag, a directory, a token
carrying quotes/braces/prose, and a lone relative path with a slash (an
explicit ./, ../, / or ~/ still runs, as do env-var prefixes). A refused span
is `unrunnable` with reason `not-a-command` - its own outcome, never FAILED,
and it never reaches a shell.

It deliberately does NOT scan on to a later span. "The first span that LOOKS
like a command" invents commands out of prose: in that same example brief it
would have run `whoami` and `login`, two real binaries a sentence happens to
name. An absent measurement is honest; a guessed one is not.

The shape check applies to prose spans only. Inside a shell-tagged fence the
author has already declared shell, so `[ -f x ] || exit 1` still runs.

The rubric follows: a NOT RUN result is never on its own a finding. The
Partial row now describes half-built DELIVERED CODE, and the reviewer gets a
table of the three reason strings - no-command, placeholder, not-a-command -
with what each says about the sentence rather than about the code.

Not covered, stated for the record: a multi-token span whose first token is a
non-executable file (`tests/golden/login.stdout --check`) still runs, and a
criterion whose command is real but whose binary is absent still reports the
shell's exit 127 - that is a true measurement of a missing binary, not a
parse artifact.

Red first: 4 runner tests + 1 doc-consistency pin failed before this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

agents/brief-conformance-reviewer.md

@@ -86,7 +86,7 @@ context is needed) and classify coverage:
 | Coverage | Meaning | Finding emitted |
 |----------|---------|-----------------|
 | **Full** | Code change visibly implements the criterion AND its check result is `PASS` | none |
-| **Partial** | Some pieces present but the verification path is incomplete (e.g., the check result is `NOT RUN` because the criterion declares no command, or the command exists but tests are missing) | `MISSING_TEST` (MAJOR) or step-specific finding |
+| **Partial** | Some pieces present but the DELIVERED CODE leaves the criterion half-built (e.g. the behaviour is implemented but no test covers it) | `MISSING_TEST` (MAJOR) or step-specific finding |
 | **Missing** | No delivered code maps to this criterion | `UNIMPLEMENTED_CRITERION` (BLOCKER) |
 | **Broken** | The check result is `FAILED` or `BLOCKED`, or the code is structurally wrong for the criterion | `BROKEN_SUCCESS_CRITERION` (BLOCKER) |
 
@@ -94,6 +94,21 @@ A `FAILED` result is decisive: cite its exit code and output line in the
 finding's `detail`. A `NOT RUN` result is NOT decisive in either direction —
 it is the absence of a measurement, so it can never support **Full**.
 
+**A `NOT RUN` result is never on its own a finding.** It says nothing about
+the code; it says the criterion's sentence held nothing to run. The Evidence
+column names the reason the runner gave:
+
+| Reason | What it means |
+|---|---|
+| `no-command` | the criterion is prose — a human judges it by reading |
+| `placeholder` | the text is a template placeholder (`{exact command}`) |
+| `not-a-command` | the sentence opened by NAMING a flag or a path (`--verbose`, `tests/`) rather than by invoking something. Running it would have produced exit 2 or exit 126 — a number about the sentence, not about the code |
+
+In every one of those cases, judge the criterion on delivered code alone and
+say in the Evidence column that no measurement exists. Emit `MISSING_TEST`
+only when the DELIVERED CODE lacks a test, never because the brief's sentence
+carried no command.
+
 Cite the criterion text in `brief_ref` (e.g., `SC3 — "review.md is
 parseable as input to /trekplan"`).
 

lib/verification/criteria-runner.mjs

@@ -95,6 +95,38 @@ function sectionLines(markdown, heading) {
 // `- [ ] text`, `- [x] text`, `- text`, `* text`, `1. text` -> text
 const BULLET = /^\s*(?:[-*]|\d+\.)\s+(?:\[[ xX]\]\s+)?(.*)$/;
 
+const ENV_ASSIGN = /^[A-Za-z_][A-Za-z0-9_]*=\S*\s+/;
+
+/**
+ * Whether a backticked span is a command to RUN or something the sentence
+ * merely NAMES. A plan's template puts the command first, but a brief's
+ * criterion usually opens with the thing under discussion — a flag
+ * (`--verbose`), a path (`tests/`) — and running those produced exit 2
+ * ("invalid option") and exit 126 ("is a directory"), which the review rubric
+ * reads as decisive: a BLOCKER invented out of prose. Measured 2026-09-18:
+ * 3 of the 6 criteria in the repo's own example brief.
+ *
+ * Shape only, no filesystem lookup, so a span parses the same way everywhere.
+ * And deliberately NO scanning on to a later span: "the first span that looks
+ * like a command" invents commands out of prose — in that same example brief
+ * it would have run `whoami` and `login`, two real binaries a sentence happens
+ * to name. An absent measurement is the honest answer; a guessed one is not.
+ */
+export function looksLikeCommand(span) {
+  let rest = String(span ?? '').trim();
+  while (ENV_ASSIGN.test(rest)) rest = rest.replace(ENV_ASSIGN, '');
+  const token = rest.split(/\s+/)[0] ?? '';
+  if (token === '') return false;
+  if (token.startsWith('-')) return false;                      // a flag
+  if (token.endsWith('/')) return false;                        // a directory
+  if (!/^[A-Za-z0-9_.@+~/-]+$/.test(token)) return false;       // quotes, braces, prose
+  if (!/[A-Za-z0-9]/.test(token)) return false;
+  // A lone relative path with a slash is a file the sentence names. An explicit
+  // `./`, `../`, `/` or `~/` is an invocation and still runs.
+  if (rest === token && token.includes('/') && !/^(\.{1,2}\/|\/|~\/)/.test(token)) return false;
+  return true;
+}
+
 // The first backtick-delimited span on the line is the command by convention —
 // both templates put it first and a second span holds the expected output.
 function firstCommand(text) {
@@ -103,6 +135,7 @@ function firstCommand(text) {
   const raw = m[1].trim();
   // Template placeholders (`{exact command}`) are not commands.
   if (raw === '' || /^\{.*\}$/.test(raw)) return { command: null, reason: 'placeholder' };
+  if (!looksLikeCommand(raw)) return { command: null, reason: 'not-a-command' };
   return { command: raw, reason: '' };
 }
 

tests/lib/criteria-runner.test.mjs

@@ -433,3 +433,86 @@ test('a report that merely quotes a plan outline has no ## Verification section'
   assert.equal(report.error?.code, 'NO_VERIFICATION_SECTION');
   assert.equal(report.summary.ok, false);
 });
+
+// --- a span is not a command just because it is in backticks ---------------
+//
+// "The first backtick span is the command" is right for a plan (the template
+// puts it first) and wrong for a brief, whose sentence usually OPENS with the
+// thing under discussion: a flag, a path. Running those produced `/bin/sh: --:
+// invalid option` (exit 2) and "is a directory" (exit 126), and the review
+// rubric reads a FAILED result as decisive -> a BLOCKER invented out of prose.
+// Measured 2026-09-18: 3 of the 6 criteria in the repo's own example brief.
+
+const EXAMPLE_BRIEF = join(ROOT, 'examples', '01-add-verbose-flag', 'brief.md');
+
+test('parseSuccessCriteria: a leading flag is not a command — the criterion is NOT RUN', () => {
+  const md = '## Success Criteria\n\n- `--verbose` works in any position: `app --verbose run`\n';
+  const [sc] = parseSuccessCriteria(md);
+  assert.equal(sc.command, null);
+  assert.equal(sc.reason, 'not-a-command');
+});
+
+test('parseSuccessCriteria: a bare directory or file path is not a command', () => {
+  const md = [
+    '## Success Criteria',
+    '',
+    '- Existing tests in `tests/` continue to pass',
+    '- The golden file `tests/golden/login.stdout` is unchanged',
+    '',
+  ].join('\n');
+  for (const sc of parseSuccessCriteria(md)) {
+    assert.equal(sc.command, null, `${sc.label} must not be run as a command`);
+    assert.equal(sc.reason, 'not-a-command');
+  }
+});
+
+test('parseSuccessCriteria: a real command, an explicit path and env prefixes still run', () => {
+  const md = [
+    '## Success Criteria',
+    '',
+    '- All tests pass: `npm test`',
+    '- The build script runs: `./scripts/build.sh --ci`',
+    '- Absolute paths run: `/usr/bin/true`',
+    '- Env prefixes are not the command: `CI=1 npm test`',
+    '',
+  ].join('\n');
+  assert.deepEqual(
+    parseSuccessCriteria(md).map((c) => c.command),
+    ['npm test', './scripts/build.sh --ci', '/usr/bin/true', 'CI=1 npm test'],
+  );
+});
+
+test("the repo's own example brief yields ZERO parse artifacts", () => {
+  const criteria = parseSuccessCriteria(readFileSync(EXAMPLE_BRIEF, 'utf8'));
+  assert.equal(criteria.length, 6);
+  for (const c of criteria) {
+    if (c.command === null) continue;
+    assert.ok(!c.command.startsWith('-'), `${c.label} would run a flag: ${c.command}`);
+    assert.ok(!/^\S+\/(\s|$)/.test(c.command), `${c.label} would run a path: ${c.command}`);
+  }
+  // SC3/SC4 open on a flag, SC6 on a directory: three criteria that used to be
+  // FAILED with a parse exit code and are now an absent measurement.
+  for (const label of ['SC3', 'SC4', 'SC6']) {
+    const c = criteria.find((x) => x.label === label);
+    assert.equal(c.command, null, `${label}: ${c.command}`);
+    assert.equal(c.reason, 'not-a-command');
+  }
+});
+
+test('runCriteria: a not-a-command criterion is unrunnable and says why — never failed', () => {
+  const criteria = parseSuccessCriteria('## Success Criteria\n\n- `--verbose` is accepted everywhere\n');
+  const exec = execDouble({});
+  const [r] = runCriteria(criteria, { exec, screen: allowAll });
+  assert.equal(r.status, 'unrunnable');
+  assert.notEqual(r.status, 'failed');
+  assert.equal(r.exitCode, null);
+  assert.match(r.output, /not-a-command/);
+  assert.deepEqual(exec.calls, [], 'a span that is not a command must never reach a shell');
+});
+
+test('a shell-tagged fence declares commands even when a line is not command-shaped', () => {
+  const md = '## Verification\n\n```bash\n[ -f README.md ] || exit 1\n```\n';
+  const [v] = parsePlanVerification(md);
+  assert.equal(v.command, '[ -f README.md ] || exit 1',
+    'inside an explicitly shell-tagged fence the author HAS declared shell; the shape check belongs to prose spans only');
+});

tests/lib/doc-consistency.test.mjs

@@ -1868,6 +1868,26 @@ test('D-04: the conformance reviewer judges a supplied result, and never runs a
   assert.match(a, /Success-criteria check results/, 'the reviewer must document the supplied evidence as an input');
 });
 
+// Follow-up to D-04 (PM checkpoint 2026-09-18): the runner now reports a criterion
+// whose sentence named a flag or a path as NOT RUN with reason `not-a-command`,
+// instead of running it and reporting exit 2 / exit 126. The rubric must not turn
+// that absent measurement into a finding of its own - it judges delivered code.
+test('D-04b: a NOT RUN check result is never, on its own, a defect in the rubric', () => {
+  const a = read('agents/brief-conformance-reviewer.md');
+  assert.ok(
+    !/NOT RUN` because the criterion declares no command/.test(a),
+    'a criterion that declares no runnable command may no longer read as an incomplete verification path',
+  );
+  assert.match(
+    a, /A `NOT RUN` result is never on its own a finding/,
+    'the rubric must say in-band that an absent measurement is not a defect',
+  );
+  assert.match(
+    a, /not-a-command/,
+    'the reviewer must be told the reason string the runner emits for a sentence that named a flag or a path',
+  );
+});
+
 test('D-04: trekreview runs the success-criteria commands and hands the reviewer the result', () => {
   const t = read('commands/trekreview.md');
   const phase = (t.split("\n## Phase 4.5 — ")[1] || '').split('\n## ')[0];