fix(verification): a backticked span is only run when it IS a command
"The first backtick span is the command" is right for a plan, whose template
puts the command first, and wrong for a brief, whose criterion usually opens
by NAMING the thing under discussion. Measured 2026-09-18 on the repo's own
example brief: 5 of 6 criteria FAILED, 3 of them parse artifacts - `--verbose`
run as a command gave exit 2 ("invalid option"), `tests/` gave exit 126 ("is a
directory"). The rubric reads a FAILED result as decisive, so each one became
a BROKEN_SUCCESS_CRITERION BLOCKER about prose.
looksLikeCommand() screens the span by SHAPE only - no filesystem lookup, so a
span parses the same everywhere. Refused: a leading flag, a directory, a token
carrying quotes/braces/prose, and a lone relative path with a slash (an
explicit ./, ../, / or ~/ still runs, as do env-var prefixes). A refused span
is `unrunnable` with reason `not-a-command` - its own outcome, never FAILED,
and it never reaches a shell.
It deliberately does NOT scan on to a later span. "The first span that LOOKS
like a command" invents commands out of prose: in that same example brief it
would have run `whoami` and `login`, two real binaries a sentence happens to
name. An absent measurement is honest; a guessed one is not.
The shape check applies to prose spans only. Inside a shell-tagged fence the
author has already declared shell, so `[ -f x ] || exit 1` still runs.
The rubric follows: a NOT RUN result is never on its own a finding. The
Partial row now describes half-built DELIVERED CODE, and the reviewer gets a
table of the three reason strings - no-command, placeholder, not-a-command -
with what each says about the sentence rather than about the code.
Not covered, stated for the record: a multi-token span whose first token is a
non-executable file (`tests/golden/login.stdout --check`) still runs, and a
criterion whose command is real but whose binary is absent still reports the
shell's exit 127 - that is a true measurement of a missing binary, not a
parse artifact.
Red first: 4 runner tests + 1 doc-consistency pin failed before this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
0013d292d6
commit
106dcb0091
4 changed files with 152 additions and 1 deletions
|
|
@ -86,7 +86,7 @@ context is needed) and classify coverage:
|
|||
| Coverage | Meaning | Finding emitted |
|
||||
|----------|---------|-----------------|
|
||||
| **Full** | Code change visibly implements the criterion AND its check result is `PASS` | none |
|
||||
| **Partial** | Some pieces present but the verification path is incomplete (e.g., the check result is `NOT RUN` because the criterion declares no command, or the command exists but tests are missing) | `MISSING_TEST` (MAJOR) or step-specific finding |
|
||||
| **Partial** | Some pieces present but the DELIVERED CODE leaves the criterion half-built (e.g. the behaviour is implemented but no test covers it) | `MISSING_TEST` (MAJOR) or step-specific finding |
|
||||
| **Missing** | No delivered code maps to this criterion | `UNIMPLEMENTED_CRITERION` (BLOCKER) |
|
||||
| **Broken** | The check result is `FAILED` or `BLOCKED`, or the code is structurally wrong for the criterion | `BROKEN_SUCCESS_CRITERION` (BLOCKER) |
|
||||
|
||||
|
|
@ -94,6 +94,21 @@ A `FAILED` result is decisive: cite its exit code and output line in the
|
|||
finding's `detail`. A `NOT RUN` result is NOT decisive in either direction —
|
||||
it is the absence of a measurement, so it can never support **Full**.
|
||||
|
||||
**A `NOT RUN` result is never on its own a finding.** It says nothing about
|
||||
the code; it says the criterion's sentence held nothing to run. The Evidence
|
||||
column names the reason the runner gave:
|
||||
|
||||
| Reason | What it means |
|
||||
|---|---|
|
||||
| `no-command` | the criterion is prose — a human judges it by reading |
|
||||
| `placeholder` | the text is a template placeholder (`{exact command}`) |
|
||||
| `not-a-command` | the sentence opened by NAMING a flag or a path (`--verbose`, `tests/`) rather than by invoking something. Running it would have produced exit 2 or exit 126 — a number about the sentence, not about the code |
|
||||
|
||||
In every one of those cases, judge the criterion on delivered code alone and
|
||||
say in the Evidence column that no measurement exists. Emit `MISSING_TEST`
|
||||
only when the DELIVERED CODE lacks a test, never because the brief's sentence
|
||||
carried no command.
|
||||
|
||||
Cite the criterion text in `brief_ref` (e.g., `SC3 — "review.md is
|
||||
parseable as input to /trekplan"`).
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue