fix(execute): one definition of the step's test files, read as decision 20 sets it

Review finding cf241526.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-10-13 19:00:09 +02:00
commit 0142034979
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 135 additions and 18 deletions

View file

@ -186,7 +186,7 @@ Extract every `### Step N: {description}` heading (in order). For each step, ext
- **Reuses** — existing code to leverage (informational)
- **Test first** — the test the step writes before it builds: its `File:`
(existing | new), what it `Verifies:`, an optional `Pattern:` (required from
plan_version ≥ 1.8, see below). The step's test files are every file on the `Test first:` `File:` line (below plan_version 1.9 every backticked span on that line with the shape of a path; from 1.9 the backticked paths the line opens with).
plan_version ≥ 1.8, see below). Its files are the step's test files, as Sub-step B defines them.
- **Verify** — command to run after implementation: the first backticked
span of the field. It is also the step's **test command**: proof of red
(Sub-step B) and the done line (Sub-step G) run it, so it must be one
@ -1451,7 +1451,7 @@ is not proven red at its start point and green at its result does not pass.
skill is the advice; the gate below, `red-proof.mjs`, is the guarantee, and it
stays a command whatever the skill says.
**The step's test files** are every file on the `Test first:` `File:` line (below plan_version 1.9 every backticked span on that line with the shape of a path; from 1.9 the backticked paths the line opens with). The
**The step's test files** are every file on the `Test first:` `File:` line: the backticked paths that stand right after `File:`, separated only by commas or spaces; a path in the description after them is not a test file (every plan_version; operator decision 20, 2026-10-12). `red-proof.mjs --plan --step` reads the same list from the plan and refuses a `--tests` that differs (exit 2, `tests-mismatch`); on that answer the executor runs it once more with the list it printed (`plan_tests`), and a second exit 2 is read as the exit-2 row below says. The lock's `--files` is the list red proof accepted. The
plan template lists them apart from `Files:`, so everywhere below **the step's
scope = its `Files:` + its test files**: both are built, both are committed at
the checkpoint, and neither is a scope finding.
@ -1469,12 +1469,12 @@ already rejects such a step (`PLAN_STEP_VERIFY_NOT_RUNNABLE`).
(the test must FAIL at the start point), locks the test files, and only then
implements. A `tests_only: true` answer stops the helper before it builds
(it does not rewrite the test); the executor's own run below then sends the
step up to the plan. The plan's `Files:` alone decides `tests_only`. The step's test files are every file on the `Test first:` `File:` line (below plan_version 1.9 every backticked span on that line with the shape of a path; from 1.9 the backticked paths the line opens with).
step up to the plan. The plan's `Files:` alone decides `tests_only`. `--tests` names the step's test files, as Sub-step B defines them.
2. **The executor proves it both ways after the helper returns:**
```bash
node ${CLAUDE_PLUGIN_ROOT}/lib/verification/red-proof.mjs --base {start_sha} --cmd "{Verify command}" --tests {test files, ONE comma-separated argument} --plan "{plan_path}" --step {N} --json
```
The step's test files are every file on the `Test first:` `File:` line (below plan_version 1.9 every backticked span on that line with the shape of a path; from 1.9 the backticked paths the line opens with). The test must FAIL against the start point and PASS against the result
`--tests` names the step's test files, as Sub-step B defines them. The test must FAIL against the start point and PASS against the result
(the working tree), and a **control run** must PASS: the start-point copy
with the step's whole change laid on top. The start point is a temp
`git clone --shared` checked out at `start_sha` (a real git checkout), with
@ -1489,7 +1489,7 @@ already rejects such a step (`PLAN_STEP_VERIFY_NOT_RUNNABLE`).
| 1, `not-red-at-base`, `tests_only: true` | `"rejected"` | every file in the step's `Files:` is a test (the plan validator's `PLAN_STEP_TESTS_ONLY`, the same reading). The plan's `Files:` alone decides: the step's diff is not read, and a path changed outside `Files:` is the controller's scope finding (Sub-step C). The behaviour its tests cover already exists at the start point (an earlier step built it, or it was there before the plan), so the step can never be red and no rewrite of the test can fix that. The plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4), with no fix round — the planner moves the test into the step that builds the behaviour, or makes a test meant to pass from the start the `Regression:` of the step that changes the code |
| 1, `not-red-at-base`, otherwise | `"rejected"` | the test passes without the change, so the TEST is wrong: a finding for Sub-step F, and that fix round is the one exception to the lock — the executor releases the step's test lock for it, the fix helper rewrites the test, passes the `--red-only` gate and locks again. `tests_only: null` (the plan or the step could not be read) counts here too |
| 1, `not-green-at-result` | `"rejected"` | the change does not make the test pass: a finding for Sub-step F |
| 2 | `"not-run"` | never a pass and never proven. **plan_version ≥ 1.8:** when the reason is the command (the `Verify:` is not one test-runner invocation), the plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4); any other reason (a missing test file, an unresolvable start point, a control run that stays red) is a finding for Sub-step F. **Below 1.8:** a step with no `Test first:`, or whose `Verify:` is not a test-runner command, is tolerated as `"not-run"` — counted in the final report among the steps without proof of red, never called proven; any other reason is a finding |
| 2 | `"not-run"` | never a pass and never proven. **`tests-mismatch` first:** a reason that starts `tests-mismatch` (the `--tests` given differs from the plan's `Test first:` `File:` line) is neither the plan's failure nor a finding at first: the executor reruns once with the list red proof printed (`plan_tests`) and reads that answer; only a second exit 2 goes on below. **plan_version ≥ 1.8:** when the reason is the command (the `Verify:` is not one test-runner invocation), the plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4); any other reason (a missing test file, an unresolvable start point, a control run that stays red) is a finding for Sub-step F. **Below 1.8:** a step with no `Test first:`, or whose `Verify:` is not a test-runner command, is tolerated as `"not-run"` — counted in the final report among the steps without proof of red, never called proven; any other reason is a finding |
The exit code is the verdict, the output text is not. `base.tap` (when the
runner prints TAP) says what the red was: `fail 1 of 1` on a node test file is
@ -1497,7 +1497,7 @@ usually a load error, not a named failing test — say so in the step report.
**The test lock (K8).** After its red gate the helper runs
`node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs lock --files {test files} --step {N} --cmd "{Verify command}"`.
The step's test files are every file on the `Test first:` `File:` line (below plan_version 1.9 every backticked span on that line with the shape of a path; from 1.9 the backticked paths the line opens with). From then on the guard refuses Write/Edit/NotebookEdit on those files, so the
`--files` names the step's test files, as Sub-step B defines them (the list red proof accepted). From then on the guard refuses Write/Edit/NotebookEdit on those files, so the
code moves to meet the test and never the reverse. The executor reads
`node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs status` and records
`steps.{N}.test_lock = "locked"` when it exits 0 and names the step's files.