open/voyage
1
0
Fork 0
Code Issues Releases 13 Activity

fix(execute): below 1.8 a plan that names no test file gets the files the helper wrote

Browse source 
Operator decision 41 (14.10); review finding 7b3b59ed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-10-14 18:17:24 +02:00
commit 829da487ec
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
6 changed files with 90 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

4
agents/step-implementer.md
View file

@ -51,7 +51,7 @@ and never touch a file the step does not list.
2. **Prove it red before you build.** Run, from the repository root:
`node {voyage_root}/lib/verification/red-proof.mjs --red-only --base {start_sha} --cmd "<the step's Verify command>" --plan "{plan}" --step {step}`
(Red proof reads the step's test files from the plan and prints the list
it ran on a `tests:` line; give it no `--tests` (operator decision 30).)
it ran on a `tests:` line; give it no `--tests` (operator decision 30), except in the one case below.)
Exit 0 = the test fails at the start point: continue. Exit 1 = the test
already passes at the start point, so it proves nothing: rewrite the test
until it fails for the reason the step exists, and run the check again.
@ -62,7 +62,7 @@ and never touch a file the step does not list.
the report and stop. The executor sends the step up to the plan.
Exit 2 = it could not run. When its reason starts `plan-unreadable`, the
plan path or the step number you were given is wrong: do not build; say
so in the report and stop (the executor corrects its call). Exit 2 otherwise (the
so in the report and stop (the executor corrects its call). When its reason starts `no test files given` and the plan_version is below 1.8 (a `File:` line there may name its test without backticks, so the plan names no test file), run the same command again with `--tests <the test files you wrote, comma-joined>` added (operator decision 41, 2026-10-14) and read that answer as above; it prints the files you wrote on its `tests:` line. From 1.8 the rules below apply unchanged. Exit 2 otherwise (the
Verify command is not one test-runner invocation, or a file is missing):
on `plan_version` ≥ 1.8 do not build; say so in the report and stop.
Below 1.8, write "red proof not run: {reason}" in the report and go on to

4
commands/trekexecute.md
View file

@ -1451,7 +1451,7 @@ is not proven red at its start point and green at its result does not pass.
skill is the advice; the gate below, `red-proof.mjs`, is the guarantee, and it
stays a command whatever the skill says.
**The step's test files** are every file on the `Test first:` `File:` line: the backticked paths that stand right after `File:`, separated only by commas or spaces, and below plan_version 1.9 also each further path after a note in brackets such as `(new)` or after `and`, `plus` or `+`, as long as it is clearly a test (the first path that is not ends the list; operator decision 31); a path in the description after them is not a test file (every plan_version; operator decision 20, 2026-10-12). `red-proof.mjs --plan --step` reads that list from the plan, so neither the helper nor the executor passes `--tests` (operator decision 30, 2026-10-14); it prints the list it ran (`tests`), refuses a `--tests` that differs (exit 2, `tests-mismatch`, the plan's list in `plan_tests`) and a plan or step it cannot read (exit 2, `plan-unreadable`). The lock's `--files` is the list red proof printed as `tests`. The
**The step's test files** are every file on the `Test first:` `File:` line: the backticked paths that stand right after `File:`, separated only by commas or spaces, and below plan_version 1.9 also each further path after a note in brackets such as `(new)` or after `and`, `plus` or `+`, as long as it is clearly a test (the first path that is not ends the list; operator decision 31); a path in the description after them is not a test file (every plan_version; operator decision 20, 2026-10-12). `red-proof.mjs --plan --step` reads that list from the plan, so neither the helper nor the executor passes `--tests` (operator decision 30, 2026-10-14); it prints the list it ran (`tests`), refuses a `--tests` that differs (exit 2, `tests-mismatch`, the plan's list in `plan_tests`) and a plan or step it cannot read (exit 2, `plan-unreadable`). Only when the plan names no test file (below plan_version 1.8 a `File:` line may name its test without backticks) does red proof answer `no test files given`; then the helper runs it again with `--tests` naming the test files it wrote, and the executor runs its own check again with `--tests` naming the files on the `tests:` line the helper copied into its report (operator decision 41, 2026-10-14). The lock's `--files` is the list red proof printed as `tests`. The
plan template lists them apart from `Files:`, so everywhere below **the step's
scope = its `Files:` + its test files**: both are built, both are committed at
the checkpoint, and neither is a scope finding.
@ -1489,7 +1489,7 @@ already rejects such a step (`PLAN_STEP_VERIFY_NOT_RUNNABLE`).
| 1, `not-red-at-base`, `tests_only: true` | `"rejected"` | every file in the step's `Files:` is a test (the plan validator's `PLAN_STEP_TESTS_ONLY`, the same reading). The plan's `Files:` alone decides: the step's diff is not read, and a path changed outside `Files:` is the controller's scope finding (Sub-step C). The behaviour its tests cover already exists at the start point (an earlier step built it, or it was there before the plan), so the step can never be red and no rewrite of the test can fix that. The plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4), with no fix round — the planner moves the test into the step that builds the behaviour, or makes a test meant to pass from the start the `Regression:` of the step that changes the code |
| 1, `not-red-at-base`, otherwise | `"rejected"` | the test passes without the change, so the TEST is wrong: a finding for Sub-step F, and that fix round is the one exception to the lock — the executor releases the step's test lock for it, the fix helper rewrites the test, passes the `--red-only` gate and locks again |
| 1, `not-green-at-result` | `"rejected"` | the change does not make the test pass: a finding for Sub-step F |
| 2 | `"not-run"` | never a pass and never proven. **`plan-unreadable` first:** a reason that starts `plan-unreadable` means the executor's own `--plan` or `--step` is wrong (a path, a step number), which no helper can fix: the executor corrects `{plan_path}` to the absolute path of the plan file it loaded in Phase 1 and `{N}` to the number in the step's own `### Step N:` heading. When the helper's report shows it built (proven and locked), the executor reruns only its own red proof with the corrected call. When the helper stopped on the same answer before it built, the executor records the restart (progress `red_proof: "not-run"`, the reason in the step's `error`, and a decision-log entry with `cost_if_wrong` "the step's first helper run"), and starts the step again with a fresh build helper (Sub-step C, round label `build (restart)`; `attempts` +1 and `iterations_remaining` −1 as for any launch; the fresh helper starts from the tree as the first helper left it, nothing is reverted, and it keeps one copy of each test case it writes), once, outside the fix rounds (`fix_rounds` does not change); a step whose `error` already holds a `plan-unreadable` restart, on a `--resume` too, does not restart again; a second `plan-unreadable` goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4), and the upward file's question is red proof's reason (the plan path or step heading it could not read), not a step assumption. A `tests-mismatch` (a `--tests` that differs from the plan's list, in `plan_tests`) cannot come from the documented call, which passes no `--tests`; when it does, it is a finding for Sub-step F. **plan_version ≥ 1.8:** when the reason is the command (the `Verify:` is not one test-runner invocation), the plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4); any other reason (a missing test file, an unresolvable start point, a control run that stays red) is a finding for Sub-step F. **Below 1.8:** a step with no `Test first:`, or whose `Verify:` is not a test-runner command, is tolerated as `"not-run"` — counted in the final report among the steps without proof of red, never called proven; any other reason is a finding |
| 2 | `"not-run"` | never a pass and never proven. **`plan-unreadable` first:** a reason that starts `plan-unreadable` means the executor's own `--plan` or `--step` is wrong (a path, a step number), which no helper can fix: the executor corrects `{plan_path}` to the absolute path of the plan file it loaded in Phase 1 and `{N}` to the number in the step's own `### Step N:` heading. When the helper's report shows it built (proven and locked), the executor reruns only its own red proof with the corrected call. When the helper stopped on the same answer before it built, the executor records the restart (progress `red_proof: "not-run"`, the reason in the step's `error`, and a decision-log entry with `cost_if_wrong` "the step's first helper run"), and starts the step again with a fresh build helper (Sub-step C, round label `build (restart)`; `attempts` +1 and `iterations_remaining` −1 as for any launch; the fresh helper starts from the tree as the first helper left it, nothing is reverted, and it keeps one copy of each test case it writes), once, outside the fix rounds (`fix_rounds` does not change); a step whose `error` already holds a `plan-unreadable` restart, on a `--resume` too, does not restart again; a second `plan-unreadable` goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4), and the upward file's question is red proof's reason (the plan path or step heading it could not read), not a step assumption. A `tests-mismatch` (a `--tests` that differs from the plan's list, in `plan_tests`) cannot come from the documented call, which passes no `--tests`; when it does, it is a finding for Sub-step F. **plan_version ≥ 1.8:** when the reason is the command (the `Verify:` is not one test-runner invocation), the plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4); any other reason (a missing test file, an unresolvable start point, a control run that stays red) is a finding for Sub-step F. **Below 1.8:** a step with no `Test first:`, or whose `Verify:` is not a test-runner command, is tolerated as `"not-run"` — counted in the final report among the steps without proof of red, never called proven; `no test files given` on a step that has `Test first:` is not a finding: when the helper's report has a `tests:` line, the executor reruns its check with `--tests` naming the files on the `tests:` line in the helper's report, as Sub-step B says (operator decision 41); without such a line, or on a second `no test files given`, the step is tolerated as `"not-run"` like a step with no `Test first:`; any other reason is a finding |
The exit code is the verdict, the output text is not. `base.tap` (when the
runner prints TAP) says what the red was: `fail 1 of 1` on a node test file is

2
docs/HANDOVER-CONTRACTS.md
View file

@ -270,7 +270,7 @@ prose form (relaxed in S26 so the template's prose emission parses).
- **K10 — `Regression:`** (optional step field): ONE backticked command the criteria runner's allowlist runs, with no pipe and no `&&` (first backticked span, as for `Verify:`). The executor runs it green before the step's checkpoint; a step without it runs nothing new. `Verify:` stays the step's own red-first test. One that can never run is `PLAN_STEP_REGRESSION_NOT_RUNNABLE` (an error; a warning under `--soft`).
- **K11 — `On failure:`** opens with one of `revert`, `retry`, `skip`, `escalate` (case-insensitive; a leading `**`, `_`, backtick or `-` is ignored, other leading characters are not; `onFailureAction` in `lib/parsers/plan-schema.mjs`), the words `/trekexecute` reads; any other opening is the warning `PLAN_STEP_ON_FAILURE_UNKNOWN`. A missing `On failure:` is unchanged (the executor defaults to `escalate`).
- **A step of only tests** (order 10, voyage-lekerepo trial 3, 2026-10-09): when every file in a step's `Files:` is a test (a file on its `Test first:` `File:` line, or a path `lib/util/test-path.mjs` reads as a test: under `tests/`, `test/`, `__tests__/`, `spec/`, `specs/`, or named `*.test.*`, `*.spec.*`, `*_test.*`, `*_spec.*`, `test_*.py`, but never a document such as `docs/specs/api.md`; `stepTestsOnly` in `lib/parsers/plan-schema.mjs`), the warning is `PLAN_STEP_TESTS_ONLY`: the behaviour its tests cover exists at its start point, so it can never be proven red. A step without `Files:` is not judged. At run time `red-proof.mjs --plan <plan.md> --step <N>` puts the same reading on a `not-red-at-base` as `tests_only`: the plan's `Files:` alone decides, and the step's diff is not read (operator 2026-10-11), and `/trekexecute` sends a `tests_only: true` step straight up to the plan.
- **The `Test first:` files** (operator decision 20, 2026-10-12): a step's test files are the backticked paths that stand right after `File:` on its `Test first:` `File:` line, separated only by commas or whitespace; a path in the description after them (`` covers `parse()` in `src/parser.mjs` ``) is no test file. This holds at every plan_version (`testFirstFiles` in `lib/parsers/plan-schema.mjs`; from 1.9 the line is read by the grammar of invariant 8). Below plan_version 1.9 the run also goes on past a note in brackets (`(new)`, `*(new)*`, `(existing …)`) and the words `and`, `plus` and `+`, to a further path only when it is clearly a test (as `lib/util/test-path.mjs` and the tests-only check read it); a path that is not ends the run (operator decision 28, 2026-10-13, and operator decision 31, 2026-10-14). `red-proof.mjs --plan <plan.md> --step <N>` reads them from the plan, and the documented call passes no `--tests` (operator decision 30, 2026-10-14): it prints the list it ran (`tests`); a given `--tests` that differs is `not-run` with reason `tests-mismatch` and the plan's list in `plan_tests`; a plan or step it cannot read is `not-run` with reason `plan-unreadable`.
- **The `Test first:` files** (operator decision 20, 2026-10-12): a step's test files are the backticked paths that stand right after `File:` on its `Test first:` `File:` line, separated only by commas or whitespace; a path in the description after them (`` covers `parse()` in `src/parser.mjs` ``) is no test file. This holds at every plan_version (`testFirstFiles` in `lib/parsers/plan-schema.mjs`; from 1.9 the line is read by the grammar of invariant 8). Below plan_version 1.9 the run also goes on past a note in brackets (`(new)`, `*(new)*`, `(existing …)`) and the words `and`, `plus` and `+`, to a further path only when it is clearly a test (as `lib/util/test-path.mjs` and the tests-only check read it); a path that is not ends the run (operator decision 28, 2026-10-13, and operator decision 31, 2026-10-14). `red-proof.mjs --plan <plan.md> --step <N>` reads them from the plan, and the documented call passes no `--tests` (operator decision 30, 2026-10-14): it prints the list it ran (`tests`); a given `--tests` that differs is `not-run` with reason `tests-mismatch` and the plan's list in `plan_tests`; a plan or step it cannot read is `not-run` with reason `plan-unreadable`; only when the plan names no test file (below plan_version 1.8 a `File:` line may name its test without backticks) does it answer `no test files given`, and then the helper runs it again with `--tests` naming the test files it wrote, and the executor with the files on the `tests:` line in the helper's report (operator decision 41, 2026-10-14).
8. **plan_version 1.9 only** (1.8 and 1.7 plans are read as before, with one warning, `PLAN_STEP_PATH_UNTICKED`, for a path their reader found outside backticks; decision 16, operator 2026-10-11) — the file first on every line:
- the entries of a step's `Files:` field are its first non-empty line and each list-item line (`-`, `*`, `+`, `1.`, `1)`); a line that is no list item continues the entry above it, when the entry so far ends with a comma, its text joins that entry with one space before the entry is read, so a list wrapped after a comma keeps its paths; otherwise it is description, never read, and so is every later wrapped line of that entry (operator decision 29, 2026-10-14); an entry with nothing on its own line (a `File:` line with nothing after it) takes its first wrapped line;
- each entry opens with its backticked path(s), separated only by commas or whitespace; everything from the first other text on is description and is never read (`` `a.test.mjs` - covers `src/parse.mjs` `` names one file), so every path stands before any annotation (`` `src/a.mjs`, `src/b.mjs` (new) ``, never `` `src/a.mjs` (new), `src/b.mjs` ``);

7
lib/verification/red-proof.mjs
View file

@ -27,9 +27,10 @@
// `tests-mismatch` and the plan's list in `plan_tests` (review cf241526). A plan
// or step that cannot be read is `not-run` with a reason that starts
// `plan-unreadable`, whether or not `--tests` is given (operator decision 30,
// 2026-10-14). An empty plan list (a 1.7 step without `Test first:`) changes
// nothing. Every result from the point the test list is settled on carries it in
// `tests` (the list it ran); the text output prints it on a ` tests:` line, and
// 2026-10-14). An empty plan list (a 1.7 step without `Test first:`, or below
// 1.8 a `File:` line without backticks) changes nothing here: the caller gives
// `--tests` (operator decision 41). Every result from the point the test list
// is settled on carries it in `tests` (the list it ran); the text output prints it on a ` tests:` line, and
// `plan_tests` on a ` plan_tests:` line, comma-joined as `--tests` takes it.
//
// Exit 0 = proven, 1 = rejected (`not-red-at-base` or `not-green-at-result`),

28
tests/e2e-files-trial4.sh
View file

@ -28,6 +28,10 @@
# line " plan_tests: tests/parser.test.mjs").
# 13. red-proof --red-only on a 1.8 plan whose File line names two tests,
# each with a (new) note (exit 0; tests is both, in order; decision 31).
# 14. red-proof --red-only on a 1.7 plan whose File line names its test
# without backticks: no --tests (exit 2, no test files given, no tests
# line), then --tests with the file the helper wrote (exit 0, the line
# " tests: tests/parser.test.mjs"; decision 41).
# Prints "e2e: ok" and exits 0, or names the failed check and exits 1.
# bash 3.2 compatible, ASCII only.
@ -193,7 +197,12 @@ writeFileSync(dir + "/p13.md", buildPlan({ steps: [{
filesRaw: "`src/parser.mjs`, `tests/a.test.mjs`, `tests/b.test.mjs`",
testFirstRaw: " - File: `tests/a.test.mjs` (new), `tests/b.test.mjs` (new)",
}, e2e] }));
' "$TMP" || fail "could not write the plans for checks 8 to 13"
writeFileSync(dir + "/p14.md", buildPlan({ version: "1.7", steps: [{
file: "src/parser.mjs",
filesRaw: "`src/parser.mjs`, `tests/parser.test.mjs`",
testFirstRaw: " - File: tests/parser.test.mjs (new)",
}, e2e] }));
' "$TMP" || fail "could not write the plans for checks 8 to 14"
out="$(node lib/validators/plan-validator.mjs --strict --json "$TMP/p8.md" 2>&1)"
rc=$?
@ -256,5 +265,22 @@ rc=$?
tests_equal "$TMP/rp13.json" '["tests/a.test.mjs","tests/b.test.mjs"]' \
|| fail "13: red-proof tests is not the two test paths in order: $(cat "$TMP/rp13.json")"
# ---- 14. decision 41: a 1.7 plan that names no test file, then --tests ----
out="$(node lib/verification/red-proof.mjs --red-only --cwd "$R" --base HEAD --cmd 'node --test tests/parser.test.mjs' --plan "$TMP/p14.md" --step 1 2>&1)"
rc=$?
[ "$rc" -eq 2 ] || fail "14: red-proof on a 1.7 plan that names no test file: exit $rc, expected 2: $out"
echo "$out" | grep -q 'no test files given' \
|| fail "14: red-proof on a 1.7 plan that names no test file: no 'no test files given' in: $out"
if echo "$out" | grep -Eq '^ tests: [^ ]'; then
fail "14: red-proof on a 1.7 plan that names no test file: a tests line in: $out"
fi
out="$(node lib/verification/red-proof.mjs --red-only --cwd "$R" --base HEAD --cmd 'node --test tests/parser.test.mjs' --plan "$TMP/p14.md" --step 1 --tests tests/parser.test.mjs 2>&1)"
rc=$?
[ "$rc" -eq 0 ] || fail "14: red-proof on a 1.7 plan with --tests the helper wrote: exit $rc, expected 0: $out"
echo "$out" | grep -Fqx ' tests: tests/parser.test.mjs' \
|| fail "14: red-proof on a 1.7 plan with --tests the helper wrote: no line ' tests: tests/parser.test.mjs' in: $out"
echo 'e2e: ok'
exit 0

54
tests/lib/test-first-multi-file.test.mjs
View file

@ -346,6 +346,60 @@ test('prose: no red-proof call line in either file passes --tests (operator deci
}
});
test('prose: below 1.8 a plan that names no test file gets --tests from the files the helper wrote (operator decision 41, review 7b3b59ed)', () => {
const esc = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
const phrase = (s) => new RegExp(s.split(/\s+/).map(esc).join('\\s+'));
const d41 = /\boperator\s+decision\s+41(?!\d)/;
const NTFG = '`no test files given`';
const has = (text, s, where) => assert.match(text, phrase(s), `${where} holds "${s}"`);
// The helper: the sentence that holds `no test files given`.
const h = readRel(HELPER);
const at = h.search(phrase(NTFG));
assert.ok(at >= 0, 'the helper names `no test files given`');
const before = h.slice(0, at);
const ends = [...before.matchAll(/\.\s/g)];
const start = ends.length ? ends[ends.length - 1].index + 1 : 0;
const rest = h.slice(at);
const stop = rest.search(/\.(\s|$)/);
const sentence = h.slice(start, stop < 0 ? undefined : at + stop + 1);
has(sentence, 'below 1.8', 'the helper sentence');
has(sentence, '`--tests <the test files you wrote', 'the helper sentence');
assert.match(sentence, d41, 'the helper sentence names operator decision 41');
const pu = h.match(/plan-unreadable[^.]*do not build/);
assert.ok(pu && pu.index < at, 'the helper sentence stands after the plan-unreadable sentence');
// The executor: the definition line and the exit-2 row.
const lines = readRel(EXEC).split('\n');
const def = lines.find((l) => l.startsWith("**The step's test files** are"));
assert.ok(def, 'the executor definition line is there');
for (const s of [NTFG, 'naming the test files it wrote', 'the `tests:` line the helper copied into its report']) {
has(def, s, 'the executor definition line');
}
assert.match(def, d41, 'the executor definition line names operator decision 41');
assert.ok(def.endsWith(' The'), `the definition line still ends with " The": ${def.slice(-40)}`);
const row = lines.find((l) => l.startsWith('| 2 | `"not-run"`'));
assert.ok(row, 'the exit-2 row is there');
const r = row.search(phrase(NTFG));
assert.ok(r >= 0, 'the exit-2 row names `no test files given`');
const after = row.slice(r);
for (const s of ['is not a finding', 'the `tests:` line in the helper\'s report', 'tolerated as `"not-run"`']) {
has(after, s, 'the exit-2 row after `no test files given`');
}
assert.match(after, d41, 'the exit-2 row names operator decision 41');
// Handover 4, invariant 7.
const ho = readRel(['docs', 'HANDOVER-CONTRACTS.md']);
const a7 = ho.indexOf('7. **Every plan_version');
const a8 = ho.indexOf('8. **', a7);
assert.ok(a7 >= 0 && a8 > a7, 'Handover 4 invariants 7 and 8 are there');
const inv7 = ho.slice(a7, a8);
for (const s of [NTFG, 'naming the test files it wrote', 'the `tests:` line in the helper\'s report']) {
has(inv7, s, 'Handover 4 invariant 7');
}
assert.match(inv7, d41, 'Handover 4 invariant 7 names operator decision 41');
});
test('prose: the helper stops on plan-unreadable before the 1.8 branches and locks the printed list', () => {
const text = readRel(HELPER);
const pu = text.match(/plan-unreadable[^.]*do not build/);