open/voyage
1
0
Fork 0
Code Issues Releases 13 Activity

fix: round 6 of the release review of 6.3.0

Browse source 
Round 6 (v6.2.0..c3641ff, WARN: 11 MAJOR, 3 MINOR; 0 BLOCKER). The round-5
fixes hold (code-correctness-reviewer); three of the new MAJOR were side
effects of running checks with no shell. Each fixed behind a test that
failed first on c3641ff.

- git checks stay in the repo too: no path that climbs out with .., starts
  at / or reads <rev>:../x, and no --no-index (both reviewers).
- A trailing `# comment` on a shell-fence line is dropped: with no shell,
  its words made an honest check NOT RUN, among them the first check of the
  shipped example plan.
- A glob reaches `node --test` only on Node 21 and later, which expands it
  itself; on an older Node it is refused with that reason (engines >=18).
- With no shell, exit 126/127 is the test program's own: red proof reads it
  as a red (the control run still tells an environment fault apart), the
  done line and the test lock as a failure.
- A runner that could not start (not installed) is NOT RUN in the criteria
  runner and not-run (exit 2) in regression-check, as red-proof and
  done-line already said; never a red suite.
- Tests-only: a backticked `Makefile` alone in its entry is a path, and red
  proof confirms the plan's tests-only reading against the step's own diff
  (the altitude finding: the diff is a fact, the Files: prose is not).
- release-check stops a suite that never ends (15 minutes) and reads it as
  not measured (exit 2).
- The two lines this release added to pre-bash-executor.mjs are ASCII.
- MINOR: git and the other runners share one plain-word pattern.

Not in this round: MINOR f00a2eb8 (shared norm/ticked-path and isMain
helpers) and MINOR on plan-review-dedup's repeated tokenizing.

Suite 2194 (2192/0/2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-10-11 01:07:47 +02:00
commit 45aa9e107a
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
17 changed files with 262 additions and 62 deletions
Download patch file Download diff file Expand all files Collapse all files

3
agents/brief-conformance-reviewer.md
View file

@ -98,7 +98,7 @@ A criterion whose command was **outside the allowlist** is reported as
`NOT RUN` too: the runner executes a criterion only when its command is a known
test runner (`npm test`, `node --test`, `pytest`, `bash tests/<script>.sh`, a
read-only git subcommand, …) carrying no shell operator. Everything else never
reaches a shell. Like any `NOT RUN` it is the ABSENCE of a measurement, never
runs (and what runs, runs with no shell: its words reach the runner as written). Like any `NOT RUN` it is the ABSENCE of a measurement, never
evidence about the code, and never a finding of its own.
**A `NOT RUN` result is never on its own a finding.** It says nothing about
@ -111,6 +111,7 @@ column names the reason the runner gave:
| `placeholder` | the text is a template placeholder (`{exact command}`) |
| `not-a-command` | the sentence opened by NAMING a flag or a path (`--verbose`, `tests/`) rather than by invoking something. Running it would have produced exit 2 or exit 126 — a number about the sentence, not about the code |
| `outside the allowlist` | the command is not a known test-runner invocation, or it carries a shell operator. It was never run — say so; do not treat it as a broken criterion |
| `the program could not start` | the runner is not installed where the review ran. Nothing was measured — say so; do not treat it as a broken criterion |
In every one of those cases, judge the criterion on delivered code alone and
say in the Evidence column that no measurement exists. Emit `MISSING_TEST`

4
agents/brief-reviewer.md
View file

@ -62,8 +62,8 @@ is one test-runner invocation with no pipe, redirect or `&&`: `npm test`,
`python -m pytest`, `uv run pytest`, `cargo test`, `go test`, `make test`,
`bash tests/<script>.sh` (or `/bin/bash tests/<script>.sh`), or `git status|log|diff|show|ls-files`.
It runs with no shell, as plain words: no quote, `$`, brace, leading `~` or
`@`, no glob outside `node --test`, no `..` or absolute path, and only the
runner's read-only options.
`@`, no glob outside `node --test` (Node 21+), no `..` or absolute path, and
only the runner's read-only options.
Anything else — the product's own binary (`node bin/x.mjs …`), `jq`, `grep`,
`git grep`, a pipe — is reported NOT RUN. Never PROPOSE such a command as a
criterion. When a criterion needs one, propose wrapping it in a script under

9
commands/trekexecute.md
View file

@ -1486,7 +1486,7 @@ already rejects such a step (`PLAN_STEP_VERIFY_NOT_RUNNABLE`).
| Exit | `steps.{N}.red_proof` | Executor |
|------|-----------------------|----------|
| 0 | `"proven"` | continue |
| 1, `not-red-at-base`, `tests_only: true` | `"rejected"` | every file in the step's `Files:` is a test (the plan validator's `PLAN_STEP_TESTS_ONLY`, the same reading): the behaviour its tests cover already exists at the start point (an earlier step built it, or it was there before the plan), so the step can never be red and no rewrite of the test can fix that. The plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4), with no fix round — the planner moves the test into the step that builds the behaviour, or makes a test meant to pass from the start the `Regression:` of the step that changes the code |
| 1, `not-red-at-base`, `tests_only: true` | `"rejected"` | every file in the step's `Files:` is a test (the plan validator's `PLAN_STEP_TESTS_ONLY`, the same reading), and so is every file the step changed since its start point: the behaviour its tests cover already exists at the start point (an earlier step built it, or it was there before the plan), so the step can never be red and no rewrite of the test can fix that. The plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4), with no fix round — the planner moves the test into the step that builds the behaviour, or makes a test meant to pass from the start the `Regression:` of the step that changes the code |
| 1, `not-red-at-base`, otherwise | `"rejected"` | the test passes without the change, so the TEST is wrong: a finding for Sub-step F, and that fix round is the one exception to the lock — the executor releases the step's test lock for it, the fix helper rewrites the test, passes the `--red-only` gate and locks again. `tests_only: null` (the plan or the step could not be read) counts here too |
| 1, `not-green-at-result` | `"rejected"` | the change does not make the test pass: a finding for Sub-step F |
| 2 | `"not-run"` | never a pass and never proven. **plan_version ≥ 1.8:** when the reason is the command (the `Verify:` is not one test-runner invocation), the plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4); any other reason (a missing test file, an unresolvable start point, a control run that stays red) is a finding for Sub-step F. **Below 1.8:** a step with no `Test first:`, or whose `Verify:` is not a test-runner command, is tolerated as `"not-run"` — counted in the final report among the steps without proof of red, never called proven; any other reason is a finding |
@ -1742,7 +1742,7 @@ CLI already speaks progress.json's words (`"blocked"` → `"unrunnable"`,
|------|------------------------|----------|
| 0 | `"passed"`, or `"absent"` when the step has no `Regression:` (nothing new happens) | continue |
| 1 | `"failed"` (the suite went red) or `"unrunnable"` (not one allowlisted command; `"blocked"` by the denylist counts here too) | the step is not done: a finding for Sub-step F, exactly like a red Verify — it goes into the fix rounds, never to the checkpoint |
| 2 | `"not-run"` | the CLI could not run (plan unreadable, no such step): a finding for Sub-step F, never green |
| 2 | `"not-run"` | the CLI could not run (plan unreadable, no such step), or the runner could not start (not installed): a finding for Sub-step F, never green |
#### Sub-step E — Control
@ -1986,8 +1986,9 @@ If any fail: record which failed. Include in final report.
Run the criteria runner over the plan file parsed in Phase 1
(`{plan_path}`). It parses the `## Verification` section, screens every command
through the plugin's own PreToolUse denylist before it reaches a shell, and
runs what survives — in the FOREGROUND, never backgrounded. (A command spawned
through the plugin's own PreToolUse denylist before it runs (with no shell:
its words reach the runner as written), and runs what survives — in the
FOREGROUND, never backgrounded. (A command spawned
from node does not pass through the Bash tool, so that hook cannot fire by
itself; the runner invokes it over its documented stdin protocol instead of
carrying a second copy of the denylist.)

7
commands/trekplan.md
View file

@ -793,9 +793,10 @@ Write the plan following the template structure. The plan MUST include:
grep`) — with no pipe, redirect or `&&`. It runs with no shell: its words
reach the runner as written, so a word with a quote, backslash, `$`,
brace, or a leading `~` or `@` is refused, a glob only reaches
`node --test` (which expands its own patterns), no path climbs out with
`..` or starts at `/`, each runner takes a short list of read-only options,
and nothing may follow `make test`. Anything else is reported NOT
`node --test` on Node 21 and later (which expands its own patterns), no
path climbs out with `..` or starts at `/` (git included), each runner
takes a short list of read-only options, and nothing may follow
`make test`. A trailing `# comment` on a line of a shell fence is dropped. Anything else is reported NOT
RUN, never executed and never failed. A check that needs more than that is
wrapped in a script under `tests/` and declared as `bash tests/<script>.sh`.
9. **Estimated Scope** — File counts and complexity rating.

2
docs/HANDOVER-CONTRACTS.md
View file

@ -269,7 +269,7 @@ prose form (relaxed in S26 so the template's prose emission parses).
7. **Every plan_version, no version bump** (a plan without the field validates unchanged; voyage-lekerepo trial 2, 2026-10-08):
- **K10 — `Regression:`** (optional step field): ONE backticked command the criteria runner's allowlist runs, with no pipe and no `&&` (first backticked span, as for `Verify:`). The executor runs it green before the step's checkpoint; a step without it runs nothing new. `Verify:` stays the step's own red-first test. One that can never run is `PLAN_STEP_REGRESSION_NOT_RUNNABLE` (an error; a warning under `--soft`).
- **K11 — `On failure:`** opens with one of `revert`, `retry`, `skip`, `escalate` (case-insensitive; a leading `**`, `_`, backtick or `-` is ignored, other leading characters are not; `onFailureAction` in `lib/parsers/plan-schema.mjs`), the words `/trekexecute` reads; any other opening is the warning `PLAN_STEP_ON_FAILURE_UNKNOWN`. A missing `On failure:` is unchanged (the executor defaults to `escalate`).
- **A step of only tests** (order 10, voyage-lekerepo trial 3, 2026-10-09): when every file in a step's `Files:` is a test (its `Test first:` file, or a path `lib/util/test-path.mjs` reads as a test: under `tests/`, `test/`, `__tests__/`, `spec/`, `specs/`, or named `*.test.*`, `*.spec.*`, `*_test.*`, `*_spec.*`, `test_*.py`, but never a document such as `docs/specs/api.md`; `stepTestsOnly` in `lib/parsers/plan-schema.mjs`), the warning is `PLAN_STEP_TESTS_ONLY`: the behaviour its tests cover exists at its start point, so it can never be proven red. A step without `Files:` is not judged. At run time `red-proof.mjs --plan <plan.md> --step <N>` puts the same reading on a `not-red-at-base` as `tests_only`, and `/trekexecute` sends a `tests_only: true` step straight up to the plan.
- **A step of only tests** (order 10, voyage-lekerepo trial 3, 2026-10-09): when every file in a step's `Files:` is a test (its `Test first:` file, or a path `lib/util/test-path.mjs` reads as a test: under `tests/`, `test/`, `__tests__/`, `spec/`, `specs/`, or named `*.test.*`, `*.spec.*`, `*_test.*`, `*_spec.*`, `test_*.py`, but never a document such as `docs/specs/api.md`; `stepTestsOnly` in `lib/parsers/plan-schema.mjs`), the warning is `PLAN_STEP_TESTS_ONLY`: the behaviour its tests cover exists at its start point, so it can never be proven red. A step without `Files:` is not judged. At run time `red-proof.mjs --plan <plan.md> --step <N>` puts the same reading on a `not-red-at-base` as `tests_only`, true only when every file the step changed since its start point is a test too, and `/trekexecute` sends a `tests_only: true` step straight up to the plan.
**Validation strategy:**

4
hooks/scripts/pre-bash-executor.mjs
View file

@ -289,9 +289,9 @@ try {
}
// In the permission mode where Claude Code shows no prompts, an "ask" reaches
// no one and the call runs unattended. Everything this guard would ask about —
// no one and the call runs unattended. Everything this guard would ask about -
// merge into main, push, tag/release, destructive git, and a command it could
// not read — is refused there instead, with the way forward (operator
// not read - is refused there instead, with the way forward (operator
// decisions 2026-10-09: first merge and push, then the other two classes).
// /trekexecute relies on this: its main-merge gate is a lock, and this guard's
// question is the one question.

26
lib/parsers/plan-schema.mjs
View file

@ -293,7 +293,8 @@ export function onFailureAction(text) {
*
* - A field with a backticked path names its backticked spans that pass the
* path test (a `.`, `/` or `\`; no URL, option or call like `parse()`),
* wherever they sit (`a.mjs`/`b.mjs`, inside a note). Its descriptions are
* wherever they sit (`a.mjs`/`b.mjs`, inside a note), and a backticked name
* with no extension that stands alone in its entry (`Makefile`). Its descriptions are
* not read: "covers pass/fail", "JSON, YAML" or a directory named in passing
* is no path. An entry of its own with no backticks (", .gitignore") counts
* by its first word, and only when that is clearly a file or a directory.
@ -331,8 +332,11 @@ export function fieldPaths(text) {
for (const raw of masked.split(/[,\n]/)) {
const entry = raw.trim().replace(/^[-*]\s+/, '');
if (entry === '') continue;
const ticked = [...entry.matchAll(/\u0000(\d+)\u0000/g)].map((m) => Number(m[1])).filter((i) => isTicked[i]);
const inEntry = [...entry.matchAll(/\u0000(\d+)\u0000/g)].map((m) => Number(m[1]));
const ticked = inEntry.filter((i) => isTicked[i]);
if (ticked.length) { out.push(...ticked.map((i) => spans[i])); continue; }
// A backticked `Makefile` alone in its entry is a file (round 6).
if (inEntry.length === 1 && /^\u0000\d+\u0000$/.test(entry) && BARE_NAME.test(spans[inEntry[0]])) { out.push(spans[inEntry[0]]); continue; }
const words = entry.split(/\s+/).filter((w) => !w.includes('\u0000')).map(plainWord).filter(Boolean);
if (tickedField) {
const first = plainWord(entry.split(/\s+/)[0]);
@ -371,15 +375,23 @@ export function stepTestsOnly(body) {
return stepTestsOnlyFromFields(parseStepFields(body));
}
/** `stepTestsOnly` for a caller that already holds the step's parsed fields. */
export function stepTestsOnlyFromFields(f) {
const files = f.has('Files') ? fieldPaths(f.get('Files').text) : [];
/**
* Whether a path counts as a test for this step: its own `Test first:` file, or
* a path that is clearly a test. red-proof.mjs uses it on the step's diff.
* @returns {(path: string) => boolean}
*/
export function stepTestFile(f) {
const testFirst = f.has('Test first')
? [...f.get('Test first').text.matchAll(/\bFile:\**[ \t]*`([^`\n]+)`/gi)].map((m) => norm(m[1].trim()))
: [];
const isTest = (p) => testFirst.includes(p) ||
return (p) => testFirst.includes(p) ||
(isTestPath(p) && (UNDER_TESTS_RE.test(p) || !(DOC_RE.test(p) || DATA_RE.test(p))));
return { testsOnly: files.length > 0 && files.every(isTest), files };
}
/** `stepTestsOnly` for a caller that already holds the step's parsed fields. */
export function stepTestsOnlyFromFields(f) {
const files = f.has('Files') ? fieldPaths(f.get('Files').text) : [];
return { testsOnly: files.length > 0 && files.every(stepTestFile(f)), files };
}
// ---- sections (fence-aware) ------------------------------------------------

3
lib/util/test-lock.mjs
View file

@ -180,7 +180,8 @@ function main(argv) {
if (!screen.allowed) fail(2, `command blocked: ${screen.rule}`);
// No shell: the words the screen checked are the words that run.
const r = runCommand(testCmd, { cwd: cur.repo.root, timeoutMs: DEFAULT_TIMEOUT_MS });
if (r.error || r.status === null || r.status === 126 || r.status === 127) {
// With no shell, 126/127 is the test program's own exit: a red, not "could not run".
if (r.error || r.status === null) {
fail(2, `the test command did not run to an exit code (${r.error?.message ?? r.signal ?? `exit ${r.status}`})`);
}
if (r.status !== 0) fail(1, `the test command exited ${r.status}; the lock stays unmarked`);

54
lib/verification/criteria-runner.mjs
View file

@ -161,11 +161,14 @@ export function firstCommand(text) {
// A command line inside a shell-tagged fence. A console block may prefix the
// line with a `$` or `>` prompt; blank and comment-only lines declare nothing.
// A `#` root prompt is NOT stripped: it is indistinguishable from a comment,
// and running a comment is the worse of the two mistakes.
// and running a comment is the worse of the two mistakes. A trailing comment
// (`npm test # all green`) is not part of the command: a shell dropped it,
// and with no shell (runCommand) its words would reach the screen and make an
// honest check NOT RUN (release review of 6.3.0, round 6).
function fencedCommand(text) {
const line = text.trim();
if (line === '' || line.startsWith('#')) return null;
return line.replace(/^[$>]\s+/, '');
return line.replace(/^[$>]\s+/, '').replace(/\s+#.*$/, '');
}
function parseSection(markdown, heading, prefix) {
@ -292,9 +295,6 @@ const SHELL_OPERATORS = /[;&|<>]|\$\(|`/;
// Read-only git subcommands: they report, they do not write a repository.
const READONLY_GIT = new Set(['status', 'log', 'diff', 'show', 'ls-files']);
// A word that means the same to git with or without a shell (refs like HEAD~1
// and HEAD^, formats like %h, paths).
const GIT_LITERAL_WORD = /^[A-Za-z0-9_@%^~:.,=+/-]+$/;
// `--output`, `--output=<file>`, or an abbreviation of it git may accept.
function writesOutput(word) {
const name = word.split('=')[0];
@ -304,14 +304,18 @@ function writesOutput(word) {
const deny = (reason) => ({ allowed: false, reason });
const permit = () => ({ allowed: true, reason: '' });
// A word that means the same with or without a shell.
// A word that means the same with or without a shell (also git's refs like
// HEAD~1 and HEAD^, formats like %h, paths).
const PLAIN_WORD = /^[A-Za-z0-9_@%^~:.,=+/-]+$/;
// The same, with a glob in it: passed on only to `node --test`, which expands
// its own patterns (sh would have expanded `.?` to `..`).
// its own patterns from Node 21 on (sh would have expanded `.?` to `..`).
const GLOB_WORD = /^[A-Za-z0-9_./][A-Za-z0-9_@%^~:.,=+/*?-]*$/;
const takesGlobs = (t) => t[0] === 'node' && t[1] === '--test';
const NODE_GLOB_MAJOR = 21;
const runningNodeMajor = () => Number(process.versions.node.split('.')[0]);
// A path outside the repo: one that climbs with `..` or starts at `/`.
const leavesRepo = (w) => w.startsWith('/') || w.split('/').includes('..');
// The same for a git word, which may also name a path as `<rev>:<path>`.
const gitLeavesRepo = (w) => leavesRepo(w) || (w.includes(':') && leavesRepo(w.slice(w.indexOf(':') + 1)));
// After each runner's fixed words: the options it may take (`name` or
// `name=value`; `value` lists the ones whose value may be the next word, and
@ -415,9 +419,13 @@ const RUNNERS = {
// diff/log/show write to a file (release review of 6.3.0, 2026-10-10: a
// plan's check could overwrite any file, first plainly, then through
// quoting while a shell still ran it).
const odd = t.find((w) => !GIT_LITERAL_WORD.test(w));
const odd = t.find((w) => !PLAIN_WORD.test(w));
if (odd !== undefined) return deny(`git with a word that means something else in a shell (${odd}): only plain words run, as written`);
if (t.some(writesOutput)) return deny('git with --output writes a file, so it is not a read-only check');
// A check runs in the repo: no path that climbs out or starts at /, and no
// --no-index, which makes git diff compare any two files (round 6).
const out = t.slice(2).find((w) => w === '--no-index' || (!w.startsWith('-') && gitLeavesRepo(w)));
if (out !== undefined) return deny(`git with a path outside the repo (${out}): a check runs in the repo`);
return permit();
},
};
@ -463,12 +471,18 @@ function screenShape(command, opts, writeTime) {
const form = RUNNERS[tokens[0]](tokens, cwd, writeTime);
if (!form.allowed || tokens[0] === 'git') return form;
const words = tokens.slice(1);
const odd = words.find((w) => !PLAIN_WORD.test(w) && !(takesGlobs(tokens) && GLOB_WORD.test(w)));
const nodeTest = tokens[0] === 'node' && tokens[1] === '--test';
const nodeMajor = opts.nodeMajor ?? runningNodeMajor();
const odd = words.find((w) => !PLAIN_WORD.test(w) && !(nodeTest && GLOB_WORD.test(w)));
if (odd !== undefined) {
return deny(GLOB_WORD.test(odd)
? `a glob (${odd}): no shell runs this, and \`${tokens[0]}\` does not expand one itself; name the files, or use \`node --test\``
: `a word that means something else in a shell (${odd}): only plain words run, as written`);
}
const glob = nodeTest && nodeMajor < NODE_GLOB_MAJOR ? words.find((w) => !PLAIN_WORD.test(w)) : undefined;
if (glob !== undefined) {
return deny(`a glob (${glob}): node ${nodeMajor} does not expand one itself (Node ${NODE_GLOB_MAJOR} and later do), and no shell runs this; name the files`);
}
const at = words.find((w) => w.startsWith('@'));
if (at !== undefined) return deny(`a word that starts with @ (${at}): pytest reads @file as a file of more arguments`);
const out = words.find((w) => !w.startsWith('-') && leavesRepo(w));
@ -496,11 +510,13 @@ export function runCommand(command, { cwd, timeoutMs = DEFAULT_TIMEOUT_MS } = {}
// --- running ----------------------------------------------------------------
// A program that could not start (not installed: ENOENT) or was killed by the
// timeout has no exit code; its error is the output, so the row says why.
// A program that could not start (not installed: ENOENT) has no exit code and
// measured nothing: `startError` says so, and runCriteria reports it NOT RUN.
// One killed by the timeout did start, and a hang is a red (`failed`).
function defaultExec(command, { cwd, timeoutMs }) {
const r = runCommand(command, { cwd, timeoutMs });
const why = r.error ? `${r.error.message}\n` : r.signal ? `killed (${r.signal})\n` : '';
if (r.status === null && r.signal === null && r.error) return { status: null, stdout: '', stderr: '', startError: r.error.message };
const why = r.signal ? `killed (${r.signal})\n` : '';
return { status: r.status, stdout: r.stdout ?? '', stderr: `${r.stderr ?? ''}${why}` };
}
@ -513,9 +529,10 @@ function cap(text, max) {
* Run each criterion that has a command. Returns one result per criterion:
* passed the command exited 0
* failed the command exited non-zero (or was killed by the timeout)
* blocked the executor denylist denied it — it never reached a shell
* blocked the executor denylist denied it — it never ran
* unrunnable there is nothing to run: no command (placeholder or prose-only
* criterion), or a command outside the allowlist
* criterion), a command outside the allowlist, or a runner that
* could not start (`startError`: not installed)
*/
export function runCriteria(criteria, opts = {}) {
const exec = opts.exec ?? defaultExec;
@ -545,6 +562,9 @@ export function runCriteria(criteria, opts = {}) {
return { ...base, status: 'blocked', exitCode: null, output: `blocked: ${verdict.rule}` };
}
const r = exec(c.command, { cwd, timeoutMs });
if (r.startError) {
return { ...base, status: 'unrunnable', exitCode: null, output: `not runnable: the program could not start (${r.startError})`, startError: r.startError };
}
const output = cap(`${r.stdout ?? ''}${r.stderr ?? ''}`, maxOutput);
return {
...base,
@ -766,8 +786,8 @@ export function formatCriteriaEvidence(rep) {
out.push(
'**NOT RUN is not a pass.** You cannot run these commands yourself — never infer, assume or ' +
'reconstruct a result for a criterion that has none. A criterion whose result is FAILED or ' +
'BLOCKED is `Broken`; a criterion marked NOT RUN declared no command, or a command outside ' +
'the allowlist of test runners, so it never reached a shell — that is the ABSENCE of a ' +
'BLOCKED is `Broken`; a criterion marked NOT RUN declared no command, a command outside ' +
'the allowlist of test runners, or a runner that could not start, so it never ran — that is the ABSENCE of a ' +
'measurement and never on its own a finding; a criterion with no result is judged on ' +
'delivered code alone, and you say so in the Evidence column.'
);

5
lib/verification/done-line.mjs
View file

@ -7,7 +7,8 @@
// appended to the step's report file. Exit 0 = written, 1 = the command failed
// (nothing written), 2 = could not run (nothing written; never green): no
// --step, a command outside the test-runner allowlist, a command that cannot
// start, a working tree that is not a git repo.
// start (not installed) or was killed, a working tree that is not a git repo.
// With no shell, an exit 126/127 is the test program's own and a failure.
//
// `--check --step <N> --report <file> [--commit <sha>]`: exit 0 when the report
// carries a well-formed line for step N (at exactly that commit, with
@ -57,7 +58,7 @@ export function writeDoneLine({ cwd, step, cmd, report, timeoutMs = DEFAULT_TIME
if (head.status !== 0) return { status: 'not-run', reason: `no git HEAD in ${dir}` };
// No shell: the words the screen checked are the words that run.
const r = runCommand(cmd, { cwd: dir, timeoutMs });
if (r.error || r.status === null || r.status === 126 || r.status === 127) {
if (r.error || r.status === null) {
return { status: 'not-run', reason: `the command did not run to an exit code (${r.error?.message ?? r.signal ?? `exit ${r.status}`})`, exit: r.status };
}
if (r.status !== 0) return { status: 'failed', reason: `the test command exited ${r.status}`, exit: r.status };

42
lib/verification/red-proof.mjs
View file

@ -15,8 +15,9 @@
// run, and only the full run is authoritative.
//
// `--plan <plan.md> --step <N>` adds `tests_only` to a `not-red-at-base`: true
// when every file in the step's `Files:` is a test, so the step can never be
// red and goes up to the plan (see `testsOnly` below).
// when every file in the step's `Files:` is a test and so is every file the
// step changed since its start point, so the step can never be red and goes
// up to the plan (see `testsOnly` below).
//
// Exit 0 = proven, 1 = rejected (`not-red-at-base` or `not-green-at-result`),
// 2 = could not run (`not-run`: no test files, an unknown start point, a command
@ -57,11 +58,11 @@
import { cpSync, existsSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, symlinkSync } from 'node:fs';
import { spawnSync } from 'node:child_process';
import { tmpdir } from 'node:os';
import { dirname, isAbsolute, join, resolve } from 'node:path';
import { dirname, isAbsolute, join, relative, resolve, sep } from 'node:path';
import { fileURLToPath } from 'node:url';
import { allowedCommand, runCommand, screenCommand } from './criteria-runner.mjs';
import { commandEnv } from '../util/command-env.mjs';
import { sliceSteps, stepTestsOnly } from '../parsers/plan-schema.mjs';
import { parseStepFields, sliceSteps, stepTestFile, stepTestsOnlyFromFields } from '../parsers/plan-schema.mjs';
export const DEFAULT_TIMEOUT_MS = 900_000;
const TAIL = 2000;
@ -110,23 +111,42 @@ const notRun = (reason, extra = {}) => ({ status: 'not-run', reason, ...extra })
// the executor sends the step up to the plan instead of into fix rounds
// (voyage-lekerepo trial 3, 2026-10-09). `null` = the plan or the step could
// not be read; without `--plan` the field is absent and nothing new happens.
function testsOnly(o) {
//
// The plan's `Files:` is prose, read by a reader that can misread it; the
// step's own diff is a fact. So a step the plan calls tests-only is so only
// when every file it changed since the start point is a test too (release
// review of 6.3.0, round 6). Before building (`--red-only`) the diff holds the
// step's test alone, so there the plan decides. The plan file itself and
// Voyage's own files under .claude/ (progress, approvals) are not the step's change.
function testsOnly(o, cwd, baseSha) {
if (!o.plan) return {};
let text;
try { text = readFileSync(o.plan, 'utf8'); }
catch (e) { return { tests_only: null, tests_only_note: `cannot read the plan: ${e.message}` }; }
const step = sliceSteps(text).find((s) => Number(s.n) === Number(o.step));
if (!step) return { tests_only: null, tests_only_note: `no "### Step ${o.step}:" in the plan` };
return { tests_only: stepTestsOnly(step.body).testsOnly };
const fields = parseStepFields(step.body);
if (!stepTestsOnlyFromFields(fields).testsOnly) return { tests_only: false };
const isTest = stepTestFile(fields);
const given = new Set(o.tests ?? []);
const changed = [
...lines(git(cwd, ['diff', '--name-only', '--no-renames', baseSha]).stdout),
...lines(git(cwd, ['ls-files', '--others', '--exclude-standard']).stdout),
];
const planRel = relative(cwd, realpathSync(o.plan)).split(sep).join('/');
const other = [...new Set(changed)].filter((p) => p !== planRel && !p.startsWith('.claude/') && !given.has(p) && !isTest(p));
if (other.length === 0) return { tests_only: true };
return { tests_only: false, tests_only_note: `the plan's Files: names only tests, but the step's diff also changes ${other.slice(0, 5).join(', ')}` };
}
// A program that could not start (a spawn error: not installed), a signal (the
// timeout), or 126/127 from a script that could not run its own command is not
// a test verdict at all.
// A program that could not start (a spawn error: not installed) or a signal
// (the timeout) is not a test verdict at all. With no shell, an exit 126/127
// is the test program's own (a script whose tool the step has not built yet):
// a red like any other, and the control run tells an environment fault apart
// (release review of 6.3.0, round 6).
function unrunnable(r) {
if (r.spawnError) return `the command could not start: ${r.spawnError}`;
if (r.exit === null) return `the command was killed (${r.signal ?? 'timeout'})`;
if (r.exit === 126 || r.exit === 127) return `the command could not be run (exit ${r.exit})`;
return null;
}
@ -221,7 +241,7 @@ export function proveRed(o) {
const baseBroken = unrunnable(base);
if (baseBroken) return notRun(`at the start point: ${baseBroken}`, { base, result, linked });
const out = { base_sha: baseSha, linked, base, result };
if (base.exit === 0) return { status: 'rejected', reason: 'not-red-at-base', ...testsOnly(o), ...out };
if (base.exit === 0) return { status: 'rejected', reason: 'not-red-at-base', ...testsOnly(o, cwd, baseSha), ...out };
if (o.redOnly) return { status: 'proven', reason: 'red at the start point (--red-only: the result was not run)', red_kind: redKind(base), base_sha: baseSha, linked, base };
if (result.exit !== 0) return { status: 'rejected', reason: 'not-green-at-result', ...out };

10
lib/verification/regression-check.mjs
View file

@ -9,7 +9,9 @@
// 1 = it failed, the executor denylist blocked it, or it is not runnable
// (no command, or a command outside the test-runner allowlist): the
// step is not done, and that is a finding for the fix rounds
// 2 = could not run: bad usage, a plan that cannot be read, no step N
// 2 = could not run: bad usage, a plan that cannot be read, no step N, or
// a runner that could not start (not installed): nothing was measured,
// as red-proof and done-line say for the same case
// JSON on stdout: {step, status, progress_status, command, exit_code, output}
// status: passed | failed | blocked | unrunnable | absent | error
// progress_status: the value progress.json takes for steps.N.regression
@ -55,13 +57,15 @@ export function checkRegression({ planText, step, cwd = process.cwd(), run = (c,
if (!field) return withProgress({ ...base, status: 'absent', output: 'no Regression: field — nothing to run', exit: EXIT.absent });
const { command, reason } = firstCommand(field.text);
const [r] = run([{ label: `R${step}`, text: field.text, command, reason }], { cwd });
// A runner that could not start measured nothing: not a red suite (round 6).
const status = r.startError ? 'error' : r.status;
return withProgress({
step,
status: r.status,
status,
command: r.command ?? null,
exit_code: r.exitCode ?? null,
output: r.output ?? '',
exit: EXIT[r.status] ?? EXIT.error,
exit: EXIT[status] ?? EXIT.error,
});
}

14
scripts/release-check.mjs
View file

@ -156,10 +156,16 @@ function checkTree(repo) {
return { id: 'tree', status: 'pass', detail: 'working tree clean' };
}
function checkSuite(repo, treeClean) {
// A suite that never ends is stopped, as every other check runner here stops
// its command (15 minutes), and read as not measured (release review of
// 6.3.0, round 6): the operator always gets an answer.
export const SUITE_TIMEOUT_MS = 900_000;
function checkSuite(repo, treeClean, timeoutMs = SUITE_TIMEOUT_MS) {
if (!treeClean) return { id: 'suite', status: 'fail', detail: 'suite not run: the tree has uncommitted changes, so it would not measure HEAD' };
// commandEnv: under a parent `node --test` the suite would report to it and exit 0 whatever it found.
const r = spawnSync('npm', ['test'], { cwd: repo, encoding: 'utf8', maxBuffer: 256 * 1024 * 1024, env: commandEnv() });
const r = spawnSync('npm', ['test'], { cwd: repo, encoding: 'utf8', maxBuffer: 256 * 1024 * 1024, env: commandEnv(), timeout: timeoutMs });
if (r.signal) return { id: 'suite', status: 'error', detail: `npm test was killed (${r.signal}) after the ${Math.round(timeoutMs / 1000)} s timeout — not measured` };
if (r.error) return { id: 'suite', status: 'error', detail: `npm test could not start: ${r.error.message} — not measured` };
const out = `${r.stdout ?? ''}\n${r.stderr ?? ''}`;
const summary = out.split('\n').filter((l) => /^ℹ (tests|pass|fail|skipped) /.test(l)).map((l) => l.slice(2));
@ -263,7 +269,7 @@ function resolveVersion(repo, explicit) {
* Run every check. Pure of process exit; the CLI below prints and exits.
* @returns {{ exit: 0|1|2, version: string|null, repo: string, checks: object[], reason?: string }}
*/
export function releaseCheck({ repo, review, version: explicitVersion }) {
export function releaseCheck({ repo, review, version: explicitVersion, suiteTimeoutMs }) {
const top = git(repo, ['rev-parse', '--show-toplevel']);
if (top.error || top.status !== 0) return { exit: EXIT.UNDECIDED, version: null, repo, checks: [], reason: `${repo} is not a git repository` };
// The plugin's own directory, not the git top level: a plugin may sit in a
@ -281,7 +287,7 @@ export function releaseCheck({ repo, review, version: explicitVersion }) {
const undecided = [tree, rev, coverage, door].filter((c) => c.status === 'error');
const suite = undecided.length > 0
? { id: 'suite', status: 'skip', detail: `not run: ${undecided.map((c) => c.id).join(', ')} could not be measured, so the answer is exit 2 already` }
: checkSuite(root, tree.status === 'pass');
: checkSuite(root, tree.status === 'pass', suiteTimeoutMs);
const checks = [tree, suite, rev, coverage, door];
let exit = EXIT.READY;

2
tests/fixtures/jsonl-schemas.md vendored
View file

@ -22,7 +22,7 @@
| trekbrief-stats | ts, task, slug, mode, interview_turns, review_iterations, brief_quality, research_topics, auto_research, auto_result, intent_approved, project_dir, order_id, order_claimed_at, order_reason | commands/trekbrief.md (orchestrator-emit Phase 7; order_* added by lib/stats/stats-append.mjs) | trekbrief.md:1130-1145 | profile, phase_models, profile_source | none |
| trekresearch-stats | ts, question, mode, scope, engine, slug, project_dir, brief_path, dimensions, dimensions_baseline, dimensions_baseline_preserved, effort, conv_turns, empty_turns, unique_sources, agents_local, agents_external, gemini_used, confidence, contradictions, open_questions | lib/stats/stats-append.mjs (trekresearch.md Stats tracking) | trekresearch.md:782-808 | profile, phase_models, parallel_agents, external_research_enabled, profile_source | none |
| trekplan-stats | ts, task, mode, slug, brief_path, project_dir, codebase_size, codebase_files, agents_deployed, deep_dives, research_briefs_used, research_scout_used, critic_verdict, guardian_verdict, outcome | lib/stats/stats-append.mjs (trekplan.md Phase 12) | trekplan.md:1115-1135 | profile, phase_models, parallel_agents, profile_source | none |
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2891-2913 | profile, phase_models, profile_source | none |
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2892-2914 | profile, phase_models, profile_source | none |
| trekexecute-stats (autonomy events) | ts, event, known_event, payload | lib/stats/event-emit.mjs `emit()` | event-emit.mjs:78-99 | payload.profile, payload.phase_models, payload.profile_source | none |
| trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | lib/stats/stats-append.mjs (trekreview.md Phase 8) | trekreview.md:628-632 | profile, phase_models, profile_source | none |

4
tests/lib/criteria-runner.test.mjs
View file

@ -394,7 +394,9 @@ const SHELL_BLOCK_PLAN = [
test('parsePlanVerification: a fenced shell block yields one criterion per command line', () => {
const criteria = parsePlanVerification(SHELL_BLOCK_PLAN);
assert.deepEqual(criteria.map((c) => c.label), ['V1', 'V2']);
assert.equal(criteria[0].command, 'npm test # all green');
// A trailing comment is not part of the command: with no shell to drop it,
// its words would make the check NOT RUN (release review of 6.3.0, round 6).
assert.equal(criteria[0].command, 'npm test');
assert.equal(criteria[1].command, 'node --test tests/lib/x.test.mjs');
});

122
tests/lib/review-630.test.mjs
View file

@ -448,8 +448,9 @@ test('review 630 r5 / da7584e7: a check runs without a shell - its words reach t
assert.equal(r.status, 'passed', r.output);
assert.equal(r.output, '~/x\n*.sh\n$HOME\n', 'no shell expanded a tilde, a glob or a variable');
const [missing] = runCriteria([{ label: 'X2', text: 'x', command: 'no-such-runner-r630 --x' }], { cwd: dir, ...PASS_SCREENS });
assert.equal(missing.status, 'failed', 'a runner that is not installed is a failure, never a pass');
assert.match(missing.output, /ENOENT|not found/i, missing.output);
// Round 6: a runner that could not start is NOT RUN (no measurement), never a pass and never a red.
assert.equal(missing.status, 'unrunnable', 'a runner that is not installed is not run, never a pass');
assert.match(missing.output, /could not start.*ENOENT/i, missing.output);
} finally { rmSync(dir, { recursive: true, force: true }); }
});
@ -586,3 +587,120 @@ test('review 630 r5 / 41c43285 (operator 2026-10-09): the example plan keeps ste
const warned = r.warnings.filter((w) => w.code === 'PLAN_STEP_TESTS_ONLY').map((w) => w.message);
assert.equal(warned.length, 2, `the operator kept the warning on steps 1-2: ${JSON.stringify(warned)}`);
});
// ==== round 6 of the release review (2026-10-11), each red on c3641ff ===========
function gitRepo(prefix) {
const dir = mkdtempSync(join(tmpdir(), prefix));
for (const args of [['init', '-q'], ['config', 'user.email', 't@example.invalid'], ['config', 'user.name', 't'], ['config', 'commit.gpgsign', 'false']]) {
assert.equal(spawnSync('git', args, { cwd: dir }).status, 0);
}
return dir;
}
const commitAll = (dir, msg) => {
assert.equal(spawnSync('git', ['add', '-A'], { cwd: dir }).status, 0);
assert.equal(spawnSync('git', ['commit', '-q', '-m', msg], { cwd: dir }).status, 0);
return spawnSync('git', ['rev-parse', 'HEAD'], { cwd: dir, encoding: 'utf8' }).stdout.trim();
};
test('review 630 r6 / criteria-runner.mjs:464: a git check stays in the repo too', async () => {
const { allowedCommand } = await import('../../lib/verification/criteria-runner.mjs');
for (const cmd of ['git diff /etc/hosts /dev/null', 'git diff ../x tests/a.test.mjs', 'git show HEAD:../x', 'git log -- /etc',
'git diff --no-index tests/a tests/b']) {
assert.equal(allowedCommand(cmd, { cwd: ROOT }).allowed, false, `${cmd} must not run`);
}
for (const cmd of ['git diff HEAD~1..HEAD --stat', 'git show HEAD:package.json', 'git log -- lib/', 'git show @~1 --stat']) {
const r = allowedCommand(cmd, { cwd: ROOT });
assert.equal(r.allowed, true, `the control: ${cmd} (${r.reason})`);
}
});
test('review 630 r6 / criteria-runner.mjs:168: a trailing # comment in a shell fence is not part of the command', async () => {
const { parsePlanVerification, allowedCommand } = await import('../../lib/verification/criteria-runner.mjs');
const plan = '## Verification\n\n```bash\nnpm test # all green\nnode --test tests/a.test.mjs #1\n```\n';
assert.deepEqual(parsePlanVerification(plan).map((c) => c.command), ['npm test', 'node --test tests/a.test.mjs']);
const example = parsePlanVerification(read('examples/01-add-verbose-flag/plan.md'))[0];
assert.equal(example.command, 'node --test tests/');
assert.equal(allowedCommand(example.command, { cwd: ROOT }).allowed, true, "the shipped example's first check runs again");
});
test('review 630 r6 / criteria-runner.mjs:466: a glob reaches node --test only where node expands it (Node 21+)', async () => {
const { allowedCommand } = await import('../../lib/verification/criteria-runner.mjs');
const old = allowedCommand('node --test tests/lib/*.test.mjs', { cwd: ROOT, nodeMajor: 20 });
assert.equal(old.allowed, false);
assert.match(old.reason, /21/);
assert.equal(allowedCommand('node --test tests/lib/*.test.mjs', { cwd: ROOT, nodeMajor: 22 }).allowed, true);
assert.equal(allowedCommand('node --test tests/lib/review-630.test.mjs', { cwd: ROOT, nodeMajor: 20 }).allowed, true, 'the control: no glob, any Node');
});
test('review 630 r6 / plan-schema.mjs:318: a backticked file name with no extension is a path', () => {
const step = (files) => `- **Files:** ${files}\n- **Changes:** x\n`;
assert.deepEqual(stepTestsOnly(step('`tests/build.test.mjs`, `Makefile`')), { testsOnly: false, files: ['tests/build.test.mjs', 'Makefile'] });
assert.deepEqual(stepTestsOnly(step('`spec/a_spec.rb`, `Gemfile`')).testsOnly, false);
assert.deepEqual(stepTestsOnly(step('`tests/a.test.mjs` — covers `parse` and `JSON`')).files, ['tests/a.test.mjs'], 'a span inside a description is no path');
});
test('review 630 r6 / plan-schema.mjs:329 (altitude): red proof checks tests-only against the step\'s own diff', async () => {
const { proveRed } = await import('../../lib/verification/red-proof.mjs');
const work = mkdtempSync(join(tmpdir(), 'r630-diff-'));
const repo = gitRepo('r630-diff-repo-');
try {
writeFileSync(join(repo, 'README.md'), 'x\n');
const base = commitAll(repo, 'base');
mkdirSync(join(repo, 'tests'));
writeFileSync(join(repo, 'tests', 'a.test.mjs'), "import { test } from 'node:test';\ntest('a', () => {});\n");
writeFileSync(join(repo, 'Makefile'), 'test:\n\tnode --test tests/\n');
// The prose reads only the test (a plain Makefile beside backticked paths is a description word).
const plan = join(work, 'plan.md');
writeFileSync(plan, '## Implementation Plan\n\n### Step 1: x\n\n- **Files:** `tests/a.test.mjs`, Makefile\n- **Changes:** x\n');
const o = { cwd: repo, base, cmd: 'node --test tests/a.test.mjs', tests: ['tests/a.test.mjs'], plan, step: 1 };
const r = proveRed(o);
assert.equal(r.reason, 'not-red-at-base', JSON.stringify(r));
assert.equal(r.tests_only, false, `the step's diff changes Makefile: ${JSON.stringify(r.tests_only_note ?? null)}`);
assert.match(r.tests_only_note ?? '', /Makefile/);
rmSync(join(repo, 'Makefile'));
assert.equal(proveRed(o).tests_only, true, 'the control: a step whose diff holds only its test is tests-only');
} finally { rmSync(work, { recursive: true, force: true }); rmSync(repo, { recursive: true, force: true }); }
});
test('review 630 r6 / red-proof.mjs:129: exit 127 from the test program itself is a red, not "could not run"', async () => {
const { proveRed } = await import('../../lib/verification/red-proof.mjs');
const { writeDoneLine } = await import('../../lib/verification/done-line.mjs');
const repo = gitRepo('r630-127-');
try {
writeFileSync(join(repo, 'README.md'), 'x\n');
const base = commitAll(repo, 'base');
mkdirSync(join(repo, 'tests'));
mkdirSync(join(repo, 'bin'));
writeFileSync(join(repo, 'tests', 'e2e.sh'), './bin/tool\n');
writeFileSync(join(repo, 'bin', 'tool'), '#!/bin/sh\nexit 0\n', { mode: 0o755 });
const r = proveRed({ cwd: repo, base, cmd: 'bash tests/e2e.sh', tests: ['tests/e2e.sh'], redOnly: true });
assert.equal(r.status, 'proven', JSON.stringify(r));
assert.equal(r.base.exit, 127);
rmSync(join(repo, 'bin'), { recursive: true });
const d = writeDoneLine({ cwd: repo, step: 1, cmd: 'bash tests/e2e.sh', report: join(repo, 'report.md') });
assert.equal(d.status, 'failed', `a test that exits 127 failed: ${JSON.stringify(d)}`);
} finally { rmSync(repo, { recursive: true, force: true }); }
});
test('review 630 r6 / regression-check.mjs:64: a Regression: whose runner cannot start is not-run, not a red suite', async () => {
const { checkRegression } = await import('../../lib/verification/regression-check.mjs');
const empty = mkdtempSync(join(tmpdir(), 'r630-path-'));
const saved = process.env.PATH;
try {
process.env.PATH = empty;
const planText = '## Implementation Plan\n\n### Step 1: x\n\n- **Files:** `src/a.mjs`\n- **Regression:** `vitest run`\n';
const r = checkRegression({ planText, step: 1, cwd: ROOT });
assert.equal(r.status, 'error', JSON.stringify(r));
assert.equal(r.progress_status, 'not-run');
assert.equal(r.exit, 2);
assert.match(r.output, /could not start/);
} finally { process.env.PATH = saved; rmSync(empty, { recursive: true, force: true }); }
});
test('review 630 r6 / hooks are ASCII: the lines this release added to pre-bash-executor.mjs', () => {
const src = read('hooks/scripts/pre-bash-executor.mjs');
const block = src.slice(src.indexOf('Everything this guard would ask about') - 200, src.indexOf('Everything this guard would ask about') + 300);
assert.ok(block.includes('Everything this guard would ask about'));
assert.doesNotMatch(block, /[^\x00-\x7f]/, 'a hook script is ASCII (~/.claude/CLAUDE.md, Miljø)');
});

13
tests/scripts/release-check.test.mjs
View file

@ -449,3 +449,16 @@ test('release-check r5 — a test moved into an allowed place after the review i
assert.match(r.stdout, /guard\.test\.mjs/);
} finally { fx.cleanup(); }
});
// ==== round 6 of the release review of 6.3.0 (2026-10-11) ====================
test('release-check r6 — a suite that never ends is stopped and read as not measured → 2', () => {
const fx = fixture({ testScript: 'node -e "setTimeout(() => {}, 4000)"' });
try {
const r = mod.releaseCheck({ repo: fx.repo, review: fx.review, suiteTimeoutMs: 1000 });
const suite = r.checks.find((c) => c.id === 'suite');
assert.equal(suite.status, 'error', JSON.stringify(suite));
assert.match(suite.detail, /timeout|killed/i);
assert.equal(r.exit, 2);
} finally { fx.cleanup(); }
});