open/voyage
1
0
Fork 0
Code Issues Releases 13 Activity

fix: round 6 of the release review of 6.3.0

Browse source 
Round 6 (v6.2.0..c3641ff, WARN: 11 MAJOR, 3 MINOR; 0 BLOCKER). The round-5
fixes hold (code-correctness-reviewer); three of the new MAJOR were side
effects of running checks with no shell. Each fixed behind a test that
failed first on c3641ff.

- git checks stay in the repo too: no path that climbs out with .., starts
  at / or reads <rev>:../x, and no --no-index (both reviewers).
- A trailing `# comment` on a shell-fence line is dropped: with no shell,
  its words made an honest check NOT RUN, among them the first check of the
  shipped example plan.
- A glob reaches `node --test` only on Node 21 and later, which expands it
  itself; on an older Node it is refused with that reason (engines >=18).
- With no shell, exit 126/127 is the test program's own: red proof reads it
  as a red (the control run still tells an environment fault apart), the
  done line and the test lock as a failure.
- A runner that could not start (not installed) is NOT RUN in the criteria
  runner and not-run (exit 2) in regression-check, as red-proof and
  done-line already said; never a red suite.
- Tests-only: a backticked `Makefile` alone in its entry is a path, and red
  proof confirms the plan's tests-only reading against the step's own diff
  (the altitude finding: the diff is a fact, the Files: prose is not).
- release-check stops a suite that never ends (15 minutes) and reads it as
  not measured (exit 2).
- The two lines this release added to pre-bash-executor.mjs are ASCII.
- MINOR: git and the other runners share one plain-word pattern.

Not in this round: MINOR f00a2eb8 (shared norm/ticked-path and isMain
helpers) and MINOR on plan-review-dedup's repeated tokenizing.

Suite 2194 (2192/0/2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-10-11 01:07:47 +02:00
commit 45aa9e107a
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
17 changed files with 262 additions and 62 deletions
Download patch file Download diff file Expand all files Collapse all files

54
lib/verification/criteria-runner.mjs
View file

@ -161,11 +161,14 @@ export function firstCommand(text) {
// A command line inside a shell-tagged fence. A console block may prefix the
// line with a `$` or `>` prompt; blank and comment-only lines declare nothing.
// A `#` root prompt is NOT stripped: it is indistinguishable from a comment,
// and running a comment is the worse of the two mistakes.
// and running a comment is the worse of the two mistakes. A trailing comment
// (`npm test # all green`) is not part of the command: a shell dropped it,
// and with no shell (runCommand) its words would reach the screen and make an
// honest check NOT RUN (release review of 6.3.0, round 6).
function fencedCommand(text) {
const line = text.trim();
if (line === '' || line.startsWith('#')) return null;
return line.replace(/^[$>]\s+/, '');
return line.replace(/^[$>]\s+/, '').replace(/\s+#.*$/, '');
}
function parseSection(markdown, heading, prefix) {
@ -292,9 +295,6 @@ const SHELL_OPERATORS = /[;&|<>]|\$\(|`/;
// Read-only git subcommands: they report, they do not write a repository.
const READONLY_GIT = new Set(['status', 'log', 'diff', 'show', 'ls-files']);
// A word that means the same to git with or without a shell (refs like HEAD~1
// and HEAD^, formats like %h, paths).
const GIT_LITERAL_WORD = /^[A-Za-z0-9_@%^~:.,=+/-]+$/;
// `--output`, `--output=<file>`, or an abbreviation of it git may accept.
function writesOutput(word) {
const name = word.split('=')[0];
@ -304,14 +304,18 @@ function writesOutput(word) {
const deny = (reason) => ({ allowed: false, reason });
const permit = () => ({ allowed: true, reason: '' });
// A word that means the same with or without a shell.
// A word that means the same with or without a shell (also git's refs like
// HEAD~1 and HEAD^, formats like %h, paths).
const PLAIN_WORD = /^[A-Za-z0-9_@%^~:.,=+/-]+$/;
// The same, with a glob in it: passed on only to `node --test`, which expands
// its own patterns (sh would have expanded `.?` to `..`).
// its own patterns from Node 21 on (sh would have expanded `.?` to `..`).
const GLOB_WORD = /^[A-Za-z0-9_./][A-Za-z0-9_@%^~:.,=+/*?-]*$/;
const takesGlobs = (t) => t[0] === 'node' && t[1] === '--test';
const NODE_GLOB_MAJOR = 21;
const runningNodeMajor = () => Number(process.versions.node.split('.')[0]);
// A path outside the repo: one that climbs with `..` or starts at `/`.
const leavesRepo = (w) => w.startsWith('/') || w.split('/').includes('..');
// The same for a git word, which may also name a path as `<rev>:<path>`.
const gitLeavesRepo = (w) => leavesRepo(w) || (w.includes(':') && leavesRepo(w.slice(w.indexOf(':') + 1)));
// After each runner's fixed words: the options it may take (`name` or
// `name=value`; `value` lists the ones whose value may be the next word, and
@ -415,9 +419,13 @@ const RUNNERS = {
// diff/log/show write to a file (release review of 6.3.0, 2026-10-10: a
// plan's check could overwrite any file, first plainly, then through
// quoting while a shell still ran it).
const odd = t.find((w) => !GIT_LITERAL_WORD.test(w));
const odd = t.find((w) => !PLAIN_WORD.test(w));
if (odd !== undefined) return deny(`git with a word that means something else in a shell (${odd}): only plain words run, as written`);
if (t.some(writesOutput)) return deny('git with --output writes a file, so it is not a read-only check');
// A check runs in the repo: no path that climbs out or starts at /, and no
// --no-index, which makes git diff compare any two files (round 6).
const out = t.slice(2).find((w) => w === '--no-index' || (!w.startsWith('-') && gitLeavesRepo(w)));
if (out !== undefined) return deny(`git with a path outside the repo (${out}): a check runs in the repo`);
return permit();
},
};
@ -463,12 +471,18 @@ function screenShape(command, opts, writeTime) {
const form = RUNNERS[tokens[0]](tokens, cwd, writeTime);
if (!form.allowed || tokens[0] === 'git') return form;
const words = tokens.slice(1);
const odd = words.find((w) => !PLAIN_WORD.test(w) && !(takesGlobs(tokens) && GLOB_WORD.test(w)));
const nodeTest = tokens[0] === 'node' && tokens[1] === '--test';
const nodeMajor = opts.nodeMajor ?? runningNodeMajor();
const odd = words.find((w) => !PLAIN_WORD.test(w) && !(nodeTest && GLOB_WORD.test(w)));
if (odd !== undefined) {
return deny(GLOB_WORD.test(odd)
? `a glob (${odd}): no shell runs this, and \`${tokens[0]}\` does not expand one itself; name the files, or use \`node --test\``
: `a word that means something else in a shell (${odd}): only plain words run, as written`);
}
const glob = nodeTest && nodeMajor < NODE_GLOB_MAJOR ? words.find((w) => !PLAIN_WORD.test(w)) : undefined;
if (glob !== undefined) {
return deny(`a glob (${glob}): node ${nodeMajor} does not expand one itself (Node ${NODE_GLOB_MAJOR} and later do), and no shell runs this; name the files`);
}
const at = words.find((w) => w.startsWith('@'));
if (at !== undefined) return deny(`a word that starts with @ (${at}): pytest reads @file as a file of more arguments`);
const out = words.find((w) => !w.startsWith('-') && leavesRepo(w));
@ -496,11 +510,13 @@ export function runCommand(command, { cwd, timeoutMs = DEFAULT_TIMEOUT_MS } = {}
// --- running ----------------------------------------------------------------
// A program that could not start (not installed: ENOENT) or was killed by the
// timeout has no exit code; its error is the output, so the row says why.
// A program that could not start (not installed: ENOENT) has no exit code and
// measured nothing: `startError` says so, and runCriteria reports it NOT RUN.
// One killed by the timeout did start, and a hang is a red (`failed`).
function defaultExec(command, { cwd, timeoutMs }) {
const r = runCommand(command, { cwd, timeoutMs });
const why = r.error ? `${r.error.message}\n` : r.signal ? `killed (${r.signal})\n` : '';
if (r.status === null && r.signal === null && r.error) return { status: null, stdout: '', stderr: '', startError: r.error.message };
const why = r.signal ? `killed (${r.signal})\n` : '';
return { status: r.status, stdout: r.stdout ?? '', stderr: `${r.stderr ?? ''}${why}` };
}
@ -513,9 +529,10 @@ function cap(text, max) {
* Run each criterion that has a command. Returns one result per criterion:
* passed the command exited 0
* failed the command exited non-zero (or was killed by the timeout)
* blocked the executor denylist denied it — it never reached a shell
* blocked the executor denylist denied it — it never ran
* unrunnable there is nothing to run: no command (placeholder or prose-only
* criterion), or a command outside the allowlist
* criterion), a command outside the allowlist, or a runner that
* could not start (`startError`: not installed)
*/
export function runCriteria(criteria, opts = {}) {
const exec = opts.exec ?? defaultExec;
@ -545,6 +562,9 @@ export function runCriteria(criteria, opts = {}) {
return { ...base, status: 'blocked', exitCode: null, output: `blocked: ${verdict.rule}` };
}
const r = exec(c.command, { cwd, timeoutMs });
if (r.startError) {
return { ...base, status: 'unrunnable', exitCode: null, output: `not runnable: the program could not start (${r.startError})`, startError: r.startError };
}
const output = cap(`${r.stdout ?? ''}${r.stderr ?? ''}`, maxOutput);
return {
...base,
@ -766,8 +786,8 @@ export function formatCriteriaEvidence(rep) {
out.push(
'**NOT RUN is not a pass.** You cannot run these commands yourself — never infer, assume or ' +
'reconstruct a result for a criterion that has none. A criterion whose result is FAILED or ' +
'BLOCKED is `Broken`; a criterion marked NOT RUN declared no command, or a command outside ' +
'the allowlist of test runners, so it never reached a shell — that is the ABSENCE of a ' +
'BLOCKED is `Broken`; a criterion marked NOT RUN declared no command, a command outside ' +
'the allowlist of test runners, or a runner that could not start, so it never ran — that is the ABSENCE of a ' +
'measurement and never on its own a finding; a criterion with no result is judged on ' +
'delivered code alone, and you say so in the Evidence column.'
);