fix: round 5 of the release review of 6.3.0, the structural fix
Round 5 (v6.2.0..d37e538, WARN: 11 MAJOR, 2 MINOR). The patching did not converge (the same two places every round), so the operator chose a structural fix on 2026-10-11. Each change behind a test that failed first. - A check runs with no shell. criteria-runner's runCommand starts the runner with the very words the screen checked (spawnSync(words[0], words.slice(1)), without NODE_TEST_CONTEXT); Phase 7 criteria, Regression: (its exec copy is gone, MINOR 0e98d5c7), proof of red, the done line and the test lock all run through it. No quote, glob, tilde or variable can rewrite a checked word any more. The screen stays and says why: a word that means something else in a shell (quote, backslash, $, brace, leading ~) is refused; a glob reaches only node --test, which expands its own patterns (sh read `tests/.?/` as `tests/../`); no path climbs out with .. or starts at /; a word starting with @ is refused (pytest reads @file as more arguments). - The K6 plan-file sweep has v6.2.0's own Files: reader again (516dffa): backticked paths, and plain words only in a field without them. The shared reader from round 3 planned directories a description named and dropped paths v6.2.0 planned. fieldPaths stays for the tests-only check, with a path test on spans (`parse()`, `p.parse()` are no paths), plain descriptions ignored when the field has backticked paths, dotfiles, backslash paths and glued spans read. - Skill frontmatter reads a value as YAML does: trailing space is nothing, and a comment ends a plain value, a flow list or a block indicator (`model: opus # pinned`). A comment that cuts a plain description, or is the whole value, is still a hazard. - release-check reads the diff with --no-renames: a test moved into an allowed place after the review shows as removed. - The example plan's step 2 heading no longer says "must FAIL initially"; steps 1-2 keep the tests-only warning (operator 2026-10-09,73262b6). Open: MINOR c1fef1ad (one shared isMain helper) is not in this round. Measured: 0 of 463 Files: fields with backticked paths (520 fields in 182 plans across the marketplace repos) carry another path-like word. Suite 2185 (2183/0/2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
d37e538ddc
commit
c3641ffa42
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
17 changed files with 409 additions and 111 deletions
|
|
@ -61,6 +61,9 @@ is one test-runner invocation with no pipe, redirect or `&&`: `npm test`,
|
|||
`npm run <script>`, `node --test <file>`, `vitest`, `jest`, `pytest`,
|
||||
`python -m pytest`, `uv run pytest`, `cargo test`, `go test`, `make test`,
|
||||
`bash tests/<script>.sh` (or `/bin/bash tests/<script>.sh`), or `git status|log|diff|show|ls-files`.
|
||||
It runs with no shell, as plain words: no quote, `$`, brace, leading `~` or
|
||||
`@`, no glob outside `node --test`, no `..` or absolute path, and only the
|
||||
runner's read-only options.
|
||||
Anything else — the product's own binary (`node bin/x.mjs …`), `jq`, `grep`,
|
||||
`git grep`, a pipe — is reported NOT RUN. Never PROPOSE such a command as a
|
||||
criterion. When a criterion needs one, propose wrapping it in a script under
|
||||
|
|
|
|||
|
|
@ -790,7 +790,12 @@ Write the plan following the template structure. The plan MUST include:
|
|||
`go test`, `make test`), `bash <script under tests/>` (also
|
||||
`/bin/bash <script under tests/>`, to pin the system shell), or a read-only
|
||||
`git` subcommand (`status`, `log`, `diff`, `show`, `ls-files` — not `git
|
||||
grep`) — with no pipe, redirect or `&&`. Anything else is reported NOT
|
||||
grep`) — with no pipe, redirect or `&&`. It runs with no shell: its words
|
||||
reach the runner as written, so a word with a quote, backslash, `$`,
|
||||
brace, or a leading `~` or `@` is refused, a glob only reaches
|
||||
`node --test` (which expands its own patterns), no path climbs out with
|
||||
`..` or starts at `/`, each runner takes a short list of read-only options,
|
||||
and nothing may follow `make test`. Anything else is reported NOT
|
||||
RUN, never executed and never failed. A check that needs more than that is
|
||||
wrapped in a script under `tests/` and declared as `bash tests/<script>.sh`.
|
||||
9. **Estimated Scope** — File counts and complexity rating.
|
||||
|
|
|
|||
|
|
@ -245,7 +245,7 @@ if [ ! -f "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" ]; then
|
|||
exit 2
|
||||
fi
|
||||
|
||||
# Every command is screened twice before it reaches a shell: the runner's own
|
||||
# Every command is screened twice before it runs (with no shell): the runner's own
|
||||
# ALLOWLIST of test runners (npm test, node --test, pytest, bash tests/<script>,
|
||||
# a read-only git subcommand) reports anything else as NOT RUN, and the executor
|
||||
# denylist (catastrophe) reports BLOCKED. Neither is ever run. Foreground only.
|
||||
|
|
|
|||
|
|
@ -85,7 +85,7 @@ or stability harness comes before behavior change.
|
|||
must_contain: []
|
||||
```
|
||||
|
||||
### Step 2: Add stability test (must FAIL initially — verbose not yet wired)
|
||||
### Step 2: Add stability test (passes today — the canary for step 5 onwards)
|
||||
|
||||
- **Files:** `tests/cli-no-verbose-stability.test.mjs` (new file)
|
||||
- **Changes:** Three subtests, one per golden file. Each runs `node src/cli.mjs <cmd> ...` and asserts stdout `===` `readFileSync('tests/golden/<cmd>.stdout')`. The test should PASS today (no behavior change yet) — it's the canary for step 5 onwards.
|
||||
|
|
|
|||
|
|
@ -28,7 +28,7 @@
|
|||
},
|
||||
{
|
||||
"n": 2,
|
||||
"title": "Add stability test (must FAIL initially — verbose not yet wired)",
|
||||
"title": "Add stability test (passes today — the canary for step 5 onwards)",
|
||||
"status": "completed",
|
||||
"started_at": "2026-05-01T10:16:01Z",
|
||||
"completed_at": "2026-05-01T10:18:42Z",
|
||||
|
|
|
|||
|
|
@ -285,48 +285,65 @@ export function onFailureAction(text) {
|
|||
// ---- a step that holds nothing but tests -----------------------------------
|
||||
|
||||
/**
|
||||
* The paths a field names, in order. A backticked span is a path as written,
|
||||
* parentheses and all (`app/(auth)/x.sh`), also with punctuation after it
|
||||
* (`a.mjs`.) and inside a note ("(fixture: `t/a.json`)"). In the plain text a
|
||||
* parenthesised note with no span in it is not a path ("(new: covers x, y)"),
|
||||
* and every word that looks like a path is one: a name with an extension, a
|
||||
* path with a slash, a directory (`docs/`) or a glob (`src/*.mjs`). A word
|
||||
* with no slash and no extension counts only alone in its entry and
|
||||
* capitalised, like `Makefile`; "format" or "v6.3.0" in a description is no
|
||||
* path. Release review of 6.3.0, rounds 1-4.
|
||||
* The paths a step's `Files:` field names, in order, for the tests-only check
|
||||
* below (the K6 sweep keeps v6.2.0's own reader: plan-scope-sweep.mjs,
|
||||
* operator 2026-10-11). The convention in the plan template is a backticked
|
||||
* path, and in every plan measured (520 Files: fields in 182 plans, 11.10)
|
||||
* a field with backticked paths had no other path-like word beside them:
|
||||
*
|
||||
* - A field with a backticked path names its backticked spans that pass the
|
||||
* path test (a `.`, `/` or `\`; no URL, option or call like `parse()`),
|
||||
* wherever they sit (`a.mjs`/`b.mjs`, inside a note). Its descriptions are
|
||||
* not read: "covers pass/fail", "JSON, YAML" or a directory named in passing
|
||||
* is no path. An entry of its own with no backticks (", .gitignore") counts
|
||||
* by its first word, and only when that is clearly a file or a directory.
|
||||
* - A field with no backticked path names every path-like word: a name with an
|
||||
* extension, a dotfile, a path with a slash, a directory (`docs/`) or a glob
|
||||
* (`src/*.mjs`); a parenthesised note is not read. A word with no slash and
|
||||
* no extension counts only alone in its entry and capitalised (`Makefile`);
|
||||
* "format" or "v6.3.0" in a description is no path.
|
||||
*
|
||||
* A backslash is read as a slash, a leading `./` is dropped. Release review of
|
||||
* 6.3.0, rounds 1-5.
|
||||
*/
|
||||
const SEG = '[\\w@.*?()-]';
|
||||
const PATH_WORD = new RegExp(`^(?:${SEG}+\\/)+${SEG}*$|^[\\w@*?-][\\w@.*?-]*\\.[A-Za-z*][A-Za-z0-9*]*$`);
|
||||
const PATH_WORD = new RegExp(`^(?:${SEG}+\\/)+${SEG}*$|^[\\w@*?-][\\w@.*?-]*\\.[A-Za-z*][A-Za-z0-9*]*$|^\\.[\\w-][\\w.-]*$`);
|
||||
// Abbreviations that PATH_WORD would read as a name with an extension.
|
||||
const NOT_A_PATH = /^(?:e\.g|i\.e|n\.b|cf|etc|vs)$/i;
|
||||
const BARE_NAME = /^[A-Z][\w.-]*$/;
|
||||
const dotSlash = (p) => p.replace(/^\.\//, '');
|
||||
/**
|
||||
* The one reader of a step's `Files:` field: this file's tests-only check and
|
||||
* the plan-file sweep (lib/verification/plan-scope-sweep.mjs) both use it, so
|
||||
* the two never judge different lists for one step. A leading `./` is dropped.
|
||||
*/
|
||||
// The last segment is a dotfile or a name with an extension, or the path is a directory.
|
||||
const FILE_OR_DIR = /(?:^|\/)(?:\.[\w-][\w.-]*|[^/]*\.[A-Za-z*][A-Za-z0-9*]*)$|\/$/;
|
||||
const norm = (p) => p.replace(/\\/g, '/').replace(/^\.\//, '');
|
||||
// A backticked span that is a path, not a URL, an option or a call (`parse()`, `p.parse()`).
|
||||
const tickedPath = (s) => /[./\\]/.test(s) && !/^https?:/.test(s) && !s.startsWith('-') && !/^[\w$.]+\(.*\)$/.test(s);
|
||||
// A plain word without the brackets and punctuation around it, `\` read as `/`.
|
||||
const plainWord = (w) => w.replace(/^[(\[]+/, '').replace(/[.,;:)\]]+$/, '').replace(/\\/g, '/');
|
||||
const isPathWord = (w) => PATH_WORD.test(w) && !NOT_A_PATH.test(w);
|
||||
|
||||
export function fieldPaths(text) {
|
||||
const spans = [];
|
||||
const masked = String(text ?? '')
|
||||
.replace(/`([^`\n]+)`/g, (_, p) => `\u0000${spans.push(p.trim()) - 1}\u0000`)
|
||||
.replace(/(^|\s)\(([^()\n]*)\)/g, (m, pre, inner) => (inner.includes('\u0000') ? m : `${pre} `));
|
||||
const isTicked = spans.map(tickedPath);
|
||||
const tickedField = isTicked.some(Boolean);
|
||||
const out = [];
|
||||
for (const raw of masked.split(/[,\n]/)) {
|
||||
const entry = raw.trim().replace(/^[-*]\s+/, '');
|
||||
if (entry === '') continue;
|
||||
const words = entry.split(/\s+/);
|
||||
const found = [];
|
||||
for (const word of words) {
|
||||
const w = word.replace(/^[(\[]+/, '').replace(/[.,;:)\]]+$/, '');
|
||||
const span = /^\u0000(\d+)\u0000$/.exec(w);
|
||||
if (span) found.push(spans[Number(span[1])]);
|
||||
else if (PATH_WORD.test(w) && !NOT_A_PATH.test(w)) found.push(w);
|
||||
const ticked = [...entry.matchAll(/\u0000(\d+)\u0000/g)].map((m) => Number(m[1])).filter((i) => isTicked[i]);
|
||||
if (ticked.length) { out.push(...ticked.map((i) => spans[i])); continue; }
|
||||
const words = entry.split(/\s+/).filter((w) => !w.includes('\u0000')).map(plainWord).filter(Boolean);
|
||||
if (tickedField) {
|
||||
const first = plainWord(entry.split(/\s+/)[0]);
|
||||
if (!first.includes('\u0000') && isPathWord(first) && FILE_OR_DIR.test(first)) out.push(first);
|
||||
continue;
|
||||
}
|
||||
if (found.length === 0 && words.length === 1 && BARE_NAME.test(words[0])) found.push(words[0]);
|
||||
const found = words.filter(isPathWord);
|
||||
if (found.length === 0 && words.length === 1 && entry.split(/\s+/).length === 1 && BARE_NAME.test(words[0])) found.push(words[0]);
|
||||
out.push(...found);
|
||||
}
|
||||
return out.filter((p) => p && !p.includes('\u0000')).map(dotSlash);
|
||||
return out.filter(Boolean).map(norm);
|
||||
}
|
||||
|
||||
// A document or a data file outside tests/ is never the test here:
|
||||
|
|
@ -358,7 +375,7 @@ export function stepTestsOnly(body) {
|
|||
export function stepTestsOnlyFromFields(f) {
|
||||
const files = f.has('Files') ? fieldPaths(f.get('Files').text) : [];
|
||||
const testFirst = f.has('Test first')
|
||||
? [...f.get('Test first').text.matchAll(/\bFile:\**[ \t]*`([^`\n]+)`/gi)].map((m) => dotSlash(m[1].trim()))
|
||||
? [...f.get('Test first').text.matchAll(/\bFile:\**[ \t]*`([^`\n]+)`/gi)].map((m) => norm(m[1].trim()))
|
||||
: [];
|
||||
const isTest = (p) => testFirst.includes(p) ||
|
||||
(isTestPath(p) && (UNDER_TESTS_RE.test(p) || !(DOC_RE.test(p) || DATA_RE.test(p))));
|
||||
|
|
|
|||
|
|
@ -10,6 +10,14 @@
|
|||
// real one. Claude Code loads a skill whose YAML does not parse "with no
|
||||
// fields set", so the description is gone and the skill cannot match; a trap
|
||||
// here is an error, not a style note.
|
||||
//
|
||||
// A value is read as YAML reads it: trailing space on the line is nothing, and
|
||||
// a comment (` #`, or `#` at the start) ends a plain value, a flow list or a
|
||||
// block indicator (`model: opus # pinned` is `opus`). Two comments are still
|
||||
// traps: one that is the whole value (the value is empty), and one that cuts a
|
||||
// plain `description`, the text Claude Code routes on (`Use when issue #42
|
||||
// …` would route on "Use when issue"); quote the description if "#" belongs to
|
||||
// it. Release review of 6.3.0, rounds 3-5.
|
||||
|
||||
const KEY_RE = /^([A-Za-z0-9_-]+):(.*)$/;
|
||||
const BLOCK_RE = /^([|>])([+-]?)\s*$/;
|
||||
|
|
@ -27,7 +35,7 @@ export function splitFile(text) {
|
|||
return { ok: true, fmLines: lines.slice(1, end), body: lines.slice(end + 1).join('\n'), bodyStartLine: end + 2 };
|
||||
}
|
||||
|
||||
/** The YAML traps in one plain (unquoted, single-line) scalar value. */
|
||||
/** The YAML traps in one plain (unquoted, single-line) scalar value, meant whole. */
|
||||
export function plainScalarHazards(value) {
|
||||
const out = [];
|
||||
if (/:\s/.test(value) || /:$/.test(value)) out.push('unquoted ": " (YAML reads a mapping and the line fails)');
|
||||
|
|
@ -39,6 +47,24 @@ export function plainScalarHazards(value) {
|
|||
return out;
|
||||
}
|
||||
|
||||
// Where a YAML comment starts in a value: `#` at the start or after a space or
|
||||
// tab, outside a quoted item (a quote opens only at the start of a token, so
|
||||
// the apostrophe in `don't` is text). -1 when there is none.
|
||||
function commentAt(v) {
|
||||
let q = null;
|
||||
for (let i = 0; i < v.length; i++) {
|
||||
const c = v[i];
|
||||
if (q) {
|
||||
if (q === '"' && c === '\\') i++;
|
||||
else if (c === q) q = null;
|
||||
continue;
|
||||
}
|
||||
if ((c === '"' || c === "'") && (i === 0 || /[\s,[]/.test(v[i - 1]))) { q = c; continue; }
|
||||
if (c === '#' && (i === 0 || v[i - 1] === ' ' || v[i - 1] === '\t')) return i;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
// A quoted value ends at its closing quote; after it only a YAML comment may
|
||||
// follow (`name: "x" # pinned`). Inside double quotes a backslash escapes the
|
||||
// next character; inside single quotes '' is one quote.
|
||||
|
|
@ -101,7 +127,7 @@ export function readFrontmatterFields(text) {
|
|||
const m = KEY_RE.exec(line);
|
||||
if (!m) { hazards.push({ line: lineNo, message: `not a "key: value" line: ${line.slice(0, 60)}` }); i++; continue; }
|
||||
const key = m[1];
|
||||
const rest = m[2].replace(/^[ \t]+/, '');
|
||||
const rest = m[2].trim();
|
||||
if (m[2] !== '' && !/^[ \t]/.test(m[2])) hazards.push({ line: lineNo, message: `"${key}:" needs a space after the colon` });
|
||||
if (key in fields) hazards.push({ line: lineNo, message: `duplicate key "${key}"` });
|
||||
|
||||
|
|
@ -111,16 +137,23 @@ export function readFrontmatterFields(text) {
|
|||
while (j < fmLines.length && (fmLines[j].trim() === '' || /^\s/.test(fmLines[j]))) { cont.push(fmLines[j]); j++; }
|
||||
while (cont.length && cont[cont.length - 1].trim() === '') { cont.pop(); }
|
||||
|
||||
const block = BLOCK_RE.exec(rest);
|
||||
// A quoted value keeps its own comment rule (unquote); any other value
|
||||
// ends where a comment starts.
|
||||
const quoted = rest[0] === '"' || rest[0] === "'";
|
||||
const cut = quoted ? -1 : commentAt(rest);
|
||||
const value = cut < 0 ? rest : rest.slice(0, cut).trimEnd();
|
||||
|
||||
const block = BLOCK_RE.exec(value);
|
||||
if (block) {
|
||||
fields[key] = { value: blockValue(cont, block[1], block[2]), line: lineNo, style: 'block' };
|
||||
} else if (rest === '') {
|
||||
} else if (value === '') {
|
||||
const items = cont.filter((l) => l.trim() !== '');
|
||||
if (items.length && items.every((l) => /^\s+-\s/.test(l))) {
|
||||
fields[key] = { value: items.map((l) => l.replace(/^\s+-\s+/, '').trim()), line: lineNo, style: 'list' };
|
||||
} else {
|
||||
fields[key] = { value: items.join('\n'), line: lineNo, style: items.length ? 'map' : 'empty' };
|
||||
}
|
||||
if (cut === 0 && !items.length) hazards.push({ line: lineNo, message: `${key}: "#" at the start is a YAML comment: the value is empty` });
|
||||
} else {
|
||||
if (cont.length) hazards.push({ line: lineNo, message: `"${key}" has a value and indented lines under it` });
|
||||
const q = unquote(rest);
|
||||
|
|
@ -129,15 +162,18 @@ export function readFrontmatterFields(text) {
|
|||
fields[key] = { value: rest, line: lineNo, style: 'broken' };
|
||||
} else if (q) {
|
||||
fields[key] = { value: q.value, line: lineNo, style: 'quoted' };
|
||||
} else if (/^\[.*\]$/.test(rest)) {
|
||||
const inner = rest.slice(1, -1).trim();
|
||||
} else if (/^\[.*\]$/.test(value)) {
|
||||
const inner = value.slice(1, -1).trim();
|
||||
fields[key] = {
|
||||
value: inner === '' ? [] : inner.split(',').map((s) => { const t = s.trim(); const u = unquote(t); return u && !u.error ? u.value : t; }),
|
||||
line: lineNo, style: 'flow',
|
||||
};
|
||||
} else {
|
||||
for (const h of plainScalarHazards(rest)) hazards.push({ line: lineNo, message: `${key}: ${h}` });
|
||||
fields[key] = { value: rest, line: lineNo, style: 'plain' };
|
||||
for (const h of plainScalarHazards(value)) hazards.push({ line: lineNo, message: `${key}: ${h}` });
|
||||
if (cut > 0 && key === 'description') {
|
||||
hazards.push({ line: lineNo, message: `${key}: " #" starts a YAML comment: the description is cut to "${value}"; quote it if "#" belongs to it` });
|
||||
}
|
||||
fields[key] = { value, line: lineNo, style: 'plain' };
|
||||
}
|
||||
}
|
||||
i = j;
|
||||
|
|
|
|||
|
|
@ -42,10 +42,9 @@ import { join, resolve, dirname } from 'node:path';
|
|||
import { fileURLToPath } from 'node:url';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { resolveRepo, canonicalPath } from '../verification/repo-tools.mjs';
|
||||
import { allowedCommand, screenCommand } from '../verification/criteria-runner.mjs';
|
||||
import { commandEnv, DEFAULT_TIMEOUT_MS } from '../verification/red-proof.mjs';
|
||||
import { allowedCommand, runCommand, screenCommand } from '../verification/criteria-runner.mjs';
|
||||
import { DEFAULT_TIMEOUT_MS } from '../verification/red-proof.mjs';
|
||||
|
||||
export const LOCK_VERSION = 1;
|
||||
|
||||
|
|
@ -179,9 +178,8 @@ function main(argv) {
|
|||
if (!allowed.allowed) fail(2, `command not run: ${allowed.reason}`);
|
||||
const screen = screenCommand(testCmd);
|
||||
if (!screen.allowed) fail(2, `command blocked: ${screen.rule}`);
|
||||
const r = spawnSync('/bin/sh', ['-c', testCmd], {
|
||||
cwd: cur.repo.root, env: commandEnv(), encoding: 'utf8', timeout: DEFAULT_TIMEOUT_MS, maxBuffer: 64 * 1024 * 1024,
|
||||
});
|
||||
// No shell: the words the screen checked are the words that run.
|
||||
const r = runCommand(testCmd, { cwd: cur.repo.root, timeoutMs: DEFAULT_TIMEOUT_MS });
|
||||
if (r.error || r.status === null || r.status === 126 || r.status === 127) {
|
||||
fail(2, `the test command did not run to an exit code (${r.error?.message ?? r.signal ?? `exit ${r.status}`})`);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -21,7 +21,17 @@
|
|||
// tests/<script>.sh`, a read-only git subcommand, …) — anything else is NOT
|
||||
// RUN, never a failure — and what survives is then screened through the
|
||||
// plugin's own PreToolUse denylist (hooks/scripts/pre-bash-executor.mjs) —
|
||||
// catastrophe — before it reaches a shell.
|
||||
// catastrophe — before it runs.
|
||||
//
|
||||
// No shell. A command runs as its words (`runCommand`): the runner is started
|
||||
// directly with the words the screen checked, so no quoting, glob, tilde,
|
||||
// brace or variable can turn them into other words after the check. Five
|
||||
// review rounds of 6.3.0 found one more way /bin/sh rewrote a checked word
|
||||
// (quotes, tilde, a glob that reached `..`); with no shell there is nothing
|
||||
// left to rewrite (operator 2026-10-11). The same function runs every check
|
||||
// Voyage runs: Phase 7 criteria, `Regression:` (regression-check.mjs), proof
|
||||
// of red (red-proof.mjs), the done line (done-line.mjs), the test lock
|
||||
// (lib/util/test-lock.mjs).
|
||||
// Commands spawned from here do not pass through the Bash tool, so that hook
|
||||
// would not otherwise fire; the hook is invoked over its documented stdin
|
||||
// protocol rather than copied, so there is exactly one denylist.
|
||||
|
|
@ -257,13 +267,21 @@ export function screenCommand(command, opts = {}) {
|
|||
// - It is not a sandbox. `npm test`, `npm run <script>` and `make test` run
|
||||
// whatever the repo's own package.json/Makefile says they run, including a
|
||||
// script that pushes. That is the repo's responsibility, not this file's.
|
||||
// - It is not an option parser. Each runner takes only plain words (no
|
||||
// quote, backslash, `$` or brace; a glob only inside a path) and a short
|
||||
// list of options that only read (RUNNER_ARGS); an option that writes a
|
||||
// file or loads code (`--test-reporter-destination`, `--outputFile`,
|
||||
// `--junitxml`, `--basetemp`, `--import`, `--config`, `-o`, `-exec`) is on
|
||||
// no list (release review of 6.3.0, round 3: rounds 1-2 closed this for
|
||||
// git only).
|
||||
// - It is not an option parser. Each runner takes only plain words and a
|
||||
// short list of options that only read (RUNNER_ARGS); an option that
|
||||
// writes a file or loads code (`--test-reporter-destination`,
|
||||
// `--outputFile`, `--junitxml`, `--basetemp`, `--import`, `--config`,
|
||||
// `-o`, `-exec`) is on no list (release review of 6.3.0, round 3: rounds
|
||||
// 1-2 closed this for git only).
|
||||
// - Plain words: the words reach the runner as written, so a word that
|
||||
// would mean something else in a shell — a quote, backslash, `$`, brace,
|
||||
// a leading `~` — is refused rather than passed on with a meaning its
|
||||
// author did not intend. A glob is passed on only to `node --test`, which
|
||||
// expands its own patterns (Node 21 and later); no other runner here
|
||||
// expands one, so elsewhere it is refused. A path that climbs out with
|
||||
// `..` or starts at `/` is refused: a check runs in the repo. A word that
|
||||
// starts with `@` is refused: pytest (8.2 and later) reads `@file` as a
|
||||
// file of more arguments, options the list would refuse.
|
||||
// - It rejects plenty of honest commands: an env prefix (`CI=1 npm test`), a
|
||||
// project's own binary (`./build/app --check`), anything piped. A check
|
||||
// that needs one of those is declared through a runner instead —
|
||||
|
|
@ -274,7 +292,8 @@ const SHELL_OPERATORS = /[;&|<>]|\$\(|`/;
|
|||
|
||||
// Read-only git subcommands: they report, they do not write a repository.
|
||||
const READONLY_GIT = new Set(['status', 'log', 'diff', 'show', 'ls-files']);
|
||||
// A word /bin/sh hands to git unchanged (refs like HEAD~1 and HEAD^, formats like %h, paths).
|
||||
// A word that means the same to git with or without a shell (refs like HEAD~1
|
||||
// and HEAD^, formats like %h, paths).
|
||||
const GIT_LITERAL_WORD = /^[A-Za-z0-9_@%^~:.,=+/-]+$/;
|
||||
// `--output`, `--output=<file>`, or an abbreviation of it git may accept.
|
||||
function writesOutput(word) {
|
||||
|
|
@ -285,10 +304,14 @@ function writesOutput(word) {
|
|||
const deny = (reason) => ({ allowed: false, reason });
|
||||
const permit = () => ({ allowed: true, reason: '' });
|
||||
|
||||
// A word /bin/sh hands on unchanged, or a path with a glob in it: a glob can
|
||||
// only expand to names that start with the word's first character, so it
|
||||
// cannot make an option out of a path.
|
||||
const PLAIN_WORD = /^(?:[A-Za-z0-9_@%^~:.,=+/-]+|[A-Za-z0-9_./][A-Za-z0-9_@%^~:.,=+/*?-]*)$/;
|
||||
// A word that means the same with or without a shell.
|
||||
const PLAIN_WORD = /^[A-Za-z0-9_@%^~:.,=+/-]+$/;
|
||||
// The same, with a glob in it: passed on only to `node --test`, which expands
|
||||
// its own patterns (sh would have expanded `.?` to `..`).
|
||||
const GLOB_WORD = /^[A-Za-z0-9_./][A-Za-z0-9_@%^~:.,=+/*?-]*$/;
|
||||
const takesGlobs = (t) => t[0] === 'node' && t[1] === '--test';
|
||||
// A path outside the repo: one that climbs with `..` or starts at `/`.
|
||||
const leavesRepo = (w) => w.startsWith('/') || w.split('/').includes('..');
|
||||
|
||||
// After each runner's fixed words: the options it may take (`name` or
|
||||
// `name=value`; `value` lists the ones whose value may be the next word, and
|
||||
|
|
@ -386,14 +409,14 @@ const RUNNERS = {
|
|||
'/bin/bash': (t) => (isTestScript(t[1]) ? permit() : deny('/bin/bash, but not a script under tests/')),
|
||||
git: (t) => {
|
||||
if (!READONLY_GIT.has(t[1])) return deny(`git, but not a read-only subcommand first (${[...READONLY_GIT].join('|')})`);
|
||||
// The command runs through /bin/sh, which unquotes and expands what it is
|
||||
// given, so a check on the words here holds only for words the shell passes
|
||||
// through unchanged: no quote, backslash, `$`, glob or brace. Then
|
||||
// `--output` (and an abbreviation of it) is refused: it makes diff/log/show
|
||||
// write to a file (release review of 6.3.0, 2026-10-10: a plan's check
|
||||
// could overwrite any file, first plainly, then through quoting).
|
||||
// The words reach git as written (no shell), and only words that mean the
|
||||
// same with or without one run: no quote, backslash, `$`, glob or brace.
|
||||
// Then `--output` (and an abbreviation of it) is refused: it makes
|
||||
// diff/log/show write to a file (release review of 6.3.0, 2026-10-10: a
|
||||
// plan's check could overwrite any file, first plainly, then through
|
||||
// quoting while a shell still ran it).
|
||||
const odd = t.find((w) => !GIT_LITERAL_WORD.test(w));
|
||||
if (odd !== undefined) return deny(`git with a word the shell would rewrite (${odd}): only plain words are checked`);
|
||||
if (odd !== undefined) return deny(`git with a word that means something else in a shell (${odd}): only plain words run, as written`);
|
||||
if (t.some(writesOutput)) return deny('git with --output writes a file, so it is not a read-only check');
|
||||
return permit();
|
||||
},
|
||||
|
|
@ -431,27 +454,54 @@ function screenShape(command, opts, writeTime) {
|
|||
const operator = text.match(SHELL_OPERATORS);
|
||||
if (operator) return deny(`a shell operator (${operator[0]}) — one criterion declares one command, unpiped`);
|
||||
|
||||
const tokens = text.trim().split(/\s+/);
|
||||
const tokens = commandWords(text);
|
||||
if (!Object.hasOwn(RUNNERS, tokens[0])) return deny(`\`${tokens[0]}\` is not a known test runner`);
|
||||
// /bin/sh expands a tilde at the start of a word (and after = or : in an
|
||||
// assignment), so such a word is not one the shell passes on unchanged.
|
||||
// A shell expands a tilde at the start of a word (and after = or : in an
|
||||
// assignment); no shell runs this, so the runner would get the ~ as written.
|
||||
const tilde = tokens.find((w) => /(^|[=:])~/.test(w));
|
||||
if (tilde !== undefined) return deny(`a word the shell would rewrite (${tilde}): a leading ~ is expanded`);
|
||||
if (tilde !== undefined) return deny(`a word that means something else in a shell (${tilde}): a leading ~ is not expanded here`);
|
||||
const form = RUNNERS[tokens[0]](tokens, cwd, writeTime);
|
||||
if (!form.allowed || tokens[0] === 'git') return form;
|
||||
const odd = tokens.find((w) => !PLAIN_WORD.test(w));
|
||||
if (odd !== undefined) return deny(`a word the shell would rewrite (${odd}): only plain words are checked`);
|
||||
const words = tokens.slice(1);
|
||||
const odd = words.find((w) => !PLAIN_WORD.test(w) && !(takesGlobs(tokens) && GLOB_WORD.test(w)));
|
||||
if (odd !== undefined) {
|
||||
return deny(GLOB_WORD.test(odd)
|
||||
? `a glob (${odd}): no shell runs this, and \`${tokens[0]}\` does not expand one itself; name the files, or use \`node --test\``
|
||||
: `a word that means something else in a shell (${odd}): only plain words run, as written`);
|
||||
}
|
||||
const at = words.find((w) => w.startsWith('@'));
|
||||
if (at !== undefined) return deny(`a word that starts with @ (${at}): pytest reads @file as a file of more arguments`);
|
||||
const out = words.find((w) => !w.startsWith('-') && leavesRepo(w));
|
||||
if (out !== undefined) return deny(`a path outside the repo (${out}): a check runs in the repo, with no .. and no absolute path`);
|
||||
return Object.hasOwn(RUNNER_ARGS, tokens[0]) ? screenArgs(tokens, RUNNER_ARGS[tokens[0]]) : permit();
|
||||
}
|
||||
|
||||
/** The words of a command: what the screen checks is exactly what runs. */
|
||||
export function commandWords(command) {
|
||||
return String(command ?? '').trim().split(/\s+/);
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a screened command with no shell: its first word is the program (found
|
||||
* on PATH), the rest are its arguments, as written. Without NODE_TEST_CONTEXT
|
||||
* (lib/util/command-env.mjs): started under `node --test`, a nested `node
|
||||
* --test` reported to the parent and exited 0 on a red test, a false green
|
||||
* (order 10, 2026-10-09). Returns spawnSync's result: `status` is null when
|
||||
* the program could not start (`error`) or was killed (`signal`).
|
||||
*/
|
||||
export function runCommand(command, { cwd, timeoutMs = DEFAULT_TIMEOUT_MS } = {}) {
|
||||
const [program, ...args] = commandWords(command);
|
||||
return spawnSync(program, args, { cwd, env: commandEnv(), encoding: 'utf8', timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 });
|
||||
}
|
||||
|
||||
// --- running ----------------------------------------------------------------
|
||||
|
||||
// Without NODE_TEST_CONTEXT (lib/util/command-env.mjs): started under
|
||||
// `node --test`, a nested `node --test` criterion reported to the parent and
|
||||
// exited 0 on a red test - a false green (order 10, 2026-10-09).
|
||||
// A program that could not start (not installed: ENOENT) or was killed by the
|
||||
// timeout has no exit code; its error is the output, so the row says why.
|
||||
function defaultExec(command, { cwd, timeoutMs }) {
|
||||
const r = spawnSync('/bin/sh', ['-c', command], { cwd, encoding: 'utf8', timeout: timeoutMs, env: commandEnv() });
|
||||
return { status: r.status, stdout: r.stdout ?? '', stderr: r.stderr ?? '' };
|
||||
const r = runCommand(command, { cwd, timeoutMs });
|
||||
const why = r.error ? `${r.error.message}\n` : r.signal ? `killed (${r.signal})\n` : '';
|
||||
return { status: r.status, stdout: r.stdout ?? '', stderr: `${r.stderr ?? ''}${why}` };
|
||||
}
|
||||
|
||||
function cap(text, max) {
|
||||
|
|
|
|||
|
|
@ -23,8 +23,8 @@ import { appendFileSync, existsSync, readFileSync, realpathSync } from 'node:fs'
|
|||
import { spawnSync } from 'node:child_process';
|
||||
import { resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { allowedCommand, screenCommand } from './criteria-runner.mjs';
|
||||
import { commandEnv, DEFAULT_TIMEOUT_MS, repoRoot } from './red-proof.mjs';
|
||||
import { allowedCommand, runCommand, screenCommand } from './criteria-runner.mjs';
|
||||
import { DEFAULT_TIMEOUT_MS, repoRoot } from './red-proof.mjs';
|
||||
|
||||
export function doneLineRe(step, commit) {
|
||||
const sha = commit ? String(commit).replace(/[^0-9a-f]/g, '') : '[0-9a-f]{40}';
|
||||
|
|
@ -55,7 +55,8 @@ export function writeDoneLine({ cwd, step, cmd, report, timeoutMs = DEFAULT_TIME
|
|||
if (!screen.allowed) return { status: 'not-run', reason: `command blocked: ${screen.rule}` };
|
||||
const head = spawnSync('git', ['rev-parse', '--verify', '--quiet', 'HEAD'], { cwd: dir, encoding: 'utf8' });
|
||||
if (head.status !== 0) return { status: 'not-run', reason: `no git HEAD in ${dir}` };
|
||||
const r = spawnSync('/bin/sh', ['-c', cmd], { cwd: dir, env: commandEnv(), encoding: 'utf8', timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 });
|
||||
// No shell: the words the screen checked are the words that run.
|
||||
const r = runCommand(cmd, { cwd: dir, timeoutMs });
|
||||
if (r.error || r.status === null || r.status === 126 || r.status === 127) {
|
||||
return { status: 'not-run', reason: `the command did not run to an exit code (${r.error?.message ?? r.signal ?? `exit ${r.status}`})`, exit: r.status };
|
||||
}
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ import { readFileSync, existsSync, realpathSync } from 'node:fs';
|
|||
import { spawnSync } from 'node:child_process';
|
||||
import { resolve, relative, dirname, isAbsolute, sep } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { sliceSteps, parseStepFields, fieldPaths } from '../parsers/plan-schema.mjs';
|
||||
import { sliceSteps, parseStepFields } from '../parsers/plan-schema.mjs';
|
||||
import { parseManifest } from '../parsers/manifest-yaml.mjs';
|
||||
|
||||
// Files Voyage's own process writes while planning and executing.
|
||||
|
|
@ -44,8 +44,9 @@ export const PROCESS_FILES = Object.freeze([
|
|||
]);
|
||||
|
||||
const BACKTICK_SPAN = /`([^`\n]+)`/g;
|
||||
// A backticked path in `Test first:` (spaces allowed: `docs/my file.md`).
|
||||
// Backticked: spaces allowed (`docs/my file.md`); bare: no whitespace.
|
||||
const looksLikeTickedPath = s => /[./]/.test(s) && !/^https?:/.test(s) && !s.startsWith('-') && s.trim() === s;
|
||||
const looksLikePath = s => !/\s/.test(s) && looksLikeTickedPath(s);
|
||||
|
||||
function pathsIn(text) {
|
||||
const out = [];
|
||||
|
|
@ -61,9 +62,20 @@ export function plannedPaths(planText) {
|
|||
for (const s of sliceSteps(planText)) {
|
||||
const f = parseStepFields(s.body);
|
||||
const files = f.get('Files');
|
||||
// One reader with the tests-only check (plan-schema.mjs fieldPaths):
|
||||
// backticked spans as written, else every path-like word.
|
||||
if (files) for (const p of fieldPaths(files.text)) out.add(norm(p));
|
||||
// v6.2.0's own reading, kept as it was (operator 2026-10-11): every
|
||||
// backticked span that looks like a path, and plain words only when the
|
||||
// field has none. A field that names its paths in backticks plans those,
|
||||
// never a path its description mentions. Release review of 6.3.0, rounds
|
||||
// 3-5: reading Files: through the tests-only check's reader
|
||||
// (plan-schema.mjs fieldPaths) dropped paths v6.2.0 planned and planned
|
||||
// directories a description named, so K6 has its own reader again; the
|
||||
// two answer different questions (what may change; is every named file a test).
|
||||
if (files) {
|
||||
const ticked = pathsIn(files.text);
|
||||
// Older plans write bare paths: `- Files: a.ts, b.ts`.
|
||||
const bare = ticked.length ? [] : files.text.split(/[,\s]+/).filter(looksLikePath);
|
||||
for (const p of [...ticked, ...bare]) out.add(norm(p));
|
||||
}
|
||||
const tf = f.get('Test first');
|
||||
if (tf) {
|
||||
for (const line of tf.text.split('\n')) {
|
||||
|
|
|
|||
|
|
@ -42,6 +42,7 @@
|
|||
// (criteria-runner.mjs `allowedCommand`), and it is screened through the
|
||||
// plugin's own PreToolUse denylist (`screenCommand`): a command spawned from
|
||||
// node never passes through the Bash tool, so the hook cannot fire by itself.
|
||||
// It runs with no shell (`runCommand`): the words checked are the words run.
|
||||
//
|
||||
// When the output is TAP (`--test-reporter=tap`, pytest-tap, ...), the counts
|
||||
// `# tests N` / `# fail N` are reported per run. They show WHAT the red was:
|
||||
|
|
@ -58,7 +59,7 @@ import { spawnSync } from 'node:child_process';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { dirname, isAbsolute, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { allowedCommand, screenCommand } from './criteria-runner.mjs';
|
||||
import { allowedCommand, runCommand, screenCommand } from './criteria-runner.mjs';
|
||||
import { commandEnv } from '../util/command-env.mjs';
|
||||
import { sliceSteps, stepTestsOnly } from '../parsers/plan-schema.mjs';
|
||||
|
||||
|
|
@ -90,8 +91,9 @@ export function repoRoot(dir) {
|
|||
// failing test (independent review, 2026-10-08).
|
||||
const LOAD_ERROR = /^\s*SyntaxError: |ERR_MODULE_NOT_FOUND|does not provide an export named|Cannot find module|ERR_REQUIRE_ESM|ModuleNotFoundError|ImportError: /m;
|
||||
|
||||
// No shell: the words the screen checked are the words that run.
|
||||
function run(cmd, cwd, timeoutMs) {
|
||||
const r = spawnSync('/bin/sh', ['-c', cmd], { cwd, env: commandEnv(), encoding: 'utf8', timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 });
|
||||
const r = runCommand(cmd, { cwd, timeoutMs });
|
||||
const output = `${r.stdout ?? ''}${r.stderr ?? ''}`;
|
||||
return { exit: r.status, signal: r.signal ?? null, tap: tap(output), tail: output.slice(-TAIL), spawnError: r.error ? r.error.message : null, loadError: LOAD_ERROR.test(output) };
|
||||
}
|
||||
|
|
@ -118,12 +120,13 @@ function testsOnly(o) {
|
|||
return { tests_only: stepTestsOnly(step.body).testsOnly };
|
||||
}
|
||||
|
||||
// A shell that could not find or execute the command (126/127), a signal (the
|
||||
// timeout), or a spawn error is not a test verdict at all.
|
||||
// A program that could not start (a spawn error: not installed), a signal (the
|
||||
// timeout), or 126/127 from a script that could not run its own command is not
|
||||
// a test verdict at all.
|
||||
function unrunnable(r) {
|
||||
if (r.spawnError) return `the command could not start: ${r.spawnError}`;
|
||||
if (r.exit === null) return `the command was killed (${r.signal ?? 'timeout'})`;
|
||||
if (r.exit === 126 || r.exit === 127) return `the shell could not run the command (exit ${r.exit})`;
|
||||
if (r.exit === 126 || r.exit === 127) return `the command could not be run (exit ${r.exit})`;
|
||||
return null;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -26,24 +26,17 @@
|
|||
// The runner reports `exitCode`; this CLI writes `exit_code`, the key
|
||||
// progress.json uses (C15). Nothing else is translated.
|
||||
//
|
||||
// The command runs the way the runner runs it (`/bin/sh -c`), but without
|
||||
// NODE_TEST_CONTEXT in its environment (lib/util/command-env.mjs): measured
|
||||
// 2026-10-08, a red `node --test` regression read as passed when this CLI was
|
||||
// itself started under `node --test`, because the nested run reported to the
|
||||
// parent and exited 0.
|
||||
// The command runs exactly as the runner runs a criterion: its own run, with
|
||||
// no shell and without NODE_TEST_CONTEXT (criteria-runner.mjs `runCommand`).
|
||||
// This file had a copy of that code, and a fix made in one copy would have
|
||||
// left K10 regressions running differently from Phase 7 criteria (release
|
||||
// review of 6.3.0, round 5).
|
||||
|
||||
import { readFileSync, realpathSync } from 'node:fs';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { sliceSteps, parseStepFields } from '../parsers/plan-schema.mjs';
|
||||
import { firstCommand, runCriteria } from './criteria-runner.mjs';
|
||||
import { commandEnv } from '../util/command-env.mjs';
|
||||
|
||||
function exec(command, { cwd, timeoutMs }) {
|
||||
const r = spawnSync('/bin/sh', ['-c', command], { cwd, encoding: 'utf8', timeout: timeoutMs, env: commandEnv() });
|
||||
return { status: r.status, stdout: r.stdout ?? '', stderr: r.stderr ?? '' };
|
||||
}
|
||||
|
||||
export const REGRESSION_FIELD = 'Regression';
|
||||
const EXIT = Object.freeze({ passed: 0, absent: 0, failed: 1, blocked: 1, unrunnable: 1, error: 2 });
|
||||
|
|
@ -54,7 +47,7 @@ const withProgress = (r) => ({ ...r, progress_status: PROGRESS[r.status] ?? 'not
|
|||
* @param {{ planText: string, step: number, cwd?: string, run?: Function }} o
|
||||
* @returns {{ step: number, status: string, command: string|null, exit_code: number|null, output: string, exit: 0|1|2 }}
|
||||
*/
|
||||
export function checkRegression({ planText, step, cwd = process.cwd(), run = (c, o) => runCriteria(c, { ...o, exec }) }) {
|
||||
export function checkRegression({ planText, step, cwd = process.cwd(), run = (c, o) => runCriteria(c, o) }) {
|
||||
const base = { step, command: null, exit_code: null, output: '' };
|
||||
const found = sliceSteps(planText).find((s) => Number(s.n) === step);
|
||||
if (!found) return withProgress({ ...base, status: 'error', output: `no "### Step ${step}:" in the plan`, exit: EXIT.error });
|
||||
|
|
|
|||
|
|
@ -206,7 +206,9 @@ function checkCoverage(repo, reviewText) {
|
|||
|
||||
// --relative: the paths, and the files that count, are those under the plugin's
|
||||
// own directory, also when it sits in a subdirectory of a larger repo.
|
||||
const names = git(repo, ['diff', '--relative', '--name-only', full, 'HEAD']);
|
||||
// --no-renames: a file moved into an allowed place after the review shows
|
||||
// both its old path and its new one, so its removal is seen (round 5).
|
||||
const names = git(repo, ['diff', '--relative', '--no-renames', '--name-only', full, 'HEAD']);
|
||||
if (names.status !== 0) return { id: 'coverage', status: 'error', detail: `git diff failed: ${names.stderr.trim()}` };
|
||||
const changed = names.stdout.split('\n').filter(Boolean);
|
||||
const outside = changed.filter((f) => !RELEASE_BOOKKEEPING.includes(f) && !PROCESS_PREFIXES.some((p) => f.startsWith(p)));
|
||||
|
|
@ -221,7 +223,7 @@ function checkCoverage(repo, reviewText) {
|
|||
if (!same) outside.push(`${file} (lines other than the version)`);
|
||||
}
|
||||
if (changed.includes('README.md')) {
|
||||
const d = git(repo, ['diff', '--relative', '-U0', full, 'HEAD', '--', 'README.md']);
|
||||
const d = git(repo, ['diff', '--relative', '--no-renames', '-U0', full, 'HEAD', '--', 'README.md']);
|
||||
if (d.status !== 0) return { id: 'coverage', status: 'error', detail: `git diff of README.md failed: ${d.stderr.trim()} — not measured` };
|
||||
if (!onlyBadgeVersion(changedLines(d.stdout))) outside.push('README.md (lines other than the version badge)');
|
||||
}
|
||||
|
|
|
|||
|
|
@ -113,7 +113,8 @@ verifiable by running a specific command or observing a specific system behavior
|
|||
- {criterion — e.g., "No TypeScript errors: `npm run typecheck` exits 0"}
|
||||
|
||||
One command per criterion, from the runner's allowlist (`npm test`, `npm run <script>`,
|
||||
`node --test`, `pytest`, `bash tests/<script>.sh`, …), with no pipe or `&&`.
|
||||
`node --test`, `pytest`, `bash tests/<script>.sh`, …), with no pipe or `&&`,
|
||||
in plain words (it runs with no shell: no quote, `$` or glob outside `node --test`).
|
||||
Anything else is wrapped in a script under `tests/`; otherwise the review
|
||||
reports the criterion as NOT RUN.
|
||||
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@
|
|||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { cpSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
|
@ -139,11 +139,14 @@ test('review 630 / 2261016d: a SKILL.md of 499 lines ending in a newline passes
|
|||
|
||||
// ---- cb34c32e: regression-check takes commandEnv from the shared util ------------
|
||||
|
||||
test('review 630 / cb34c32e: regression-check imports commandEnv from lib/util/command-env.mjs', () => {
|
||||
test('review 630 / cb34c32e: regression-check takes commandEnv from lib/util/command-env.mjs', () => {
|
||||
// Round 5 (MINOR 0e98d5c7): regression-check has no exec of its own any more;
|
||||
// it runs through the criteria runner's runCommand, which takes commandEnv
|
||||
// from lib/util/command-env.mjs.
|
||||
const src = read('lib/verification/regression-check.mjs');
|
||||
assert.match(src, /import \{ commandEnv \} from '\.\.\/util\/command-env\.mjs';/);
|
||||
assert.doesNotMatch(src, /from '\.\/red-proof\.mjs'/);
|
||||
assert.ok(src.indexOf('function exec(') > src.lastIndexOf('\nimport '), 'exec is defined after the imports');
|
||||
assert.match(src, /runCriteria\(c, o\)/);
|
||||
assert.match(read('lib/verification/criteria-runner.mjs'), /import \{ commandEnv \} from '\.\.\/util\/command-env\.mjs';/);
|
||||
});
|
||||
|
||||
// ---- CLAUDE.md names the mode-read exemptions as they are -----------------------
|
||||
|
|
@ -331,11 +334,15 @@ test('review 630 r3: a fixture document under tests/ is a test file, and "e.g."
|
|||
assert.deepEqual(stepTestsOnly(step('`./tests/a.test.mjs`')).files, ['tests/a.test.mjs'], './ is stripped');
|
||||
});
|
||||
|
||||
test('review 630 r3: the plan-file sweep reads Files: with the same reader as the tests-only check', async () => {
|
||||
test('review 630 r3: the plan-file sweep and the tests-only check both drop a leading ./', async () => {
|
||||
// Round 3 had the sweep read Files: through the tests-only reader; the
|
||||
// operator reversed that on 2026-10-11 (the K6 sweep keeps v6.2.0's reader,
|
||||
// round 5). What the round-3 finding named still holds: `./` is dropped by both.
|
||||
const { plannedPaths } = await import('../../lib/verification/plan-scope-sweep.mjs');
|
||||
const plan = '## Implementation Plan\n\n### Step 1: x\n\n- **Files:** `./src/a.mjs`, tests/b.test.mjs + src/c.mjs\n- **Changes:** x\n';
|
||||
const plan = '## Implementation Plan\n\n### Step 1: x\n\n- **Files:** `./src/a.mjs`, `./tests/b.test.mjs`\n- **Changes:** x\n';
|
||||
const body = plan.slice(plan.indexOf('### Step 1'));
|
||||
assert.deepEqual(plannedPaths(plan), [...stepTestsOnly(body).files].sort());
|
||||
assert.deepEqual(plannedPaths(plan), ['src/a.mjs', 'tests/b.test.mjs']);
|
||||
assert.deepEqual(stepTestsOnly(body).files, ['src/a.mjs', 'tests/b.test.mjs']);
|
||||
});
|
||||
|
||||
test('review 630 r3: --steps-from refuses an appended step with no Verify: on a 1.8 plan too', () => {
|
||||
|
|
@ -423,3 +430,159 @@ test('review 630 r4: no tilde word, and nothing after `make test`', async () =>
|
|||
assert.equal(allowedCommand(cmd, { cwd: ROOT }).allowed, true, `the control: ${cmd}`);
|
||||
}
|
||||
});
|
||||
|
||||
// ==== round 5 of the release review (2026-10-11), each red on d37e538 ===========
|
||||
// Operator 2026-10-11: no more patching of the same two places. A check runs
|
||||
// with no shell (its words reach the runner as written), and the K6 sweep has
|
||||
// its own v6.2.0 reader again; the shared reader stays for the tests-only check.
|
||||
|
||||
const PASS_SCREENS = { allow: () => ({ allowed: true, reason: '' }), screen: () => ({ allowed: true, rule: '' }) };
|
||||
|
||||
test('review 630 r5 / da7584e7: a check runs without a shell - its words reach the runner as written', async () => {
|
||||
const { runCriteria } = await import('../../lib/verification/criteria-runner.mjs');
|
||||
const dir = mkdtempSync(join(tmpdir(), 'r630-noshell-'));
|
||||
try {
|
||||
mkdirSync(join(dir, 'tests'));
|
||||
writeFileSync(join(dir, 'tests', 'args.sh'), 'printf "%s\\n" "$@"\n');
|
||||
const [r] = runCriteria([{ label: 'X1', text: 'x', command: 'bash tests/args.sh ~/x *.sh $HOME' }], { cwd: dir, ...PASS_SCREENS });
|
||||
assert.equal(r.status, 'passed', r.output);
|
||||
assert.equal(r.output, '~/x\n*.sh\n$HOME\n', 'no shell expanded a tilde, a glob or a variable');
|
||||
const [missing] = runCriteria([{ label: 'X2', text: 'x', command: 'no-such-runner-r630 --x' }], { cwd: dir, ...PASS_SCREENS });
|
||||
assert.equal(missing.status, 'failed', 'a runner that is not installed is a failure, never a pass');
|
||||
assert.match(missing.output, /ENOENT|not found/i, missing.output);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('review 630 r5 / c25f8704: a glob cannot climb out of tests/ or the repo', async () => {
|
||||
const { runCriteria, allowedCommand } = await import('../../lib/verification/criteria-runner.mjs');
|
||||
const base = mkdtempSync(join(tmpdir(), 'r630-escape-'));
|
||||
const repo = join(base, 'repo');
|
||||
try {
|
||||
mkdirSync(join(repo, 'tests'), { recursive: true });
|
||||
mkdirSync(join(repo, 'outside'));
|
||||
writeFileSync(join(repo, 'outside', 'evil.sh'), 'echo ESCAPED > escaped-bash.txt\n');
|
||||
writeFileSync(join(repo, 'tests', 'a.test.mjs'), "import { test } from 'node:test';\ntest('in', () => {});\n");
|
||||
writeFileSync(join(base, 'outside.test.mjs'),
|
||||
`import { test } from 'node:test';\nimport { writeFileSync } from 'node:fs';\ntest('out', () => { writeFileSync(${JSON.stringify(join(base, 'escaped-node.txt'))}, 'x'); });\n`);
|
||||
const results = runCriteria([
|
||||
{ label: 'E1', text: 'x', command: 'bash tests/.?/outside/evil.sh' },
|
||||
{ label: 'E2', text: 'x', command: 'node --test .*' },
|
||||
], { cwd: repo });
|
||||
assert.equal(existsSync(join(repo, 'escaped-bash.txt')), false, `bash ran a script outside tests/: ${JSON.stringify(results[0])}`);
|
||||
assert.equal(existsSync(join(base, 'escaped-node.txt')), false, `node --test ran a test file outside the repo: ${JSON.stringify(results[1])}`);
|
||||
assert.notEqual(results[0].status, 'passed');
|
||||
for (const cmd of ['node --test ../outside.test.mjs', 'node --test tests/../../outside.test.mjs', 'node --test /tmp/x.test.mjs',
|
||||
'pytest ../tests', 'go test ../...', 'npm test -- ../x.test.mjs', 'jest /abs/x.test.js', 'vitest run ../x', 'bash tests/*.sh',
|
||||
'pytest tests/test_*.py', 'jest src/*.test.js', 'npm test -- tests/*.mjs']) {
|
||||
assert.equal(allowedCommand(cmd, { cwd: ROOT }).allowed, false, `${cmd} must not run`);
|
||||
}
|
||||
for (const cmd of ['node --test tests/lib/*.test.mjs', 'node --test tests/**/*.test.mjs', 'go test ./...', '/bin/bash tests/fixtures/criteria-exit-0.sh',
|
||||
'node --test tests/', 'pytest tests/', 'git diff HEAD~1..HEAD --stat']) {
|
||||
const r = allowedCommand(cmd, { cwd: ROOT });
|
||||
assert.equal(r.allowed, true, `the control: ${cmd} (${r.reason})`);
|
||||
}
|
||||
} finally { rmSync(base, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('review 630 r5 / 8f3cf84d: a word starting with @ is refused (pytest reads @file as more arguments)', async () => {
|
||||
const { allowedCommand } = await import('../../lib/verification/criteria-runner.mjs');
|
||||
for (const cmd of ['pytest @tests/args.txt', 'python -m pytest @tests/args.txt', 'python3 -m pytest -q @a', 'uv run pytest @a', 'node --test @a', 'jest @a']) {
|
||||
const r = allowedCommand(cmd, { cwd: ROOT });
|
||||
assert.equal(r.allowed, false, `${cmd} must not run`);
|
||||
assert.match(r.reason, /@/);
|
||||
}
|
||||
});
|
||||
|
||||
test('review 630 r5 / da7584e7: every place that runs a check runs it with no shell', () => {
|
||||
for (const rel of ['lib/verification/criteria-runner.mjs', 'lib/verification/regression-check.mjs', 'lib/verification/red-proof.mjs',
|
||||
'lib/verification/done-line.mjs', 'lib/util/test-lock.mjs']) {
|
||||
const src = read(rel);
|
||||
assert.doesNotMatch(src, /spawnSync\(\s*['"](?:\/bin\/)?(?:ba)?sh['"]/, `${rel} starts a shell`);
|
||||
assert.doesNotMatch(src, /shell:\s*true/, `${rel} asks spawn for a shell`);
|
||||
if (rel !== 'lib/verification/criteria-runner.mjs') assert.match(src, /\brunCommand\b/, `${rel} runs through criteria-runner's runCommand`);
|
||||
}
|
||||
assert.doesNotMatch(read('lib/verification/regression-check.mjs'), /function exec\(/, 'regression-check has no exec of its own (MINOR 0e98d5c7)');
|
||||
});
|
||||
|
||||
test('review 630 r5 / da7584e7: the done line runs its command with no shell', async () => {
|
||||
// sh reads `**` as `*` (one level); node --test expands its own pattern at
|
||||
// every depth. Only with no shell does the top-level test run.
|
||||
const { writeDoneLine } = await import('../../lib/verification/done-line.mjs');
|
||||
const repo = mkdtempSync(join(tmpdir(), 'r630-done-'));
|
||||
try {
|
||||
mkdirSync(join(repo, 'tests', 'sub'), { recursive: true });
|
||||
writeFileSync(join(repo, 'tests', 'sub', 'deep.test.mjs'), "import { test } from 'node:test';\ntest('deep', () => {});\n");
|
||||
writeFileSync(join(repo, 'tests', 'top.test.mjs'),
|
||||
`import { test } from 'node:test';\nimport { writeFileSync } from 'node:fs';\ntest('top', () => { writeFileSync(${JSON.stringify(join(repo, 'top-ran.txt'))}, 'x'); });\n`);
|
||||
for (const args of [['init', '-q'], ['-c', 'user.email=t@example.invalid', '-c', 'user.name=t', 'commit', '-q', '--allow-empty', '-m', 'x']]) {
|
||||
assert.equal(spawnSync('git', args, { cwd: repo }).status, 0);
|
||||
}
|
||||
const r = writeDoneLine({ cwd: repo, step: 1, cmd: 'node --test tests/**/*.test.mjs', report: join(repo, 'report.md') });
|
||||
assert.equal(r.status, 'written', JSON.stringify(r));
|
||||
assert.equal(existsSync(join(repo, 'top-ran.txt')), true, 'the pattern reached node as written');
|
||||
} finally { rmSync(repo, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('review 630 r5 / ff80c7ef 5dc53c7e 02cea4aa: the K6 sweep plans what v6.2.0 planned, with its own reader', async () => {
|
||||
const { plannedPaths } = await import('../../lib/verification/plan-scope-sweep.mjs');
|
||||
const plan = (files) => `## Implementation Plan\n\n### Step 1: x\n\n- **Files:** ${files}\n- **Changes:** x\n`;
|
||||
const cases = [
|
||||
['`src/a.mjs` — replaces the helper in lib/util.mjs and touches config.json; see Node.js docs', ['src/a.mjs']],
|
||||
['`src/cli.mjs` — follows the pattern in lib/', ['src/cli.mjs']],
|
||||
['.gitignore, src/a.mjs', ['.gitignore', 'src/a.mjs']],
|
||||
['src\\cli.mjs', ['src/cli.mjs']],
|
||||
['`src/a.mjs`/`src/b.mjs`', ['src/a.mjs', 'src/b.mjs']],
|
||||
['`src/a.mjs`—adds', ['src/a.mjs']],
|
||||
['`src/cli.mjs`: add the flag; `src/b.mjs`.', ['src/b.mjs', 'src/cli.mjs']],
|
||||
];
|
||||
for (const [files, want] of cases) assert.deepEqual(plannedPaths(plan(files)), want, files);
|
||||
assert.doesNotMatch(read('lib/verification/plan-scope-sweep.mjs'), /import \{[^}]*\bfieldPaths\b/, 'K6 does not read Files: through the tests-only reader');
|
||||
});
|
||||
|
||||
test('review 630 r5 / a4c8005d 02cea4aa: the tests-only check takes only what passes a path test', () => {
|
||||
const step = (files) => `- **Files:** ${files}\n- **Changes:** x\n`;
|
||||
const cases = [
|
||||
['`tests/parse.test.mjs` (asserts `parse()` throws)', true, ['tests/parse.test.mjs']],
|
||||
['`tests/a.test.mjs` — covers pass/fail', true, ['tests/a.test.mjs']],
|
||||
['`tests/a.test.mjs` — covers JSON, YAML', true, ['tests/a.test.mjs']],
|
||||
['`tests/a.test.mjs` — calls `parser.parse()` twice', true, ['tests/a.test.mjs']],
|
||||
['`tests/a.test.mjs`, .gitignore', false, ['tests/a.test.mjs', '.gitignore']],
|
||||
['tests/a.test.mjs, .gitignore', false, ['tests/a.test.mjs', '.gitignore']],
|
||||
['`tests/a.test.mjs`/`src/b.mjs`', false, ['tests/a.test.mjs', 'src/b.mjs']],
|
||||
['`tests\\a.test.mjs`', true, ['tests/a.test.mjs']],
|
||||
['tests\\a.test.mjs, src\\cli.mjs', false, ['tests/a.test.mjs', 'src/cli.mjs']],
|
||||
['Makefile.', false, ['Makefile']],
|
||||
];
|
||||
for (const [files, testsOnly, want] of cases) assert.deepEqual(stepTestsOnly(step(files)), { testsOnly, files: want }, files);
|
||||
});
|
||||
|
||||
test('review 630 r5 / e1ff583a f582b9c2: frontmatter reads trailing space and a comment after a plain value as YAML does', async () => {
|
||||
const { readFrontmatterFields } = await import('../../lib/skills/skill-frontmatter.mjs');
|
||||
const r = readFrontmatterFields('---\nname: "debugging" \nmodel: opus # operator pin\neffort: high # pinned\ntags: [a] \n' +
|
||||
'allowed-tools: [Read, Grep] # read only\ndescription: | # folded below\n Use when x.\n---\nbody\n');
|
||||
assert.deepEqual(r.hazards, [], JSON.stringify(r.hazards));
|
||||
assert.equal(r.fields.name.value, 'debugging');
|
||||
assert.equal(r.fields.model.value, 'opus');
|
||||
assert.equal(r.fields.effort.value, 'high');
|
||||
assert.deepEqual(r.fields.tags.value, ['a']);
|
||||
assert.deepEqual(r.fields['allowed-tools'].value, ['Read', 'Grep']);
|
||||
assert.equal(r.fields.description.value, 'Use when x.\n');
|
||||
// The control: a description is the text Claude Code routes on, so a comment
|
||||
// that cuts it is still a hazard (quote it), and so is a value that is only a comment.
|
||||
for (const fm of ['description: Use when x happens #1.', 'description: Use when x # note', 'name: # x']) {
|
||||
assert.ok(readFrontmatterFields(`---\n${fm}\n---\nbody\n`).hazards.length > 0, fm);
|
||||
}
|
||||
});
|
||||
|
||||
test('review 630 r5 / 41c43285 (operator 2026-10-09): the example plan keeps steps 1-2, and step 2 no longer says it must fail', () => {
|
||||
const plan = read('examples/01-add-verbose-flag/plan.md');
|
||||
const heading = plan.match(/^### Step 2: .*$/m)[0];
|
||||
assert.doesNotMatch(heading, /must FAIL/i, heading);
|
||||
assert.match(plan.slice(plan.indexOf(heading)), /should PASS today/, 'the step body still says it passes today');
|
||||
const progress = JSON.parse(read('examples/01-add-verbose-flag/progress.json'));
|
||||
const title = JSON.stringify(progress).match(/"title":"(Add stability test[^"]*)"/)?.[1];
|
||||
assert.equal(`### Step 2: ${title}`, heading, 'progress.json carries the same step title');
|
||||
const r = validatePlanContent(plan, { strict: false });
|
||||
const warned = r.warnings.filter((w) => w.code === 'PLAN_STEP_TESTS_ONLY').map((w) => w.message);
|
||||
assert.equal(warned.length, 2, `the operator kept the warning on steps 1-2: ${JSON.stringify(warned)}`);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -435,3 +435,17 @@ test('release-check r4 — other text on the badge line is not release bookkeepi
|
|||
assert.match(r.stdout, /README\.md/);
|
||||
} finally { fx.cleanup(); }
|
||||
});
|
||||
|
||||
// ==== round 5 of the release review of 6.3.0 (2026-10-11) ====================
|
||||
|
||||
test('release-check r5 — a test moved into an allowed place after the review is still a removal → 1', () => {
|
||||
const fx = fixture({ files: { 'guard.test.mjs': 'export const guarded = true;\n' } });
|
||||
try {
|
||||
mkdirSync(join(fx.repo, '.claude', 'voyage'), { recursive: true });
|
||||
git(fx.repo, 'mv', 'guard.test.mjs', '.claude/voyage/guard.test.mjs');
|
||||
git(fx.repo, 'commit', '-q', '-m', 'move the test out of the suite');
|
||||
const r = run(fx);
|
||||
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||
assert.match(r.stdout, /guard\.test\.mjs/);
|
||||
} finally { fx.cleanup(); }
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue