fix(verification): the runner refuses writes a brief may not perform
/trekreview now runs the commands a BRIEF declares, and a brief is an artifact
that can arrive from outside the repo. Measured 2026-09-18 on 6cafb4c: the
executor denylist stopped a download piped into a shell, but the remote-writing
git subcommand and a recursive delete of a path both RAN. The denylist screens
catastrophe (root deletion, fork bombs, mkfs); it was never meant to screen an
artifact under review.
A second screen, in the runner and ahead of the denylist, refuses four classes:
- a remote-writing git subcommand. The subcommand is found by walking git's own
options (`-C`, `-c`, `--git-dir`, ... take a value), so `git status` and
`git log` still run and `git -C sub push` does not.
- a recursive delete: any `rm` carrying `-r`/`-rf`/`--recursive`. A plain
`rm build/artifact.txt` still runs.
- a download piped straight into a shell (also caught by the denylist; pinned
here so the runner does not depend on another file for it).
- a write outside the working tree. `/dev/null`-class devices are fine, and so
is anything under the working tree; `~/...`, an absolute path elsewhere, and
a target carrying an unexpanded `$VAR` are refused - the runner cannot know
where a variable points, and guessing is how a screen stops screening.
A refusal is its own outcome, REFUSED_BY_POLICY: the command never reaches a
shell, and `summary.ok` is false in both plan and brief mode. For the reviewer,
REFUSED is like NOT RUN - the absence of a measurement, never on its own a
finding - and the rubric and the evidence block both say so.
Chosen deliberately, and it is stricter than today's habit: writing scratch to
/tmp is refused too. The repo's own example plan does `> /tmp/out`. Verification
output belongs in the working tree; exempting the whole system temp dir would
have made the rule unstatable, since a working tree created under /tmp then
contains its own escape hatch.
NOT covered, stated rather than implied:
- other writing git subcommands (tag, remote, config, gc) - only push is listed
- writes through a wrapper: `sh -c '...'`, `xargs`, `find -exec`, a Makefile
target, a script the criterion invokes. The screen reads the command it is
given, not what that command goes on to do.
- `>` inside a quoted string reads as a redirect, so a criterion echoing a
literal `>` is refused. Fail-closed, on purpose.
- the whole surface still runs with the invoking process's permissions; this is
a refusal list, not a sandbox.
The denylist-layer test now uses a stand-in command with a screen double: the
refusal list catches a recursive delete first, so naming one there would have
stopped exercising the denylist layer at all.
Red first: the 6 new tests failed before this change (`refuseCommand` did not
exist), and the fixture brief's four writes ran.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
a7af76e5ff
commit
52b87978cb
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
6 changed files with 286 additions and 17 deletions
|
|
@ -16,8 +16,11 @@
|
|||
// Fail-closed everywhere: a criterion that cannot run — placeholder text, no
|
||||
// command, a screen that is unavailable — is NEVER reported as passed.
|
||||
//
|
||||
// Screening: every command is screened through the plugin's own PreToolUse
|
||||
// denylist (hooks/scripts/pre-bash-executor.mjs) before it reaches a shell.
|
||||
// Screening, two layers: every command is first screened against this file's
|
||||
// own refusal list (writes — git push, recursive delete, pipe-to-shell,
|
||||
// redirection outside the working tree), then through the plugin's own
|
||||
// PreToolUse denylist (hooks/scripts/pre-bash-executor.mjs) — catastrophe —
|
||||
// before it reaches a shell.
|
||||
// Commands spawned from here do not pass through the Bash tool, so that hook
|
||||
// would not otherwise fire; the hook is invoked over its documented stdin
|
||||
// protocol rather than copied, so there is exactly one denylist.
|
||||
|
|
@ -27,11 +30,12 @@
|
|||
// node lib/verification/criteria-runner.mjs --brief <brief.md> [--json|--evidence] [--cwd D] [--timeout MS]
|
||||
// --evidence emits the markdown block /trekreview hands to the conformance
|
||||
// reviewer; --json emits the whole report. They are two shapes of one run.
|
||||
// Exit: 0 = ok, 1 = a criterion failed / was blocked / could not run, 2 = usage
|
||||
// Exit: 0 = ok, 1 = a criterion failed / was refused / blocked / could not run, 2 = usage
|
||||
// or read error (deliberately distinct from "a criterion failed").
|
||||
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { resolve, dirname, join } from 'node:path';
|
||||
import { resolve, dirname, join, sep } from 'node:path';
|
||||
import { homedir } from 'node:os';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
|
||||
|
|
@ -209,6 +213,114 @@ export function screenCommand(command, opts = {}) {
|
|||
};
|
||||
}
|
||||
|
||||
// --- refusing ---------------------------------------------------------------
|
||||
|
||||
// The executor denylist screens CATASTROPHE (rm -rf /, fork bombs, mkfs). It
|
||||
// was never meant to screen an artifact that arrives from outside the repo,
|
||||
// and /trekreview now runs the commands a BRIEF declares. Measured 2026-09-18:
|
||||
// `git push origin main` and `rm -rf <path>` both ran; only the pipe-to-shell
|
||||
// was stopped. This list screens WRITES, and it is pinned by
|
||||
// tests/lib/criteria-runner.test.mjs so it cannot quietly shrink.
|
||||
//
|
||||
// A refusal is its own outcome, REFUSED_BY_POLICY: the command never reaches a
|
||||
// shell, and a refused criterion is never ok in either mode.
|
||||
|
||||
// Segments of a command that each start at command position.
|
||||
const SEPARATORS = /(?:\|\||&&|[;|&\n])/;
|
||||
const ENV_PREFIX = /^[A-Za-z_][A-Za-z0-9_]*=\S*$/;
|
||||
|
||||
// git's own options before the subcommand; these take a separate value.
|
||||
const GIT_VALUE_OPTS = new Set(['-C', '-c', '--git-dir', '--work-tree', '--namespace', '--exec-path']);
|
||||
// git subcommands that write somewhere this process does not own.
|
||||
const WRITING_GIT_SUBCOMMANDS = new Set(['push']);
|
||||
|
||||
const RECURSIVE_FLAG = /^(?:-[A-Za-z]*r[A-Za-z]*|--recursive)$/;
|
||||
const PIPE_TO_SHELL =
|
||||
/\b(?:curl|wget|fetch)\b[^|]*\|\s*(?:sudo\s+)?(?:[\w./-]*\/)?(?:ba|z|k|da|a)?sh\b/;
|
||||
|
||||
// `>` / `>>` and their target. `2>&1` and `>&2` duplicate a descriptor rather
|
||||
// than open a file, so `&` is excluded.
|
||||
const REDIRECT = /(?:^|[^>&\w])\d?>>?\s*(?!&)(?:"([^"]*)"|'([^']*)'|([^\s;|&<>]+))/g;
|
||||
const WRITABLE_DEVICES = new Set(['/dev/null', '/dev/stdout', '/dev/stderr', '/dev/tty']);
|
||||
|
||||
const basenameOf = (token) => token.split('/').pop();
|
||||
|
||||
function segments(command) {
|
||||
return String(command).split(SEPARATORS).map((seg) => seg.trim()).filter(Boolean);
|
||||
}
|
||||
|
||||
function tokensOf(segment) {
|
||||
const t = segment.split(/\s+/).filter(Boolean);
|
||||
while (t.length > 0 && ENV_PREFIX.test(t[0])) t.shift();
|
||||
return t;
|
||||
}
|
||||
|
||||
function gitSubcommand(tokens) {
|
||||
let i = 1;
|
||||
while (i < tokens.length) {
|
||||
const t = tokens[i];
|
||||
if (!t.startsWith('-')) return t;
|
||||
if (GIT_VALUE_OPTS.has(t)) { i += 2; continue; }
|
||||
i += 1;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function redirectTargets(command) {
|
||||
const out = [];
|
||||
for (const m of String(command).matchAll(REDIRECT)) out.push(m[1] ?? m[2] ?? m[3]);
|
||||
return out.filter(Boolean);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a redirect target lands inside `cwd`. A target that cannot be
|
||||
* resolved statically — one carrying an unexpanded `$VAR` — is treated as
|
||||
* outside: the runner cannot know where it points, and guessing is how a
|
||||
* screen stops screening.
|
||||
*/
|
||||
function writesOutside(target, cwd) {
|
||||
if (WRITABLE_DEVICES.has(target)) return false;
|
||||
if (target.includes('$') || target.includes('`')) return true;
|
||||
const expanded = target.startsWith('~') ? join(homedir(), target.slice(1)) : target;
|
||||
const abs = resolve(cwd, expanded);
|
||||
return abs !== cwd && !abs.startsWith(cwd.endsWith(sep) ? cwd : cwd + sep);
|
||||
}
|
||||
|
||||
/**
|
||||
* Screen one command against the refusal list. Returns `{refused, rule}`.
|
||||
* `cwd` is the working tree the criteria run in; writes outside it are refused.
|
||||
*/
|
||||
export function refuseCommand(command, opts = {}) {
|
||||
const cwd = resolve(opts.cwd ?? process.cwd());
|
||||
const text = String(command ?? '');
|
||||
const refuse = (rule) => ({ refused: true, rule });
|
||||
|
||||
if (PIPE_TO_SHELL.test(text)) return refuse('a download piped straight into a shell');
|
||||
|
||||
for (const segment of segments(text)) {
|
||||
const tokens = tokensOf(segment);
|
||||
if (tokens.length === 0) continue;
|
||||
const name = basenameOf(tokens[0]);
|
||||
if (name === 'git') {
|
||||
const sub = gitSubcommand(tokens);
|
||||
if (sub && WRITING_GIT_SUBCOMMANDS.has(sub)) {
|
||||
return refuse(`a remote-writing git subcommand (git ${sub})`);
|
||||
}
|
||||
}
|
||||
if (name === 'rm' && tokens.slice(1).some((t) => RECURSIVE_FLAG.test(t))) {
|
||||
return refuse('a recursive delete (rm -r)');
|
||||
}
|
||||
}
|
||||
|
||||
for (const target of redirectTargets(text)) {
|
||||
if (writesOutside(target, cwd)) {
|
||||
return refuse(`a write outside the working tree (${target})`);
|
||||
}
|
||||
}
|
||||
|
||||
return { refused: false, rule: '' };
|
||||
}
|
||||
|
||||
// --- running ----------------------------------------------------------------
|
||||
|
||||
function defaultExec(command, { cwd, timeoutMs }) {
|
||||
|
|
@ -225,12 +337,14 @@ function cap(text, max) {
|
|||
* Run each criterion that has a command. Returns one result per criterion:
|
||||
* passed the command exited 0
|
||||
* failed the command exited non-zero (or was killed by the timeout)
|
||||
* blocked the screen denied it — it never reached a shell
|
||||
* refused the refusal list denied it — it never reached a shell
|
||||
* blocked the executor denylist denied it — it never reached a shell
|
||||
* unrunnable there is no command to run (placeholder or prose-only criterion)
|
||||
*/
|
||||
export function runCriteria(criteria, opts = {}) {
|
||||
const exec = opts.exec ?? defaultExec;
|
||||
const screen = opts.screen ?? screenCommand;
|
||||
const refuse = opts.refuse ?? refuseCommand;
|
||||
const cwd = opts.cwd ?? process.cwd();
|
||||
const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
||||
const maxOutput = opts.maxOutput ?? DEFAULT_MAX_OUTPUT;
|
||||
|
|
@ -241,6 +355,10 @@ export function runCriteria(criteria, opts = {}) {
|
|||
if (!c.command) {
|
||||
return { ...base, status: 'unrunnable', exitCode: null, output: `not runnable: ${c.reason || 'no command'}` };
|
||||
}
|
||||
const refusal = refuse(c.command, { cwd });
|
||||
if (refusal.refused) {
|
||||
return { ...base, status: 'refused', exitCode: null, output: `REFUSED_BY_POLICY: ${refusal.rule}` };
|
||||
}
|
||||
const verdict = screen(c.command, { hookPath });
|
||||
if (!verdict.allowed) {
|
||||
return { ...base, status: 'blocked', exitCode: null, output: `blocked: ${verdict.rule}` };
|
||||
|
|
@ -270,11 +388,13 @@ export function summarize(results, opts = {}) {
|
|||
passed: count('passed'),
|
||||
failed: count('failed'),
|
||||
blocked: count('blocked'),
|
||||
refused: count('refused'),
|
||||
unrunnable: count('unrunnable'),
|
||||
};
|
||||
summary.ok =
|
||||
summary.failed === 0 &&
|
||||
summary.blocked === 0 &&
|
||||
summary.refused === 0 &&
|
||||
(!requireCommand || (summary.unrunnable === 0 && summary.total > 0));
|
||||
return summary;
|
||||
}
|
||||
|
|
@ -335,7 +455,7 @@ export function runSuccessCriteriaChecks(briefPath, opts = {}) {
|
|||
|
||||
// --- rendering --------------------------------------------------------------
|
||||
|
||||
const MARK = { passed: 'PASS', failed: 'FAILED', blocked: 'BLOCKED', unrunnable: 'NOT RUN' };
|
||||
const MARK = { passed: 'PASS', failed: 'FAILED', blocked: 'BLOCKED', refused: 'REFUSED', unrunnable: 'NOT RUN' };
|
||||
|
||||
export function render(rep) {
|
||||
const out = [];
|
||||
|
|
@ -343,7 +463,7 @@ export function render(rep) {
|
|||
out.push(`criteria-runner: ${s.ok ? 'OK' : 'NOT OK'} — ${rep.source} ${rep.heading}`);
|
||||
if (rep.error) out.push(` error: ${rep.error.code} — ${rep.error.message}`);
|
||||
out.push(
|
||||
` ${s.passed} passed · ${s.failed} failed · ${s.blocked} blocked · ${s.unrunnable} not run (of ${s.total})`
|
||||
` ${s.passed} passed · ${s.failed} failed · ${s.blocked} blocked · ${s.refused} refused · ${s.unrunnable} not run (of ${s.total})`
|
||||
);
|
||||
for (const r of rep.results) {
|
||||
const code = r.exitCode === null ? '' : ` (exit ${r.exitCode})`;
|
||||
|
|
@ -385,15 +505,17 @@ export function formatCriteriaEvidence(rep) {
|
|||
const s = rep.summary;
|
||||
out.push('');
|
||||
out.push(
|
||||
`${s.passed} passed · ${s.failed} failed · ${s.blocked} blocked · ${s.unrunnable} not run (of ${s.total}).`
|
||||
`${s.passed} passed · ${s.failed} failed · ${s.blocked} blocked · ${s.refused} refused · ${s.unrunnable} not run (of ${s.total}).`
|
||||
);
|
||||
}
|
||||
out.push('');
|
||||
out.push(
|
||||
'**NOT RUN is not a pass.** You cannot run these commands yourself — never infer, assume or ' +
|
||||
'reconstruct a result for a criterion that has none. A criterion whose result is FAILED or ' +
|
||||
'BLOCKED is `Broken`; a criterion with no result is judged on delivered code alone, and you ' +
|
||||
'say so in the Evidence column.'
|
||||
'BLOCKED is `Broken`; a criterion marked REFUSED was refused by policy before it reached a ' +
|
||||
'shell, so like NOT RUN it is the ABSENCE of a measurement and never on its own a finding; ' +
|
||||
'a criterion with no result is judged on delivered code alone, and you say so in the ' +
|
||||
'Evidence column.'
|
||||
);
|
||||
return out.join('\n');
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue