fix(verification): the criteria runner screens with an ALLOWLIST, not a denylist

A denylist in front of /bin/sh is whack-a-mole. Measured 2026-09-18, end to
end through both screens: 5 of 11 named evasions ran with real effect - a
`command` prefix reached git, an escaped `rm` inside a shell fence deleted a
directory, `find -delete` deleted a file, `>|` and `tee` wrote outside the
working tree, a python one-liner deleted the whole tree - and 19 of 28 got
past the refusal list on its own. Every quoting, aliasing and indirection form
of the shell is another mole.

So the screen is now an ALLOWLIST. A criterion runs only when its first word
is a known test runner (npm test, npm run <script package.json declares>,
node --test, vitest, jest, pytest, python -m pytest, uv run pytest,
cargo test, go test, make test, bash <script under tests/>, a read-only git
subcommand) AND the command carries no shell operator and no newline.
Everything else is NOT RUN with the reason said out loud: never run, and never
reported as a failure either - an absent measurement is not a finding. That
also closes the smaller hole in the same file: a bare word a sentence merely
names (`whoami`, `login`, `package.json`) is no longer executed, because it is
not a runner.

REFUSED_BY_POLICY is gone with the list that produced it; a command outside
the allowlist is `unrunnable`, which in plan mode still fells the run and in
brief mode is reported to the reviewer as an absent measurement.

What the allowlist deliberately does NOT do, said in the file and in the
reviewer's rubric: it is not a sandbox. `npm test`, `npm run <script>` and
`make test` run whatever the repo's own package.json/Makefile says they run,
including a script that pushes - that is the repo's responsibility. And it
rejects honest commands too: an env prefix, a project's own binary, anything
piped. A check that needs one of those is declared through
`bash tests/<script>.sh`, the documented way in.

Red first: 6 of the new tests fail against the previous runner (measured with
an always-allow shim so the module still loads), including the end-to-end one
where the canary directory was deleted and files were written outside the
tree. The fixtures move from `true`/`false` to two allowlisted shell fixtures,
because `false` is no longer a runner - the fail case must still be a real
non-zero exit, not an unrun criterion.

Suite 1148 (1146/0/2).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 02:51:40 +02:00
commit 02243c6365
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
12 changed files with 350 additions and 255 deletions

View file

@ -1288,9 +1288,10 @@ if [ ! -f "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" ]; then
exit 2
fi
# The working tree the criteria run in. It is also the boundary the runner's
# refusal list measures a write against, so it is resolved, never left to the
# process cwd - and a repo-less checkout falls back rather than passing "".
# The working tree the criteria run in. It is where every allowlisted command
# is executed and where a `bash tests/<script>.sh` criterion resolves, so it is
# resolved, never left to the process cwd - and a repo-less checkout falls back
# rather than passing "".
CRITERIA_CWD="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
node "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" --plan "{plan_path}" \
@ -1302,7 +1303,7 @@ The exit code is the verdict, and it is not advisory:
| Exit | Meaning | Executor |
|------|---------|----------|
| 0 | every criterion passed | `plan_verification.status = "passed"`; continue |
| 1 | a criterion failed, was refused by policy, was blocked, or could not run | `plan_verification.status = "failed"`; progress `status: "failed"` |
| 1 | a criterion failed, was blocked, or could not run (no command, or a command outside the allowlist of test runners) | `plan_verification.status = "failed"`; progress `status: "failed"` |
| 2 | the runner itself could not run | `plan_verification.status = "not-run"`; treated exactly like exit 1 |
A plan with no `## Verification` section exits 1 with
@ -1313,7 +1314,7 @@ Record in the progress file (additive-optional; unknown keys are tolerated by
`progress-validator.mjs`, so a legacy progress file still validates):
- `plan_verification.status` — `"passed" | "failed" | "not-run"`
- `plan_verification.summary` — `{total, passed, failed, blocked, refused, unrunnable}`
- `plan_verification.summary` — `{total, passed, failed, blocked, unrunnable}`
- `plan_verification.failed_criteria` — `[{label, command, exit_code}]`
**A failing criterion FELLS the run.** It is not "recorded and included in the

View file

@ -217,9 +217,9 @@ if [ ! -f "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" ]; then
fi
# Every command is screened twice before it reaches a shell: the runner's own
# refusal list (writes - push, recursive delete, pipe-to-shell, redirection
# outside the working tree) reports REFUSED, and the executor denylist
# (catastrophe) reports BLOCKED. Neither is ever run. Foreground only.
# ALLOWLIST of test runners (npm test, node --test, pytest, bash tests/<script>,
# a read-only git subcommand) reports anything else as NOT RUN, and the executor
# denylist (catastrophe) reports BLOCKED. Neither is ever run. Foreground only.
# The working tree the criteria run in - the same resolution trekexecute
# Phase 7 uses, so one criterion cannot resolve two ways in the two phases.
CRITERIA_CWD="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
@ -228,8 +228,9 @@ node "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" \
--brief "{brief_path}" --evidence --cwd "$CRITERIA_CWD"
```
Exit 0 means every criterion passed; exit 1 means at least one failed, was
refused by policy, was blocked by the executor denylist, or had no command;
Exit 0 means every criterion passed; exit 1 means at least one failed or was
blocked by the executor denylist; a criterion with no command, or one outside
the allowlist of test runners, is reported as NOT RUN;
exit 2 means the runner could not run. **The exit
code does not stop the review** — a failing criterion is exactly what the review
exists to find. Capture stdout as `sc_evidence_block`.