Kjell Tore Guttormsen
02243c6365
A denylist in front of /bin/sh is whack-a-mole. Measured 2026-09-18, end to end through both screens: 5 of 11 named evasions ran with real effect - a `command` prefix reached git, an escaped `rm` inside a shell fence deleted a directory, `find -delete` deleted a file, `>|` and `tee` wrote outside the working tree, a python one-liner deleted the whole tree - and 19 of 28 got past the refusal list on its own. Every quoting, aliasing and indirection form of the shell is another mole. So the screen is now an ALLOWLIST. A criterion runs only when its first word is a known test runner (npm test, npm run <script package.json declares>, node --test, vitest, jest, pytest, python -m pytest, uv run pytest, cargo test, go test, make test, bash <script under tests/>, a read-only git subcommand) AND the command carries no shell operator and no newline. Everything else is NOT RUN with the reason said out loud: never run, and never reported as a failure either - an absent measurement is not a finding. That also closes the smaller hole in the same file: a bare word a sentence merely names (`whoami`, `login`, `package.json`) is no longer executed, because it is not a runner. REFUSED_BY_POLICY is gone with the list that produced it; a command outside the allowlist is `unrunnable`, which in plan mode still fells the run and in brief mode is reported to the reviewer as an absent measurement. What the allowlist deliberately does NOT do, said in the file and in the reviewer's rubric: it is not a sandbox. `npm test`, `npm run <script>` and `make test` run whatever the repo's own package.json/Makefile says they run, including a script that pushes - that is the repo's responsibility. And it rejects honest commands too: an env prefix, a project's own binary, anything piped. A check that needs one of those is declared through `bash tests/<script>.sh`, the documented way in. Red first: 6 of the new tests fail against the previous runner (measured with an always-allow shim so the module still loads), including the end-to-end one where the canary directory was deleted and files were written outside the tree. The fixtures move from `true`/`false` to two allowlisted shell fixtures, because `false` is no longer a runner - the fail case must still be a real non-zero exit, not an unrun criterion. Suite 1148 (1146/0/2). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
30 lines
885 B
Markdown
30 lines
885 B
Markdown
---
|
|
task: criteria-runner fixture — a brief whose success criteria are mixed
|
|
slug: criteria-runner-fixture-brief
|
|
brief_version: "2.2"
|
|
framing: new-direction
|
|
---
|
|
|
|
# Task Brief: criteria-runner fixture
|
|
|
|
Fixture only. Consumed by `tests/lib/criteria-runner.test.mjs`. It is the
|
|
falsifying case for D-04: a read-only reviewer cannot run these commands, so
|
|
/trekreview runs them and hands over the result.
|
|
|
|
## TL;DR
|
|
|
|
Three success criteria: one passes, one fails on purpose, one is prose only.
|
|
|
|
## Goal
|
|
|
|
Exercise the brief side of the criteria runner.
|
|
|
|
## Success Criteria
|
|
|
|
- The runner reaches the shell: `bash tests/fixtures/criteria-exit-0.sh` exits 0
|
|
- The failing case is visible: `bash tests/fixtures/criteria-exit-1.sh` exits 0 (FAILS on purpose — exit 1)
|
|
- No new runtime dependencies are introduced
|
|
|
|
## Non-Goals
|
|
|
|
- Being executed by the pipeline. This file is a test fixture.
|