voyage-lekerepo trial (2026-10-06) point 7, operator decision 2026-10-08
(«grønt merke på låsen»): the stop guard blocked Stop three times while the
executor waited for the controller, with the step's test already green.
- test-lock.mjs green --cmd "<Verify>" runs the step's test command (same
allowlist + denylist as done-line) and records green_at only on exit 0;
pending clears it; a new lock starts without it.
- stop-test-lock lets a green-marked lock stop; the write guard still
refuses the locked test files.
- trekexecute: D1 marks green after a PASS; each fix round marks pending.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
voyage-lekerepo end-to-end trial (2026-10-06), report points 8, 10, 12, 15, 16:
- approval-log: «Approve (Recommended)» / «Godkjenn (anbefalt)» is the
approval it labels; the suffix never turns a hollow answer into a yes.
- trekexecute Phase 8: the state-file block imports atomic-write through
${CLAUDE_PLUGIN_ROOT}; a relative import only ran with cwd = plugin root
(the test runs the block from another directory).
- trekreview, trekresearch, trekcontinue: stats go through stats-append like
the other commands, and a failed write is reported, never skipped
silently (closes the open STATE finding for all three). The schema
fixture's stale line refs for these and trekplan are re-measured.
- annotate prints an absolute path, so file://{$ANNOT_HTML} is a live link.
- red-proof: `red_kind` = load-error | test-failure on a proven red.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
voyage-lekerepo end-to-end trial (2026-10-06), report points 2, 3, 4, 9:
11 of 13 brief criteria were NOT RUN at review with no warning in /trekbrief;
a plan `git grep` criterion passed validator, gate and three critic rounds,
then felled the run in Phase 7; the End-to-end field and Phase 7's text
search never met; a prose Verify left the test command undefined.
- criteria-runner: runnableCheck() screens a declared check without running
it (same allowlist; an npm run script is not looked up at write time);
/bin/bash tests/<script> allowed; the plan report carries end_to_end,
verified when a passed criterion runs a step's End-to-end command (the
older text form still counts).
- plan-validator 1.8: PLAN_VERIFICATION_NOT_RUNNABLE,
PLAN_STEP_VERIFY_NOT_RUNNABLE, PLAN_END_TO_END_NOT_RUNNABLE,
PLAN_END_TO_END_NOT_VERIFIED (the gate inherits them).
- brief-validator: BRIEF_SC_NOT_RUNNABLE warning; /trekbrief offers a
tests/ script; brief-reviewer no longer proposes unrunnable commands.
- trekexecute: Phase 7 copies end_to_end from the runner; Verify's test
command is its first backticked span.
Measured on the trial's own brief and plan: 11 warnings (the 11 NOT RUN),
and the plan stops on V5 git grep and the unlisted End-to-end command.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CLAUDE.md command table: each row from its owner (trekplan from order 3,
trekexecute from order 1, trekreview from order 5).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An independent review of abc3083 found 15 defects, the two worst verified
in scratch repos. Each fix started with a test that failed.
- Proof of red could be "proven" by the environment: a start point
unpacked with git archive had no node_modules and no .git, so any test
failed there. The start point is now a git clone --shared checked out at
start_sha, gitignored top-level entries are linked in, and a control run
(start point + the change) must pass, or the result is not-run.
- The Test first file was never committed (the template lists it apart
from Files:) and leaked into the next step's diff. The step's scope is
now Files: + test files, staged at the checkpoint.
- New files outside Files: were invisible to every check. Every untracked
file is registered with --intent-to-add; a revert unregisters them.
- The floor guard paired renames (foo.test.js -> .bak read as clean) and
missed skipIf and node's t.skip/t.todo; prose files no longer count as
stubs.
- Phase 7.5 audited from the last step's start_sha; it now uses the run's
session_start_sha.
- The done line is written after the checkpoint and checked with --commit.
- fix_rounds never passes 5; attempts are never reset; re-control keeps
the spec/quality verdicts and earlier floor-guard rulings; the closed
stop list maps every existing stop; failed vs stopped made consistent;
the controller's effort is said, not assumed.
Tests: 1470 (1468 pass, 0 fail, 2 skipped).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/trekexecute no longer builds steps itself. Each plan step is built by one
fresh helper (new agent step-implementer: opus, effort high, no Agent tool)
and judged by one controller (a general helper on opus) that reads the
diff from a recorded start_sha and gives two verdicts, spec and quality.
Fix rounds go up, maximum 5: three fixes by a fresh helper, then back to
the plan, then to the operator. No cheaper-model ladder.
Proof is a command, never the model's word:
- lib/verification/red-proof.mjs: the step's test must fail at the start
point and pass at the result (exit 0/1/2; 2 is never green). Reproduces
the 02243c6 red-first measurement (6 of 55) through the shim fixture.
- lib/verification/test-integrity.mjs: the floor guard on the diff
(removed asserts, skipped/deleted tests, muted checks, stubs, loosened
thresholds). Over the last 150 commits: 149 clean, 1 real finding.
- lib/verification/done-line.mjs: DONE step N only when the test command
exits 0.
Test first is required from plan_version 1.8. The test lock (K8) is used
through its CLI and reported as not-locked while it is missing. A run
stops only on a closed list of four classes; every other call goes into a
decision log with its cost if wrong. Without a passing run-the-thing
criterion the result is not verified end to end, never green.
Handover 5 gains optional per-step start_sha, report_path, red_proof,
test_lock, fix_rounds (0-5) and run-level decisions and end_to_end; old
progress files still validate and resume.
No isolation on the helper: measured, a worktree-isolated subagent
branches from the default branch (02f320f), not the executor's HEAD
(2c89a6b), so it would miss the earlier steps.
Borrowings credited in commands/trekexecute.md ## Credits (Superpowers,
ai-native-sdlc, Kiro, GitHub Spec Kit, agent-skills, Cherny).
Tests: 1411 -> 1457 (1455 pass, 0 fail, 2 skipped).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ordre 5 (wave 1, P11 + P5), review_version 1.1.
- K7: `verdict: ALLOW | WARN | BLOCK` is required in review.md (strict
error, soft warning for 1.0 reviews). New CLI lib/review/verdict-gate.mjs:
0 = ALLOW/WARN, 1 = BLOCK, 2 = could not decide (never green); a verdict
greener than the file's own BLOCKERs or JSON block is exit 2. Realpath
main guard (the old file:// guard exits 0 under a symlink - measured).
- No finding is struck for length: succinctness trims only MINOR/SUGGESTION
text (catalogue tier), keeps BLOCKER/MAJOR whole, counts trims; the
coordinator lists what it declined to judge.
- Claude Code's /code-review and /security-review run as the engine via the
Skill tool (never --fix/--comment/--post/ultra); lib/review/engine-findings.mjs
turns their findings into a reviewer payload; two catalogue keys
(CODE_REVIEW_DEFECT, SECURITY_REVIEW_VULNERABILITY), catalogue 12 -> 14.
An engine that did not run forbids ALLOW.
- Every BLOCKER is written as a numbered `### Step R<n>` with its source
finding (strict error REVIEW_MISSING_REMEDIATION_STEP).
- P5: lib/review/recurrence.mjs - a finding class in two review.md files
(current + git history, no new store) becomes one proposal, routed by the
catalogue's memory_route; written only between Voyage markers in
.claude/rules/voyage-review-lessons.md under a size budget, CLAUDE.md gets
a pointer only; the operator approves every line (Phase 8.7).
- Workflow substrate: catalogue enum in step; a missing reviewer or engine
forbids ALLOW there too (it returned ALLOW on zero findings before).
- Fixtures review-run-A/B said WARN over a BLOCKER; now BLOCK.
Borrowed and credited in code: Superpowers (declined-to-judge), ai-native-sdlc
(cap only nits; mistake/fix/check), Spec Kit (managed block; converge steps),
Pocock (retro routing), Kiro (verdict file + gate; evidence per lesson),
Cherny (engine -> brief control order; review -> memory; budget).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
P1 (order 2, wave 1). /trekbrief puts the task in a size class
(ceremony: small|bounded|architectural, K1; heaviest in doubt, said
aloud, overridable, only moved up) that sets the question budget and the
review cap, and asks one round of questions: facts looked up, never
asked; recommendation first; ranked by impact x uncertainty with a
visible Deferred list; stop when the next three answers can be predicted.
"høres bra ut", "gjør som du vil" and an empty answer are not approval.
K2: lib/util/approval-log.mjs, an append-only hash-chained log in the
target repo (.claude/voyage/approvals.jsonl) bound to the approved
content's sha256; append/verify (0/1/2), answer, digest. Committed by
lib/util/artifact-commit.mjs: the log always, the brief and plan only
with artifact_store: git (operator decision 2026-10-06). /trekplan
Phase 1 halts on a broken, diverged, missing or mismatched log entry;
Phase 11 logs and commits an explicit plan approval before any handoff.
Both new brief fields are optional: no brief_version bump. The
main-module guard in intent-approval.mjs and brief-validator.mjs now
compares real paths. Borrowings credited in the prose and module headers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/trekreview of 5680591..84a0621 (verdict WARN: 0 BLOCKER, 3 MAJOR, 1 MINOR,
1 MAJOR suppressed as unverified for length only). All closed red -> green:
- Suppressed MAJOR 1d0171a7 (verified live): Phase 8 spliced {plan-path}
into a single-quoted --payload. A lone quote broke the shell parse (event
lost, no written:false to report); a quoted pair ran a command
substitution and still reported written:true. event-emit now reads
`--payload -` from stdin and the three Phase 8 blocks pass the payload
through a quoted heredoc. Checked under bash and zsh.
- 570ba19b: the SC5 set-step extractor read line by line and never saw the
wrapped Step 0 sentinel (`blocked`); it now matches across the break and
the test asserts `blocked` is read.
- c69a865a: the Phase 9 line_ref in jsonl-schemas.md was 9 lines off. Both
trekexecute rows are now pinned to the exact span by a test.
- 0004fa23 (MINOR): the gates_mode_flag_override block says what to do on
written:false (print the reason, continue).
- emit() docstring no longer says failures are swallowed silently.
- 7ffaede5: chosen to KEEP the prose-only legacy `passed` tolerance, because
a public-plugin run started under the 6.1.0 prose would otherwise re-run
every finished step on resume; recorded as an accepted deviation in the
(local-only) brief. No code change.
Suite 1400 (1398/0/2), was 1393.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 9 said 'append to ${CLAUDE_PLUGIN_DATA}/trekexecute-stats.jsonl; if it is not set, skip silently' - the Bash tool env never carries it. It now runs stats-append.mjs trekexecute with the record on stdin, the /trekbrief Phase 7 shape, and reports a written:false. The gates_mode_flag_override record was prose with no writer; it now has an event-emit block (fixed literal payload: --gates is a boolean flag) and a documented shape {ts, event, known_event, payload:{flag, signal_effort}}. Phase 8's three event blocks spliced ${PLAN_PATH} (set in an earlier Bash call that does not survive) and ${WAVE_COUNT} (never assigned) into a double-quoted payload, so a literal run printed payload-not-json, and the decline reason was operator free text in a shell argument. Payloads are now single-quoted with {plan-path} / {wave-count} placeholders and the literal label hold-for-review; each call runs on its own and a written:false is printed and passed. docs/architecture.md and the jsonl-schemas rows (writer, measured line_refs) follow.
Red: 7 of 7 new tests fail on the pre-fix text (Phase 9 writer + no 'skip silently', Phase 9 record lands, no bare ${CLAUDE_PLUGIN_DATA}/...jsonl target, no shell expansion in --payload, the three Phase 8 blocks land known events when run as written in a fresh shell, the written:false sentence, the override writer and shape).
Verify 198/198 (stats-landing, main-merge-gate, gates-flag-coverage, profile-stats-fields, plugin-root-resolution, doc-consistency, no-claude-code-launch). Suite 1383 (1381/0/2).
Voyage order 20261005T104513Z-1250514346, plan step 3/5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/trekbrief Phase 7 says the writer adds order_id / order_claimed_at /
order_reason (never typed by the model) and that the yardstick joins use on
order_id; the fixture row lists them in `fields`, EXPORT_DENYLIST names them,
and its stale line_ref (657-672) now reads 961-976; the allowlist header
names them as dropped; docs/architecture.md Stats explains them.
/trekbrief --approve now runs Phase 7 after a successful stamp, with fixed
values (mode "approve", 0 turns, brief_quality/research_topics/auto_research
from the brief's frontmatter, auto_result "manual", intent_approved true).
Valgt fordi PM 04.10 (point 3): fix the code, not the criterion - the
approval must leave the trekbrief record. Measured hole: okf-wiki v1-k has
brief-approved 04.10 17:12Z and trekplan 17:29Z, but only a 03.10 trekbrief
record with intent_approved false. CLAUDE.md, README.md and
docs/command-modes.md say so. The manual path is unchanged.
npm test 1309 -> 1362 (1360 pass / 0 fail / 2 skip).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Operator decision 27.09.2026: Voyage works the PM way, so nothing in it
starts Claude Code. Work that needs another session becomes an order plus
a starter line the user gives a session they open themselves; Voyage
never starts the session.
/trekexecute's multi-session path (Phase 2.6) now prepares one wave at a
time: worktree per session, then lib/util/session-orders.mjs writes one
order per session (worktree, steps, done-condition, starter) plus
orders/wave.json, and the run stops. Running the same command again reads
each session's progress file (status mode), merges sequentially with
push-before-cleanup as before, and prepares the next wave. Phase 2.5 skips
the destructive pre-flight while a wave is in flight. Phase 7.6 recovery is
handed over the same way. Hard rule 21: never start Claude Code.
Removed with it, because they existed only for launched children: the
ANTHROPIC_API_KEY billing check (and its dry-run line and Hard rule 1
exception), the hooks-fire version precheck, the safety preamble, the
scoped-tool/turn/budget flags, templates/headless-launch-template.md, the
decomposer's launch.sh (now orders.md), /trekplan's "launch", the README
billing and MCP tuning sections, and scripts/q3-cache-prefix-experiment.mjs
(an experiment that spawned the binary; its results stay in docs/).
/trekresearch's deep-research pre-gate no longer runs the binary for its
version; the post-hoc check was already the authoritative guard.
Chose orders + starters over in-session subagents because the order's done
criterion asks for exactly that, and because /trekexecute disallows the
Agent tool on purpose (Hard rule 10); reopening that is a separate change.
Gate: 54 -> 0 hit lines outside EXEMPT. npm test 1250 (1245 pass, 0 fail,
5 skipped: 3 are the retired-reference scan without its local list).
verify.sh 7/0. The fixture run (tests/lib/session-orders.test.mjs) writes
two orders and wave.json for wave 1 and leaves a trap binary untouched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Veikart steg 1, økt 1 av 2. The three holes measured in the grounds:
(1) ## Intent / ## Goal were only checked for existence, (2) no approval
marker existed, (3) brief-approved was emitted only on the auto path.
- lib/validators/intent-approval.mjs: --check / --stamp. The marker is
intent_approved_hash (sha256 over normalized ## Intent + ## Goal) +
intent_approved_at. Editing either section after approval → STALE.
- /trekplan Phase 1 (Read the brief, both --brief and --project): runs
--check and HALTS on BRIEF_INTENT_NOT_APPROVED / _STALE / _INVALID with
the remedy spelled out; an unrunnable check halts too.
- /trekbrief Phase 4h (before the Phase 5 fork, asked even in --quick):
shows Intent + Goal verbatim, AskUserQuestion Approve / Revise / Leave;
only "Approve" runs --stamp. --stamp emits brief-approved, so the manual
(default) path records it; the auto path's own emission is removed.
New mode /trekbrief --approve <project-dir> = Phase 4h alone.
- README, CLAUDE.md, command-modes, HANDOVER-CONTRACTS §Handover 1,
jsonl-schemas (trekbrief-stats gains intent_approved).
Valgt ingen brief_version-bump fordi skjemaendringen er rent additiv (to
valgfrie felt) og kravet sitter i /trekplan — enhver produsents brief kan
godkjennes via /trekbrief --approve uten produsentendring. Valgt eget
--check-kall i stedet for et flagg på brief-validator fordi --brief-stien
i dag ikke kjører validatoren i det hele tatt; et nytt validatorkall der
ville også stoppe på andre feil. Valgt fallback-datamappe = målestokkens
(plugins/data/voyage-…) fordi CLAUDE_PLUGIN_DATA er tom i Bash-miljøet og
event-emits egen fallback er stille skip — slik ble brief-approved 0 records.
What the marker does NOT prove (module header, command prose, contract):
same user, same machine — any session can stamp. A trace, not a signature.
Suite 1183 → 1201 (1199/0/2). Mutants M1–M5 (stale check off, check
always valid, gate line removed, stamp emits nothing, no normalization)
each fell ≥ 1 test. yardstick unchanged: RED, 1 of 3 countable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One command, `node scripts/yardstick.mjs`, prints use, trust and
interrupts. Each row gives countable YES/NO, the value, the denominator and
the source it was read from. A fourth row asks whether the end-state gate can
be fooled by a mutant that keeps the module's signature. Exit is non-zero
while any number is not countable, the mutant is not felled, or the verdict
over the first five deliveries is not judged. Nothing counts a delivery yet,
so today it is RED for the right reasons:
use NO no order in the queue carries work-class
trust YES current ALLOW streak 0 of 37 (BLOCK 14, WARN 12, ALLOW 11,
last 2026-09-05 BLOCK, ordered by ts, read as JSON)
interrupts NO 0 user_input records of ~141k lines in 5 files, 0 emitter
call sites outside lib/stats/ and tests/
mutants 0 of 1 signature-preserving mutants felled: on M9 (every export
returns {status:'PASSED'}, every named test only checks typeof)
the gate closes both D-03 and D-04
It only measures. The gate, the emitter and the order writer are untouched.
The mutant row is meant to stay red until the gate learns to fell M9.
Choices, and why:
- Use counts over the order queue (pending, archive/ and claimed/), because
an order is the unit of work. The denominator is orders dated on or after a
cutoff (default 2026-09-21) that carry `work-class: new`. They count as
having gone through Voyage when a Voyage run in the stats has the order's
`slug:`. Both fields are the contract the order writer must meet. No order
carries them today, so the number is NOT COUNTABLE, not 0 %.
- Mailboxes whose names start with a dot are mailboxes too. A shell `*`
glob skips them, which is why a hand count of `coord/*/orders` comes out
lower. The row names them so a re-measure can reconcile.
- M9 must be LIVE before its row can count. Its typeof tests have to pass on
the M9 tree, checked by a runner independent of the gate, so a broken
fixture can never read as "felled".
Also: tests/fixtures/red-first/02243c6-always-allow-shim.mjs, the shim that
makes the red-first claim of 02243c6 reproducible (6 of 55). The recipe is in
its header.
Red first: the new test file failed to load before scripts/yardstick.mjs
existed, and a dot-mailbox test was red before the census learned about
dot-mailboxes. Suite 1161 -> 1183 (1181/0/2), also green on a clean tree
exported from the index.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cosmetic close came back one level up. The probe reads the test file from
the same tree it measures, so a four-line file holding two EMPTY tests with the
two named names closed D-03 and D-04 on a tree where `lib/verification/` did
not exist at all - "defects 0 of 7, registry intact" (measured 2026-09-18).
First a two-line stub exporting the right symbols; then an empty test with the
right name. A name is always forgeable.
So the named test is now run twice. Once on the tree, as before - and once in a
sandbox where the module the condition declares in `stubs` is replaced by a
stub exporting the same names, all inert. If the test still passes there, it
binds the name and not the behaviour, and the condition THROWS: NOT FELLABLE,
counted open. A condition that declares no `stubs`, or names a module that is
not there, cannot fire either.
The sandbox is a symlink overlay: every entry of the tree is symlinked, and
only the test file and the stubbed module are materialised for real - Node
resolves an ESM import through the realpath, so a symlinked test file would
import the original module and never see the mutant. Nothing is ever written
inside the measured tree, and the only directory removed is the one this code
made under the system temp dir (pinned by a test).
M7 is now a permanent mutant beside M6, in two forms: the checkpoint's own
reproduction (unfixed tree + empty named tests) and the harder one (the real
module present, so the stub can be built and the empty test passes against it).
Both report `defects 2 of 7`. A positive control pins that D-03/D-04 still
CLOSE on the real tree, so "not closed" everywhere cannot read as a working
probe.
The frozen denominator moves a third time, deliberately, and its `why` no
longer claims authority it does not have: the second and third amendments were
maintenance decisions by the maintainer, not operator decisions, and the
tracked file now says exactly that.
Measured after:
real tree node scripts/end-state-gate.mjs -> defects 0 of 7, intact, exit 1
M6 (stub) 8d1669e + current gate/registry/frozen + lib/cosmetic/stub.mjs -> 2 of 7
M7 (empty) 8d1669e + current gate/registry/frozen + 3-line
tests/lib/criteria-runner.test.mjs with the two named tests -> 2 of 7
Red first: 4 of the new tests failed before the change. Suite 1154 (1152/0/2).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A denylist in front of /bin/sh is whack-a-mole. Measured 2026-09-18, end to
end through both screens: 5 of 11 named evasions ran with real effect - a
`command` prefix reached git, an escaped `rm` inside a shell fence deleted a
directory, `find -delete` deleted a file, `>|` and `tee` wrote outside the
working tree, a python one-liner deleted the whole tree - and 19 of 28 got
past the refusal list on its own. Every quoting, aliasing and indirection form
of the shell is another mole.
So the screen is now an ALLOWLIST. A criterion runs only when its first word
is a known test runner (npm test, npm run <script package.json declares>,
node --test, vitest, jest, pytest, python -m pytest, uv run pytest,
cargo test, go test, make test, bash <script under tests/>, a read-only git
subcommand) AND the command carries no shell operator and no newline.
Everything else is NOT RUN with the reason said out loud: never run, and never
reported as a failure either - an absent measurement is not a finding. That
also closes the smaller hole in the same file: a bare word a sentence merely
names (`whoami`, `login`, `package.json`) is no longer executed, because it is
not a runner.
REFUSED_BY_POLICY is gone with the list that produced it; a command outside
the allowlist is `unrunnable`, which in plan mode still fells the run and in
brief mode is reported to the reviewer as an absent measurement.
What the allowlist deliberately does NOT do, said in the file and in the
reviewer's rubric: it is not a sandbox. `npm test`, `npm run <script>` and
`make test` run whatever the repo's own package.json/Makefile says they run,
including a script that pushes - that is the repo's responsibility. And it
rejects honest commands too: an env prefix, a project's own binary, anything
piped. A check that needs one of those is declared through
`bash tests/<script>.sh`, the documented way in.
Red first: 6 of the new tests fail against the previous runner (measured with
an always-allow shim so the module still loads), including the end-to-end one
where the canary directory was deleted and files were written outside the
tree. The fixtures move from `true`/`false` to two allowlisted shell fixtures,
because `false` is no longer a runner - the fail case must still be a real
non-zero exit, not an unrun criterion.
Suite 1148 (1146/0/2).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/trekreview now runs the commands a BRIEF declares, and a brief is an artifact
that can arrive from outside the repo. Measured 2026-09-18 on 6cafb4c: the
executor denylist stopped a download piped into a shell, but the remote-writing
git subcommand and a recursive delete of a path both RAN. The denylist screens
catastrophe (root deletion, fork bombs, mkfs); it was never meant to screen an
artifact under review.
A second screen, in the runner and ahead of the denylist, refuses four classes:
- a remote-writing git subcommand. The subcommand is found by walking git's own
options (`-C`, `-c`, `--git-dir`, ... take a value), so `git status` and
`git log` still run and `git -C sub push` does not.
- a recursive delete: any `rm` carrying `-r`/`-rf`/`--recursive`. A plain
`rm build/artifact.txt` still runs.
- a download piped straight into a shell (also caught by the denylist; pinned
here so the runner does not depend on another file for it).
- a write outside the working tree. `/dev/null`-class devices are fine, and so
is anything under the working tree; `~/...`, an absolute path elsewhere, and
a target carrying an unexpanded `$VAR` are refused - the runner cannot know
where a variable points, and guessing is how a screen stops screening.
A refusal is its own outcome, REFUSED_BY_POLICY: the command never reaches a
shell, and `summary.ok` is false in both plan and brief mode. For the reviewer,
REFUSED is like NOT RUN - the absence of a measurement, never on its own a
finding - and the rubric and the evidence block both say so.
Chosen deliberately, and it is stricter than today's habit: writing scratch to
/tmp is refused too. The repo's own example plan does `> /tmp/out`. Verification
output belongs in the working tree; exempting the whole system temp dir would
have made the rule unstatable, since a working tree created under /tmp then
contains its own escape hatch.
NOT covered, stated rather than implied:
- other writing git subcommands (tag, remote, config, gc) - only push is listed
- writes through a wrapper: `sh -c '...'`, `xargs`, `find -exec`, a Makefile
target, a script the criterion invokes. The screen reads the command it is
given, not what that command goes on to do.
- `>` inside a quoted string reads as a redirect, so a criterion echoing a
literal `>` is refused. Fail-closed, on purpose.
- the whole surface still runs with the invoking process's permissions; this is
a refusal list, not a sandbox.
The denylist-layer test now uses a stand-in command with a screen double: the
refusal list catches a recursive delete first, so naming one there would have
stopped exercising the denylist layer at all.
Red first: the 6 new tests failed before this change (`refuseCommand` did not
exist), and the fixture brief's four writes ran.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured 2026-09-18 (PM checkpoint on 6cafb4c, mutant M6): a two-line file
export function runPlanVerification() { return { ok: true }; }
export function formatCriteriaEvidence() { return ''; }
dropped into an otherwise UNFIXED tree closed both D-03 and D-04, and the gate
reported "defects 0 of 7, registry integrity: intact". The same checkpoint felled
5 of 5 mutants with the SUITE and 0 of 5 with the GATE. Both probes called
themselves `behaviour` while asking a grep whether an export name was present -
a phrase probe pointed at code.
A check condition may now name a test instead of a pattern:
{ "test": "tests/lib/criteria-runner.test.mjs", "name": "<exact test name>",
"expect": "fails" }
The gate spawns the test runner on that one test and reads its TAP line.
`expect: "fails"` holds - the defect stays open - while the test does not pass.
A missing file, a name that matches nothing, a runner that will not start: all
throw, which counts as NOT FELLABLE and therefore open. A check that cannot fire
is never "fixed".
The child's `NODE_TEST_*` env is stripped. The gate usually runs UNDER the test
runner, and an inherited `NODE_TEST_CONTEXT` makes the grandchild report over
the parent's IPC channel instead of stdout - the TAP line would never arrive and
the probe would silently stop felling anything.
M6 is now a permanent mutant in the gate's own test, so the cosmetic close
cannot come back. Verified end to end against a real stub tree:
git archive 8d1669e | tar -x -C $TMP
cp scripts/end-state-gate.mjs scripts/end-state-registry.json $TMP/scripts/
cp tests/fixtures/end-state-frozen.json $TMP/tests/fixtures/; cp STATE.md $TMP/
printf 'export function runPlanVerification…' > $TMP/lib/cosmetic/stub.mjs
node scripts/end-state-gate.mjs --root $TMP
-> defects 2 of 7, both NOT FELLABLE, exit 1
The gate also now states the limit out loud, as its own line above the table:
a behaviour probe proves the capability WORKS; that Phase 7 and Phase 4.5 CALL
it is pinned by TEXT in doc-consistency, not proven deterministically - real
proof is a headless plugin-eval run against a fixture plan (week 40).
FROZEN DENOMINATOR AMENDED, second time in one day, by the work order that
carries this change: D-03 and D-04 signatures moved because their checks
changed on purpose. tests/fixtures/end-state-frozen.json and the FROZEN literal
in tests/scripts/end-state-gate.test.mjs are updated together, both dated, both
saying why. No entry was removed.
Gate after this change: defects 0 of 7, registry integrity intact, exit 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The operator decided on 2026-09-18 that the two checks must stop measuring a
phrase and start measuring behaviour. The reason is the gate's own caveat: a
phrase probe closes on rewording, so "Phase 7 no longer says 'Skip for
trekplans'" and "the rubric no longer says 'exists and passes'" could both have
been satisfied by an edit that changed no behaviour at all. Both entries now
point at code a test exercises:
D-03 lib/**/*.mjs exports runPlanVerification
D-04 lib/**/*.mjs exports formatCriteriaEvidence
Verified open against the pre-fix tree (8d1669e) and closed against this one —
a check that cannot be seen to fire is not evidence.
What the probes can and cannot prove is now stated by the gate itself rather
than left to the reader. `PROBE_NOTES` carries one note per kind, the detail
line prints a note for every kind PRESENT in the row (and none for a kind no
entry declares), and the open/closed tags name the kind instead of labelling
`phrase` alone and leaving every other kind unlabelled. A behaviour probe
proves the capability exists; it does not prove a prose phase calls it — the
wiring is pinned by tests/lib/doc-consistency.test.mjs, and the gate says so.
THE DENOMINATOR MOVED ON PURPOSE. `tests/fixtures/end-state-frozen.json` and
the `FROZEN` literal in `tests/scripts/end-state-gate.test.mjs` carry new
signatures for D-03 and D-04, amended together and dated in both places. That
is the freeze mechanism working, not a bypass: the integrity check went
VIOLATED the moment the checks changed and stayed red until the amendment was
written down. The list may change — only as a decision said out loud. Nothing
else about the denominator moved: seven defect ids, three experiment ids, and
the agents/decisions/freeze configuration are unchanged, and no entry was
removed.
Gate after: defects 0 of 7, registry intact, still RED and exit 1 on the other
four rows — which is correct.
Suite: 1120 (1118/0/2), up 3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rubric required `brief-conformance-reviewer` to classify a Success
Criterion as Full only when "its verification command/test exists and passes".
Its tools are `Read`, `Glob`, `Grep`. It cannot run anything, so "passes" was
either guessed from the command's mere existence or quietly downgraded to
"exists" — a BLOCKER-tier rule key resting on an impression.
The reviewer stays read-only — a reviewer that executes the code it reviews is
not an independent reviewer. The command does the running instead:
- `/trekreview` Phase 4.5 runs the brief's `## Success Criteria` commands
through `lib/verification/criteria-runner.mjs --brief --evidence` and captures
the block as `sc_evidence_block`, pasted verbatim into the reviewer prompt in
Phase 5. The exit code does not stop the review — a failing criterion is
exactly what the review exists to find.
- `formatCriteriaEvidence` builds that block in code: one row per criterion with
the command, the exit code and the first output line. Chose a code-built block
over an orchestrator-written summary so the orchestrator cannot narrate a pass
that never happened.
- The rubric now judges the supplied result: `PASS` supports Full, `FAILED` /
`BLOCKED` is `Broken` with the exit code cited, and `NOT RUN` is the absence
of a measurement — never evidence in either direction.
- Phase 4.5 is skipped in `quick` mode: that mode does not launch the
conformance reviewer, so there is nobody to hand the result to.
Red first: seven tests in `tests/lib/criteria-runner.test.mjs` against a
committed brief fixture whose three criteria pass, fail, and are prose-only.
The two doc pins were verified red against the pre-fix files (rubric asked
"exists and passes"; no Phase 4.5; the block reached nobody).
Suite: 1117 (1115/0/2), up 9.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A trekplan's `## Verification` section is where the brief's success criteria
land. Phase 7 opened with "**Skip for trekplans.**", and only the multi-session
wave path (Phase 2.6 Step 3) ran master verification. A plan executed in ONE
session therefore reported `completed` without ever running the criteria it was
measured against — the executor's own belief was the only evidence.
The check now exists as code, not as an instruction:
- `lib/verification/criteria-runner.mjs` parses the criteria an artifact
DECLARES (a plan's `## Verification`, a brief's `## Success Criteria`), runs
each command, and returns a verdict built from exit codes. Fail-closed
throughout: a placeholder, a prose-only criterion, or an unavailable screen
is `unrunnable`/`blocked`, never `passed`. A plan with no `## Verification`
section exits 1 — a plan that promises no end-to-end check cannot be reported
as verified.
- Every command is screened through the plugin's own PreToolUse denylist
(`hooks/scripts/pre-bash-executor.mjs`) before it reaches a shell. Chose
invoking that hook over its documented stdin protocol rather than copying its
rules, because a command spawned from node never passes through the Bash tool
and so the hook cannot fire by itself — this keeps exactly one denylist.
- Phase 7 is now "Exit / verification check": session specs run the exit
condition, trekplans run the criteria runner. Phase 4's entry-condition skip
for trekplans stands — a plan carries no entry condition; the exit side is
not symmetrical.
- A failing criterion FELLS the run: `plan_verification.status != "passed"`
forbids `result: completed`. That is clause 2 of the stop-signal contract,
now enforced on the single-session path too.
Red first: `tests/lib/criteria-runner.test.mjs` (26 tests) against two committed
fixture plans, one of which declares a criterion that fails on purpose. The
doc pin in `tests/lib/doc-consistency.test.mjs` guards the wiring — a capability
no phase calls is the same defect wearing a lib/ file; verified red against the
pre-fix Phase 7 (skip present, runner absent, no fell-the-run clause).
Suite: 1108 (1106/0/2), up 27 from 1081.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An independent adversarial review of the end-state gate reproduced a GREEN result, exit 0,
with the gate's 18 tests passing. Only the registry and the STATE markers were edited; every
other file stayed byte-identical. The judged party owned the denominator, and nothing
guarded it. This change closes that finding and three more, and reopens a defect the gate
had closed too early.
Registry integrity (the blocker):
- tests/fixtures/end-state-frozen.json freezes the denominator. It holds the 7 defect ids
and 3 experiment ids, each with a sha256 signature of its check (canonical JSON, key
order irrelevant), plus the exact agents / decisions / freeze configuration.
- The rule: an entry may close; it may never disappear or have its check changed. New
entries are allowed. spawnSites may never include agents/.
- The test file pins the manifest literally, with a comment that the denominator was
frozen on 2026-09-17 and that changing it is a decision. The gate verifies the registry
against the manifest on every run.
- A violation makes the gate red and names what moved. A missing manifest makes integrity
n/a, which is also red.
Dormant means "never spawned", not "never mentioned":
- An agent counts as spawned only when a command names it in a spawn instruction: a row of
a table headed `Agent`, a line that starts with Launch/Spawn, or a **name** block followed
by a Prompt: line.
- Prose, negations, HTML comments and fenced code never count. Block quotes cannot match
either, because every form is anchored at the start of the line.
- Re-measured under this definition: still 1 of 20 (synthesis-agent). Per agent, every other
spawnable agent has a spawn instruction in at least one command.
A fifth tally, feat-after-freeze:
- It counts commits after the `end-state-freeze` tag whose subject matches
^feat(\(|!|:).
- The row is n/a, and therefore red, when there is no tag yet, the root is not a git work
tree, or the root is not the top of its work tree. It is never 0.
Honest output:
- Every registry entry now declares its probe kind (phrase or byte).
- Open phrase probes are labelled as such, and closed ids are listed. The output states that
a closed phrase probe is evidence, not proof of behaviour. D-03 and D-04 stay open and
labelled; how they are fixed is still to be decided.
- The decisions row now says it counts ticks, not verified decisions.
D-07 reopened:
- Its check now also covers README.md and commands/trekresearch.md, where the "orchestrator
runs on opus" claim still lives. README even ships a sed recipe for `model: opus` lines no
command has.
- Defects are 3 of 7 until the separate docs fix lands.
Three surviving review mutants were killed with tests:
- an empty glob over an existing dir is silent;
- the header counts n/a rows as zero;
- a dormant-row error reads as 0.
Measured on a clean export of the index:
- Mutation harness, adapted from the review's: review code mutants 18 of 18 killed. Four of
them were re-targeted at the equivalent new code. M02 survived until a test with an
intact registry plus a missing freeze tag was added.
- New-logic mutants: 12 of 12 killed. One equivalent mutant (block-quote stripping) was
removed together with the dead clause it targeted.
- Registry attacks: 7 of 7 killed.
- The review's combined green attack (registry + STATE only) is now RED, exit 1:
"defects: D-03 is frozen but missing from the registry".
- Gate tests 38/38. Suite 1059 -> 1079 (1077/0/2).
- Gate: RED, defects 3 of 7, experiments 3 of 3, dormant 1 of 20; decisions and
feat-after-freeze are n/a in a clean export.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The SC asks two things of an effort: high run: that it discovered at least
one dimension AND that the dimension list in the output brief is a TRUE
SUPERSET of the interview-derived ones. activationCheck computed
dimensions - dimensions_baseline >= 1 and returned ok on that alone. A
count delta says nothing about membership: a run that dropped two
interview dimensions and appended three discovered ones is +1 and passed
the check while violating the second half outright. Supersetness was
asserted only by Phase 4.5's prose contract that discovery appends -
nothing read it.
The stats record cannot carry the dimension names that would show it
directly: names are free prose, and lib/exporters/field-allowlist.mjs
denies prose by omission (question, project_dir, brief_path are already
excluded for that reason). So the run attests membership with
dimensions_baseline_preserved, a boolean set in a new Phase 4.5 step 5,
and the gate refuses to call activation OK without it. An ABSENT
attestation fails rather than falling back to the old count-only rule -
otherwise legacy rows would keep passing on the defect.
Producer side wired end to end: the record format, the measurement-fields
prose (five fields -> six), the exporter allowlist, the jsonl-schemas
fixture row, and the --activation-check comment in
docs/storm-measurement.md.
Review finding d2786604458207a5a73478cdcb6a54bbdb92141d (MINOR).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuGhWAbWyRFBFeemfhxoVv
Scores committed agent-run fixtures against the golden corpus at
(file, rule_key) granularity, building on the deterministic coordinator
contract (4a). Offline: committed reviewer payloads, no live agent spawn,
no LLM, no network (the LLM-in-the-loop grading is the separate 4c tier).
- lib/review/gold-scorer.mjs: scoreFindings (precision/recall/f1 at
(file,rule_key) granularity, line+severity ignored) + scoreVerdict; pure,
with documented vacuous-set conventions.
- tests/fixtures/bakeoff-rich/runs/run-perfect.json: committed run that
reproduces all 5 seeded gold findings through runContract.
- tests/lib/gold-eval.test.mjs: the scoring RUN (precision/recall/f1 = 1.0,
verdict == expected_verdict BLOCK, nothing suppressed/skipped).
- lib/util/test-census.mjs: third census category (goldEval) — a scoring run
is neither behavior coverage nor a doc-pin; honest-count invariant now 3-way.
- docs/eval-corpus/README.md: 4b moved from Future hardening to implemented.
Suite 809 -> 822 (820/0/2). gold-scorer covers TP+FP+FN+degenerate paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJQYC5vpkJWxndS55vQQZ6
Run the full T2 §5 prose-vs-Workflow /trekreview bake-off (operator GO,
choice "a"): 3 runs/arm on a rich-finding JWT-auth fixture, resolving the
smoke's 0-finding limitation.
Deliverables:
- tests/fixtures/bakeoff-rich/ — JWT-auth brief + diff with 5 seeded blatant,
brief-traceable issues (varied severity/rule_key, one dual-flaggable).
- scripts/bakeoff-armA-merge.mjs — Arm A (prose) validate (NW1) + triplet-dedup,
matching Arm B's dedup exactly.
- scripts/bakeoff-fidelity.mjs — cross-arm + within-arm + granularity-ladder
fidelity analysis over the structured arm outputs.
- docs/T2-bakeoff-results.md §Full run — the T2 §5 verdict.
Result (3 runs/arm, both arms ran the coordinator):
- Verdict fidelity EQUIVALENT — all 6 runs BLOCK, cross-arm verdict-match 1.0.
- Finding-set: substrate is fidelity-neutral. Cross-arm jaccard 0.41 (triplet)
→ 0.71 (file,rule_key) → 1.0 (file); cross-arm ≈ within-arm at every
granularity. Issue coverage 5/5 in 6/6 runs. Low triplet jaccard is
line-citation noise shared by both arms, not a substrate effect.
- Token +4.4% (Arm B vs A; <=+15%). Classifier interference 0 at 9-agent
concurrency. JSON-robustness: Arm B schema-forced; Arm A 6/6 valid via NW1.
- VERDICT POSITIVE → S11 proceeds with opt-in --workflow flag.
Caveat (per plan posture): strict triplet-jaccard>=0.7 flag is 0/9, a
metric-calibration artifact (both arms sub-0.7 against themselves), not a
regression. Residual: F4 auto/bypass explicit-mode check (mode not settable
in-session).
Suite green (662/660 pass, 2 skip); plugin validate clean (modulo the
pre-existing root-CLAUDE.md warning). No production code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
Build the prose-vs-Workflow bake-off machinery for /trekreview Phase 5-6 and
run a 1-run/arm smoke to de-risk before the full measurement (operator posture:
build + smoke, then pause for go/no-go on the full >=3-runs/arm run).
New:
- lib/review/fidelity-diff.mjs (+ tests) — the PRIMARY metric: parse two
review.md (or two structured arm outputs) and compare verdict + jaccard over
(file,line,rule_key)-IDs + per-finding severity/rule_key. Reuses jaccard +
frontmatter + NW1 findings-schema + finding-id. fidelityDiffStructured avoids
rendering review.md per run.
- scripts/trekreview-armB.workflow.mjs — Arm B: Phase 5-6 as a Workflow
(parallel([conformance, correctness]) schema-forced -> JS dedup-by-triplet ->
agent(review-coordinator) verdict schema). Path-based input via args (reviewers
carry Read). Inlines dedup + the 12-key rule_key enum (scripts have no imports).
- tests/fixtures/bakeoff/ — committable fixture: real diff of b149538 (NW1) +
brief reconstructed from plan S9. Both arms review the same pinned input.
- docs/T2-bakeoff-results.md — smoke results + verdict + go/no-go recommendation.
Smoke result: SMOKE PASS. Arm B runs the full pipeline (3 agents) with ZERO
classifier interference; fidelity EQUIVALENT to Arm A at the verdict level
(both ALLOW; jaccard 1.0). Caveat: the clean TDD'd fixture yielded ~0 findings,
so finding-SET fidelity was not stressed (only verdict fidelity proven). A
reviewer-level divergence appeared (Arm B raised 1 raw finding, coordinator
filtered it; Arm A raised 0) — to be quantified in the full run on a
richer-finding-surface fixture. NOT the T2 §5 POSITIVE/NEGATIVE verdict.
Suite 647 -> 662 (660 pass / 2 skip / 0 fail; +15 fidelity-diff). claude plugin
validate clean (known root-CLAUDE.md warning only). Plan: docs/W1-narrow-wins-plan.md S10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
The v4.2/v4.3 bespoke playground SPA (~388 KB), the /trekrevise command,
Handover 8 (annotation → revision), the supporting lib/ modules
(anchor-parser, annotation-digest, markdown-write, revision-guard), the
Playwright e2e suite, and the @playwright/test / @axe-core/playwright
devDeps are removed. A browser walkthrough found the playground borderline
unusable, and it duplicated the official /playground plugin's
document-critique / diff-review templates.
In their place: scripts/render-artifact.mjs — a small, zero-dependency
renderer that turns a brief/plan/review .md into a self-contained,
design-system-styled, zero-network .html (frontmatter folded into a
<details> block). /trekbrief, /trekplan, and /trekreview call it on their
last step and print the file:// link; to annotate, run /playground
(document-critique) on the .md and paste the generated prompt back.
Resolves the v4.3.1-deferred findings as moot (their target files are
deleted). npm test green: 509 tests, 507 pass, 0 fail, 2 skipped.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Step 20 of v4.1 — implements drift detection in plan-validator.mjs per
brief Assumptions block 7: "Mismatch (e.g. korrupt manuell endring)
emitterer MANIFEST_PROFILE_DRIFT-warning fra plan-validator i --strict-modus."
Logic (after validateAllManifests in validatePlanContent):
1. Strict-mode only — soft mode never emits drift warnings.
2. Plan frontmatter must declare 'profile: <name>' to establish baseline.
3. For each step manifest, if profile_used is set AND differs from plan
profile, emit warning (NOT error) with code MANIFEST_PROFILE_DRIFT
and location 'step N: profile_used = X, plan profile = Y'.
Forward-compat preserved: drift is a warning, plan remains valid:true.
Operators see the drift in --strict mode without parsing breaking.
New files:
tests/validators/plan-validator-profile-drift.test.mjs — 4 tests
tests/fixtures/plan-profile-drift.md — drift fixture
Tests verify:
1. drift detected in strict mode → MANIFEST_PROFILE_DRIFT in warnings
2. drift NOT detected in soft mode → strict gate honored
3. matching profile → no drift warning
4. no plan-level profile → drift detection silent (no baseline)
Tests: 479 pass + 2 skipped (Docker not installed).
