voyage-lekerepo end-to-end trial (2026-10-06), report points 2, 3, 4, 9:
11 of 13 brief criteria were NOT RUN at review with no warning in /trekbrief;
a plan `git grep` criterion passed validator, gate and three critic rounds,
then felled the run in Phase 7; the End-to-end field and Phase 7's text
search never met; a prose Verify left the test command undefined.
- criteria-runner: runnableCheck() screens a declared check without running
it (same allowlist; an npm run script is not looked up at write time);
/bin/bash tests/<script> allowed; the plan report carries end_to_end,
verified when a passed criterion runs a step's End-to-end command (the
older text form still counts).
- plan-validator 1.8: PLAN_VERIFICATION_NOT_RUNNABLE,
PLAN_STEP_VERIFY_NOT_RUNNABLE, PLAN_END_TO_END_NOT_RUNNABLE,
PLAN_END_TO_END_NOT_VERIFIED (the gate inherits them).
- brief-validator: BRIEF_SC_NOT_RUNNABLE warning; /trekbrief offers a
tests/ script; brief-reviewer no longer proposes unrunnable commands.
- trekexecute: Phase 7 copies end_to_end from the runner; Verify's test
command is its first backticked span.
Measured on the trial's own brief and plan: 11 warnings (the 11 NOT RUN),
and the plan stops on V5 git grep and the unlisted End-to-end command.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CLAUDE.md command table: each row from its owner (trekplan from order 3,
trekexecute from order 1, trekreview from order 5).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CLAUDE.md: the review paragraph (order 5) and the guards paragraph
(order 4) are both kept (planned union line).
The launch gate went red on the merged tree: post-approval-log reads
the session's permission_mode, and its regression test feeds that value.
Both were untracked when the branch ran its suite, so the gate never saw
them there. Operator decision 2026-10-06: a counted exemption in a new
class (mode-read: 3 + 1 hits), and the binding-rule sentence in CLAUDE.md
names it. An extra hit in either file turns the gate red again (checked).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An independent review of abc3083 found 15 defects, the two worst verified
in scratch repos. Each fix started with a test that failed.
- Proof of red could be "proven" by the environment: a start point
unpacked with git archive had no node_modules and no .git, so any test
failed there. The start point is now a git clone --shared checked out at
start_sha, gitignored top-level entries are linked in, and a control run
(start point + the change) must pass, or the result is not-run.
- The Test first file was never committed (the template lists it apart
from Files:) and leaked into the next step's diff. The step's scope is
now Files: + test files, staged at the checkpoint.
- New files outside Files: were invisible to every check. Every untracked
file is registered with --intent-to-add; a revert unregisters them.
- The floor guard paired renames (foo.test.js -> .bak read as clean) and
missed skipIf and node's t.skip/t.todo; prose files no longer count as
stubs.
- Phase 7.5 audited from the last step's start_sha; it now uses the run's
session_start_sha.
- The done line is written after the checkpoint and checked with --commit.
- fix_rounds never passes 5; attempts are never reset; re-control keeps
the spec/quality verdicts and earlier floor-guard rulings; the closed
stop list maps every existing stop; failed vs stopped made consistent;
the controller's effort is said, not assumed.
Tests: 1470 (1468 pass, 0 fail, 2 skipped).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wave 1, order 3 (P3 plan mode as the start, P2 design as its own part, P9
plan side). Contracts K3-K6 for the later merges.
- plan_version 1.8 (1.7 plans stay valid): Test first on every step, one
End-to-end ("run the thing") step, optional Interface in / Interface out /
Review focus fields, a ceremony class in the header (absent =
architectural), and for architectural plans a ## Design part: module map
without cycles, architecture rules that name their enforcer and the step
that proves them, a requirements check asked as keep / change.
- K3 lib/validators/plan-gate.mjs: the critic verdict is recorded INSIDE the
plan (Claude Code's plan approval hands a hook only the plan file's text),
bound to the hash taken before the reviewers read it; 0 go / 1 stop / 2
could not run; at most 3 critic rounds, then stop and ask.
- K5 lib/verification/arch-rules.mjs: runs the target repo's own
dependency-cruiser or import-linter (installs nothing; 2 = not measured,
never green); --prove --rule shows pass -> fail on a planted violation of
that rule -> pass. Verified against dependency-cruiser 18.5.0 and
import-linter 2.15; madge 8.0.0 left out (missed a planted .mjs cycle).
- K6 lib/verification/plan-scope-sweep.mjs: changed files no step names.
- /trekplan Phases 8-10: design part, bounded critic loop, the gate before
"execute", a plan-mode path through ExitPlanMode (not yet run end to end
inside plan mode). plan-critic 10 -> 12 dimensions; scope-guardian checks
the ceremony.
- Fixed on the way: plan-validator and plan-review-dedup used the
file://argv[1] main guard (no-op under a symlink or a space); CRLF plans
failed every manifest; the dedup dropped finding severity.
Borrowed, credited in place: obra/superpowers, Kiro (docs), addyosmani/
agent-skills, mattpocock/skills, GitHub Spec Kit, bashebr/ai-native-sdlc,
Boris Cherny's public threads.
Suite 1497 (1495 / 0 / 2), 86 new tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wave-1 order 4 (P7, P12, the P9 test lock, the P3 plan-approval guard).
All command hooks; none runs a model.
- P12: pre-bash-executor answers permissionDecision "ask" before merge
into main, push, tag/publish and destructive git (holds in auto mode).
post-approval-log writes exactly what the person was asked to the K2
approval log: merge/push to main as `merge`, tag/publish as `release`,
with a one-hour expiry; nothing when no person was asked.
- P3: pre-plan-gate runs the K3 plan gate on a Voyage plan leaving plan
mode and sends a refused plan back; "not measured" asks, never passes.
- P9: lib/util/test-lock.mjs (K8). pre-write-executor refuses edits to
locked test files; stop-test-lock refuses "done" for the locking
session (stop_hook_active guarded).
- P7: post-edit-check runs the target repo's own formatter/linter per
edit, only during a Voyage plan run or after opt-in
(.claude/voyage-guards.json); lib/verification/step-check.mjs (K9) fails
the step on findings; no tool reads "not measured", never green.
- STORM cap matcher also names `Agent` (the tool's name since 2.1.63).
- Mod precondition stated in README, SECURITY.md and every PreToolUse stop
guard: Voyage's guards are deterministic only when no installed mod
handles `tool.check`.
K2, K3 and K5 are stubbed in tests until orders 2 and 3 are merged.
Two independent review rounds; every finding has a regression test.
Suite 1600 (1598 pass / 0 fail / 2 skipped).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/trekexecute no longer builds steps itself. Each plan step is built by one
fresh helper (new agent step-implementer: opus, effort high, no Agent tool)
and judged by one controller (a general helper on opus) that reads the
diff from a recorded start_sha and gives two verdicts, spec and quality.
Fix rounds go up, maximum 5: three fixes by a fresh helper, then back to
the plan, then to the operator. No cheaper-model ladder.
Proof is a command, never the model's word:
- lib/verification/red-proof.mjs: the step's test must fail at the start
point and pass at the result (exit 0/1/2; 2 is never green). Reproduces
the 02243c6 red-first measurement (6 of 55) through the shim fixture.
- lib/verification/test-integrity.mjs: the floor guard on the diff
(removed asserts, skipped/deleted tests, muted checks, stubs, loosened
thresholds). Over the last 150 commits: 149 clean, 1 real finding.
- lib/verification/done-line.mjs: DONE step N only when the test command
exits 0.
Test first is required from plan_version 1.8. The test lock (K8) is used
through its CLI and reported as not-locked while it is missing. A run
stops only on a closed list of four classes; every other call goes into a
decision log with its cost if wrong. Without a passing run-the-thing
criterion the result is not verified end to end, never green.
Handover 5 gains optional per-step start_sha, report_path, red_proof,
test_lock, fix_rounds (0-5) and run-level decisions and end_to_end; old
progress files still validate and resume.
No isolation on the helper: measured, a worktree-isolated subagent
branches from the default branch (02f320f), not the executor's HEAD
(2c89a6b), so it would miss the earlier steps.
Borrowings credited in commands/trekexecute.md ## Credits (Superpowers,
ai-native-sdlc, Kiro, GitHub Spec Kit, agent-skills, Cherny).
Tests: 1411 -> 1457 (1455 pass, 0 fail, 2 skipped).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured 2026-09-18: nine ordinary shapes of a plan's `## Verification`
section parsed to zero criteria - an untagged fence, a ```text fence, a
markdown table, `## Verification (acceptance)`, `## Verification:`,
`### Verification`, an unclosed fence earlier in the document. Every one came
out as `0 of 0`, NOT OK, exit 1, and Phase 7 then forbade `result: completed`
without anyone being told that the FORMAT, not the code, was the problem.
"The section is empty" and "I cannot read this format" are different facts.
Three changes, one hole:
- The runner reports `NO_CRITERIA` with a source line (`plan.md:NN`) when the
section is there and nothing in it parsed, and names the two forms it does
read. Same for a brief's `## Success Criteria`, so the evidence block the
conformance reviewer gets says which of the two it is looking at rather than
showing an empty table.
- Phase 7 says it out loud instead of failing silently: report the source line
and the two forms, and say that the plan is what failed there, not the run.
- `/trekplan` now pins what it produces to what the runner reads: the heading
is exactly `## Verification`, the criteria are a bullet whose first
backticked span is the command or a shell-tagged fence, and the command must
be one the allowlist runs. A doc-consistency test holds the writer and the
reader together, so a runner that learns a new form must update the source.
Honest about the round trip: the two round-trip tests were GREEN on arrival -
the template already writes the bullet form the runner reads. What was missing
was not the format but the PIN: `/trekplan` mandated neither the heading string
nor the format, so a plan could satisfy the command's own instructions and
still parse to nothing. The tests now hold that.
Red first: 5 of the 7 new tests failed before the change (3 NO_CRITERIA, 2
doc-consistency); the 2 round-trip tests are guards, and said so above.
Suite 1161 (1159/0/2). Gate unchanged: defects 0 of 7, intact, exit 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A denylist in front of /bin/sh is whack-a-mole. Measured 2026-09-18, end to
end through both screens: 5 of 11 named evasions ran with real effect - a
`command` prefix reached git, an escaped `rm` inside a shell fence deleted a
directory, `find -delete` deleted a file, `>|` and `tee` wrote outside the
working tree, a python one-liner deleted the whole tree - and 19 of 28 got
past the refusal list on its own. Every quoting, aliasing and indirection form
of the shell is another mole.
So the screen is now an ALLOWLIST. A criterion runs only when its first word
is a known test runner (npm test, npm run <script package.json declares>,
node --test, vitest, jest, pytest, python -m pytest, uv run pytest,
cargo test, go test, make test, bash <script under tests/>, a read-only git
subcommand) AND the command carries no shell operator and no newline.
Everything else is NOT RUN with the reason said out loud: never run, and never
reported as a failure either - an absent measurement is not a finding. That
also closes the smaller hole in the same file: a bare word a sentence merely
names (`whoami`, `login`, `package.json`) is no longer executed, because it is
not a runner.
REFUSED_BY_POLICY is gone with the list that produced it; a command outside
the allowlist is `unrunnable`, which in plan mode still fells the run and in
brief mode is reported to the reviewer as an absent measurement.
What the allowlist deliberately does NOT do, said in the file and in the
reviewer's rubric: it is not a sandbox. `npm test`, `npm run <script>` and
`make test` run whatever the repo's own package.json/Makefile says they run,
including a script that pushes - that is the repo's responsibility. And it
rejects honest commands too: an env prefix, a project's own binary, anything
piped. A check that needs one of those is declared through
`bash tests/<script>.sh`, the documented way in.
Red first: 6 of the new tests fail against the previous runner (measured with
an always-allow shim so the module still loads), including the end-to-end one
where the canary directory was deleted and files were written outside the
tree. The fixtures move from `true`/`false` to two allowlisted shell fixtures,
because `false` is no longer a runner - the fail case must still be a real
non-zero exit, not an unrun criterion.
Suite 1148 (1146/0/2).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/trekreview now runs the commands a BRIEF declares, and a brief is an artifact
that can arrive from outside the repo. Measured 2026-09-18 on 6cafb4c: the
executor denylist stopped a download piped into a shell, but the remote-writing
git subcommand and a recursive delete of a path both RAN. The denylist screens
catastrophe (root deletion, fork bombs, mkfs); it was never meant to screen an
artifact under review.
A second screen, in the runner and ahead of the denylist, refuses four classes:
- a remote-writing git subcommand. The subcommand is found by walking git's own
options (`-C`, `-c`, `--git-dir`, ... take a value), so `git status` and
`git log` still run and `git -C sub push` does not.
- a recursive delete: any `rm` carrying `-r`/`-rf`/`--recursive`. A plain
`rm build/artifact.txt` still runs.
- a download piped straight into a shell (also caught by the denylist; pinned
here so the runner does not depend on another file for it).
- a write outside the working tree. `/dev/null`-class devices are fine, and so
is anything under the working tree; `~/...`, an absolute path elsewhere, and
a target carrying an unexpanded `$VAR` are refused - the runner cannot know
where a variable points, and guessing is how a screen stops screening.
A refusal is its own outcome, REFUSED_BY_POLICY: the command never reaches a
shell, and `summary.ok` is false in both plan and brief mode. For the reviewer,
REFUSED is like NOT RUN - the absence of a measurement, never on its own a
finding - and the rubric and the evidence block both say so.
Chosen deliberately, and it is stricter than today's habit: writing scratch to
/tmp is refused too. The repo's own example plan does `> /tmp/out`. Verification
output belongs in the working tree; exempting the whole system temp dir would
have made the rule unstatable, since a working tree created under /tmp then
contains its own escape hatch.
NOT covered, stated rather than implied:
- other writing git subcommands (tag, remote, config, gc) - only push is listed
- writes through a wrapper: `sh -c '...'`, `xargs`, `find -exec`, a Makefile
target, a script the criterion invokes. The screen reads the command it is
given, not what that command goes on to do.
- `>` inside a quoted string reads as a redirect, so a criterion echoing a
literal `>` is refused. Fail-closed, on purpose.
- the whole surface still runs with the invoking process's permissions; this is
a refusal list, not a sandbox.
The denylist-layer test now uses a stand-in command with a screen double: the
refusal list catches a recursive delete first, so naming one there would have
stopped exercising the denylist layer at all.
Red first: the 6 new tests failed before this change (`refuseCommand` did not
exist), and the fixture brief's four writes ran.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"The first backtick span is the command" is right for a plan, whose template
puts the command first, and wrong for a brief, whose criterion usually opens
by NAMING the thing under discussion. Measured 2026-09-18 on the repo's own
example brief: 5 of 6 criteria FAILED, 3 of them parse artifacts - `--verbose`
run as a command gave exit 2 ("invalid option"), `tests/` gave exit 126 ("is a
directory"). The rubric reads a FAILED result as decisive, so each one became
a BROKEN_SUCCESS_CRITERION BLOCKER about prose.
looksLikeCommand() screens the span by SHAPE only - no filesystem lookup, so a
span parses the same everywhere. Refused: a leading flag, a directory, a token
carrying quotes/braces/prose, and a lone relative path with a slash (an
explicit ./, ../, / or ~/ still runs, as do env-var prefixes). A refused span
is `unrunnable` with reason `not-a-command` - its own outcome, never FAILED,
and it never reaches a shell.
It deliberately does NOT scan on to a later span. "The first span that LOOKS
like a command" invents commands out of prose: in that same example brief it
would have run `whoami` and `login`, two real binaries a sentence happens to
name. An absent measurement is honest; a guessed one is not.
The shape check applies to prose spans only. Inside a shell-tagged fence the
author has already declared shell, so `[ -f x ] || exit 1` still runs.
The rubric follows: a NOT RUN result is never on its own a finding. The
Partial row now describes half-built DELIVERED CODE, and the reviewer gets a
table of the three reason strings - no-command, placeholder, not-a-command -
with what each says about the sentence rather than about the code.
Not covered, stated for the record: a multi-token span whose first token is a
non-executable file (`tests/golden/login.stdout --check`) still runs, and a
criterion whose command is real but whose binary is absent still reports the
shell's exit 127 - that is a true measurement of a missing binary, not a
parse artifact.
Red first: 4 runner tests + 1 doc-consistency pin failed before this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runner read bullet lines only. The repo's own example plan writes its
whole acceptance run as a fenced bash block, so `## Verification` parsed to
ZERO criteria, the runner exited 1, and Phase 7 forbade `result: completed`
- a correct plan felled every single-session run. Measured 2026-09-18 on
6cafb4c: none of the repo's plan artifacts exited 0.
Fences are now read twice over, for two opposite reasons:
- a `## ` heading INSIDE a fence is quoted text and no longer opens a
section. examples/02-real-cli/REGENERATED.md is a report that quotes a
plan outline inside one fence; it used to yield an empty section that
read as "0 of 0", and now yields the honest NO_VERIFICATION_SECTION.
- a shell-tagged fence inside the section holds the commands. The tag list
is closed (bash/sh/shell/zsh/console/shell-session): an untagged fence is
more often expected OUTPUT than input, and inventing a criterion from
output is the failure this file exists to prevent.
Blank and comment-only lines inside the block declare nothing. A `$`/`>`
console prompt is stripped; a `#` root prompt is NOT, because it cannot be
told from a comment and running a comment is the worse mistake.
Measured after the fix (parse only - one example names a fictional CLI):
examples/01 6 criteria, plan-template 2 (both placeholders, correctly NOT
RUN), the two runner fixtures 2 each, plan-run-C 1, REGENERATED.md 0 with
NO_VERIFICATION_SECTION.
Divergence from the order's premise, stated for the record: it said 2 of 3
example plans write `## Verification` as a fenced bash block. Ground truth
is 1 of 3 - REGENERATED.md has no section of its own at all - and the
example plan's block holds 6 command lines, not 5.
Red first: 4 of the 6 new tests failed before this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rubric required `brief-conformance-reviewer` to classify a Success
Criterion as Full only when "its verification command/test exists and passes".
Its tools are `Read`, `Glob`, `Grep`. It cannot run anything, so "passes" was
either guessed from the command's mere existence or quietly downgraded to
"exists" — a BLOCKER-tier rule key resting on an impression.
The reviewer stays read-only — a reviewer that executes the code it reviews is
not an independent reviewer. The command does the running instead:
- `/trekreview` Phase 4.5 runs the brief's `## Success Criteria` commands
through `lib/verification/criteria-runner.mjs --brief --evidence` and captures
the block as `sc_evidence_block`, pasted verbatim into the reviewer prompt in
Phase 5. The exit code does not stop the review — a failing criterion is
exactly what the review exists to find.
- `formatCriteriaEvidence` builds that block in code: one row per criterion with
the command, the exit code and the first output line. Chose a code-built block
over an orchestrator-written summary so the orchestrator cannot narrate a pass
that never happened.
- The rubric now judges the supplied result: `PASS` supports Full, `FAILED` /
`BLOCKED` is `Broken` with the exit code cited, and `NOT RUN` is the absence
of a measurement — never evidence in either direction.
- Phase 4.5 is skipped in `quick` mode: that mode does not launch the
conformance reviewer, so there is nobody to hand the result to.
Red first: seven tests in `tests/lib/criteria-runner.test.mjs` against a
committed brief fixture whose three criteria pass, fail, and are prose-only.
The two doc pins were verified red against the pre-fix files (rubric asked
"exists and passes"; no Phase 4.5; the block reached nobody).
Suite: 1117 (1115/0/2), up 9.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A trekplan's `## Verification` section is where the brief's success criteria
land. Phase 7 opened with "**Skip for trekplans.**", and only the multi-session
wave path (Phase 2.6 Step 3) ran master verification. A plan executed in ONE
session therefore reported `completed` without ever running the criteria it was
measured against — the executor's own belief was the only evidence.
The check now exists as code, not as an instruction:
- `lib/verification/criteria-runner.mjs` parses the criteria an artifact
DECLARES (a plan's `## Verification`, a brief's `## Success Criteria`), runs
each command, and returns a verdict built from exit codes. Fail-closed
throughout: a placeholder, a prose-only criterion, or an unavailable screen
is `unrunnable`/`blocked`, never `passed`. A plan with no `## Verification`
section exits 1 — a plan that promises no end-to-end check cannot be reported
as verified.
- Every command is screened through the plugin's own PreToolUse denylist
(`hooks/scripts/pre-bash-executor.mjs`) before it reaches a shell. Chose
invoking that hook over its documented stdin protocol rather than copying its
rules, because a command spawned from node never passes through the Bash tool
and so the hook cannot fire by itself — this keeps exactly one denylist.
- Phase 7 is now "Exit / verification check": session specs run the exit
condition, trekplans run the criteria runner. Phase 4's entry-condition skip
for trekplans stands — a plan carries no entry condition; the exit side is
not symmetrical.
- A failing criterion FELLS the run: `plan_verification.status != "passed"`
forbids `result: completed`. That is clause 2 of the stop-signal contract,
now enforced on the single-session path too.
Red first: `tests/lib/criteria-runner.test.mjs` (26 tests) against two committed
fixture plans, one of which declares a criterion that fails on purpose. The
doc pin in `tests/lib/doc-consistency.test.mjs` guards the wiring — a capability
no phase calls is the same defect wearing a lib/ file; verified red against the
pre-fix Phase 7 (skip present, runner absent, no fell-the-run clause).
Suite: 1108 (1106/0/2), up 27 from 1081.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
