Operator rule (2026-10-05): standard is Opus 5.5 at reasoning effort High. Four retrieval agents (architecture-mapper, dependency-tracer, git-historian, task-finder) pinned effort: medium in frontmatter, which overrides the session level, so every /trekplan exploration ran part of its swarm below the floor. They came from fdd3ad8 on the premise that unpinned agents inherit high; e4e37f9 corrected that (unpinned inherits the session; Opus 5.5 defaults to medium), so the reason for medium was already gone. All four now pin high; the other agents keep following the session (operator's choice). docs/profiles.md (one bullet for the eight pinned agents), CLAUDE.md and README.md updated; the model-default sentence stays. Dated decision records and CHANGELOG untouched.
Red: 4 of 5 new tests fail on the pre-fix files - effort floor over all agents (offenders: the four), the four pin exactly high, no live doc says retrieval runs at medium (hits docs/profiles.md:76, CLAUDE.md:85, README.md:806), profiles.md names the four as high. Known-positive control green on arrival.
Verify 153/153 (agent-frontmatter, doc-consistency). SC9 probe: npm test green under a scratch HOME with CLAUDE_PLUGIN_DATA empty, nothing under its plugin data dir. Suite 1393 (1391/0/2).
The change is repo-true, not installed-true, until a release.
Voyage order 20261005T104513Z-1250514346, plan step 5/5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The progress file /trekexecute's prose described did not pass the command's own validator: run status in-progress and no partial, no plan_version (required), step status running and passed. Measured on the 4 real progress files: 3 used passed, 2 failed on plan_version. The prose now speaks the validator's vocabulary, site by site: schema (plan_version = the value Phase 2 parsed; run status in_progress|completed|failed|stopped|partial), fresh-mode warning, both resume rules, step start in_progress, step finish completed, the escalation staging rule, the Phase 7.5 commit count (status=completed - left alone, every successful run would read as drift and be overridden to partial) and the final write (adds partial; a sandbox-blocked run writes stopped, report result blocked). Untouched on purpose: plan_verification.status, session-state / NEXT-SESSION-PROMPT status, stats and report result, the tolerant session-orders reader list, Phase 2.6's running state.
Valgt: tolerate a legacy passed on resume (one sentence after each resume rule) because Voyage is a public plugin and an in-flight run written under the old prose would otherwise re-execute finished steps. The validator gets no alias. Valgt: fresh mode also warns for stopped and partial - both are resumable, so they now get the --resume hint instead of a silent overwrite (intended behaviour change).
Red: 3 of 5 new tests fail on the pre-fix text - SC4 (documented progress file validates 0/0 per run status), SC5 (offenders at trekexecute.md 1011, 1055, 1062, 1164, 1169, 1326, 1352, 1489), the legacy-sentence count. Regression guard and known-positive control green on arrival.
Verify 125/125 (trekexecute-progress-schema, trekexecute, session-orders, criteria-runner, progress-validator). Suite 1388 (1386/0/2).
Voyage order 20261005T104513Z-1250514346, plan step 4/5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 9 said 'append to ${CLAUDE_PLUGIN_DATA}/trekexecute-stats.jsonl; if it is not set, skip silently' - the Bash tool env never carries it. It now runs stats-append.mjs trekexecute with the record on stdin, the /trekbrief Phase 7 shape, and reports a written:false. The gates_mode_flag_override record was prose with no writer; it now has an event-emit block (fixed literal payload: --gates is a boolean flag) and a documented shape {ts, event, known_event, payload:{flag, signal_effort}}. Phase 8's three event blocks spliced ${PLAN_PATH} (set in an earlier Bash call that does not survive) and ${WAVE_COUNT} (never assigned) into a double-quoted payload, so a literal run printed payload-not-json, and the decline reason was operator free text in a shell argument. Payloads are now single-quoted with {plan-path} / {wave-count} placeholders and the literal label hold-for-review; each call runs on its own and a written:false is printed and passed. docs/architecture.md and the jsonl-schemas rows (writer, measured line_refs) follow.
Red: 7 of 7 new tests fail on the pre-fix text (Phase 9 writer + no 'skip silently', Phase 9 record lands, no bare ${CLAUDE_PLUGIN_DATA}/...jsonl target, no shell expansion in --payload, the three Phase 8 blocks land known events when run as written in a fresh shell, the written:false sentence, the override writer and shape).
Verify 198/198 (stats-landing, main-merge-gate, gates-flag-coverage, profile-stats-fields, plugin-root-resolution, doc-consistency, no-claude-code-launch). Suite 1383 (1381/0/2).
Voyage order 20261005T104513Z-1250514346, plan step 3/5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
event-emit returned {written:false, reason:'CLAUDE_PLUGIN_DATA unset'} whenever the harness did not export the variable - the normal case for the Bash tool - so /trekexecute's Phase 8 events never landed. It now resolves the directory through stats-append's resolveStatsDir (one resolver; opts.path still wins, so intent-approval is unchanged). The CLI uses the realpath isMain() guard (the file:// guard was a silent no-op under a symlink or a path with a space) and exits 1 on every written:false - usage error, payload-not-json, failed write - instead of always 0. KNOWN_EVENTS gains main-merge-approved, main-merge-declined and gates_mode_flag_override, which /trekexecute emits by design. Header, intent-approval docstring (comment only), docs/operations.md and the README autonomy sentence (wrong state names, 'records each transition') updated.
Red: 10 of 20 tests fail on the pre-fix code - 7 new (CLI lands with CLAUDE_PLUGIN_DATA empty / absent, unwritable target exit 1, usage error exit 1, symlink, space path, known events) and 3 inverted contracts. Regression guards (green on arrival): never throws when unwritable; no second copy of the fallback directory name.
Inverted (the contract this changes, not deleted): emit with no CLAUDE_PLUGIN_DATA (silent skip) -> lands under a pinned HOME; never throws when unwritable -> a genuinely unwritable target (env {} now resolves to the real home); resolveStatsPath({}) === null -> the fallback path; malformed --payload exit 0 -> exit 1.
Readers checked: otel-export keys on 'event', yardstick counts only user_input, cache-analyzer and research-loop-cap name event-emit only in comments. Verify 268/268 (stats-event-emit, intent-approval, doc-consistency, otel-export-validators, yardstick). Suite 1376 (1374/0/2); non-hook lines in the real trekexecute-stats.jsonl 60 before and after.
Voyage order 20261005T104513Z-1250514346, plan step 2/5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/trekexecute writes run status stopped (dirty-tree stop, entry-condition stop, escalation) and step status blocked (Step 0 sandbox sentinel); the validator rejected the first and warned on the second. Both are real states lib/util/session-orders.mjs already reads (ENDED_BAD), so the validator learns them. The step-status list is exported as PROGRESS_STEP_STATUSES (was an inline literal) so tests check prose against it instead of keeping a copy. docs/HANDOVER-CONTRACTS.md Handover 5 rows updated.
Red: 3 of 4 new tests fail on the pre-fix code (stopped run status, PROGRESS_STEP_STATUSES export, blocked step status). Regression guard (green on arrival): every run status but completed stays resumable.
progress-validator 16/16. Suite 1368 (1366/0/2).
Voyage order 20261005T104513Z-1250514346, plan step 1/5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/trekreview of e7ac34b..3f073b7 (WARN, 1 MAJOR, finding
986c939f93808a274b5e4ecd2c5318fd422a3be4): baseDir's accepting branch
was never exercised, though every production /trekbrief record carries
project_dir. One unit test (attributeOrder from a non-git parent with
projectDir inside it) and one end-to-end landing test (the Phase 7 line
with a record carrying project_dir).
Both pass on HEAD and both fail on the always-cwd mutant (measured,
then reverted). Suite 1364 (1362/0/2).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/trekbrief Phase 7 says the writer adds order_id / order_claimed_at /
order_reason (never typed by the model) and that the yardstick joins use on
order_id; the fixture row lists them in `fields`, EXPORT_DENYLIST names them,
and its stale line_ref (657-672) now reads 961-976; the allowlist header
names them as dropped; docs/architecture.md Stats explains them.
/trekbrief --approve now runs Phase 7 after a successful stamp, with fixed
values (mode "approve", 0 turns, brief_quality/research_topics/auto_research
from the brief's frontmatter, auto_result "manual", intent_approved true).
Valgt fordi PM 04.10 (point 3): fix the code, not the criterion - the
approval must leave the trekbrief record. Measured hole: okf-wiki v1-k has
brief-approved 04.10 17:12Z and trekplan 17:29Z, but only a 03.10 trekbrief
record with intent_approved false. CLAUDE.md, README.md and
docs/command-modes.md say so. The manual path is unchanged.
npm test 1309 -> 1362 (1360 pass / 0 fail / 2 skip).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
order_id / order_claimed_at / order_reason on the five surfaces: (1) record
format = the writer (pinned in stats-landing), (2) Phase 7 prose with "the
writer adds", (3) the field-allowlist header names them as dropped and
TREKBRIEF_ALLOWED leaves them out, (4) the trekbrief-stats fixture row's
`fields` column + EXPORT_DENYLIST, (5) docs/architecture.md Stats. Plus: the
row's line_ref must contain the Phase 7 stats-append line (it said 657-672,
the line is at 958); /trekbrief --approve reaches Phase 7 with mode "approve"
(PM 04.10, point 3) and CLAUDE.md, README.md, docs/command-modes.md say so.
Red: 7 doc pins fail. Green on arrival: the yardstick-header pin (Step 6
already rewrote the header) and the otel DROP guard (the fields were never
allowlisted).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
measureUse({ queue, runs, cutoff, countingSince }): runs come from
orderRuns(stats) - trekbrief-stats.jsonl only. YES = a trekbrief record names
the order's id under one slug that names no other order; declared NO = a
new-work order carrying <!-- arbeidsform: ordre --> on its own line; no
record or a disputed record = NOT MEASURED, never a no (PM 04.10, point 1).
countingSince null -> NOT COUNTABLE, every order not measured; the window
starts at the latest of cutoff, MARKER_SINCE and the counting start.
facts.ordersNamedByARecord is the known-positive diagnostic (SC8).
Existing use tests get countingSince = MARKER_SINCE (their window unchanged)
and records instead of slugs; their denominators now read YES + declared NO.
Replaced: "a new-work order without a slug leaves the numerator NOT
MEASURED" - the slug join it guarded is gone. New exports are read through a
namespace import, so the trust/interrupts/mutants tests still run.
Red: 24 of 43 fail (every use test, the two measureAll tests, the symlink
CLI test); 19 pass, none of them new.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stats-landing gains a baseEnv() (process.env minus CLAUDE_COORD_DIR) at every
spawn - the real voyage mailbox holds a claimed order while the suite runs -
and runLine takes a cwd and extra env. New: Phase 7's own line, run in a temp
git repo against a temp coord dir, must land order_id / order_claimed_at /
order_reason (one claimed -> the id; zero -> keys present, null, none-claimed;
two -> several-claimed:2; model-supplied order_* keys replaced; no coord ->
coord-missing). The CLI must also write when run through a symlink or from a
path with a space.
Red on behaviour: 7 of 8 new tests fail against today's writer (no order keys;
the file:// guard is a silent no-op under a symlink and under a space -
measured in a scratch repro 04.10). Guard green on arrival: a trekplan record
gains no order keys.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
20 tests for lib/stats/claimed-order.mjs (new): exactly one claimed order is
an attribution (stem id + the .claim sidecar's claimed-at); zero, several
(an unparseable file still counts), a to: naming another repo, a missing
coord dir or no git repo is null plus a reason - never a guess. Mailboxes
mirror coord-order-claim.sh (basename of the toplevel) plus the worktree's
main repo; _* is reserved; an inherited GIT_DIR and a project_dir outside
cwd cannot steer it.
Red: the module does not exist (20 of 20 fail, each on its own - the module
is loaded per test). Against a null stub in a scratch copy, 18 fail on
behaviour; 2 are guards green on arrival (13 _-prefixed mailbox never read,
14 the recorder only reads).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured 2026-10-03: the two new-work orders on the real queue carry no `slug:`
(0 of 690 orders do), so "0 of 2" was forced by the missing join key, not by
work going around Voyage. An absence without a possible match is "not
measured", never 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The use number read `work-class` from the order frontmatter; 0 of 690 orders
carry it, so the denominator was always empty. PM rule 02.10: the order
marker `<!-- loesningsvalg: plan|prove -->` is new work, `ikke-nytt` is not,
counted from 2026-09-28 (enforced by pre-order-loesningsvalg.sh since then).
Unmarked orders and orders before the start are "not measured", never a no.
Red against today's code: MARKER_SINCE is not exported, and with it stubbed in
a scratch copy all five new use tests fail on behaviour.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Verified 2026-10-02 against code.claude.com/docs/en/model-config and
/sub-agents: opus -> Opus 5.5 and sonnet -> Sonnet 5.5 (both default
effort medium), fable -> Fable 5.1 (default high), and a subagent
without effort: inherits the session level. Seven tracked files still
say Opus 4.8 / Sonnet 4.6 / Fable 5. History keeps its names; a
known-positive control proves the query can find them there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lifted target (operator, 2026-09-27; PM order 44111111): Voyage + the
operator's way of working level with or ahead of related frameworks in all
14 areas, and every Voyage part a documented Claude Code primitive covers
removed or with a measured reason to stay.
scripts/comparison-gate.mjs reads both denominators from their source, never
from the table: the areas from the first column of the PM report's table,
the parts from git ls-files (one part per eval case). It fells the table on
missing evidence, never on a verdict: a cited file must exist inside its
root and hold the cited lines, a pinned clone must sit at its pin, a web
citation needs a verbatim quote, an absence needs zero hits and a
known-positive control. Green also requires an independent reader. The
table itself is local-only (docs/comparison.local.json).
Measured today on 7439dac: RED, 0 evidence problems; 2 of 14 areas against
all five references (Superpowers 11, Pocock 14, agent-skills 11, Cherny 10,
playbook 2); tidy 5 of 64 covered parts resolved.
23 tests in tests/scripts/comparison-gate.test.mjs; six mutants felled (no
evidence check, no line check, no reader check, denominator from the table,
no tidy completeness, no pin check). Suite 1278 -> 1301 (1299/0/2).
[skip-docs]: an operator measurement instrument like scripts/yardstick.mjs,
whose table is local-only - no user-facing behaviour changes, so README and
CLAUDE.md have nothing true to say about it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Bash tool runs the user's shell; on macOS that is zsh, whose NOMATCH
option fails an unmatched session-* glob with exit 1. A first run has no
worktrees yet, so Check 4 stops every first multi-session run and every
re-run after a declined main-merge gate. Red under /bin/zsh, green under
/bin/bash.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 3 of the review of the session-order flow:
- the orders dir is cleared only under the main-merge gate's confirm
(wave path, next-wave none, Step 4 passed); Step 6 keeps it, and a
declined gate keeps the merged-waves record
- the gate text is true on the wave path: the waves are already merged
on the current branch, nothing is finalized, and re-running the same
command (not --resume) returns to the gate; the single-session
--resume statement stays
- the report footer names the multi-session way to continue
- wave.json stores the branch the wave was prepared on, and Step 3's
status call (executed from the doc in a temp repo) refuses another
branch, keeping wave.json
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One invariant closes the data-loss class: Phase 2.55, 2.6 and 7.6 remove
worktrees only with `git worktree remove` without force and branches only
with `git branch -d`. Whatever git refuses (a worktree with changes, an
unmerged or checked-out branch) is kept, reported with what the user can
do, and never retried with force; every unmerged trek/{slug}/* branch
survives every failure path. Step 1 stops on a branch or worktree left
by an earlier attempt instead of deleting it. After a merged wave, a
progress file is dropped from its worktree only when untracked and
byte-identical to its archived copy, so the happy path still cleans up.
B1: Step 3 refuses to merge into a dirty main checkout, and tells a
refused merge (no MERGE_HEAD: stop, keep wave.json) from a real conflict
(abort, Step 5 keeps the branches).
B2: Phase 2.5, Check 4 and Step 0 take the main checkout from the first
`git worktree list` entry and refuse to run from a linked worktree.
B3: wave.json and merged-waves.json carry the plan's sha256 and refuse
another plan or version, naming the run that holds the orders dir and
the clear command; Step 0 stops on that. Step 6 clears the orders dir
when every wave is merged AND Step 4 passed, no longer keyed on the
Phase 8 label the wave path cannot reach.
Also: every session-orders.mjs call runs from the main checkout with a
main-relative plan path; Check 1 no longer counts .claude/trekplan-sessions/
as a dirty tree (it blocked every re-run after a failed wave); a stuck
wave has a documented non-force way out; Hard rule 12 states the
invariant. The test file only narrows one call-site filter that also
matched Step 0's plugin-root existence check.
npm test: 1273 tests, 1271 pass, 0 fail, 2 skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 2 of the review of the session-order flow. One invariant closes the
class: Phase 2.55, 2.6 and 7.6 remove worktrees only without force and
branches only with -d; a refusal keeps the work and is reported.
These tests execute the bash blocks of commands/trekexecute.md in
throwaway repos with linked worktrees:
- Check 4 and Step 5 remove a clean worktree and a merged branch, keep an
unmerged branch and a worktree with untracked files, and list them
- Step 1 stops on a branch or worktree left by an earlier attempt
- Step 3 stops on a dirty main checkout (B1) and tells a refused merge
(no MERGE_HEAD) from a real conflict; after a merged wave it records
the wave and drops only identical, untracked, archived progress copies
- Phase 2.5, Check 4 and Step 0 resolve the main checkout and refuse a
linked worktree (B2); Step 0 derives a main-relative plan path
- Check 1 does not count .claude/trekplan-sessions/ as a dirty tree
- wave.json and merged-waves.json carry the plan sha256 and refuse
another plan or version, naming the run and the clear command (B3);
Step 6, not Phase 8, clears on every wave merged + Step 4 passed
One test is a structural guard over the text (no force flag in those
phases); the executed blocks pin the behavior.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Operator decision 2026-10-02: keep the test set that a person runs
through Anthropic's plugin eval tool. Voyage starts nothing; the gate
counts that entry exactly, and the binding rule and the gate header now
say so instead of claiming historical mentions only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An independent review of the 6.0 session-order flow found seven defects.
These tests pin the fixed behavior:
- status classifies each session as completed, running (no file, pending,
in-progress, in_progress, or any unknown status) or ended-bad (failed,
stopped, partial, blocked, unreadable), keeping the raw status; a failed
session next to a running one reports running, and Step 3 must branch on
running before any failure cleanup
- archive copies each progress file into orders/ before worktrees go
- a merged-waves record drives next-wave, survives Step 5, and clear
resets it (refused while wave.json exists, and for another plan)
- the Phase 2.5 guard is anchored at the git toplevel; Check 4's guard is
executed from a subdirectory and must refuse while wave.json exists
- Phase 7.6 names no deleted worktree and promises no unwired re-run
- Hard rule 12 carries the awaiting-sessions exception
- progress paths are asserted literally; the old self-referential check
stayed green under a basename(planPath) mutation, the new one goes red
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The four blocks in trekexecute (x2), trekresearch and trekreview read
${CLAUDE_PLUGIN_ROOT:-}, which Claude Code never substitutes (it replaces
the literal ${CLAUDE_PLUGIN_ROOT}), and the fallback returns the
version-less cache/<marketplace>/voyage directory, so the guard after it
exits 2 every time. Red on both conditions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Operator decision 27.09.2026: Voyage works the PM way, so nothing in it
starts Claude Code. Work that needs another session becomes an order plus
a starter line the user gives a session they open themselves; Voyage
never starts the session.
/trekexecute's multi-session path (Phase 2.6) now prepares one wave at a
time: worktree per session, then lib/util/session-orders.mjs writes one
order per session (worktree, steps, done-condition, starter) plus
orders/wave.json, and the run stops. Running the same command again reads
each session's progress file (status mode), merges sequentially with
push-before-cleanup as before, and prepares the next wave. Phase 2.5 skips
the destructive pre-flight while a wave is in flight. Phase 7.6 recovery is
handed over the same way. Hard rule 21: never start Claude Code.
Removed with it, because they existed only for launched children: the
ANTHROPIC_API_KEY billing check (and its dry-run line and Hard rule 1
exception), the hooks-fire version precheck, the safety preamble, the
scoped-tool/turn/budget flags, templates/headless-launch-template.md, the
decomposer's launch.sh (now orders.md), /trekplan's "launch", the README
billing and MCP tuning sections, and scripts/q3-cache-prefix-experiment.mjs
(an experiment that spawned the binary; its results stay in docs/).
/trekresearch's deep-research pre-gate no longer runs the binary for its
version; the post-hoc check was already the authoritative guard.
Chose orders + starters over in-session subagents because the order's done
criterion asks for exactly that, and because /trekexecute disallows the
Agent tool on purpose (Hard rule 10); reopening that is a separate change.
Gate: 54 -> 0 hit lines outside EXEMPT. npm test 1250 (1245 pass, 0 fail,
5 skipped: 3 are the retired-reference scan without its local list).
verify.sh 7/0. The fixture run (tests/lib/session-orders.test.mjs) writes
two orders and wave.json for wave 1 and leaves a trap binary untouched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node:test gate that scans every tracked file (the tree a release tag
publishes) for any way to start a Claude Code session or model run: the
CLI with a flag or prompt, `claude plugin eval`, spawn/exec of the binary,
the Agent SDK / claude-code package / GitHub Action, and the permission
bypass flags. Plugin management (install, validate, marketplace add)
starts no session and is not a hit.
The scanner is proven on a known-positive fixture that exercises every
pattern, and on look-alikes that must not hit. Historical mentions (class
c) stand only in EXEMPT, each with a reason and its exact hit count, so a
new hit in an exempt file fails as well. Two entries are the operator's
own eval run through Anthropic's tool; they are listed as their own class
so the decision stays visible.
Red on 57e620b: 54 hit lines in 14 files outside EXEMPT (a: 34 in 6 files,
b: 18 in 7 files, a descriptive comment: 2 in 1 file). EXEMPT holds 16
files and 46 hit lines, and matches exactly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The annotation surface's provenance is now described as the pattern it
follows (pencil toggle, intent buttons, popover form, structured export)
rather than by naming a repository that no longer exists. Touches the
plugin/package descriptions, README, docs/architecture.md, the v5.0.3
CHANGELOG entry, and two comments in scripts/annotate.mjs and its test.
Retired-reference check: 12 hits in 7 files -> 0 (content and path names).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node:test pin that scans every tracked text file and path for the terms
in a gitignored local list (tests/.excluded-terms.local.txt), so the terms
never live in tracked text. The scanner is validated against a planted
known-positive (content and path name), an empty scan fails closed, the
named EXEMPT list is empty and rot-checked, and without the local list the
scan is skipped with an explicit message.
Red on 0be152a with the list present: 12 hits in 7 files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on 687710a: a correct case-6 reply said "**FAIL**: ... did not pass
... The other checks passed." With lower case accepted, says-pass matches that
reply, so a wrong case-8 reply of the same shape would pass the case. The test
pins that a reply whose capitalised verdict is the opposite one is refused,
while the opposite word in lower case (a side clause) or negated is not.
Red on 687710a: 11 of 12 in tests/lib/proevesett.test.mjs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PM re-measurement of ea66a98 had case 8 pass 10 of 12. Both failures were
correct replies that gave the verdict only in lower case ("passed"), and
\bPASS(?:ED)?\b is case-sensitive. The test pins lower-case pass/passes/passed
and fail/fails/failed as verdicts, keeps "not passed", "doesn't pass", "failed"
refused by says-pass, and checks that each case's recorded reply does not
satisfy the other case's grader.
Red on ea66a98: 10 of 11 in tests/lib/proevesett.test.mjs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Known-positive for case 4, in a scratch copy (never this tree):
brief-validator.mjs with the BRIEF_V51_MISSING_SIGNALS check made
'if (false)'. The case still passed 3 of 3: the child read the rule in
the command prose, halted, named phase_signals, and even found the
planted 'if (false)'. So the halt alone does not show the validator
fired, and no grader checked what expected_outcome already promised
('brief-validator ... reports BRIEF_V51_MISSING_SIGNALS').
New test: case 4 needs a names-code grader reading the validator's JSON
code from the trace (same form as the intent case). Red: 1 of 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scaffold writes a current intent_approved_hash (sha256 over ## Intent +
## Goal) into the case-4 brief. Chose a literal marker over running
intent-approval.mjs --stamp in the scaffold because --stamp emits a
brief-approved record into the live stats the yardstick reads: every eval
run would count as an approval.
New grader no-intent-halt (regex, trace, not_contains
code\W{1,12}BRIEF_INTENT_): a halt at the intent gate fails the case
instead of passing it for the wrong gate.
says-fail / says-pass take FAILED / PASSED as the verdict word. v2's word
boundary dropped them. The one says-fail failure (PM, 1 of 7) is not
explained: its trace was deleted, and 0 of 13 kept replies used another
spelling than FAIL.
Case 4 alone, 10 runs: 10 of 10. In 3 of 10 the child ran the intent gate
first, passed it and went on to the brief-validator. That order halted the
case before. Before the fix, same day: 5 of 6.
Green: tests/lib/proevesett.test.mjs 9 of 9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Case 4 (plan-halts-without-phase-signals) measured on dc9b480, 6 runs:
5 of 6. The failing run called only intent-approval.mjs, halted on
BRIEF_INTENT_NOT_APPROVED, and so failed validator-ran and
halt-names-signals: the fixture has no intent marker, so the case can
measure the wrong gate. PM: 6 of 12 on dc9b480, 7 of 10 on 66e1fa1.
New tests: the case-4 fixture must pass intent-approval --check and fail
only BRIEF_V51_MISSING_SIGNALS; a not_contains grader must fail the case
on any BRIEF_INTENT_* code in the trace. says-fail / says-pass must take
FAILED / PASSED as the verdict word (v2's word boundary dropped them).
Red: 3 of 9 in tests/lib/proevesett.test.mjs fail, each for its own reason.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Offline structure tests for evals/ (the suite itself needs headless runs):
- no case reaches the steg 1 intent gate (PM: 0 traces touched it). The
new case's scaffold is in: its brief passes brief-validator --soft
(2.1 WITH phase_signals) and --check gives BRIEF_INTENT_NOT_APPROVED —
that test is green, so the fixture reaches the gate.
- review-requires-project: trekreview.md composes 'Error: --project <dir>
is required.' in prose (8/10 in the PM run, backticks broke the regex),
and its arg-parser line passes "$@", which the Bash tool never has.
- no-error-code misses REVIEW_WRONG_TYPE; PASS/FAIL graders are raw
substrings; no-write cannot see a write through Bash.
7 tests, 6 red.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
M4: the halt regex matched any **halt** in a 2500-char window, so the PM's
mutant (first **halt** → "continue anyway") survived. Two new tests pin the
sentence right after the gate command and the could-not-run halt. Mutant
run: old test ok, both new tests not ok.
M5: nothing guarded 2045432's allowlist line. New test: trekbrief keeps
intent_approved, drops task/project_dir. Mutant (line removed): not ok.
M3 (red): the header claims a reflow keeps the approval; joining lines is
STALE. Pins the honest wording + the STALE behaviour.
M7 (red): the trekreview exemption is not documented as a bypass of the
same trust class as self-stamping (header, HANDOVER-CONTRACTS, trekplan).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
M1: headings inside fenced code blocks and HTML comments are text, not
headings, and ## Intent / ## Goal must each appear exactly once. A second
section, an example section in a code block or comment, or a '## ' line in
a code block inside Intent no longer carries the old approval: check gives
INVALID (duplicate) or STALE, stamp refuses a duplicate. A plain brief
hashes exactly as before (pinned), so no stamped brief goes stale.
Re-run of the PM's 29 probes: vi-a, vi-b, vii-a, vii-b now stop; the rest
unchanged. The PM's p7c probe was a no-op (its replace string is not in the
base brief); vii-c is covered by its own test, red on 66e1fa1.
M2: /trekbrief --approve takes <project-dir | brief-file>. New
intent-approval.mjs --resolve decides: a directory → <dir>/brief.md and
/trekplan --project; a file → itself and /trekplan --brief. Phase 4h and the
stamp line use {BRIEF_PATH}; /trekplan's halt table names
/trekbrief --approve {brief_path}, which works on both brief modes, and so
does the gate's hint. HANDOVER-CONTRACTS' "open to every producer's brief"
is now true. Chose fixing the path over rewording the contract because two
real briefs (docs/*-brief.md) had no approval path at all.
Red ff760ed 9/29 → green 29/29. Suite 1219: 1217 pass / 0 fail / 2 skip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
M1 (PM probes vi-a, vi-b, vii-a, vii-b, vii-c): extractSection takes the first
## Intent and does not see code blocks or HTML comments, so an added, hidden
or cut-off section rides on the old approval. 5 red. vii-d (comment inside
Intent) and the plain-brief hash are pinned green so the fix cannot move them.
M2: --approve resolves only <dir>/brief.md; a brief planned with
--brief docs/x-brief.md has no approval path but a raw --stamp. 4 red.
The stamp helper now substitutes {BRIEF_PATH} as well as {PROJECT_DIR}.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 7 of /trekbrief and Phase 12 of /trekplan say "skip silently" when
CLAUDE_PLUGIN_DATA is unset, and the Bash tool env never carries it — the
same silent skip that left brief-approved at 0 records. Økt 2's countable
form (brief-approved + intent_approved: true + a trekplan record with the
same slug) cannot be counted until those records land where the yardstick
reads. 7 tests, 7 red: no stats-append line in either command.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Veikart steg 1, økt 1 av 2. The three holes measured in the grounds:
(1) ## Intent / ## Goal were only checked for existence, (2) no approval
marker existed, (3) brief-approved was emitted only on the auto path.
- lib/validators/intent-approval.mjs: --check / --stamp. The marker is
intent_approved_hash (sha256 over normalized ## Intent + ## Goal) +
intent_approved_at. Editing either section after approval → STALE.
- /trekplan Phase 1 (Read the brief, both --brief and --project): runs
--check and HALTS on BRIEF_INTENT_NOT_APPROVED / _STALE / _INVALID with
the remedy spelled out; an unrunnable check halts too.
- /trekbrief Phase 4h (before the Phase 5 fork, asked even in --quick):
shows Intent + Goal verbatim, AskUserQuestion Approve / Revise / Leave;
only "Approve" runs --stamp. --stamp emits brief-approved, so the manual
(default) path records it; the auto path's own emission is removed.
New mode /trekbrief --approve <project-dir> = Phase 4h alone.
- README, CLAUDE.md, command-modes, HANDOVER-CONTRACTS §Handover 1,
jsonl-schemas (trekbrief-stats gains intent_approved).
Valgt ingen brief_version-bump fordi skjemaendringen er rent additiv (to
valgfrie felt) og kravet sitter i /trekplan — enhver produsents brief kan
godkjennes via /trekbrief --approve uten produsentendring. Valgt eget
--check-kall i stedet for et flagg på brief-validator fordi --brief-stien
i dag ikke kjører validatoren i det hele tatt; et nytt validatorkall der
ville også stoppe på andre feil. Valgt fallback-datamappe = målestokkens
(plugins/data/voyage-…) fordi CLAUDE_PLUGIN_DATA er tom i Bash-miljøet og
event-emits egen fallback er stille skip — slik ble brief-approved 0 records.
What the marker does NOT prove (module header, command prose, contract):
same user, same machine — any session can stamp. A trace, not a signature.
Suite 1183 → 1201 (1199/0/2). Mutants M1–M5 (stale check off, check
always valid, gate line removed, stamp emits nothing, no normalization)
each fell ≥ 1 test. yardstick unchanged: RED, 1 of 3 countable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Veikart steg 1, økt 1 av 2. 18 tests, all failing today:
(a) a brief without an approval marker → the gate line in
commands/trekplan.md exits non-zero (today: no gate, planning proceeds);
(b) marker + ## Intent or ## Goal edited after approval → stale, stops;
(c) a valid marker passes (known negative), and edits outside the two
sections or whitespace-only reflow keep it valid;
(d) the stamp runs before the Phase 5 manual/auto fork and emits
brief-approved, so the manual (default) path records it too.
The gate and stamp lines are extracted from the command prose and
executed, so removing the line or stubbing the module fells the test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One command, `node scripts/yardstick.mjs`, prints use, trust and
interrupts. Each row gives countable YES/NO, the value, the denominator and
the source it was read from. A fourth row asks whether the end-state gate can
be fooled by a mutant that keeps the module's signature. Exit is non-zero
while any number is not countable, the mutant is not felled, or the verdict
over the first five deliveries is not judged. Nothing counts a delivery yet,
so today it is RED for the right reasons:
use NO no order in the queue carries work-class
trust YES current ALLOW streak 0 of 37 (BLOCK 14, WARN 12, ALLOW 11,
last 2026-09-05 BLOCK, ordered by ts, read as JSON)
interrupts NO 0 user_input records of ~141k lines in 5 files, 0 emitter
call sites outside lib/stats/ and tests/
mutants 0 of 1 signature-preserving mutants felled: on M9 (every export
returns {status:'PASSED'}, every named test only checks typeof)
the gate closes both D-03 and D-04
It only measures. The gate, the emitter and the order writer are untouched.
The mutant row is meant to stay red until the gate learns to fell M9.
Choices, and why:
- Use counts over the order queue (pending, archive/ and claimed/), because
an order is the unit of work. The denominator is orders dated on or after a
cutoff (default 2026-09-21) that carry `work-class: new`. They count as
having gone through Voyage when a Voyage run in the stats has the order's
`slug:`. Both fields are the contract the order writer must meet. No order
carries them today, so the number is NOT COUNTABLE, not 0 %.
- Mailboxes whose names start with a dot are mailboxes too. A shell `*`
glob skips them, which is why a hand count of `coord/*/orders` comes out
lower. The row names them so a re-measure can reconcile.
- M9 must be LIVE before its row can count. Its typeof tests have to pass on
the M9 tree, checked by a runner independent of the gate, so a broken
fixture can never read as "felled".
Also: tests/fixtures/red-first/02243c6-always-allow-shim.mjs, the shim that
makes the red-first claim of 02243c6 reproducible (6 of 55). The recipe is in
its header.
Red first: the new test file failed to load before scripts/yardstick.mjs
existed, and a dot-mailbox test was red before the census learned about
dot-mailboxes. Suite 1161 -> 1183 (1181/0/2), also green on a clean tree
exported from the index.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`origin` is a public mirror. Comments in tracked test files named the internal
review mechanism by its role rather than the measurement they came from; the
same class as the frozen file's overclaim fixed in 83d82f1. Replaced with what
they actually are: a re-measurement on a named commit, with the date.
The one remaining occurrence is the assertion that FORBIDS those phrases in
tests/fixtures/end-state-frozen.json - a guard has to spell out what it guards.
Suite 1161 (1159/0/2). Comment-only change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured 2026-09-18: nine ordinary shapes of a plan's `## Verification`
section parsed to zero criteria - an untagged fence, a ```text fence, a
markdown table, `## Verification (acceptance)`, `## Verification:`,
`### Verification`, an unclosed fence earlier in the document. Every one came
out as `0 of 0`, NOT OK, exit 1, and Phase 7 then forbade `result: completed`
without anyone being told that the FORMAT, not the code, was the problem.
"The section is empty" and "I cannot read this format" are different facts.
Three changes, one hole:
- The runner reports `NO_CRITERIA` with a source line (`plan.md:NN`) when the
section is there and nothing in it parsed, and names the two forms it does
read. Same for a brief's `## Success Criteria`, so the evidence block the
conformance reviewer gets says which of the two it is looking at rather than
showing an empty table.
- Phase 7 says it out loud instead of failing silently: report the source line
and the two forms, and say that the plan is what failed there, not the run.
- `/trekplan` now pins what it produces to what the runner reads: the heading
is exactly `## Verification`, the criteria are a bullet whose first
backticked span is the command or a shell-tagged fence, and the command must
be one the allowlist runs. A doc-consistency test holds the writer and the
reader together, so a runner that learns a new form must update the source.
Honest about the round trip: the two round-trip tests were GREEN on arrival -
the template already writes the bullet form the runner reads. What was missing
was not the format but the PIN: `/trekplan` mandated neither the heading string
nor the format, so a plan could satisfy the command's own instructions and
still parse to nothing. The tests now hold that.
Red first: 5 of the 7 new tests failed before the change (3 NO_CRITERIA, 2
doc-consistency); the 2 round-trip tests are guards, and said so above.
Suite 1161 (1159/0/2). Gate unchanged: defects 0 of 7, intact, exit 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The cosmetic close came back one level up. The probe reads the test file from
the same tree it measures, so a four-line file holding two EMPTY tests with the
two named names closed D-03 and D-04 on a tree where `lib/verification/` did
not exist at all - "defects 0 of 7, registry intact" (measured 2026-09-18).
First a two-line stub exporting the right symbols; then an empty test with the
right name. A name is always forgeable.
So the named test is now run twice. Once on the tree, as before - and once in a
sandbox where the module the condition declares in `stubs` is replaced by a
stub exporting the same names, all inert. If the test still passes there, it
binds the name and not the behaviour, and the condition THROWS: NOT FELLABLE,
counted open. A condition that declares no `stubs`, or names a module that is
not there, cannot fire either.
The sandbox is a symlink overlay: every entry of the tree is symlinked, and
only the test file and the stubbed module are materialised for real - Node
resolves an ESM import through the realpath, so a symlinked test file would
import the original module and never see the mutant. Nothing is ever written
inside the measured tree, and the only directory removed is the one this code
made under the system temp dir (pinned by a test).
M7 is now a permanent mutant beside M6, in two forms: the checkpoint's own
reproduction (unfixed tree + empty named tests) and the harder one (the real
module present, so the stub can be built and the empty test passes against it).
Both report `defects 2 of 7`. A positive control pins that D-03/D-04 still
CLOSE on the real tree, so "not closed" everywhere cannot read as a working
probe.
The frozen denominator moves a third time, deliberately, and its `why` no
longer claims authority it does not have: the second and third amendments were
maintenance decisions by the maintainer, not operator decisions, and the
tracked file now says exactly that.
Measured after:
real tree node scripts/end-state-gate.mjs -> defects 0 of 7, intact, exit 1
M6 (stub) 8d1669e + current gate/registry/frozen + lib/cosmetic/stub.mjs -> 2 of 7
M7 (empty) 8d1669e + current gate/registry/frozen + 3-line
tests/lib/criteria-runner.test.mjs with the two named tests -> 2 of 7
Red first: 4 of the new tests failed before the change. Suite 1154 (1152/0/2).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A denylist in front of /bin/sh is whack-a-mole. Measured 2026-09-18, end to
end through both screens: 5 of 11 named evasions ran with real effect - a
`command` prefix reached git, an escaped `rm` inside a shell fence deleted a
directory, `find -delete` deleted a file, `>|` and `tee` wrote outside the
working tree, a python one-liner deleted the whole tree - and 19 of 28 got
past the refusal list on its own. Every quoting, aliasing and indirection form
of the shell is another mole.
So the screen is now an ALLOWLIST. A criterion runs only when its first word
is a known test runner (npm test, npm run <script package.json declares>,
node --test, vitest, jest, pytest, python -m pytest, uv run pytest,
cargo test, go test, make test, bash <script under tests/>, a read-only git
subcommand) AND the command carries no shell operator and no newline.
Everything else is NOT RUN with the reason said out loud: never run, and never
reported as a failure either - an absent measurement is not a finding. That
also closes the smaller hole in the same file: a bare word a sentence merely
names (`whoami`, `login`, `package.json`) is no longer executed, because it is
not a runner.
REFUSED_BY_POLICY is gone with the list that produced it; a command outside
the allowlist is `unrunnable`, which in plan mode still fells the run and in
brief mode is reported to the reviewer as an absent measurement.
What the allowlist deliberately does NOT do, said in the file and in the
reviewer's rubric: it is not a sandbox. `npm test`, `npm run <script>` and
`make test` run whatever the repo's own package.json/Makefile says they run,
including a script that pushes - that is the repo's responsibility. And it
rejects honest commands too: an env prefix, a project's own binary, anything
piped. A check that needs one of those is declared through
`bash tests/<script>.sh`, the documented way in.
Red first: 6 of the new tests fail against the previous runner (measured with
an always-allow shim so the module still loads), including the end-to-end one
where the canary directory was deleted and files were written outside the
tree. The fixtures move from `true`/`false` to two allowlisted shell fixtures,
because `false` is no longer a runner - the fail case must still be a real
non-zero exit, not an unrun criterion.
Suite 1148 (1146/0/2).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/trekreview passed --cwd and trekexecute Phase 7 did not, so the same criterion
could resolve two ways in the two phases (PM checkpoint 2026-09-18, MINOR).
It stopped being cosmetic with the refusal list: --cwd is the boundary a write
is measured against, so leaving it unset silently moves that boundary to
whatever the process cwd happens to be. Both phases now resolve
CRITERIA_CWD="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
- one form, with a fallback, so a repo-less checkout falls back instead of
passing an empty --cwd. Pinned in tests/lib/doc-consistency.test.mjs.
Red first: the new pin failed on both phases before this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/trekreview now runs the commands a BRIEF declares, and a brief is an artifact
that can arrive from outside the repo. Measured 2026-09-18 on 6cafb4c: the
executor denylist stopped a download piped into a shell, but the remote-writing
git subcommand and a recursive delete of a path both RAN. The denylist screens
catastrophe (root deletion, fork bombs, mkfs); it was never meant to screen an
artifact under review.
A second screen, in the runner and ahead of the denylist, refuses four classes:
- a remote-writing git subcommand. The subcommand is found by walking git's own
options (`-C`, `-c`, `--git-dir`, ... take a value), so `git status` and
`git log` still run and `git -C sub push` does not.
- a recursive delete: any `rm` carrying `-r`/`-rf`/`--recursive`. A plain
`rm build/artifact.txt` still runs.
- a download piped straight into a shell (also caught by the denylist; pinned
here so the runner does not depend on another file for it).
- a write outside the working tree. `/dev/null`-class devices are fine, and so
is anything under the working tree; `~/...`, an absolute path elsewhere, and
a target carrying an unexpanded `$VAR` are refused - the runner cannot know
where a variable points, and guessing is how a screen stops screening.
A refusal is its own outcome, REFUSED_BY_POLICY: the command never reaches a
shell, and `summary.ok` is false in both plan and brief mode. For the reviewer,
REFUSED is like NOT RUN - the absence of a measurement, never on its own a
finding - and the rubric and the evidence block both say so.
Chosen deliberately, and it is stricter than today's habit: writing scratch to
/tmp is refused too. The repo's own example plan does `> /tmp/out`. Verification
output belongs in the working tree; exempting the whole system temp dir would
have made the rule unstatable, since a working tree created under /tmp then
contains its own escape hatch.
NOT covered, stated rather than implied:
- other writing git subcommands (tag, remote, config, gc) - only push is listed
- writes through a wrapper: `sh -c '...'`, `xargs`, `find -exec`, a Makefile
target, a script the criterion invokes. The screen reads the command it is
given, not what that command goes on to do.
- `>` inside a quoted string reads as a redirect, so a criterion echoing a
literal `>` is refused. Fail-closed, on purpose.
- the whole surface still runs with the invoking process's permissions; this is
a refusal list, not a sandbox.
The denylist-layer test now uses a stand-in command with a screen double: the
refusal list catches a recursive delete first, so naming one there would have
stopped exercising the denylist layer at all.
Red first: the 6 new tests failed before this change (`refuseCommand` did not
exist), and the fixture brief's four writes ran.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured 2026-09-18 (PM checkpoint on 6cafb4c, mutant M6): a two-line file
export function runPlanVerification() { return { ok: true }; }
export function formatCriteriaEvidence() { return ''; }
dropped into an otherwise UNFIXED tree closed both D-03 and D-04, and the gate
reported "defects 0 of 7, registry integrity: intact". The same checkpoint felled
5 of 5 mutants with the SUITE and 0 of 5 with the GATE. Both probes called
themselves `behaviour` while asking a grep whether an export name was present -
a phrase probe pointed at code.
A check condition may now name a test instead of a pattern:
{ "test": "tests/lib/criteria-runner.test.mjs", "name": "<exact test name>",
"expect": "fails" }
The gate spawns the test runner on that one test and reads its TAP line.
`expect: "fails"` holds - the defect stays open - while the test does not pass.
A missing file, a name that matches nothing, a runner that will not start: all
throw, which counts as NOT FELLABLE and therefore open. A check that cannot fire
is never "fixed".
The child's `NODE_TEST_*` env is stripped. The gate usually runs UNDER the test
runner, and an inherited `NODE_TEST_CONTEXT` makes the grandchild report over
the parent's IPC channel instead of stdout - the TAP line would never arrive and
the probe would silently stop felling anything.
M6 is now a permanent mutant in the gate's own test, so the cosmetic close
cannot come back. Verified end to end against a real stub tree:
git archive 8d1669e | tar -x -C $TMP
cp scripts/end-state-gate.mjs scripts/end-state-registry.json $TMP/scripts/
cp tests/fixtures/end-state-frozen.json $TMP/tests/fixtures/; cp STATE.md $TMP/
printf 'export function runPlanVerification…' > $TMP/lib/cosmetic/stub.mjs
node scripts/end-state-gate.mjs --root $TMP
-> defects 2 of 7, both NOT FELLABLE, exit 1
The gate also now states the limit out loud, as its own line above the table:
a behaviour probe proves the capability WORKS; that Phase 7 and Phase 4.5 CALL
it is pinned by TEXT in doc-consistency, not proven deterministically - real
proof is a headless plugin-eval run against a fixture plan (week 40).
FROZEN DENOMINATOR AMENDED, second time in one day, by the work order that
carries this change: D-03 and D-04 signatures moved because their checks
changed on purpose. tests/fixtures/end-state-frozen.json and the FROZEN literal
in tests/scripts/end-state-gate.test.mjs are updated together, both dated, both
saying why. No entry was removed.
Gate after this change: defects 0 of 7, registry integrity intact, exit 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"The first backtick span is the command" is right for a plan, whose template
puts the command first, and wrong for a brief, whose criterion usually opens
by NAMING the thing under discussion. Measured 2026-09-18 on the repo's own
example brief: 5 of 6 criteria FAILED, 3 of them parse artifacts - `--verbose`
run as a command gave exit 2 ("invalid option"), `tests/` gave exit 126 ("is a
directory"). The rubric reads a FAILED result as decisive, so each one became
a BROKEN_SUCCESS_CRITERION BLOCKER about prose.
looksLikeCommand() screens the span by SHAPE only - no filesystem lookup, so a
span parses the same everywhere. Refused: a leading flag, a directory, a token
carrying quotes/braces/prose, and a lone relative path with a slash (an
explicit ./, ../, / or ~/ still runs, as do env-var prefixes). A refused span
is `unrunnable` with reason `not-a-command` - its own outcome, never FAILED,
and it never reaches a shell.
It deliberately does NOT scan on to a later span. "The first span that LOOKS
like a command" invents commands out of prose: in that same example brief it
would have run `whoami` and `login`, two real binaries a sentence happens to
name. An absent measurement is honest; a guessed one is not.
The shape check applies to prose spans only. Inside a shell-tagged fence the
author has already declared shell, so `[ -f x ] || exit 1` still runs.
The rubric follows: a NOT RUN result is never on its own a finding. The
Partial row now describes half-built DELIVERED CODE, and the reviewer gets a
table of the three reason strings - no-command, placeholder, not-a-command -
with what each says about the sentence rather than about the code.
Not covered, stated for the record: a multi-token span whose first token is a
non-executable file (`tests/golden/login.stdout --check`) still runs, and a
criterion whose command is real but whose binary is absent still reports the
shell's exit 127 - that is a true measurement of a missing binary, not a
parse artifact.
Red first: 4 runner tests + 1 doc-consistency pin failed before this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runner read bullet lines only. The repo's own example plan writes its
whole acceptance run as a fenced bash block, so `## Verification` parsed to
ZERO criteria, the runner exited 1, and Phase 7 forbade `result: completed`
- a correct plan felled every single-session run. Measured 2026-09-18 on
6cafb4c: none of the repo's plan artifacts exited 0.
Fences are now read twice over, for two opposite reasons:
- a `## ` heading INSIDE a fence is quoted text and no longer opens a
section. examples/02-real-cli/REGENERATED.md is a report that quotes a
plan outline inside one fence; it used to yield an empty section that
read as "0 of 0", and now yields the honest NO_VERIFICATION_SECTION.
- a shell-tagged fence inside the section holds the commands. The tag list
is closed (bash/sh/shell/zsh/console/shell-session): an untagged fence is
more often expected OUTPUT than input, and inventing a criterion from
output is the failure this file exists to prevent.
Blank and comment-only lines inside the block declare nothing. A `$`/`>`
console prompt is stripped; a `#` root prompt is NOT, because it cannot be
told from a comment and running a comment is the worse mistake.
Measured after the fix (parse only - one example names a fictional CLI):
examples/01 6 criteria, plan-template 2 (both placeholders, correctly NOT
RUN), the two runner fixtures 2 each, plan-run-C 1, REGENERATED.md 0 with
NO_VERIFICATION_SECTION.
Divergence from the order's premise, stated for the record: it said 2 of 3
example plans write `## Verification` as a fenced bash block. Ground truth
is 1 of 3 - REGENERATED.md has no section of its own at all - and the
example plan's block holds 6 command lines, not 5.
Red first: 4 of the 6 new tests failed before this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The operator decided on 2026-09-18 that the two checks must stop measuring a
phrase and start measuring behaviour. The reason is the gate's own caveat: a
phrase probe closes on rewording, so "Phase 7 no longer says 'Skip for
trekplans'" and "the rubric no longer says 'exists and passes'" could both have
been satisfied by an edit that changed no behaviour at all. Both entries now
point at code a test exercises:
D-03 lib/**/*.mjs exports runPlanVerification
D-04 lib/**/*.mjs exports formatCriteriaEvidence
Verified open against the pre-fix tree (8d1669e) and closed against this one —
a check that cannot be seen to fire is not evidence.
What the probes can and cannot prove is now stated by the gate itself rather
than left to the reader. `PROBE_NOTES` carries one note per kind, the detail
line prints a note for every kind PRESENT in the row (and none for a kind no
entry declares), and the open/closed tags name the kind instead of labelling
`phrase` alone and leaving every other kind unlabelled. A behaviour probe
proves the capability exists; it does not prove a prose phase calls it — the
wiring is pinned by tests/lib/doc-consistency.test.mjs, and the gate says so.
THE DENOMINATOR MOVED ON PURPOSE. `tests/fixtures/end-state-frozen.json` and
the `FROZEN` literal in `tests/scripts/end-state-gate.test.mjs` carry new
signatures for D-03 and D-04, amended together and dated in both places. That
is the freeze mechanism working, not a bypass: the integrity check went
VIOLATED the moment the checks changed and stayed red until the amendment was
written down. The list may change — only as a decision said out loud. Nothing
else about the denominator moved: seven defect ids, three experiment ids, and
the agents/decisions/freeze configuration are unchanged, and no entry was
removed.
Gate after: defects 0 of 7, registry intact, still RED and exit 1 on the other
four rows — which is correct.
Suite: 1120 (1118/0/2), up 3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rubric required `brief-conformance-reviewer` to classify a Success
Criterion as Full only when "its verification command/test exists and passes".
Its tools are `Read`, `Glob`, `Grep`. It cannot run anything, so "passes" was
either guessed from the command's mere existence or quietly downgraded to
"exists" — a BLOCKER-tier rule key resting on an impression.
The reviewer stays read-only — a reviewer that executes the code it reviews is
not an independent reviewer. The command does the running instead:
- `/trekreview` Phase 4.5 runs the brief's `## Success Criteria` commands
through `lib/verification/criteria-runner.mjs --brief --evidence` and captures
the block as `sc_evidence_block`, pasted verbatim into the reviewer prompt in
Phase 5. The exit code does not stop the review — a failing criterion is
exactly what the review exists to find.
- `formatCriteriaEvidence` builds that block in code: one row per criterion with
the command, the exit code and the first output line. Chose a code-built block
over an orchestrator-written summary so the orchestrator cannot narrate a pass
that never happened.
- The rubric now judges the supplied result: `PASS` supports Full, `FAILED` /
`BLOCKED` is `Broken` with the exit code cited, and `NOT RUN` is the absence
of a measurement — never evidence in either direction.
- Phase 4.5 is skipped in `quick` mode: that mode does not launch the
conformance reviewer, so there is nobody to hand the result to.
Red first: seven tests in `tests/lib/criteria-runner.test.mjs` against a
committed brief fixture whose three criteria pass, fail, and are prose-only.
The two doc pins were verified red against the pre-fix files (rubric asked
"exists and passes"; no Phase 4.5; the block reached nobody).
Suite: 1117 (1115/0/2), up 9.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
