The cosmetic close came back one level up. The probe reads the test file from the same tree it measures, so a four-line file holding two EMPTY tests with the two named names closed D-03 and D-04 on a tree where `lib/verification/` did not exist at all - "defects 0 of 7, registry intact" (measured 2026-09-18). First a two-line stub exporting the right symbols; then an empty test with the right name. A name is always forgeable. So the named test is now run twice. Once on the tree, as before - and once in a sandbox where the module the condition declares in `stubs` is replaced by a stub exporting the same names, all inert. If the test still passes there, it binds the name and not the behaviour, and the condition THROWS: NOT FELLABLE, counted open. A condition that declares no `stubs`, or names a module that is not there, cannot fire either. The sandbox is a symlink overlay: every entry of the tree is symlinked, and only the test file and the stubbed module are materialised for real - Node resolves an ESM import through the realpath, so a symlinked test file would import the original module and never see the mutant. Nothing is ever written inside the measured tree, and the only directory removed is the one this code made under the system temp dir (pinned by a test). M7 is now a permanent mutant beside M6, in two forms: the checkpoint's own reproduction (unfixed tree + empty named tests) and the harder one (the real module present, so the stub can be built and the empty test passes against it). Both report `defects 2 of 7`. A positive control pins that D-03/D-04 still CLOSE on the real tree, so "not closed" everywhere cannot read as a working probe. The frozen denominator moves a third time, deliberately, and its `why` no longer claims authority it does not have: the second and third amendments were maintenance decisions by the maintainer, not operator decisions, and the tracked file now says exactly that. Measured after: real tree node scripts/end-state-gate.mjs -> defects 0 of 7, intact, exit 1 M6 (stub)8d1669e+ current gate/registry/frozen + lib/cosmetic/stub.mjs -> 2 of 7 M7 (empty)8d1669e+ current gate/registry/frozen + 3-line tests/lib/criteria-runner.test.mjs with the two named tests -> 2 of 7 Red first: 4 of the new tests failed before the change. Suite 1154 (1152/0/2). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
182 lines
7.1 KiB
JSON
182 lines
7.1 KiB
JSON
{
|
|
"defects": [
|
|
{
|
|
"id": "D-01",
|
|
"summary": "commands/trekplan.md still references TeamCreate/TeamDelete (allowed-tools and the execute-with-team path); both tools were removed in Claude Code 2.1.178, so that path always falls back to sequential",
|
|
"closesWhen": "trekplan.md no longer names TeamCreate or TeamDelete",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekplan.md",
|
|
"pattern": "\\bTeam(Create|Delete)\\b",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-02",
|
|
"summary": "commands/trekresearch.md tells the swarm engine to run the `### Bridge agent` block, but that heading was removed together with gemini-bridge",
|
|
"closesWhen": "the dangling reference is gone (or the heading exists again)",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekresearch.md",
|
|
"pattern": "`### Bridge agent`",
|
|
"expect": "match"
|
|
},
|
|
{
|
|
"path": "commands/trekresearch.md",
|
|
"pattern": "^### Bridge agent",
|
|
"flags": "m",
|
|
"expect": "no-match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-03",
|
|
"summary": "commands/trekexecute.md never runs a trekplan's `## Verification` (where the brief's success criteria land) on the single-session path: Phase 7 says 'Skip for trekplans', and only the multi-session wave path runs master verification",
|
|
"closesWhen": "a named test PROVES the single-session path fells a run on a criterion that does not hold (behaviour probe: the gate RUNS tests/lib/criteria-runner.test.mjs, so a stub that only exports the symbol does not close it, and the named test must itself FELL a stub of lib/verification/criteria-runner.mjs, so an EMPTY test with the right name does not close it either; that Phase 7 CALLS it is pinned by tests/lib/doc-consistency.test.mjs, not by this probe)",
|
|
"probe": "behaviour",
|
|
"check": [
|
|
{
|
|
"test": "tests/lib/criteria-runner.test.mjs",
|
|
"name": "runPlanVerification: a plan whose success criterion FAILS fells the run",
|
|
"expect": "fails",
|
|
"stubs": [
|
|
"lib/verification/criteria-runner.mjs"
|
|
]
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-04",
|
|
"summary": "agents/brief-conformance-reviewer.md must judge whether a success criterion's verification command 'exists and passes', but its tools are Read/Glob/Grep, so it cannot run anything",
|
|
"closesWhen": "a named test PROVES the evidence block the conformance reviewer judges is BUILT IN CODE from real exit codes (behaviour probe: the gate RUNS tests/lib/criteria-runner.test.mjs, so a stub that only exports the symbol does not close it, and the named test must itself FELL a stub of lib/verification/criteria-runner.mjs, so an EMPTY test with the right name does not close it either; that Phase 4.5 CALLS it is pinned by tests/lib/doc-consistency.test.mjs, not by this probe)",
|
|
"probe": "behaviour",
|
|
"check": [
|
|
{
|
|
"test": "tests/lib/criteria-runner.test.mjs",
|
|
"name": "formatCriteriaEvidence: one row per criterion, with command and exit code",
|
|
"expect": "fails",
|
|
"stubs": [
|
|
"lib/verification/criteria-runner.mjs"
|
|
]
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-05",
|
|
"summary": "commands/trekplan.md allowed-tools lists TaskCreate/TaskUpdate, which Claude Code 2.1.233 no longer offers on Opus 4.8 / Sonnet 5 / Fable 5 and newer by default (runtime effect not measured)",
|
|
"closesWhen": "trekplan.md allowed-tools no longer lists TaskCreate or TaskUpdate",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekplan.md",
|
|
"pattern": "^allowed-tools:.*\\bTask(Create|Update)\\b",
|
|
"flags": "m",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-06",
|
|
"summary": "lib/review/gold-scorer.mjs contains a literal NUL byte, so git treats the file as binary and hides it from diffs and --numstat",
|
|
"closesWhen": "the file contains no NUL byte (write the separator as an escape sequence)",
|
|
"probe": "byte",
|
|
"check": [
|
|
{
|
|
"path": "lib/review/gold-scorer.mjs",
|
|
"pattern": "\\u0000",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-07",
|
|
"summary": "CLAUDE.md, README.md and commands/trekresearch.md claim the /trek* orchestrators run on opus (README even ships a sed recipe for `model: opus` lines no command has), but since v5.9.0 no command pins model: and the orchestrator follows the session model",
|
|
"closesWhen": "none of the three files claims an opus orchestrator any more, or the commands pin opus again",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": [
|
|
"CLAUDE.md",
|
|
"README.md",
|
|
"commands/trekresearch.md"
|
|
],
|
|
"pattern": "(^\\| `/trek[a-z]+` \\|.*\\| opus \\|\\s*$)|(\\^model: opus\\$)|(default for `/trekbrief`[\\s\\S]{0,80}?is `opus`)|(orchestrator runs on Opus)",
|
|
"flags": "m",
|
|
"expect": "match"
|
|
},
|
|
{
|
|
"path": "commands/*.md",
|
|
"pattern": "^model:",
|
|
"flags": "m",
|
|
"expect": "no-match"
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"experiments": [
|
|
{
|
|
"id": "E-01",
|
|
"summary": "STORM dimension discovery + bounded research loop ships default-off behind VOYAGE_STORM_ENABLED; adoption is gated on a pre-registered measurement that has not run",
|
|
"closesWhen": "the env gate is gone from the pipeline surface: adopted as default, or the loop is removed",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": [
|
|
"commands/*.md",
|
|
"agents/*.md",
|
|
"lib/**/*.mjs",
|
|
"hooks/**/*.mjs"
|
|
],
|
|
"pattern": "VOYAGE_STORM_ENABLED",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "E-02",
|
|
"summary": "`/trekresearch --engine deep-research` delegates to Claude Code's /deep-research, which is manual-only since 2.1.218, so the opt-in always falls back to the swarm",
|
|
"closesWhen": "the --engine opt-in is removed from trekresearch.md",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekresearch.md",
|
|
"pattern": "--engine\\b",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "E-03",
|
|
"summary": "the delegated-orchestration head-to-head (T1 §5) is designed but has no recorded decision",
|
|
"closesWhen": "T1 §5 carries a STATUS block (run, or declined)",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "docs/T1-cc26-delegated-orchestration.md",
|
|
"section": "## 5.",
|
|
"pattern": "^> \\*\\*STATUS:",
|
|
"flags": "m",
|
|
"expect": "no-match"
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"agents": {
|
|
"dir": "agents",
|
|
"spawnSites": "commands/*.md",
|
|
"referenceMarker": "Reference document, not a spawnable capability"
|
|
},
|
|
"decisions": {
|
|
"file": "STATE.md",
|
|
"section": "## Åpne operatørbeslutninger",
|
|
"open": "^- \\[ \\] ",
|
|
"closed": "^- \\[x\\] "
|
|
},
|
|
"freeze": {
|
|
"tag": "end-state-freeze",
|
|
"featPattern": "^feat(\\(|!|:)"
|
|
}
|
|
}
|