Kjell Tore Guttormsen
a7af76e5ff
Measured 2026-09-18 (PM checkpoint on6cafb4c, mutant M6): a two-line file export function runPlanVerification() { return { ok: true }; } export function formatCriteriaEvidence() { return ''; } dropped into an otherwise UNFIXED tree closed both D-03 and D-04, and the gate reported "defects 0 of 7, registry integrity: intact". The same checkpoint felled 5 of 5 mutants with the SUITE and 0 of 5 with the GATE. Both probes called themselves `behaviour` while asking a grep whether an export name was present - a phrase probe pointed at code. A check condition may now name a test instead of a pattern: { "test": "tests/lib/criteria-runner.test.mjs", "name": "<exact test name>", "expect": "fails" } The gate spawns the test runner on that one test and reads its TAP line. `expect: "fails"` holds - the defect stays open - while the test does not pass. A missing file, a name that matches nothing, a runner that will not start: all throw, which counts as NOT FELLABLE and therefore open. A check that cannot fire is never "fixed". The child's `NODE_TEST_*` env is stripped. The gate usually runs UNDER the test runner, and an inherited `NODE_TEST_CONTEXT` makes the grandchild report over the parent's IPC channel instead of stdout - the TAP line would never arrive and the probe would silently stop felling anything. M6 is now a permanent mutant in the gate's own test, so the cosmetic close cannot come back. Verified end to end against a real stub tree: git archive8d1669e| tar -x -C $TMP cp scripts/end-state-gate.mjs scripts/end-state-registry.json $TMP/scripts/ cp tests/fixtures/end-state-frozen.json $TMP/tests/fixtures/; cp STATE.md $TMP/ printf 'export function runPlanVerification…' > $TMP/lib/cosmetic/stub.mjs node scripts/end-state-gate.mjs --root $TMP -> defects 2 of 7, both NOT FELLABLE, exit 1 The gate also now states the limit out loud, as its own line above the table: a behaviour probe proves the capability WORKS; that Phase 7 and Phase 4.5 CALL it is pinned by TEXT in doc-consistency, not proven deterministically - real proof is a headless plugin-eval run against a fixture plan (week 40). FROZEN DENOMINATOR AMENDED, second time in one day, by the work order that carries this change: D-03 and D-04 signatures moved because their checks changed on purpose. tests/fixtures/end-state-frozen.json and the FROZEN literal in tests/scripts/end-state-gate.test.mjs are updated together, both dated, both saying why. No entry was removed. Gate after this change: defects 0 of 7, registry integrity intact, exit 1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
176 lines
6.7 KiB
JSON
176 lines
6.7 KiB
JSON
{
|
|
"defects": [
|
|
{
|
|
"id": "D-01",
|
|
"summary": "commands/trekplan.md still references TeamCreate/TeamDelete (allowed-tools and the execute-with-team path); both tools were removed in Claude Code 2.1.178, so that path always falls back to sequential",
|
|
"closesWhen": "trekplan.md no longer names TeamCreate or TeamDelete",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekplan.md",
|
|
"pattern": "\\bTeam(Create|Delete)\\b",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-02",
|
|
"summary": "commands/trekresearch.md tells the swarm engine to run the `### Bridge agent` block, but that heading was removed together with gemini-bridge",
|
|
"closesWhen": "the dangling reference is gone (or the heading exists again)",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekresearch.md",
|
|
"pattern": "`### Bridge agent`",
|
|
"expect": "match"
|
|
},
|
|
{
|
|
"path": "commands/trekresearch.md",
|
|
"pattern": "^### Bridge agent",
|
|
"flags": "m",
|
|
"expect": "no-match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-03",
|
|
"summary": "commands/trekexecute.md never runs a trekplan's `## Verification` (where the brief's success criteria land) on the single-session path: Phase 7 says 'Skip for trekplans', and only the multi-session wave path runs master verification",
|
|
"closesWhen": "a named test PROVES the single-session path fells a run on a criterion that does not hold (behaviour probe: the gate RUNS tests/lib/criteria-runner.test.mjs, so a stub that only exports the symbol does not close it; that Phase 7 CALLS it is pinned by tests/lib/doc-consistency.test.mjs, not by this probe)",
|
|
"probe": "behaviour",
|
|
"check": [
|
|
{
|
|
"test": "tests/lib/criteria-runner.test.mjs",
|
|
"name": "runPlanVerification: a plan whose success criterion FAILS fells the run",
|
|
"expect": "fails"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-04",
|
|
"summary": "agents/brief-conformance-reviewer.md must judge whether a success criterion's verification command 'exists and passes', but its tools are Read/Glob/Grep, so it cannot run anything",
|
|
"closesWhen": "a named test PROVES the evidence block the conformance reviewer judges is BUILT IN CODE from real exit codes (behaviour probe: the gate RUNS tests/lib/criteria-runner.test.mjs, so a stub that only exports the symbol does not close it; that Phase 4.5 CALLS it is pinned by tests/lib/doc-consistency.test.mjs, not by this probe)",
|
|
"probe": "behaviour",
|
|
"check": [
|
|
{
|
|
"test": "tests/lib/criteria-runner.test.mjs",
|
|
"name": "formatCriteriaEvidence: one row per criterion, with command and exit code",
|
|
"expect": "fails"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-05",
|
|
"summary": "commands/trekplan.md allowed-tools lists TaskCreate/TaskUpdate, which Claude Code 2.1.233 no longer offers on Opus 4.8 / Sonnet 5 / Fable 5 and newer by default (runtime effect not measured)",
|
|
"closesWhen": "trekplan.md allowed-tools no longer lists TaskCreate or TaskUpdate",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekplan.md",
|
|
"pattern": "^allowed-tools:.*\\bTask(Create|Update)\\b",
|
|
"flags": "m",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-06",
|
|
"summary": "lib/review/gold-scorer.mjs contains a literal NUL byte, so git treats the file as binary and hides it from diffs and --numstat",
|
|
"closesWhen": "the file contains no NUL byte (write the separator as an escape sequence)",
|
|
"probe": "byte",
|
|
"check": [
|
|
{
|
|
"path": "lib/review/gold-scorer.mjs",
|
|
"pattern": "\\u0000",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-07",
|
|
"summary": "CLAUDE.md, README.md and commands/trekresearch.md claim the /trek* orchestrators run on opus (README even ships a sed recipe for `model: opus` lines no command has), but since v5.9.0 no command pins model: and the orchestrator follows the session model",
|
|
"closesWhen": "none of the three files claims an opus orchestrator any more, or the commands pin opus again",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": [
|
|
"CLAUDE.md",
|
|
"README.md",
|
|
"commands/trekresearch.md"
|
|
],
|
|
"pattern": "(^\\| `/trek[a-z]+` \\|.*\\| opus \\|\\s*$)|(\\^model: opus\\$)|(default for `/trekbrief`[\\s\\S]{0,80}?is `opus`)|(orchestrator runs on Opus)",
|
|
"flags": "m",
|
|
"expect": "match"
|
|
},
|
|
{
|
|
"path": "commands/*.md",
|
|
"pattern": "^model:",
|
|
"flags": "m",
|
|
"expect": "no-match"
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"experiments": [
|
|
{
|
|
"id": "E-01",
|
|
"summary": "STORM dimension discovery + bounded research loop ships default-off behind VOYAGE_STORM_ENABLED; adoption is gated on a pre-registered measurement that has not run",
|
|
"closesWhen": "the env gate is gone from the pipeline surface: adopted as default, or the loop is removed",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": [
|
|
"commands/*.md",
|
|
"agents/*.md",
|
|
"lib/**/*.mjs",
|
|
"hooks/**/*.mjs"
|
|
],
|
|
"pattern": "VOYAGE_STORM_ENABLED",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "E-02",
|
|
"summary": "`/trekresearch --engine deep-research` delegates to Claude Code's /deep-research, which is manual-only since 2.1.218, so the opt-in always falls back to the swarm",
|
|
"closesWhen": "the --engine opt-in is removed from trekresearch.md",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekresearch.md",
|
|
"pattern": "--engine\\b",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "E-03",
|
|
"summary": "the delegated-orchestration head-to-head (T1 §5) is designed but has no recorded decision",
|
|
"closesWhen": "T1 §5 carries a STATUS block (run, or declined)",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "docs/T1-cc26-delegated-orchestration.md",
|
|
"section": "## 5.",
|
|
"pattern": "^> \\*\\*STATUS:",
|
|
"flags": "m",
|
|
"expect": "no-match"
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"agents": {
|
|
"dir": "agents",
|
|
"spawnSites": "commands/*.md",
|
|
"referenceMarker": "Reference document, not a spawnable capability"
|
|
},
|
|
"decisions": {
|
|
"file": "STATE.md",
|
|
"section": "## Åpne operatørbeslutninger",
|
|
"open": "^- \\[ \\] ",
|
|
"closed": "^- \\[x\\] "
|
|
},
|
|
"freeze": {
|
|
"tag": "end-state-freeze",
|
|
"featPattern": "^feat(\\(|!|:)"
|
|
}
|
|
}
|