Kjell Tore Guttormsen
09feb415b3
An independent adversarial review of the end-state gate reproduced a GREEN result, exit 0, with the gate's 18 tests passing. Only the registry and the STATE markers were edited; every other file stayed byte-identical. The judged party owned the denominator, and nothing guarded it. This change closes that finding and three more, and reopens a defect the gate had closed too early. Registry integrity (the blocker): - tests/fixtures/end-state-frozen.json freezes the denominator. It holds the 7 defect ids and 3 experiment ids, each with a sha256 signature of its check (canonical JSON, key order irrelevant), plus the exact agents / decisions / freeze configuration. - The rule: an entry may close; it may never disappear or have its check changed. New entries are allowed. spawnSites may never include agents/. - The test file pins the manifest literally, with a comment that the denominator was frozen on 2026-09-17 and that changing it is a decision. The gate verifies the registry against the manifest on every run. - A violation makes the gate red and names what moved. A missing manifest makes integrity n/a, which is also red. Dormant means "never spawned", not "never mentioned": - An agent counts as spawned only when a command names it in a spawn instruction: a row of a table headed `Agent`, a line that starts with Launch/Spawn, or a **name** block followed by a Prompt: line. - Prose, negations, HTML comments and fenced code never count. Block quotes cannot match either, because every form is anchored at the start of the line. - Re-measured under this definition: still 1 of 20 (synthesis-agent). Per agent, every other spawnable agent has a spawn instruction in at least one command. A fifth tally, feat-after-freeze: - It counts commits after the `end-state-freeze` tag whose subject matches ^feat(\(|!|:). - The row is n/a, and therefore red, when there is no tag yet, the root is not a git work tree, or the root is not the top of its work tree. It is never 0. Honest output: - Every registry entry now declares its probe kind (phrase or byte). - Open phrase probes are labelled as such, and closed ids are listed. The output states that a closed phrase probe is evidence, not proof of behaviour. D-03 and D-04 stay open and labelled; how they are fixed is still to be decided. - The decisions row now says it counts ticks, not verified decisions. D-07 reopened: - Its check now also covers README.md and commands/trekresearch.md, where the "orchestrator runs on opus" claim still lives. README even ships a sed recipe for `model: opus` lines no command has. - Defects are 3 of 7 until the separate docs fix lands. Three surviving review mutants were killed with tests: - an empty glob over an existing dir is silent; - the header counts n/a rows as zero; - a dormant-row error reads as 0. Measured on a clean export of the index: - Mutation harness, adapted from the review's: review code mutants 18 of 18 killed. Four of them were re-targeted at the equivalent new code. M02 survived until a test with an intact registry plus a missing freeze tag was added. - New-logic mutants: 12 of 12 killed. One equivalent mutant (block-quote stripping) was removed together with the dead clause it targeted. - Registry attacks: 7 of 7 killed. - The review's combined green attack (registry + STATE only) is now RED, exit 1: "defects: D-03 is frozen but missing from the registry". - Gate tests 38/38. Suite 1059 -> 1079 (1077/0/2). - Gate: RED, defects 3 of 7, experiments 3 of 3, dormant 1 of 20; decisions and feat-after-freeze are n/a in a clean export. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
184 lines
6.4 KiB
JSON
184 lines
6.4 KiB
JSON
{
|
|
"defects": [
|
|
{
|
|
"id": "D-01",
|
|
"summary": "commands/trekplan.md still references TeamCreate/TeamDelete (allowed-tools and the execute-with-team path); both tools were removed in Claude Code 2.1.178, so that path always falls back to sequential",
|
|
"closesWhen": "trekplan.md no longer names TeamCreate or TeamDelete",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekplan.md",
|
|
"pattern": "\\bTeam(Create|Delete)\\b",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-02",
|
|
"summary": "commands/trekresearch.md tells the swarm engine to run the `### Bridge agent` block, but that heading was removed together with gemini-bridge",
|
|
"closesWhen": "the dangling reference is gone (or the heading exists again)",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekresearch.md",
|
|
"pattern": "`### Bridge agent`",
|
|
"expect": "match"
|
|
},
|
|
{
|
|
"path": "commands/trekresearch.md",
|
|
"pattern": "^### Bridge agent",
|
|
"flags": "m",
|
|
"expect": "no-match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-03",
|
|
"summary": "commands/trekexecute.md never runs a trekplan's `## Verification` (where the brief's success criteria land) on the single-session path: Phase 7 says 'Skip for trekplans', and only the multi-session wave path runs master verification",
|
|
"closesWhen": "Phase 7 no longer skips trekplans (proxy: the fix must also make that path run the plan's Verification; Phase 4's entry-condition skip is legitimate and out of scope)",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekexecute.md",
|
|
"section": "## Phase 7 —",
|
|
"pattern": "^\\*\\*Skip for trekplans\\.\\*\\*",
|
|
"flags": "m",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-04",
|
|
"summary": "agents/brief-conformance-reviewer.md must judge whether a success criterion's verification command 'exists and passes', but its tools are Read/Glob/Grep, so it cannot run anything",
|
|
"closesWhen": "the rubric no longer asks it to judge 'passes', or the reviewer can execute the command",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "agents/brief-conformance-reviewer.md",
|
|
"pattern": "exists and passes",
|
|
"expect": "match"
|
|
},
|
|
{
|
|
"path": "agents/brief-conformance-reviewer.md",
|
|
"pattern": "^tools:.*\"Bash\"",
|
|
"flags": "m",
|
|
"expect": "no-match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-05",
|
|
"summary": "commands/trekplan.md allowed-tools lists TaskCreate/TaskUpdate, which Claude Code 2.1.233 no longer offers on Opus 4.8 / Sonnet 5 / Fable 5 and newer by default (runtime effect not measured)",
|
|
"closesWhen": "trekplan.md allowed-tools no longer lists TaskCreate or TaskUpdate",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekplan.md",
|
|
"pattern": "^allowed-tools:.*\\bTask(Create|Update)\\b",
|
|
"flags": "m",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-06",
|
|
"summary": "lib/review/gold-scorer.mjs contains a literal NUL byte, so git treats the file as binary and hides it from diffs and --numstat",
|
|
"closesWhen": "the file contains no NUL byte (write the separator as an escape sequence)",
|
|
"probe": "byte",
|
|
"check": [
|
|
{
|
|
"path": "lib/review/gold-scorer.mjs",
|
|
"pattern": "\\u0000",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "D-07",
|
|
"summary": "CLAUDE.md, README.md and commands/trekresearch.md claim the /trek* orchestrators run on opus (README even ships a sed recipe for `model: opus` lines no command has), but since v5.9.0 no command pins model: and the orchestrator follows the session model",
|
|
"closesWhen": "none of the three files claims an opus orchestrator any more, or the commands pin opus again",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": [
|
|
"CLAUDE.md",
|
|
"README.md",
|
|
"commands/trekresearch.md"
|
|
],
|
|
"pattern": "(^\\| `/trek[a-z]+` \\|.*\\| opus \\|\\s*$)|(\\^model: opus\\$)|(default for `/trekbrief`[\\s\\S]{0,80}?is `opus`)|(orchestrator runs on Opus)",
|
|
"flags": "m",
|
|
"expect": "match"
|
|
},
|
|
{
|
|
"path": "commands/*.md",
|
|
"pattern": "^model:",
|
|
"flags": "m",
|
|
"expect": "no-match"
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"experiments": [
|
|
{
|
|
"id": "E-01",
|
|
"summary": "STORM dimension discovery + bounded research loop ships default-off behind VOYAGE_STORM_ENABLED; adoption is gated on a pre-registered measurement that has not run",
|
|
"closesWhen": "the env gate is gone from the pipeline surface: adopted as default, or the loop is removed",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": [
|
|
"commands/*.md",
|
|
"agents/*.md",
|
|
"lib/**/*.mjs",
|
|
"hooks/**/*.mjs"
|
|
],
|
|
"pattern": "VOYAGE_STORM_ENABLED",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "E-02",
|
|
"summary": "`/trekresearch --engine deep-research` delegates to Claude Code's /deep-research, which is manual-only since 2.1.218, so the opt-in always falls back to the swarm",
|
|
"closesWhen": "the --engine opt-in is removed from trekresearch.md",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "commands/trekresearch.md",
|
|
"pattern": "--engine\\b",
|
|
"expect": "match"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "E-03",
|
|
"summary": "the delegated-orchestration head-to-head (T1 §5) is designed but has no recorded decision",
|
|
"closesWhen": "T1 §5 carries a STATUS block (run, or declined)",
|
|
"probe": "phrase",
|
|
"check": [
|
|
{
|
|
"path": "docs/T1-cc26-delegated-orchestration.md",
|
|
"section": "## 5.",
|
|
"pattern": "^> \\*\\*STATUS:",
|
|
"flags": "m",
|
|
"expect": "no-match"
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"agents": {
|
|
"dir": "agents",
|
|
"spawnSites": "commands/*.md",
|
|
"referenceMarker": "Reference document, not a spawnable capability"
|
|
},
|
|
"decisions": {
|
|
"file": "STATE.md",
|
|
"section": "## Åpne operatørbeslutninger",
|
|
"open": "^- \\[ \\] ",
|
|
"closed": "^- \\[x\\] "
|
|
},
|
|
"freeze": {
|
|
"tag": "end-state-freeze",
|
|
"featPattern": "^feat(\\(|!|:)"
|
|
}
|
|
}
|