fix(end-state): the D-03/D-04 probes RUN a named test instead of grepping for a symbol
Measured 2026-09-18 (PM checkpoint on6cafb4c, mutant M6): a two-line file export function runPlanVerification() { return { ok: true }; } export function formatCriteriaEvidence() { return ''; } dropped into an otherwise UNFIXED tree closed both D-03 and D-04, and the gate reported "defects 0 of 7, registry integrity: intact". The same checkpoint felled 5 of 5 mutants with the SUITE and 0 of 5 with the GATE. Both probes called themselves `behaviour` while asking a grep whether an export name was present - a phrase probe pointed at code. A check condition may now name a test instead of a pattern: { "test": "tests/lib/criteria-runner.test.mjs", "name": "<exact test name>", "expect": "fails" } The gate spawns the test runner on that one test and reads its TAP line. `expect: "fails"` holds - the defect stays open - while the test does not pass. A missing file, a name that matches nothing, a runner that will not start: all throw, which counts as NOT FELLABLE and therefore open. A check that cannot fire is never "fixed". The child's `NODE_TEST_*` env is stripped. The gate usually runs UNDER the test runner, and an inherited `NODE_TEST_CONTEXT` makes the grandchild report over the parent's IPC channel instead of stdout - the TAP line would never arrive and the probe would silently stop felling anything. M6 is now a permanent mutant in the gate's own test, so the cosmetic close cannot come back. Verified end to end against a real stub tree: git archive8d1669e| tar -x -C $TMP cp scripts/end-state-gate.mjs scripts/end-state-registry.json $TMP/scripts/ cp tests/fixtures/end-state-frozen.json $TMP/tests/fixtures/; cp STATE.md $TMP/ printf 'export function runPlanVerification…' > $TMP/lib/cosmetic/stub.mjs node scripts/end-state-gate.mjs --root $TMP -> defects 2 of 7, both NOT FELLABLE, exit 1 The gate also now states the limit out loud, as its own line above the table: a behaviour probe proves the capability WORKS; that Phase 7 and Phase 4.5 CALL it is pinned by TEXT in doc-consistency, not proven deterministically - real proof is a headless plugin-eval run against a fixture plan (week 40). FROZEN DENOMINATOR AMENDED, second time in one day, by the work order that carries this change: D-03 and D-04 signatures moved because their checks changed on purpose. tests/fixtures/end-state-frozen.json and the FROZEN literal in tests/scripts/end-state-gate.test.mjs are updated together, both dated, both saying why. No entry was removed. Gate after this change: defects 0 of 7, registry integrity intact, exit 1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
106dcb0091
commit
a7af76e5ff
4 changed files with 158 additions and 15 deletions
|
|
@ -35,26 +35,26 @@
|
|||
{
|
||||
"id": "D-03",
|
||||
"summary": "commands/trekexecute.md never runs a trekplan's `## Verification` (where the brief's success criteria land) on the single-session path: Phase 7 says 'Skip for trekplans', and only the multi-session wave path runs master verification",
|
||||
"closesWhen": "a runnable plan-verification runner exists in lib/ (behaviour probe: it proves the capability exists and is exercised by a test, not that Phase 7 calls it — the wiring is pinned by tests/lib/doc-consistency.test.mjs)",
|
||||
"closesWhen": "a named test PROVES the single-session path fells a run on a criterion that does not hold (behaviour probe: the gate RUNS tests/lib/criteria-runner.test.mjs, so a stub that only exports the symbol does not close it; that Phase 7 CALLS it is pinned by tests/lib/doc-consistency.test.mjs, not by this probe)",
|
||||
"probe": "behaviour",
|
||||
"check": [
|
||||
{
|
||||
"path": "lib/**/*.mjs",
|
||||
"pattern": "export function runPlanVerification",
|
||||
"expect": "no-match"
|
||||
"test": "tests/lib/criteria-runner.test.mjs",
|
||||
"name": "runPlanVerification: a plan whose success criterion FAILS fells the run",
|
||||
"expect": "fails"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "D-04",
|
||||
"summary": "agents/brief-conformance-reviewer.md must judge whether a success criterion's verification command 'exists and passes', but its tools are Read/Glob/Grep, so it cannot run anything",
|
||||
"closesWhen": "the evidence block the conformance reviewer judges is BUILT IN CODE (behaviour probe: it proves /trekreview can hand over a real exit code instead of asking a Read/Glob/Grep agent whether a command passes; the rubric rewrite and the hand-over are pinned by tests/lib/doc-consistency.test.mjs)",
|
||||
"closesWhen": "a named test PROVES the evidence block the conformance reviewer judges is BUILT IN CODE from real exit codes (behaviour probe: the gate RUNS tests/lib/criteria-runner.test.mjs, so a stub that only exports the symbol does not close it; that Phase 4.5 CALLS it is pinned by tests/lib/doc-consistency.test.mjs, not by this probe)",
|
||||
"probe": "behaviour",
|
||||
"check": [
|
||||
{
|
||||
"path": "lib/**/*.mjs",
|
||||
"pattern": "export function formatCriteriaEvidence",
|
||||
"expect": "no-match"
|
||||
"test": "tests/lib/criteria-runner.test.mjs",
|
||||
"name": "formatCriteriaEvidence: one row per criterion, with command and exit code",
|
||||
"expect": "fails"
|
||||
}
|
||||
]
|
||||
},
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue