fix(end-state): a behaviour probe must FELL a stub, not just name a test
The cosmetic close came back one level up. The probe reads the test file from the same tree it measures, so a four-line file holding two EMPTY tests with the two named names closed D-03 and D-04 on a tree where `lib/verification/` did not exist at all - "defects 0 of 7, registry intact" (measured 2026-09-18). First a two-line stub exporting the right symbols; then an empty test with the right name. A name is always forgeable. So the named test is now run twice. Once on the tree, as before - and once in a sandbox where the module the condition declares in `stubs` is replaced by a stub exporting the same names, all inert. If the test still passes there, it binds the name and not the behaviour, and the condition THROWS: NOT FELLABLE, counted open. A condition that declares no `stubs`, or names a module that is not there, cannot fire either. The sandbox is a symlink overlay: every entry of the tree is symlinked, and only the test file and the stubbed module are materialised for real - Node resolves an ESM import through the realpath, so a symlinked test file would import the original module and never see the mutant. Nothing is ever written inside the measured tree, and the only directory removed is the one this code made under the system temp dir (pinned by a test). M7 is now a permanent mutant beside M6, in two forms: the checkpoint's own reproduction (unfixed tree + empty named tests) and the harder one (the real module present, so the stub can be built and the empty test passes against it). Both report `defects 2 of 7`. A positive control pins that D-03/D-04 still CLOSE on the real tree, so "not closed" everywhere cannot read as a working probe. The frozen denominator moves a third time, deliberately, and its `why` no longer claims authority it does not have: the second and third amendments were maintenance decisions by the maintainer, not operator decisions, and the tracked file now says exactly that. Measured after: real tree node scripts/end-state-gate.mjs -> defects 0 of 7, intact, exit 1 M6 (stub)8d1669e+ current gate/registry/frozen + lib/cosmetic/stub.mjs -> 2 of 7 M7 (empty)8d1669e+ current gate/registry/frozen + 3-line tests/lib/criteria-runner.test.mjs with the two named tests -> 2 of 7 Red first: 4 of the new tests failed before the change. Suite 1154 (1152/0/2). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
02243c6365
commit
83d82f121d
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 252 additions and 45 deletions
|
|
@ -19,7 +19,11 @@
|
|||
//
|
||||
// Usage: node scripts/end-state-gate.mjs [--json] [--root <dir>]
|
||||
|
||||
import { readFileSync, existsSync, readdirSync, statSync, realpathSync } from 'node:fs';
|
||||
import {
|
||||
readFileSync, writeFileSync, existsSync, readdirSync, statSync, realpathSync,
|
||||
mkdirSync, mkdtempSync, symlinkSync, rmSync,
|
||||
} from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, resolve, dirname, relative, sep, basename } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { createHash } from 'node:crypto';
|
||||
|
|
@ -79,19 +83,103 @@ export const TEST_TIMEOUT_MS = 120_000;
|
|||
|
||||
const escapeRegExp = (t) => String(t).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
|
||||
// Every `export` name a module declares, so a stub can offer the same surface
|
||||
// and none of the behaviour.
|
||||
const EXPORT_NAME = /^\s*export\s+(?:async\s+)?(?:function\*?|const|let|var|class)\s+([A-Za-z_$][\w$]*)/gm;
|
||||
|
||||
/**
|
||||
* A condition that RUNS a named test and reads its result.
|
||||
* A stub of a module: the same export names, every one of them inert.
|
||||
*
|
||||
* This is the mutant a behaviour probe must FELL. A test that still passes
|
||||
* when the module it names does nothing is not measuring that module — the
|
||||
* empty test that closed D-03 and D-04 on a tree without `lib/verification/`
|
||||
* is the limiting case (measured 2026-09-18).
|
||||
*/
|
||||
export function stubModule(source) {
|
||||
const names = new Set();
|
||||
for (const m of String(source).matchAll(EXPORT_NAME)) names.add(m[1]);
|
||||
const lines = [...names].map((n) => `export const ${n} = undefined;`);
|
||||
if (/^\s*export\s+default\b/m.test(source)) lines.push('export default undefined;');
|
||||
return `${lines.join('\n')}\n`;
|
||||
}
|
||||
|
||||
// Build `dest` as a shallow copy of `root`: every entry is a SYMLINK to the
|
||||
// original, except the files in `files`, which are written for real (and whose
|
||||
// parent directories are therefore real directories too). Node resolves an
|
||||
// ESM import through the realpath, so a symlinked test file would import the
|
||||
// ORIGINAL module and never see the stub — only the files that must differ are
|
||||
// materialised. Nothing is ever written inside the measured tree.
|
||||
function overlay(root, dest, files) {
|
||||
const realDirs = new Set();
|
||||
for (const rel of files.keys()) {
|
||||
const parts = rel.split('/');
|
||||
for (let i = 0; i < parts.length - 1; i++) realDirs.add(parts.slice(0, i + 1).join('/'));
|
||||
}
|
||||
const build = (relDir) => {
|
||||
const absSrc = relDir === '' ? root : join(root, relDir);
|
||||
const absDst = relDir === '' ? dest : join(dest, relDir);
|
||||
mkdirSync(absDst, { recursive: true });
|
||||
if (!existsSync(absSrc)) return;
|
||||
for (const name of readdirSync(absSrc)) {
|
||||
const childRel = relDir === '' ? name : `${relDir}/${name}`;
|
||||
if (realDirs.has(childRel)) { build(childRel); continue; }
|
||||
if (files.has(childRel)) continue;
|
||||
symlinkSync(join(absSrc, name), join(absDst, name));
|
||||
}
|
||||
};
|
||||
build('');
|
||||
for (const [rel, body] of files) writeFileSync(join(dest, rel), body);
|
||||
}
|
||||
|
||||
// Run ONE named test out of one test file and report whether it passed.
|
||||
// `null` = the runner produced no TAP line for that name (it did not run at
|
||||
// all), which is never a pass.
|
||||
function runNamedTest(root, testRel, name) {
|
||||
// The gate itself often runs UNDER the test runner (the gate's own test
|
||||
// spawns it). `NODE_TEST_CONTEXT` inherited into this child makes it report
|
||||
// over the parent's IPC channel instead of stdout, and the TAP line would
|
||||
// never arrive — a probe that silently stops felling anything.
|
||||
const env = { ...process.env };
|
||||
for (const key of Object.keys(env)) {
|
||||
if (key.startsWith('NODE_TEST_')) delete env[key];
|
||||
}
|
||||
const r = spawnSync(
|
||||
process.execPath,
|
||||
['--test', '--test-reporter=tap', `--test-name-pattern=^${escapeRegExp(name)}$`, testRel],
|
||||
{ cwd: root, encoding: 'utf8', timeout: TEST_TIMEOUT_MS, env },
|
||||
);
|
||||
if (r.error) throw new Error(`test runner did not run ${testRel}: ${r.error.message}`);
|
||||
// The file itself reports as a subtest too, so the summary counters are
|
||||
// ambiguous; the TAP line carrying the exact name is not. TAP escapes `#`
|
||||
// and `\`, so unescape before comparing.
|
||||
for (const line of String(r.stdout ?? '').split('\n')) {
|
||||
const m = line.match(/^\s*(not )?ok \d+ - (.*)$/);
|
||||
if (!m) continue;
|
||||
if (m[2].replace(/\\(.)/g, '$1').trim() !== name) continue;
|
||||
return m[1] === undefined;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* A condition that RUNS a named test, reads its result, and requires that the
|
||||
* same test FELLS a stub of the module it claims to bind.
|
||||
*
|
||||
* A grep condition can only ask whether a string is present, so a "behaviour"
|
||||
* probe built from one is a phrase probe pointed at code: measured 2026-09-18,
|
||||
* a two-line file exporting `runPlanVerification` and `formatCriteriaEvidence`
|
||||
* closed D-03 and D-04 on a tree where nothing was fixed. This spawns the test
|
||||
* runner on one named test and reads the TAP line for it.
|
||||
* probe built from one is a phrase probe pointed at code: measured
|
||||
* 2026-09-18, a two-line file exporting `runPlanVerification` and
|
||||
* `formatCriteriaEvidence` closed D-03 and D-04 on a tree where nothing was
|
||||
* fixed. Running a NAMED test closed that hole and opened a smaller one one
|
||||
* level up: two EMPTY tests with the right names closed them again. A name is
|
||||
* always forgeable; felling a mutant is not. So the named test is run twice —
|
||||
* once on the tree, once against a stub of each module in `stubs` — and it
|
||||
* must not pass the second time.
|
||||
*
|
||||
* `expect: "passes"` holds when the named test passes; `"fails"` holds when it
|
||||
* does not. A test file that is missing, a name that matches nothing, a runner
|
||||
* that cannot start: all THROW, which the caller counts as NOT FELLABLE and
|
||||
* therefore open. A check that cannot fire is never "fixed".
|
||||
* that cannot start, a missing or unfelled stub target: all THROW, which the
|
||||
* caller counts as NOT FELLABLE and therefore open. A check that cannot fire
|
||||
* is never "fixed".
|
||||
*/
|
||||
export function evaluateTestCondition(root, cond) {
|
||||
if (cond.expect !== 'passes' && cond.expect !== 'fails') {
|
||||
|
|
@ -100,33 +188,33 @@ export function evaluateTestCondition(root, cond) {
|
|||
if (typeof cond.name !== 'string' || cond.name.trim() === '') {
|
||||
throw new Error('a test condition must name exactly one test');
|
||||
}
|
||||
if (!Array.isArray(cond.stubs) || cond.stubs.length === 0) {
|
||||
throw new Error('a test condition must declare `stubs`: the module(s) whose stub the named test must fell');
|
||||
}
|
||||
const file = join(root, cond.test);
|
||||
if (!existsSync(file)) throw new Error(`test file not found: ${cond.test}`);
|
||||
// The gate itself often runs UNDER the test runner (the gate's own test
|
||||
// spawns it). `NODE_TEST_CONTEXT` inherited into this child makes it report
|
||||
// over the parent's IPC channel instead of stdout, and the TAP line would
|
||||
// never arrive — a probe that silently stops fellling anything.
|
||||
const env = { ...process.env };
|
||||
for (const key of Object.keys(env)) {
|
||||
if (key.startsWith('NODE_TEST_')) delete env[key];
|
||||
}
|
||||
const r = spawnSync(
|
||||
process.execPath,
|
||||
['--test', '--test-reporter=tap', `--test-name-pattern=^${escapeRegExp(cond.name)}$`, file],
|
||||
{ cwd: root, encoding: 'utf8', timeout: TEST_TIMEOUT_MS, env },
|
||||
);
|
||||
if (r.error) throw new Error(`test runner did not run ${cond.test}: ${r.error.message}`);
|
||||
// The file itself reports as a subtest too, so the summary counters are
|
||||
// ambiguous; the TAP line carrying the exact name is not. TAP escapes `#`
|
||||
// and `\`, so unescape before comparing.
|
||||
let passed = null;
|
||||
for (const line of String(r.stdout ?? '').split('\n')) {
|
||||
const m = line.match(/^\s*(not )?ok \d+ - (.*)$/);
|
||||
if (!m) continue;
|
||||
if (m[2].replace(/\\(.)/g, '$1').trim() !== cond.name) continue;
|
||||
passed = m[1] === undefined;
|
||||
break;
|
||||
|
||||
for (const target of cond.stubs) {
|
||||
const abs = join(root, target);
|
||||
if (!existsSync(abs)) throw new Error(`stub target not found: ${target}`);
|
||||
const sandbox = mkdtempSync(join(tmpdir(), 'end-state-mutant-'));
|
||||
try {
|
||||
overlay(root, sandbox, new Map([
|
||||
[cond.test, readFileSync(file, 'utf8')],
|
||||
[target, stubModule(readFileSync(abs, 'utf8'))],
|
||||
]));
|
||||
if (runNamedTest(sandbox, cond.test, cond.name) === true) {
|
||||
throw new Error(
|
||||
`"${cond.name}" still passes against a stub of ${target} — it binds the name, not the behaviour`,
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
// Only ever a directory this function made, under the system temp dir.
|
||||
if (sandbox.startsWith(realpathSync(tmpdir()))) rmSync(sandbox, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
const passed = runNamedTest(root, cond.test, cond.name);
|
||||
if (passed === null) throw new Error(`no test named "${cond.name}" ran in ${cond.test}`);
|
||||
return cond.expect === 'passes' ? passed : !passed;
|
||||
}
|
||||
|
|
@ -234,7 +322,7 @@ export const PROBE_NOTES = {
|
|||
phrase:
|
||||
'a phrase probe closes on rewording: a closed phrase probe is evidence, not proof of behaviour',
|
||||
behaviour:
|
||||
'a behaviour probe RUNS a named test and reads its result, so a stub that only exports the right symbol does not close it; it proves the capability WORKS, not that a prose phase calls it',
|
||||
'a behaviour probe RUNS a named test AND re-runs it against a stub of the module it binds, which it must fell: neither a stub that exports the right symbols nor an empty test with the right name closes it; it proves the capability WORKS, not that a prose phase calls it',
|
||||
byte:
|
||||
"a byte probe reads the file's bytes, so it closes only on a real change to them",
|
||||
};
|
||||
|
|
|
|||
|
|
@ -35,26 +35,32 @@
|
|||
{
|
||||
"id": "D-03",
|
||||
"summary": "commands/trekexecute.md never runs a trekplan's `## Verification` (where the brief's success criteria land) on the single-session path: Phase 7 says 'Skip for trekplans', and only the multi-session wave path runs master verification",
|
||||
"closesWhen": "a named test PROVES the single-session path fells a run on a criterion that does not hold (behaviour probe: the gate RUNS tests/lib/criteria-runner.test.mjs, so a stub that only exports the symbol does not close it; that Phase 7 CALLS it is pinned by tests/lib/doc-consistency.test.mjs, not by this probe)",
|
||||
"closesWhen": "a named test PROVES the single-session path fells a run on a criterion that does not hold (behaviour probe: the gate RUNS tests/lib/criteria-runner.test.mjs, so a stub that only exports the symbol does not close it, and the named test must itself FELL a stub of lib/verification/criteria-runner.mjs, so an EMPTY test with the right name does not close it either; that Phase 7 CALLS it is pinned by tests/lib/doc-consistency.test.mjs, not by this probe)",
|
||||
"probe": "behaviour",
|
||||
"check": [
|
||||
{
|
||||
"test": "tests/lib/criteria-runner.test.mjs",
|
||||
"name": "runPlanVerification: a plan whose success criterion FAILS fells the run",
|
||||
"expect": "fails"
|
||||
"expect": "fails",
|
||||
"stubs": [
|
||||
"lib/verification/criteria-runner.mjs"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "D-04",
|
||||
"summary": "agents/brief-conformance-reviewer.md must judge whether a success criterion's verification command 'exists and passes', but its tools are Read/Glob/Grep, so it cannot run anything",
|
||||
"closesWhen": "a named test PROVES the evidence block the conformance reviewer judges is BUILT IN CODE from real exit codes (behaviour probe: the gate RUNS tests/lib/criteria-runner.test.mjs, so a stub that only exports the symbol does not close it; that Phase 4.5 CALLS it is pinned by tests/lib/doc-consistency.test.mjs, not by this probe)",
|
||||
"closesWhen": "a named test PROVES the evidence block the conformance reviewer judges is BUILT IN CODE from real exit codes (behaviour probe: the gate RUNS tests/lib/criteria-runner.test.mjs, so a stub that only exports the symbol does not close it, and the named test must itself FELL a stub of lib/verification/criteria-runner.mjs, so an EMPTY test with the right name does not close it either; that Phase 4.5 CALLS it is pinned by tests/lib/doc-consistency.test.mjs, not by this probe)",
|
||||
"probe": "behaviour",
|
||||
"check": [
|
||||
{
|
||||
"test": "tests/lib/criteria-runner.test.mjs",
|
||||
"name": "formatCriteriaEvidence: one row per criterion, with command and exit code",
|
||||
"expect": "fails"
|
||||
"expect": "fails",
|
||||
"stubs": [
|
||||
"lib/verification/criteria-runner.mjs"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue