open/voyage
1
0
Fork 0
Code Issues Releases 5 Activity

fix(end-state): a behaviour probe must FELL a stub, not just name a test

Browse source 
The cosmetic close came back one level up. The probe reads the test file from
the same tree it measures, so a four-line file holding two EMPTY tests with the
two named names closed D-03 and D-04 on a tree where `lib/verification/` did
not exist at all - "defects 0 of 7, registry intact" (measured 2026-09-18).
First a two-line stub exporting the right symbols; then an empty test with the
right name. A name is always forgeable.

So the named test is now run twice. Once on the tree, as before - and once in a
sandbox where the module the condition declares in `stubs` is replaced by a
stub exporting the same names, all inert. If the test still passes there, it
binds the name and not the behaviour, and the condition THROWS: NOT FELLABLE,
counted open. A condition that declares no `stubs`, or names a module that is
not there, cannot fire either.

The sandbox is a symlink overlay: every entry of the tree is symlinked, and
only the test file and the stubbed module are materialised for real - Node
resolves an ESM import through the realpath, so a symlinked test file would
import the original module and never see the mutant. Nothing is ever written
inside the measured tree, and the only directory removed is the one this code
made under the system temp dir (pinned by a test).

M7 is now a permanent mutant beside M6, in two forms: the checkpoint's own
reproduction (unfixed tree + empty named tests) and the harder one (the real
module present, so the stub can be built and the empty test passes against it).
Both report `defects 2 of 7`. A positive control pins that D-03/D-04 still
CLOSE on the real tree, so "not closed" everywhere cannot read as a working
probe.

The frozen denominator moves a third time, deliberately, and its `why` no
longer claims authority it does not have: the second and third amendments were
maintenance decisions by the maintainer, not operator decisions, and the
tracked file now says exactly that.

Measured after:
  real tree      node scripts/end-state-gate.mjs -> defects 0 of 7, intact, exit 1
  M6 (stub)      8d1669e + current gate/registry/frozen + lib/cosmetic/stub.mjs -> 2 of 7
  M7 (empty)     8d1669e + current gate/registry/frozen + 3-line
                 tests/lib/criteria-runner.test.mjs with the two named tests -> 2 of 7
Red first: 4 of the new tests failed before the change. Suite 1154 (1152/0/2).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 02:56:22 +02:00
commit 83d82f121d
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 252 additions and 45 deletions
Download patch file Download diff file Expand all files Collapse all files

133
tests/scripts/end-state-gate.test.mjs
View file

@ -43,12 +43,17 @@ const FROZEN = {
'D-01': '48f7f5cca070091e8f9b9ca0c7cdb7a1880733691935c25d07ea1d3e8d508fc9',
'D-02': 'dfc94dba04a9116ae9be2097a0c5a9d8313b02c7de1ca6b49a472c1823a6d1e1',
// Amended 2026-09-18 (operator decision): D-03/D-04 became behaviour probes.
// Amended again the same day, by work order after the PM checkpoint: those
// Amended a second time the same day, as a maintenance decision: those
// behaviour probes were greps for an export name, and a two-line stub
// exporting both names closed them on an unfixed tree (mutant M6, pinned
// below). They now RUN a named test, so both signatures moved on purpose.
'D-03': '4eb4c7e447d927b389c7537c2e76d10e6c53ae46c12a61aed1c19adde2ffeb08',
'D-04': '32e3c2aba4b1dc8c41581bf849f39658552e4e8b251a40824bf9dd75c846d19a',
// below). They now RUN a named test.
// Amended a third time the same day, same standing: a named test can be
// EMPTY, and two empty tests with the right names closed both defects on a
// tree without lib/verification/ (mutant M7, pinned below). Each condition
// now also names the module whose STUB its test must fell. Every amendment
// moved a signature on purpose and removed nothing.
'D-03': '98698ce74c243353713786b8604160f6672568b090884e9428d9e5e9fbcad89d',
'D-04': 'df11a2e95db7b014ce077db0473e5dde4195ace92d8077cacdc1cee273947b47',
'D-05': '8fe6df52b43496c3302400507cb98002dcf8d2a12d92085e14bda4c7819bc7ba',
'D-06': '72a787c8154d1c18849e9856dcf1a67703d6b5541f88db81d6bab13cab14b418',
'D-07': '1bf3e0cd36c621720697475b0a3fd7db78611d66b58a9ce035e7695e57c69f1b',
@ -781,18 +786,24 @@ const M6_STUB = [
'',
].join('\n');
// A fixture pair: a module, and a test that BINDS it. `green one` passes on
// the tree and fails against a stub of the module - which is what makes it a
// behaviour probe rather than a name.
const MODULE = 'export function answer() { return 42; }\n';
const TEST_FILE = [
"import { test } from 'node:test';",
"import { strict as assert } from 'node:assert';",
"test('green one', () => { assert.ok(true); });",
"import { answer } from '../lib/m.mjs';",
"test('green one', () => { assert.equal(answer(), 42); });",
"test('red one', () => { assert.ok(false, 'red on purpose'); });",
'',
].join('\n');
test('a test condition RUNS the named test and reads its result', () => {
const dir = fixture({ 'tests/t.test.mjs': TEST_FILE });
const dir = fixture({ 'tests/t.test.mjs': TEST_FILE, 'lib/m.mjs': MODULE });
try {
const cond = (name, expect) => ({ test: 'tests/t.test.mjs', name, expect });
const cond = (name, expect) => ({ test: 'tests/t.test.mjs', name, expect, stubs: ['lib/m.mjs'] });
assert.equal(evaluateCondition(dir, cond('green one', 'passes')), true);
assert.equal(evaluateCondition(dir, cond('green one', 'fails')), false);
assert.equal(evaluateCondition(dir, cond('red one', 'passes')), false);
@ -801,11 +812,11 @@ test('a test condition RUNS the named test and reads its result', () => {
});
test('a test condition naming a test that does not run is NOT FELLABLE, never closed', () => {
const dir = fixture({ 'tests/t.test.mjs': TEST_FILE });
const dir = fixture({ 'tests/t.test.mjs': TEST_FILE, 'lib/m.mjs': MODULE });
try {
for (const cond of [
{ test: 'tests/t.test.mjs', name: 'no such test', expect: 'fails' },
{ test: 'tests/missing.test.mjs', name: 'green one', expect: 'fails' },
{ test: 'tests/t.test.mjs', name: 'no such test', expect: 'fails', stubs: ['lib/m.mjs'] },
{ test: 'tests/missing.test.mjs', name: 'green one', expect: 'fails', stubs: ['lib/m.mjs'] },
]) {
const r = evaluateCheck(dir, [cond]);
assert.equal(r.status, 'not-fellable', JSON.stringify(cond));
@ -835,3 +846,105 @@ test('the gate states out loud that the wiring is pinned by text, not proven', (
assert.match(r.stdout, /pinned by TEXT/);
assert.match(r.stdout, /plugin-eval/);
});
// --- M7: a probe must bind the test's CONTENT, not its name ---------------
//
// Measured 2026-09-18 (PM checkpoint on e1e7bdf): the cosmetic close came back
// one level up. The probe reads the test file from the SAME tree it measures,
// so a four-line file holding two EMPTY tests with the two named names —
// test('runPlanVerification: a plan whose success criterion FAILS fells the run', () => {});
// test('formatCriteriaEvidence: one row per criterion, with command and exit code', () => {});
// — closed D-03 and D-04 on a tree where `lib/verification/` did not exist at
// all: "defects 0 of 7, registry intact". An empty test is the new two-line
// stub. The only form that cannot be closed by a name is one that requires the
// named test to FELL a mutant: the gate runs it a second time against a tree
// where the module it binds is replaced by a stub exporting the same names and
// doing nothing. A test that passes against both is not a behaviour probe.
const EMPTY_NAMED_TESTS = [
"import test from 'node:test';",
"test('runPlanVerification: a plan whose success criterion FAILS fells the run', () => {});",
"test('formatCriteriaEvidence: one row per criterion, with command and exit code', () => {});",
'',
].join('\n');
const REAL_RUNNER = join(ROOT, 'lib', 'verification', 'criteria-runner.mjs');
const REAL_RUNNER_TEST = join(ROOT, 'tests', 'lib', 'criteria-runner.test.mjs');
test('M7 mutant: two EMPTY tests with the right names do NOT close D-03 or D-04', () => {
// The checkpoint's reproduction: an unfixed tree (no lib/verification/) plus
// the four-line test file.
const dir = fixture({ 'tests/lib/criteria-runner.test.mjs': EMPTY_NAMED_TESTS });
try {
const entries = loadRegistry(ROOT).defects.filter((e) => e.id === 'D-03' || e.id === 'D-04');
assert.equal(entries.length, 2, 'D-03 and D-04 are still in the registry');
for (const e of entries) {
const r = evaluateCheck(dir, e.check);
assert.equal(r.status, 'not-fellable', `${e.id} closed on a tree where nothing is fixed`);
assert.ok(r.detail.length > 0, 'the gate must say WHY the check could not fire');
}
} finally { cleanup(dir); }
});
test('M7 mutant: an empty named test does not close the defect even when the module IS present', () => {
// The harder case: the real module is there, so the stub CAN be built — and
// the empty test passes against it, which is exactly the proof that the test
// binds the name and nothing else.
const dir = fixture({
'tests/lib/criteria-runner.test.mjs': EMPTY_NAMED_TESTS,
'lib/verification/criteria-runner.mjs': readFileSync(REAL_RUNNER, 'utf8'),
});
try {
for (const e of loadRegistry(ROOT).defects.filter((x) => x.id === 'D-03' || x.id === 'D-04')) {
const r = evaluateCheck(dir, e.check);
assert.equal(r.status, 'not-fellable', `${e.id} closed on an empty test`);
assert.match(r.detail, /stub/i, 'the gate must name the mutant the test failed to fell');
}
} finally { cleanup(dir); }
});
test("D-03/D-04 still CLOSE on the real tree — the named tests do fell a stub", () => {
// The positive control. Without it, "not closed" everywhere would read as a
// working probe when the probe is simply broken.
for (const e of loadRegistry(ROOT).defects.filter((x) => x.id === 'D-03' || x.id === 'D-04')) {
const r = evaluateCheck(ROOT, e.check);
assert.equal(r.status, 'closed', `${e.id} is not closed on the real tree: ${r.detail}`);
}
});
test('a test condition must name the module its test binds, or it cannot fire', () => {
const dir = fixture({ 'tests/t.test.mjs': TEST_FILE, 'lib/m.mjs': MODULE });
try {
for (const cond of [
{ test: 'tests/t.test.mjs', name: 'green one', expect: 'passes' },
{ test: 'tests/t.test.mjs', name: 'green one', expect: 'passes', stubs: [] },
{ test: 'tests/t.test.mjs', name: 'green one', expect: 'passes', stubs: ['lib/gone.mjs'] },
]) {
const r = evaluateCheck(dir, [cond]);
assert.equal(r.status, 'not-fellable', JSON.stringify(cond));
assert.ok(r.detail.length > 0, 'the gate must say WHY');
}
} finally { cleanup(dir); }
});
test('the stub mutant leaves the measured tree untouched', () => {
const dir = fixture({ 'tests/t.test.mjs': TEST_FILE, 'lib/m.mjs': MODULE });
try {
const before = readFileSync(join(dir, 'lib', 'm.mjs'), 'utf8');
evaluateCheck(dir, [{ test: 'tests/t.test.mjs', name: 'green one', expect: 'passes', stubs: ['lib/m.mjs'] }]);
assert.equal(readFileSync(join(dir, 'lib', 'm.mjs'), 'utf8'), before, 'the mutation must happen in a copy, never in place');
} finally { cleanup(dir); }
});
test('the frozen file claims no authority it does not have', () => {
// MINOR from the same checkpoint: the tracked frozen file said the second
// amendment was "operator-authorised by work order". No operator authorised
// it; it followed from a maintenance decision. A file whose whole job is to
// be an explicit decision trail may not overclaim in either direction.
const why = loadFrozen(ROOT).why;
assert.ok(why.length > 0, 'the frozen manifest must say WHY it changed');
assert.ok(
!/operator-authoris|operator-authoriz|by work order|PM checkpoint/i.test(why),
'the frozen file may not claim operator authorisation, and coordination metadata belongs in the local plan',
);
});