open/repo-mailbox
1
0
Fork 0
Code Issues Releases 2 Activity

test(readme): the selftest numbers rot loudly, against the suites' own summaries

Browse source 
Order 20260905T053602Z-6743615726-from-.claude (.claude, 2026-09-05), asking
for the check this repo recommended when it re-measured the README a run
earlier. The badge and the five `## Development` comments rotted twice in a
row - 529 carried from 0.25.0, then a badge saying 868 beside comments summing
to 792, two different wrong sums of the same fact on the same screen - because
nothing compared them to anything.

It lives in tests/selftest.test.mjs, not in a bash suite, and the choice was
measured rather than assumed. The order's parenthetical pointed at whichever
suite already pins README/catalog invariants; no such suite exists -
`grep -ln README scripts/*selftest*.sh` returns board-selftest.sh alone, on two
incidental hits (a prose comment and a research/README.md fixture). This
wrapper is the only place where all five numbers exist at once in a run that
already happens. runSuite() captures each suite's own summary line, so the
truth source is the line the suite prints. A check inside one suite could see
its own total but would have to RE-RUN the other four - 212s sequentially,
measured 2026-09-05 under /bin/bash 3.2 (coord 16, board 169, route 12,
orders 5, guard 10) - and grepping `check` calls out of the scripts is both the
second copy of the counting the order warned against and a wrong one, since
those calls sit inside loops.

Three properties are deliberate. The badge is compared against the MEASURED
sum, never against the five README comments: a badge agreeing with five stale
comments is the 868-beside-792 shape one layer down. A suite that stops
printing a recognisable summary FAILS the check rather than being skipped -
an absent measurement must not read as a matching one. And the check adds no
bash check anywhere, so the five counts and the 893 badge are unchanged by its
arrival, exactly as the order expects; a counted self-check would have had to
compare against PASS+FAIL+1 and would break for whoever adds a check after it.

Ground truth on this HEAD, run before anything was written: coord 257,
board 393, route 73, orders 116, guard 54 = 893, 0 failed in all five - the
README was already correct, so the red step is the mutation. Mutation-verified
in both directions: 73 -> 74 on the route comment gives "README says
route-selftest has 74 checks; it reported 73"; 893 -> 894 on the badge gives
"README's badge says 894 selftest checks; the five suites reported 893";
restored, npm test is 12/12 green.

Bounded gap, stated rather than closed: CLAUDE.md's own copies of the five
counts are a second surface carrying the same numbers and are NOT checked.
Measured and left alone - widening the check to it was not ordered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-05 20:33:55 +02:00
commit a1ef1fb555
3 changed files with 127 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

32
CLAUDE.md
View file

@ -1464,6 +1464,36 @@ obligations in another repo.
and re-measured after: 11 of 11 now match. Only the Install block was
converted; converting the other 39 indented blocks was not ordered and
would bury a two-line fix in a whole-file diff.
- **README's selftest numbers are pinned by `tests/selftest.test.mjs`, and the
check lives there rather than in one of the five bash suites for a measured
reason.** The badge and the five `## Development` comments rotted twice in a
row - 529 carried from 0.25.0, then a badge saying 868 beside comments summing
to 792, two different wrong sums of the same fact on the same screen - because
nothing compared them to anything. The order behind the fix
(`20260905T053602Z-6743615726`) suggested hosting it in whichever suite
already pins README/catalog invariants; measured before choosing, **no such
suite exists** (`grep -ln README scripts/*selftest*.sh` returns
`board-selftest.sh` alone, on two incidental hits - a prose comment and a
`research/README.md` fixture). This wrapper is the only place where all five
numbers exist at once in a run that already happens: `runSuite()` captures
each suite's own summary line, so the truth source is the line the suite
prints, never a re-count. A check inside one suite could see its own total but
would have to RE-RUN the other four (**212s sequentially**, measured
2026-09-05 under `/bin/bash` 3.2) to see theirs, and grepping `check` calls out
of the scripts is both the second copy of the counting and a wrong one, since
those calls sit inside loops. The badge is compared against the MEASURED sum,
not against the five README comments - a badge agreeing with five stale
comments is the 868-beside-792 shape one layer down - and a suite that stops
printing a summary FAILS the check rather than being skipped, because an
absent measurement must not read as a matching one. It adds no bash check, so
the five counts and the 893 badge are unchanged by its arrival.
Mutation-verified in both directions: `73 -> 74` on the route comment gives
"README says route-selftest has 74 checks; it reported 73", `893 -> 894` on
the badge gives "README's badge says 894 selftest checks; the five suites
reported 893", and the restored file is 12/12 green. **Bounded gap, stated
rather than closed:** `CLAUDE.md`'s own copies of the five counts (in this
section, above) are NOT checked - a second surface carrying the same numbers,
measured and left alone because widening the check to it was not ordered.
- Conventional Commits: `type(scope): description`.
## Commands
@ -1471,7 +1501,7 @@ obligations in another repo.
- Test: `bash scripts/coord-selftest.sh`, `bash scripts/board-selftest.sh`,
`bash scripts/route-selftest.sh`, `bash scripts/orders-selftest.sh` and
`bash scripts/state-line-guard-selftest.sh` (or `npm test`, the Node wrapper
around all five plus the hook tests)
around all five plus the hook tests and the README-number check)
- Order queue smoke test: `CLAUDE_COORD_DIR=$(mktemp -d) bash
scripts/coord-order-send.sh --to smoke --from tester --subject s --message m`
then `CLAUDE_COORD_DIR=<same> bash scripts/coord-order-inbox.sh --repo smoke`

2
README.md
View file

@ -200,7 +200,7 @@ Note that raising the inbox's priority (Rule 7) deliberately does **not** widen
bash scripts/route-selftest.sh # 73 checks, incl. the route->board round trip
bash scripts/orders-selftest.sh # 116 checks, incl. the 20-way barriered claim race
bash scripts/state-line-guard-selftest.sh # 54 checks, incl. the Edit replace_all projection and the ratchet
npm test # all five selftests plus the hook tests, via node --test
npm test # all five selftests, the hook tests, and the README-number check
TDD is the house rule: every behavior change lands with a failing selftest check first.

101
tests/selftest.test.mjs
View file

@ -4,7 +4,7 @@
import { test } from 'node:test';
import assert from 'node:assert';
import { execFileSync } from 'node:child_process';
import { mkdtempSync, mkdirSync, writeFileSync, existsSync } from 'node:fs';
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, existsSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { basename, dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
@ -12,8 +12,31 @@ import { fileURLToPath } from 'node:url';
const root = join(dirname(fileURLToPath(import.meta.url)), '..');
const hook = join(root, 'hooks', 'scripts', 'session-start.mjs');
// Every bash suite already prints its own total on its last line, and this
// wrapper already runs all five. Capturing that line here is what makes the
// README's numbers testable without a SECOND copy of the counting: nothing
// re-counts `check` calls (loops make that undecidable anyway) and nothing
// re-runs a suite to read a number the run in progress is already printing.
// The five suites cost 212s sequentially, measured 2026-09-05 under /bin/bash
// 3.2 - the marginal cost of the README check is zero because it consumes a
// run that happens regardless.
const summaries = new Map();
function runSuite(name) {
const script = join(root, 'scripts', `${name}-selftest.sh`);
try {
summaries.set(name, execFileSync('bash', [script], { encoding: 'utf8' }));
} catch (err) {
// Record what the suite managed to print before failing, then let the
// failure through: a red suite must stay red here, and the README check
// below still gets a number to compare rather than a silent absence.
if (typeof err.stdout === 'string') summaries.set(name, err.stdout);
throw err;
}
}
test('coord bash selftest passes', () => {
execFileSync('bash', [join(root, 'scripts', 'coord-selftest.sh')], { encoding: 'utf8' });
runSuite('coord');
});
// board.sh reads this plugin's mailbox for its INN column, so the board ships
@ -22,7 +45,7 @@ test('coord bash selftest passes', () => {
// through CLAUDE_PLUGIN_ROOT, so a board.sh that exists only in
// ~/.claude/scripts/ would be missing on exactly the path production uses.
test('board bash selftest passes', () => {
execFileSync('bash', [join(root, 'scripts', 'board-selftest.sh')], { encoding: 'utf8' });
runSuite('board');
});
// route.sh is the WRITER for the next-cost field board.sh already reads, so its
@ -31,7 +54,7 @@ test('board bash selftest passes', () => {
// CLAUDE_PLUGIN_ROOT, and a calculator proven only elsewhere is unproven on the
// one path production uses.
test('route bash selftest passes', () => {
execFileSync('bash', [join(root, 'scripts', 'route-selftest.sh')], { encoding: 'utf8' });
runSuite('route');
});
// pre-state-line-guard.mjs is a PreToolUse hook, so like session-start.mjs it
@ -39,7 +62,7 @@ test('route bash selftest passes', () => {
// CLAUDE_PLUGIN_ROOT, and a guard proven only elsewhere is unproven on the
// path production actually runs.
test('state-line-guard bash selftest passes', () => {
execFileSync('bash', [join(root, 'scripts', 'state-line-guard-selftest.sh')], { encoding: 'utf8' });
runSuite('state-line-guard');
});
// The order queue is the second channel beside the mailbox, with the opposite
@ -48,7 +71,7 @@ test('state-line-guard bash selftest passes', () => {
// CLAUDE_PLUGIN_ROOT, so a queue proven only elsewhere is unproven where it
// runs.
test('orders bash selftest passes', () => {
execFileSync('bash', [join(root, 'scripts', 'orders-selftest.sh')], { encoding: 'utf8' });
runSuite('orders');
});
// The engine refuses to invent an identity from the cwd, but the hook is the
@ -188,3 +211,69 @@ test('hook keeps mail and orders in separate blocks, mail first', () => {
assert.ok(ctx.includes('UNTRUSTED DATA'), 'the mail block lost its authorization framing');
assert.ok(ctx.includes('OPERATOR-AUTHORIZED'), 'the order block lost its authorization framing');
});
// --- README's selftest numbers must rot loudly ------------------------------
//
// The badge and the five `## Development` comments are the only public claim
// about how much this engine is pinned by, and they are the number furthest
// from the meter: they rotted twice in a row (529 from 0.25.0; then a badge
// saying 868 beside comments summing to 792 - two different wrong sums of the
// same fact, neither matching the other, on the same screen). Nothing caught
// either, because nothing compared them to anything.
//
// It lives HERE rather than in one of the five bash suites, and the choice is
// not arbitrary. The order's parenthetical suggested the suite that already
// pins README/catalog invariants; measured before choosing, no such suite
// exists - `grep -ln README scripts/*selftest*.sh` returns board-selftest.sh
// alone, on two incidental hits (a prose comment and a `research/README.md`
// fixture). Of the places that could host it, this wrapper is the only one
// where all five numbers exist at once in a run that already happens: a check
// inside a suite could see its own count but would have to RE-RUN the other
// four (212s, measured 2026-09-05) to see theirs, and reading counters out of
// the scripts is the second copy of the counting this check was asked not to
// be. `check` calls sit inside loops, so a static count is not merely a second
// copy - it is a wrong one.
//
// The truth source is each suite's own summary line, verbatim, and a suite
// that stops printing one FAILS here rather than being skipped: an absent
// measurement must not read as a matching one.
function suiteTotal(name) {
const out = summaries.get(name);
assert.ok(out !== undefined,
`${name}-selftest produced no captured output: its total was never measured, ` +
'so the README comparison below would be resting on nothing');
// Two summary grammars, both already in the tree: coord prints
// `PASS=N FAIL=M`, the other four print `<name>-selftest: N passed, M failed`
// and orders adds `, S skipped (of T checks)`. README documents the TOTAL
// number of checks, so skipped ones count.
let m = out.match(/^\S+-selftest: (\d+) passed, (\d+) failed(?:, (\d+) skipped)?/m);
if (m) return Number(m[1]) + Number(m[2]) + Number(m[3] ?? 0);
m = out.match(/^PASS=(\d+) FAIL=(\d+)/m);
assert.ok(m, `${name}-selftest printed no summary line this parser recognises`);
return Number(m[1]) + Number(m[2]);
}
test('README states the selftest counts the suites actually reported', () => {
const readme = readFileSync(join(root, 'README.md'), 'utf8');
const suites = ['coord', 'board', 'route', 'orders', 'state-line-guard'];
let sum = 0;
for (const name of suites) {
const measured = suiteTotal(name);
sum += measured;
const line = readme.match(
new RegExp(`^\\s*bash scripts/${name}-selftest\\.sh\\s+#\\s+(\\d+) checks`, 'm'));
assert.ok(line,
`README's ## Development block has no "N checks" comment for ${name}-selftest.sh`);
assert.equal(Number(line[1]), measured,
`README says ${name}-selftest has ${line[1]} checks; it reported ${measured}`);
}
// The badge is the sum, and it is compared against the MEASURED total rather
// than against the five README comments: a badge agreeing with five stale
// comments is exactly the 868-beside-792 shape, one layer down.
const badge = readme.match(/badge\/selftest_checks-(\d+)-/);
assert.ok(badge, 'README has no selftest_checks badge to check');
assert.equal(Number(badge[1]), sum,
`README's badge says ${badge[1]} selftest checks; the five suites reported ${sum}`);
});