test(readme): the selftest numbers rot loudly, against the suites' own summaries

Order 20260905T053602Z-6743615726-from-.claude (.claude, 2026-09-05), asking
for the check this repo recommended when it re-measured the README a run
earlier. The badge and the five `## Development` comments rotted twice in a
row - 529 carried from 0.25.0, then a badge saying 868 beside comments summing
to 792, two different wrong sums of the same fact on the same screen - because
nothing compared them to anything.

It lives in tests/selftest.test.mjs, not in a bash suite, and the choice was
measured rather than assumed. The order's parenthetical pointed at whichever
suite already pins README/catalog invariants; no such suite exists -
`grep -ln README scripts/*selftest*.sh` returns board-selftest.sh alone, on two
incidental hits (a prose comment and a research/README.md fixture). This
wrapper is the only place where all five numbers exist at once in a run that
already happens. runSuite() captures each suite's own summary line, so the
truth source is the line the suite prints. A check inside one suite could see
its own total but would have to RE-RUN the other four - 212s sequentially,
measured 2026-09-05 under /bin/bash 3.2 (coord 16, board 169, route 12,
orders 5, guard 10) - and grepping `check` calls out of the scripts is both the
second copy of the counting the order warned against and a wrong one, since
those calls sit inside loops.

Three properties are deliberate. The badge is compared against the MEASURED
sum, never against the five README comments: a badge agreeing with five stale
comments is the 868-beside-792 shape one layer down. A suite that stops
printing a recognisable summary FAILS the check rather than being skipped -
an absent measurement must not read as a matching one. And the check adds no
bash check anywhere, so the five counts and the 893 badge are unchanged by its
arrival, exactly as the order expects; a counted self-check would have had to
compare against PASS+FAIL+1 and would break for whoever adds a check after it.

Ground truth on this HEAD, run before anything was written: coord 257,
board 393, route 73, orders 116, guard 54 = 893, 0 failed in all five - the
README was already correct, so the red step is the mutation. Mutation-verified
in both directions: 73 -> 74 on the route comment gives "README says
route-selftest has 74 checks; it reported 73"; 893 -> 894 on the badge gives
"README's badge says 894 selftest checks; the five suites reported 893";
restored, npm test is 12/12 green.

Bounded gap, stated rather than closed: CLAUDE.md's own copies of the five
counts are a second surface carrying the same numbers and are NOT checked.
Measured and left alone - widening the check to it was not ordered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-05 20:33:55 +02:00
commit a1ef1fb555
3 changed files with 127 additions and 8 deletions

View file

@ -1464,6 +1464,36 @@ obligations in another repo.
and re-measured after: 11 of 11 now match. Only the Install block was
converted; converting the other 39 indented blocks was not ordered and
would bury a two-line fix in a whole-file diff.
- **README's selftest numbers are pinned by `tests/selftest.test.mjs`, and the
check lives there rather than in one of the five bash suites for a measured
reason.** The badge and the five `## Development` comments rotted twice in a
row - 529 carried from 0.25.0, then a badge saying 868 beside comments summing
to 792, two different wrong sums of the same fact on the same screen - because
nothing compared them to anything. The order behind the fix
(`20260905T053602Z-6743615726`) suggested hosting it in whichever suite
already pins README/catalog invariants; measured before choosing, **no such
suite exists** (`grep -ln README scripts/*selftest*.sh` returns
`board-selftest.sh` alone, on two incidental hits - a prose comment and a
`research/README.md` fixture). This wrapper is the only place where all five
numbers exist at once in a run that already happens: `runSuite()` captures
each suite's own summary line, so the truth source is the line the suite
prints, never a re-count. A check inside one suite could see its own total but
would have to RE-RUN the other four (**212s sequentially**, measured
2026-09-05 under `/bin/bash` 3.2) to see theirs, and grepping `check` calls out
of the scripts is both the second copy of the counting and a wrong one, since
those calls sit inside loops. The badge is compared against the MEASURED sum,
not against the five README comments - a badge agreeing with five stale
comments is the 868-beside-792 shape one layer down - and a suite that stops
printing a summary FAILS the check rather than being skipped, because an
absent measurement must not read as a matching one. It adds no bash check, so
the five counts and the 893 badge are unchanged by its arrival.
Mutation-verified in both directions: `73 -> 74` on the route comment gives
"README says route-selftest has 74 checks; it reported 73", `893 -> 894` on
the badge gives "README's badge says 894 selftest checks; the five suites
reported 893", and the restored file is 12/12 green. **Bounded gap, stated
rather than closed:** `CLAUDE.md`'s own copies of the five counts (in this
section, above) are NOT checked - a second surface carrying the same numbers,
measured and left alone because widening the check to it was not ordered.
- Conventional Commits: `type(scope): description`.
## Commands
@ -1471,7 +1501,7 @@ obligations in another repo.
- Test: `bash scripts/coord-selftest.sh`, `bash scripts/board-selftest.sh`,
`bash scripts/route-selftest.sh`, `bash scripts/orders-selftest.sh` and
`bash scripts/state-line-guard-selftest.sh` (or `npm test`, the Node wrapper
around all five plus the hook tests)
around all five plus the hook tests and the README-number check)
- Order queue smoke test: `CLAUDE_COORD_DIR=$(mktemp -d) bash
scripts/coord-order-send.sh --to smoke --from tester --subject s --message m`
then `CLAUDE_COORD_DIR=<same> bash scripts/coord-order-inbox.sh --repo smoke`