feat(harness): the procedure rebuilt ten times gets the four controls it kept skipping

Point 7 decided: green-before/red-after becomes a standalone script, not a
pytest fixture and not prose. Ten hand-builds (2x oekt 23, 4x 31, 2x 32, 1x 33,
1x 37) is not the argument on its own -- typing was never the cost. The argument
is that four of its controls fail SILENTLY, and prose cannot enforce any:

  the anchor occurring exactly once, so the mutation lands on the seam alone;
  the node id having actually run -- MEASURED here, a mistyped id exits 4, which
    is non-zero and therefore MIMICS red to any harness asking "rc != 0?"
    (and `--collect-only -q` exits 0 on an id that does not exist, so that is
    not an existence check either);
  the restore read back FROM DISK and sha256-compared, because under .venv/
    nothing is tracked and `git status` never sees the mutation;
  the redness landing where the proof is about.

NOT a fixture. The suite runs every session, so a fixture that writes to disk
turns every interrupted run into a mutated tree -- and it would mutate modules
the running process already imported. The mutation class that CAN live in the
suite (mutate a copy in memory, call the guard directly) already does, as
test_guard_red_when_*, and needed no tool at all.

THE ORDERING QUESTION, ANSWERED RATHER THAN LEFT AN IMPLEMENTATION DETAIL.
"Positive controls before negatives" was never in tension with this procedure.
It only looked that way because "the control" was read as one thing when it is
three. --red must be green BEFORE (the measuring apparatus: the id resolves and
passes right now) and red AFTER (the measurement itself -- "can this go red at
all?" is not answerable until the mutation exists, so this is not a control
that ran late). --green must hold in BOTH runs; if it reds, the mutation landed
wider than the seam and the target's redness attributes to nothing. The order
follows from what each one measures.

VALUE-PROVED IN BOTH DIRECTIONS -- AND THE FIRST VERSION FAILED IT. Gating on
"is it an AssertionError?" rejected three of the four real proofs run against
the tool itself: a legitimately red test dies as AssertionError, as
`Failed: DID NOT RAISE`, or as a custom exception. That is modelling pytest
instead of reading it (oekt 28), and a gate that refuses real evidence gets
switched off, which is worse than none. Replaced with --red-at: the caller pins
the line, the tool checks the pin against pytest's real output, and an unpinned
red is reported with its type rather than silently blessed.

Five seams proved by the harness, each restored byte-identical: rc=4 read as
red · the anchor check disabled · the green-before check disabled · the restore
verification disabled · and the parity extension below. They died of
AssertionError, NotAValueProof, Failed, Failed and AssertionError -- the spread
that killed the first gate. Negative half: a docstring-only mutation stays green
and is refused as NOT a value proof; a mistyped id raises; a 16-occurrence
anchor is refused before a byte is written.

Two honesty guards caught this commit on the way in, both correctly. The README
anchor guard reddened on a new heading -- its ids are ground truth measured
against the published surface and may not be re-derived, so the heading became
bold text instead. The README/CLI parity guard reddened because the harness is a
second command-line surface that is neither a portfolio_optimiser_claude module
nor third-party tooling: its help is now captured too, with a positive control,
so its documented flags stay MEASURED rather than exempted.

923 -> 950 tests. ruff, mypy strict (src + scripts), full suite green, offline.
`git status` clean before and after every mutation run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-08-25 08:57:06 +02:00
commit c267bfa255
5 changed files with 925 additions and 2 deletions

View file

@ -45,7 +45,8 @@ Pakkehåndtering: `uv`. Pydantic for kontrakts-validering.
- Sync: `uv sync`
- Test: `uv run pytest`
- Lint: `uv run ruff check .` + `uv run ruff format .`
- Type: `uv run mypy src`
- Type: `uv run mypy src scripts`
- Bevis en søm: `uv run python scripts/mutation_harness.py --help` (grønn-før/rød-etter)
## Arbeidsflyt (invarianter)
- **Rent teknisk rammeverk:** deployer eier DPIA/ROS/behandlingsformål — kun tekniske
@ -54,6 +55,16 @@ Pakkehåndtering: `uv`. Pydantic for kontrakts-validering.
artefakt påstår mer enn implementasjonen gjør; skriptede stand-ins merkes som det.
- **Deterministisk validator obligatorisk + blokkerende** (§3 steg 4); golden-suiten
(`shared/examples/bygg-energi-mikro/golden.json`) er ENESTE fasit (§7).
- **Mutasjonsharnesset er verktøyet, ikke en huskeregel** (beslutning 2026-08-25, etter ti
håndbygg): `scripts/mutation_harness.py` kjører grønn-før/rød-etter mot det EKTE treet og
håndhever de fire kontrollene som ellers svikter stille — unikt anker, node-id-en faktisk
kjørt (pytest rc=4 er IKKE rødt), restaurering sha256-verifisert fra disk, og rødhet på en
pinnet linje. Tre roller, og rekkefølgen følger av hva hver måler: `--red` grønn FØR
(måleapparatets positivkontroll) og rød ETTER (selve målingen, ikke besvarbar før
mutasjonen finnes), `--green` grønn i BEGGE (treffer den, landet mutasjonen for bredt).
Verktøyet ligger bevisst UTENFOR `tests/` — suiten kjører hver økt, og en fixture som
skriver til disk gjør enhver avbrutt kjøring til et mutert tre. Mutasjon av en KOPI i minnet
hører derimot hjemme i suiten og er der alt (`test_guard_red_when_*`).
- **Load-bearing tester** (§11): hver søm bevises med en test som blir RØD når sømmen
detaches — grønn-men-død er feilmoden regelen finnes for.
- **Stoppkriterier + budsjett-tak påkrevd ved oppstart** (§8; `contracts.py` fail-fast).