feat(harness): the procedure rebuilt ten times gets the four controls it kept skipping

Point 7 decided: green-before/red-after becomes a standalone script, not a
pytest fixture and not prose. Ten hand-builds (2x oekt 23, 4x 31, 2x 32, 1x 33,
1x 37) is not the argument on its own -- typing was never the cost. The argument
is that four of its controls fail SILENTLY, and prose cannot enforce any:

  the anchor occurring exactly once, so the mutation lands on the seam alone;
  the node id having actually run -- MEASURED here, a mistyped id exits 4, which
    is non-zero and therefore MIMICS red to any harness asking "rc != 0?"
    (and `--collect-only -q` exits 0 on an id that does not exist, so that is
    not an existence check either);
  the restore read back FROM DISK and sha256-compared, because under .venv/
    nothing is tracked and `git status` never sees the mutation;
  the redness landing where the proof is about.

NOT a fixture. The suite runs every session, so a fixture that writes to disk
turns every interrupted run into a mutated tree -- and it would mutate modules
the running process already imported. The mutation class that CAN live in the
suite (mutate a copy in memory, call the guard directly) already does, as
test_guard_red_when_*, and needed no tool at all.

THE ORDERING QUESTION, ANSWERED RATHER THAN LEFT AN IMPLEMENTATION DETAIL.
"Positive controls before negatives" was never in tension with this procedure.
It only looked that way because "the control" was read as one thing when it is
three. --red must be green BEFORE (the measuring apparatus: the id resolves and
passes right now) and red AFTER (the measurement itself -- "can this go red at
all?" is not answerable until the mutation exists, so this is not a control
that ran late). --green must hold in BOTH runs; if it reds, the mutation landed
wider than the seam and the target's redness attributes to nothing. The order
follows from what each one measures.

VALUE-PROVED IN BOTH DIRECTIONS -- AND THE FIRST VERSION FAILED IT. Gating on
"is it an AssertionError?" rejected three of the four real proofs run against
the tool itself: a legitimately red test dies as AssertionError, as
`Failed: DID NOT RAISE`, or as a custom exception. That is modelling pytest
instead of reading it (oekt 28), and a gate that refuses real evidence gets
switched off, which is worse than none. Replaced with --red-at: the caller pins
the line, the tool checks the pin against pytest's real output, and an unpinned
red is reported with its type rather than silently blessed.

Five seams proved by the harness, each restored byte-identical: rc=4 read as
red · the anchor check disabled · the green-before check disabled · the restore
verification disabled · and the parity extension below. They died of
AssertionError, NotAValueProof, Failed, Failed and AssertionError -- the spread
that killed the first gate. Negative half: a docstring-only mutation stays green
and is refused as NOT a value proof; a mistyped id raises; a 16-occurrence
anchor is refused before a byte is written.

Two honesty guards caught this commit on the way in, both correctly. The README
anchor guard reddened on a new heading -- its ids are ground truth measured
against the published surface and may not be re-derived, so the heading became
bold text instead. The README/CLI parity guard reddened because the harness is a
second command-line surface that is neither a portfolio_optimiser_claude module
nor third-party tooling: its help is now captured too, with a positive control,
so its documented flags stay MEASURED rather than exempted.

923 -> 950 tests. ruff, mypy strict (src + scripts), full suite green, offline.
`git status` clean before and after every mutation run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-08-25 08:57:06 +02:00
commit c267bfa255
5 changed files with 925 additions and 2 deletions

View file

@ -351,6 +351,47 @@ rather than borrowing the installed one — red the moment the stamp reads the e
instead of the producing client — and the committed S10 record stays un-back-filled),
and `test_sdk_isolation.py` (local config cannot capture the checker).
**Proving it — `scripts/mutation_harness.py`.**
A test that *claims* to be load-bearing is a claim. The proof is to detach the seam, run the
test, and require it to go **red** — then restore the tree and verify the restore. That
procedure was hand-written ten times before it became a script, and the reason it became one
is not the typing. It is that four of its controls fail *silently*:
| Control | What it catches | Why prose cannot enforce it |
|---|---|---|
| Anchor occurs exactly once | A mutation landing wider than the seam | Nothing complains when it matches twice |
| The node id actually ran | A mistyped id — `pytest` exits **4**, which is non-zero and so *mimics* red | `--collect-only -q` exits **0** on an id that does not exist |
| Restore read back from disk and sha256-compared | A mutation left behind under `.venv/`, which `git status` does not track | The run still prints "restored" |
| Redness lands on a pinned line | A red caused by a broken import rather than by the seam | Any garbage edit reds a test |
Three roles, and the order follows from what each one measures — not from a convention:
- `--red` must be green **before** (the measuring apparatus: the id resolves and passes right
now) and red **after** (the measurement itself — *can this go red at all?* is not answerable
until the mutation exists).
- `--green` must hold in **both** runs. If a control reds, the mutation landed wider than the
seam and the target's redness attributes to nothing.
```bash
uv run python scripts/mutation_harness.py \
--target src/portfolio_optimiser_claude/some_module.py \
--anchor 'the exact text, unique in the file' \
--replacement 'what detaches the seam' \
--red tests/test_x_loadbearing.py::TestY::test_the_seam \
--green tests/test_x_loadbearing.py::TestY::test_the_population
```
It refuses rather than reports: an ambiguous anchor, a target already red, an id pytest never
collected, a control that reddened, and a mutation the target did not notice are each a named
error, never a quiet pass. `tests/test_mutation_harness.py` proves it in both directions — it
catches a known-detaching mutation and refuses to call a harmless one a proof.
The *other* mutation class — mutate a copy in memory and call the guard directly — belongs in
the suite and is already there (`test_guard_red_when_*`). This tool deliberately lives outside
`tests/`: the suite runs constantly, and a fixture that writes to disk would turn every
interrupted run into a mutated tree.
## The ingest layer — CSV and SQL, in front of the loop
The method spec forbids query-time retrieval against the bundle (§3 Step 1), so **data