open/portfolio-optimiser-claude
1
0
Fork 0
Code Issues Releases 1 Activity

feat(harness): the procedure rebuilt ten times gets the four controls it kept skipping

Browse source 
Point 7 decided: green-before/red-after becomes a standalone script, not a
pytest fixture and not prose. Ten hand-builds (2x oekt 23, 4x 31, 2x 32, 1x 33,
1x 37) is not the argument on its own -- typing was never the cost. The argument
is that four of its controls fail SILENTLY, and prose cannot enforce any:

  the anchor occurring exactly once, so the mutation lands on the seam alone;
  the node id having actually run -- MEASURED here, a mistyped id exits 4, which
    is non-zero and therefore MIMICS red to any harness asking "rc != 0?"
    (and `--collect-only -q` exits 0 on an id that does not exist, so that is
    not an existence check either);
  the restore read back FROM DISK and sha256-compared, because under .venv/
    nothing is tracked and `git status` never sees the mutation;
  the redness landing where the proof is about.

NOT a fixture. The suite runs every session, so a fixture that writes to disk
turns every interrupted run into a mutated tree -- and it would mutate modules
the running process already imported. The mutation class that CAN live in the
suite (mutate a copy in memory, call the guard directly) already does, as
test_guard_red_when_*, and needed no tool at all.

THE ORDERING QUESTION, ANSWERED RATHER THAN LEFT AN IMPLEMENTATION DETAIL.
"Positive controls before negatives" was never in tension with this procedure.
It only looked that way because "the control" was read as one thing when it is
three. --red must be green BEFORE (the measuring apparatus: the id resolves and
passes right now) and red AFTER (the measurement itself -- "can this go red at
all?" is not answerable until the mutation exists, so this is not a control
that ran late). --green must hold in BOTH runs; if it reds, the mutation landed
wider than the seam and the target's redness attributes to nothing. The order
follows from what each one measures.

VALUE-PROVED IN BOTH DIRECTIONS -- AND THE FIRST VERSION FAILED IT. Gating on
"is it an AssertionError?" rejected three of the four real proofs run against
the tool itself: a legitimately red test dies as AssertionError, as
`Failed: DID NOT RAISE`, or as a custom exception. That is modelling pytest
instead of reading it (oekt 28), and a gate that refuses real evidence gets
switched off, which is worse than none. Replaced with --red-at: the caller pins
the line, the tool checks the pin against pytest's real output, and an unpinned
red is reported with its type rather than silently blessed.

Five seams proved by the harness, each restored byte-identical: rc=4 read as
red · the anchor check disabled · the green-before check disabled · the restore
verification disabled · and the parity extension below. They died of
AssertionError, NotAValueProof, Failed, Failed and AssertionError -- the spread
that killed the first gate. Negative half: a docstring-only mutation stays green
and is refused as NOT a value proof; a mistyped id raises; a 16-occurrence
anchor is refused before a byte is written.

Two honesty guards caught this commit on the way in, both correctly. The README
anchor guard reddened on a new heading -- its ids are ground truth measured
against the published surface and may not be re-derived, so the heading became
bold text instead. The README/CLI parity guard reddened because the harness is a
second command-line surface that is neither a portfolio_optimiser_claude module
nor third-party tooling: its help is now captured too, with a positive control,
so its documented flags stay MEASURED rather than exempted.

923 -> 950 tests. ruff, mypy strict (src + scripts), full suite green, offline.
`git status` clean before and after every mutation run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-08-25 08:57:06 +02:00
commit c267bfa255
5 changed files with 925 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

13
CLAUDE.md
View file

@ -45,7 +45,8 @@ Pakkehåndtering: `uv`. Pydantic for kontrakts-validering.
- Sync: `uv sync`
- Test: `uv run pytest`
- Lint: `uv run ruff check .` + `uv run ruff format .`
- Type: `uv run mypy src`
- Type: `uv run mypy src scripts`
- Bevis en søm: `uv run python scripts/mutation_harness.py --help` (grønn-før/rød-etter)
## Arbeidsflyt (invarianter)
- **Rent teknisk rammeverk:** deployer eier DPIA/ROS/behandlingsformål — kun tekniske
@ -54,6 +55,16 @@ Pakkehåndtering: `uv`. Pydantic for kontrakts-validering.
artefakt påstår mer enn implementasjonen gjør; skriptede stand-ins merkes som det.
- **Deterministisk validator obligatorisk + blokkerende** (§3 steg 4); golden-suiten
(`shared/examples/bygg-energi-mikro/golden.json`) er ENESTE fasit (§7).
- **Mutasjonsharnesset er verktøyet, ikke en huskeregel** (beslutning 2026-08-25, etter ti
håndbygg): `scripts/mutation_harness.py` kjører grønn-før/rød-etter mot det EKTE treet og
håndhever de fire kontrollene som ellers svikter stille — unikt anker, node-id-en faktisk
kjørt (pytest rc=4 er IKKE rødt), restaurering sha256-verifisert fra disk, og rødhet på en
pinnet linje. Tre roller, og rekkefølgen følger av hva hver måler: `--red` grønn FØR
(måleapparatets positivkontroll) og rød ETTER (selve målingen, ikke besvarbar før
mutasjonen finnes), `--green` grønn i BEGGE (treffer den, landet mutasjonen for bredt).
Verktøyet ligger bevisst UTENFOR `tests/` — suiten kjører hver økt, og en fixture som
skriver til disk gjør enhver avbrutt kjøring til et mutert tre. Mutasjon av en KOPI i minnet
hører derimot hjemme i suiten og er der alt (`test_guard_red_when_*`).
- **Load-bearing tester** (§11): hver søm bevises med en test som blir RØD når sømmen
detaches — grønn-men-død er feilmoden regelen finnes for.
- **Stoppkriterier + budsjett-tak påkrevd ved oppstart** (§8; `contracts.py` fail-fast).

41
README.md
View file

@ -351,6 +351,47 @@ rather than borrowing the installed one — red the moment the stamp reads the e
instead of the producing client — and the committed S10 record stays un-back-filled),
and `test_sdk_isolation.py` (local config cannot capture the checker).
**Proving it — `scripts/mutation_harness.py`.**
A test that *claims* to be load-bearing is a claim. The proof is to detach the seam, run the
test, and require it to go **red** — then restore the tree and verify the restore. That
procedure was hand-written ten times before it became a script, and the reason it became one
is not the typing. It is that four of its controls fail *silently*:
| Control | What it catches | Why prose cannot enforce it |
|---|---|---|
| Anchor occurs exactly once | A mutation landing wider than the seam | Nothing complains when it matches twice |
| The node id actually ran | A mistyped id — `pytest` exits **4**, which is non-zero and so *mimics* red | `--collect-only -q` exits **0** on an id that does not exist |
| Restore read back from disk and sha256-compared | A mutation left behind under `.venv/`, which `git status` does not track | The run still prints "restored" |
| Redness lands on a pinned line | A red caused by a broken import rather than by the seam | Any garbage edit reds a test |
Three roles, and the order follows from what each one measures — not from a convention:
- `--red` must be green **before** (the measuring apparatus: the id resolves and passes right
now) and red **after** (the measurement itself — *can this go red at all?* is not answerable
until the mutation exists).
- `--green` must hold in **both** runs. If a control reds, the mutation landed wider than the
seam and the target's redness attributes to nothing.
```bash
uv run python scripts/mutation_harness.py \
--target src/portfolio_optimiser_claude/some_module.py \
--anchor 'the exact text, unique in the file' \
--replacement 'what detaches the seam' \
--red tests/test_x_loadbearing.py::TestY::test_the_seam \
--green tests/test_x_loadbearing.py::TestY::test_the_population
```
It refuses rather than reports: an ambiguous anchor, a target already red, an id pytest never
collected, a control that reddened, and a mutation the target did not notice are each a named
error, never a quiet pass. `tests/test_mutation_harness.py` proves it in both directions — it
catches a known-detaching mutation and refuses to call a harmless one a proof.
The *other* mutation class — mutate a copy in memory and call the guard directly — belongs in
the suite and is already there (`test_guard_red_when_*`). This tool deliberately lives outside
`tests/`: the suite runs constantly, and a fixture that writes to disk would turn every
interrupted run into a mutated tree.
## The ingest layer — CSV and SQL, in front of the loop
The method spec forbids query-time retrieval against the bundle (§3 Step 1), so **data

385
scripts/mutation_harness.py Executable file
View file

@ -0,0 +1,385 @@
#!/usr/bin/env python3
"""Green-before / red-after, with the four silent controls made loud.
This repo does not accept "the test is load-bearing" as a claim. It mutates
what the test rests on, runs the test, and requires it to go RED -- then
restores the tree and proves the restore was byte-identical. That procedure was
hand-built ten times before it became this file, and the reason it became a
file is not the typing. It is that four of its controls fail SILENTLY:
1. AN ANCHOR THAT IS NOT UNIQUE mutates more than the seam, so the redness is
not attributable to the seam.
2. A MISTYPED NODE ID never runs the test. pytest exits 4, which is non-zero,
so a harness that asks "rc != 0?" reads a typo as a proof. (MEASURED
2026-08-25: green rc=0, real failure rc=1, unknown id or broken collection
rc=4, nothing collected rc=5. And `--collect-only -q` exits 0 on an id
that does not exist, so it is not an existence check either.)
3. AN UNVERIFIED RESTORE leaves the tree mutated and says "restored". Under
`.venv/` nothing is tracked, so `git status` will not catch it.
4. RED ON THE WRONG LINE -- a NameError or a broken import -- is achievable by
any garbage edit and proves nothing about the seam.
Each one turns a broken query into a positive-looking fact, which is the defect
class the operating model names. Prose cannot enforce them; this can.
THREE ROLES, AND THE ORDER FOLLOWS FROM WHAT EACH ONE MEASURES. The rule
"positive controls before negatives" was never in tension with this procedure;
it only looked that way because "the control" was read as one thing when it is
three:
* BEFORE, mandatory -- the measuring apparatus. Every node id is collected and
GREEN right now. Without this, "red after" may be red because the id was
always wrong. This IS the positive control, and it runs first.
* AFTER, by definition -- the measurement. "Can this test go red at all?" is
not answerable until the mutation exists. This is not a control that ran
late; it is the thing being measured.
* BOTH, must stay green -- the population/collateral control. If it reds, the
mutation landed wider than the seam and the target's redness is void.
SCOPE. This is for mutating the REAL tree and running pytest as a subprocess.
The other mutation class -- mutate a copy in memory, call the guard directly --
already lives in the suite as ``test_guard_red_when_*`` and needs no tool. This
one deliberately does NOT live in the suite: the suite runs every session, and a
fixture that writes to disk turns every interrupted run into a mutated tree.
Usage:
python scripts/mutation_harness.py \
--target .venv/lib/python3.x/site-packages/some/module.py \
--anchor 'the exact text, unique in the file' \
--replacement 'what detaches the seam' \
--red tests/test_x.py::TestY::test_the_seam \
--green tests/test_x.py::TestY::test_the_population
"""
from __future__ import annotations
import argparse
import hashlib
import re
import subprocess
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import Literal, Mapping, Sequence
Verdict = Literal["green", "red", "not-collected"]
# MEASURED 2026-08-25, pytest 8, this repo's .venv. The point of the table is
# that 4 and 5 are NOT red -- they are the apparatus reporting that it never
# measured anything.
_GREEN = 0
_FAILED = 1
_USAGE_OR_COLLECTION_ERROR = 4
_NOTHING_COLLECTED = 5
# Reads the exception NAME off pytest's `E <dotted.Name>: <msg>` lines. Kept
# deliberately dumb: it reports what pytest emitted, and nothing downstream is
# allowed to gate on the value (see CONTROL 4). `E assert 1 == 2` has no
# colon-terminated name and correctly yields nothing.
_ERROR_LINE = re.compile(r"^E\s+([A-Za-z_][\w.]*)\s*:", re.MULTILINE)
class HarnessError(Exception):
"""Base for every refusal. A refusal is a result, never a silent pass."""
class AnchorNotUnique(HarnessError):
"""The anchor does not occur exactly once, so the mutation is not targeted."""
class NodeNotCollected(HarnessError):
"""pytest never ran the id. Non-zero here mimics red and must not be read as it."""
class NotGreenBefore(HarnessError):
"""A target was already red, so redness after is not caused by the mutation."""
class ControlWentRed(HarnessError):
"""A control reddened: the mutation landed wider than the seam."""
class NotAValueProof(HarnessError):
"""The mutation ran, and the seam did not care. That is a finding, not an error."""
class RestoreFailed(HarnessError):
"""The bytes on disk are not the bytes we started with. Loudest possible failure."""
@dataclass(frozen=True)
class Outcome:
node_id: str
returncode: int
verdict: Verdict
error_types: tuple[str, ...]
output: str = ""
@property
def summary(self) -> str:
kinds = ", ".join(self.error_types) if self.error_types else "no named exception"
return f"{self.verdict} (rc={self.returncode}, {kinds})"
@dataclass(frozen=True)
class Mutation:
target: Path
anchor: str
replacement: str
@dataclass(frozen=True)
class Report:
mutation: Mutation
before: tuple[Outcome, ...]
after: tuple[Outcome, ...]
sha256_before: str
sha256_after: str
targets: tuple[str, ...] = field(default_factory=tuple)
controls: tuple[str, ...] = field(default_factory=tuple)
pins: dict[str, str] = field(default_factory=dict)
@property
def after_by_id(self) -> dict[str, Outcome]:
return {o.node_id: o for o in self.after}
@property
def before_by_id(self) -> dict[str, Outcome]:
return {o.node_id: o for o in self.before}
@property
def value_proved(self) -> bool:
after = self.after_by_id
return bool(self.targets) and all(after[n].verdict == "red" for n in self.targets)
def classify(returncode: int, output: str, node_id: str = "") -> Outcome:
"""Map a pytest exit code to a verdict, keeping "never ran" out of "red".
The whole tool turns on this function. Everything else is bookkeeping.
"""
if returncode == _GREEN:
verdict: Verdict = "green"
elif returncode == _FAILED:
verdict = "red"
elif returncode in (_USAGE_OR_COLLECTION_ERROR, _NOTHING_COLLECTED):
verdict = "not-collected"
else:
# Unknown code: refuse to guess which side of the line it falls on.
verdict = "not-collected"
return Outcome(
node_id=node_id,
returncode=returncode,
verdict=verdict,
error_types=tuple(dict.fromkeys(m.rsplit(".", 1)[-1] for m in _ERROR_LINE.findall(output))),
output=output,
)
def run_node(node_id: str, cwd: Path) -> Outcome:
completed = subprocess.run(
[sys.executable, "-m", "pytest", node_id, "-q", "--no-header", "-p", "no:cacheprovider"],
cwd=cwd,
capture_output=True,
text=True,
)
return classify(completed.returncode, completed.stdout + completed.stderr, node_id)
def _sha256(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def _verify_restore(target: Path, original: bytes, expected_sha: str) -> str:
"""Write the original bytes back and prove from DISK that they are there.
Reads the file again rather than trusting the write, because the failure
this exists to catch is precisely a write that did not fully land.
"""
target.write_bytes(original)
actual = _sha256(target.read_bytes())
if actual != expected_sha:
raise RestoreFailed(
f"restore of {target} is NOT byte-identical: expected {expected_sha}, got {actual}. "
"The tree is dirty -- fix it by hand before trusting anything else."
)
return actual
def prove(
mutation: Mutation,
expect_red: Sequence[str],
expect_green: Sequence[str] = (),
*,
red_at: Mapping[str, str] | None = None,
cwd: Path | None = None,
) -> Report:
"""Run the full procedure, refusing loudly at every control it fails."""
targets = tuple(expect_red)
controls = tuple(expect_green)
pins = dict(red_at or {})
if not targets:
raise ValueError(
"a proof with no --red target reds nothing and proves nothing; name at least one"
)
work_dir = Path(cwd) if cwd is not None else Path.cwd()
target_file = mutation.target
# CONTROL 1 -- the anchor, checked before a single byte is written.
original = target_file.read_bytes()
text = original.decode("utf-8")
occurrences = text.count(mutation.anchor)
if occurrences != 1:
raise AnchorNotUnique(
f"anchor must occur exactly once in {target_file}; measured {occurrences} "
f"occurrence(s). A wider anchor mutates more than the seam."
)
sha_before = _sha256(original)
node_ids = list(dict.fromkeys([*targets, *controls]))
# CONTROL 2 -- the measuring apparatus, and it runs FIRST.
before = tuple(run_node(n, work_dir) for n in node_ids)
for outcome in before:
if outcome.verdict == "not-collected":
raise NodeNotCollected(
f"pytest never ran {outcome.node_id} (rc={outcome.returncode}). "
"That exit is non-zero but it is not red -- the id is wrong, or "
"collection is broken. Nothing measured."
)
if outcome.verdict == "red":
raise NotGreenBefore(
f"{outcome.node_id} was ALREADY red before the mutation, so red after "
"is not caused by it. Fix the tree, then measure."
)
mutated = text.replace(mutation.anchor, mutation.replacement, 1)
try:
target_file.write_text(mutated, encoding="utf-8")
after = tuple(run_node(n, work_dir) for n in node_ids)
finally:
# CONTROL 3 -- the restore, verified from disk, in `finally` or not at all.
sha_after = _verify_restore(target_file, original, sha_before)
report = Report(
mutation=mutation,
before=before,
after=after,
sha256_before=sha_before,
sha256_after=sha_after,
targets=targets,
controls=controls,
pins=pins,
)
results = report.after_by_id
for node_id in controls:
if results[node_id].verdict != "green":
raise ControlWentRed(
f"control {node_id} went {results[node_id].verdict} under the mutation. "
"The mutation landed wider than the seam, so the target's redness "
"attributes to nothing."
)
for node_id in targets:
outcome = results[node_id]
if outcome.verdict == "not-collected":
raise NodeNotCollected(
f"after the mutation, pytest could not collect {node_id} "
f"(rc={outcome.returncode}). Collection broke -- which any garbage edit "
"achieves. That is the loudest way to prove nothing."
)
if outcome.verdict == "green":
raise NotAValueProof(
f"{node_id} stayed GREEN under the mutation. The seam it claims to guard "
"is detached and the test did not notice -- green-but-dead, measured."
)
# CONTROL 4 -- red, but WHERE? Not derivable from the exception type: a
# legitimately red test may die as AssertionError, as `Failed: DID NOT
# RAISE`, or as a custom exception. So the caller PINS the line the
# proof is about and this checks the pin against pytest's real output.
# Unpinned, the type is reported instead of silently blessed.
pin = pins.get(node_id)
if pin is not None and pin not in outcome.output:
raise NotAValueProof(
f"{node_id} went red ({outcome.summary}), but the failure output does "
f"not contain the pinned location {pin!r}. It died somewhere else than "
"the line this proof is about."
)
return report
def _format(report: Report) -> str:
lines = [
f"VALUE-PROVED {report.mutation.target}",
f" anchor {report.mutation.anchor!r} -> {report.mutation.replacement!r}",
f" restored sha256 {report.sha256_after} (byte-identical, verified from disk)",
]
for node_id in report.targets:
outcome = report.after_by_id[node_id]
pinned = f", pinned at {report.pins[node_id]!r}" if node_id in report.pins else ""
lines.append(f" TARGET green before / red after {node_id}")
lines.append(f" died of: {outcome.summary}{pinned}")
for node_id in report.controls:
lines.append(f" CONTROL green before / green after {node_id}")
return "\n".join(lines)
def main(argv: Sequence[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="mutation_harness.py",
description="Prove a test is load-bearing: green before, red after, restore verified.",
)
parser.add_argument("--target", required=True, type=Path, help="file to mutate in place")
parser.add_argument("--anchor", required=True, help="exact text, must occur exactly once")
parser.add_argument("--replacement", required=True, help="what the anchor becomes")
parser.add_argument(
"--red",
action="append",
default=[],
metavar="NODE_ID",
help="test that must be green before and red after (repeatable)",
)
parser.add_argument(
"--green",
action="append",
default=[],
metavar="NODE_ID",
help="control that must stay green in both runs (repeatable)",
)
parser.add_argument(
"--red-at",
action="append",
default=[],
metavar="NODE_ID=TEXT",
help="require this text in that target's failure output (repeatable)",
)
parser.add_argument("--cwd", type=Path, default=None, help="directory to run pytest from")
args = parser.parse_args(argv)
try:
report = prove(
Mutation(target=args.target, anchor=args.anchor, replacement=args.replacement),
expect_red=args.red,
expect_green=args.green,
red_at=dict(pin.split("=", 1) for pin in args.red_at),
cwd=args.cwd,
)
except HarnessError as exc:
print(f"{type(exc).__name__}: {exc}")
if isinstance(exc, NotAValueProof):
print("NOT a value proof.")
return 1
except ValueError as exc:
print(f"ValueError: {exc}")
return 2
print(_format(report))
return 0
if __name__ == "__main__": # pragma: no cover
raise SystemExit(main())

44
tests/test_cli_paritet_loadbearing.py
View file

@ -25,9 +25,11 @@ from __future__ import annotations
import argparse
import contextlib
import importlib
import importlib.util
import io
import json
import re
import sys
from pathlib import Path
from typing import Callable
@ -558,6 +560,28 @@ def _full_help(module_name: str) -> str:
return text
def _harness_help() -> str:
"""Capture ``scripts/mutation_harness.py --help``.
Loaded from its published path rather than as a package module, because
that path IS the contract: the harness is a standalone script by design
(it must not run inside the suite it mutates for), so the README's claim
about its flags can only be checked against the file an operator runs.
"""
path = Path(__file__).resolve().parents[1] / "scripts" / "mutation_harness.py"
spec = importlib.util.spec_from_file_location("_harness_for_parity", path)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
# Register BEFORE exec: @dataclass resolves annotations through
# sys.modules[cls.__module__], which is None for an unregistered module.
sys.modules[spec.name] = module
spec.loader.exec_module(module)
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer), contextlib.suppress(SystemExit):
module.main(["--help"])
return buffer.getvalue()
def _readme_documented_modules() -> list[str]:
return sorted(set(_MODULE.findall(README.read_text(encoding="utf-8"))))
@ -587,14 +611,32 @@ class TestReadmeClaimsMatchTheCli:
def test_every_documented_flag_exists_in_a_documented_cli(self) -> None:
# RED the moment the README claims a flag the code does not offer —
# the drift K12 exists to close, kept closed from here on.
available = "\n".join(_full_help(name) for name in _readme_documented_modules())
available = "\n".join(
[*(_full_help(name) for name in _readme_documented_modules()), _harness_help()]
)
assert "--bundle" in available, "help capture is broken — the grep would be vacuous"
assert "--anchor" in available, (
"harness help missing — its README flags would be unmeasured"
)
undelivered = sorted(flag for flag in _readme_documented_flags() if flag not in available)
assert undelivered == [], (
f"README documents flags no CLI offers: {undelivered} — "
"either wire them or stop claiming them (§1)"
)
def test_the_harness_help_is_captured_and_the_detector_still_fires(self) -> None:
# The README documents a second command-line surface -- the mutation
# harness -- and it is NOT a `portfolio_optimiser_claude.*` module, so
# the module walk above cannot see it. It is also not third-party
# tooling, so the foreign-marker skip would be a lie. Capturing its
# --help keeps its documented flags MEASURED rather than exempted.
help_text = _harness_help()
assert "--target" in help_text, "harness help capture is broken -- the grep is vacuous"
assert "--red-at" in help_text
# POSITIVE CONTROL: the same query must be able to MISS, or the
# inclusion above proves nothing about flags the harness lacks.
assert "--tomorrows-flag" not in help_text
def test_the_operator_surfaces_are_all_documented(self) -> None:
# The other direction, bounded to the flags K12 promises the operator
# can drive from the command line: the run entrance's collecting

444
tests/test_mutation_harness.py Normal file
View file

@ -0,0 +1,444 @@
"""The green-before / red-after procedure, made a tool instead of a memory.
WHY THIS EXISTS. This repo proves a test is load-bearing by mutating what it
rests on and watching it go red. That procedure has been hand-built ten times
(2x oekt 23, 4x oekt 31, 2x oekt 32, 1x oekt 33, 1x oekt 37) and rewritten from
memory in a scratchpad every time. Four of its controls fail SILENTLY, which is
exactly the class STATE's operating model calls oensket 4 of the verification
law -- a negative result from a broken query, consumed as a positive fact:
1. the anchor is not unique, so the mutation lands somewhere else too;
2. the node id was mistyped, so pytest never ran the test and the non-zero
exit MIMICS red (MEASURED 2026-08-25: rc=4, not rc=1 -- distinguishable,
but only if something looks at the number);
3. the restore is not verified, so the tree keeps the mutation;
4. the test goes red on a NameError or a broken import rather than on the
assertion the proof is about (oekt 33: "not on a NameError, not on the
control").
Prose cannot enforce any of those. That is the whole argument for a tool.
WHAT THIS IS NOT. The other mutation class -- mutate a COPY held in memory and
call the guard directly -- already has a home in this suite and needs no tool
(``test_guard_red_when_*``, five files). This tool is for the class that mutates
the REAL tree and runs pytest as a subprocess, which is why it must NOT live
inside the suite as a fixture: the suite runs every session, and a fixture that
writes to disk makes every interrupted run a mutated tree.
So every test below runs against a SANDBOX built in ``tmp_path``. Nothing here
mutates this repo. That is a property of the design, not a precaution.
"""
from __future__ import annotations
import hashlib
import importlib.util
import sys
from pathlib import Path
from types import ModuleType
import pytest
HARNESS_PATH = Path(__file__).resolve().parents[1] / "scripts" / "mutation_harness.py"
def _load_harness() -> ModuleType:
"""Import the tool from its published path, not from a package alias.
The path is part of the contract: the order asked for a standalone script,
and a test that imported it through some other name would stay green if the
script moved out from under the operator running it by hand.
"""
assert HARNESS_PATH.is_file(), f"the harness is not at its documented path: {HARNESS_PATH}"
spec = importlib.util.spec_from_file_location("mutation_harness", HARNESS_PATH)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
sys.modules["mutation_harness"] = module
spec.loader.exec_module(module)
return module
mh = _load_harness()
# --------------------------------------------------------------------------
# The sandbox: a seam, a test that rests on it, and a control that does not.
# --------------------------------------------------------------------------
SEAM_SOURCE = '''\
"""A stand-in seam with one uniquely anchored behaviour."""
PREFIX = "mcp__"
def qualify(server: str, tool: str) -> str:
return f"{PREFIX}{server}__{tool}"
def population() -> list[str]:
return ["record_call", "flush_calls"]
'''
SEAM_TESTS = """\
from seam import population, qualify
def test_the_population_is_readable_and_is_two() -> None:
# CONTROL: green before AND after. If a mutation reds this, it landed wider
# than the seam and the target's redness proves nothing.
assert len(population()) == 2
def test_the_name_carries_the_server(){ANNOT} -> None:
# TARGET: green before, red after.
assert qualify("recorder", "record_call") == "mcp__recorder__record_call"
""".replace("{ANNOT}", "")
TARGET_ID = "test_seam.py::test_the_name_carries_the_server"
CONTROL_ID = "test_seam.py::test_the_population_is_readable_and_is_two"
@pytest.fixture()
def sandbox(tmp_path: Path) -> Path:
(tmp_path / "seam.py").write_text(SEAM_SOURCE, encoding="utf-8")
(tmp_path / "test_seam.py").write_text(SEAM_TESTS, encoding="utf-8")
return tmp_path
def _sha256(path: Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()
# --------------------------------------------------------------------------
# 1. Classification: the layer where oensket 4 actually bites.
# --------------------------------------------------------------------------
class TestExitCodesAreClassifiedNotJustTestedForZero:
def test_the_four_measured_exit_codes_map_to_three_distinct_verdicts(self) -> None:
# MEASURED 2026-08-25 against pytest 8 in this repo's .venv. Kept as a
# table so the mapping reads as the measurement, not as a belief.
assert mh.classify(0, "1 passed").verdict == "green"
assert mh.classify(1, "E AssertionError: nope").verdict == "red"
assert mh.classify(4, "no tests ran").verdict == "not-collected"
assert mh.classify(5, "no tests collected").verdict == "not-collected"
def test_a_mistyped_node_id_is_never_read_as_red(self) -> None:
# THE control this tool exists for. rc=4 is non-zero, so any harness
# that asks "rc != 0?" calls a typo a proof. This one cannot.
outcome = mh.classify(4, "no tests ran in 0.59s")
assert outcome.verdict == "not-collected"
assert outcome.verdict != "red"
def test_the_error_type_is_read_off_the_output_not_assumed(self) -> None:
assert mh.classify(1, "E AssertionError: x").error_types == ("AssertionError",)
assert mh.classify(1, "E NameError: name 'q' is not defined").error_types == (
"NameError",
)
def test_dotted_and_non_error_suffixed_failures_are_read_too(self) -> None:
# MEASURED, and it cost a rewrite. The first version of this tool gated
# on "is it an AssertionError?" and REJECTED three of the four real
# proofs run against itself, because a legitimately red test can die as
# `Failed: DID NOT RAISE`, or as a dotted custom exception. A gate that
# refuses real evidence gets switched off, which is worse than none --
# so the type is REPORTED, never used to model what pytest may emit
# (oekt 28: the fixture is the tool's emission, not a model of it).
assert mh.classify(1, "E Failed: DID NOT RAISE").error_types == ("Failed",)
assert mh.classify(1, "E mutation_harness.NotAValueProof: stayed green").error_types == (
"NotAValueProof",
)
assert mh.classify(1, "E assert 1 == 2").error_types == ()
# --------------------------------------------------------------------------
# 2. The anchor, checked before anything is written.
# --------------------------------------------------------------------------
class TestTheAnchorIsProvenUniqueBeforeTheFileIsTouched:
def test_an_ambiguous_anchor_is_refused_and_the_file_is_untouched(self, sandbox: Path) -> None:
target = sandbox / "seam.py"
before = _sha256(target)
with pytest.raises(mh.AnchorNotUnique) as excinfo:
mh.prove(
mh.Mutation(target=target, anchor="server", replacement="srv"),
expect_red=[TARGET_ID],
cwd=sandbox,
)
assert "3" in str(excinfo.value) or "occurrence" in str(excinfo.value).lower()
assert _sha256(target) == before, "refusal must happen BEFORE the write"
def test_an_absent_anchor_is_refused(self, sandbox: Path) -> None:
target = sandbox / "seam.py"
before = _sha256(target)
with pytest.raises(mh.AnchorNotUnique):
mh.prove(
mh.Mutation(target=target, anchor="not in this file", replacement="x"),
expect_red=[TARGET_ID],
cwd=sandbox,
)
assert _sha256(target) == before
def test_a_proof_with_no_target_is_vacuous_and_refused(self, sandbox: Path) -> None:
# A harness run that reds nothing proves nothing, and would otherwise
# report success. Refuse it rather than emit an empty proof.
with pytest.raises(ValueError):
mh.prove(
mh.Mutation(target=sandbox / "seam.py", anchor='PREFIX = "mcp__"', replacement=""),
expect_red=[],
cwd=sandbox,
)
# --------------------------------------------------------------------------
# 3. The value proof of the tool itself: it catches a known-red mutation and
# refuses to call a known-harmless one a proof.
# --------------------------------------------------------------------------
DETACHING = ('PREFIX = "mcp__"', 'PREFIX = ""')
HARMLESS = ("A stand-in seam", "A stand-in seam (comment touched)")
class TestTheToolIsValueProvedInBothDirections:
def test_a_detaching_mutation_is_caught_and_reported_as_a_value_proof(
self, sandbox: Path
) -> None:
target = sandbox / "seam.py"
report = mh.prove(
mh.Mutation(target=target, anchor=DETACHING[0], replacement=DETACHING[1]),
expect_red=[TARGET_ID],
expect_green=[CONTROL_ID],
cwd=sandbox,
)
assert report.value_proved
assert all(o.verdict == "green" for o in report.before)
assert report.after_by_id[TARGET_ID].verdict == "red"
assert report.after_by_id[TARGET_ID].error_types == ("AssertionError",)
assert report.after_by_id[CONTROL_ID].verdict == "green"
def test_a_harmless_mutation_is_not_flagged_as_a_proof(self, sandbox: Path) -> None:
# The other half of the value proof, and the one that is easy to skip:
# a tool that reports success on everything reports nothing.
target = sandbox / "seam.py"
with pytest.raises(mh.NotAValueProof) as excinfo:
mh.prove(
mh.Mutation(target=target, anchor=HARMLESS[0], replacement=HARMLESS[1]),
expect_red=[TARGET_ID],
expect_green=[CONTROL_ID],
cwd=sandbox,
)
assert TARGET_ID in str(excinfo.value)
assert "green" in str(excinfo.value).lower()
def test_the_target_must_be_green_before_or_it_is_not_a_measurement(
self, sandbox: Path
) -> None:
# POSITIVE CONTROL OF THE MEASURING APPARATUS, and it runs FIRST. If the
# target is already red, "red after" is not caused by the mutation.
(sandbox / "test_seam.py").write_text(
SEAM_TESTS.replace('== "mcp__recorder__record_call"', '== "already wrong"'),
encoding="utf-8",
)
with pytest.raises(mh.NotGreenBefore):
mh.prove(
mh.Mutation(
target=sandbox / "seam.py", anchor=DETACHING[0], replacement=DETACHING[1]
),
expect_red=[TARGET_ID],
cwd=sandbox,
)
def test_a_mistyped_node_id_raises_instead_of_proving_anything(self, sandbox: Path) -> None:
with pytest.raises(mh.NodeNotCollected) as excinfo:
mh.prove(
mh.Mutation(
target=sandbox / "seam.py", anchor=DETACHING[0], replacement=DETACHING[1]
),
expect_red=["test_seam.py::test_this_id_does_not_exist"],
cwd=sandbox,
)
assert "test_this_id_does_not_exist" in str(excinfo.value)
def test_a_mutation_that_breaks_the_import_is_not_a_value_proof(self, sandbox: Path) -> None:
# rc=4 AFTER the mutation means collection broke, which any garbage
# edit achieves. It is the loudest possible way to prove nothing.
with pytest.raises(mh.NodeNotCollected):
mh.prove(
mh.Mutation(
target=sandbox / "seam.py",
anchor='PREFIX = "mcp__"',
replacement="import definitely_not_a_module",
),
expect_red=[TARGET_ID],
cwd=sandbox,
)
def test_a_pinned_red_location_is_verified_against_the_real_output(self, sandbox: Path) -> None:
# oekt 33 said the redness must land on the line the proof is about
# ("not on a NameError, not on the control"). That is not derivable
# from the exception type -- it is derivable from WHERE it died. So the
# caller pins it and the tool checks the pin against pytest's actual
# output, rather than guessing from a type it does not control.
report = mh.prove(
mh.Mutation(target=sandbox / "seam.py", anchor=DETACHING[0], replacement=DETACHING[1]),
expect_red=[TARGET_ID],
red_at={TARGET_ID: "mcp__recorder__record_call"},
cwd=sandbox,
)
assert report.value_proved
def test_a_red_that_misses_the_pinned_location_is_refused(self, sandbox: Path) -> None:
with pytest.raises(mh.NotAValueProof) as excinfo:
mh.prove(
mh.Mutation(
target=sandbox / "seam.py", anchor=DETACHING[0], replacement=DETACHING[1]
),
expect_red=[TARGET_ID],
red_at={TARGET_ID: "a line this failure never prints"},
cwd=sandbox,
)
assert "a line this failure never prints" in str(excinfo.value)
def test_a_red_on_a_broken_module_is_still_reported_with_its_type(self, sandbox: Path) -> None:
# The honest limit, stated rather than papered over: a NameError inside
# a test that RAN is red, and the tool does not silently bless or
# reject it -- it names the type so the operator can see what died.
# (A mutation that breaks COLLECTION is a different case and is refused
# outright above: rc=4 is not red.)
report = mh.prove(
mh.Mutation(
target=sandbox / "seam.py",
anchor=' return f"{PREFIX}{server}__{tool}"',
replacement=" return undefined_name",
),
expect_red=[TARGET_ID],
cwd=sandbox,
)
assert report.after_by_id[TARGET_ID].error_types == ("NameError",)
assert "NameError" in mh._format(report)
def test_a_control_that_goes_red_fails_the_proof(self, sandbox: Path) -> None:
# The mutation lands wider than the seam: the population control reds
# too, so the target's redness is not attributable to the seam.
with pytest.raises(mh.ControlWentRed) as excinfo:
mh.prove(
mh.Mutation(
target=sandbox / "seam.py",
anchor='return ["record_call", "flush_calls"]',
replacement='return ["record_call"]',
),
expect_red=[CONTROL_ID],
expect_green=[CONTROL_ID],
cwd=sandbox,
)
assert CONTROL_ID in str(excinfo.value)
# --------------------------------------------------------------------------
# 4. The restore, verified in both directions, git or no git.
# --------------------------------------------------------------------------
class TestTheRestoreIsVerifiedNotAssumed:
def test_the_target_is_restored_byte_identical_after_a_successful_proof(
self, sandbox: Path
) -> None:
target = sandbox / "seam.py"
before = _sha256(target)
report = mh.prove(
mh.Mutation(target=target, anchor=DETACHING[0], replacement=DETACHING[1]),
expect_red=[TARGET_ID],
cwd=sandbox,
)
assert _sha256(target) == before
assert report.sha256_before == before == report.sha256_after
def test_the_target_is_restored_even_when_the_proof_raises(self, sandbox: Path) -> None:
# The restore lives in `finally` or it does not exist. Every failure
# path above would otherwise leave the tree mutated.
target = sandbox / "seam.py"
before = _sha256(target)
with pytest.raises(mh.NotAValueProof):
mh.prove(
mh.Mutation(target=target, anchor=HARMLESS[0], replacement=HARMLESS[1]),
expect_red=[TARGET_ID],
cwd=sandbox,
)
assert _sha256(target) == before, "a failed proof must not leave a mutated tree"
def test_the_restore_does_not_go_through_git(self, sandbox: Path) -> None:
# The .venv case (oekt 37): the target is not tracked, so `git status`
# would never catch the mutation and `git checkout --` cannot undo it.
# The harness holds the original bytes itself, which is why the same
# mechanism covers tracked and untracked targets alike.
assert not (sandbox / ".git").exists()
target = sandbox / "seam.py"
before = _sha256(target)
mh.prove(
mh.Mutation(target=target, anchor=DETACHING[0], replacement=DETACHING[1]),
expect_red=[TARGET_ID],
cwd=sandbox,
)
assert _sha256(target) == before
def test_a_corrupted_restore_is_reported_rather_than_passed_over(self, sandbox: Path) -> None:
# Force the one failure the sha256 check exists to catch. Without it,
# a partial write leaves a silently wrong tree and the run still says
# "restored".
target = sandbox / "seam.py"
with pytest.raises(mh.RestoreFailed):
mh._verify_restore(target, b"not what was written", "0" * 64)
# --------------------------------------------------------------------------
# 5. The command line, wired to the same code path the tests prove.
# --------------------------------------------------------------------------
class TestTheCommandLineReachesTheProvenCodePath:
def test_a_successful_proof_exits_zero_and_says_what_it_measured(
self, sandbox: Path, capsys: pytest.CaptureFixture[str]
) -> None:
code = mh.main(
[
"--target",
str(sandbox / "seam.py"),
"--anchor",
DETACHING[0],
"--replacement",
DETACHING[1],
"--red",
TARGET_ID,
"--green",
CONTROL_ID,
"--cwd",
str(sandbox),
]
)
out = capsys.readouterr().out
assert code == 0
assert "VALUE-PROVED" in out
assert TARGET_ID in out
assert _sha256(sandbox / "seam.py") in out, "the verified restore is part of the report"
def test_a_failed_proof_exits_non_zero_and_names_the_reason(
self, sandbox: Path, capsys: pytest.CaptureFixture[str]
) -> None:
code = mh.main(
[
"--target",
str(sandbox / "seam.py"),
"--anchor",
HARMLESS[0],
"--replacement",
HARMLESS[1],
"--red",
TARGET_ID,
"--cwd",
str(sandbox),
]
)
out = capsys.readouterr().out
assert code != 0
assert "NOT a value proof" in out or "NotAValueProof" in out