feat(harness): the procedure rebuilt ten times gets the four controls it kept skipping
Point 7 decided: green-before/red-after becomes a standalone script, not a
pytest fixture and not prose. Ten hand-builds (2x oekt 23, 4x 31, 2x 32, 1x 33,
1x 37) is not the argument on its own -- typing was never the cost. The argument
is that four of its controls fail SILENTLY, and prose cannot enforce any:
the anchor occurring exactly once, so the mutation lands on the seam alone;
the node id having actually run -- MEASURED here, a mistyped id exits 4, which
is non-zero and therefore MIMICS red to any harness asking "rc != 0?"
(and `--collect-only -q` exits 0 on an id that does not exist, so that is
not an existence check either);
the restore read back FROM DISK and sha256-compared, because under .venv/
nothing is tracked and `git status` never sees the mutation;
the redness landing where the proof is about.
NOT a fixture. The suite runs every session, so a fixture that writes to disk
turns every interrupted run into a mutated tree -- and it would mutate modules
the running process already imported. The mutation class that CAN live in the
suite (mutate a copy in memory, call the guard directly) already does, as
test_guard_red_when_*, and needed no tool at all.
THE ORDERING QUESTION, ANSWERED RATHER THAN LEFT AN IMPLEMENTATION DETAIL.
"Positive controls before negatives" was never in tension with this procedure.
It only looked that way because "the control" was read as one thing when it is
three. --red must be green BEFORE (the measuring apparatus: the id resolves and
passes right now) and red AFTER (the measurement itself -- "can this go red at
all?" is not answerable until the mutation exists, so this is not a control
that ran late). --green must hold in BOTH runs; if it reds, the mutation landed
wider than the seam and the target's redness attributes to nothing. The order
follows from what each one measures.
VALUE-PROVED IN BOTH DIRECTIONS -- AND THE FIRST VERSION FAILED IT. Gating on
"is it an AssertionError?" rejected three of the four real proofs run against
the tool itself: a legitimately red test dies as AssertionError, as
`Failed: DID NOT RAISE`, or as a custom exception. That is modelling pytest
instead of reading it (oekt 28), and a gate that refuses real evidence gets
switched off, which is worse than none. Replaced with --red-at: the caller pins
the line, the tool checks the pin against pytest's real output, and an unpinned
red is reported with its type rather than silently blessed.
Five seams proved by the harness, each restored byte-identical: rc=4 read as
red · the anchor check disabled · the green-before check disabled · the restore
verification disabled · and the parity extension below. They died of
AssertionError, NotAValueProof, Failed, Failed and AssertionError -- the spread
that killed the first gate. Negative half: a docstring-only mutation stays green
and is refused as NOT a value proof; a mistyped id raises; a 16-occurrence
anchor is refused before a byte is written.
Two honesty guards caught this commit on the way in, both correctly. The README
anchor guard reddened on a new heading -- its ids are ground truth measured
against the published surface and may not be re-derived, so the heading became
bold text instead. The README/CLI parity guard reddened because the harness is a
second command-line surface that is neither a portfolio_optimiser_claude module
nor third-party tooling: its help is now captured too, with a positive control,
so its documented flags stay MEASURED rather than exempted.
923 -> 950 tests. ruff, mypy strict (src + scripts), full suite green, offline.
`git status` clean before and after every mutation run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
56164f5f07
commit
c267bfa255
5 changed files with 925 additions and 2 deletions
|
|
@ -25,9 +25,11 @@ from __future__ import annotations
|
|||
import argparse
|
||||
import contextlib
|
||||
import importlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Callable
|
||||
|
||||
|
|
@ -558,6 +560,28 @@ def _full_help(module_name: str) -> str:
|
|||
return text
|
||||
|
||||
|
||||
def _harness_help() -> str:
|
||||
"""Capture ``scripts/mutation_harness.py --help``.
|
||||
|
||||
Loaded from its published path rather than as a package module, because
|
||||
that path IS the contract: the harness is a standalone script by design
|
||||
(it must not run inside the suite it mutates for), so the README's claim
|
||||
about its flags can only be checked against the file an operator runs.
|
||||
"""
|
||||
path = Path(__file__).resolve().parents[1] / "scripts" / "mutation_harness.py"
|
||||
spec = importlib.util.spec_from_file_location("_harness_for_parity", path)
|
||||
assert spec is not None and spec.loader is not None
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
# Register BEFORE exec: @dataclass resolves annotations through
|
||||
# sys.modules[cls.__module__], which is None for an unregistered module.
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer), contextlib.suppress(SystemExit):
|
||||
module.main(["--help"])
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
def _readme_documented_modules() -> list[str]:
|
||||
return sorted(set(_MODULE.findall(README.read_text(encoding="utf-8"))))
|
||||
|
||||
|
|
@ -587,14 +611,32 @@ class TestReadmeClaimsMatchTheCli:
|
|||
def test_every_documented_flag_exists_in_a_documented_cli(self) -> None:
|
||||
# RED the moment the README claims a flag the code does not offer —
|
||||
# the drift K12 exists to close, kept closed from here on.
|
||||
available = "\n".join(_full_help(name) for name in _readme_documented_modules())
|
||||
available = "\n".join(
|
||||
[*(_full_help(name) for name in _readme_documented_modules()), _harness_help()]
|
||||
)
|
||||
assert "--bundle" in available, "help capture is broken — the grep would be vacuous"
|
||||
assert "--anchor" in available, (
|
||||
"harness help missing — its README flags would be unmeasured"
|
||||
)
|
||||
undelivered = sorted(flag for flag in _readme_documented_flags() if flag not in available)
|
||||
assert undelivered == [], (
|
||||
f"README documents flags no CLI offers: {undelivered} — "
|
||||
"either wire them or stop claiming them (§1)"
|
||||
)
|
||||
|
||||
def test_the_harness_help_is_captured_and_the_detector_still_fires(self) -> None:
|
||||
# The README documents a second command-line surface -- the mutation
|
||||
# harness -- and it is NOT a `portfolio_optimiser_claude.*` module, so
|
||||
# the module walk above cannot see it. It is also not third-party
|
||||
# tooling, so the foreign-marker skip would be a lie. Capturing its
|
||||
# --help keeps its documented flags MEASURED rather than exempted.
|
||||
help_text = _harness_help()
|
||||
assert "--target" in help_text, "harness help capture is broken -- the grep is vacuous"
|
||||
assert "--red-at" in help_text
|
||||
# POSITIVE CONTROL: the same query must be able to MISS, or the
|
||||
# inclusion above proves nothing about flags the harness lacks.
|
||||
assert "--tomorrows-flag" not in help_text
|
||||
|
||||
def test_the_operator_surfaces_are_all_documented(self) -> None:
|
||||
# The other direction, bounded to the flags K12 promises the operator
|
||||
# can drive from the command line: the run entrance's collecting
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue