open/portfolio-optimiser-claude
1
0
Fork 0
Code Issues Releases 1 Activity

test(validator): the gate was pinned in its arithmetic, not in its decision [mutation-ok]

Browse source 
[mutation-ok] rationale: no mutation is pinned in code. `src/` is byte-identical
to HEAD (git diff HEAD -- src/ is empty; `if False` occurs 0 times under src/).
The three matches the guard found are PROSE in the docs note and the test file's
docstring, quoting the detach mutation that was run and restored. Every harness
run in this session verified its restore by sha256 from disk.

S2.7 is the D7 mirroring queue's topmost unmeasured candidate. The MAF sibling
tightened its validator in two halves -- (a) a structural block on
`claimed > nominal_feasible`, (b) an IR invariant `low <= unit_cost <= high`.
Both halves are GATED here on D-A pkt. 1 + a commons pull, and the defect they
answer is confirmed on our side as C-F2. So neither is built. The question this
answers is the one that is answerable offline: is today's boundary -- "the ONE
numeric gate is p90" -- load-bearing?

One rule, structurally: a claim is numerically bounded in exactly two places in
src, each with its own spec role (ir.py:50 §7.1, validator.py:68 §3 Step 4).
The sibling's drift shape does not exist here.

But the coverage splits cleanly across the rule. Measured with
scripts/mutation_harness.py, denominator tests/ (all 955), each run
sha256-restored: everything the gate COMPUTES is red, because the golden
fixture freezes it -- policy cap, band branch, band endpoint order, MC seed,
p90 cut point, nominal_feasible. Everything the gate DECIDES WITH is
green-but-dead -- bound to p10, bound to nominal_feasible, and loosening the
comparison each left all 955 green. The golden freezes what the validator
produces, so it cannot help with the one thing it does not observe: which bound
the gate reads. Swapping p90 for nominal_feasible IS the gated S2.7 half (a),
and it would have landed with the suite green, before D-A was decided.

Closed by tests/test_validator_gate_loadbearing.py (8 tests, 955 -> 963). Each
clause is green before and red after exactly its own mutation, with the golden
figures green in BOTH runs -- which shows mechanically that the mutation moved
the decision, not the arithmetic. The two IR tests are pinned with --red-at
against the invariant's own message, since they die in a helper. The AST
population control was proved against a BEHAVIOUR-PRESERVING mutation (the gate
widened to a logically equivalent conjunction) with all three behavioural
controls green: a new gate site is invisible to any behavioural test, which is
why it is there.

Two things the measurement gave in addition. Under the containment mutation the
golden test stayed green, confirming mechanically that half (b) is
golden-compatible when D-A lands. And the IR carries no ORDER on band endpoints
either -- strictly more than C-F2 names: (1.40, 0.70) is accepted, and while
random.uniform still draws from [0.70, 1.40], it walks the seeded stream
backwards, which is a different p90 (120456.91 against 121057.09).

Honest limit: pinning that a claim above nominal_feasible validates today is not
an endorsement of it. C-F2 calls that a MAJOR spec-level defect and the fix is
gated, not declined. These tests make the gated work arrive as a visible red
test and a decision, never as a silent swap. No src change, no spec text
touched, the fasit untouched.

Dated under the D7 frame: work AFTER 2026-08-09, not independent convergence.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 00:07:19 +02:00
commit 544655b4c8
2 changed files with 287 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

220
tests/test_validator_gate_loadbearing.py Normal file
View file

@ -0,0 +1,220 @@
"""S2.7: the validator's numeric gate — is the DECISION pinned, or only the ARITHMETIC?
The D7 mirroring queue (``docs/2026-08-18-funn-koeer-og-gater.md § D7-speilingskøen``)
carries S2.7 because the MAF sibling tightened its validator in two halves: (a) a
structural block on ``claimed > nominal_feasible``, and (b) an IR invariant
``low <= unit_cost <= high``. Both halves are GATED here on D-A pkt. 1 + a commons
pull (parity plan row 12), and the defect they answer is confirmed on our side as
C-F2 (``docs/review-2026-07.md``). So this file does NOT build either half. The
mirroring question it answers is the one that is answerable offline today: **is
today's boundary — "the ONE numeric gate is p90" — load-bearing?**
MEASURED 2026-09-07 with ``scripts/mutation_harness.py``, denominator ``tests/``
(the whole suite, 955 tests), every run restored sha256-verified:
- Detaching the gate outright (``if ... > p90:`` -> ``if False:``) is RED.
``test_validator.py`` catches it. The seam has an edge guard.
- Every clause of WHAT THE GATE COMPUTES is RED, and the golden fixture is why:
the policy cap (``0.30`` -> ``0.31``), the band branch (ignore bands), the band
endpoint order, the Monte Carlo seed, the p90 cut point index, and the
``nominal_feasible`` formula each reddened the suite.
- Every clause of WHAT THE GATE DECIDES WITH is GREEN-BUT-DEAD. Swapping the bound
to ``p10``, swapping it to ``nominal_feasible``, and loosening the comparison from
``>`` to ``>=`` each left all 955 tests green. Nothing noticed.
That split is the finding, and it is sharper than "the seam is thin": the golden
fixture freezes every number the validator PRODUCES, so it cannot help with the one
thing it does not observe — which bound the gate reads. The consequence is concrete.
Swapping ``p90`` for ``nominal_feasible`` IS the gated S2.7 half (a). It would land
with the suite green, silently, before D-A has been decided — and a gate change that
arrives without a red test is exactly what this repo's §11 rule exists to prevent.
The mutations were shown to change behaviour before their greens were read as holes
(the økt 39 trap: the harness reports a no-op and an undetected seam identically).
All three boundary inputs are reachable and were run: a claim exactly at p90 exists
(degenerate band, 0.30 x 1000 == 300.0), and with the golden band a claim of 100 000
validates while ``nominal_feasible`` is 90 000 — C-F2's first counterexample,
reproduced live at p90 = 121 057.09, together with its second (claim 55 000 against
nominal 30 000, p90 = 65 058.49).
HONEST LIMIT — what this does NOT say. Pinning that a claim above ``nominal_feasible``
validates TODAY is not an endorsement of that behaviour; C-F2 calls it a MAJOR
spec-level defect and the fix is gated, not declined. These tests pin the boundary so
that the gated work must arrive as a visible red test and a decision, never as a
silent swap. Nor do they touch the sibling's code or the frozen spec. And the band
tests below assert only that today's IR ACCEPTS a band that excludes ``unit_cost``;
the golden bundle's own band ([0.70, 1.40]) encloses it, so half (b) remains
golden-compatible when D-A lands.
Dated under the D7 frame: this is work AFTER 2026-08-09 and must NOT be read as
independent convergence with the sibling.
"""
from __future__ import annotations
import ast
from pathlib import Path
import pytest
from pydantic import ValidationError
from portfolio_optimiser_claude.ir import SavingsProposal
from portfolio_optimiser_claude.validator import (
Rejection,
ValidatedProposal,
validate_proposal,
)
SRC_PKG = Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser_claude"
# The golden bundle's own shape (shared/examples/bygg-energi-mikro/validator-input.json):
# one item, a real spread band, so p10 < nominal_feasible < p90 are three DISTINCT
# bounds and a test can tell which one the gate reads. A degenerate band collapses
# all three onto the same number, which is why test_validator.py's fixture — correct
# for what it pins — cannot answer this question.
_QUANTITY = 300_000.0
_UNIT_COST = 1.0
_BAND = (0.70, 1.40)
def _proposal(claimed: float, band: tuple[float, float] = _BAND) -> SavingsProposal:
"""Build through the CONSTRUCTOR, never ``model_copy``.
``model_copy(update=...)`` does not re-run pydantic's model validators, so a
proposal assembled that way would sail past the very invariant the band tests
below are pinned against — a layer you do not run is not a seam (økt 37).
"""
return SavingsProposal(
project_id="BYGG-KONTOR-NORD",
measure="S2.7 gate probe",
affected_items=[{"code": "EL", "quantity": _QUANTITY, "unit_cost": _UNIT_COST}],
claimed_saving_nok=claimed,
assumptions={"EL": band},
)
def _bounds() -> ValidatedProposal:
"""Learn today's bounds from the validator itself, then pin the DECISION to them.
Deliberately not hardcoded: the numbers are the golden suite's job (§7), and
reading them from a run keeps every test below attributable to which bound the
gate READS, never to what it computes. The percentiles do not depend on the
claim, so probing with a token claim is sound.
"""
outcome = validate_proposal(_proposal(1.0))
assert isinstance(outcome, ValidatedProposal), "the probe must validate to expose bounds"
return outcome
class TestThereIsExactlyOneNumericGateOnAClaim:
"""The population control, and it runs first: every clause below is worth only
as much as the claim that there is exactly ONE place where a gate can be moved."""
def test_a_claim_is_bounded_in_exactly_two_places_with_distinct_roles(self) -> None:
# The sibling's S2.7 shape made measurable here. Two numeric bounds on a
# claim exist BY DESIGN and answer different specs: the schema invariant
# (claim <= the items' own total, §7.1, ir.py) and the validator gate
# (claim <= the optimistic feasible bound, §3 Step 4, validator.py). RED the
# day a third appears somewhere else in src — a claim gated in a place no
# one is looking is how a weak gate hides. Read from the AST, not prose.
sites: dict[str, int] = {}
for module in sorted(SRC_PKG.glob("*.py")):
tree = ast.parse(module.read_text(encoding="utf-8"))
found = [
node
for node in ast.walk(tree)
if isinstance(node, ast.Compare)
and isinstance(node.left, ast.Attribute)
and node.left.attr == "claimed_saving_nok"
]
if found:
sites[module.name] = len(found)
assert sites == {"ir.py": 1, "validator.py": 1}, (
f"a claim is now numerically bounded in {sites} — the two-rule design "
"(schema invariant + validator gate) has drifted"
)
def test_the_gate_rejects_a_claim_above_the_optimistic_bound(self) -> None:
# POSITIVE CONTROL FOR EVERY NEGATIVE BELOW. Each test that follows asserts
# that some claim VALIDATES; that is vacuous unless rejection demonstrably
# happens at all on this path and on this band. RED under detaching the gate
# (`if ... > p90:` -> `if False:`).
outcome = validate_proposal(_proposal(_bounds().p90 * 1.05))
assert isinstance(outcome, Rejection)
class TestWhichBoundTheGateReadsIsLoadBearing:
"""One test per clause of ``if proposal.claimed_saving_nok > p90:`` — each proved
RED under exactly the mutation it names, with the golden figures green in both
runs (the mutation moved the DECISION, not the arithmetic)."""
def test_the_bound_is_the_optimistic_p90_not_the_nominal_feasible(self) -> None:
# RED under `> p90` -> `> nominal_feasible`, which IS the gated S2.7 half (a).
# C-F2's counterexample, pinned as today's boundary: a self-authored band can
# carry a claim above the closed-form nominal bound and it validates. This
# test does not bless that; it makes the gated fix cost a visible red test
# instead of arriving silently.
bounds = _bounds()
assert bounds.nominal_feasible < bounds.p90, "non-vacuity: the two bounds must differ"
claimed = (bounds.nominal_feasible + bounds.p90) / 2
outcome = validate_proposal(_proposal(claimed))
assert isinstance(outcome, ValidatedProposal)
assert outcome.claimed_saving_nok > outcome.nominal_feasible
def test_the_bound_is_the_optimistic_p90_not_a_central_percentile(self) -> None:
# RED under `> p90` -> `> p50` (and under `> p10`). §3 Step 4 makes the gate
# the OPTIMISTIC edge of the simulation: it blocks what is infeasible even at
# the favourable end, and deliberately does not arbitrate the middle. Sliding
# the bound down to a central percentile is a different policy, not a fix.
bounds = _bounds()
assert bounds.p50 < bounds.p90, "non-vacuity: the two bounds must differ"
outcome = validate_proposal(_proposal((bounds.p50 + bounds.p90) / 2))
assert isinstance(outcome, ValidatedProposal)
def test_the_comparison_is_strict_so_a_claim_exactly_at_p90_validates(self) -> None:
# RED under `>` -> `>=`. The bound is INCLUSIVE: p90 is a feasible value, not
# the first infeasible one, and `test_bygg_energi_mikro.py` states the same
# boundary from the other side (`claimed_saving_nok <= p90`). Reachable, not
# theoretical: a degenerate band collapses p90 onto 0.30 x the fixed total.
bounds = _bounds()
outcome = validate_proposal(_proposal(bounds.p90))
assert isinstance(outcome, ValidatedProposal)
assert outcome.claimed_saving_nok == outcome.p90
class TestTodaysIRAcceptsABandItDoesNotCheck:
"""S2.7 half (b) — the invariant ``low <= unit_cost <= high`` does NOT exist here.
Pinned as today's boundary so the gated work cannot land silently, each proved RED
under the ir.py mutation that adds exactly the invariant it names."""
def test_a_band_that_excludes_the_unit_cost_is_accepted_today(self) -> None:
# RED under adding `low <= unit_cost <= high` to the IR's model validator.
# C-F2's second counterexample: a band entirely ABOVE unit_cost inflates every
# sample, so the optimistic bound is drawn from costs the item never had.
excluding_band = (1.8, 2.2)
assert not (excluding_band[0] <= _UNIT_COST <= excluding_band[1]), "non-vacuity"
outcome = validate_proposal(_proposal(1.0, band=excluding_band))
assert isinstance(outcome, ValidatedProposal)
assert outcome.p90 > _bounds().p90
def test_a_reversed_band_is_accepted_today(self) -> None:
# RED under adding `low <= high` to the IR's model validator. Strictly
# stronger than C-F2, which only names containment: the endpoints carry no
# ORDER either. `random.uniform(1.40, 0.70)` still draws from [0.70, 1.40],
# so this does not corrupt the range — it walks the seeded stream backwards,
# which is a DIFFERENT p90 (measured: 120 456.91 against 121 057.09). A band
# whose meaning depends on argument order is not yet a band.
reversed_band = (_BAND[1], _BAND[0])
assert reversed_band[0] > reversed_band[1], "non-vacuity"
outcome = validate_proposal(_proposal(1.0, band=reversed_band))
assert isinstance(outcome, ValidatedProposal)
assert outcome.p90 != pytest.approx(_bounds().p90)
def test_the_ir_does_still_refuse_a_malformed_band(self) -> None:
# POSITIVE CONTROL for the two negatives above: "the IR accepts it" is only
# a finding if the IR refuses anything at all about bands. It refuses
# non-finite and negative endpoints (R-2 hardening, ir.py `_BandEndpoint`).
with pytest.raises(ValidationError):
_proposal(1.0, band=(-1.0, 1.4))
with pytest.raises(ValidationError):
_proposal(1.0, band=(0.7, float("inf")))