open/portfolio-optimiser-claude
1
0
Fork 0
Code Issues Releases 1 Activity

test(baseline): the baseline was loaded, and the judge could never reach it

Browse source 
S4.0, the fifth D7 mirroring candidate, measured with scripts/mutation_harness.py
against the real tree. C-F3 says "nothing in the bundle format carries a cost
baseline to reconcile against". Measured, that is too strong: validator-input.json
carries ENERGI-TOTAL-EL at 300 000 NOK and it is loaded on every run path. The
defect is not that the baseline is missing -- it is that no path connects it to
the judge.

A SavingsProposal comes into being in exactly three places (model-authored parse,
bundle baseline, re-read output; AST-measured over all 27 src files, positive
control: the same query finds validate_proposal's one call site). Only the first
reaches the validator. The only field read off the baseline directly is
project_id; the quantities, unit costs and uncertainty bands are schema-validated
and never read again.

Seven mutations, every one green-before/red-after with the restore sha256-verified
from disk:

  M1 baseline swapped for the weakest schema-valid variant -- 3 targets RED,
     all 974 pre-existing tests GREEN in both runs
  M2 the carried annotation narrowed to object -- AST ratchet RED, behaviour GREEN
  M3 the S10 path stops loading the baseline -- loader population RED
  M4 the validator gains a baseline parameter -- signature ratchet RED, golden GREEN
  M5 the call-site argument becomes a keyword -- provenance control RED
  M6 fail-closed code-set gate (the gated fix's own shape) -- boundary RED, golden GREEN
  M7 retrieval stops reading the codes -- disjointness RED

M4 and M6 leaving the golden green is itself the finding that S4.0 is
golden-compatible when D-A pkt. 2 lands.

A measuring trap was caught by the harness rather than by memory: the first M1
flattened the cost lines to 1.0x1.0, which broke the IR invariant (claim 30 000 >
total 1.0) and turned the run-entrance tests red. That redness attributed to
pydantic, not to the seam. The collateral control refused the proof.

New beyond C-F3: the fabricated line's code set is DISJOINT from the bundle's, and
retrieval computes that disjointness to rank experience. The system holds the
evidence that would expose the fabrication and spends it on ranking, never on
deciding.

No src/ change -- the fix is GATED on D-A pkt. 2 plus a commons amendment. Tests
pin the boundary so the gated work must arrive as a visible red test.

974 -> 984, strict superset, 0 node ids lost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-13 00:18:46 +02:00
commit b955bf274e
2 changed files with 307 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

217
tests/test_cost_baseline_loadbearing.py Normal file
View file

@ -0,0 +1,217 @@
"""S4.0: the cost-baseline seam — does the bundle's OWN cost baseline reach the judge?
The D7 mirroring queue (``docs/2026-08-18-funn-koeer-og-gater.md § D7-speilingskøen``)
carries S4.0 because the MAF sibling anchors ``affected_items`` against a cost
baseline. On our side that defect is confirmed as C-F3 (``docs/review-2026-07.md``,
MAJOR, spec-level: a fabricated cost line validates a 2.9 MNOK claim) and the fix —
a fail-closed reconciliation stage against a baseline projection — is GATED on D-A
pkt. 2 + a commons amendment for ``cost-baseline.json`` (parity plan row 19). So this
file does NOT build the fix. It answers the question that is answerable offline
today: **is today's boundary — "every cost figure the validator judges comes from the
proposal itself" — load-bearing?**
MEASURED 2026-09-13 with ``scripts/mutation_harness.py``, denominator ``tests/`` (the
whole suite, 974 tests before this file), every run restored sha256-verified. The
population was AST-measured over all 27 ``src/*.py`` files first (positive control:
the same query finds ``validate_proposal``'s single call site in ``loop.py``).
- A ``SavingsProposal`` comes into being in exactly THREE places, with three
provenances: the model-authored parse (``loop.py``), the bundle's baseline
projection (``ir.py``), and the re-read system output (``hitl.py``). Only the FIRST
reaches ``validate_proposal``. That is C-F3 stated as a measurement.
- The baseline IS loaded on every run path — and the only field read off it directly
is ``project_id`` (4 sites). Everything else leaves the load through
``CandidateFeatures.from_proposal``, which reads the codes, the measure and the
claim. The QUANTITIES, the UNIT COSTS and the UNCERTAINTY BANDS are schema-validated
and then never read again by anything.
- So the fix's own input — the bundle's cost lines — is present in memory at judgement
time and structurally unreachable from the judge. The consequence is concrete:
narrowing ``ComposedRunContext.ir_projection`` to the one field anyone reads would
leave all 974 tests green and silently delete the only cost baseline the gated S4.0
work has to reconcile against.
The mutations were shown to change behaviour BEFORE their greens were read as holes
(the økt 39 trap: the harness reports a no-op and an undetected seam identically).
Scaling the carried baseline's quantities by 1000 changes what the composed context
holds — the assertions below observe it — while every pre-existing test stays green.
NEW BEYOND C-F3 — the bundle DOES carry a cost baseline, and retrieval already sees
it. C-F3 says "nothing in the bundle format carries a cost baseline to reconcile
against"; measured here, ``validator-input.json`` carries ``ENERGI-TOTAL-EL`` at
300 000 NOK, and a fabricated line's code is visible to the retrieval layer as a
DISJOINT code set (similarity signal) at the same moment the validator judges it on
its own arithmetic. The system holds the evidence that would expose the fabrication,
spends it on ranking experience, and never on deciding.
HONEST LIMIT — what this does NOT say. Pinning that the baseline arrives intact is
not a claim that it is USED; it is not, and C-F3 stands as a MAJOR defect. These
tests pin the boundary so the gated work must arrive as a visible red test and a D-A
pkt. 2 decision, never as a silent swap — the same ratchet role
``test_ingest_stamp_conformance_loadbearing.py`` plays upstream. No ``src/`` change is
made here, no spec text is touched, and the golden fixture cannot help: it freezes
what the validator COMPUTES from a proposal, and the baseline is not an input to that
computation at all.
Dated under the D7 frame: this is work AFTER 2026-08-09 and must NOT be read as
independent convergence with the sibling.
"""
from __future__ import annotations
import ast
import json
from pathlib import Path
from portfolio_optimiser_claude.experience import CandidateFeatures
from portfolio_optimiser_claude.ir import SavingsProposal, load_validator_input
from portfolio_optimiser_claude.run import compose_run_context
from portfolio_optimiser_claude.validator import ValidatedProposal, validate_proposal
SRC_PKG = Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser_claude"
BUNDLE = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
# The bundle's own cost lines, read from the file rather than restated — a restated
# expectation would pass against a projection that never touched the bundle.
_RAW_BASELINE = json.loads((BUNDLE / "validator-input.json").read_text(encoding="utf-8"))
# A line the bundle knows nothing about: C-F3's fabricated code, at a magnitude that
# dwarfs the whole building's annual energy cost. This is the input the gated fix
# must reject; today it validates.
FABRICATED = SavingsProposal(
project_id="BYGG-KONTOR-NORD",
measure="diktet tiltak",
affected_items=[{"code": "XX-DIKTET", "quantity": 1_000_000, "unit_cost": 10}],
claimed_saving_nok=2_900_000,
)
def _cost_lines(proposal: SavingsProposal) -> list[tuple[str, float, float]]:
return [(i.code, i.quantity, i.unit_cost) for i in proposal.affected_items]
def _module_ast(name: str) -> ast.Module:
return ast.parse((SRC_PKG / name).read_text(encoding="utf-8"))
class TestTheBaselineReachesTheRunPath:
"""The cost lines the gated fix needs must survive composition INTACT."""
def test_composed_context_carries_the_bundles_cost_lines_verbatim(self) -> None:
composed = compose_run_context(BUNDLE, None, k=3)
expected = [
(item["code"], float(item["quantity"]), float(item["unit_cost"]))
for item in _RAW_BASELINE["affected_items"]
]
assert _cost_lines(composed.ir_projection) == expected
def test_composed_context_carries_the_uncertainty_bands_verbatim(self) -> None:
# SPLIT from the cost lines on purpose: a red test only proves its FIRST
# assert, and the bands are a separate half of the baseline — the fix's
# reconciliation needs the spread, not only the point estimate.
composed = compose_run_context(BUNDLE, None, k=3)
expected = {
code: (float(low), float(high))
for code, (low, high) in _RAW_BASELINE["assumptions"].items()
}
assert composed.ir_projection.assumptions == expected
def test_the_baseline_total_is_the_bundles_own_figure(self) -> None:
# The one number a reconciliation stage would compare a claim against. It is
# a property of the BUNDLE, never of whatever the model proposes.
composed = compose_run_context(BUNDLE, None, k=3)
total = sum(i.quantity * i.unit_cost for i in composed.ir_projection.affected_items)
assert total == 300_000.0
class TestTheBaselineIsStructurallyHeld:
"""Nothing behavioural reads these fields, so the holding is pinned structurally."""
def test_the_composed_context_holds_the_whole_typed_ir(self) -> None:
# The ratchet against a silent narrowing: `project_id` is the only field any
# caller reads off the projection, so shrinking the carried type to a string
# is invisible to every behavioural test — and deletes the baseline.
module = _module_ast("run.py")
annotations = {
node.target.id: ast.unparse(node.annotation)
for cls in ast.walk(module)
if isinstance(cls, ast.ClassDef) and cls.name == "ComposedRunContext"
for node in cls.body
if isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name)
}
# Positive control: the query CAN find fields — it finds the siblings too.
assert "citations" in annotations and "context" in annotations
assert annotations["ir_projection"] == "SavingsProposal"
def test_every_composition_path_loads_the_baseline(self) -> None:
# Two composition paths exist (the shared `compose_run_context` and S10's
# own hand-written one). The sibling's drift form would be one of them
# dropping the load; both are pinned so a divergence cannot be silent.
loaders = sorted(
path.name
for path in SRC_PKG.glob("*.py")
if any(
isinstance(node, ast.Call)
and isinstance(node.func, ast.Name)
and node.func.id == "load_validator_input"
for node in ast.walk(ast.parse(path.read_text(encoding="utf-8")))
)
)
assert loaders == ["experience.py", "run.py", "run_s10.py"]
class TestTodaysBoundaryIsUnanchored:
"""C-F3's boundary, pinned so the gated fix must arrive as a red test."""
def test_the_validator_judges_the_proposal_alone(self) -> None:
# The fix adds a baseline argument here. That MUST be a visible red test and
# a D-A pkt. 2 decision, never a silent swap.
module = _module_ast("validator.py")
signature = next(
node.args
for node in ast.walk(module)
if isinstance(node, ast.FunctionDef) and node.name == "validate_proposal"
)
parameters = [arg.arg for arg in signature.args + signature.kwonlyargs]
assert parameters == ["proposal"]
def test_a_fabricated_cost_code_outside_the_bundle_still_validates(self) -> None:
# C-F3's run proof, pinned: the claim clears the gate on arithmetic derived
# entirely from the line that invented itself.
outcome = validate_proposal(FABRICATED)
assert isinstance(outcome, ValidatedProposal)
assert outcome.claimed_saving_nok == 2_900_000.0
def test_the_fabricated_line_dwarfs_the_baseline_that_was_in_memory(self) -> None:
# SPLIT from the validation above: this is the magnitude claim, and it is the
# half that shows the baseline was AVAILABLE, not merely absent.
baseline = load_validator_input(BUNDLE)
baseline_total = sum(i.quantity * i.unit_cost for i in baseline.affected_items)
fabricated_total = sum(i.quantity * i.unit_cost for i in FABRICATED.affected_items)
assert fabricated_total > 30 * baseline_total
assert FABRICATED.claimed_saving_nok > 9 * baseline_total
def test_retrieval_sees_the_disjoint_code_set_the_gate_never_consults(self) -> None:
# NEW BEYOND C-F3: the evidence exists and is spent on ranking. The code sets
# are disjoint — a signal the retrieval layer computes and the judge ignores.
baseline_codes = CandidateFeatures.from_proposal(load_validator_input(BUNDLE))
fabricated_codes = CandidateFeatures.from_proposal(FABRICATED)
assert baseline_codes.affected_codes == {"ENERGI-TOTAL-EL"}
assert baseline_codes.affected_codes.isdisjoint(fabricated_codes.affected_codes)
class TestOnlyTheAuthoredProposalIsJudged:
"""The population control: which of the three provenances reaches the judge."""
def test_the_validator_is_called_on_the_generated_candidate_only(self) -> None:
# Three constructions of a SavingsProposal exist (model-authored, bundle
# baseline, re-read output). Exactly one call site judges, and its argument
# is the parse-path variable — so the baseline provenance cannot be judged.
module = _module_ast("loop.py")
call_arguments = [
[ast.unparse(arg) for arg in node.args]
for node in ast.walk(module)
if isinstance(node, ast.Call)
and isinstance(node.func, ast.Name)
and node.func.id == "validate_proposal"
]
assert call_arguments == [["proposal"]]