open/portfolio-optimiser-claude
1
0
Fork 0
Code Issues Releases 1 Activity

test(baseline): the baseline was loaded, and the judge could never reach it

Browse source 
S4.0, the fifth D7 mirroring candidate, measured with scripts/mutation_harness.py
against the real tree. C-F3 says "nothing in the bundle format carries a cost
baseline to reconcile against". Measured, that is too strong: validator-input.json
carries ENERGI-TOTAL-EL at 300 000 NOK and it is loaded on every run path. The
defect is not that the baseline is missing -- it is that no path connects it to
the judge.

A SavingsProposal comes into being in exactly three places (model-authored parse,
bundle baseline, re-read output; AST-measured over all 27 src files, positive
control: the same query finds validate_proposal's one call site). Only the first
reaches the validator. The only field read off the baseline directly is
project_id; the quantities, unit costs and uncertainty bands are schema-validated
and never read again.

Seven mutations, every one green-before/red-after with the restore sha256-verified
from disk:

  M1 baseline swapped for the weakest schema-valid variant -- 3 targets RED,
     all 974 pre-existing tests GREEN in both runs
  M2 the carried annotation narrowed to object -- AST ratchet RED, behaviour GREEN
  M3 the S10 path stops loading the baseline -- loader population RED
  M4 the validator gains a baseline parameter -- signature ratchet RED, golden GREEN
  M5 the call-site argument becomes a keyword -- provenance control RED
  M6 fail-closed code-set gate (the gated fix's own shape) -- boundary RED, golden GREEN
  M7 retrieval stops reading the codes -- disjointness RED

M4 and M6 leaving the golden green is itself the finding that S4.0 is
golden-compatible when D-A pkt. 2 lands.

A measuring trap was caught by the harness rather than by memory: the first M1
flattened the cost lines to 1.0x1.0, which broke the IR invariant (claim 30 000 >
total 1.0) and turned the run-entrance tests red. That redness attributed to
pydantic, not to the seam. The collateral control refused the proof.

New beyond C-F3: the fabricated line's code set is DISJOINT from the bundle's, and
retrieval computes that disjointness to rank experience. The system holds the
evidence that would expose the fabrication and spends it on ranking, never on
deciding.

No src/ change -- the fix is GATED on D-A pkt. 2 plus a commons amendment. Tests
pin the boundary so the gated work must arrive as a visible red test.

974 -> 984, strict superset, 0 node ids lost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-13 00:18:46 +02:00
commit b955bf274e
2 changed files with 307 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

92
docs/2026-08-18-funn-koeer-og-gater.md
View file

@ -35,11 +35,11 @@ spørringen som ble kjørt** — ikke fila den ble kjørt mot.
## D7-speilingskøen
Åtte kandidater for speiling mellom D7-søsknene. **Ingen er besluttet** — de står som
kandidater, ikke som planlagt arbeid. **Fire er målt, 4 står igjen:**
kandidater, ikke som planlagt arbeid. **Fem er målt, 3 står igjen:**
- ~~S2.7~~ — **MÅLT 2026-09-07, se under**
- ~~S3.2~~ — **MÅLT 2026-09-12, se under**
- S4.0 (`126807a`)
- ~~S4.0~~ (`126807a`) — **MÅLT 2026-09-13, se under**
- (p) `to_ore` — TO kallsteder
- ~~(a)/(i) `unquote_scalar`~~ — **MÅLT 2026-08-31, se under**
- `mandate.py`
@ -265,6 +265,94 @@ involvert i nøklingen i det hele tatt** — hvilket er nettopp derfor den ikke
**Datering (D7-rammen):** arbeid ETTER 2026-08-09 — skal **ikke** leses som uavhengig konvergens.
### S4.0 — baselinen er lastet, og den er strukturelt utenfor dommerens rekkevidde (målt 2026-09-13)
Kandidaten står i køen fordi søskenet forankrer `affected_items` mot en kostbaseline. Defekten
er bekreftet her som C-F3 (`docs/review-2026-07.md`, MAJOR, spec-nivå: en diktet kostlinje
validerer en 2,9 MNOK-claim), og fiksen — en fail-closed avstemmings-stage — er **GATET** på
D-A pkt. 2 + et commons-amendment for `cost-baseline.json` (paritetsplanens rad 19).
Speilings-spørsmålet som KAN besvares offline i dag er derfor: **er dagens grense — «hvert
kosttall validatoren dømmer på stammer fra forslaget selv» — load-bearing?**
**Populasjonen først.** En `SavingsProposal` blir til i nøyaktig TRE steder, med hver sin
proveniens: det modell-forfattede parset (`loop.py:89`), bundelens baseline-projeksjon
(`ir.py:64`) og det re-leste system-outputet (`hitl.py:132`). AST-målt over alle 27
`src/*.py` (positiv kontroll: samme spørring finner `validate_proposal`s ENE kallsted,
`loop.py:284`). **Kun den FØRSTE når validatoren.** Det er C-F3 uttrykt som en måling.
Baselinen LASTES på hver komposisjonssti — `run.py:123`, `run_s10.py:71`, `experience.py:132`
— og det eneste feltet noen leser direkte av den er `project_id` (fire steder). Alt annet
forlater lastingen gjennom `CandidateFeatures.from_proposal`, som leser kodene, `measure` og
claimen. **`quantity`, `unit_cost` og usikkerhets-bandene er skjema-validert og deretter
aldri lest igjen av noe.** Fiksens egen input ligger altså i minnet i samme
`ComposedRunContext` som dommen felles fra, og ingen sti fører den dit.
**Målingene.** Med `scripts/mutation_harness.py`, nevner `tests/` (hele den gamle suiten,
974 tester), hver kjøring sha256-restaurert fra disk:
- **M1 — baselinen erstattes med den svakeste skjemagyldige varianten** (kodene, `measure`,
claimen og `project_id` bevart; `quantity`/`unit_cost` flatet, band tømt): tre nye tester
RØDE, og **alle 974 gamle GRØNNE i begge kjøringer**. Bundelens kosttall kan byttes ut på
run-stien uten at én eneste eksisterende test merker det.
- **M2 — `ComposedRunContext.ir_projection: SavingsProposal` → `object`**: AST-testen RØD,
alle oppførselstester GRØNNE. En stille innsnevring av det bårne feltet er usynlig for
oppførsel — og ville slettet den eneste kostbaselinen på run-stien.
- **M3 — S10-stien slutter å kalle `load_validator_input`** (oppførselsbevarende alias):
loader-populasjonen RØD, `test_s10_run_layer.py` + `test_preflight.py` GRØNNE.
- **M4 — validatoren får en `baseline`-parameter** (den gatede fiksens signatur, default
`None`): signatur-ratchet-en RØD, **goldenen GRØNN** — som mekanisk bekrefter at S4.0s
signatur-halvdel er golden-kompatibel når D-A pkt. 2 lander.
- **M5 — kallstedets argument blir et keyword**: proveniens-kontrollen RØD, oppførselen
uendret.
- **M6 — fail-closed kodesett-gate i validatoren** (fiksens EGEN form, hardkodet baseline):
grense-testen RØD, **goldenen GRØNN i begge**.
- **M7 — retrieval slutter å lese kodene**: disjunkthets-testen RØD, golden + validator GRØNNE.
**En måletrap unngått — mutasjonen må vises å endre oppførsel, ikke bare å være skrevet**
(økt 39, her i motsatt retning). Første M1-forsøk flatet kostlinjene til `1.0 × 1.0` og
gjorde `test_run_entrance_loadbearing.py` RØD. Det så ut som dekning, men var det ikke:
total 1,0 < claim 30 000 bryter IR-invarianten (`ir.py:46`), så mutasjonen var en
**konstruksjonsfeil**, ikke en baseline-fjerning, og rødheten attribuerte til pydantic.
Harnessets kollateral-kontroll fanget den. Den korrekte mutasjonen holder invarianten — og
da er alle 974 grønne.
**NYTT UTOVER C-F3 — bundelen BÆRER en kostbaseline, og retrieval ser den allerede.** C-F3
formulerer defekten som «ingenting i bundle-formatet bærer en kostbaseline å avstemme mot».
Målt her er det for sterkt: `validator-input.json` bærer `ENERGI-TOTAL-EL` à 300 000 NOK, og
den lastes på hver run-sti. C-F3s kjørte bevis er reprodusert med review-ens egne tall (claim
2 900 000, degenererte percentiler 3 000 000) og skjerpet: det diktede kostgrunnlaget er
**33,3× hele bundelens baseline**, claimen er **9,67× byggets totale årlige energikost**, og
den diktede koden har **null overlapp** med bundelens kodesett. Det overlappet BEREGNES —
`CandidateFeatures` bruker det til å rangere erfaring. **Systemet holder altså beviset som
ville avslørt dikteringen, bruker det på rangering, og aldri på å dømme.** Defekten er ikke
at baselinen mangler; den er at den ikke er koblet til dommeren.
**Hvor bredt fiksen slår ut (en telling, ikke et load-bearing-bevis — derfor ikke harnesset,
men git-verifisert restaurering):** med fiksens form hardkodet til bundelens kodesett er **30
av 974** røde. Fiksen er altså bredt synlig i suiten, ikke stille. Ærlig grense: det tallet er
målt på en HARDKODET baseline; den ekte fiksen ville lest bundelens baseline per kjøring, og
hver testfixtures egen baseline ville da definert sitt eget kodesett — så 30 er et tak på
støyen, ikke et estimat på arbeidet.
**Pinnet av** `tests/test_cost_baseline_loadbearing.py` (10 tester, 974 → 984, strengt
supersett: 0 tapte node-id-er). To testpar er SPLITTET fordi en rød test bare beviser sin
FØRSTE assert: kostlinjene fra bandene, og valideringen av den diktede linja fra dens
magnitude.
**Ærlig grense — hva dette IKKE sier.** Å pinne at baselinen ankommer intakt er ingen påstand
om at den BRUKES; det gjør den ikke, og C-F3 står som MAJOR. Testene pinner grensen slik at
det gatede arbeidet MÅ ankomme som en synlig rød test og en D-A pkt. 2-beslutning — samme
ratchet-rolle `test_ingest_stamp_conformance_loadbearing.py` har oppstrøms. Ingen
`src/`-endring er gjort, ingen spec-tekst rørt, og goldenen kan ikke hjelpe: den fryser hva
validatoren REGNER UT av et forslag, og baselinen er ikke en input til den beregningen i det
hele tatt. De to magnitude-/disjunkthets-påstandene er dessuten faktapåstander om
bundle-fixturen og deler anker med goldenen — de er konsistens-tester, ikke uavhengige
søm-bevis, og er merket som det over.
**Datering (D7-rammen):** arbeid ETTER 2026-08-09 — skal **ikke** leses som uavhengig
konvergens.
Rammen rundt køen: å lese søskenets kode er tillatt (`3bdf7f0`), men kopiering skal kun skje
der det tjener løsningen, aldri som snarvei. **Uavhengighets-beviset er DATERT** t.o.m.
2026-08-09; arbeid etter den datoen kan ikke leses som uavhengig konvergens.

217
tests/test_cost_baseline_loadbearing.py Normal file
View file

@ -0,0 +1,217 @@
"""S4.0: the cost-baseline seam — does the bundle's OWN cost baseline reach the judge?
The D7 mirroring queue (``docs/2026-08-18-funn-koeer-og-gater.md § D7-speilingskøen``)
carries S4.0 because the MAF sibling anchors ``affected_items`` against a cost
baseline. On our side that defect is confirmed as C-F3 (``docs/review-2026-07.md``,
MAJOR, spec-level: a fabricated cost line validates a 2.9 MNOK claim) and the fix —
a fail-closed reconciliation stage against a baseline projection — is GATED on D-A
pkt. 2 + a commons amendment for ``cost-baseline.json`` (parity plan row 19). So this
file does NOT build the fix. It answers the question that is answerable offline
today: **is today's boundary — "every cost figure the validator judges comes from the
proposal itself" — load-bearing?**
MEASURED 2026-09-13 with ``scripts/mutation_harness.py``, denominator ``tests/`` (the
whole suite, 974 tests before this file), every run restored sha256-verified. The
population was AST-measured over all 27 ``src/*.py`` files first (positive control:
the same query finds ``validate_proposal``'s single call site in ``loop.py``).
- A ``SavingsProposal`` comes into being in exactly THREE places, with three
provenances: the model-authored parse (``loop.py``), the bundle's baseline
projection (``ir.py``), and the re-read system output (``hitl.py``). Only the FIRST
reaches ``validate_proposal``. That is C-F3 stated as a measurement.
- The baseline IS loaded on every run path — and the only field read off it directly
is ``project_id`` (4 sites). Everything else leaves the load through
``CandidateFeatures.from_proposal``, which reads the codes, the measure and the
claim. The QUANTITIES, the UNIT COSTS and the UNCERTAINTY BANDS are schema-validated
and then never read again by anything.
- So the fix's own input — the bundle's cost lines — is present in memory at judgement
time and structurally unreachable from the judge. The consequence is concrete:
narrowing ``ComposedRunContext.ir_projection`` to the one field anyone reads would
leave all 974 tests green and silently delete the only cost baseline the gated S4.0
work has to reconcile against.
The mutations were shown to change behaviour BEFORE their greens were read as holes
(the økt 39 trap: the harness reports a no-op and an undetected seam identically).
Scaling the carried baseline's quantities by 1000 changes what the composed context
holds — the assertions below observe it — while every pre-existing test stays green.
NEW BEYOND C-F3 — the bundle DOES carry a cost baseline, and retrieval already sees
it. C-F3 says "nothing in the bundle format carries a cost baseline to reconcile
against"; measured here, ``validator-input.json`` carries ``ENERGI-TOTAL-EL`` at
300 000 NOK, and a fabricated line's code is visible to the retrieval layer as a
DISJOINT code set (similarity signal) at the same moment the validator judges it on
its own arithmetic. The system holds the evidence that would expose the fabrication,
spends it on ranking experience, and never on deciding.
HONEST LIMIT — what this does NOT say. Pinning that the baseline arrives intact is
not a claim that it is USED; it is not, and C-F3 stands as a MAJOR defect. These
tests pin the boundary so the gated work must arrive as a visible red test and a D-A
pkt. 2 decision, never as a silent swap — the same ratchet role
``test_ingest_stamp_conformance_loadbearing.py`` plays upstream. No ``src/`` change is
made here, no spec text is touched, and the golden fixture cannot help: it freezes
what the validator COMPUTES from a proposal, and the baseline is not an input to that
computation at all.
Dated under the D7 frame: this is work AFTER 2026-08-09 and must NOT be read as
independent convergence with the sibling.
"""
from __future__ import annotations
import ast
import json
from pathlib import Path
from portfolio_optimiser_claude.experience import CandidateFeatures
from portfolio_optimiser_claude.ir import SavingsProposal, load_validator_input
from portfolio_optimiser_claude.run import compose_run_context
from portfolio_optimiser_claude.validator import ValidatedProposal, validate_proposal
SRC_PKG = Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser_claude"
BUNDLE = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
# The bundle's own cost lines, read from the file rather than restated — a restated
# expectation would pass against a projection that never touched the bundle.
_RAW_BASELINE = json.loads((BUNDLE / "validator-input.json").read_text(encoding="utf-8"))
# A line the bundle knows nothing about: C-F3's fabricated code, at a magnitude that
# dwarfs the whole building's annual energy cost. This is the input the gated fix
# must reject; today it validates.
FABRICATED = SavingsProposal(
project_id="BYGG-KONTOR-NORD",
measure="diktet tiltak",
affected_items=[{"code": "XX-DIKTET", "quantity": 1_000_000, "unit_cost": 10}],
claimed_saving_nok=2_900_000,
)
def _cost_lines(proposal: SavingsProposal) -> list[tuple[str, float, float]]:
return [(i.code, i.quantity, i.unit_cost) for i in proposal.affected_items]
def _module_ast(name: str) -> ast.Module:
return ast.parse((SRC_PKG / name).read_text(encoding="utf-8"))
class TestTheBaselineReachesTheRunPath:
"""The cost lines the gated fix needs must survive composition INTACT."""
def test_composed_context_carries_the_bundles_cost_lines_verbatim(self) -> None:
composed = compose_run_context(BUNDLE, None, k=3)
expected = [
(item["code"], float(item["quantity"]), float(item["unit_cost"]))
for item in _RAW_BASELINE["affected_items"]
]
assert _cost_lines(composed.ir_projection) == expected
def test_composed_context_carries_the_uncertainty_bands_verbatim(self) -> None:
# SPLIT from the cost lines on purpose: a red test only proves its FIRST
# assert, and the bands are a separate half of the baseline — the fix's
# reconciliation needs the spread, not only the point estimate.
composed = compose_run_context(BUNDLE, None, k=3)
expected = {
code: (float(low), float(high))
for code, (low, high) in _RAW_BASELINE["assumptions"].items()
}
assert composed.ir_projection.assumptions == expected
def test_the_baseline_total_is_the_bundles_own_figure(self) -> None:
# The one number a reconciliation stage would compare a claim against. It is
# a property of the BUNDLE, never of whatever the model proposes.
composed = compose_run_context(BUNDLE, None, k=3)
total = sum(i.quantity * i.unit_cost for i in composed.ir_projection.affected_items)
assert total == 300_000.0
class TestTheBaselineIsStructurallyHeld:
"""Nothing behavioural reads these fields, so the holding is pinned structurally."""
def test_the_composed_context_holds_the_whole_typed_ir(self) -> None:
# The ratchet against a silent narrowing: `project_id` is the only field any
# caller reads off the projection, so shrinking the carried type to a string
# is invisible to every behavioural test — and deletes the baseline.
module = _module_ast("run.py")
annotations = {
node.target.id: ast.unparse(node.annotation)
for cls in ast.walk(module)
if isinstance(cls, ast.ClassDef) and cls.name == "ComposedRunContext"
for node in cls.body
if isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name)
}
# Positive control: the query CAN find fields — it finds the siblings too.
assert "citations" in annotations and "context" in annotations
assert annotations["ir_projection"] == "SavingsProposal"
def test_every_composition_path_loads_the_baseline(self) -> None:
# Two composition paths exist (the shared `compose_run_context` and S10's
# own hand-written one). The sibling's drift form would be one of them
# dropping the load; both are pinned so a divergence cannot be silent.
loaders = sorted(
path.name
for path in SRC_PKG.glob("*.py")
if any(
isinstance(node, ast.Call)
and isinstance(node.func, ast.Name)
and node.func.id == "load_validator_input"
for node in ast.walk(ast.parse(path.read_text(encoding="utf-8")))
)
)
assert loaders == ["experience.py", "run.py", "run_s10.py"]
class TestTodaysBoundaryIsUnanchored:
"""C-F3's boundary, pinned so the gated fix must arrive as a red test."""
def test_the_validator_judges_the_proposal_alone(self) -> None:
# The fix adds a baseline argument here. That MUST be a visible red test and
# a D-A pkt. 2 decision, never a silent swap.
module = _module_ast("validator.py")
signature = next(
node.args
for node in ast.walk(module)
if isinstance(node, ast.FunctionDef) and node.name == "validate_proposal"
)
parameters = [arg.arg for arg in signature.args + signature.kwonlyargs]
assert parameters == ["proposal"]
def test_a_fabricated_cost_code_outside_the_bundle_still_validates(self) -> None:
# C-F3's run proof, pinned: the claim clears the gate on arithmetic derived
# entirely from the line that invented itself.
outcome = validate_proposal(FABRICATED)
assert isinstance(outcome, ValidatedProposal)
assert outcome.claimed_saving_nok == 2_900_000.0
def test_the_fabricated_line_dwarfs_the_baseline_that_was_in_memory(self) -> None:
# SPLIT from the validation above: this is the magnitude claim, and it is the
# half that shows the baseline was AVAILABLE, not merely absent.
baseline = load_validator_input(BUNDLE)
baseline_total = sum(i.quantity * i.unit_cost for i in baseline.affected_items)
fabricated_total = sum(i.quantity * i.unit_cost for i in FABRICATED.affected_items)
assert fabricated_total > 30 * baseline_total
assert FABRICATED.claimed_saving_nok > 9 * baseline_total
def test_retrieval_sees_the_disjoint_code_set_the_gate_never_consults(self) -> None:
# NEW BEYOND C-F3: the evidence exists and is spent on ranking. The code sets
# are disjoint — a signal the retrieval layer computes and the judge ignores.
baseline_codes = CandidateFeatures.from_proposal(load_validator_input(BUNDLE))
fabricated_codes = CandidateFeatures.from_proposal(FABRICATED)
assert baseline_codes.affected_codes == {"ENERGI-TOTAL-EL"}
assert baseline_codes.affected_codes.isdisjoint(fabricated_codes.affected_codes)
class TestOnlyTheAuthoredProposalIsJudged:
"""The population control: which of the three provenances reaches the judge."""
def test_the_validator_is_called_on_the_generated_candidate_only(self) -> None:
# Three constructions of a SavingsProposal exist (model-authored, bundle
# baseline, re-read output). Exactly one call site judges, and its argument
# is the parse-path variable — so the baseline provenance cannot be judged.
module = _module_ast("loop.py")
call_arguments = [
[ast.unparse(arg) for arg in node.args]
for node in ast.walk(module)
if isinstance(node, ast.Call)
and isinstance(node.func, ast.Name)
and node.func.id == "validate_proposal"
]
assert call_arguments == [["proposal"]]