portfolio-optimiser/tests/test_prepass_cli_door_loadbearing.py
Kjell Tore Guttormsen 183191e51c docs(prepass): before/after on K2, the mutation battery and the invariant row
Steg 8. Ingen produksjonskode i denne commiten utover de TRE testarmene tre GROENNE
mutasjoner tvang fram.

MAALING (`docs/2026-09-07-okf-prepass-i-debatten.md`), instrumentet validert mot en
kjent-positiv FOER foerste tall (K2 rotlisting = 3 954 tegn / 1 495 o200k-tokens,
S7a-3s publiserte tall eksakt; skriptet asserterer paa det og nekter aa rapportere
ellers):
- KUTTET paa levert K2: 629 = 621 + 8, tre regler navngitt. Payload paa disk 40 425
  tok; renderingen debatten ser 5 162 tok -- 87 % av kostnaden er withheld-lista, som
  aldri naar prompten.
- FOER/ETTER, tre armer fordi "FOER" ikke er ett tall. Like-for-like (en debatt som
  GAAR stigen mot det deklarerte kuttet): 10 prompter / 7 031 tok -> 3 prompter /
  10 641 tok. **+51 %, og dokumentet paastaar ikke at dette er en besparelse.** Det som
  kjoepes er nevnerne, et re-maalbart `ref`, og at stempelet slutter aa overdrive:
  siteringer 629 -> 8.
- SPOERSMAALET BASEN IKKE SVARER PAA: aatte arkitekttegninger, og **5,5x dyrere** enn
  det gode spoersmaalet (28 312 mot 5 162 tok). En TOM leveranse er bevis for fravaer;
  en FULL er IKKE bevis for tilstedevaerelse -- og med verktoeyene trukket har debatten
  ingen vei til aa oppdage det selv. Uttalt som den reelle handelen, ikke oppdaget i
  drift.

LOAD-BEARING: 32 mutasjoner, ALLE ROEDE mot HELE suiten, maks en per Bash-kall,
restaurert fra `scratchpad/` med `shasum -c` (aldri `git checkout`). Groenn kontroll
**1467 passed / 5 skipped** (fra 1387/5; **+80 node-ider, 0 fjernet**, maalt med `comm`
mot en baseline tatt foer foerste commit). Golden `shasum -a 1` av INNHOLDET =
ea8c534773acdbe41ae68f2c55724d69aaf8be4f, BYTE-UENDRET. `pyproject.toml` uroert.

TRE MUTASJONER VAR GROENNE FOERST, og alle tre var TESTFEIL -- ikke soemfeil. Ingen ble
droppet:
- M9: injeksjonsarmen satte `text_sha256` fra den INJISERTE teksten, saa
  `text_sha256`-grenen fyrte i stedet, og armen matchet paa "text" -- en DELSTRENG av
  `text_sha256`. En angriper kontrollerer begge avledede medlemmer, saa fixturen setter
  naa `text_sha256` til digesten av den EKTE teksten; da er likhetssjekken det eneste
  som staar i veien. Ny parallell arm beviser at de to sjekkene ikke er en sjekk.
- M18: fixturbasen navigerer til fire ikke-verdict-konsepter og payloadet leverer ALLE
  fire, saa "siter de leverte" og "siter alt navigerbart" ga SAMME sett. Armen flytter
  naa ett utdrag til `withheld` og asserterer `delivered < navigable` FOER den maaler.
- M32: uten raden falt kjoeringen gjennom til "--explore requires --explore-config",
  som ogsaa navngir `--explore`. Oekt 57s regel ordrett -- assert aldri paa en
  delstreng to nekter deler. Armen bruker naa en argv `--explore` ellers ville blitt
  AKSEPTERT paa.

Co-Authored-By: Claude <claude-opus-5>
2026-09-07 14:39:50 +02:00

445 lines
16 KiB
Python

"""Load-bearing gate for the ``--prepass-payload`` operator door (order 20260907T080223Z).
Two halves, and each has its own failure mode.
- **The wiring.** A flag that is parsed, validated and never passed to ``run_project`` is the F4
silent-drop class: the operator sees rc 0 and a navigating run. So one arm asserts the value
ARRIVES, by spying on the dispatch rather than by reading an exit code.
- **The refusals.** Six surfaces would otherwise accept the flag and do nothing with it. Each is
refused BY NAME and each arm is paired with an **rc-0 control on an argv that would otherwise
be ACCEPTED** — without which a red arm can come from the fixture rather than from the row.
"""
from __future__ import annotations
import json
import shutil
from pathlib import Path
from typing import Any
import pytest
import portfolio_optimiser.run as run_module
from portfolio_optimiser.run import main
FIXTURE = Path(__file__).parent / "fixtures" / "prepass" / "bygg-energi-mikro-fixture.payload.json"
SHIPPED_BASE = Path(__file__).parent.parent / "shared" / "examples" / "bygg-energi-mikro"
PROJECT_ID = "BYGG-KONTOR-NORD"
_PROPOSAL = json.dumps(
{
"project_id": PROJECT_ID,
"measure": "energy_efficiency",
"claimed_saving_nok": 30000,
"affected_items": [{"code": "ENERGI-TOTAL-EL", "quantity": 120000.0, "unit_cost": 1.25}],
"assumptions": {},
}
)
def _base(tmp_path: Path) -> str:
root = tmp_path / "mounted-under-another-name"
shutil.copytree(SHIPPED_BASE, root)
index = root / "index.md"
lines = index.read_text(encoding="utf-8").split("\n")
lines.insert(1, "bundle_id: bygg-energi-mikro-fixture")
index.write_text("\n".join(lines), encoding="utf-8")
return str(root)
def _payload_file(tmp_path: Path) -> str:
path = tmp_path / "payload.json"
path.write_text(FIXTURE.read_text(encoding="utf-8"), encoding="utf-8")
return str(path)
def _docs(tmp_path: Path) -> str:
d = tmp_path / "docs"
d.mkdir(exist_ok=True)
(d / "cost.txt").write_text("Energitiltak i kontorbygg.", encoding="utf-8")
return str(d)
def _replies(tmp_path: Path) -> str:
path = tmp_path / "replies.json"
path.write_text(
json.dumps({"proposer": _PROPOSAL, "checker": "VERDICT: APPROVE"}), encoding="utf-8"
)
return str(path)
def _run_argv(tmp_path: Path, *extra: str) -> list[str]:
"""An argv the CLI ACCEPTS — the control every refusal arm below is measured against."""
return [
PROJECT_ID,
"--bundle-dir",
_base(tmp_path),
"--docs-dir",
_docs(tmp_path),
"--scripted-replies",
_replies(tmp_path),
*extra,
]
def _refuse_model(monkeypatch: pytest.MonkeyPatch) -> None:
"""Any model client construction becomes a failure, so a refusal that fired AFTER the spend is
distinguishable from one that fired before it. At the exit code the two look identical."""
def refuse(profile: Any) -> Any:
raise AssertionError("a model client was built despite a refusal") # pragma: no cover
monkeypatch.setattr(run_module, "_default_factory", refuse)
# --- the wiring ------------------------------------------------------------------------------
def test_the_flag_reaches_run_project(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Parsed-and-dropped and parsed-and-passed are the same exit code. This arm reads the
argument, not the outcome."""
seen: list[Any] = []
original = run_module.run_project
async def spy(*args: Any, **kwargs: Any) -> Any:
seen.append(kwargs.get("prepass_payload"))
return await original(*args, **kwargs)
monkeypatch.setattr(run_module, "run_project", spy)
rc = main(_run_argv(tmp_path, "--prepass-payload", _payload_file(tmp_path)))
assert rc == 0
assert seen and seen[0] is not None
assert seen[0].bundle.bundle_id == "bygg-energi-mikro-fixture"
def test_the_notice_reaches_stdout(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
rc = main(_run_argv(tmp_path, "--prepass-payload", _payload_file(tmp_path)))
assert rc == 0
assert "DECLARED CUT" in capsys.readouterr().out
def test_the_dry_run_dispatch_is_threaded_too(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""``DryRunReport.prepass`` and the dry-run notice call are dead code without this."""
# ``--scripted-replies`` and ``--live-dry-run`` are a documented contradiction, so the dry-run
# argv is built without it.
rc = main(
[
PROJECT_ID,
"--bundle-dir",
_base(tmp_path),
"--docs-dir",
_docs(tmp_path),
"--live-dry-run",
"--prepass-payload",
_payload_file(tmp_path),
]
)
assert rc == 0
assert "DECLARED CUT" in capsys.readouterr().out
def test_a_missing_payload_file_refuses_without_a_traceback(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
_refuse_model(monkeypatch)
rc = main(_run_argv(tmp_path, "--prepass-payload", str(tmp_path / "nope.json")))
assert rc == 1
assert "run refused:" in capsys.readouterr().err
def test_a_malformed_payload_file_refuses_without_a_traceback(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
_refuse_model(monkeypatch)
bad = tmp_path / "bad.json"
bad.write_text("{not json", encoding="utf-8")
rc = main(_run_argv(tmp_path, "--prepass-payload", str(bad)))
assert rc == 1
assert "run refused:" in capsys.readouterr().err
def test_a_payload_for_another_base_refuses_before_any_model_call(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
"""``PrepassRefused`` is a ``ValueError``, so it lands on the refusal surface and not the
crash channel — and the refusal happens before a client is built."""
_refuse_model(monkeypatch)
raw = json.loads(FIXTURE.read_text(encoding="utf-8"))
raw["bundle"]["bundle_id"] = "a-different-corpus"
path = tmp_path / "other.json"
path.write_text(json.dumps(raw), encoding="utf-8")
rc = main(_run_argv(tmp_path, "--prepass-payload", str(path)))
assert rc == 1
assert "run refused:" in capsys.readouterr().err
# --- the six refusals, each with an rc-0 control ---------------------------------------------
def test_the_flag_requires_a_bundle_dir(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
_refuse_model(monkeypatch)
rc = main(
[
PROJECT_ID,
"--docs-dir",
_docs(tmp_path),
"--scripted-replies",
_replies(tmp_path),
"--prepass-payload",
_payload_file(tmp_path),
]
)
assert rc == 1
assert "--bundle-dir" in capsys.readouterr().err
def test_it_is_refused_in_portfolio_mode_by_name(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
"""NAMING ``--portfolio``, never the shared ``--prepass-payload`` token: a dropped row falls
through to the ``--bundle-dir`` requirement, whose message names the flag too — so an arm
asserting on the shared token would be green against the mutation it exists for."""
_refuse_model(monkeypatch)
rc = main(
[
"--portfolio",
"--scripted-replies",
_replies(tmp_path),
"--prepass-payload",
_payload_file(tmp_path),
]
)
assert rc == 1
assert "--portfolio" in capsys.readouterr().err
def test_portfolio_mode_without_the_flag_is_accepted(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""The rc-0 control for the arm above."""
rc = main(["--portfolio", "--scripted-replies", _replies(tmp_path)])
assert rc == 0
def _ledger(tmp_path: Path) -> str:
"""A JSON **ARRAY**. An object is refused by the ledger loader itself, which would make every
report arm below red for the wrong reason (measured in økt 89)."""
path = tmp_path / "ledger.json"
path.write_text(json.dumps([]), encoding="utf-8")
return str(path)
def test_it_is_refused_in_report_mode(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
"""Report mode returns ABOVE every dispatch, so an omission here is a silent DROP."""
_refuse_model(monkeypatch)
rc = main(
["--report", "--ledger", _ledger(tmp_path), "--prepass-payload", _payload_file(tmp_path)]
)
assert rc == 1
assert "--report" in capsys.readouterr().err
def test_report_mode_without_the_flag_is_accepted(tmp_path: Path) -> None:
"""The rc-0 control for the arm above."""
assert main(["--report", "--ledger", _ledger(tmp_path)]) == 0
def test_it_is_refused_with_proposals_from_mandate(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
"""That mode returns above the debate, so the flag would be silently inert."""
_refuse_model(monkeypatch)
mandate = tmp_path / "m.json"
mandate.write_text(
json.dumps(
{
"objective": "x",
"approaches": [
{
"id": "a1",
"label": "energy_efficiency",
"rationale": "r",
"affected_codes": ["ENERGI-TOTAL-EL"],
"claimed_saving_nok": 1000,
}
],
"allow_own_proposals": False,
}
),
encoding="utf-8",
)
rc = main(
_run_argv(
tmp_path,
"--proposals-from-mandate",
"--mandate",
str(mandate),
"--derive-cost-baseline",
"--prepass-payload",
_payload_file(tmp_path),
)
)
assert rc == 1
assert "--proposals-from-mandate" in capsys.readouterr().err
def test_it_is_refused_with_a_dimension_config(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
"""The pre-pass has no dimension concept, so its CUT is unscoped. Composing them would mean
po discarding excerpts the declaration counted as delivered — which makes the payload's own
denominators wrong for the run that published them."""
_refuse_model(monkeypatch)
dim = tmp_path / "dim.json"
dim.write_text(
json.dumps(
{
"id": "energi",
"label": "Energi",
"allowed_measure_types": ["energy_efficiency"],
}
),
encoding="utf-8",
)
rc = main(
_run_argv(
tmp_path, "--dimension-config", str(dim), "--prepass-payload", _payload_file(tmp_path)
)
)
assert rc == 1
assert "--dimension-config" in capsys.readouterr().err
def test_a_dimension_config_without_the_flag_is_accepted(tmp_path: Path) -> None:
"""The rc-0 control: the two flags are each fine alone."""
dim = tmp_path / "dim.json"
dim.write_text(
json.dumps(
{
"id": "energi",
"label": "Energi",
"allowed_measure_types": ["energy_efficiency"],
}
),
encoding="utf-8",
)
assert main(_run_argv(tmp_path, "--dimension-config", str(dim))) == 0
def _explore_config(tmp_path: Path) -> str:
path = tmp_path / "explore.json"
path.write_text(
json.dumps(
{
"max_rounds": 2,
"max_tokens": 5000,
"max_stall_count": 1,
"max_reset_count": 1,
"max_plan_revisions": 0,
"enable_plan_review": False,
}
),
encoding="utf-8",
)
return str(path)
def test_it_is_refused_with_explore(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
"""The exploration reads the WHOLE base with all four navigator tools and then hands its
mandate to a debate told it is under a declared cut — this seam's own grounds for withdrawing
the tools, one caller over.
**The argv is one ``--explore`` would otherwise be ACCEPTED on**, and the assertion is on
wording only THIS refusal produces. MEASURED: without ``--explore-config`` the run falls
through to "--explore requires --explore-config", which also contains the token
``--explore`` — so an arm asserting on that token alone stays green against the very mutation
it exists for (økt 57's own lesson: never assert on a substring two refusals share).
"""
_refuse_model(monkeypatch)
rc = main(
_run_argv(
tmp_path,
"--explore",
"finn tiltak",
"--explore-config",
_explore_config(tmp_path),
"--prepass-payload",
_payload_file(tmp_path),
)
)
assert rc == 1
assert "cannot be combined" in capsys.readouterr().err
def test_a_plain_run_without_the_flag_is_accepted(tmp_path: Path) -> None:
"""The rc-0 control shared by the arms that add exactly one flag to this argv."""
assert main(_run_argv(tmp_path)) == 0
# --- the hosted surface, deliberately untouched ------------------------------------------------
def test_the_hosted_surface_refuses_the_field_without_being_edited() -> None:
"""The brief's Non-Goal (MAJOR-4 / S7b precedent): the field enters NONE of the three sets, so
the generic ``unknown field(s)`` 400 already answers it and Fase 4e's two halves stand."""
from portfolio_optimiser import hosting
assert "prepass_payload" not in hosting._ALLOWED_FIELDS
with pytest.raises(ValueError, match="unknown field"):
hosting._run_kwargs({"project_id": PROJECT_ID, "prepass_payload": "/x.json"})
def test_the_phase_4e_partitions_still_hold() -> None:
"""The negative half: every forwarded field is a real ``run_project`` parameter and every
consumed one is not. Adding a parameter without touching hosting must not break it."""
import inspect
from portfolio_optimiser import hosting
from portfolio_optimiser.run import run_project
parameters = set(inspect.signature(run_project).parameters)
assert set(hosting._REQUIRED_FIELDS) <= parameters
assert set(hosting._OPTIONAL_FIELDS) <= parameters
assert set(hosting._CONSUMED_FIELDS).isdisjoint(parameters)
# --- the README block --------------------------------------------------------------------------
def _readme_block(flag: str) -> str:
"""The prose block for ONE flag, extracted rather than substring-matched: ``--portfolio`` and
``--report`` occur all over the README, so a file-wide search cannot tell a documented refusal
from an unrelated mention."""
readme = (Path(__file__).parent.parent / "README.md").read_text(encoding="utf-8")
start = readme.index(f"(`{flag}`)")
end = readme.find("\n **", start)
return readme[start : end if end != -1 else len(readme)]
def test_the_readme_documents_the_flag_and_every_partner_refusal() -> None:
block = _readme_block("--prepass-payload")
for partner in (
"--bundle-dir",
"--portfolio",
"--report",
"--proposals-from-mandate",
"--dimension-config",
"--explore",
):
assert partner in block, partner
# Customer-facing terminology: never "OKF bundle" on a published surface.
assert "OKF bundle" not in block
def test_the_extractor_finds_a_block_that_has_existed_since_f4() -> None:
"""The known-positive control: an extractor that silently finds nothing would make the arm
above green against a README with no block at all."""
assert "--explore" in _readme_block("--plan-review")