open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 4 Activity

fix(v1-gate): row 2 binds a round to the run's own artefact family, and says out loud what it still cannot prove

Browse source 
The 18.09 re-measurement took row 2 to 3 of 3 GREEN on a tree this product had never
run in: four handwritten outcome.json, four handwritten <run_id>-coverage.json in an
outbox the forger named in those same files, and os.utime for the ordering. "Round 0
must be a named real run" was implemented as "a file with that name exists" — which
touch satisfies. The attack is committed as a red test in b769537.

Three bindings, chosen because each removes one of the forger's three moves:

1. The outbox is DERIVED, never declared. It is <rounds-dir>/<n>/outbox/, and an
   outcome.json that names one is refused by name. A path a round file chooses is a
   path it can point at a directory the same hand just wrote.

2. "The run exists" now means the run's own artefacts agree WITH EACH OTHER on content
   the gate recomputes (verify_run). Every evaluated approach has the proposal/outcome
   pair write_outbox actually persists; outcome_type IS the coverage status, reason IS
   its detail, a validated row's figure IS the proposal's own claimed_saving_nok, the
   provenance stamp follows the same branch, and verdict_id is RE-MINTED here from the
   proposal's own IR with the product's one minting rule (A5) rather than read. A
   not_evaluated approach wrote neither file, and an artefact naming an approach the
   coverage omits belongs to another run. verdicts._features_from_ir is made public for
   this: a second private copy in the gate could drift from the rule the run stamped
   with, which would turn the binding into a coincidence.

3. mtime decides nothing. The run's time is the round's declared ran_at (ISO-8601 with
   zone, required). An mtime is not evidence — it is a filesystem attribute one utime
   call sets.

What this does NOT do is prove a run happened. Nothing in a directory can: the outbox
writers are byte-deterministic and carry no clock by contract. So row 2 states its own
limit on every run instead of leaving GREEN to imply it (Row.attests / RUN_ATTESTATION):
that a run was actually made, and when, is the operator's to confirm. The cost of a
forgery moves from touch to reproducing the product's own artefact set, minting rule
included.

Measured, in a scratch copy, never in the work tree — four new mutants in this class,
each one line, each felled by the whole suite:
  M7  if "outbox" in data:          -> if False:   1 failed (declares its own outbox)
  M8  if verdict_id != minted:      -> if False:   1 failed (key is not the IR's)
  M9  if strays:                    -> if False:   1 failed (artefact of another run)
  M10 ran_at                        -> coverage mtime  18 failed (incl. the utime test)
Control, same scratch copy, unmutated: 1993 passed, 10 skipped, 5 xfailed.
Work tree, re-run after git add: uv run pytest -q -> 1998 passed, 5 skipped, 5 xfailed.
Gate: uv run python -m portfolio_optimiser.evals.v1_gate -> exit 1, row 2 RED (0 of 3).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 15:32:14 +02:00
commit a362504108
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
5 changed files with 389 additions and 42 deletions
Download patch file Download diff file Expand all files Collapse all files

192
tests/test_v1_gate.py
View file

@ -15,6 +15,7 @@ from __future__ import annotations
import ast
import json
import os
import shutil
import subprocess
import sys
from pathlib import Path
@ -24,7 +25,9 @@ import pytest
from portfolio_optimiser import frozen_bundles
from portfolio_optimiser.evals import v1_gate as gate
from portfolio_optimiser.ir import AffectedItem, SavingsProposal
from portfolio_optimiser.validator import UNSUPPORTED_REASON
from portfolio_optimiser.verdicts import features_from_ir, verdict_key
_REPO = Path(__file__).resolve().parents[1]
_CONFIG = gate.load_config()
@ -98,6 +101,54 @@ def _coverage_row(row: dict[str, Any]) -> dict[str, Any]:
return {"id": row["id"], "status": status, "detail": _DETAIL[row["stage"]], "saving_nok": None}
def _ir(aid: str, nok: float | None) -> dict[str, Any]:
"""The proposal IR a run would have written for this approach — built through the real model
so the fixture cannot drift from the shape ``outbox.write_outbox`` actually persists, which is
the shape the gate re-mints the verdict key from."""
saving = 1.0 if nok is None else float(nok)
return SavingsProposal(
project_id="p",
measure=f"tiltak {aid}",
affected_items=[AffectedItem(code=f"K-{aid}", quantity=1.0, unit_cost=saving)],
claimed_saving_nok=saving,
).model_dump()
def _run_family(outbox: Path, run_id: str, coverage: list[dict[str, Any]]) -> None:
"""One proposal/outcome pair per EVALUATED approach, agreeing with its coverage row — the
artefact family ``gate.verify_run`` re-checks. ``verdict_id`` is minted with the product's own
rule (A5), because that is the field the gate mints again rather than reads."""
for cov in coverage:
if cov["status"] == "not_evaluated":
continue
aid = cov["id"]
ir = _ir(aid, cov.get("saving_nok"))
stem = f"{run_id}-{aid}"
_write(
outbox / f"{stem}-proposal.json",
{
"run_id": run_id,
"approach_id": aid,
"proposal": ir,
"provenance": {
"validator_decision": (
"rejected" if cov["status"] == "rejected" else "validated"
)
},
},
)
_write(
outbox / f"{stem}-outcome.json",
{
"run_id": run_id,
"approach_id": aid,
"outcome_type": cov["status"],
"reason": cov.get("detail", ""),
"verdict_id": verdict_key(features_from_ir(ir)),
},
)
def _outcome(
round_dir: Path,
rows: list[dict[str, Any]],
@ -106,19 +157,24 @@ def _outcome(
at: int | None = None,
coverage: list[dict[str, Any]] | None = None,
) -> None:
"""The round's outcome file AND the run it names: an outbox holding the run's own coverage,
stamped at a fixed time (the run's time is the coverage file's)."""
"""The round's outcome file AND the run it names: the run's OWN outbox inside the round,
holding its coverage and one artefact pair per evaluated approach. The run's time is DECLARED
(``ran_at``) — an mtime is not evidence, and the gate no longer reads one."""
run_id = f"r{round_dir.name}"
outbox = round_dir.parent / "runs" / run_id
outbox = round_dir / "outbox"
if outbox.is_dir():
shutil.rmtree(outbox) # a re-written round brings its own run, not the last one's leftovers
rendered = [_coverage_row(r) for r in rows] if coverage is None else coverage
_write(outbox / f"{run_id}-coverage.json", {"rows": rendered, "stop_reason": ""})
stamp = _T0 + (int(round_dir.name) * 20 if at is None else at)
os.utime(outbox / f"{run_id}-coverage.json", (stamp, stamp))
_write(
outbox / f"{run_id}-coverage.json",
{"run_id": run_id, "rows": rendered, "stop_reason": ""},
)
_run_family(outbox, run_id, rendered)
_write(
round_dir / "outcome.json",
{
"run_id": run_id,
"outbox": f"../runs/{run_id}",
"ran_at": _iso(int(round_dir.name) * 20 if at is None else at),
"approaches": rows,
"removed": list(removed),
},
@ -356,7 +412,7 @@ def test_m1_a_change_below_one_percent_is_noise(tmp_path: Path) -> None:
def test_m1_an_outcome_must_name_a_run_that_exists_and_agree_with_it(tmp_path: Path) -> None:
root = _green_rounds(tmp_path)
run = root / "runs" / "r1" / "r1-coverage.json"
run = root / "1" / "outbox" / "r1-coverage.json"
run.unlink()
assert "finnes ikke" in " ".join(gate.score_changes(root, 3, _AI).exceptions)
_outcome(
@ -451,6 +507,126 @@ def test_m6_a_handwritten_outbox_is_not_a_run(tmp_path: Path) -> None:
assert (row.k, row.status) == (0, gate.RED), row.exceptions
def _edit(path: Path, **changes: Any) -> None:
data = json.loads(path.read_text(encoding="utf-8"))
data.update(changes)
_write(path, data)
def _mut_declares_its_own_outbox(root: Path) -> None:
_edit(root / "1" / "outcome.json", outbox=str(root / "1" / "outbox"))
def _mut_coverage_is_another_run(root: Path) -> None:
_edit(root / "1" / "outbox" / "r1-coverage.json", run_id="r9")
def _mut_the_approach_has_no_artefact(root: Path) -> None:
(root / "1" / "outbox" / "r1-a1-outcome.json").unlink()
def _mut_the_key_is_not_the_irs(root: Path) -> None:
_edit(
root / "1" / "outbox" / "r1-a1-outcome.json",
verdict_id=verdict_key(features_from_ir(_ir("a1", 424242.0))),
)
def _mut_the_stamp_contradicts_the_row(root: Path) -> None:
path = root / "1" / "outbox" / "r1-a1-proposal.json"
data = json.loads(path.read_text(encoding="utf-8"))
data["provenance"]["validator_decision"] = "rejected"
_write(path, data)
def _mut_the_outcome_type_contradicts_the_row(root: Path) -> None:
_edit(root / "1" / "outbox" / "r1-a1-outcome.json", outcome_type="rejected")
def _mut_the_figure_is_not_the_proposals(root: Path) -> None:
"""The IR says one figure and the coverage another — with the key RE-MINTED off the changed
IR, so this arm is felled by the figures and not by the key check standing in front of it."""
path = root / "1" / "outbox" / "r1-a1-proposal.json"
data = json.loads(path.read_text(encoding="utf-8"))
data["proposal"] = _ir("a1", 424242.0)
_write(path, data)
_edit(
root / "1" / "outbox" / "r1-a1-outcome.json",
verdict_id=verdict_key(features_from_ir(data["proposal"])),
)
def _mut_the_reason_is_not_the_rows(root: Path) -> None:
_edit(root / "0" / "outbox" / "r0-a1-outcome.json", reason="en annen grunn")
def _mut_an_artefact_belongs_to_another_run(root: Path) -> None:
_run_family(
root / "1" / "outbox",
"r1",
[{"id": "a9", "status": "validated", "detail": "", "saving_nok": 5.0}],
)
def _mut_an_unevaluated_approach_has_artefacts(root: Path) -> None:
_outcome(
root / "1",
[_row("a1", True, 1000.0, "f1"), _row("a2", False, None, stage="not_evaluated")],
)
_run_family(
root / "1" / "outbox",
"r1",
[{"id": "a2", "status": "rejected", "detail": "x", "saving_nok": None}],
)
def _mut_the_run_time_has_no_zone(root: Path) -> None:
_edit(root / "1" / "outcome.json", ran_at="2026-09-18T10:00:00")
@pytest.mark.parametrize(
("mutate", "marker"),
[
(_mut_declares_its_own_outbox, "outbox kan ikke oppgis"),
(_mut_coverage_is_another_run, "coverage-fila er kjøring"),
(_mut_the_approach_has_no_artefact, "r1-a1-outcome.json mangler"),
(_mut_the_key_is_not_the_irs, "verdict_id er ikke nøkkelen"),
(_mut_the_stamp_contradicts_the_row, "validator_decision"),
(_mut_the_outcome_type_contradicts_the_row, "utfallet er"),
(_mut_the_figure_is_not_the_proposals, "forslaget selv sier"),
(_mut_the_reason_is_not_the_rows, "utfallets grunn"),
(_mut_an_artefact_belongs_to_another_run, "coverage ikke nevner"),
(_mut_an_unevaluated_approach_has_artefacts, "ble ikke evaluert, men kjøringen skrev"),
(_mut_the_run_time_has_no_zone, "ran_at er ikke et ISO"),
],
)
def test_m6_the_run_family_must_stand_up_to_itself(
tmp_path: Path, mutate: Any, marker: str
) -> None:
"""One binding at a time, on a tree that is green until the mutation lands — and each arm is
read back by the reason it was refused for, so an arm cannot pass because some OTHER rule
happened to fire. The rc-0 control is the unmutated tree, asserted first."""
root = _green_rounds(tmp_path)
assert gate.score_changes(root, 3, _AI).status == gate.GREEN
mutate(root)
row = gate.score_changes(root, 3, _AI)
assert row.status == gate.RED
assert any(marker in x for x in row.exceptions), (marker, row.exceptions)
def test_m6_an_mtime_is_not_evidence_and_no_longer_decides_anything(tmp_path: Path) -> None:
"""The forger's third move was ``os.utime``. Every artefact in a green tree is stamped far in
the future, in the wrong order, and row 2 does not move: the run's time is the round's
declared ``ran_at``, and the row says so out loud."""
root = _green_rounds(tmp_path)
for n, path in enumerate(sorted(root.rglob("*.json"))):
os.utime(path, (_T0 - n * 1000, _T0 - n * 1000))
row = gate.score_changes(root, 3, _AI)
assert (row.k, row.status) == (3, gate.GREEN), row.exceptions
assert row.attests == (gate.RUN_ATTESTATION,)
assert "bekrefter operatøren" in gate.RUN_ATTESTATION
def _outcome_obj(rows: list[dict[str, Any]], removed: dict[str, set[str]] | None = None) -> Any:
from datetime import datetime, timezone