feat(simulation): the demo's gate is anchored to real cost lines (P4 pkt. 0)

The validator can reconcile a proposal against the project's actual cost lines
(S4.0 stage 0), but only when the knowledge base ships a cost-baseline.json —
and no bundle under shared/ has one. So on stage the gate reasoned only about
numbers the proposal supplied itself.

The reserve can never receive the file in shared/ (pull-only subtree, and demo
criterion 8 requires the goldens byte-unchanged). That is a placement
constraint, not an impossibility: materialize_anchored_bundle copies the bundle
and adds the file outside shared/, and the run path reads it through exactly
the seam a delivered bundle would use.

The baseline is DERIVED IN CODE from the scripted register, never typed beside
it — two sources of the same numbers drift, and drift is precisely what the
10 % probe models. On GO day the direction reverses (plan P3 b). Both scripted
replies must state the same cost lines or ValueError: were they to differ,
hypothesis #1 would be falsified by stage 0 instead of by P90 — the same
REJECTED line on screen, a different mechanism behind it.

10 % probe, measured: baseline x 1.10 -> FORKASTET at stage 0, before the
solver; corrected -> FORESLÅTT. Criterion 6 re-measured (stdout byte-identical
across two runs); stderr unchanged at 6 lines. The ONLY diff against the
un-anchored demo is the new KUNNSKAPSBASE block — everything else is
byte-identical, which is the problem: an anchoring nobody can see is one nobody
can check. Hence it is printed, and hence `provenance` is a required argument.
769 -> 775 passed.

Five mutations red + green control. The measurement failed the TEST first:
"ingen kostbaseline erklært" CONTAINS "kostbaseline erklært", and
ENERGI-TOTAL-EL already appears in the Step-2 line, so both assertions survived
the detach mutation. The two branches now share no wording.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GD6Y2Y23NZZxPYtSRoCmst
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 14:35:07 +02:00
commit 1522e2aaaa
5 changed files with 364 additions and 4 deletions

View file

@ -0,0 +1,176 @@
"""P4 pkt. 0 — the demo's RESERVE bundle must anchor the deterministic gate to real cost lines.
The gap (egnethetsreview Funn 1, corrected by objection I1): the validator CAN reconcile a proposal
against the project's actual cost lines (S4.0, stage 0), but only when the knowledge base ships a
``cost-baseline.json``. No bundle under ``shared/examples/`` has that file so in the demo the
validator reasoned only about numbers the proposal itself supplied, and an internally consistent
hallucination would clear the gate on stage.
The reserve cannot receive the file IN ``shared/``: the subtree is pull-only and demo criterion 8
requires the commons-owned goldens byte-unchanged. But that is a PLACEMENT constraint, not an
impossibility the demo already runs on a COPY of the bundle, so a copy-and-extend variant gives an
anchored run without touching commons.
**Direction of derivation, and why it matters.** Here the baseline is derived FROM the scripted
register: the reserve's numbers are synthetic, so the script is the only ground truth available.
On GO day the direction reverses (plan P3 b) the register's numbers are written FROM the
delivered ``cost-baseline.json``. Deriving in code, not by hand, is what stops the two from drifting
apart; drift is precisely the failure the 10 % test below models.
**The 10 % test** is the answer to "you generated the ground truth from the answer, so of course it
passes": deviate the baseline beyond the 5 % tolerance and the same, unchanged script must be
FORKASTET at stage 0 before the solver while the undeviated run is FORESLÅTT.
"""
from __future__ import annotations
import json
import subprocess
import sys
import pytest
from portfolio_optimiser import okf
from portfolio_optimiser.ir import CostBaseline, CostBaselineLine
from portfolio_optimiser.simulation import (
ScriptedCandidate,
_default_bundle_dir,
baseline_from_scripted_candidate,
materialize_anchored_bundle,
simulate_learning_loop,
)
from portfolio_optimiser.validator import Rejection, ValidatedProposal
def _deviated(baseline: CostBaseline, factor: float) -> CostBaseline:
"""The same baseline with every quantity scaled — the delivered numbers disagreeing with the
script's by ``factor``, which is exactly the GO-day risk this models."""
return CostBaseline(
project_id=baseline.project_id,
items={
code: CostBaselineLine(quantity=line.quantity * factor, unit_cost=line.unit_cost)
for code, line in baseline.items.items()
},
)
async def test_the_anchored_reserve_runs_the_whole_demo(tmp_path) -> None:
"""CONTROL: with the baseline derived from the script, the anchored reserve behaves exactly as
the demo narrates hypothesis #1 falsified by the P90 stage, the corrected one validated.
This is the control that gives the 10 % test its meaning: a gate that rejects everything proves
nothing. It also pins WHICH stage rejects hypothesis #1 — if stage 0 started rejecting it, demo
criterion 2 would still show a REJECTED and a VALIDATED line while silently demonstrating a
different mechanism."""
bundle = materialize_anchored_bundle(tmp_path / "forankret")
result = await simulate_learning_loop(str(bundle), str(tmp_path))
assert isinstance(result.run_a.outcome, ValidatedProposal)
assert isinstance(result.run_b.outcome, ValidatedProposal)
assert result.run_a.refinements, "no falsification was fed back — Step 5 is not being shown"
assert "exceeds P90 feasible" in result.run_a.refinements[0].reason, (
"hypothesis #1 was rejected by some other stage than the P90 one the demo narrates"
)
async def test_a_deviating_baseline_forkaster_the_demo_run_before_the_solver(tmp_path) -> None:
"""LOAD-BEARING (the 10 % test): when the project's declared cost lines deviate by 10 % from
the numbers the script asserts, the run is FORKASTET at stage 0 with the reconciliation
reason, not the P90 one.
Goes RED the moment the demo stops being anchored: without the ``cost-baseline.json`` in the
bundle the run path passes ``baseline=None``, stage 0 is skipped, and this same deviating
number changes nothing at all (the run ends FORESLÅTT, as ``test_..._runs_the_whole_demo``
above shows). The script is byte-identical in both tests only the declared baseline moves."""
baseline = _deviated(baseline_from_scripted_candidate(_only_candidate()), 1.10)
bundle = materialize_anchored_bundle(tmp_path / "forankret", baseline=baseline)
result = await simulate_learning_loop(str(bundle), str(tmp_path))
outcome = result.run_a.outcome
assert isinstance(outcome, Rejection), (
"a proposal 10 % away from the project's declared cost lines was NOT rejected — the "
"deterministic gate is not anchored to the baseline"
)
assert "outside the 5.0% tolerance" in outcome.reason
assert "ENERGI-TOTAL-EL" in outcome.reason
assert "P90" not in outcome.reason, (
"rejected by the solver stage, not by the reconciliation stage 0 that must run BEFORE it"
)
def test_the_reserve_itself_ships_no_baseline(tmp_path) -> None:
"""The materializer must ADD something the reserve genuinely lacks — and must leave the
commons-owned bundle alone (criterion 8: the goldens stay byte-unchanged)."""
reserve = _default_bundle_dir()
assert okf.load_optional_cost_baseline(str(reserve)) is None, (
"the shared reserve now ships a cost baseline — the copy-and-extend variant is obsolete "
"and this whole seam should be re-measured"
)
bundle = materialize_anchored_bundle(tmp_path / "forankret")
assert okf.load_optional_cost_baseline(str(bundle)) is not None, (
"the materialized bundle is not readable by okf's own loader — the filename has drifted"
)
assert okf.load_optional_cost_baseline(str(reserve)) is None
def test_the_baseline_is_derived_from_the_scripted_register() -> None:
"""The baseline is DERIVED from the script's own cost lines, never typed alongside them: a
hand-written copy is a second source of the same numbers, and two sources drift."""
candidate = _only_candidate()
derived = baseline_from_scripted_candidate(candidate)
for reply in (candidate.overclaimed, candidate.corrected):
for item in json.loads(reply)["affected_items"]:
line = derived.items[item["code"]]
assert (line.quantity, line.unit_cost) == (item["quantity"], item["unit_cost"])
assert derived.project_id == candidate.project_id
def test_a_candidate_whose_two_replies_disagree_is_refused() -> None:
"""Validation, never repair. The two scripted replies must state the SAME cost lines: were they
to differ, hypothesis #1 would be rejected by stage 0 instead of by P90, and the demo's
REJECTED line would silently come from another mechanism than the one it narrates."""
candidate = _only_candidate()
skewed = ScriptedCandidate(
project_id=candidate.project_id,
overclaimed=candidate.overclaimed.replace("300000", "310000"),
corrected=candidate.corrected,
flip_key=candidate.flip_key,
)
with pytest.raises(ValueError):
baseline_from_scripted_candidate(skewed)
def test_the_demo_entry_point_runs_the_anchored_reserve() -> None:
"""LOAD-BEARING on the CALL SITE: the thing the operator actually runs on stage must be the
anchored variant. Goes RED if ``main`` is pointed back at the plain reserve.
The declared baseline is printed because an anchoring nobody can see is an anchoring nobody can
check: every other line of the demo is byte-identical whether the gate is anchored or not.
**The first form of this test was vacuous, and the mutation caught it.** It asserted
``"kostbaseline erklært" in stdout`` but the un-anchored branch read "ingen kostbaseline
erklært", which CONTAINS that substring; and ``"ENERGI-TOTAL-EL" in stdout`` holds either way,
because the Step-2 line prints the proposal's own cost lines. Both survived the mutation. The
assertions below name the whole declared line and rule the other branch out explicitly."""
proc = subprocess.run(
[sys.executable, "-m", "portfolio_optimiser.simulation"],
capture_output=True,
text=True,
check=False,
)
assert proc.returncode == 0, proc.stderr
assert "kostbaseline erklært (ENERGI-TOTAL-EL 300000 x 1)" in proc.stdout, (
"the demo ran against a bundle with no cost baseline — the deterministic gate on stage is "
"reasoning only about the numbers the proposal supplied itself"
)
assert "validatorens stage 0 avstemmer" in proc.stdout
assert "uten kostbaseline" not in proc.stdout
def _only_candidate() -> ScriptedCandidate:
from portfolio_optimiser.simulation import _CANDIDATES, _PROJECT_ID
(candidate,) = [c for c in _CANDIDATES if c.project_id == _PROJECT_ID]
return candidate