feat(simulation): the demo's gate is anchored to real cost lines (P4 pkt. 0)
The validator can reconcile a proposal against the project's actual cost lines (S4.0 stage 0), but only when the knowledge base ships a cost-baseline.json — and no bundle under shared/ has one. So on stage the gate reasoned only about numbers the proposal supplied itself. The reserve can never receive the file in shared/ (pull-only subtree, and demo criterion 8 requires the goldens byte-unchanged). That is a placement constraint, not an impossibility: materialize_anchored_bundle copies the bundle and adds the file outside shared/, and the run path reads it through exactly the seam a delivered bundle would use. The baseline is DERIVED IN CODE from the scripted register, never typed beside it — two sources of the same numbers drift, and drift is precisely what the 10 % probe models. On GO day the direction reverses (plan P3 b). Both scripted replies must state the same cost lines or ValueError: were they to differ, hypothesis #1 would be falsified by stage 0 instead of by P90 — the same REJECTED line on screen, a different mechanism behind it. 10 % probe, measured: baseline x 1.10 -> FORKASTET at stage 0, before the solver; corrected -> FORESLÅTT. Criterion 6 re-measured (stdout byte-identical across two runs); stderr unchanged at 6 lines. The ONLY diff against the un-anchored demo is the new KUNNSKAPSBASE block — everything else is byte-identical, which is the problem: an anchoring nobody can see is one nobody can check. Hence it is printed, and hence `provenance` is a required argument. 769 -> 775 passed. Five mutations red + green control. The measurement failed the TEST first: "ingen kostbaseline erklært" CONTAINS "kostbaseline erklært", and ENERGI-TOTAL-EL already appears in the Step-2 line, so both assertions survived the detach mutation. The two branches now share no wording. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GD6Y2Y23NZZxPYtSRoCmst
This commit is contained in:
parent
1e11dcb96c
commit
1522e2aaaa
5 changed files with 364 additions and 4 deletions
176
tests/test_anchored_reserve_loadbearing.py
Normal file
176
tests/test_anchored_reserve_loadbearing.py
Normal file
|
|
@ -0,0 +1,176 @@
|
|||
"""P4 pkt. 0 — the demo's RESERVE bundle must anchor the deterministic gate to real cost lines.
|
||||
|
||||
The gap (egnethetsreview Funn 1, corrected by objection I1): the validator CAN reconcile a proposal
|
||||
against the project's actual cost lines (S4.0, stage 0), but only when the knowledge base ships a
|
||||
``cost-baseline.json``. No bundle under ``shared/examples/`` has that file — so in the demo the
|
||||
validator reasoned only about numbers the proposal itself supplied, and an internally consistent
|
||||
hallucination would clear the gate on stage.
|
||||
|
||||
The reserve cannot receive the file IN ``shared/``: the subtree is pull-only and demo criterion 8
|
||||
requires the commons-owned goldens byte-unchanged. But that is a PLACEMENT constraint, not an
|
||||
impossibility — the demo already runs on a COPY of the bundle, so a copy-and-extend variant gives an
|
||||
anchored run without touching commons.
|
||||
|
||||
**Direction of derivation, and why it matters.** Here the baseline is derived FROM the scripted
|
||||
register: the reserve's numbers are synthetic, so the script is the only ground truth available.
|
||||
On GO day the direction reverses (plan P3 b) — the register's numbers are written FROM the
|
||||
delivered ``cost-baseline.json``. Deriving in code, not by hand, is what stops the two from drifting
|
||||
apart; drift is precisely the failure the 10 % test below models.
|
||||
|
||||
**The 10 % test** is the answer to "you generated the ground truth from the answer, so of course it
|
||||
passes": deviate the baseline beyond the 5 % tolerance and the same, unchanged script must be
|
||||
FORKASTET at stage 0 — before the solver — while the undeviated run is FORESLÅTT.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser import okf
|
||||
from portfolio_optimiser.ir import CostBaseline, CostBaselineLine
|
||||
from portfolio_optimiser.simulation import (
|
||||
ScriptedCandidate,
|
||||
_default_bundle_dir,
|
||||
baseline_from_scripted_candidate,
|
||||
materialize_anchored_bundle,
|
||||
simulate_learning_loop,
|
||||
)
|
||||
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
||||
|
||||
|
||||
def _deviated(baseline: CostBaseline, factor: float) -> CostBaseline:
|
||||
"""The same baseline with every quantity scaled — the delivered numbers disagreeing with the
|
||||
script's by ``factor``, which is exactly the GO-day risk this models."""
|
||||
return CostBaseline(
|
||||
project_id=baseline.project_id,
|
||||
items={
|
||||
code: CostBaselineLine(quantity=line.quantity * factor, unit_cost=line.unit_cost)
|
||||
for code, line in baseline.items.items()
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
async def test_the_anchored_reserve_runs_the_whole_demo(tmp_path) -> None:
|
||||
"""CONTROL: with the baseline derived from the script, the anchored reserve behaves exactly as
|
||||
the demo narrates — hypothesis #1 falsified by the P90 stage, the corrected one validated.
|
||||
|
||||
This is the control that gives the 10 % test its meaning: a gate that rejects everything proves
|
||||
nothing. It also pins WHICH stage rejects hypothesis #1 — if stage 0 started rejecting it, demo
|
||||
criterion 2 would still show a REJECTED and a VALIDATED line while silently demonstrating a
|
||||
different mechanism."""
|
||||
bundle = materialize_anchored_bundle(tmp_path / "forankret")
|
||||
result = await simulate_learning_loop(str(bundle), str(tmp_path))
|
||||
|
||||
assert isinstance(result.run_a.outcome, ValidatedProposal)
|
||||
assert isinstance(result.run_b.outcome, ValidatedProposal)
|
||||
assert result.run_a.refinements, "no falsification was fed back — Step 5 is not being shown"
|
||||
assert "exceeds P90 feasible" in result.run_a.refinements[0].reason, (
|
||||
"hypothesis #1 was rejected by some other stage than the P90 one the demo narrates"
|
||||
)
|
||||
|
||||
|
||||
async def test_a_deviating_baseline_forkaster_the_demo_run_before_the_solver(tmp_path) -> None:
|
||||
"""LOAD-BEARING (the 10 % test): when the project's declared cost lines deviate by 10 % from
|
||||
the numbers the script asserts, the run is FORKASTET at stage 0 — with the reconciliation
|
||||
reason, not the P90 one.
|
||||
|
||||
Goes RED the moment the demo stops being anchored: without the ``cost-baseline.json`` in the
|
||||
bundle the run path passes ``baseline=None``, stage 0 is skipped, and this same deviating
|
||||
number changes nothing at all (the run ends FORESLÅTT, as ``test_..._runs_the_whole_demo``
|
||||
above shows). The script is byte-identical in both tests — only the declared baseline moves."""
|
||||
baseline = _deviated(baseline_from_scripted_candidate(_only_candidate()), 1.10)
|
||||
bundle = materialize_anchored_bundle(tmp_path / "forankret", baseline=baseline)
|
||||
result = await simulate_learning_loop(str(bundle), str(tmp_path))
|
||||
|
||||
outcome = result.run_a.outcome
|
||||
assert isinstance(outcome, Rejection), (
|
||||
"a proposal 10 % away from the project's declared cost lines was NOT rejected — the "
|
||||
"deterministic gate is not anchored to the baseline"
|
||||
)
|
||||
assert "outside the 5.0% tolerance" in outcome.reason
|
||||
assert "ENERGI-TOTAL-EL" in outcome.reason
|
||||
assert "P90" not in outcome.reason, (
|
||||
"rejected by the solver stage, not by the reconciliation stage 0 that must run BEFORE it"
|
||||
)
|
||||
|
||||
|
||||
def test_the_reserve_itself_ships_no_baseline(tmp_path) -> None:
|
||||
"""The materializer must ADD something the reserve genuinely lacks — and must leave the
|
||||
commons-owned bundle alone (criterion 8: the goldens stay byte-unchanged)."""
|
||||
reserve = _default_bundle_dir()
|
||||
assert okf.load_optional_cost_baseline(str(reserve)) is None, (
|
||||
"the shared reserve now ships a cost baseline — the copy-and-extend variant is obsolete "
|
||||
"and this whole seam should be re-measured"
|
||||
)
|
||||
|
||||
bundle = materialize_anchored_bundle(tmp_path / "forankret")
|
||||
assert okf.load_optional_cost_baseline(str(bundle)) is not None, (
|
||||
"the materialized bundle is not readable by okf's own loader — the filename has drifted"
|
||||
)
|
||||
assert okf.load_optional_cost_baseline(str(reserve)) is None
|
||||
|
||||
|
||||
def test_the_baseline_is_derived_from_the_scripted_register() -> None:
|
||||
"""The baseline is DERIVED from the script's own cost lines, never typed alongside them: a
|
||||
hand-written copy is a second source of the same numbers, and two sources drift."""
|
||||
candidate = _only_candidate()
|
||||
derived = baseline_from_scripted_candidate(candidate)
|
||||
|
||||
for reply in (candidate.overclaimed, candidate.corrected):
|
||||
for item in json.loads(reply)["affected_items"]:
|
||||
line = derived.items[item["code"]]
|
||||
assert (line.quantity, line.unit_cost) == (item["quantity"], item["unit_cost"])
|
||||
assert derived.project_id == candidate.project_id
|
||||
|
||||
|
||||
def test_a_candidate_whose_two_replies_disagree_is_refused() -> None:
|
||||
"""Validation, never repair. The two scripted replies must state the SAME cost lines: were they
|
||||
to differ, hypothesis #1 would be rejected by stage 0 instead of by P90, and the demo's
|
||||
REJECTED line would silently come from another mechanism than the one it narrates."""
|
||||
candidate = _only_candidate()
|
||||
skewed = ScriptedCandidate(
|
||||
project_id=candidate.project_id,
|
||||
overclaimed=candidate.overclaimed.replace("300000", "310000"),
|
||||
corrected=candidate.corrected,
|
||||
flip_key=candidate.flip_key,
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
baseline_from_scripted_candidate(skewed)
|
||||
|
||||
|
||||
def test_the_demo_entry_point_runs_the_anchored_reserve() -> None:
|
||||
"""LOAD-BEARING on the CALL SITE: the thing the operator actually runs on stage must be the
|
||||
anchored variant. Goes RED if ``main`` is pointed back at the plain reserve.
|
||||
|
||||
The declared baseline is printed because an anchoring nobody can see is an anchoring nobody can
|
||||
check: every other line of the demo is byte-identical whether the gate is anchored or not.
|
||||
|
||||
**The first form of this test was vacuous, and the mutation caught it.** It asserted
|
||||
``"kostbaseline erklært" in stdout`` — but the un-anchored branch read "ingen kostbaseline
|
||||
erklært", which CONTAINS that substring; and ``"ENERGI-TOTAL-EL" in stdout`` holds either way,
|
||||
because the Step-2 line prints the proposal's own cost lines. Both survived the mutation. The
|
||||
assertions below name the whole declared line and rule the other branch out explicitly."""
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "-m", "portfolio_optimiser.simulation"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
assert "kostbaseline erklært (ENERGI-TOTAL-EL 300000 x 1)" in proc.stdout, (
|
||||
"the demo ran against a bundle with no cost baseline — the deterministic gate on stage is "
|
||||
"reasoning only about the numbers the proposal supplied itself"
|
||||
)
|
||||
assert "validatorens stage 0 avstemmer" in proc.stdout
|
||||
assert "uten kostbaseline" not in proc.stdout
|
||||
|
||||
|
||||
def _only_candidate() -> ScriptedCandidate:
|
||||
from portfolio_optimiser.simulation import _CANDIDATES, _PROJECT_ID
|
||||
|
||||
(candidate,) = [c for c in _CANDIDATES if c.project_id == _PROJECT_ID]
|
||||
return candidate
|
||||
Loading…
Add table
Add a link
Reference in a new issue