feat(simulation): the demo's gate is anchored to real cost lines (P4 pkt. 0)
The validator can reconcile a proposal against the project's actual cost lines (S4.0 stage 0), but only when the knowledge base ships a cost-baseline.json — and no bundle under shared/ has one. So on stage the gate reasoned only about numbers the proposal supplied itself. The reserve can never receive the file in shared/ (pull-only subtree, and demo criterion 8 requires the goldens byte-unchanged). That is a placement constraint, not an impossibility: materialize_anchored_bundle copies the bundle and adds the file outside shared/, and the run path reads it through exactly the seam a delivered bundle would use. The baseline is DERIVED IN CODE from the scripted register, never typed beside it — two sources of the same numbers drift, and drift is precisely what the 10 % probe models. On GO day the direction reverses (plan P3 b). Both scripted replies must state the same cost lines or ValueError: were they to differ, hypothesis #1 would be falsified by stage 0 instead of by P90 — the same REJECTED line on screen, a different mechanism behind it. 10 % probe, measured: baseline x 1.10 -> FORKASTET at stage 0, before the solver; corrected -> FORESLÅTT. Criterion 6 re-measured (stdout byte-identical across two runs); stderr unchanged at 6 lines. The ONLY diff against the un-anchored demo is the new KUNNSKAPSBASE block — everything else is byte-identical, which is the problem: an anchoring nobody can see is one nobody can check. Hence it is printed, and hence `provenance` is a required argument. 769 -> 775 passed. Five mutations red + green control. The measurement failed the TEST first: "ingen kostbaseline erklært" CONTAINS "kostbaseline erklært", and ENERGI-TOTAL-EL already appears in the Step-2 line, so both assertions survived the detach mutation. The two branches now share no wording. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GD6Y2Y23NZZxPYtSRoCmst
This commit is contained in:
parent
1e11dcb96c
commit
1522e2aaaa
5 changed files with 364 additions and 4 deletions
21
CLAUDE.md
21
CLAUDE.md
|
|
@ -301,6 +301,27 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
tvetydighet · detach `project_id`-argumentet. **Flip-key-testen ble skrevet om under målingen** —
|
||||
første form asserterte på FØRSTE register-oppføring, der «den matchede kandidatens nøkkel» og
|
||||
«`candidates[0]`s nøkkel» sammenfaller; den kunne ikke skille de to implementasjonene.
|
||||
- **Demoen kjører FORANKRET, og baselinen DERIVERES fra manuset (P4 pkt. 0):** før dette regnet
|
||||
validatoren i demoen kun på tall forslaget selv oppga — S4.0-forankringen aktiveres bare når
|
||||
kunnskapsbasen shipper `cost-baseline.json`, og ingen bundle under `shared/` har den. Reserven kan
|
||||
aldri få fila DER (pull-only subtree + kriterium 8 krever goldenene byte-uendret), men det er en
|
||||
*plasserings*-begrensning: `materialize_anchored_bundle` KOPIERER bundelen og legger fila til
|
||||
utenfor `shared/`, og kjørestien (`run.py` → `load_optional_cost_baseline`) er da NØYAKTIG samme
|
||||
søm en levert bundle ville brukt. **Retningen på avledningen er bærende:** reservens tall er
|
||||
syntetiske, så manus-registeret er eneste grunnsannhet — `baseline_from_scripted_candidate`
|
||||
avleder i KODE, aldri en andre håndskrevet kopi av de samme tallene (to kilder drifter, og drift
|
||||
er nøyaktig det 10 %-prøven modellerer). På GO-dagen snus retningen (plan P3 b: registeret skrives
|
||||
FRA levert fil). Begge skriptede svar må oppgi SAMME kostlinjer (`ValueError` ellers): var de
|
||||
ulike, ville hypotese #1 blitt avvist av stage 0 istedenfor av P90 — samme REJECTED-linje på
|
||||
skjermen, annen mekanisme. **Forankringen er usynlig i alt annet stdout** (målt: eneste diff mot
|
||||
uforankret er KUNNSKAPSBASE-blokka), derfor printes den erklærte baselinen, og derfor er
|
||||
`provenance` et PÅKREVD argument til `_baseline_lines` — kallstedet som velger bundelen er det
|
||||
eneste som vet hvor tallene kom fra. Load-bearing MÅLT
|
||||
(`tests/test_anchored_reserve_loadbearing.py`), fem mutasjoner røde + grønn kontroll: detach
|
||||
main-wiringen · detach fil-skrivingen · la filnavnet drifte · detach to-svars-enigheten · returner
|
||||
et literal i stedet for det avledede. **Målingen felte TESTEN først** (samme klasse som 08-06):
|
||||
«ingen kostbaseline erklært» INNEHOLDER «kostbaseline erklært», og `ENERGI-TOTAL-EL` står allerede
|
||||
i Steg 2-linja — begge assertene overlevde detach-mutasjonen. De to grenene deler nå ingen ordlyd.
|
||||
- **Delt ekspert-persona som Agent Skill (§8, framework-nøytral):** ekspert-reviewer-personaen bor i
|
||||
`shared/skills/expert-reviewer/` (`SKILL.md` + `references/example-verdict.json`) og er den ENE
|
||||
delte artefakten begge stacker instansierer reviewer-en fra. `shared/` forblir REN DATA — MAF-siden
|
||||
|
|
|
|||
|
|
@ -66,6 +66,13 @@ reviewer would days later — is merged from disk into the next run's store befo
|
|||
formed (write file → merge → fold). Both markers are present in Run B's prompt and absent from
|
||||
Run A's. Nothing crosses in memory.
|
||||
|
||||
The run is **anchored**: the demo copies the example knowledge base and adds a `cost-baseline.json`
|
||||
— the project's actual cost lines — so the validator's first stage reconciles every cost line a
|
||||
proposal claims against them, before the solver runs. The declared baseline is printed at the top,
|
||||
because it is the one property the rest of the output looks identical with or without. Those
|
||||
numbers are synthetic, derived from the scripted replies themselves; a knowledge base that ships
|
||||
its own baseline replaces them through the same seam, with no code change.
|
||||
|
||||
**3 — Run the loop over a knowledge base, with answers you supply.** Write the stand-in replies,
|
||||
then point the CLI at the bundle:
|
||||
|
||||
|
|
|
|||
|
|
@ -195,7 +195,8 @@ et planlagt utfall, ikke en krise — generalprøve nr. 0 mandag har allerede ve
|
|||
|
||||
### P4 — Forankret prøvekjøring + fire kriterier + to setninger (FORSKUTTERES i helgen; onsdag gjenstår kun re-måling mot valgt innhold) ☐
|
||||
**[0,75–1,25 økt · Opus 5/low–medium · alt under bygges/måles mot mikro-reserven i helgen — se §5]**
|
||||
0. **Forankret prøvekjøring (NY 08-07, I1 — ukas største måletekniske risiko, nå TIDLIGST):**
|
||||
0. **Forankret prøvekjøring ✔ (2026-08-09)** — se UTFØRT-blokka under punkt 5.
|
||||
**(NY 08-07, I1 — ukas største måletekniske risiko, nå TIDLIGST):**
|
||||
lag en repo-lokal kopier-og-utvid-bundle — mikro-reservens innhold + `cost-baseline.json` i
|
||||
S4.0-formatet fixturen definerer (`project_id` + `items{code:{quantity,unit_cost}}`), med
|
||||
baseline-tall avledet FRA manus-registerets tall (samme disiplin som P3 b, speilvendt). Kjør
|
||||
|
|
@ -228,6 +229,36 @@ et planlagt utfall, ikke en krise — generalprøve nr. 0 mandag har allerede ve
|
|||
install-flate som endres på frysedagen ville krevd ny fresh-clone-måling ETTER frys —
|
||||
selvmotsigende.
|
||||
|
||||
**UTFØRT — punkt 0 (2026-08-09).** `materialize_anchored_bundle` kopierer reserven og legger til
|
||||
`cost-baseline.json` utenfor `shared/`; `main()` kjører den varianten, og hele demoløpet er kjørt
|
||||
mot den. **Baselinen avledes i KODE fra manus-registeret** (`baseline_from_scripted_candidate`),
|
||||
ikke skrevet ved siden av det — på GO-dagen snus retningen (punkt b i P3), og en håndskrevet kopi
|
||||
ville vært den andre kilden som drifter. Begge skriptede svar må oppgi samme kostlinjer, ellers
|
||||
`ValueError`: var de ulike, ville hypotese #1 blitt avvist av stage 0 istedenfor av P90, og demoens
|
||||
REJECTED-linje kommet fra en annen mekanisme enn den den forteller om.
|
||||
|
||||
**10 %-prøven, målt:** bevisst avvik (baseline ×1,10) → `FORKASTET — quantity 300000 for cost code
|
||||
'ENERGI-TOTAL-EL' is outside the 5.0% tolerance around the baseline quantity 330000`, i stage 0,
|
||||
FØR løseren. Korrigert (manus = baseline) → `FORESLÅTT — LED-retrofit av kontorbelysning: 30000 NOK`.
|
||||
Kontroll: hypotese #1 avvises fortsatt av P90-stagen, så demo-kriterium 2 viser samme mekanisme som
|
||||
før. Suite 769 → 775 passed / 4 skipped.
|
||||
|
||||
**Kriterium 6 re-målt:** stdout byte-identisk mellom to kjøringer; stderr uendret 6 linjer. Eneste
|
||||
diff mot uforankret demo er den nye fire-linjers KUNNSKAPSBASE-blokka — **alt annet er
|
||||
byte-identisk, og det er selve problemet:** forankringen er usynlig, derfor må den printes, og
|
||||
derfor er `provenance` et påkrevd argument (kallstedet som velger bundelen er det eneste som vet
|
||||
hvor tallene kom fra). Punkt 4s NO-GO-setning står nå ordrett på skjermen.
|
||||
|
||||
**Målingen felte TESTEN først.** Første form av entry-point-testen asserterte
|
||||
`"kostbaseline erklært" in stdout` — men den uforankrede grenen sa «ingen kostbaseline erklært»,
|
||||
som INNEHOLDER strengen; og `ENERGI-TOTAL-EL` står allerede i Steg 2-linja. Detach-mutasjonen gikk
|
||||
GRØNN. De to grenene deler nå ingen ordlyd, og asserten navngir hele linja. Fem mutasjoner røde +
|
||||
grønn kontroll (`tests/test_anchored_reserve_loadbearing.py`).
|
||||
|
||||
**GO-dagens endring er ÉN blokk i `main()`:** `bundle = materialize_anchored_bundle(...)` +
|
||||
`provenance = _RESERVE_PROVENANCE` byttes mot den leverte bundelens sti og dens egen provenans.
|
||||
Kjørestien er uendret — `run.py` leser baselinen fra hvilken som helst bundle-katalog.
|
||||
|
||||
### P4.5 — Demo-runbook: ÉN side operatøren følger på scenen (NY 08-07, I5) ☐
|
||||
**[0,25 økt · Opus 5/low · produseres VED frysen onsdag, ETTER generalprøve ×2 — så den matcher frosset output]**
|
||||
Det som skal SIES torsdag ligger i dag på fire steder: demo-uke-planen §1 (tre ærlighets-punkter),
|
||||
|
|
@ -303,7 +334,7 @@ og risikoen ligger tidlig med slakk bak seg.
|
|||
| Dag | Innhold |
|
||||
|---|---|
|
||||
| fre 7. | **Planrevisjonen (I1–I6)** ✔ + **P1/S1.a**: Steg 7-innboksen inn i demoløpet (økt 1) |
|
||||
| lør 8.–søn 9. | **P2/S1.b**: innholdsgaten (økt 2, evt. 3) + **P4-forskuddet UTVIDET** (0,75–1,25 økt), i denne rekkefølgen: forankret prøvekjøring + 10 %-prøven (pkt. 0, I1) → `[project.scripts]` (pkt. 5, I3) → stderr-demping FØR pinning (pkt. 2, I2) → fresh-clone (pkt. 1) → golden-transkript-mekanikk (pkt. 3) → de to ærlighets-setningene (pkt. 4) — alt bygges og måles mot den FORANKREDE mikro-reserven NÅ |
|
||||
| lør 8.–søn 9. | **P2/S1.b**: innholdsgaten (økt 2, evt. 3) + **P4-forskuddet UTVIDET** (0,75–1,25 økt), i denne rekkefølgen: forankret prøvekjøring + 10 %-prøven (pkt. 0, I1) **✔ 08-09** → `[project.scripts]` (pkt. 5, I3) → stderr-demping FØR pinning (pkt. 2, I2) → fresh-clone (pkt. 1) → golden-transkript-mekanikk (pkt. 3) → de to ærlighets-setningene (pkt. 4) — alt bygges og måles mot den FORANKREDE mikro-reserven NÅ |
|
||||
| man 10. | **Generalprøve nr. 0 mot forankret reserve**: alle kriterier ende-til-ende → NO-GO-utfallet er ferdig verifisert FØR tirsdag; rest-slakk er buffer |
|
||||
| tir 11. | **P3**: noter pre-pull-hash → subtree pull → GO/NO-GO (**abortsti I4:** rød suite/goldens → `git reset --hard <pre-pull-hash>` + NO-GO senest **kl. 18:00**) + RE-måling: kriterium 1+2 mot NY bundle; manus-tall skrives FRA levert `cost-baseline.json`; 10 %-prøven gjentatt; goldens byte-uendret |
|
||||
| ons 12. | **P4** re-målt mot valgt innhold → generalprøve ×2 → **FRYS** → **P4.5: demo-runbook (I5)** → **S1.c: versjonssynk + CHANGELOG + tag `v1.0.0`** (uten `[project.scripts]` — den ligger i helgen, I3) |
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ client is MAF-side scaffolding; it is NOT part of the framework-neutral ``shared
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
from collections.abc import Awaitable, Callable, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
|
|
@ -36,7 +37,8 @@ from agent_framework import (
|
|||
)
|
||||
from agent_framework_openai import OpenAIChatCompletionClient
|
||||
|
||||
from portfolio_optimiser.ir import SavingsProposal
|
||||
from portfolio_optimiser import okf
|
||||
from portfolio_optimiser.ir import AffectedItem, CostBaseline, CostBaselineLine, SavingsProposal
|
||||
from portfolio_optimiser.persona import load_persona_example
|
||||
from portfolio_optimiser.run import RunResult, run_project
|
||||
from portfolio_optimiser.shared_root import shared_root
|
||||
|
|
@ -167,6 +169,89 @@ _CANDIDATES: tuple[ScriptedCandidate, ...] = (
|
|||
|
||||
_proposer_reply = scripted_proposer(_CANDIDATES)
|
||||
|
||||
# The filename ``okf``'s loaders default to. Kept local rather than reaching into ``okf``'s private
|
||||
# constant; the coupling is measured, not assumed — a drifted name makes the materialized bundle
|
||||
# un-anchored, which ``test_anchored_reserve_loadbearing`` turns red.
|
||||
_COST_BASELINE_FILE = "cost-baseline.json"
|
||||
_ANCHORED_DIR_NAME = "forankret-reserve"
|
||||
_RESERVE_PROVENANCE = "tallene er syntetiske — avledet av demo-manuset, ikke levert av et fagmiljø"
|
||||
|
||||
|
||||
def baseline_from_scripted_candidate(candidate: ScriptedCandidate) -> CostBaseline:
|
||||
"""Derive a project's cost baseline FROM the scripted register's own cost lines (P4 pkt. 0).
|
||||
|
||||
The reserve bundle's numbers are synthetic, so the script is the only ground truth there is;
|
||||
deriving in code rather than typing the same numbers into a second file is what keeps the two
|
||||
from drifting apart. **On GO day the direction reverses** (plan P3 b): the register is written
|
||||
FROM the delivered ``cost-baseline.json``, and this function is not used.
|
||||
|
||||
Both scripted replies must state the SAME cost lines, or ``ValueError``. Validation, never
|
||||
repair: were they to differ, hypothesis #1 would be falsified by the reconciliation stage
|
||||
instead of by the P90 stage, and the demo's REJECTED line would come from another mechanism
|
||||
than the one it narrates — visible on screen as the same line either way.
|
||||
"""
|
||||
lines = {
|
||||
item.code: CostBaselineLine(quantity=item.quantity, unit_cost=item.unit_cost)
|
||||
for item in (
|
||||
AffectedItem.model_validate(raw)
|
||||
for raw in json.loads(candidate.corrected)["affected_items"]
|
||||
)
|
||||
}
|
||||
overclaimed = {
|
||||
raw["code"]: (raw["quantity"], raw["unit_cost"])
|
||||
for raw in json.loads(candidate.overclaimed)["affected_items"]
|
||||
}
|
||||
if overclaimed != {code: (line.quantity, line.unit_cost) for code, line in lines.items()}:
|
||||
raise ValueError(
|
||||
f"scripted candidate {candidate.project_id} states different cost lines in its two "
|
||||
"replies; a baseline derived from one of them would falsify the other at stage 0"
|
||||
)
|
||||
return CostBaseline(project_id=candidate.project_id, items=lines)
|
||||
|
||||
|
||||
def _reserve_baseline() -> CostBaseline:
|
||||
"""The reserve bundle's baseline: the registry entry for the reserve's project, never
|
||||
``_CANDIDATES[0]`` — the registry is a set of DATA entries whose order carries no meaning, and
|
||||
an index would silently anchor the demo to another project once a second entry lands."""
|
||||
(candidate,) = [c for c in _CANDIDATES if c.project_id == _PROJECT_ID]
|
||||
return baseline_from_scripted_candidate(candidate)
|
||||
|
||||
|
||||
def materialize_anchored_bundle(
|
||||
dest: str | Path,
|
||||
*,
|
||||
source: str | Path | None = None,
|
||||
baseline: CostBaseline | None = None,
|
||||
) -> Path:
|
||||
"""Copy the reserve bundle and ADD the ``cost-baseline.json`` it cannot be given in place — the
|
||||
copy-and-extend variant that anchors the deterministic gate (S4.0 stage 0) for the demo.
|
||||
|
||||
``shared/`` is a pull-only subtree and demo criterion 8 requires the commons-owned goldens
|
||||
byte-unchanged, so the reserve can never ship the file itself. That is a PLACEMENT constraint,
|
||||
not an impossibility: the run already reads the baseline from whichever bundle directory it is
|
||||
handed (``run.py`` -> ``okf.load_optional_cost_baseline``), so an extended copy outside
|
||||
``shared/`` is anchored by exactly the same seam a delivered bundle would use.
|
||||
|
||||
``baseline`` defaults to the one derived from the scripted register; a caller passes its own to
|
||||
model a DELIVERED baseline that disagrees with the script (the 10 % test).
|
||||
"""
|
||||
src = Path(source) if source is not None else _default_bundle_dir()
|
||||
out = Path(dest)
|
||||
shutil.copytree(src, out)
|
||||
resolved = baseline if baseline is not None else _reserve_baseline()
|
||||
payload = {
|
||||
"_note": (
|
||||
"SYNTHETIC cost baseline, materialized for the demo (P4 pkt. 0) — NOT delivered data. "
|
||||
"Derived from the scripted register in portfolio_optimiser.simulation so the two "
|
||||
"cannot drift. The source bundle is never modified."
|
||||
),
|
||||
**resolved.model_dump(),
|
||||
}
|
||||
(out / _COST_BASELINE_FILE).write_text(
|
||||
json.dumps(payload, indent=2, ensure_ascii=False) + "\n", encoding="utf-8"
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
# The checker's debate turn ends with the gate marker the run parses (run._checker_verdict).
|
||||
_CHECKER_APPROVE = "Tallene er innenfor feasibelt område og resonnementet holder. VERDICT: APPROVE"
|
||||
|
|
@ -550,6 +635,37 @@ def _run_trace_lines(result: RunResult, *, marker: str, marker_in_prompt: bool)
|
|||
return lines
|
||||
|
||||
|
||||
def _baseline_lines(bundle_dir: Path, provenance: str) -> list[str]:
|
||||
"""What the knowledge base DECLARES about the project's own cost lines.
|
||||
|
||||
Read off the BUNDLE, not off a ``RunResult`` — which is why it is printed by ``main`` and not by
|
||||
``_run_trace_lines`` (whose contract is that every value comes from the run). Whether the run
|
||||
then USES the baseline is not something a screen can show: every other line of the demo is
|
||||
byte-identical anchored or not, so that property is measured by
|
||||
``tests/test_anchored_reserve_loadbearing.py`` instead.
|
||||
|
||||
``provenance`` is a required argument, not a default: the caller who chooses the bundle is the
|
||||
only one who knows where its numbers came from, and saying so is the honesty claim itself.
|
||||
|
||||
The two branches share no wording (measured: an "ingen kostbaseline erklært" phrasing CONTAINS
|
||||
"kostbaseline erklært", which made the entry-point test pass with the anchoring detached)."""
|
||||
baseline = okf.load_optional_cost_baseline(str(bundle_dir))
|
||||
if baseline is None:
|
||||
return [
|
||||
f"KUNNSKAPSBASE: {bundle_dir.name} — uten kostbaseline",
|
||||
" validatoren regner kun på tallene forslaget selv oppgir",
|
||||
]
|
||||
items = ", ".join(
|
||||
f"{code} {line.quantity:g} x {line.unit_cost:g}"
|
||||
for code, line in sorted(baseline.items.items())
|
||||
)
|
||||
return [
|
||||
f"KUNNSKAPSBASE: {bundle_dir.name} — kostbaseline erklært ({items})",
|
||||
" validatorens stage 0 avstemmer forslagets kostlinjer mot disse, FØR løseren",
|
||||
f" {provenance}",
|
||||
]
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int: # pragma: no cover - console trace
|
||||
"""Run the simulation against the energi bundle in a throwaway temp dir and print an honest,
|
||||
readable trace. Invoke: ``uv run python -m portfolio_optimiser.simulation``."""
|
||||
|
|
@ -558,7 +674,12 @@ def main(argv: list[str] | None = None) -> int: # pragma: no cover - console tr
|
|||
import tempfile
|
||||
|
||||
work = tempfile.mkdtemp(prefix="po-sim-")
|
||||
result = asyncio.run(simulate_learning_loop(str(_default_bundle_dir()), work))
|
||||
# THE call site (P4 pkt. 0): the demo runs the ANCHORED reserve — the shared bundle plus the
|
||||
# cost baseline it cannot be given in place. On GO day these two lines point at the delivered
|
||||
# bundle and state ITS provenance instead; the run path's seam is the same either way.
|
||||
bundle = materialize_anchored_bundle(Path(work) / _ANCHORED_DIR_NAME)
|
||||
provenance = _RESERVE_PROVENANCE
|
||||
result = asyncio.run(simulate_learning_loop(str(bundle), work))
|
||||
|
||||
print("=" * 78)
|
||||
print("OFFLINE SIMULERING — skriptede agent-svar, INGEN ekte modell.")
|
||||
|
|
@ -566,6 +687,10 @@ def main(argv: list[str] | None = None) -> int: # pragma: no cover - console tr
|
|||
print("Beviser IKKE at en levende modell ville produsert dette — forslag og dom er skriptet.")
|
||||
print("=" * 78)
|
||||
|
||||
print()
|
||||
for line in _baseline_lines(bundle, provenance):
|
||||
print(line)
|
||||
|
||||
# Run A walks steps 1-7 of the method; the promotion between the runs IS step 8. Run B is not
|
||||
# re-numbered — it re-runs the same eight steps, and what the demo needs from it is the ONE
|
||||
# thing that changed: the marker now reaches the hypothesis prompt.
|
||||
|
|
|
|||
176
tests/test_anchored_reserve_loadbearing.py
Normal file
176
tests/test_anchored_reserve_loadbearing.py
Normal file
|
|
@ -0,0 +1,176 @@
|
|||
"""P4 pkt. 0 — the demo's RESERVE bundle must anchor the deterministic gate to real cost lines.
|
||||
|
||||
The gap (egnethetsreview Funn 1, corrected by objection I1): the validator CAN reconcile a proposal
|
||||
against the project's actual cost lines (S4.0, stage 0), but only when the knowledge base ships a
|
||||
``cost-baseline.json``. No bundle under ``shared/examples/`` has that file — so in the demo the
|
||||
validator reasoned only about numbers the proposal itself supplied, and an internally consistent
|
||||
hallucination would clear the gate on stage.
|
||||
|
||||
The reserve cannot receive the file IN ``shared/``: the subtree is pull-only and demo criterion 8
|
||||
requires the commons-owned goldens byte-unchanged. But that is a PLACEMENT constraint, not an
|
||||
impossibility — the demo already runs on a COPY of the bundle, so a copy-and-extend variant gives an
|
||||
anchored run without touching commons.
|
||||
|
||||
**Direction of derivation, and why it matters.** Here the baseline is derived FROM the scripted
|
||||
register: the reserve's numbers are synthetic, so the script is the only ground truth available.
|
||||
On GO day the direction reverses (plan P3 b) — the register's numbers are written FROM the
|
||||
delivered ``cost-baseline.json``. Deriving in code, not by hand, is what stops the two from drifting
|
||||
apart; drift is precisely the failure the 10 % test below models.
|
||||
|
||||
**The 10 % test** is the answer to "you generated the ground truth from the answer, so of course it
|
||||
passes": deviate the baseline beyond the 5 % tolerance and the same, unchanged script must be
|
||||
FORKASTET at stage 0 — before the solver — while the undeviated run is FORESLÅTT.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser import okf
|
||||
from portfolio_optimiser.ir import CostBaseline, CostBaselineLine
|
||||
from portfolio_optimiser.simulation import (
|
||||
ScriptedCandidate,
|
||||
_default_bundle_dir,
|
||||
baseline_from_scripted_candidate,
|
||||
materialize_anchored_bundle,
|
||||
simulate_learning_loop,
|
||||
)
|
||||
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
||||
|
||||
|
||||
def _deviated(baseline: CostBaseline, factor: float) -> CostBaseline:
|
||||
"""The same baseline with every quantity scaled — the delivered numbers disagreeing with the
|
||||
script's by ``factor``, which is exactly the GO-day risk this models."""
|
||||
return CostBaseline(
|
||||
project_id=baseline.project_id,
|
||||
items={
|
||||
code: CostBaselineLine(quantity=line.quantity * factor, unit_cost=line.unit_cost)
|
||||
for code, line in baseline.items.items()
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
async def test_the_anchored_reserve_runs_the_whole_demo(tmp_path) -> None:
|
||||
"""CONTROL: with the baseline derived from the script, the anchored reserve behaves exactly as
|
||||
the demo narrates — hypothesis #1 falsified by the P90 stage, the corrected one validated.
|
||||
|
||||
This is the control that gives the 10 % test its meaning: a gate that rejects everything proves
|
||||
nothing. It also pins WHICH stage rejects hypothesis #1 — if stage 0 started rejecting it, demo
|
||||
criterion 2 would still show a REJECTED and a VALIDATED line while silently demonstrating a
|
||||
different mechanism."""
|
||||
bundle = materialize_anchored_bundle(tmp_path / "forankret")
|
||||
result = await simulate_learning_loop(str(bundle), str(tmp_path))
|
||||
|
||||
assert isinstance(result.run_a.outcome, ValidatedProposal)
|
||||
assert isinstance(result.run_b.outcome, ValidatedProposal)
|
||||
assert result.run_a.refinements, "no falsification was fed back — Step 5 is not being shown"
|
||||
assert "exceeds P90 feasible" in result.run_a.refinements[0].reason, (
|
||||
"hypothesis #1 was rejected by some other stage than the P90 one the demo narrates"
|
||||
)
|
||||
|
||||
|
||||
async def test_a_deviating_baseline_forkaster_the_demo_run_before_the_solver(tmp_path) -> None:
|
||||
"""LOAD-BEARING (the 10 % test): when the project's declared cost lines deviate by 10 % from
|
||||
the numbers the script asserts, the run is FORKASTET at stage 0 — with the reconciliation
|
||||
reason, not the P90 one.
|
||||
|
||||
Goes RED the moment the demo stops being anchored: without the ``cost-baseline.json`` in the
|
||||
bundle the run path passes ``baseline=None``, stage 0 is skipped, and this same deviating
|
||||
number changes nothing at all (the run ends FORESLÅTT, as ``test_..._runs_the_whole_demo``
|
||||
above shows). The script is byte-identical in both tests — only the declared baseline moves."""
|
||||
baseline = _deviated(baseline_from_scripted_candidate(_only_candidate()), 1.10)
|
||||
bundle = materialize_anchored_bundle(tmp_path / "forankret", baseline=baseline)
|
||||
result = await simulate_learning_loop(str(bundle), str(tmp_path))
|
||||
|
||||
outcome = result.run_a.outcome
|
||||
assert isinstance(outcome, Rejection), (
|
||||
"a proposal 10 % away from the project's declared cost lines was NOT rejected — the "
|
||||
"deterministic gate is not anchored to the baseline"
|
||||
)
|
||||
assert "outside the 5.0% tolerance" in outcome.reason
|
||||
assert "ENERGI-TOTAL-EL" in outcome.reason
|
||||
assert "P90" not in outcome.reason, (
|
||||
"rejected by the solver stage, not by the reconciliation stage 0 that must run BEFORE it"
|
||||
)
|
||||
|
||||
|
||||
def test_the_reserve_itself_ships_no_baseline(tmp_path) -> None:
|
||||
"""The materializer must ADD something the reserve genuinely lacks — and must leave the
|
||||
commons-owned bundle alone (criterion 8: the goldens stay byte-unchanged)."""
|
||||
reserve = _default_bundle_dir()
|
||||
assert okf.load_optional_cost_baseline(str(reserve)) is None, (
|
||||
"the shared reserve now ships a cost baseline — the copy-and-extend variant is obsolete "
|
||||
"and this whole seam should be re-measured"
|
||||
)
|
||||
|
||||
bundle = materialize_anchored_bundle(tmp_path / "forankret")
|
||||
assert okf.load_optional_cost_baseline(str(bundle)) is not None, (
|
||||
"the materialized bundle is not readable by okf's own loader — the filename has drifted"
|
||||
)
|
||||
assert okf.load_optional_cost_baseline(str(reserve)) is None
|
||||
|
||||
|
||||
def test_the_baseline_is_derived_from_the_scripted_register() -> None:
|
||||
"""The baseline is DERIVED from the script's own cost lines, never typed alongside them: a
|
||||
hand-written copy is a second source of the same numbers, and two sources drift."""
|
||||
candidate = _only_candidate()
|
||||
derived = baseline_from_scripted_candidate(candidate)
|
||||
|
||||
for reply in (candidate.overclaimed, candidate.corrected):
|
||||
for item in json.loads(reply)["affected_items"]:
|
||||
line = derived.items[item["code"]]
|
||||
assert (line.quantity, line.unit_cost) == (item["quantity"], item["unit_cost"])
|
||||
assert derived.project_id == candidate.project_id
|
||||
|
||||
|
||||
def test_a_candidate_whose_two_replies_disagree_is_refused() -> None:
|
||||
"""Validation, never repair. The two scripted replies must state the SAME cost lines: were they
|
||||
to differ, hypothesis #1 would be rejected by stage 0 instead of by P90, and the demo's
|
||||
REJECTED line would silently come from another mechanism than the one it narrates."""
|
||||
candidate = _only_candidate()
|
||||
skewed = ScriptedCandidate(
|
||||
project_id=candidate.project_id,
|
||||
overclaimed=candidate.overclaimed.replace("300000", "310000"),
|
||||
corrected=candidate.corrected,
|
||||
flip_key=candidate.flip_key,
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
baseline_from_scripted_candidate(skewed)
|
||||
|
||||
|
||||
def test_the_demo_entry_point_runs_the_anchored_reserve() -> None:
|
||||
"""LOAD-BEARING on the CALL SITE: the thing the operator actually runs on stage must be the
|
||||
anchored variant. Goes RED if ``main`` is pointed back at the plain reserve.
|
||||
|
||||
The declared baseline is printed because an anchoring nobody can see is an anchoring nobody can
|
||||
check: every other line of the demo is byte-identical whether the gate is anchored or not.
|
||||
|
||||
**The first form of this test was vacuous, and the mutation caught it.** It asserted
|
||||
``"kostbaseline erklært" in stdout`` — but the un-anchored branch read "ingen kostbaseline
|
||||
erklært", which CONTAINS that substring; and ``"ENERGI-TOTAL-EL" in stdout`` holds either way,
|
||||
because the Step-2 line prints the proposal's own cost lines. Both survived the mutation. The
|
||||
assertions below name the whole declared line and rule the other branch out explicitly."""
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "-m", "portfolio_optimiser.simulation"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
assert "kostbaseline erklært (ENERGI-TOTAL-EL 300000 x 1)" in proc.stdout, (
|
||||
"the demo ran against a bundle with no cost baseline — the deterministic gate on stage is "
|
||||
"reasoning only about the numbers the proposal supplied itself"
|
||||
)
|
||||
assert "validatorens stage 0 avstemmer" in proc.stdout
|
||||
assert "uten kostbaseline" not in proc.stdout
|
||||
|
||||
|
||||
def _only_candidate() -> ScriptedCandidate:
|
||||
from portfolio_optimiser.simulation import _CANDIDATES, _PROJECT_ID
|
||||
|
||||
(candidate,) = [c for c in _CANDIDATES if c.project_id == _PROJECT_ID]
|
||||
return candidate
|
||||
Loading…
Add table
Add a link
Reference in a new issue