feat(simulation): the demo's gate is anchored to real cost lines (P4 pkt. 0)
The validator can reconcile a proposal against the project's actual cost lines (S4.0 stage 0), but only when the knowledge base ships a cost-baseline.json — and no bundle under shared/ has one. So on stage the gate reasoned only about numbers the proposal supplied itself. The reserve can never receive the file in shared/ (pull-only subtree, and demo criterion 8 requires the goldens byte-unchanged). That is a placement constraint, not an impossibility: materialize_anchored_bundle copies the bundle and adds the file outside shared/, and the run path reads it through exactly the seam a delivered bundle would use. The baseline is DERIVED IN CODE from the scripted register, never typed beside it — two sources of the same numbers drift, and drift is precisely what the 10 % probe models. On GO day the direction reverses (plan P3 b). Both scripted replies must state the same cost lines or ValueError: were they to differ, hypothesis #1 would be falsified by stage 0 instead of by P90 — the same REJECTED line on screen, a different mechanism behind it. 10 % probe, measured: baseline x 1.10 -> FORKASTET at stage 0, before the solver; corrected -> FORESLÅTT. Criterion 6 re-measured (stdout byte-identical across two runs); stderr unchanged at 6 lines. The ONLY diff against the un-anchored demo is the new KUNNSKAPSBASE block — everything else is byte-identical, which is the problem: an anchoring nobody can see is one nobody can check. Hence it is printed, and hence `provenance` is a required argument. 769 -> 775 passed. Five mutations red + green control. The measurement failed the TEST first: "ingen kostbaseline erklært" CONTAINS "kostbaseline erklært", and ENERGI-TOTAL-EL already appears in the Step-2 line, so both assertions survived the detach mutation. The two branches now share no wording. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GD6Y2Y23NZZxPYtSRoCmst
This commit is contained in:
parent
1e11dcb96c
commit
1522e2aaaa
5 changed files with 364 additions and 4 deletions
21
CLAUDE.md
21
CLAUDE.md
|
|
@ -301,6 +301,27 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
||||||
tvetydighet · detach `project_id`-argumentet. **Flip-key-testen ble skrevet om under målingen** —
|
tvetydighet · detach `project_id`-argumentet. **Flip-key-testen ble skrevet om under målingen** —
|
||||||
første form asserterte på FØRSTE register-oppføring, der «den matchede kandidatens nøkkel» og
|
første form asserterte på FØRSTE register-oppføring, der «den matchede kandidatens nøkkel» og
|
||||||
«`candidates[0]`s nøkkel» sammenfaller; den kunne ikke skille de to implementasjonene.
|
«`candidates[0]`s nøkkel» sammenfaller; den kunne ikke skille de to implementasjonene.
|
||||||
|
- **Demoen kjører FORANKRET, og baselinen DERIVERES fra manuset (P4 pkt. 0):** før dette regnet
|
||||||
|
validatoren i demoen kun på tall forslaget selv oppga — S4.0-forankringen aktiveres bare når
|
||||||
|
kunnskapsbasen shipper `cost-baseline.json`, og ingen bundle under `shared/` har den. Reserven kan
|
||||||
|
aldri få fila DER (pull-only subtree + kriterium 8 krever goldenene byte-uendret), men det er en
|
||||||
|
*plasserings*-begrensning: `materialize_anchored_bundle` KOPIERER bundelen og legger fila til
|
||||||
|
utenfor `shared/`, og kjørestien (`run.py` → `load_optional_cost_baseline`) er da NØYAKTIG samme
|
||||||
|
søm en levert bundle ville brukt. **Retningen på avledningen er bærende:** reservens tall er
|
||||||
|
syntetiske, så manus-registeret er eneste grunnsannhet — `baseline_from_scripted_candidate`
|
||||||
|
avleder i KODE, aldri en andre håndskrevet kopi av de samme tallene (to kilder drifter, og drift
|
||||||
|
er nøyaktig det 10 %-prøven modellerer). På GO-dagen snus retningen (plan P3 b: registeret skrives
|
||||||
|
FRA levert fil). Begge skriptede svar må oppgi SAMME kostlinjer (`ValueError` ellers): var de
|
||||||
|
ulike, ville hypotese #1 blitt avvist av stage 0 istedenfor av P90 — samme REJECTED-linje på
|
||||||
|
skjermen, annen mekanisme. **Forankringen er usynlig i alt annet stdout** (målt: eneste diff mot
|
||||||
|
uforankret er KUNNSKAPSBASE-blokka), derfor printes den erklærte baselinen, og derfor er
|
||||||
|
`provenance` et PÅKREVD argument til `_baseline_lines` — kallstedet som velger bundelen er det
|
||||||
|
eneste som vet hvor tallene kom fra. Load-bearing MÅLT
|
||||||
|
(`tests/test_anchored_reserve_loadbearing.py`), fem mutasjoner røde + grønn kontroll: detach
|
||||||
|
main-wiringen · detach fil-skrivingen · la filnavnet drifte · detach to-svars-enigheten · returner
|
||||||
|
et literal i stedet for det avledede. **Målingen felte TESTEN først** (samme klasse som 08-06):
|
||||||
|
«ingen kostbaseline erklært» INNEHOLDER «kostbaseline erklært», og `ENERGI-TOTAL-EL` står allerede
|
||||||
|
i Steg 2-linja — begge assertene overlevde detach-mutasjonen. De to grenene deler nå ingen ordlyd.
|
||||||
- **Delt ekspert-persona som Agent Skill (§8, framework-nøytral):** ekspert-reviewer-personaen bor i
|
- **Delt ekspert-persona som Agent Skill (§8, framework-nøytral):** ekspert-reviewer-personaen bor i
|
||||||
`shared/skills/expert-reviewer/` (`SKILL.md` + `references/example-verdict.json`) og er den ENE
|
`shared/skills/expert-reviewer/` (`SKILL.md` + `references/example-verdict.json`) og er den ENE
|
||||||
delte artefakten begge stacker instansierer reviewer-en fra. `shared/` forblir REN DATA — MAF-siden
|
delte artefakten begge stacker instansierer reviewer-en fra. `shared/` forblir REN DATA — MAF-siden
|
||||||
|
|
|
||||||
|
|
@ -66,6 +66,13 @@ reviewer would days later — is merged from disk into the next run's store befo
|
||||||
formed (write file → merge → fold). Both markers are present in Run B's prompt and absent from
|
formed (write file → merge → fold). Both markers are present in Run B's prompt and absent from
|
||||||
Run A's. Nothing crosses in memory.
|
Run A's. Nothing crosses in memory.
|
||||||
|
|
||||||
|
The run is **anchored**: the demo copies the example knowledge base and adds a `cost-baseline.json`
|
||||||
|
— the project's actual cost lines — so the validator's first stage reconciles every cost line a
|
||||||
|
proposal claims against them, before the solver runs. The declared baseline is printed at the top,
|
||||||
|
because it is the one property the rest of the output looks identical with or without. Those
|
||||||
|
numbers are synthetic, derived from the scripted replies themselves; a knowledge base that ships
|
||||||
|
its own baseline replaces them through the same seam, with no code change.
|
||||||
|
|
||||||
**3 — Run the loop over a knowledge base, with answers you supply.** Write the stand-in replies,
|
**3 — Run the loop over a knowledge base, with answers you supply.** Write the stand-in replies,
|
||||||
then point the CLI at the bundle:
|
then point the CLI at the bundle:
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -195,7 +195,8 @@ et planlagt utfall, ikke en krise — generalprøve nr. 0 mandag har allerede ve
|
||||||
|
|
||||||
### P4 — Forankret prøvekjøring + fire kriterier + to setninger (FORSKUTTERES i helgen; onsdag gjenstår kun re-måling mot valgt innhold) ☐
|
### P4 — Forankret prøvekjøring + fire kriterier + to setninger (FORSKUTTERES i helgen; onsdag gjenstår kun re-måling mot valgt innhold) ☐
|
||||||
**[0,75–1,25 økt · Opus 5/low–medium · alt under bygges/måles mot mikro-reserven i helgen — se §5]**
|
**[0,75–1,25 økt · Opus 5/low–medium · alt under bygges/måles mot mikro-reserven i helgen — se §5]**
|
||||||
0. **Forankret prøvekjøring (NY 08-07, I1 — ukas største måletekniske risiko, nå TIDLIGST):**
|
0. **Forankret prøvekjøring ✔ (2026-08-09)** — se UTFØRT-blokka under punkt 5.
|
||||||
|
**(NY 08-07, I1 — ukas største måletekniske risiko, nå TIDLIGST):**
|
||||||
lag en repo-lokal kopier-og-utvid-bundle — mikro-reservens innhold + `cost-baseline.json` i
|
lag en repo-lokal kopier-og-utvid-bundle — mikro-reservens innhold + `cost-baseline.json` i
|
||||||
S4.0-formatet fixturen definerer (`project_id` + `items{code:{quantity,unit_cost}}`), med
|
S4.0-formatet fixturen definerer (`project_id` + `items{code:{quantity,unit_cost}}`), med
|
||||||
baseline-tall avledet FRA manus-registerets tall (samme disiplin som P3 b, speilvendt). Kjør
|
baseline-tall avledet FRA manus-registerets tall (samme disiplin som P3 b, speilvendt). Kjør
|
||||||
|
|
@ -228,6 +229,36 @@ et planlagt utfall, ikke en krise — generalprøve nr. 0 mandag har allerede ve
|
||||||
install-flate som endres på frysedagen ville krevd ny fresh-clone-måling ETTER frys —
|
install-flate som endres på frysedagen ville krevd ny fresh-clone-måling ETTER frys —
|
||||||
selvmotsigende.
|
selvmotsigende.
|
||||||
|
|
||||||
|
**UTFØRT — punkt 0 (2026-08-09).** `materialize_anchored_bundle` kopierer reserven og legger til
|
||||||
|
`cost-baseline.json` utenfor `shared/`; `main()` kjører den varianten, og hele demoløpet er kjørt
|
||||||
|
mot den. **Baselinen avledes i KODE fra manus-registeret** (`baseline_from_scripted_candidate`),
|
||||||
|
ikke skrevet ved siden av det — på GO-dagen snus retningen (punkt b i P3), og en håndskrevet kopi
|
||||||
|
ville vært den andre kilden som drifter. Begge skriptede svar må oppgi samme kostlinjer, ellers
|
||||||
|
`ValueError`: var de ulike, ville hypotese #1 blitt avvist av stage 0 istedenfor av P90, og demoens
|
||||||
|
REJECTED-linje kommet fra en annen mekanisme enn den den forteller om.
|
||||||
|
|
||||||
|
**10 %-prøven, målt:** bevisst avvik (baseline ×1,10) → `FORKASTET — quantity 300000 for cost code
|
||||||
|
'ENERGI-TOTAL-EL' is outside the 5.0% tolerance around the baseline quantity 330000`, i stage 0,
|
||||||
|
FØR løseren. Korrigert (manus = baseline) → `FORESLÅTT — LED-retrofit av kontorbelysning: 30000 NOK`.
|
||||||
|
Kontroll: hypotese #1 avvises fortsatt av P90-stagen, så demo-kriterium 2 viser samme mekanisme som
|
||||||
|
før. Suite 769 → 775 passed / 4 skipped.
|
||||||
|
|
||||||
|
**Kriterium 6 re-målt:** stdout byte-identisk mellom to kjøringer; stderr uendret 6 linjer. Eneste
|
||||||
|
diff mot uforankret demo er den nye fire-linjers KUNNSKAPSBASE-blokka — **alt annet er
|
||||||
|
byte-identisk, og det er selve problemet:** forankringen er usynlig, derfor må den printes, og
|
||||||
|
derfor er `provenance` et påkrevd argument (kallstedet som velger bundelen er det eneste som vet
|
||||||
|
hvor tallene kom fra). Punkt 4s NO-GO-setning står nå ordrett på skjermen.
|
||||||
|
|
||||||
|
**Målingen felte TESTEN først.** Første form av entry-point-testen asserterte
|
||||||
|
`"kostbaseline erklært" in stdout` — men den uforankrede grenen sa «ingen kostbaseline erklært»,
|
||||||
|
som INNEHOLDER strengen; og `ENERGI-TOTAL-EL` står allerede i Steg 2-linja. Detach-mutasjonen gikk
|
||||||
|
GRØNN. De to grenene deler nå ingen ordlyd, og asserten navngir hele linja. Fem mutasjoner røde +
|
||||||
|
grønn kontroll (`tests/test_anchored_reserve_loadbearing.py`).
|
||||||
|
|
||||||
|
**GO-dagens endring er ÉN blokk i `main()`:** `bundle = materialize_anchored_bundle(...)` +
|
||||||
|
`provenance = _RESERVE_PROVENANCE` byttes mot den leverte bundelens sti og dens egen provenans.
|
||||||
|
Kjørestien er uendret — `run.py` leser baselinen fra hvilken som helst bundle-katalog.
|
||||||
|
|
||||||
### P4.5 — Demo-runbook: ÉN side operatøren følger på scenen (NY 08-07, I5) ☐
|
### P4.5 — Demo-runbook: ÉN side operatøren følger på scenen (NY 08-07, I5) ☐
|
||||||
**[0,25 økt · Opus 5/low · produseres VED frysen onsdag, ETTER generalprøve ×2 — så den matcher frosset output]**
|
**[0,25 økt · Opus 5/low · produseres VED frysen onsdag, ETTER generalprøve ×2 — så den matcher frosset output]**
|
||||||
Det som skal SIES torsdag ligger i dag på fire steder: demo-uke-planen §1 (tre ærlighets-punkter),
|
Det som skal SIES torsdag ligger i dag på fire steder: demo-uke-planen §1 (tre ærlighets-punkter),
|
||||||
|
|
@ -303,7 +334,7 @@ og risikoen ligger tidlig med slakk bak seg.
|
||||||
| Dag | Innhold |
|
| Dag | Innhold |
|
||||||
|---|---|
|
|---|---|
|
||||||
| fre 7. | **Planrevisjonen (I1–I6)** ✔ + **P1/S1.a**: Steg 7-innboksen inn i demoløpet (økt 1) |
|
| fre 7. | **Planrevisjonen (I1–I6)** ✔ + **P1/S1.a**: Steg 7-innboksen inn i demoløpet (økt 1) |
|
||||||
| lør 8.–søn 9. | **P2/S1.b**: innholdsgaten (økt 2, evt. 3) + **P4-forskuddet UTVIDET** (0,75–1,25 økt), i denne rekkefølgen: forankret prøvekjøring + 10 %-prøven (pkt. 0, I1) → `[project.scripts]` (pkt. 5, I3) → stderr-demping FØR pinning (pkt. 2, I2) → fresh-clone (pkt. 1) → golden-transkript-mekanikk (pkt. 3) → de to ærlighets-setningene (pkt. 4) — alt bygges og måles mot den FORANKREDE mikro-reserven NÅ |
|
| lør 8.–søn 9. | **P2/S1.b**: innholdsgaten (økt 2, evt. 3) + **P4-forskuddet UTVIDET** (0,75–1,25 økt), i denne rekkefølgen: forankret prøvekjøring + 10 %-prøven (pkt. 0, I1) **✔ 08-09** → `[project.scripts]` (pkt. 5, I3) → stderr-demping FØR pinning (pkt. 2, I2) → fresh-clone (pkt. 1) → golden-transkript-mekanikk (pkt. 3) → de to ærlighets-setningene (pkt. 4) — alt bygges og måles mot den FORANKREDE mikro-reserven NÅ |
|
||||||
| man 10. | **Generalprøve nr. 0 mot forankret reserve**: alle kriterier ende-til-ende → NO-GO-utfallet er ferdig verifisert FØR tirsdag; rest-slakk er buffer |
|
| man 10. | **Generalprøve nr. 0 mot forankret reserve**: alle kriterier ende-til-ende → NO-GO-utfallet er ferdig verifisert FØR tirsdag; rest-slakk er buffer |
|
||||||
| tir 11. | **P3**: noter pre-pull-hash → subtree pull → GO/NO-GO (**abortsti I4:** rød suite/goldens → `git reset --hard <pre-pull-hash>` + NO-GO senest **kl. 18:00**) + RE-måling: kriterium 1+2 mot NY bundle; manus-tall skrives FRA levert `cost-baseline.json`; 10 %-prøven gjentatt; goldens byte-uendret |
|
| tir 11. | **P3**: noter pre-pull-hash → subtree pull → GO/NO-GO (**abortsti I4:** rød suite/goldens → `git reset --hard <pre-pull-hash>` + NO-GO senest **kl. 18:00**) + RE-måling: kriterium 1+2 mot NY bundle; manus-tall skrives FRA levert `cost-baseline.json`; 10 %-prøven gjentatt; goldens byte-uendret |
|
||||||
| ons 12. | **P4** re-målt mot valgt innhold → generalprøve ×2 → **FRYS** → **P4.5: demo-runbook (I5)** → **S1.c: versjonssynk + CHANGELOG + tag `v1.0.0`** (uten `[project.scripts]` — den ligger i helgen, I3) |
|
| ons 12. | **P4** re-målt mot valgt innhold → generalprøve ×2 → **FRYS** → **P4.5: demo-runbook (I5)** → **S1.c: versjonssynk + CHANGELOG + tag `v1.0.0`** (uten `[project.scripts]` — den ligger i helgen, I3) |
|
||||||
|
|
|
||||||
|
|
@ -20,6 +20,7 @@ client is MAF-side scaffolding; it is NOT part of the framework-neutral ``shared
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
import shutil
|
import shutil
|
||||||
from collections.abc import Awaitable, Callable, Mapping, Sequence
|
from collections.abc import Awaitable, Callable, Mapping, Sequence
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
|
|
@ -36,7 +37,8 @@ from agent_framework import (
|
||||||
)
|
)
|
||||||
from agent_framework_openai import OpenAIChatCompletionClient
|
from agent_framework_openai import OpenAIChatCompletionClient
|
||||||
|
|
||||||
from portfolio_optimiser.ir import SavingsProposal
|
from portfolio_optimiser import okf
|
||||||
|
from portfolio_optimiser.ir import AffectedItem, CostBaseline, CostBaselineLine, SavingsProposal
|
||||||
from portfolio_optimiser.persona import load_persona_example
|
from portfolio_optimiser.persona import load_persona_example
|
||||||
from portfolio_optimiser.run import RunResult, run_project
|
from portfolio_optimiser.run import RunResult, run_project
|
||||||
from portfolio_optimiser.shared_root import shared_root
|
from portfolio_optimiser.shared_root import shared_root
|
||||||
|
|
@ -167,6 +169,89 @@ _CANDIDATES: tuple[ScriptedCandidate, ...] = (
|
||||||
|
|
||||||
_proposer_reply = scripted_proposer(_CANDIDATES)
|
_proposer_reply = scripted_proposer(_CANDIDATES)
|
||||||
|
|
||||||
|
# The filename ``okf``'s loaders default to. Kept local rather than reaching into ``okf``'s private
|
||||||
|
# constant; the coupling is measured, not assumed — a drifted name makes the materialized bundle
|
||||||
|
# un-anchored, which ``test_anchored_reserve_loadbearing`` turns red.
|
||||||
|
_COST_BASELINE_FILE = "cost-baseline.json"
|
||||||
|
_ANCHORED_DIR_NAME = "forankret-reserve"
|
||||||
|
_RESERVE_PROVENANCE = "tallene er syntetiske — avledet av demo-manuset, ikke levert av et fagmiljø"
|
||||||
|
|
||||||
|
|
||||||
|
def baseline_from_scripted_candidate(candidate: ScriptedCandidate) -> CostBaseline:
|
||||||
|
"""Derive a project's cost baseline FROM the scripted register's own cost lines (P4 pkt. 0).
|
||||||
|
|
||||||
|
The reserve bundle's numbers are synthetic, so the script is the only ground truth there is;
|
||||||
|
deriving in code rather than typing the same numbers into a second file is what keeps the two
|
||||||
|
from drifting apart. **On GO day the direction reverses** (plan P3 b): the register is written
|
||||||
|
FROM the delivered ``cost-baseline.json``, and this function is not used.
|
||||||
|
|
||||||
|
Both scripted replies must state the SAME cost lines, or ``ValueError``. Validation, never
|
||||||
|
repair: were they to differ, hypothesis #1 would be falsified by the reconciliation stage
|
||||||
|
instead of by the P90 stage, and the demo's REJECTED line would come from another mechanism
|
||||||
|
than the one it narrates — visible on screen as the same line either way.
|
||||||
|
"""
|
||||||
|
lines = {
|
||||||
|
item.code: CostBaselineLine(quantity=item.quantity, unit_cost=item.unit_cost)
|
||||||
|
for item in (
|
||||||
|
AffectedItem.model_validate(raw)
|
||||||
|
for raw in json.loads(candidate.corrected)["affected_items"]
|
||||||
|
)
|
||||||
|
}
|
||||||
|
overclaimed = {
|
||||||
|
raw["code"]: (raw["quantity"], raw["unit_cost"])
|
||||||
|
for raw in json.loads(candidate.overclaimed)["affected_items"]
|
||||||
|
}
|
||||||
|
if overclaimed != {code: (line.quantity, line.unit_cost) for code, line in lines.items()}:
|
||||||
|
raise ValueError(
|
||||||
|
f"scripted candidate {candidate.project_id} states different cost lines in its two "
|
||||||
|
"replies; a baseline derived from one of them would falsify the other at stage 0"
|
||||||
|
)
|
||||||
|
return CostBaseline(project_id=candidate.project_id, items=lines)
|
||||||
|
|
||||||
|
|
||||||
|
def _reserve_baseline() -> CostBaseline:
|
||||||
|
"""The reserve bundle's baseline: the registry entry for the reserve's project, never
|
||||||
|
``_CANDIDATES[0]`` — the registry is a set of DATA entries whose order carries no meaning, and
|
||||||
|
an index would silently anchor the demo to another project once a second entry lands."""
|
||||||
|
(candidate,) = [c for c in _CANDIDATES if c.project_id == _PROJECT_ID]
|
||||||
|
return baseline_from_scripted_candidate(candidate)
|
||||||
|
|
||||||
|
|
||||||
|
def materialize_anchored_bundle(
|
||||||
|
dest: str | Path,
|
||||||
|
*,
|
||||||
|
source: str | Path | None = None,
|
||||||
|
baseline: CostBaseline | None = None,
|
||||||
|
) -> Path:
|
||||||
|
"""Copy the reserve bundle and ADD the ``cost-baseline.json`` it cannot be given in place — the
|
||||||
|
copy-and-extend variant that anchors the deterministic gate (S4.0 stage 0) for the demo.
|
||||||
|
|
||||||
|
``shared/`` is a pull-only subtree and demo criterion 8 requires the commons-owned goldens
|
||||||
|
byte-unchanged, so the reserve can never ship the file itself. That is a PLACEMENT constraint,
|
||||||
|
not an impossibility: the run already reads the baseline from whichever bundle directory it is
|
||||||
|
handed (``run.py`` -> ``okf.load_optional_cost_baseline``), so an extended copy outside
|
||||||
|
``shared/`` is anchored by exactly the same seam a delivered bundle would use.
|
||||||
|
|
||||||
|
``baseline`` defaults to the one derived from the scripted register; a caller passes its own to
|
||||||
|
model a DELIVERED baseline that disagrees with the script (the 10 % test).
|
||||||
|
"""
|
||||||
|
src = Path(source) if source is not None else _default_bundle_dir()
|
||||||
|
out = Path(dest)
|
||||||
|
shutil.copytree(src, out)
|
||||||
|
resolved = baseline if baseline is not None else _reserve_baseline()
|
||||||
|
payload = {
|
||||||
|
"_note": (
|
||||||
|
"SYNTHETIC cost baseline, materialized for the demo (P4 pkt. 0) — NOT delivered data. "
|
||||||
|
"Derived from the scripted register in portfolio_optimiser.simulation so the two "
|
||||||
|
"cannot drift. The source bundle is never modified."
|
||||||
|
),
|
||||||
|
**resolved.model_dump(),
|
||||||
|
}
|
||||||
|
(out / _COST_BASELINE_FILE).write_text(
|
||||||
|
json.dumps(payload, indent=2, ensure_ascii=False) + "\n", encoding="utf-8"
|
||||||
|
)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
# The checker's debate turn ends with the gate marker the run parses (run._checker_verdict).
|
# The checker's debate turn ends with the gate marker the run parses (run._checker_verdict).
|
||||||
_CHECKER_APPROVE = "Tallene er innenfor feasibelt område og resonnementet holder. VERDICT: APPROVE"
|
_CHECKER_APPROVE = "Tallene er innenfor feasibelt område og resonnementet holder. VERDICT: APPROVE"
|
||||||
|
|
@ -550,6 +635,37 @@ def _run_trace_lines(result: RunResult, *, marker: str, marker_in_prompt: bool)
|
||||||
return lines
|
return lines
|
||||||
|
|
||||||
|
|
||||||
|
def _baseline_lines(bundle_dir: Path, provenance: str) -> list[str]:
|
||||||
|
"""What the knowledge base DECLARES about the project's own cost lines.
|
||||||
|
|
||||||
|
Read off the BUNDLE, not off a ``RunResult`` — which is why it is printed by ``main`` and not by
|
||||||
|
``_run_trace_lines`` (whose contract is that every value comes from the run). Whether the run
|
||||||
|
then USES the baseline is not something a screen can show: every other line of the demo is
|
||||||
|
byte-identical anchored or not, so that property is measured by
|
||||||
|
``tests/test_anchored_reserve_loadbearing.py`` instead.
|
||||||
|
|
||||||
|
``provenance`` is a required argument, not a default: the caller who chooses the bundle is the
|
||||||
|
only one who knows where its numbers came from, and saying so is the honesty claim itself.
|
||||||
|
|
||||||
|
The two branches share no wording (measured: an "ingen kostbaseline erklært" phrasing CONTAINS
|
||||||
|
"kostbaseline erklært", which made the entry-point test pass with the anchoring detached)."""
|
||||||
|
baseline = okf.load_optional_cost_baseline(str(bundle_dir))
|
||||||
|
if baseline is None:
|
||||||
|
return [
|
||||||
|
f"KUNNSKAPSBASE: {bundle_dir.name} — uten kostbaseline",
|
||||||
|
" validatoren regner kun på tallene forslaget selv oppgir",
|
||||||
|
]
|
||||||
|
items = ", ".join(
|
||||||
|
f"{code} {line.quantity:g} x {line.unit_cost:g}"
|
||||||
|
for code, line in sorted(baseline.items.items())
|
||||||
|
)
|
||||||
|
return [
|
||||||
|
f"KUNNSKAPSBASE: {bundle_dir.name} — kostbaseline erklært ({items})",
|
||||||
|
" validatorens stage 0 avstemmer forslagets kostlinjer mot disse, FØR løseren",
|
||||||
|
f" {provenance}",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def main(argv: list[str] | None = None) -> int: # pragma: no cover - console trace
|
def main(argv: list[str] | None = None) -> int: # pragma: no cover - console trace
|
||||||
"""Run the simulation against the energi bundle in a throwaway temp dir and print an honest,
|
"""Run the simulation against the energi bundle in a throwaway temp dir and print an honest,
|
||||||
readable trace. Invoke: ``uv run python -m portfolio_optimiser.simulation``."""
|
readable trace. Invoke: ``uv run python -m portfolio_optimiser.simulation``."""
|
||||||
|
|
@ -558,7 +674,12 @@ def main(argv: list[str] | None = None) -> int: # pragma: no cover - console tr
|
||||||
import tempfile
|
import tempfile
|
||||||
|
|
||||||
work = tempfile.mkdtemp(prefix="po-sim-")
|
work = tempfile.mkdtemp(prefix="po-sim-")
|
||||||
result = asyncio.run(simulate_learning_loop(str(_default_bundle_dir()), work))
|
# THE call site (P4 pkt. 0): the demo runs the ANCHORED reserve — the shared bundle plus the
|
||||||
|
# cost baseline it cannot be given in place. On GO day these two lines point at the delivered
|
||||||
|
# bundle and state ITS provenance instead; the run path's seam is the same either way.
|
||||||
|
bundle = materialize_anchored_bundle(Path(work) / _ANCHORED_DIR_NAME)
|
||||||
|
provenance = _RESERVE_PROVENANCE
|
||||||
|
result = asyncio.run(simulate_learning_loop(str(bundle), work))
|
||||||
|
|
||||||
print("=" * 78)
|
print("=" * 78)
|
||||||
print("OFFLINE SIMULERING — skriptede agent-svar, INGEN ekte modell.")
|
print("OFFLINE SIMULERING — skriptede agent-svar, INGEN ekte modell.")
|
||||||
|
|
@ -566,6 +687,10 @@ def main(argv: list[str] | None = None) -> int: # pragma: no cover - console tr
|
||||||
print("Beviser IKKE at en levende modell ville produsert dette — forslag og dom er skriptet.")
|
print("Beviser IKKE at en levende modell ville produsert dette — forslag og dom er skriptet.")
|
||||||
print("=" * 78)
|
print("=" * 78)
|
||||||
|
|
||||||
|
print()
|
||||||
|
for line in _baseline_lines(bundle, provenance):
|
||||||
|
print(line)
|
||||||
|
|
||||||
# Run A walks steps 1-7 of the method; the promotion between the runs IS step 8. Run B is not
|
# Run A walks steps 1-7 of the method; the promotion between the runs IS step 8. Run B is not
|
||||||
# re-numbered — it re-runs the same eight steps, and what the demo needs from it is the ONE
|
# re-numbered — it re-runs the same eight steps, and what the demo needs from it is the ONE
|
||||||
# thing that changed: the marker now reaches the hypothesis prompt.
|
# thing that changed: the marker now reaches the hypothesis prompt.
|
||||||
|
|
|
||||||
176
tests/test_anchored_reserve_loadbearing.py
Normal file
176
tests/test_anchored_reserve_loadbearing.py
Normal file
|
|
@ -0,0 +1,176 @@
|
||||||
|
"""P4 pkt. 0 — the demo's RESERVE bundle must anchor the deterministic gate to real cost lines.
|
||||||
|
|
||||||
|
The gap (egnethetsreview Funn 1, corrected by objection I1): the validator CAN reconcile a proposal
|
||||||
|
against the project's actual cost lines (S4.0, stage 0), but only when the knowledge base ships a
|
||||||
|
``cost-baseline.json``. No bundle under ``shared/examples/`` has that file — so in the demo the
|
||||||
|
validator reasoned only about numbers the proposal itself supplied, and an internally consistent
|
||||||
|
hallucination would clear the gate on stage.
|
||||||
|
|
||||||
|
The reserve cannot receive the file IN ``shared/``: the subtree is pull-only and demo criterion 8
|
||||||
|
requires the commons-owned goldens byte-unchanged. But that is a PLACEMENT constraint, not an
|
||||||
|
impossibility — the demo already runs on a COPY of the bundle, so a copy-and-extend variant gives an
|
||||||
|
anchored run without touching commons.
|
||||||
|
|
||||||
|
**Direction of derivation, and why it matters.** Here the baseline is derived FROM the scripted
|
||||||
|
register: the reserve's numbers are synthetic, so the script is the only ground truth available.
|
||||||
|
On GO day the direction reverses (plan P3 b) — the register's numbers are written FROM the
|
||||||
|
delivered ``cost-baseline.json``. Deriving in code, not by hand, is what stops the two from drifting
|
||||||
|
apart; drift is precisely the failure the 10 % test below models.
|
||||||
|
|
||||||
|
**The 10 % test** is the answer to "you generated the ground truth from the answer, so of course it
|
||||||
|
passes": deviate the baseline beyond the 5 % tolerance and the same, unchanged script must be
|
||||||
|
FORKASTET at stage 0 — before the solver — while the undeviated run is FORESLÅTT.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from portfolio_optimiser import okf
|
||||||
|
from portfolio_optimiser.ir import CostBaseline, CostBaselineLine
|
||||||
|
from portfolio_optimiser.simulation import (
|
||||||
|
ScriptedCandidate,
|
||||||
|
_default_bundle_dir,
|
||||||
|
baseline_from_scripted_candidate,
|
||||||
|
materialize_anchored_bundle,
|
||||||
|
simulate_learning_loop,
|
||||||
|
)
|
||||||
|
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
||||||
|
|
||||||
|
|
||||||
|
def _deviated(baseline: CostBaseline, factor: float) -> CostBaseline:
|
||||||
|
"""The same baseline with every quantity scaled — the delivered numbers disagreeing with the
|
||||||
|
script's by ``factor``, which is exactly the GO-day risk this models."""
|
||||||
|
return CostBaseline(
|
||||||
|
project_id=baseline.project_id,
|
||||||
|
items={
|
||||||
|
code: CostBaselineLine(quantity=line.quantity * factor, unit_cost=line.unit_cost)
|
||||||
|
for code, line in baseline.items.items()
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_the_anchored_reserve_runs_the_whole_demo(tmp_path) -> None:
|
||||||
|
"""CONTROL: with the baseline derived from the script, the anchored reserve behaves exactly as
|
||||||
|
the demo narrates — hypothesis #1 falsified by the P90 stage, the corrected one validated.
|
||||||
|
|
||||||
|
This is the control that gives the 10 % test its meaning: a gate that rejects everything proves
|
||||||
|
nothing. It also pins WHICH stage rejects hypothesis #1 — if stage 0 started rejecting it, demo
|
||||||
|
criterion 2 would still show a REJECTED and a VALIDATED line while silently demonstrating a
|
||||||
|
different mechanism."""
|
||||||
|
bundle = materialize_anchored_bundle(tmp_path / "forankret")
|
||||||
|
result = await simulate_learning_loop(str(bundle), str(tmp_path))
|
||||||
|
|
||||||
|
assert isinstance(result.run_a.outcome, ValidatedProposal)
|
||||||
|
assert isinstance(result.run_b.outcome, ValidatedProposal)
|
||||||
|
assert result.run_a.refinements, "no falsification was fed back — Step 5 is not being shown"
|
||||||
|
assert "exceeds P90 feasible" in result.run_a.refinements[0].reason, (
|
||||||
|
"hypothesis #1 was rejected by some other stage than the P90 one the demo narrates"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_deviating_baseline_forkaster_the_demo_run_before_the_solver(tmp_path) -> None:
|
||||||
|
"""LOAD-BEARING (the 10 % test): when the project's declared cost lines deviate by 10 % from
|
||||||
|
the numbers the script asserts, the run is FORKASTET at stage 0 — with the reconciliation
|
||||||
|
reason, not the P90 one.
|
||||||
|
|
||||||
|
Goes RED the moment the demo stops being anchored: without the ``cost-baseline.json`` in the
|
||||||
|
bundle the run path passes ``baseline=None``, stage 0 is skipped, and this same deviating
|
||||||
|
number changes nothing at all (the run ends FORESLÅTT, as ``test_..._runs_the_whole_demo``
|
||||||
|
above shows). The script is byte-identical in both tests — only the declared baseline moves."""
|
||||||
|
baseline = _deviated(baseline_from_scripted_candidate(_only_candidate()), 1.10)
|
||||||
|
bundle = materialize_anchored_bundle(tmp_path / "forankret", baseline=baseline)
|
||||||
|
result = await simulate_learning_loop(str(bundle), str(tmp_path))
|
||||||
|
|
||||||
|
outcome = result.run_a.outcome
|
||||||
|
assert isinstance(outcome, Rejection), (
|
||||||
|
"a proposal 10 % away from the project's declared cost lines was NOT rejected — the "
|
||||||
|
"deterministic gate is not anchored to the baseline"
|
||||||
|
)
|
||||||
|
assert "outside the 5.0% tolerance" in outcome.reason
|
||||||
|
assert "ENERGI-TOTAL-EL" in outcome.reason
|
||||||
|
assert "P90" not in outcome.reason, (
|
||||||
|
"rejected by the solver stage, not by the reconciliation stage 0 that must run BEFORE it"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_reserve_itself_ships_no_baseline(tmp_path) -> None:
|
||||||
|
"""The materializer must ADD something the reserve genuinely lacks — and must leave the
|
||||||
|
commons-owned bundle alone (criterion 8: the goldens stay byte-unchanged)."""
|
||||||
|
reserve = _default_bundle_dir()
|
||||||
|
assert okf.load_optional_cost_baseline(str(reserve)) is None, (
|
||||||
|
"the shared reserve now ships a cost baseline — the copy-and-extend variant is obsolete "
|
||||||
|
"and this whole seam should be re-measured"
|
||||||
|
)
|
||||||
|
|
||||||
|
bundle = materialize_anchored_bundle(tmp_path / "forankret")
|
||||||
|
assert okf.load_optional_cost_baseline(str(bundle)) is not None, (
|
||||||
|
"the materialized bundle is not readable by okf's own loader — the filename has drifted"
|
||||||
|
)
|
||||||
|
assert okf.load_optional_cost_baseline(str(reserve)) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_baseline_is_derived_from_the_scripted_register() -> None:
|
||||||
|
"""The baseline is DERIVED from the script's own cost lines, never typed alongside them: a
|
||||||
|
hand-written copy is a second source of the same numbers, and two sources drift."""
|
||||||
|
candidate = _only_candidate()
|
||||||
|
derived = baseline_from_scripted_candidate(candidate)
|
||||||
|
|
||||||
|
for reply in (candidate.overclaimed, candidate.corrected):
|
||||||
|
for item in json.loads(reply)["affected_items"]:
|
||||||
|
line = derived.items[item["code"]]
|
||||||
|
assert (line.quantity, line.unit_cost) == (item["quantity"], item["unit_cost"])
|
||||||
|
assert derived.project_id == candidate.project_id
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_candidate_whose_two_replies_disagree_is_refused() -> None:
|
||||||
|
"""Validation, never repair. The two scripted replies must state the SAME cost lines: were they
|
||||||
|
to differ, hypothesis #1 would be rejected by stage 0 instead of by P90, and the demo's
|
||||||
|
REJECTED line would silently come from another mechanism than the one it narrates."""
|
||||||
|
candidate = _only_candidate()
|
||||||
|
skewed = ScriptedCandidate(
|
||||||
|
project_id=candidate.project_id,
|
||||||
|
overclaimed=candidate.overclaimed.replace("300000", "310000"),
|
||||||
|
corrected=candidate.corrected,
|
||||||
|
flip_key=candidate.flip_key,
|
||||||
|
)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
baseline_from_scripted_candidate(skewed)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_demo_entry_point_runs_the_anchored_reserve() -> None:
|
||||||
|
"""LOAD-BEARING on the CALL SITE: the thing the operator actually runs on stage must be the
|
||||||
|
anchored variant. Goes RED if ``main`` is pointed back at the plain reserve.
|
||||||
|
|
||||||
|
The declared baseline is printed because an anchoring nobody can see is an anchoring nobody can
|
||||||
|
check: every other line of the demo is byte-identical whether the gate is anchored or not.
|
||||||
|
|
||||||
|
**The first form of this test was vacuous, and the mutation caught it.** It asserted
|
||||||
|
``"kostbaseline erklært" in stdout`` — but the un-anchored branch read "ingen kostbaseline
|
||||||
|
erklært", which CONTAINS that substring; and ``"ENERGI-TOTAL-EL" in stdout`` holds either way,
|
||||||
|
because the Step-2 line prints the proposal's own cost lines. Both survived the mutation. The
|
||||||
|
assertions below name the whole declared line and rule the other branch out explicitly."""
|
||||||
|
proc = subprocess.run(
|
||||||
|
[sys.executable, "-m", "portfolio_optimiser.simulation"],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
assert proc.returncode == 0, proc.stderr
|
||||||
|
assert "kostbaseline erklært (ENERGI-TOTAL-EL 300000 x 1)" in proc.stdout, (
|
||||||
|
"the demo ran against a bundle with no cost baseline — the deterministic gate on stage is "
|
||||||
|
"reasoning only about the numbers the proposal supplied itself"
|
||||||
|
)
|
||||||
|
assert "validatorens stage 0 avstemmer" in proc.stdout
|
||||||
|
assert "uten kostbaseline" not in proc.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def _only_candidate() -> ScriptedCandidate:
|
||||||
|
from portfolio_optimiser.simulation import _CANDIDATES, _PROJECT_ID
|
||||||
|
|
||||||
|
(candidate,) = [c for c in _CANDIDATES if c.project_id == _PROJECT_ID]
|
||||||
|
return candidate
|
||||||
Loading…
Add table
Add a link
Reference in a new issue