feat(validator): anchor the deterministic gate to the project's real cost baseline (S4.0)

Every stage of validate_proposal reasoned only about numbers the proposal itself
supplied, so an internally-consistent hallucination cleared the whole gate (F3).
A new stage 0 reconciles each affected_item against the project's CostBaseline
before the CBC solve: an unknown cost code is rejected, and a real code carrying
a quantity/unit_cost outside the configured tolerance (5% default, relative to
the baseline value) is rejected. Validation, never repair.

The baseline argument is OPTIONAL (None = pre-S4.0 behaviour), but both run
paths set it: the road path projects project.cost_items, the bundle path loads
cost-baseline.json when the bundle ships one. Bundles written before the
amendment stay un-anchored, so the commons-owned goldens run byte-identically;
a baseline that exists but is malformed still raises on both loaders.

F8: the method-specific cap now comes from the METHOD_CAPS registry (measure
type -> fraction, injectable) instead of an energy_efficiency string comparison.

The baseline format and tolerance semantics were decided locally — the commons
amendment (D-A pt. 2) never arrived, exactly as in S3.2. D7 mirroring stays open.

Three portfolio fixtures quoted cost codes belonging to OTHER projects; the new
gate caught them. They now quote each project's own lines, and the two copied
REPLIES tables import the single source instead of drifting from it.

Load-bearing measured (tests/test_s40_cost_baseline_loadbearing.py), six
mutations all red: detach the reconciliation stage; detach the magnitude
tolerance; detach the road wiring; detach the bundle wiring; ignore the injected
cap registry; make the optional loader tolerant of malformed content. Control:
with the road wiring detached the repaired portfolio fixtures still pass, so
they are not masking the seam. 597 -> 612 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JdwK7bQ4BZkWH4t8MRDKb4
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 17:19:31 +02:00
commit 126807aee7
16 changed files with 645 additions and 69 deletions

View file

@ -124,6 +124,26 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
(`tests/test_step32_multicandidate_loadbearing.py` + `test_step8_promotion_loadbearing.py`), fem (`tests/test_step32_multicandidate_loadbearing.py` + `test_step8_promotion_loadbearing.py`), fem
mutasjoner alle røde: detach per-dom-nøklingen · detach feltene `promote_verdict` skriver · gjør en mutasjoner alle røde: detach per-dom-nøklingen · detach feltene `promote_verdict` skriver · gjør en
delvis/uparsebar nøkkel tolerant · normaliser magnituden ved skriving · fjern fallbacken (kontroll). delvis/uparsebar nøkkel tolerant · normaliser magnituden ved skriving · fjern fallbacken (kontroll).
- **Den deterministiske gaten er FORANKRET i prosjektets faktiske kostbaseline (S4.0, F3/F8):** før
S4.0 resonnerte HVER stage kun om tall forslaget selv oppga, så en internt konsistent hallusinasjon
klarerte hele gaten. `validate_proposal(..., baseline=...)` avstemmer nå hvert `affected_item` mot
`CostBaseline` (`ir.py`) i en **stage 0 — FØR løseren** (billigst, og den eneste som skiller en
oppdiktet linje fra en ekte; å bruke en CBC-solve på tall som ikke tilhører prosjektet er arbeid på
et krav som uansett ikke kan valideres). To uavhengige avvisninger: ukjent kostkode, og ekte kode
med `quantity`/`unit_cost` utenfor `tolerance` (default 5 %, **konfig**) relativt til BASELINE-verdien.
Validering, ALDRI reparasjon — forslaget avvises, aldri stilltiende korrigert til baselinen.
**Argumentet er VALGFRITT** (`None` = pre-S4.0-oppførsel), men begge run-stier SETTER det: road-stien
fra `project.cost_items` (alltid — estimatet ER prosjektet), bundle-stien KUN når bundelen shipper
`cost-baseline.json` (`load_optional_cost_baseline`) — en pre-amendment-bundle er legitimt
uforankret, og det er dét som holder commons-goldenene byte-identiske. **Toleransen stopper ved
fravær:** en baseline som FINNES men er malformed raiser på BEGGE loaderne (å lese korrupt som
«ingen baseline» ville gitt en uforankret gate i forkledning — samme resonnement som `read_spend`).
**F8:** metode-cap-en slås opp i `METHOD_CAPS`-registeret (måletype→brøk, injiserbart), ikke mot
`energy_efficiency`-literalen — en andre metode er nå data, ikke en redigering av validatoren.
Format- og toleranse-semantikken er bestemt LOKALT (commons-amendmentet D-A pkt. 2 kom aldri, som i
S3.2); **D7-speiling ÅPEN.** Load-bearing MÅLT (`tests/test_s40_cost_baseline_loadbearing.py`), seks
mutasjoner alle røde: detach avstemmings-stagen · detach magnitude-toleransen · detach road-wiringen ·
detach bundle-wiringen · ignorer det injiserte cap-registeret · gjør den valgfrie loaderen tolerant.
- **Globalt token-tak håndheves FØR kall, aldri bare etterpå (S3.4, F10):** `PortfolioBudget` + - **Globalt token-tak håndheves FØR kall, aldri bare etterpå (S3.4, F10):** `PortfolioBudget` +
`PortfolioMeter` er ÉN ledger over hele porteføljepasset (og — seedet av `read_spend` — på tvers `PortfolioMeter` er ÉN ledger over hele porteføljepasset (og — seedet av `read_spend` — på tvers
av pass), mens per-run `Budget`/`TokenMeter` er uendret. Taket har tre tenner, med hver sin jobb: av pass), mens per-run `Budget`/`TokenMeter` er uendret. Taket har tre tenner, med hver sin jobb:

View file

@ -71,7 +71,9 @@ One run, one project, eight steps — with the learning loop closing across runs
2. **Hypothesise** — one typed candidate measure (strict IR, fail-fast schema). 2. **Hypothesise** — one typed candidate measure (strict IR, fail-fast schema).
3. **Debate** — a maker-checker pair argues the reasoning (round-capped). 3. **Debate** — a maker-checker pair argues the reasoning (round-capped).
4. **Validate** — two falsifiers on the same candidate: the **deterministic validator** gates 4. **Validate** — two falsifiers on the same candidate: the **deterministic validator** gates
the numbers (blocking, never optional) and the **checker** gates the reasoning. the numbers (blocking, never optional) and the **checker** gates the reasoning. The validator
is anchored to the project's declared cost baseline, so a proposal cannot invent the cost
lines it claims to save against.
5. **Refine** — a rejected attempt retries *informed* by the rejection reason, under hard 5. **Refine** — a rejected attempt retries *informed* by the rejection reason, under hard
attempt and token caps. Unbounded loops are forbidden everywhere. attempt and token caps. Unbounded loops are forbidden everywhere.
6. **Propose or discard** — a validated proposal with risk percentiles, or a typed rejection. 6. **Propose or discard** — a validated proposal with risk percentiles, or a typed rejection.
@ -149,6 +151,18 @@ when the seam is detached, so the loop cannot silently degrade into theater.
the merge would produce a key belonging to neither. `promote_verdict` writes all three, so a the merge would produce a key belonging to neither. `promote_verdict` writes all three, so a
promoted verdict about one candidate never surfaces for another. promoted verdict about one candidate never surfaces for another.
A bundle may also ship a **`cost-baseline.json`** — the project's actual cost lines,
`{code: {quantity, unit_cost}}` — and when it does, the deterministic validator reconciles every
affected item of a proposal against it before anything else runs. A cost code the project does not
have is rejected, and so is a real code carrying a quantity or unit cost outside the configured
tolerance (5% by default, relative to the baseline value). Without it, every stage of the gate
reasons only about numbers the proposal supplied itself, so an internally consistent hallucination
passes. The reconciliation validates; it never repairs a proposal into the baseline. A bundle that
ships no baseline is simply un-anchored and runs exactly as before, while a baseline that is
present but malformed is an error rather than a silent fall-back to un-anchored. On the
reference-domain (non-bundle) path the project's own cost items are the baseline, so those runs
are always anchored.
The **prior-verdict fold — the learning step — happens only on the `--bundle-dir` path**; a The **prior-verdict fold — the learning step — happens only on the `--bundle-dir` path**; a
plain `--docs-dir`-only run is single-shot (no fold). `--decision`/`--rationale` apply to the plain `--docs-dir`-only run is single-shot (no fold). `--decision`/`--rationale` apply to the
single-project path only and are inert in portfolio mode. **`--outbox-dir` must differ from single-project path only and are inert in portfolio mode. **`--outbox-dir` must differ from

View file

@ -0,0 +1,20 @@
---
type: project
title: "Kontorbygg B — energibaseline med erklærte kostlinjer (repo-lokal fixture)"
description: "Syntetisk energibaseline for S4.0-fixturen. Ett kontorbygg, én kandidat-kostpost (ENERGI-TOTAL-EL), og en eksplisitt cost-baseline.json den deterministiske validatoren avstemmer mot."
timestamp: 2026-08-03
---
# Kontorbygg B (repo-lokal test-fixture)
Et lite syntetisk kontorbygg med én dokumentert energibaseline. Til forskjell fra mikro-A erklærer
denne bundelen kostlinjene sine maskinlesbart i `cost-baseline.json`, slik at et forslag som siterer
en oppdiktet kostkode — eller en ekte kode med oppdiktet størrelse — blir avvist av validatorens
avstemmings-stage før løseren i det hele tatt kjøres.
## Energibaseline
- Årlig elektrisk energikostnad (`ENERGI-TOTAL-EL`): 180 000 NOK/år (syntetisk), ført som
`quantity: 180000` × `unit_cost: 1.0` i `cost-baseline.json`.
- Kandidat-tiltak: LED-retrofit av kontorbelysning.
- Modellert besparelse: ~18 000 NOK/år (innenfor validatorens feasibelt-område).

View file

@ -0,0 +1,10 @@
{
"_note": "SYNTHETIC repo-lokal kostbaseline (S4.0-fixture) — ikke ekte data. Prosjektets FAKTISKE kostlinjer: den deterministiske validatoren avstemmer et forslags affected_items mot disse (okf.load_cost_baseline -> validator._reconcile_against_baseline). Formatet er bestemt LOKALT (commons-amendmentet D-A pkt. 2 kom aldri); D7-speiling er ÅPEN.",
"project_id": "BYGG-ENERGI-BASELINE-MIKRO",
"items": {
"ENERGI-TOTAL-EL": {
"quantity": 180000,
"unit_cost": 1.0
}
}
}

View file

@ -0,0 +1,26 @@
---
type: index
okf_version: 0.1
title: "Bygg-energi baseline-mikro — repo-lokal fixture (S4.0 kostbaseline-forankring)"
description: "Minimal repo-lokal OKF-bundle som SHIPPER en cost-baseline.json, så bundle-stiens validator-forankring (S4.0, F3) kan testes ende-til-ende. Speiler bygg-energi-mikro-A, men med kostbaselinen lagt til."
tags: [fixture, S4.0, kostbaseline]
timestamp: 2026-08-03
---
# Bygg-energi baseline-mikro (repo-lokal test-fixture)
En **repo-lokal mini OKF-bundle** under pakkens `data/` (ALDRI `shared/` — subtree er PULL-ONLY).
Eneste formål: være den ene bundelen som **erklærer sin kostbaseline**, slik at
`run_project(bundle_dir=...)` faktisk forankrer den deterministiske gaten mot prosjektets virkelige
kostlinjer. De eldre bundlene bærer bevisst INGEN `cost-baseline.json` — de er kontrollen som viser
at forankringen er opt-in per bundle (pre-amendment-bundler kjører uendret).
## Innhold (progressiv disclosure)
- [bygg-kontor-baseline.md](bygg-kontor-baseline.md) — `type: project` — bygget og energibaselinen
som lese-kontekst (og `bundle_citations`-kilde).
`validator-input.json` er IR-projeksjonen (kandidatens kost-IR), og `cost-baseline.json` er
prosjektets FAKTISKE kostlinjer (`code -> {quantity, unit_cost}`) som `okf.load_cost_baseline` leser
og validatoren avstemmer `affected_items` mot. De to er bevisst adskilt: IR-projeksjonen er hva
noen *foreslår*, baselinen er hva prosjektet *er*.

View file

@ -0,0 +1,13 @@
{
"_note": "SYNTHETIC repo-lokal IR-projeksjon (S4.0-fixture) — ikke ekte data. Kandidatens kost-IR; project_id matcher det bundle-backede prosjektets id (run._project_from_bundle fail-faster ved mismatch). Kostlinjene her SPEILER cost-baseline.json med vilje: et forslag om denne kandidaten skal avstemme rent.",
"project_id": "BYGG-ENERGI-BASELINE-MIKRO",
"measure": "LED-retrofit av kontorbelysning",
"affected_items": [
{
"code": "ENERGI-TOTAL-EL",
"quantity": 180000,
"unit_cost": 1.0
}
],
"claimed_saving_nok": 18000
}

View file

@ -28,7 +28,7 @@ from agent_framework import BaseChatClient, Message
from pydantic import ValidationError from pydantic import ValidationError
from portfolio_optimiser.budget import TokenMeter from portfolio_optimiser.budget import TokenMeter
from portfolio_optimiser.ir import SavingsProposal from portfolio_optimiser.ir import CostBaseline, SavingsProposal
from portfolio_optimiser.reference_domain import Project from portfolio_optimiser.reference_domain import Project
from portfolio_optimiser.validator import ( from portfolio_optimiser.validator import (
Rejection, Rejection,
@ -110,6 +110,7 @@ async def generate_via_llm(
meter: TokenMeter, meter: TokenMeter,
*, *,
max_attempts: int = 3, max_attempts: int = 3,
baseline: CostBaseline | None = None,
) -> ValidatedProposal | Rejection: ) -> ValidatedProposal | Rejection:
"""Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds, """Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds,
the meter checked in this loop: the meter checked in this loop:
@ -123,7 +124,12 @@ async def generate_via_llm(
The only per-attempt falsifier here is the deterministic validator (the numbers). The The only per-attempt falsifier here is the deterministic validator (the numbers). The
checker is a run-level, one-shot signal (run.py, before generation); seeding generation checker is a run-level, one-shot signal (run.py, before generation); seeding generation
with the checker critique is separately scoped and NOT done here. Returns with the checker critique is separately scoped and NOT done here.
``baseline`` (S4.0) is handed straight to ``validate_proposal``, so a fabricated cost line is
falsified per ATTEMPT like any other rejection and its reason feeds the next attempt's prompt
through the SAME informed-refinement path (Step 5), which is why no new loop appears here.
Returns
``ValidatedProposal | Rejection``; never a malformed proposal; raises ``BudgetExceeded`` ``ValidatedProposal | Rejection``; never a malformed proposal; raises ``BudgetExceeded``
when the meter cap is crossed.""" when the meter cap is crossed."""
@ -146,7 +152,7 @@ async def generate_via_llm(
# accumulated history (bounded prompt growth). # accumulated history (bounded prompt growth).
messages = _build_messages(project, context, prior_rejection=last) messages = _build_messages(project, context, prior_rejection=last)
candidate = await _fetch_parsed(messages) candidate = await _fetch_parsed(messages)
result = validate_proposal(candidate) result = validate_proposal(candidate, baseline=baseline)
if isinstance(result, ValidatedProposal): if isinstance(result, ValidatedProposal):
return result return result
last = result last = result

View file

@ -24,6 +24,31 @@ class AffectedItem(BaseModel):
return self.quantity * self.unit_cost return self.quantity * self.unit_cost
class CostBaselineLine(BaseModel):
"""One line of a project's ACTUAL cost baseline: the quantity and unit cost a proposal's
``AffectedItem`` for that code must reconcile against (S4.0, F3)."""
quantity: float = Field(ge=0)
unit_cost: float = Field(gt=0)
class CostBaseline(BaseModel):
"""A project's cost baseline, keyed by cost code — the ground truth the deterministic
validator anchors ``affected_items`` to, so the gate cannot be fed hallucinated cost lines.
Deliberately a typed IR contract (not a loader-private shape): both sources project INTO
it an OKF bundle's ``cost-baseline.json`` (``okf.load_cost_baseline``) and the road
reference domain's ``cost_items`` (``validator.baseline_from_project``) — so the validator
sees ONE representation regardless of path, and the Claude-SDK sibling can mirror it (D7).
The projection/tolerance semantics were decided HERE: the commons amendment specifying
``cost-baseline.json`` never arrived, exactly as in S3.2. D7 mirroring stays OPEN.
"""
project_id: str
items: dict[str, CostBaselineLine]
class SavingsProposal(BaseModel): class SavingsProposal(BaseModel):
"""Typed IR for a candidate cost-saving measure (B1).""" """Typed IR for a candidate cost-saving measure (B1)."""

View file

@ -3,7 +3,8 @@
Reads a bundle the way OKF intends (progressive disclosure): start at ``index.md``, follow Reads a bundle the way OKF intends (progressive disclosure): start at ``index.md``, follow
intra-bundle cross-links **recursively, depth-first in first-seen link order**, parse each file's intra-bundle cross-links **recursively, depth-first in first-seen link order**, parse each file's
YAML frontmatter, classify by the one required ``type`` field. **NO** ``agent_framework``, **NO** YAML frontmatter, classify by the one required ``type`` field. **NO** ``agent_framework``, **NO**
``mcp`` pure stdlib, so the SAME navigation serves both the MAF and the Claude-SDK ``mcp`` stdlib + ``pydantic`` only (as ``dimension.py``; the typed contracts this module loads
live in ``ir.py``), so the SAME navigation serves both the MAF and the Claude-SDK
implementations unchanged (målbilde §4 vendor-neutrality). implementations unchanged (målbilde §4 vendor-neutrality).
Link resolution follows ``shared/method-spec.md`` §3 Step 1: a leading ``/`` denotes the **bundle Link resolution follows ``shared/method-spec.md`` §3 Step 1: a leading ``/`` denotes the **bundle
@ -29,10 +30,12 @@ from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from portfolio_optimiser.ir import CostBaseline
from portfolio_optimiser.retrieval import PathSecurityError, safe_resolve from portfolio_optimiser.retrieval import PathSecurityError, safe_resolve
_INDEX_NAME = "index.md" _INDEX_NAME = "index.md"
_IR_PROJECTION = "validator-input.json" _IR_PROJECTION = "validator-input.json"
_COST_BASELINE = "cost-baseline.json"
# Intra-bundle markdown cross-links: ``](target.md)``. A path separator is NOT a rejection reason — # Intra-bundle markdown cross-links: ``](target.md)``. A path separator is NOT a rejection reason —
# ``_resolve_target`` decides in-/out-of-bundle, and only escape is refused (method-spec §3 Step 1). # ``_resolve_target`` decides in-/out-of-bundle, and only escape is refused (method-spec §3 Step 1).
_LINK_RE = re.compile(r"\]\(([^)]+\.md)\)") _LINK_RE = re.compile(r"\]\(([^)]+\.md)\)")
@ -285,6 +288,39 @@ def link_in_index(bundle_dir: str, target_name: str, label: str) -> bool:
return True return True
def load_cost_baseline(bundle_dir: str, name: str = _COST_BASELINE) -> CostBaseline:
"""Load the bundle's cost baseline (``cost-baseline.json`` by default): the project's ACTUAL
cost lines (``{code: {quantity, unit_cost}}``), which the deterministic validator reconciles a
proposal's ``affected_items`` against (S4.0, F3).
Fail-fast, mirroring ``load_ir_projection`` and ``dimension.load_dimension``: a missing file
raises ``FileNotFoundError`` and malformed content raises ``pydantic.ValidationError``. A cost
baseline is authoritative gate input a tolerantly-degraded one would silently un-anchor the
gate, which is precisely the failure this stage exists to prevent. (The tolerant skip rule
belongs to the RAW verdict-inbox layer, never here.)
Use ``load_optional_cost_baseline`` where the ABSENCE of the file is legitimate."""
resolved = Path(safe_resolve(bundle_dir, name))
if not resolved.is_file():
raise FileNotFoundError(f"cost baseline not found in bundle: {name!r}")
return CostBaseline.model_validate_json(resolved.read_text(encoding="utf-8"))
def load_optional_cost_baseline(bundle_dir: str, name: str = _COST_BASELINE) -> CostBaseline | None:
"""``load_cost_baseline`` where a MISSING file is legitimate: returns ``None`` instead of
raising. This is the run path's loader — a bundle authored before the baseline amendment is
simply un-anchored (``None`` = pre-S4.0 behaviour), not an error, which is what keeps every
existing bundle (including the commons-owned goldens) running byte-identically.
The tolerance stops at absence: a baseline that EXISTS but is malformed still raises. Reading a
corrupt baseline as "no baseline" would hand back an un-anchored gate under the appearance of an
anchored one (the same reasoning as ``budget.read_spend``)."""
try:
return load_cost_baseline(bundle_dir, name)
except FileNotFoundError:
return None
def load_ir_projection(bundle_dir: str, name: str = _IR_PROJECTION) -> dict[str, Any]: def load_ir_projection(bundle_dir: str, name: str = _IR_PROJECTION) -> dict[str, Any]:
"""Load the bundle's IR projection (``validator-input.json`` by default): the candidate """Load the bundle's IR projection (``validator-input.json`` by default): the candidate
measure's cost-IR (``measure``, ``affected_items``, ``claimed_saving_nok``) — the measure's cost-IR (``measure``, ``affected_items``, ``claimed_saving_nok``) — the

View file

@ -55,7 +55,7 @@ from portfolio_optimiser.generate import generate_via_llm
from portfolio_optimiser.ir import SavingsProposal from portfolio_optimiser.ir import SavingsProposal
from portfolio_optimiser.provenance import ProvenanceStamp from portfolio_optimiser.provenance import ProvenanceStamp
from portfolio_optimiser.reference_domain import Project, load_reference_projects from portfolio_optimiser.reference_domain import Project, load_reference_projects
from portfolio_optimiser.validator import Rejection, ValidatedProposal from portfolio_optimiser.validator import Rejection, ValidatedProposal, baseline_from_project
from portfolio_optimiser import okf, outbox from portfolio_optimiser import okf, outbox
from portfolio_optimiser.semretrieval import ( from portfolio_optimiser.semretrieval import (
SEMANTIC_WEIGHT_DEFAULT, SEMANTIC_WEIGHT_DEFAULT,
@ -373,9 +373,16 @@ async def run_project(
# NOT keyword chunk-stuffing; the road path keeps the chunk-retrieval data source. ``debate_tools`` # NOT keyword chunk-stuffing; the road path keeps the chunk-retrieval data source. ``debate_tools``
# is the query-time retrieval surface — empty on the bundle path (navigation already placed the # is the query-time retrieval surface — empty on the bundle path (navigation already placed the
# curated context in the prompt, and a docs_dir==bundle_dir tool would re-leak the verdict layer). # curated context in the prompt, and a docs_dir==bundle_dir tool would re-leak the verdict layer).
# S4.0 (F3): the run path SETS the validator's cost baseline, so the deterministic gate is
# anchored to the project's real cost lines instead of the ones the proposal asserts.
# * road path: the reference project's own ``cost_items`` ARE the baseline -> always anchored.
# * bundle path: anchored only when the bundle SHIPS a ``cost-baseline.json``. A bundle written
# before the amendment (every commons-owned golden) is legitimately un-anchored -> None =
# pre-S4.0 behaviour. A baseline that exists but is malformed still raises (fail-closed).
if bundle_dir is not None: if bundle_dir is not None:
bundle = okf.navigate_bundle(bundle_dir) bundle = okf.navigate_bundle(bundle_dir)
project = _project_from_bundle(bundle_dir, project_id, bundle=bundle) project = _project_from_bundle(bundle_dir, project_id, bundle=bundle)
baseline = okf.load_optional_cost_baseline(bundle_dir)
# §4.1a context-scope: agents read ONLY dimension-scoped bundle knowledge (Step-3 filter); # §4.1a context-scope: agents read ONLY dimension-scoped bundle knowledge (Step-3 filter);
# dimension=None keeps the full context, byte-identical to before. # dimension=None keeps the full context, byte-identical to before.
context = okf.bundle_context(bundle, dimension=dimension.id if dimension else None) context = okf.bundle_context(bundle, dimension=dimension.id if dimension else None)
@ -383,6 +390,7 @@ async def run_project(
debate_tools: list[Any] = [] debate_tools: list[Any] = []
else: else:
project = _project_by_id(project_id) project = _project_by_id(project_id)
baseline = baseline_from_project(project)
chunks = retrieve_chunks("cost saving measure", docs_dir, top_k) chunks = retrieve_chunks("cost saving measure", docs_dir, top_k)
citations = [chunk_dict_to_citation(c) for c in chunks] citations = [chunk_dict_to_citation(c) for c in chunks]
context = "\n".join(c["snippet"] for c in chunks) context = "\n".join(c["snippet"] for c in chunks)
@ -480,7 +488,9 @@ async def run_project(
# 5. Structured candidate -> blocking validation on the NUMBERS; token bound = the meter. # 5. Structured candidate -> blocking validation on the NUMBERS; token bound = the meter.
proposer_client = factory("proposer") proposer_client = factory("proposer")
validator_outcome = await generate_via_llm(proposer_client, project, gen_context, meter) validator_outcome = await generate_via_llm(
proposer_client, project, gen_context, meter, baseline=baseline
)
proposal = validator_outcome.proposal proposal = validator_outcome.proposal
# 6. First-class provenance stamp (authoritative; independent of MAF Annotation). # 6. First-class provenance stamp (authoritative; independent of MAF Annotation).

View file

@ -24,13 +24,13 @@ from __future__ import annotations
import random import random
import statistics import statistics
import warnings import warnings
from collections.abc import Callable from collections.abc import Callable, Mapping
from contextlib import contextmanager from contextlib import contextmanager
from dataclasses import dataclass from dataclasses import dataclass
import pulp import pulp
from portfolio_optimiser.ir import AffectedItem, SavingsProposal from portfolio_optimiser.ir import AffectedItem, CostBaseline, CostBaselineLine, SavingsProposal
from portfolio_optimiser.reference_domain import Project from portfolio_optimiser.reference_domain import Project
MAX_SAVING_FRACTION = 0.30 MAX_SAVING_FRACTION = 0.30
@ -46,6 +46,21 @@ INDEPENDENT gate: it can reject a proposal the generic P90 stage passes (not red
fraction is calibrated against the reference domain; the CONDITION (a method-scoped stricter cap) is fraction is calibrated against the reference domain; the CONDITION (a method-scoped stricter cap) is
the encoded rule. Returns the same ``Rejection`` type a validator stage, not a new gate.""" the encoded rule. Returns the same ``Rejection`` type a validator stage, not a new gate."""
METHOD_CAPS: dict[str, float] = {_ENERGY_METHOD_MEASURE: _ENERGY_METHOD_MAX_FRACTION}
"""S4.0 (F8): the method-cap REGISTRY — measure type -> method-scoped max saving fraction. The
rule used to be an ``if proposal.measure == "energy_efficiency"`` branch, so encoding a second
assessment method meant editing the validator. It is now data: a caller passes its own registry
(``validate_proposal(..., method_caps=...)``), keyed by the measure type a dimension admits
(``dimension.allowed_measure_types``), and the built-in entry stays the default so the Step-9
behaviour is unchanged. Deliberately NOT a config file yet the deliverable is the key-by-config
seam (90%-prinsippet), not a settings format."""
BASELINE_TOLERANCE_DEFAULT = 0.05
"""S4.0: the relative deviation a reconciled ``AffectedItem`` may show against its cost-baseline
line (5%). A tolerance is needed at all because a proposer restates magnitudes in prose-derived,
rounded form; it is small because its whole purpose is to leave no room for a FABRICATED magnitude.
Config, not policy: every caller can tighten or loosen it per run (``tolerance=``)."""
_MC_SAMPLES = 512 _MC_SAMPLES = 512
_MC_SEED = 20260624 _MC_SEED = 20260624
@ -122,9 +137,83 @@ def _monte_carlo(
return deciles[0], deciles[4], deciles[8] # P10, P50, P90 return deciles[0], deciles[4], deciles[8] # P10, P50, P90
def validate_proposal(proposal: SavingsProposal) -> ValidatedProposal | Rejection: def baseline_from_project(project: Project) -> CostBaseline:
"""Project a road reference-domain ``Project``'s ``cost_items`` into the ``CostBaseline``
contract the road-path counterpart of ``okf.load_cost_baseline`` (S4.0). The road path always
HAS its baseline (the estimate is the project), so this projection is total: no optional
variant, and a run on this path is always anchored."""
return CostBaseline(
project_id=project.id,
items={
ci.code: CostBaselineLine(quantity=ci.quantity, unit_cost=ci.unit_cost)
for ci in project.cost_items
},
)
def _reconcile_against_baseline(
proposal: SavingsProposal, baseline: CostBaseline, tolerance: float
) -> Rejection | None:
"""S4.0 (F3): every affected item must correspond to a REAL line of the project's cost baseline.
Two independent failures, both fail-closed:
* the cost code is absent from the baseline a fabricated line;
* the code is real but its ``quantity``/``unit_cost`` deviates from the baseline line by more
than ``tolerance`` (relative to the BASELINE value, which is the ground truth) a real code
carrying a fabricated magnitude.
Returns the first ``Rejection`` (validator's own type — never a new gate), or ``None`` when the
proposal reconciles. Items are checked in their stated order so the reason is deterministic.
A validation, never a repair: the proposal is rejected, not silently corrected to the baseline."""
for item in proposal.affected_items:
line = baseline.items.get(item.code)
if line is None:
return Rejection(
proposal=proposal,
reason=(
f"unknown cost code {item.code!r}: not in project {baseline.project_id}'s "
f"cost baseline ({len(baseline.items)} known codes)"
),
)
for field, claimed, actual in (
("quantity", item.quantity, line.quantity),
("unit_cost", item.unit_cost, line.unit_cost),
):
if abs(claimed - actual) > tolerance * actual:
return Rejection(
proposal=proposal,
reason=(
f"{field} {claimed:g} for cost code {item.code!r} is outside the "
f"{tolerance:.1%} tolerance around the baseline {field} {actual:g}"
),
)
return None
def validate_proposal(
proposal: SavingsProposal,
*,
baseline: CostBaseline | None = None,
tolerance: float = BASELINE_TOLERANCE_DEFAULT,
method_caps: Mapping[str, float] | None = None,
) -> ValidatedProposal | Rejection:
"""Deterministic blocking validation. Returns a ``ValidatedProposal`` only when the """Deterministic blocking validation. Returns a ``ValidatedProposal`` only when the
claim is feasible; otherwise a ``Rejection`` that cannot be consumed as validated.""" claim is feasible; otherwise a ``Rejection`` that cannot be consumed as validated.
``baseline`` (S4.0, F3) anchors the gate to the project's ACTUAL cost lines: without it every
stage reasons only about numbers the proposal supplied itself, so an internally-consistent
hallucination clears the gate. It is OPTIONAL ``None`` is exactly the pre-S4.0 behaviour, so a
caller with no baseline (a bundle authored before the amendment) is unchanged but both run
paths SET it. ``tolerance`` is the reconciliation's config knob; ``method_caps`` overrides the
built-in method-cap registry (F8)."""
# Stage 0 (S4.0): reconcile against the cost baseline BEFORE the solver. It is the cheapest
# stage and the only one that can tell a fabricated line from a real one — spending a CBC solve
# on numbers that do not belong to the project is work on a claim that cannot be validated.
if baseline is not None:
blocked = _reconcile_against_baseline(proposal, baseline, tolerance)
if blocked is not None:
return blocked
# Stage 1 (Pydantic) already ran at construction. Stage 2: real CBC solve. # Stage 1 (Pydantic) already ran at construction. Stage 2: real CBC solve.
nominal = _solve_max_feasible(proposal.affected_items, MAX_SAVING_FRACTION) nominal = _solve_max_feasible(proposal.affected_items, MAX_SAVING_FRACTION)
# Stage 3: Monte Carlo percentiles of the feasible saving. # Stage 3: Monte Carlo percentiles of the feasible saving.
@ -152,15 +241,18 @@ def validate_proposal(proposal: SavingsProposal) -> ValidatedProposal | Rejectio
# Stage 5 (Step 9, SC7-B): a method-specific rule STRICTER than the generic cap. A proposal in # Stage 5 (Step 9, SC7-B): a method-specific rule STRICTER than the generic cap. A proposal in
# the energy method (IPMVP Option A) must clear a lower, method-scoped feasible — an INDEPENDENT # the energy method (IPMVP Option A) must clear a lower, method-scoped feasible — an INDEPENDENT
# gate that can reject a proposal the P90 stage passed. Same ``Rejection`` type, not a new gate. # gate that can reject a proposal the P90 stage passed. Same ``Rejection`` type, not a new gate.
if proposal.measure == _ENERGY_METHOD_MEASURE: # F8 (S4.0): the cap is looked up in a REGISTRY keyed by measure type (config), not compared
method_feasible = _ENERGY_METHOD_MAX_FRACTION * sum( # against the ``energy_efficiency`` literal — a second assessment method is now data, not an
it.total for it in proposal.affected_items # edit to this function. The built-in registry keeps the Step-9 behaviour identical.
) caps = METHOD_CAPS if method_caps is None else method_caps
method_fraction = caps.get(proposal.measure)
if method_fraction is not None:
method_feasible = method_fraction * sum(it.total for it in proposal.affected_items)
if proposal.claimed_saving_nok > method_feasible: if proposal.claimed_saving_nok > method_feasible:
return Rejection( return Rejection(
proposal=proposal, proposal=proposal,
reason=( reason=(
f"claimed {proposal.claimed_saving_nok:.0f} exceeds the {_ENERGY_METHOD_MEASURE} " f"claimed {proposal.claimed_saving_nok:.0f} exceeds the {proposal.measure} "
f"method cap {method_feasible:.0f} (stricter than the generic P90)" f"method cap {method_feasible:.0f} (stricter than the generic P90)"
), ),
) )

View file

@ -8,11 +8,13 @@ the gated live arm (Step 14).
from __future__ import annotations from __future__ import annotations
import json
from collections.abc import Callable, Sequence from collections.abc import Callable, Sequence
import pytest import pytest
from agent_framework import BaseChatClient from agent_framework import BaseChatClient
from portfolio_optimiser.reference_domain import load_reference_projects
from portfolio_optimiser.simulation import ScriptedChatClient from portfolio_optimiser.simulation import ScriptedChatClient
from portfolio_optimiser.verdicts import VerdictStore, seed_store from portfolio_optimiser.verdicts import VerdictStore, seed_store
@ -57,29 +59,63 @@ def make_client_factory() -> Callable[..., Callable[[str], BaseChatClient]]:
return _make return _make
# A generic VALID SavingsProposal reply for any project not present in a portfolio reply map: _DEFAULT_CLAIM = 20_000
# affected total = 1 x 100_000 = 100_000, P90 = 0.30 x 100_000 = 30_000, claimed 20_000 <= both # Last-resort reply for a prompt naming NO known reference project (the anchored per-project
# (Pydantic affected-total invariant and the validator P90 gate) -> always validates. # fallback below cannot be built then). Kept for that case only.
_PORTFOLIO_DEFAULT_REPLY = ( _PORTFOLIO_DEFAULT_REPLY = (
'{"measure":"Reduce scope","affected_items":' '{"measure":"Reduce scope","affected_items":'
'[{"code":"01.1","quantity":1,"unit_cost":100000}],"claimed_saving_nok":20000}' '[{"code":"01.1","quantity":1,"unit_cost":100000}],"claimed_saving_nok":20000}'
) )
def _anchored_default_replies() -> dict[str, str]:
"""A VALID default proposal PER reference project, quoting that project's OWN first cost line
verbatim (S4.0): since the road path anchors the validator to ``project.cost_items``, a generic
reply carrying an invented magnitude for code ``01.1`` is now correctly rejected as a
fabricated cost line. Anchoring the fixture is the fix; weakening the gate is not.
``claimed_saving_nok`` stays ``20_000`` for every project, exactly as the single generic reply
claimed before, so every ledger/goal/budget assertion built on that figure is unchanged. Each
project's first line is ``01.1 Rigg og drift`` at >= 480 000 NOK, so P90 (>= 144 000) clears the
claim on every project."""
replies: dict[str, str] = {}
for project in load_reference_projects():
line = project.cost_items[0]
replies[project.id] = json.dumps(
{
"measure": "Reduce scope",
"affected_items": [
{"code": line.code, "quantity": line.quantity, "unit_cost": line.unit_cost}
],
"claimed_saving_nok": _DEFAULT_CLAIM,
}
)
return replies
class _ProjectAwareUsageChatClient(ScriptedChatClient): class _ProjectAwareUsageChatClient(ScriptedChatClient):
"""Selects its reply by scanning the incoming prompt for a known ``project_id`` substring (the """Selects its reply by scanning the incoming prompt for a known ``project_id`` substring (the
prompt embeds ``project.id`` at run.py:162 and generate.py:48), falling back to a default valid prompt embeds ``project.id`` at run.py:162 and generate.py:48), falling back to a default valid
proposal so ``run_portfolio``'s single ``client_factory`` stays production-shaped while tests proposal so ``run_portfolio``'s single ``client_factory`` stays production-shaped while tests
vary the proposal per project. A THIN subclass: the prompt-scan lives in its selector, the shared vary the proposal per project. A THIN subclass: the prompt-scan lives in its selector, the shared
``_inner_get_response`` body in the canonical.""" ``_inner_get_response`` body in the canonical.
The fallback is itself project-aware (S4.0): a prompt naming a reference project gets that
project's baseline-anchored default reply, so an un-mapped project still produces a proposal the
anchored validator admits. Only a prompt naming NO known project falls through to
``default_reply``."""
def __init__( def __init__(
self, replies: dict[str, str], *, default_reply: str, tokens_per_reply: int = 8 self, replies: dict[str, str], *, default_reply: str, tokens_per_reply: int = 8
) -> None: ) -> None:
table = dict(replies) table = dict(replies)
anchored = _anchored_default_replies()
def _select(blob: str, _role: str) -> str: def _select(blob: str, _role: str) -> str:
return next((r for pid, r in table.items() if pid in blob), default_reply) explicit = next((r for pid, r in table.items() if pid in blob), None)
if explicit is not None:
return explicit
return next((r for pid, r in anchored.items() if pid in blob), default_reply)
super().__init__( super().__init__(
reply_selector=_select, default_reply=default_reply, tokens_per_reply=tokens_per_reply reply_selector=_select, default_reply=default_reply, tokens_per_reply=tokens_per_reply

View file

@ -28,29 +28,34 @@ _OFFLINE_MODEL_MAP = {
# The synthetic reply IS the proposal: generate._parse_ir builds affected_items (each with its # The synthetic reply IS the proposal: generate._parse_ir builds affected_items (each with its
# own quantity/unit_cost) straight from this JSON, and the validator's P90 = 0.30 x Σ(qty·unit_cost) # own quantity/unit_cost) straight from this JSON, and the validator's P90 = 0.30 x Σ(qty·unit_cost)
# ONLY when ``assumptions`` is empty (degenerate Monte Carlo, validator.py:108-113). All three # ONLY when ``assumptions`` is empty (degenerate Monte Carlo, validator.py). All three replies
# replies therefore OMIT ``assumptions`` and carry explicit magnitudes so each # therefore OMIT ``assumptions`` and carry explicit magnitudes so each ``claimed_saving_nok`` <= P90.
# ``claimed_saving_nok`` <= P90. Verified against validator + ir: #
# FV42-GSV-E1 Σ=1,482,500 P90=444,750 claimed 200,000 -> validates # S4.0: every line below quotes a cost line the project ACTUALLY has, verbatim from
# RV13-RAS-TP Σ= 756,000 P90=226,800 claimed 130,000 -> validates (decoy) # reference_projects.json — the road path now anchors the validator to ``project.cost_items``, so a
# BRU-LAKS-REHAB Σ=2,580,500 P90=774,150 claimed 210,000 -> validates # reply quoting another project's code (which these fixtures used to do) is rejected as a fabricated
# cost line. Verified against reference_projects.json + validator + ir:
# FV42-GSV-E1 01.1 1x850,000 + 05.2 4300x215 Σ=1,774,500 P90=532,350 claimed 200,000
# RV13-RAS-TP 22.4 610x3,850 Σ=2,348,500 P90=704,550 claimed 130,000 (decoy)
# BRU-LAKS-REHAB 01.1 1x620,000 + 87.3 640x980 Σ=1,247,200 P90=374,160 claimed 210,000
# ``measure`` is byte-identical "Reduce scope" for FV42+BRU (measure-match is exact string # ``measure`` is byte-identical "Reduce scope" for FV42+BRU (measure-match is exact string
# equality, verdicts.py:68) and "Material substitution" for the decoy, so the BRU<->FV42 pair # equality, verdicts.py:68) and "Material substitution" for the decoy, so the BRU<->FV42 pair still
# overlaps (shared code 05.2 + measure + magnitude bucket) while the decoy does not. # overlaps (shared code 01.1 + measure + magnitude bucket) while the decoy does not. 01.1 replaces
# 05.2 as the shared code because it is the only code both projects genuinely carry.
REPLIES = { REPLIES = {
"FV42-GSV-E1": ( "FV42-GSV-E1": (
'{"measure":"Reduce scope","affected_items":[' '{"measure":"Reduce scope","affected_items":['
'{"code":"05.2","quantity":4300,"unit_cost":215},' '{"code":"01.1","quantity":1,"unit_cost":850000},'
'{"code":"03.1","quantity":1800,"unit_cost":310}],"claimed_saving_nok":200000}' '{"code":"05.2","quantity":4300,"unit_cost":215}],"claimed_saving_nok":200000}'
), ),
"RV13-RAS-TP": ( "RV13-RAS-TP": (
'{"measure":"Material substitution","affected_items":[' '{"measure":"Material substitution","affected_items":['
'{"code":"88.2","quantity":180,"unit_cost":4200}],"claimed_saving_nok":130000}' '{"code":"22.4","quantity":610,"unit_cost":3850}],"claimed_saving_nok":130000}'
), ),
"BRU-LAKS-REHAB": ( "BRU-LAKS-REHAB": (
'{"measure":"Reduce scope","affected_items":[' '{"measure":"Reduce scope","affected_items":['
'{"code":"05.2","quantity":4300,"unit_cost":215},' '{"code":"01.1","quantity":1,"unit_cost":620000},'
'{"code":"07.4","quantity":2400,"unit_cost":690}],"claimed_saving_nok":210000}' '{"code":"87.3","quantity":640,"unit_cost":980}],"claimed_saving_nok":210000}'
), ),
} }

View file

@ -26,6 +26,7 @@ from pathlib import Path
import pytest import pytest
from agent_framework import Agent from agent_framework import Agent
from test_portfolio import REPLIES
from portfolio_optimiser.budget import ( from portfolio_optimiser.budget import (
Budget, Budget,
@ -42,25 +43,11 @@ from portfolio_optimiser.run import run_portfolio
_PORTFOLIO_IDS = ["FV42-GSV-E1", "RV13-RAS-TP", "BRU-LAKS-REHAB"] _PORTFOLIO_IDS = ["FV42-GSV-E1", "RV13-RAS-TP", "BRU-LAKS-REHAB"]
# Per-project replies (the tested constants of tests/test_portfolio.py, unchanged): all three # Per-project replies: IMPORTED from tests/test_portfolio.py rather than copied. The copy claimed to
# validate, and their claimed savings are DISTINCT — which is how a run is keyed back to its # be "the tested constants ... unchanged" and then drifted — S4.0's baseline anchoring caught it,
# project here, since ``RunResult`` carries no project id. 200000 + 130000 = the first two. # because the copies quoted cost codes belonging to OTHER projects. All three validate, and their
REPLIES = { # claimed savings are DISTINCT — which is how a run is keyed back to its project here, since
"FV42-GSV-E1": ( # ``RunResult`` carries no project id. 200000 + 130000 = the first two.
'{"measure":"Reduce scope","affected_items":['
'{"code":"05.2","quantity":4300,"unit_cost":215},'
'{"code":"03.1","quantity":1800,"unit_cost":310}],"claimed_saving_nok":200000}'
),
"RV13-RAS-TP": (
'{"measure":"Material substitution","affected_items":['
'{"code":"88.2","quantity":180,"unit_cost":4200}],"claimed_saving_nok":130000}'
),
"BRU-LAKS-REHAB": (
'{"measure":"Reduce scope","affected_items":['
'{"code":"05.2","quantity":4300,"unit_cost":215},'
'{"code":"07.4","quantity":2400,"unit_cost":690}],"claimed_saving_nok":210000}'
),
}
_FIRST_TWO_SAVING = 330000 _FIRST_TWO_SAVING = 330000
# Measured: 32 tokens per run at tokens=8. 80 total funds two runs (32 + 32 = 64) and leaves 16, # Measured: 32 tokens per run at tokens=8. 80 total funds two runs (32 + 32 = 64) and leaves 16,

View file

@ -34,6 +34,8 @@ from __future__ import annotations
import asyncio import asyncio
from typing import Any, Mapping, Sequence from typing import Any, Mapping, Sequence
from test_portfolio import REPLIES
from portfolio_optimiser.budget import PortfolioBudget, PortfolioMeter from portfolio_optimiser.budget import PortfolioBudget, PortfolioMeter
from portfolio_optimiser.run import RunFailure, run_portfolio from portfolio_optimiser.run import RunFailure, run_portfolio
from portfolio_optimiser.simulation import ScriptedChatClient from portfolio_optimiser.simulation import ScriptedChatClient
@ -49,22 +51,11 @@ _DEFAULT_REPLY = (
'[{"code":"01.1","quantity":1,"unit_cost":100000}],"claimed_saving_nok":20000}' '[{"code":"01.1","quantity":1,"unit_cost":100000}],"claimed_saving_nok":20000}'
) )
REPLIES = { # ``REPLIES`` is IMPORTED from tests/test_portfolio.py (see the import above) rather than copied —
"FV42-GSV-E1": ( # the local copy had drifted onto other projects' cost codes, which S4.0's baseline anchoring
'{"measure":"Reduce scope","affected_items":[' # rejects. ``_DEFAULT_REPLY`` above is reached only by a prompt naming none of the three mapped
'{"code":"05.2","quantity":4300,"unit_cost":215},' # projects; on the anchored road path such a reply is rejected as a fabricated cost line, which is
'{"code":"03.1","quantity":1800,"unit_cost":310}],"claimed_saving_nok":200000}' # the correct outcome for a project this fixture never described.
),
"RV13-RAS-TP": (
'{"measure":"Material substitution","affected_items":['
'{"code":"88.2","quantity":180,"unit_cost":4200}],"claimed_saving_nok":130000}'
),
"BRU-LAKS-REHAB": (
'{"measure":"Reduce scope","affected_items":['
'{"code":"05.2","quantity":4300,"unit_cost":215},'
'{"code":"07.4","quantity":2400,"unit_cost":690}],"claimed_saving_nok":210000}'
),
}
# Measured in tests/test_portfolio_budget_loadbearing.py: 4 chat calls x ``tokens`` per reply, so a # Measured in tests/test_portfolio_budget_loadbearing.py: 4 chat calls x ``tokens`` per reply, so a
# completed run costs a flat 32 tokens at tokens=8. # completed run costs a flat 32 tokens at tokens=8.

View file

@ -0,0 +1,285 @@
"""S4.0 load-bearing seam: the deterministic gate is ANCHORED to the project's real cost baseline.
Review finding F3: every stage of ``validate_proposal`` reasoned about the numbers the *proposal
itself* supplied, so a hallucinated cost line (an invented code, or a real code at an invented
magnitude) could clear the whole gate as long as its own arithmetic was internally consistent. The
reconciliation stage closes that: each ``affected_item`` must correspond to a line in the project's
cost baseline, within a configured tolerance.
Every RED here is a genuine OUTCOME FLIP, not a reason-string check: the fabricated proposals are
deliberately built to pass the P90 / nominal / method stages, so detaching the reconciliation makes
them ``ValidatedProposal`` again. Controls prove causality (a real baseline line, same shape,
validates), and the no-baseline arm proves the argument stays OPTIONAL (``None`` = pre-S4.0
behaviour, which is why the existing suite stands).
Measured detach points (see the session log): the reconciliation stage · the magnitude tolerance ·
the road-path wiring in ``run.py`` · the bundle-path wiring · the method-cap registry key (F8).
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from conftest import SyntheticUsageChatClient
from pydantic import ValidationError
from portfolio_optimiser import okf
from portfolio_optimiser.ir import AffectedItem, CostBaseline, CostBaselineLine, SavingsProposal
from portfolio_optimiser.reference_domain import load_reference_projects
from portfolio_optimiser.run import run_project
from portfolio_optimiser.validator import (
Rejection,
ValidatedProposal,
baseline_from_project,
validate_proposal,
)
# The repo-local S4.0 fixture bundle: the ONLY bundle carrying a ``cost-baseline.json`` (the pre-
# amendment bundles deliberately have none — that is the optional-argument control below).
_DATA = Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser" / "data" / "bundles"
BASELINE_BUNDLE = _DATA / "bygg-energi-baseline-mikro"
PRE_AMENDMENT_BUNDLE = _DATA / "bygg-energi-mikro-a"
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
# FV42-GSV-E1's real cost line 05.2 (Asfalt Ab11): 4300 m2 x 215 NOK. Affected total 924500 ->
# degenerate P90 = 0.30 x 924500 = 277350, so claimed 200000 clears every pre-S4.0 stage.
_REAL_CODE = "05.2"
_REAL_QTY = 4300.0
_REAL_UNIT_COST = 215.0
_REAL_CLAIM = 200000.0
# The F3 scenario: an invented code carrying a 10 MNOK line. The claim is set at exactly the generic
# feasible (0.30 x 10 MNOK) so the fabrication is numerically IMPECCABLE — every pre-S4.0 stage
# passes it. Only the baseline reconciliation can reject it, which is what makes the detach a flip.
_FAKE_CODE = "XX"
_FAKE_UNIT_COST = 10_000_000.0
_FAKE_CLAIM = 3_000_000.0
def _fv42_baseline() -> CostBaseline:
return baseline_from_project(
next(p for p in load_reference_projects() if p.id == "FV42-GSV-E1")
)
def _proposal(code: str, quantity: float, unit_cost: float, claimed: float) -> SavingsProposal:
return SavingsProposal(
project_id="FV42-GSV-E1",
measure="Reduce scope",
affected_items=[AffectedItem(code=code, quantity=quantity, unit_cost=unit_cost)],
claimed_saving_nok=claimed,
assumptions={},
)
# --- Arm 1: the reconciliation stage itself -------------------------------------------------------
def test_fabricated_cost_code_is_rejected() -> None:
"""RED (F3): a proposal citing a cost code that exists nowhere in the project's baseline is
rejected, even though its own arithmetic clears the LP/P90/nominal stages. Detach the
reconciliation stage and the SAME proposal validates."""
result = validate_proposal(
_proposal(_FAKE_CODE, 1.0, _FAKE_UNIT_COST, _FAKE_CLAIM), baseline=_fv42_baseline()
)
assert isinstance(result, Rejection), "a hallucinated cost code must never reach validated"
assert "unknown cost code" in result.reason
assert _FAKE_CODE in result.reason
def test_real_baseline_line_still_validates() -> None:
"""Causality control: the same shape of proposal on a REAL baseline line validates — so the
rejection above is caused by the code being absent from the baseline, not by the new stage
rejecting everything."""
result = validate_proposal(
_proposal(_REAL_CODE, _REAL_QTY, _REAL_UNIT_COST, _REAL_CLAIM), baseline=_fv42_baseline()
)
assert isinstance(result, ValidatedProposal)
def test_baseline_is_optional_and_none_is_pre_s40_behaviour() -> None:
"""The baseline argument is OPTIONAL: with ``None`` the fabricated proposal validates exactly as
it did before S4.0. This is the property the existing suite rests on and the reason the RED
above is a flip rather than a tightening of an already-rejecting path."""
result = validate_proposal(_proposal(_FAKE_CODE, 1.0, _FAKE_UNIT_COST, _FAKE_CLAIM))
assert isinstance(result, ValidatedProposal)
# --- Arm 2: the magnitude tolerance ---------------------------------------------------------------
def test_inflated_unit_cost_on_a_real_code_is_rejected() -> None:
"""RED: a REAL cost code at an invented unit_cost (+20%, well past the 5% default tolerance) is
rejected. Detach the tolerance check and only the code-membership test remains the inflated
line then validates, because the code itself is genuine."""
inflated = _REAL_UNIT_COST * 1.20
result = validate_proposal(
_proposal(_REAL_CODE, _REAL_QTY, inflated, _REAL_CLAIM), baseline=_fv42_baseline()
)
assert isinstance(result, Rejection)
assert "unit_cost" in result.reason and _REAL_CODE in result.reason
def test_inflated_quantity_on_a_real_code_is_rejected() -> None:
"""RED: the same for quantity — a real code at an invented quantity (+20%) is rejected."""
result = validate_proposal(
_proposal(_REAL_CODE, _REAL_QTY * 1.20, _REAL_UNIT_COST, _REAL_CLAIM),
baseline=_fv42_baseline(),
)
assert isinstance(result, Rejection)
assert "quantity" in result.reason
def test_within_tolerance_deviation_is_admitted() -> None:
"""Causality control for the tolerance: a 2% deviation (rounding-scale, inside the 5% default)
validates the rejections above are caused by the SIZE of the deviation, not by any deviation
at all."""
result = validate_proposal(
_proposal(_REAL_CODE, _REAL_QTY, _REAL_UNIT_COST * 1.02, _REAL_CLAIM),
baseline=_fv42_baseline(),
)
assert isinstance(result, ValidatedProposal)
def test_tolerance_is_configurable() -> None:
"""The tolerance is config, not a constant: the same 2% deviation is rejected under a stricter
caller-supplied tolerance."""
result = validate_proposal(
_proposal(_REAL_CODE, _REAL_QTY, _REAL_UNIT_COST * 1.02, _REAL_CLAIM),
baseline=_fv42_baseline(),
tolerance=0.001,
)
assert isinstance(result, Rejection)
# --- Arm 3: the loader (fail-fast, mirroring ``load_ir_projection``) -------------------------------
def test_bundle_baseline_loads_from_the_fixture() -> None:
baseline = okf.load_cost_baseline(str(BASELINE_BUNDLE))
assert baseline.project_id == "BYGG-ENERGI-BASELINE-MIKRO"
assert baseline.items["ENERGI-TOTAL-EL"] == CostBaselineLine(quantity=180000, unit_cost=1.0)
def test_missing_baseline_is_fail_fast_but_optional_loader_returns_none() -> None:
"""Two deliberately different contracts over the same absence: the fail-fast loader raises (it
is authoritative startup input, like ``load_ir_projection``), while the OPTIONAL loader the run
path uses returns ``None`` a bundle written before the amendment is not an error, it is simply
un-anchored."""
with pytest.raises(FileNotFoundError):
okf.load_cost_baseline(str(PRE_AMENDMENT_BUNDLE))
assert okf.load_optional_cost_baseline(str(PRE_AMENDMENT_BUNDLE)) is None
def test_malformed_baseline_raises_even_on_the_optional_path(tmp_path) -> None:
"""Fail-closed where it matters: a baseline that EXISTS but is malformed raises on BOTH loaders.
Tolerating it would silently un-anchor the gate the RAW-inbox skip rule stops at this layer."""
(tmp_path / "cost-baseline.json").write_text(
json.dumps({"project_id": "P", "items": {"01.1": {"quantity": 1}}}), encoding="utf-8"
)
with pytest.raises(ValidationError):
okf.load_optional_cost_baseline(str(tmp_path))
# --- Arm 4: the run-path wiring (road + bundle) ---------------------------------------------------
def _reply(code: str, quantity: float, unit_cost: float, claimed: float) -> str:
return json.dumps(
{
"measure": "Reduce scope",
"affected_items": [{"code": code, "quantity": quantity, "unit_cost": unit_cost}],
"claimed_saving_nok": claimed,
}
)
def _factory(reply: str):
def factory(role: str):
return SyntheticUsageChatClient(default_reply=reply)
return factory
async def test_road_path_anchors_the_gate_to_the_reference_baseline(docs_dir, fresh_store) -> None:
"""RED (road wiring): a numerically-impeccable fabricated cost line is REJECTED end-to-end
through ``run_project``. Detach the road-path baseline (stop passing it) and the same run
returns a ValidatedProposal."""
result = await run_project(
"FV42-GSV-E1",
"local",
docs_dir=docs_dir,
verdict_input=_VERDICT_INPUT,
client_factory=_factory(_reply(_FAKE_CODE, 1.0, _FAKE_UNIT_COST, _FAKE_CLAIM)),
store=fresh_store,
)
assert isinstance(result.outcome, Rejection)
assert "unknown cost code" in result.outcome.reason
async def test_road_path_control_real_line_validates(docs_dir, fresh_store) -> None:
"""Causality control for the road wiring: the real 05.2 line validates through the same path."""
result = await run_project(
"FV42-GSV-E1",
"local",
docs_dir=docs_dir,
verdict_input=_VERDICT_INPUT,
client_factory=_factory(_reply(_REAL_CODE, _REAL_QTY, _REAL_UNIT_COST, _REAL_CLAIM)),
store=fresh_store,
)
assert isinstance(result.outcome, ValidatedProposal)
async def test_bundle_path_anchors_when_the_bundle_declares_a_baseline(fresh_store) -> None:
"""RED (bundle wiring): a bundle that ships ``cost-baseline.json`` anchors its run — the
fabricated line is rejected. Detach the bundle-path load and it validates again."""
result = await run_project(
"BYGG-ENERGI-BASELINE-MIKRO",
"local",
docs_dir=str(BASELINE_BUNDLE),
bundle_dir=str(BASELINE_BUNDLE),
verdict_input=_VERDICT_INPUT,
client_factory=_factory(_reply(_FAKE_CODE, 1.0, 300000.0, 90000.0)),
store=fresh_store,
)
assert isinstance(result.outcome, Rejection)
assert "unknown cost code" in result.outcome.reason
async def test_pre_amendment_bundle_runs_unchanged(fresh_store) -> None:
"""Control + backward compatibility: the SAME fabricated reply validates against a bundle with
no ``cost-baseline.json``. Anchoring is opt-in per bundle, so every pre-S4.0 bundle (including
the commons-owned goldens) runs byte-identically to before."""
result = await run_project(
"BYGG-ENERGI-MIKRO-A",
"local",
docs_dir=str(PRE_AMENDMENT_BUNDLE),
bundle_dir=str(PRE_AMENDMENT_BUNDLE),
verdict_input=_VERDICT_INPUT,
client_factory=_factory(_reply(_FAKE_CODE, 1.0, 300000.0, 90000.0)),
store=fresh_store,
)
assert isinstance(result.outcome, ValidatedProposal)
# --- Arm 5: F8 — method caps keyed by config, not the literal measure string ----------------------
def test_method_cap_is_keyed_by_config_not_a_hardcoded_string() -> None:
"""F8: the method-specific cap comes from a REGISTRY the caller can supply. A caller-configured
cap for a measure with no built-in entry rejects a proposal the generic P90 stage passes so
the rule is keyed by configuration, not by the ``energy_efficiency`` literal."""
proposal = SavingsProposal(
project_id="P-ASFALT",
measure="asfalt_reduction",
affected_items=[AffectedItem(code="05.2", quantity=100000, unit_cost=1.0)],
claimed_saving_nok=20000,
assumptions={},
)
assert isinstance(validate_proposal(proposal), ValidatedProposal) # generic P90 = 30000
capped = validate_proposal(proposal, method_caps={"asfalt_reduction": 0.10})
assert isinstance(capped, Rejection)
assert "method cap" in capped.reason