feat(validator): anchor the deterministic gate to the project's real cost baseline (S4.0)
Every stage of validate_proposal reasoned only about numbers the proposal itself supplied, so an internally-consistent hallucination cleared the whole gate (F3). A new stage 0 reconciles each affected_item against the project's CostBaseline before the CBC solve: an unknown cost code is rejected, and a real code carrying a quantity/unit_cost outside the configured tolerance (5% default, relative to the baseline value) is rejected. Validation, never repair. The baseline argument is OPTIONAL (None = pre-S4.0 behaviour), but both run paths set it: the road path projects project.cost_items, the bundle path loads cost-baseline.json when the bundle ships one. Bundles written before the amendment stay un-anchored, so the commons-owned goldens run byte-identically; a baseline that exists but is malformed still raises on both loaders. F8: the method-specific cap now comes from the METHOD_CAPS registry (measure type -> fraction, injectable) instead of an energy_efficiency string comparison. The baseline format and tolerance semantics were decided locally — the commons amendment (D-A pt. 2) never arrived, exactly as in S3.2. D7 mirroring stays open. Three portfolio fixtures quoted cost codes belonging to OTHER projects; the new gate caught them. They now quote each project's own lines, and the two copied REPLIES tables import the single source instead of drifting from it. Load-bearing measured (tests/test_s40_cost_baseline_loadbearing.py), six mutations all red: detach the reconciliation stage; detach the magnitude tolerance; detach the road wiring; detach the bundle wiring; ignore the injected cap registry; make the optional loader tolerant of malformed content. Control: with the road wiring detached the repaired portfolio fixtures still pass, so they are not masking the seam. 597 -> 612 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JdwK7bQ4BZkWH4t8MRDKb4
This commit is contained in:
parent
012adc0a3c
commit
126807aee7
16 changed files with 645 additions and 69 deletions
20
CLAUDE.md
20
CLAUDE.md
|
|
@ -124,6 +124,26 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
(`tests/test_step32_multicandidate_loadbearing.py` + `test_step8_promotion_loadbearing.py`), fem
|
||||
mutasjoner alle røde: detach per-dom-nøklingen · detach feltene `promote_verdict` skriver · gjør en
|
||||
delvis/uparsebar nøkkel tolerant · normaliser magnituden ved skriving · fjern fallbacken (kontroll).
|
||||
- **Den deterministiske gaten er FORANKRET i prosjektets faktiske kostbaseline (S4.0, F3/F8):** før
|
||||
S4.0 resonnerte HVER stage kun om tall forslaget selv oppga, så en internt konsistent hallusinasjon
|
||||
klarerte hele gaten. `validate_proposal(..., baseline=...)` avstemmer nå hvert `affected_item` mot
|
||||
`CostBaseline` (`ir.py`) i en **stage 0 — FØR løseren** (billigst, og den eneste som skiller en
|
||||
oppdiktet linje fra en ekte; å bruke en CBC-solve på tall som ikke tilhører prosjektet er arbeid på
|
||||
et krav som uansett ikke kan valideres). To uavhengige avvisninger: ukjent kostkode, og ekte kode
|
||||
med `quantity`/`unit_cost` utenfor `tolerance` (default 5 %, **konfig**) relativt til BASELINE-verdien.
|
||||
Validering, ALDRI reparasjon — forslaget avvises, aldri stilltiende korrigert til baselinen.
|
||||
**Argumentet er VALGFRITT** (`None` = pre-S4.0-oppførsel), men begge run-stier SETTER det: road-stien
|
||||
fra `project.cost_items` (alltid — estimatet ER prosjektet), bundle-stien KUN når bundelen shipper
|
||||
`cost-baseline.json` (`load_optional_cost_baseline`) — en pre-amendment-bundle er legitimt
|
||||
uforankret, og det er dét som holder commons-goldenene byte-identiske. **Toleransen stopper ved
|
||||
fravær:** en baseline som FINNES men er malformed raiser på BEGGE loaderne (å lese korrupt som
|
||||
«ingen baseline» ville gitt en uforankret gate i forkledning — samme resonnement som `read_spend`).
|
||||
**F8:** metode-cap-en slås opp i `METHOD_CAPS`-registeret (måletype→brøk, injiserbart), ikke mot
|
||||
`energy_efficiency`-literalen — en andre metode er nå data, ikke en redigering av validatoren.
|
||||
Format- og toleranse-semantikken er bestemt LOKALT (commons-amendmentet D-A pkt. 2 kom aldri, som i
|
||||
S3.2); **D7-speiling ÅPEN.** Load-bearing MÅLT (`tests/test_s40_cost_baseline_loadbearing.py`), seks
|
||||
mutasjoner alle røde: detach avstemmings-stagen · detach magnitude-toleransen · detach road-wiringen ·
|
||||
detach bundle-wiringen · ignorer det injiserte cap-registeret · gjør den valgfrie loaderen tolerant.
|
||||
- **Globalt token-tak håndheves FØR kall, aldri bare etterpå (S3.4, F10):** `PortfolioBudget` +
|
||||
`PortfolioMeter` er ÉN ledger over hele porteføljepasset (og — seedet av `read_spend` — på tvers
|
||||
av pass), mens per-run `Budget`/`TokenMeter` er uendret. Taket har tre tenner, med hver sin jobb:
|
||||
|
|
|
|||
16
README.md
16
README.md
|
|
@ -71,7 +71,9 @@ One run, one project, eight steps — with the learning loop closing across runs
|
|||
2. **Hypothesise** — one typed candidate measure (strict IR, fail-fast schema).
|
||||
3. **Debate** — a maker-checker pair argues the reasoning (round-capped).
|
||||
4. **Validate** — two falsifiers on the same candidate: the **deterministic validator** gates
|
||||
the numbers (blocking, never optional) and the **checker** gates the reasoning.
|
||||
the numbers (blocking, never optional) and the **checker** gates the reasoning. The validator
|
||||
is anchored to the project's declared cost baseline, so a proposal cannot invent the cost
|
||||
lines it claims to save against.
|
||||
5. **Refine** — a rejected attempt retries *informed* by the rejection reason, under hard
|
||||
attempt and token caps. Unbounded loops are forbidden everywhere.
|
||||
6. **Propose or discard** — a validated proposal with risk percentiles, or a typed rejection.
|
||||
|
|
@ -149,6 +151,18 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
|||
the merge would produce a key belonging to neither. `promote_verdict` writes all three, so a
|
||||
promoted verdict about one candidate never surfaces for another.
|
||||
|
||||
A bundle may also ship a **`cost-baseline.json`** — the project's actual cost lines,
|
||||
`{code: {quantity, unit_cost}}` — and when it does, the deterministic validator reconciles every
|
||||
affected item of a proposal against it before anything else runs. A cost code the project does not
|
||||
have is rejected, and so is a real code carrying a quantity or unit cost outside the configured
|
||||
tolerance (5% by default, relative to the baseline value). Without it, every stage of the gate
|
||||
reasons only about numbers the proposal supplied itself, so an internally consistent hallucination
|
||||
passes. The reconciliation validates; it never repairs a proposal into the baseline. A bundle that
|
||||
ships no baseline is simply un-anchored and runs exactly as before, while a baseline that is
|
||||
present but malformed is an error rather than a silent fall-back to un-anchored. On the
|
||||
reference-domain (non-bundle) path the project's own cost items are the baseline, so those runs
|
||||
are always anchored.
|
||||
|
||||
The **prior-verdict fold — the learning step — happens only on the `--bundle-dir` path**; a
|
||||
plain `--docs-dir`-only run is single-shot (no fold). `--decision`/`--rationale` apply to the
|
||||
single-project path only and are inert in portfolio mode. **`--outbox-dir` must differ from
|
||||
|
|
|
|||
|
|
@ -0,0 +1,20 @@
|
|||
---
|
||||
type: project
|
||||
title: "Kontorbygg B — energibaseline med erklærte kostlinjer (repo-lokal fixture)"
|
||||
description: "Syntetisk energibaseline for S4.0-fixturen. Ett kontorbygg, én kandidat-kostpost (ENERGI-TOTAL-EL), og en eksplisitt cost-baseline.json den deterministiske validatoren avstemmer mot."
|
||||
timestamp: 2026-08-03
|
||||
---
|
||||
|
||||
# Kontorbygg B (repo-lokal test-fixture)
|
||||
|
||||
Et lite syntetisk kontorbygg med én dokumentert energibaseline. Til forskjell fra mikro-A erklærer
|
||||
denne bundelen kostlinjene sine maskinlesbart i `cost-baseline.json`, slik at et forslag som siterer
|
||||
en oppdiktet kostkode — eller en ekte kode med oppdiktet størrelse — blir avvist av validatorens
|
||||
avstemmings-stage før løseren i det hele tatt kjøres.
|
||||
|
||||
## Energibaseline
|
||||
|
||||
- Årlig elektrisk energikostnad (`ENERGI-TOTAL-EL`): 180 000 NOK/år (syntetisk), ført som
|
||||
`quantity: 180000` × `unit_cost: 1.0` i `cost-baseline.json`.
|
||||
- Kandidat-tiltak: LED-retrofit av kontorbelysning.
|
||||
- Modellert besparelse: ~18 000 NOK/år (innenfor validatorens feasibelt-område).
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
{
|
||||
"_note": "SYNTHETIC repo-lokal kostbaseline (S4.0-fixture) — ikke ekte data. Prosjektets FAKTISKE kostlinjer: den deterministiske validatoren avstemmer et forslags affected_items mot disse (okf.load_cost_baseline -> validator._reconcile_against_baseline). Formatet er bestemt LOKALT (commons-amendmentet D-A pkt. 2 kom aldri); D7-speiling er ÅPEN.",
|
||||
"project_id": "BYGG-ENERGI-BASELINE-MIKRO",
|
||||
"items": {
|
||||
"ENERGI-TOTAL-EL": {
|
||||
"quantity": 180000,
|
||||
"unit_cost": 1.0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
---
|
||||
type: index
|
||||
okf_version: 0.1
|
||||
title: "Bygg-energi baseline-mikro — repo-lokal fixture (S4.0 kostbaseline-forankring)"
|
||||
description: "Minimal repo-lokal OKF-bundle som SHIPPER en cost-baseline.json, så bundle-stiens validator-forankring (S4.0, F3) kan testes ende-til-ende. Speiler bygg-energi-mikro-A, men med kostbaselinen lagt til."
|
||||
tags: [fixture, S4.0, kostbaseline]
|
||||
timestamp: 2026-08-03
|
||||
---
|
||||
|
||||
# Bygg-energi baseline-mikro (repo-lokal test-fixture)
|
||||
|
||||
En **repo-lokal mini OKF-bundle** under pakkens `data/` (ALDRI `shared/` — subtree er PULL-ONLY).
|
||||
Eneste formål: være den ene bundelen som **erklærer sin kostbaseline**, slik at
|
||||
`run_project(bundle_dir=...)` faktisk forankrer den deterministiske gaten mot prosjektets virkelige
|
||||
kostlinjer. De eldre bundlene bærer bevisst INGEN `cost-baseline.json` — de er kontrollen som viser
|
||||
at forankringen er opt-in per bundle (pre-amendment-bundler kjører uendret).
|
||||
|
||||
## Innhold (progressiv disclosure)
|
||||
|
||||
- [bygg-kontor-baseline.md](bygg-kontor-baseline.md) — `type: project` — bygget og energibaselinen
|
||||
som lese-kontekst (og `bundle_citations`-kilde).
|
||||
|
||||
`validator-input.json` er IR-projeksjonen (kandidatens kost-IR), og `cost-baseline.json` er
|
||||
prosjektets FAKTISKE kostlinjer (`code -> {quantity, unit_cost}`) som `okf.load_cost_baseline` leser
|
||||
og validatoren avstemmer `affected_items` mot. De to er bevisst adskilt: IR-projeksjonen er hva
|
||||
noen *foreslår*, baselinen er hva prosjektet *er*.
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
{
|
||||
"_note": "SYNTHETIC repo-lokal IR-projeksjon (S4.0-fixture) — ikke ekte data. Kandidatens kost-IR; project_id matcher det bundle-backede prosjektets id (run._project_from_bundle fail-faster ved mismatch). Kostlinjene her SPEILER cost-baseline.json med vilje: et forslag om denne kandidaten skal avstemme rent.",
|
||||
"project_id": "BYGG-ENERGI-BASELINE-MIKRO",
|
||||
"measure": "LED-retrofit av kontorbelysning",
|
||||
"affected_items": [
|
||||
{
|
||||
"code": "ENERGI-TOTAL-EL",
|
||||
"quantity": 180000,
|
||||
"unit_cost": 1.0
|
||||
}
|
||||
],
|
||||
"claimed_saving_nok": 18000
|
||||
}
|
||||
|
|
@ -28,7 +28,7 @@ from agent_framework import BaseChatClient, Message
|
|||
from pydantic import ValidationError
|
||||
|
||||
from portfolio_optimiser.budget import TokenMeter
|
||||
from portfolio_optimiser.ir import SavingsProposal
|
||||
from portfolio_optimiser.ir import CostBaseline, SavingsProposal
|
||||
from portfolio_optimiser.reference_domain import Project
|
||||
from portfolio_optimiser.validator import (
|
||||
Rejection,
|
||||
|
|
@ -110,6 +110,7 @@ async def generate_via_llm(
|
|||
meter: TokenMeter,
|
||||
*,
|
||||
max_attempts: int = 3,
|
||||
baseline: CostBaseline | None = None,
|
||||
) -> ValidatedProposal | Rejection:
|
||||
"""Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds,
|
||||
the meter checked in this loop:
|
||||
|
|
@ -123,7 +124,12 @@ async def generate_via_llm(
|
|||
|
||||
The only per-attempt falsifier here is the deterministic validator (the numbers). The
|
||||
checker is a run-level, one-shot signal (run.py, before generation); seeding generation
|
||||
with the checker critique is separately scoped and NOT done here. Returns
|
||||
with the checker critique is separately scoped and NOT done here.
|
||||
|
||||
``baseline`` (S4.0) is handed straight to ``validate_proposal``, so a fabricated cost line is
|
||||
falsified per ATTEMPT like any other rejection — and its reason feeds the next attempt's prompt
|
||||
through the SAME informed-refinement path (Step 5), which is why no new loop appears here.
|
||||
Returns
|
||||
``ValidatedProposal | Rejection``; never a malformed proposal; raises ``BudgetExceeded``
|
||||
when the meter cap is crossed."""
|
||||
|
||||
|
|
@ -146,7 +152,7 @@ async def generate_via_llm(
|
|||
# accumulated history (bounded prompt growth).
|
||||
messages = _build_messages(project, context, prior_rejection=last)
|
||||
candidate = await _fetch_parsed(messages)
|
||||
result = validate_proposal(candidate)
|
||||
result = validate_proposal(candidate, baseline=baseline)
|
||||
if isinstance(result, ValidatedProposal):
|
||||
return result
|
||||
last = result
|
||||
|
|
|
|||
|
|
@ -24,6 +24,31 @@ class AffectedItem(BaseModel):
|
|||
return self.quantity * self.unit_cost
|
||||
|
||||
|
||||
class CostBaselineLine(BaseModel):
|
||||
"""One line of a project's ACTUAL cost baseline: the quantity and unit cost a proposal's
|
||||
``AffectedItem`` for that code must reconcile against (S4.0, F3)."""
|
||||
|
||||
quantity: float = Field(ge=0)
|
||||
unit_cost: float = Field(gt=0)
|
||||
|
||||
|
||||
class CostBaseline(BaseModel):
|
||||
"""A project's cost baseline, keyed by cost code — the ground truth the deterministic
|
||||
validator anchors ``affected_items`` to, so the gate cannot be fed hallucinated cost lines.
|
||||
|
||||
Deliberately a typed IR contract (not a loader-private shape): both sources project INTO
|
||||
it — an OKF bundle's ``cost-baseline.json`` (``okf.load_cost_baseline``) and the road
|
||||
reference domain's ``cost_items`` (``validator.baseline_from_project``) — so the validator
|
||||
sees ONE representation regardless of path, and the Claude-SDK sibling can mirror it (D7).
|
||||
|
||||
The projection/tolerance semantics were decided HERE: the commons amendment specifying
|
||||
``cost-baseline.json`` never arrived, exactly as in S3.2. D7 mirroring stays OPEN.
|
||||
"""
|
||||
|
||||
project_id: str
|
||||
items: dict[str, CostBaselineLine]
|
||||
|
||||
|
||||
class SavingsProposal(BaseModel):
|
||||
"""Typed IR for a candidate cost-saving measure (B1)."""
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,8 @@
|
|||
Reads a bundle the way OKF intends (progressive disclosure): start at ``index.md``, follow
|
||||
intra-bundle cross-links **recursively, depth-first in first-seen link order**, parse each file's
|
||||
YAML frontmatter, classify by the one required ``type`` field. **NO** ``agent_framework``, **NO**
|
||||
``mcp`` — pure stdlib, so the SAME navigation serves both the MAF and the Claude-SDK
|
||||
``mcp`` — stdlib + ``pydantic`` only (as ``dimension.py``; the typed contracts this module loads
|
||||
live in ``ir.py``), so the SAME navigation serves both the MAF and the Claude-SDK
|
||||
implementations unchanged (målbilde §4 vendor-neutrality).
|
||||
|
||||
Link resolution follows ``shared/method-spec.md`` §3 Step 1: a leading ``/`` denotes the **bundle
|
||||
|
|
@ -29,10 +30,12 @@ from dataclasses import dataclass
|
|||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from portfolio_optimiser.ir import CostBaseline
|
||||
from portfolio_optimiser.retrieval import PathSecurityError, safe_resolve
|
||||
|
||||
_INDEX_NAME = "index.md"
|
||||
_IR_PROJECTION = "validator-input.json"
|
||||
_COST_BASELINE = "cost-baseline.json"
|
||||
# Intra-bundle markdown cross-links: ``](target.md)``. A path separator is NOT a rejection reason —
|
||||
# ``_resolve_target`` decides in-/out-of-bundle, and only escape is refused (method-spec §3 Step 1).
|
||||
_LINK_RE = re.compile(r"\]\(([^)]+\.md)\)")
|
||||
|
|
@ -285,6 +288,39 @@ def link_in_index(bundle_dir: str, target_name: str, label: str) -> bool:
|
|||
return True
|
||||
|
||||
|
||||
def load_cost_baseline(bundle_dir: str, name: str = _COST_BASELINE) -> CostBaseline:
|
||||
"""Load the bundle's cost baseline (``cost-baseline.json`` by default): the project's ACTUAL
|
||||
cost lines (``{code: {quantity, unit_cost}}``), which the deterministic validator reconciles a
|
||||
proposal's ``affected_items`` against (S4.0, F3).
|
||||
|
||||
Fail-fast, mirroring ``load_ir_projection`` and ``dimension.load_dimension``: a missing file
|
||||
raises ``FileNotFoundError`` and malformed content raises ``pydantic.ValidationError``. A cost
|
||||
baseline is authoritative gate input — a tolerantly-degraded one would silently un-anchor the
|
||||
gate, which is precisely the failure this stage exists to prevent. (The tolerant skip rule
|
||||
belongs to the RAW verdict-inbox layer, never here.)
|
||||
|
||||
Use ``load_optional_cost_baseline`` where the ABSENCE of the file is legitimate."""
|
||||
resolved = Path(safe_resolve(bundle_dir, name))
|
||||
if not resolved.is_file():
|
||||
raise FileNotFoundError(f"cost baseline not found in bundle: {name!r}")
|
||||
return CostBaseline.model_validate_json(resolved.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def load_optional_cost_baseline(bundle_dir: str, name: str = _COST_BASELINE) -> CostBaseline | None:
|
||||
"""``load_cost_baseline`` where a MISSING file is legitimate: returns ``None`` instead of
|
||||
raising. This is the run path's loader — a bundle authored before the baseline amendment is
|
||||
simply un-anchored (``None`` = pre-S4.0 behaviour), not an error, which is what keeps every
|
||||
existing bundle (including the commons-owned goldens) running byte-identically.
|
||||
|
||||
The tolerance stops at absence: a baseline that EXISTS but is malformed still raises. Reading a
|
||||
corrupt baseline as "no baseline" would hand back an un-anchored gate under the appearance of an
|
||||
anchored one (the same reasoning as ``budget.read_spend``)."""
|
||||
try:
|
||||
return load_cost_baseline(bundle_dir, name)
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
|
||||
|
||||
def load_ir_projection(bundle_dir: str, name: str = _IR_PROJECTION) -> dict[str, Any]:
|
||||
"""Load the bundle's IR projection (``validator-input.json`` by default): the candidate
|
||||
measure's cost-IR (``measure``, ``affected_items``, ``claimed_saving_nok``) — the
|
||||
|
|
|
|||
|
|
@ -55,7 +55,7 @@ from portfolio_optimiser.generate import generate_via_llm
|
|||
from portfolio_optimiser.ir import SavingsProposal
|
||||
from portfolio_optimiser.provenance import ProvenanceStamp
|
||||
from portfolio_optimiser.reference_domain import Project, load_reference_projects
|
||||
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
||||
from portfolio_optimiser.validator import Rejection, ValidatedProposal, baseline_from_project
|
||||
from portfolio_optimiser import okf, outbox
|
||||
from portfolio_optimiser.semretrieval import (
|
||||
SEMANTIC_WEIGHT_DEFAULT,
|
||||
|
|
@ -373,9 +373,16 @@ async def run_project(
|
|||
# NOT keyword chunk-stuffing; the road path keeps the chunk-retrieval data source. ``debate_tools``
|
||||
# is the query-time retrieval surface — empty on the bundle path (navigation already placed the
|
||||
# curated context in the prompt, and a docs_dir==bundle_dir tool would re-leak the verdict layer).
|
||||
# S4.0 (F3): the run path SETS the validator's cost baseline, so the deterministic gate is
|
||||
# anchored to the project's real cost lines instead of the ones the proposal asserts.
|
||||
# * road path: the reference project's own ``cost_items`` ARE the baseline -> always anchored.
|
||||
# * bundle path: anchored only when the bundle SHIPS a ``cost-baseline.json``. A bundle written
|
||||
# before the amendment (every commons-owned golden) is legitimately un-anchored -> None =
|
||||
# pre-S4.0 behaviour. A baseline that exists but is malformed still raises (fail-closed).
|
||||
if bundle_dir is not None:
|
||||
bundle = okf.navigate_bundle(bundle_dir)
|
||||
project = _project_from_bundle(bundle_dir, project_id, bundle=bundle)
|
||||
baseline = okf.load_optional_cost_baseline(bundle_dir)
|
||||
# §4.1a context-scope: agents read ONLY dimension-scoped bundle knowledge (Step-3 filter);
|
||||
# dimension=None keeps the full context, byte-identical to before.
|
||||
context = okf.bundle_context(bundle, dimension=dimension.id if dimension else None)
|
||||
|
|
@ -383,6 +390,7 @@ async def run_project(
|
|||
debate_tools: list[Any] = []
|
||||
else:
|
||||
project = _project_by_id(project_id)
|
||||
baseline = baseline_from_project(project)
|
||||
chunks = retrieve_chunks("cost saving measure", docs_dir, top_k)
|
||||
citations = [chunk_dict_to_citation(c) for c in chunks]
|
||||
context = "\n".join(c["snippet"] for c in chunks)
|
||||
|
|
@ -480,7 +488,9 @@ async def run_project(
|
|||
|
||||
# 5. Structured candidate -> blocking validation on the NUMBERS; token bound = the meter.
|
||||
proposer_client = factory("proposer")
|
||||
validator_outcome = await generate_via_llm(proposer_client, project, gen_context, meter)
|
||||
validator_outcome = await generate_via_llm(
|
||||
proposer_client, project, gen_context, meter, baseline=baseline
|
||||
)
|
||||
proposal = validator_outcome.proposal
|
||||
|
||||
# 6. First-class provenance stamp (authoritative; independent of MAF Annotation).
|
||||
|
|
|
|||
|
|
@ -24,13 +24,13 @@ from __future__ import annotations
|
|||
import random
|
||||
import statistics
|
||||
import warnings
|
||||
from collections.abc import Callable
|
||||
from collections.abc import Callable, Mapping
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
|
||||
import pulp
|
||||
|
||||
from portfolio_optimiser.ir import AffectedItem, SavingsProposal
|
||||
from portfolio_optimiser.ir import AffectedItem, CostBaseline, CostBaselineLine, SavingsProposal
|
||||
from portfolio_optimiser.reference_domain import Project
|
||||
|
||||
MAX_SAVING_FRACTION = 0.30
|
||||
|
|
@ -46,6 +46,21 @@ INDEPENDENT gate: it can reject a proposal the generic P90 stage passes (not red
|
|||
fraction is calibrated against the reference domain; the CONDITION (a method-scoped stricter cap) is
|
||||
the encoded rule. Returns the same ``Rejection`` type — a validator stage, not a new gate."""
|
||||
|
||||
METHOD_CAPS: dict[str, float] = {_ENERGY_METHOD_MEASURE: _ENERGY_METHOD_MAX_FRACTION}
|
||||
"""S4.0 (F8): the method-cap REGISTRY — measure type -> method-scoped max saving fraction. The
|
||||
rule used to be an ``if proposal.measure == "energy_efficiency"`` branch, so encoding a second
|
||||
assessment method meant editing the validator. It is now data: a caller passes its own registry
|
||||
(``validate_proposal(..., method_caps=...)``), keyed by the measure type a dimension admits
|
||||
(``dimension.allowed_measure_types``), and the built-in entry stays the default so the Step-9
|
||||
behaviour is unchanged. Deliberately NOT a config file yet — the deliverable is the key-by-config
|
||||
seam (90%-prinsippet), not a settings format."""
|
||||
|
||||
BASELINE_TOLERANCE_DEFAULT = 0.05
|
||||
"""S4.0: the relative deviation a reconciled ``AffectedItem`` may show against its cost-baseline
|
||||
line (5%). A tolerance is needed at all because a proposer restates magnitudes in prose-derived,
|
||||
rounded form; it is small because its whole purpose is to leave no room for a FABRICATED magnitude.
|
||||
Config, not policy: every caller can tighten or loosen it per run (``tolerance=``)."""
|
||||
|
||||
_MC_SAMPLES = 512
|
||||
_MC_SEED = 20260624
|
||||
|
||||
|
|
@ -122,9 +137,83 @@ def _monte_carlo(
|
|||
return deciles[0], deciles[4], deciles[8] # P10, P50, P90
|
||||
|
||||
|
||||
def validate_proposal(proposal: SavingsProposal) -> ValidatedProposal | Rejection:
|
||||
def baseline_from_project(project: Project) -> CostBaseline:
|
||||
"""Project a road reference-domain ``Project``'s ``cost_items`` into the ``CostBaseline``
|
||||
contract — the road-path counterpart of ``okf.load_cost_baseline`` (S4.0). The road path always
|
||||
HAS its baseline (the estimate is the project), so this projection is total: no optional
|
||||
variant, and a run on this path is always anchored."""
|
||||
return CostBaseline(
|
||||
project_id=project.id,
|
||||
items={
|
||||
ci.code: CostBaselineLine(quantity=ci.quantity, unit_cost=ci.unit_cost)
|
||||
for ci in project.cost_items
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _reconcile_against_baseline(
|
||||
proposal: SavingsProposal, baseline: CostBaseline, tolerance: float
|
||||
) -> Rejection | None:
|
||||
"""S4.0 (F3): every affected item must correspond to a REAL line of the project's cost baseline.
|
||||
|
||||
Two independent failures, both fail-closed:
|
||||
|
||||
* the cost code is absent from the baseline — a fabricated line;
|
||||
* the code is real but its ``quantity``/``unit_cost`` deviates from the baseline line by more
|
||||
than ``tolerance`` (relative to the BASELINE value, which is the ground truth) — a real code
|
||||
carrying a fabricated magnitude.
|
||||
|
||||
Returns the first ``Rejection`` (validator's own type — never a new gate), or ``None`` when the
|
||||
proposal reconciles. Items are checked in their stated order so the reason is deterministic.
|
||||
A validation, never a repair: the proposal is rejected, not silently corrected to the baseline."""
|
||||
for item in proposal.affected_items:
|
||||
line = baseline.items.get(item.code)
|
||||
if line is None:
|
||||
return Rejection(
|
||||
proposal=proposal,
|
||||
reason=(
|
||||
f"unknown cost code {item.code!r}: not in project {baseline.project_id}'s "
|
||||
f"cost baseline ({len(baseline.items)} known codes)"
|
||||
),
|
||||
)
|
||||
for field, claimed, actual in (
|
||||
("quantity", item.quantity, line.quantity),
|
||||
("unit_cost", item.unit_cost, line.unit_cost),
|
||||
):
|
||||
if abs(claimed - actual) > tolerance * actual:
|
||||
return Rejection(
|
||||
proposal=proposal,
|
||||
reason=(
|
||||
f"{field} {claimed:g} for cost code {item.code!r} is outside the "
|
||||
f"{tolerance:.1%} tolerance around the baseline {field} {actual:g}"
|
||||
),
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def validate_proposal(
|
||||
proposal: SavingsProposal,
|
||||
*,
|
||||
baseline: CostBaseline | None = None,
|
||||
tolerance: float = BASELINE_TOLERANCE_DEFAULT,
|
||||
method_caps: Mapping[str, float] | None = None,
|
||||
) -> ValidatedProposal | Rejection:
|
||||
"""Deterministic blocking validation. Returns a ``ValidatedProposal`` only when the
|
||||
claim is feasible; otherwise a ``Rejection`` that cannot be consumed as validated."""
|
||||
claim is feasible; otherwise a ``Rejection`` that cannot be consumed as validated.
|
||||
|
||||
``baseline`` (S4.0, F3) anchors the gate to the project's ACTUAL cost lines: without it every
|
||||
stage reasons only about numbers the proposal supplied itself, so an internally-consistent
|
||||
hallucination clears the gate. It is OPTIONAL — ``None`` is exactly the pre-S4.0 behaviour, so a
|
||||
caller with no baseline (a bundle authored before the amendment) is unchanged — but both run
|
||||
paths SET it. ``tolerance`` is the reconciliation's config knob; ``method_caps`` overrides the
|
||||
built-in method-cap registry (F8)."""
|
||||
# Stage 0 (S4.0): reconcile against the cost baseline BEFORE the solver. It is the cheapest
|
||||
# stage and the only one that can tell a fabricated line from a real one — spending a CBC solve
|
||||
# on numbers that do not belong to the project is work on a claim that cannot be validated.
|
||||
if baseline is not None:
|
||||
blocked = _reconcile_against_baseline(proposal, baseline, tolerance)
|
||||
if blocked is not None:
|
||||
return blocked
|
||||
# Stage 1 (Pydantic) already ran at construction. Stage 2: real CBC solve.
|
||||
nominal = _solve_max_feasible(proposal.affected_items, MAX_SAVING_FRACTION)
|
||||
# Stage 3: Monte Carlo percentiles of the feasible saving.
|
||||
|
|
@ -152,15 +241,18 @@ def validate_proposal(proposal: SavingsProposal) -> ValidatedProposal | Rejectio
|
|||
# Stage 5 (Step 9, SC7-B): a method-specific rule STRICTER than the generic cap. A proposal in
|
||||
# the energy method (IPMVP Option A) must clear a lower, method-scoped feasible — an INDEPENDENT
|
||||
# gate that can reject a proposal the P90 stage passed. Same ``Rejection`` type, not a new gate.
|
||||
if proposal.measure == _ENERGY_METHOD_MEASURE:
|
||||
method_feasible = _ENERGY_METHOD_MAX_FRACTION * sum(
|
||||
it.total for it in proposal.affected_items
|
||||
)
|
||||
# F8 (S4.0): the cap is looked up in a REGISTRY keyed by measure type (config), not compared
|
||||
# against the ``energy_efficiency`` literal — a second assessment method is now data, not an
|
||||
# edit to this function. The built-in registry keeps the Step-9 behaviour identical.
|
||||
caps = METHOD_CAPS if method_caps is None else method_caps
|
||||
method_fraction = caps.get(proposal.measure)
|
||||
if method_fraction is not None:
|
||||
method_feasible = method_fraction * sum(it.total for it in proposal.affected_items)
|
||||
if proposal.claimed_saving_nok > method_feasible:
|
||||
return Rejection(
|
||||
proposal=proposal,
|
||||
reason=(
|
||||
f"claimed {proposal.claimed_saving_nok:.0f} exceeds the {_ENERGY_METHOD_MEASURE} "
|
||||
f"claimed {proposal.claimed_saving_nok:.0f} exceeds the {proposal.measure} "
|
||||
f"method cap {method_feasible:.0f} (stricter than the generic P90)"
|
||||
),
|
||||
)
|
||||
|
|
|
|||
|
|
@ -8,11 +8,13 @@ the gated live arm (Step 14).
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Callable, Sequence
|
||||
|
||||
import pytest
|
||||
from agent_framework import BaseChatClient
|
||||
|
||||
from portfolio_optimiser.reference_domain import load_reference_projects
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
from portfolio_optimiser.verdicts import VerdictStore, seed_store
|
||||
|
||||
|
|
@ -57,29 +59,63 @@ def make_client_factory() -> Callable[..., Callable[[str], BaseChatClient]]:
|
|||
return _make
|
||||
|
||||
|
||||
# A generic VALID SavingsProposal reply for any project not present in a portfolio reply map:
|
||||
# affected total = 1 x 100_000 = 100_000, P90 = 0.30 x 100_000 = 30_000, claimed 20_000 <= both
|
||||
# (Pydantic affected-total invariant and the validator P90 gate) -> always validates.
|
||||
_DEFAULT_CLAIM = 20_000
|
||||
# Last-resort reply for a prompt naming NO known reference project (the anchored per-project
|
||||
# fallback below cannot be built then). Kept for that case only.
|
||||
_PORTFOLIO_DEFAULT_REPLY = (
|
||||
'{"measure":"Reduce scope","affected_items":'
|
||||
'[{"code":"01.1","quantity":1,"unit_cost":100000}],"claimed_saving_nok":20000}'
|
||||
)
|
||||
|
||||
|
||||
def _anchored_default_replies() -> dict[str, str]:
|
||||
"""A VALID default proposal PER reference project, quoting that project's OWN first cost line
|
||||
verbatim (S4.0): since the road path anchors the validator to ``project.cost_items``, a generic
|
||||
reply carrying an invented magnitude for code ``01.1`` is now — correctly — rejected as a
|
||||
fabricated cost line. Anchoring the fixture is the fix; weakening the gate is not.
|
||||
|
||||
``claimed_saving_nok`` stays ``20_000`` for every project, exactly as the single generic reply
|
||||
claimed before, so every ledger/goal/budget assertion built on that figure is unchanged. Each
|
||||
project's first line is ``01.1 Rigg og drift`` at >= 480 000 NOK, so P90 (>= 144 000) clears the
|
||||
claim on every project."""
|
||||
replies: dict[str, str] = {}
|
||||
for project in load_reference_projects():
|
||||
line = project.cost_items[0]
|
||||
replies[project.id] = json.dumps(
|
||||
{
|
||||
"measure": "Reduce scope",
|
||||
"affected_items": [
|
||||
{"code": line.code, "quantity": line.quantity, "unit_cost": line.unit_cost}
|
||||
],
|
||||
"claimed_saving_nok": _DEFAULT_CLAIM,
|
||||
}
|
||||
)
|
||||
return replies
|
||||
|
||||
|
||||
class _ProjectAwareUsageChatClient(ScriptedChatClient):
|
||||
"""Selects its reply by scanning the incoming prompt for a known ``project_id`` substring (the
|
||||
prompt embeds ``project.id`` at run.py:162 and generate.py:48), falling back to a default valid
|
||||
proposal — so ``run_portfolio``'s single ``client_factory`` stays production-shaped while tests
|
||||
vary the proposal per project. A THIN subclass: the prompt-scan lives in its selector, the shared
|
||||
``_inner_get_response`` body in the canonical."""
|
||||
``_inner_get_response`` body in the canonical.
|
||||
|
||||
The fallback is itself project-aware (S4.0): a prompt naming a reference project gets that
|
||||
project's baseline-anchored default reply, so an un-mapped project still produces a proposal the
|
||||
anchored validator admits. Only a prompt naming NO known project falls through to
|
||||
``default_reply``."""
|
||||
|
||||
def __init__(
|
||||
self, replies: dict[str, str], *, default_reply: str, tokens_per_reply: int = 8
|
||||
) -> None:
|
||||
table = dict(replies)
|
||||
anchored = _anchored_default_replies()
|
||||
|
||||
def _select(blob: str, _role: str) -> str:
|
||||
return next((r for pid, r in table.items() if pid in blob), default_reply)
|
||||
explicit = next((r for pid, r in table.items() if pid in blob), None)
|
||||
if explicit is not None:
|
||||
return explicit
|
||||
return next((r for pid, r in anchored.items() if pid in blob), default_reply)
|
||||
|
||||
super().__init__(
|
||||
reply_selector=_select, default_reply=default_reply, tokens_per_reply=tokens_per_reply
|
||||
|
|
|
|||
|
|
@ -28,29 +28,34 @@ _OFFLINE_MODEL_MAP = {
|
|||
|
||||
# The synthetic reply IS the proposal: generate._parse_ir builds affected_items (each with its
|
||||
# own quantity/unit_cost) straight from this JSON, and the validator's P90 = 0.30 x Σ(qty·unit_cost)
|
||||
# ONLY when ``assumptions`` is empty (degenerate Monte Carlo, validator.py:108-113). All three
|
||||
# replies therefore OMIT ``assumptions`` and carry explicit magnitudes so each
|
||||
# ``claimed_saving_nok`` <= P90. Verified against validator + ir:
|
||||
# FV42-GSV-E1 Σ=1,482,500 P90=444,750 claimed 200,000 -> validates
|
||||
# RV13-RAS-TP Σ= 756,000 P90=226,800 claimed 130,000 -> validates (decoy)
|
||||
# BRU-LAKS-REHAB Σ=2,580,500 P90=774,150 claimed 210,000 -> validates
|
||||
# ONLY when ``assumptions`` is empty (degenerate Monte Carlo, validator.py). All three replies
|
||||
# therefore OMIT ``assumptions`` and carry explicit magnitudes so each ``claimed_saving_nok`` <= P90.
|
||||
#
|
||||
# S4.0: every line below quotes a cost line the project ACTUALLY has, verbatim from
|
||||
# reference_projects.json — the road path now anchors the validator to ``project.cost_items``, so a
|
||||
# reply quoting another project's code (which these fixtures used to do) is rejected as a fabricated
|
||||
# cost line. Verified against reference_projects.json + validator + ir:
|
||||
# FV42-GSV-E1 01.1 1x850,000 + 05.2 4300x215 Σ=1,774,500 P90=532,350 claimed 200,000
|
||||
# RV13-RAS-TP 22.4 610x3,850 Σ=2,348,500 P90=704,550 claimed 130,000 (decoy)
|
||||
# BRU-LAKS-REHAB 01.1 1x620,000 + 87.3 640x980 Σ=1,247,200 P90=374,160 claimed 210,000
|
||||
# ``measure`` is byte-identical "Reduce scope" for FV42+BRU (measure-match is exact string
|
||||
# equality, verdicts.py:68) and "Material substitution" for the decoy, so the BRU<->FV42 pair
|
||||
# overlaps (shared code 05.2 + measure + magnitude bucket) while the decoy does not.
|
||||
# equality, verdicts.py:68) and "Material substitution" for the decoy, so the BRU<->FV42 pair still
|
||||
# overlaps (shared code 01.1 + measure + magnitude bucket) while the decoy does not. 01.1 replaces
|
||||
# 05.2 as the shared code because it is the only code both projects genuinely carry.
|
||||
REPLIES = {
|
||||
"FV42-GSV-E1": (
|
||||
'{"measure":"Reduce scope","affected_items":['
|
||||
'{"code":"05.2","quantity":4300,"unit_cost":215},'
|
||||
'{"code":"03.1","quantity":1800,"unit_cost":310}],"claimed_saving_nok":200000}'
|
||||
'{"code":"01.1","quantity":1,"unit_cost":850000},'
|
||||
'{"code":"05.2","quantity":4300,"unit_cost":215}],"claimed_saving_nok":200000}'
|
||||
),
|
||||
"RV13-RAS-TP": (
|
||||
'{"measure":"Material substitution","affected_items":['
|
||||
'{"code":"88.2","quantity":180,"unit_cost":4200}],"claimed_saving_nok":130000}'
|
||||
'{"code":"22.4","quantity":610,"unit_cost":3850}],"claimed_saving_nok":130000}'
|
||||
),
|
||||
"BRU-LAKS-REHAB": (
|
||||
'{"measure":"Reduce scope","affected_items":['
|
||||
'{"code":"05.2","quantity":4300,"unit_cost":215},'
|
||||
'{"code":"07.4","quantity":2400,"unit_cost":690}],"claimed_saving_nok":210000}'
|
||||
'{"code":"01.1","quantity":1,"unit_cost":620000},'
|
||||
'{"code":"87.3","quantity":640,"unit_cost":980}],"claimed_saving_nok":210000}'
|
||||
),
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -26,6 +26,7 @@ from pathlib import Path
|
|||
|
||||
import pytest
|
||||
from agent_framework import Agent
|
||||
from test_portfolio import REPLIES
|
||||
|
||||
from portfolio_optimiser.budget import (
|
||||
Budget,
|
||||
|
|
@ -42,25 +43,11 @@ from portfolio_optimiser.run import run_portfolio
|
|||
|
||||
_PORTFOLIO_IDS = ["FV42-GSV-E1", "RV13-RAS-TP", "BRU-LAKS-REHAB"]
|
||||
|
||||
# Per-project replies (the tested constants of tests/test_portfolio.py, unchanged): all three
|
||||
# validate, and their claimed savings are DISTINCT — which is how a run is keyed back to its
|
||||
# project here, since ``RunResult`` carries no project id. 200000 + 130000 = the first two.
|
||||
REPLIES = {
|
||||
"FV42-GSV-E1": (
|
||||
'{"measure":"Reduce scope","affected_items":['
|
||||
'{"code":"05.2","quantity":4300,"unit_cost":215},'
|
||||
'{"code":"03.1","quantity":1800,"unit_cost":310}],"claimed_saving_nok":200000}'
|
||||
),
|
||||
"RV13-RAS-TP": (
|
||||
'{"measure":"Material substitution","affected_items":['
|
||||
'{"code":"88.2","quantity":180,"unit_cost":4200}],"claimed_saving_nok":130000}'
|
||||
),
|
||||
"BRU-LAKS-REHAB": (
|
||||
'{"measure":"Reduce scope","affected_items":['
|
||||
'{"code":"05.2","quantity":4300,"unit_cost":215},'
|
||||
'{"code":"07.4","quantity":2400,"unit_cost":690}],"claimed_saving_nok":210000}'
|
||||
),
|
||||
}
|
||||
# Per-project replies: IMPORTED from tests/test_portfolio.py rather than copied. The copy claimed to
|
||||
# be "the tested constants ... unchanged" and then drifted — S4.0's baseline anchoring caught it,
|
||||
# because the copies quoted cost codes belonging to OTHER projects. All three validate, and their
|
||||
# claimed savings are DISTINCT — which is how a run is keyed back to its project here, since
|
||||
# ``RunResult`` carries no project id. 200000 + 130000 = the first two.
|
||||
_FIRST_TWO_SAVING = 330000
|
||||
|
||||
# Measured: 32 tokens per run at tokens=8. 80 total funds two runs (32 + 32 = 64) and leaves 16,
|
||||
|
|
|
|||
|
|
@ -34,6 +34,8 @@ from __future__ import annotations
|
|||
import asyncio
|
||||
from typing import Any, Mapping, Sequence
|
||||
|
||||
from test_portfolio import REPLIES
|
||||
|
||||
from portfolio_optimiser.budget import PortfolioBudget, PortfolioMeter
|
||||
from portfolio_optimiser.run import RunFailure, run_portfolio
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
|
|
@ -49,22 +51,11 @@ _DEFAULT_REPLY = (
|
|||
'[{"code":"01.1","quantity":1,"unit_cost":100000}],"claimed_saving_nok":20000}'
|
||||
)
|
||||
|
||||
REPLIES = {
|
||||
"FV42-GSV-E1": (
|
||||
'{"measure":"Reduce scope","affected_items":['
|
||||
'{"code":"05.2","quantity":4300,"unit_cost":215},'
|
||||
'{"code":"03.1","quantity":1800,"unit_cost":310}],"claimed_saving_nok":200000}'
|
||||
),
|
||||
"RV13-RAS-TP": (
|
||||
'{"measure":"Material substitution","affected_items":['
|
||||
'{"code":"88.2","quantity":180,"unit_cost":4200}],"claimed_saving_nok":130000}'
|
||||
),
|
||||
"BRU-LAKS-REHAB": (
|
||||
'{"measure":"Reduce scope","affected_items":['
|
||||
'{"code":"05.2","quantity":4300,"unit_cost":215},'
|
||||
'{"code":"07.4","quantity":2400,"unit_cost":690}],"claimed_saving_nok":210000}'
|
||||
),
|
||||
}
|
||||
# ``REPLIES`` is IMPORTED from tests/test_portfolio.py (see the import above) rather than copied —
|
||||
# the local copy had drifted onto other projects' cost codes, which S4.0's baseline anchoring
|
||||
# rejects. ``_DEFAULT_REPLY`` above is reached only by a prompt naming none of the three mapped
|
||||
# projects; on the anchored road path such a reply is rejected as a fabricated cost line, which is
|
||||
# the correct outcome for a project this fixture never described.
|
||||
|
||||
# Measured in tests/test_portfolio_budget_loadbearing.py: 4 chat calls x ``tokens`` per reply, so a
|
||||
# completed run costs a flat 32 tokens at tokens=8.
|
||||
|
|
|
|||
285
tests/test_s40_cost_baseline_loadbearing.py
Normal file
285
tests/test_s40_cost_baseline_loadbearing.py
Normal file
|
|
@ -0,0 +1,285 @@
|
|||
"""S4.0 load-bearing seam: the deterministic gate is ANCHORED to the project's real cost baseline.
|
||||
|
||||
Review finding F3: every stage of ``validate_proposal`` reasoned about the numbers the *proposal
|
||||
itself* supplied, so a hallucinated cost line (an invented code, or a real code at an invented
|
||||
magnitude) could clear the whole gate as long as its own arithmetic was internally consistent. The
|
||||
reconciliation stage closes that: each ``affected_item`` must correspond to a line in the project's
|
||||
cost baseline, within a configured tolerance.
|
||||
|
||||
Every RED here is a genuine OUTCOME FLIP, not a reason-string check: the fabricated proposals are
|
||||
deliberately built to pass the P90 / nominal / method stages, so detaching the reconciliation makes
|
||||
them ``ValidatedProposal`` again. Controls prove causality (a real baseline line, same shape,
|
||||
validates), and the no-baseline arm proves the argument stays OPTIONAL (``None`` = pre-S4.0
|
||||
behaviour, which is why the existing suite stands).
|
||||
|
||||
Measured detach points (see the session log): the reconciliation stage · the magnitude tolerance ·
|
||||
the road-path wiring in ``run.py`` · the bundle-path wiring · the method-cap registry key (F8).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from conftest import SyntheticUsageChatClient
|
||||
from pydantic import ValidationError
|
||||
|
||||
from portfolio_optimiser import okf
|
||||
from portfolio_optimiser.ir import AffectedItem, CostBaseline, CostBaselineLine, SavingsProposal
|
||||
from portfolio_optimiser.reference_domain import load_reference_projects
|
||||
from portfolio_optimiser.run import run_project
|
||||
from portfolio_optimiser.validator import (
|
||||
Rejection,
|
||||
ValidatedProposal,
|
||||
baseline_from_project,
|
||||
validate_proposal,
|
||||
)
|
||||
|
||||
# The repo-local S4.0 fixture bundle: the ONLY bundle carrying a ``cost-baseline.json`` (the pre-
|
||||
# amendment bundles deliberately have none — that is the optional-argument control below).
|
||||
_DATA = Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser" / "data" / "bundles"
|
||||
BASELINE_BUNDLE = _DATA / "bygg-energi-baseline-mikro"
|
||||
PRE_AMENDMENT_BUNDLE = _DATA / "bygg-energi-mikro-a"
|
||||
|
||||
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
|
||||
|
||||
# FV42-GSV-E1's real cost line 05.2 (Asfalt Ab11): 4300 m2 x 215 NOK. Affected total 924500 ->
|
||||
# degenerate P90 = 0.30 x 924500 = 277350, so claimed 200000 clears every pre-S4.0 stage.
|
||||
_REAL_CODE = "05.2"
|
||||
_REAL_QTY = 4300.0
|
||||
_REAL_UNIT_COST = 215.0
|
||||
_REAL_CLAIM = 200000.0
|
||||
|
||||
# The F3 scenario: an invented code carrying a 10 MNOK line. The claim is set at exactly the generic
|
||||
# feasible (0.30 x 10 MNOK) so the fabrication is numerically IMPECCABLE — every pre-S4.0 stage
|
||||
# passes it. Only the baseline reconciliation can reject it, which is what makes the detach a flip.
|
||||
_FAKE_CODE = "XX"
|
||||
_FAKE_UNIT_COST = 10_000_000.0
|
||||
_FAKE_CLAIM = 3_000_000.0
|
||||
|
||||
|
||||
def _fv42_baseline() -> CostBaseline:
|
||||
return baseline_from_project(
|
||||
next(p for p in load_reference_projects() if p.id == "FV42-GSV-E1")
|
||||
)
|
||||
|
||||
|
||||
def _proposal(code: str, quantity: float, unit_cost: float, claimed: float) -> SavingsProposal:
|
||||
return SavingsProposal(
|
||||
project_id="FV42-GSV-E1",
|
||||
measure="Reduce scope",
|
||||
affected_items=[AffectedItem(code=code, quantity=quantity, unit_cost=unit_cost)],
|
||||
claimed_saving_nok=claimed,
|
||||
assumptions={},
|
||||
)
|
||||
|
||||
|
||||
# --- Arm 1: the reconciliation stage itself -------------------------------------------------------
|
||||
|
||||
|
||||
def test_fabricated_cost_code_is_rejected() -> None:
|
||||
"""RED (F3): a proposal citing a cost code that exists nowhere in the project's baseline is
|
||||
rejected, even though its own arithmetic clears the LP/P90/nominal stages. Detach the
|
||||
reconciliation stage and the SAME proposal validates."""
|
||||
result = validate_proposal(
|
||||
_proposal(_FAKE_CODE, 1.0, _FAKE_UNIT_COST, _FAKE_CLAIM), baseline=_fv42_baseline()
|
||||
)
|
||||
assert isinstance(result, Rejection), "a hallucinated cost code must never reach validated"
|
||||
assert "unknown cost code" in result.reason
|
||||
assert _FAKE_CODE in result.reason
|
||||
|
||||
|
||||
def test_real_baseline_line_still_validates() -> None:
|
||||
"""Causality control: the same shape of proposal on a REAL baseline line validates — so the
|
||||
rejection above is caused by the code being absent from the baseline, not by the new stage
|
||||
rejecting everything."""
|
||||
result = validate_proposal(
|
||||
_proposal(_REAL_CODE, _REAL_QTY, _REAL_UNIT_COST, _REAL_CLAIM), baseline=_fv42_baseline()
|
||||
)
|
||||
assert isinstance(result, ValidatedProposal)
|
||||
|
||||
|
||||
def test_baseline_is_optional_and_none_is_pre_s40_behaviour() -> None:
|
||||
"""The baseline argument is OPTIONAL: with ``None`` the fabricated proposal validates exactly as
|
||||
it did before S4.0. This is the property the existing suite rests on — and the reason the RED
|
||||
above is a flip rather than a tightening of an already-rejecting path."""
|
||||
result = validate_proposal(_proposal(_FAKE_CODE, 1.0, _FAKE_UNIT_COST, _FAKE_CLAIM))
|
||||
assert isinstance(result, ValidatedProposal)
|
||||
|
||||
|
||||
# --- Arm 2: the magnitude tolerance ---------------------------------------------------------------
|
||||
|
||||
|
||||
def test_inflated_unit_cost_on_a_real_code_is_rejected() -> None:
|
||||
"""RED: a REAL cost code at an invented unit_cost (+20%, well past the 5% default tolerance) is
|
||||
rejected. Detach the tolerance check and only the code-membership test remains — the inflated
|
||||
line then validates, because the code itself is genuine."""
|
||||
inflated = _REAL_UNIT_COST * 1.20
|
||||
result = validate_proposal(
|
||||
_proposal(_REAL_CODE, _REAL_QTY, inflated, _REAL_CLAIM), baseline=_fv42_baseline()
|
||||
)
|
||||
assert isinstance(result, Rejection)
|
||||
assert "unit_cost" in result.reason and _REAL_CODE in result.reason
|
||||
|
||||
|
||||
def test_inflated_quantity_on_a_real_code_is_rejected() -> None:
|
||||
"""RED: the same for quantity — a real code at an invented quantity (+20%) is rejected."""
|
||||
result = validate_proposal(
|
||||
_proposal(_REAL_CODE, _REAL_QTY * 1.20, _REAL_UNIT_COST, _REAL_CLAIM),
|
||||
baseline=_fv42_baseline(),
|
||||
)
|
||||
assert isinstance(result, Rejection)
|
||||
assert "quantity" in result.reason
|
||||
|
||||
|
||||
def test_within_tolerance_deviation_is_admitted() -> None:
|
||||
"""Causality control for the tolerance: a 2% deviation (rounding-scale, inside the 5% default)
|
||||
validates — the rejections above are caused by the SIZE of the deviation, not by any deviation
|
||||
at all."""
|
||||
result = validate_proposal(
|
||||
_proposal(_REAL_CODE, _REAL_QTY, _REAL_UNIT_COST * 1.02, _REAL_CLAIM),
|
||||
baseline=_fv42_baseline(),
|
||||
)
|
||||
assert isinstance(result, ValidatedProposal)
|
||||
|
||||
|
||||
def test_tolerance_is_configurable() -> None:
|
||||
"""The tolerance is config, not a constant: the same 2% deviation is rejected under a stricter
|
||||
caller-supplied tolerance."""
|
||||
result = validate_proposal(
|
||||
_proposal(_REAL_CODE, _REAL_QTY, _REAL_UNIT_COST * 1.02, _REAL_CLAIM),
|
||||
baseline=_fv42_baseline(),
|
||||
tolerance=0.001,
|
||||
)
|
||||
assert isinstance(result, Rejection)
|
||||
|
||||
|
||||
# --- Arm 3: the loader (fail-fast, mirroring ``load_ir_projection``) -------------------------------
|
||||
|
||||
|
||||
def test_bundle_baseline_loads_from_the_fixture() -> None:
|
||||
baseline = okf.load_cost_baseline(str(BASELINE_BUNDLE))
|
||||
assert baseline.project_id == "BYGG-ENERGI-BASELINE-MIKRO"
|
||||
assert baseline.items["ENERGI-TOTAL-EL"] == CostBaselineLine(quantity=180000, unit_cost=1.0)
|
||||
|
||||
|
||||
def test_missing_baseline_is_fail_fast_but_optional_loader_returns_none() -> None:
|
||||
"""Two deliberately different contracts over the same absence: the fail-fast loader raises (it
|
||||
is authoritative startup input, like ``load_ir_projection``), while the OPTIONAL loader the run
|
||||
path uses returns ``None`` — a bundle written before the amendment is not an error, it is simply
|
||||
un-anchored."""
|
||||
with pytest.raises(FileNotFoundError):
|
||||
okf.load_cost_baseline(str(PRE_AMENDMENT_BUNDLE))
|
||||
assert okf.load_optional_cost_baseline(str(PRE_AMENDMENT_BUNDLE)) is None
|
||||
|
||||
|
||||
def test_malformed_baseline_raises_even_on_the_optional_path(tmp_path) -> None:
|
||||
"""Fail-closed where it matters: a baseline that EXISTS but is malformed raises on BOTH loaders.
|
||||
Tolerating it would silently un-anchor the gate — the RAW-inbox skip rule stops at this layer."""
|
||||
(tmp_path / "cost-baseline.json").write_text(
|
||||
json.dumps({"project_id": "P", "items": {"01.1": {"quantity": 1}}}), encoding="utf-8"
|
||||
)
|
||||
with pytest.raises(ValidationError):
|
||||
okf.load_optional_cost_baseline(str(tmp_path))
|
||||
|
||||
|
||||
# --- Arm 4: the run-path wiring (road + bundle) ---------------------------------------------------
|
||||
|
||||
|
||||
def _reply(code: str, quantity: float, unit_cost: float, claimed: float) -> str:
|
||||
return json.dumps(
|
||||
{
|
||||
"measure": "Reduce scope",
|
||||
"affected_items": [{"code": code, "quantity": quantity, "unit_cost": unit_cost}],
|
||||
"claimed_saving_nok": claimed,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _factory(reply: str):
|
||||
def factory(role: str):
|
||||
return SyntheticUsageChatClient(default_reply=reply)
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
async def test_road_path_anchors_the_gate_to_the_reference_baseline(docs_dir, fresh_store) -> None:
|
||||
"""RED (road wiring): a numerically-impeccable fabricated cost line is REJECTED end-to-end
|
||||
through ``run_project``. Detach the road-path baseline (stop passing it) and the same run
|
||||
returns a ValidatedProposal."""
|
||||
result = await run_project(
|
||||
"FV42-GSV-E1",
|
||||
"local",
|
||||
docs_dir=docs_dir,
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(_reply(_FAKE_CODE, 1.0, _FAKE_UNIT_COST, _FAKE_CLAIM)),
|
||||
store=fresh_store,
|
||||
)
|
||||
assert isinstance(result.outcome, Rejection)
|
||||
assert "unknown cost code" in result.outcome.reason
|
||||
|
||||
|
||||
async def test_road_path_control_real_line_validates(docs_dir, fresh_store) -> None:
|
||||
"""Causality control for the road wiring: the real 05.2 line validates through the same path."""
|
||||
result = await run_project(
|
||||
"FV42-GSV-E1",
|
||||
"local",
|
||||
docs_dir=docs_dir,
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(_reply(_REAL_CODE, _REAL_QTY, _REAL_UNIT_COST, _REAL_CLAIM)),
|
||||
store=fresh_store,
|
||||
)
|
||||
assert isinstance(result.outcome, ValidatedProposal)
|
||||
|
||||
|
||||
async def test_bundle_path_anchors_when_the_bundle_declares_a_baseline(fresh_store) -> None:
|
||||
"""RED (bundle wiring): a bundle that ships ``cost-baseline.json`` anchors its run — the
|
||||
fabricated line is rejected. Detach the bundle-path load and it validates again."""
|
||||
result = await run_project(
|
||||
"BYGG-ENERGI-BASELINE-MIKRO",
|
||||
"local",
|
||||
docs_dir=str(BASELINE_BUNDLE),
|
||||
bundle_dir=str(BASELINE_BUNDLE),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(_reply(_FAKE_CODE, 1.0, 300000.0, 90000.0)),
|
||||
store=fresh_store,
|
||||
)
|
||||
assert isinstance(result.outcome, Rejection)
|
||||
assert "unknown cost code" in result.outcome.reason
|
||||
|
||||
|
||||
async def test_pre_amendment_bundle_runs_unchanged(fresh_store) -> None:
|
||||
"""Control + backward compatibility: the SAME fabricated reply validates against a bundle with
|
||||
no ``cost-baseline.json``. Anchoring is opt-in per bundle, so every pre-S4.0 bundle (including
|
||||
the commons-owned goldens) runs byte-identically to before."""
|
||||
result = await run_project(
|
||||
"BYGG-ENERGI-MIKRO-A",
|
||||
"local",
|
||||
docs_dir=str(PRE_AMENDMENT_BUNDLE),
|
||||
bundle_dir=str(PRE_AMENDMENT_BUNDLE),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(_reply(_FAKE_CODE, 1.0, 300000.0, 90000.0)),
|
||||
store=fresh_store,
|
||||
)
|
||||
assert isinstance(result.outcome, ValidatedProposal)
|
||||
|
||||
|
||||
# --- Arm 5: F8 — method caps keyed by config, not the literal measure string ----------------------
|
||||
|
||||
|
||||
def test_method_cap_is_keyed_by_config_not_a_hardcoded_string() -> None:
|
||||
"""F8: the method-specific cap comes from a REGISTRY the caller can supply. A caller-configured
|
||||
cap for a measure with no built-in entry rejects a proposal the generic P90 stage passes — so
|
||||
the rule is keyed by configuration, not by the ``energy_efficiency`` literal."""
|
||||
proposal = SavingsProposal(
|
||||
project_id="P-ASFALT",
|
||||
measure="asfalt_reduction",
|
||||
affected_items=[AffectedItem(code="05.2", quantity=100000, unit_cost=1.0)],
|
||||
claimed_saving_nok=20000,
|
||||
assumptions={},
|
||||
)
|
||||
assert isinstance(validate_proposal(proposal), ValidatedProposal) # generic P90 = 30000
|
||||
capped = validate_proposal(proposal, method_caps={"asfalt_reduction": 0.10})
|
||||
assert isinstance(capped, Rejection)
|
||||
assert "method cap" in capped.reason
|
||||
Loading…
Add table
Add a link
Reference in a new issue