feat(validator): anchor the deterministic gate to the project's real cost baseline (S4.0)

Every stage of validate_proposal reasoned only about numbers the proposal itself
supplied, so an internally-consistent hallucination cleared the whole gate (F3).
A new stage 0 reconciles each affected_item against the project's CostBaseline
before the CBC solve: an unknown cost code is rejected, and a real code carrying
a quantity/unit_cost outside the configured tolerance (5% default, relative to
the baseline value) is rejected. Validation, never repair.

The baseline argument is OPTIONAL (None = pre-S4.0 behaviour), but both run
paths set it: the road path projects project.cost_items, the bundle path loads
cost-baseline.json when the bundle ships one. Bundles written before the
amendment stay un-anchored, so the commons-owned goldens run byte-identically;
a baseline that exists but is malformed still raises on both loaders.

F8: the method-specific cap now comes from the METHOD_CAPS registry (measure
type -> fraction, injectable) instead of an energy_efficiency string comparison.

The baseline format and tolerance semantics were decided locally — the commons
amendment (D-A pt. 2) never arrived, exactly as in S3.2. D7 mirroring stays open.

Three portfolio fixtures quoted cost codes belonging to OTHER projects; the new
gate caught them. They now quote each project's own lines, and the two copied
REPLIES tables import the single source instead of drifting from it.

Load-bearing measured (tests/test_s40_cost_baseline_loadbearing.py), six
mutations all red: detach the reconciliation stage; detach the magnitude
tolerance; detach the road wiring; detach the bundle wiring; ignore the injected
cap registry; make the optional loader tolerant of malformed content. Control:
with the road wiring detached the repaired portfolio fixtures still pass, so
they are not masking the seam. 597 -> 612 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JdwK7bQ4BZkWH4t8MRDKb4
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 17:19:31 +02:00
commit 126807aee7
16 changed files with 645 additions and 69 deletions

View file

@ -0,0 +1,20 @@
---
type: project
title: "Kontorbygg B — energibaseline med erklærte kostlinjer (repo-lokal fixture)"
description: "Syntetisk energibaseline for S4.0-fixturen. Ett kontorbygg, én kandidat-kostpost (ENERGI-TOTAL-EL), og en eksplisitt cost-baseline.json den deterministiske validatoren avstemmer mot."
timestamp: 2026-08-03
---
# Kontorbygg B (repo-lokal test-fixture)
Et lite syntetisk kontorbygg med én dokumentert energibaseline. Til forskjell fra mikro-A erklærer
denne bundelen kostlinjene sine maskinlesbart i `cost-baseline.json`, slik at et forslag som siterer
en oppdiktet kostkode — eller en ekte kode med oppdiktet størrelse — blir avvist av validatorens
avstemmings-stage før løseren i det hele tatt kjøres.
## Energibaseline
- Årlig elektrisk energikostnad (`ENERGI-TOTAL-EL`): 180 000 NOK/år (syntetisk), ført som
`quantity: 180000` × `unit_cost: 1.0` i `cost-baseline.json`.
- Kandidat-tiltak: LED-retrofit av kontorbelysning.
- Modellert besparelse: ~18 000 NOK/år (innenfor validatorens feasibelt-område).

View file

@ -0,0 +1,10 @@
{
"_note": "SYNTHETIC repo-lokal kostbaseline (S4.0-fixture) — ikke ekte data. Prosjektets FAKTISKE kostlinjer: den deterministiske validatoren avstemmer et forslags affected_items mot disse (okf.load_cost_baseline -> validator._reconcile_against_baseline). Formatet er bestemt LOKALT (commons-amendmentet D-A pkt. 2 kom aldri); D7-speiling er ÅPEN.",
"project_id": "BYGG-ENERGI-BASELINE-MIKRO",
"items": {
"ENERGI-TOTAL-EL": {
"quantity": 180000,
"unit_cost": 1.0
}
}
}

View file

@ -0,0 +1,26 @@
---
type: index
okf_version: 0.1
title: "Bygg-energi baseline-mikro — repo-lokal fixture (S4.0 kostbaseline-forankring)"
description: "Minimal repo-lokal OKF-bundle som SHIPPER en cost-baseline.json, så bundle-stiens validator-forankring (S4.0, F3) kan testes ende-til-ende. Speiler bygg-energi-mikro-A, men med kostbaselinen lagt til."
tags: [fixture, S4.0, kostbaseline]
timestamp: 2026-08-03
---
# Bygg-energi baseline-mikro (repo-lokal test-fixture)
En **repo-lokal mini OKF-bundle** under pakkens `data/` (ALDRI `shared/` — subtree er PULL-ONLY).
Eneste formål: være den ene bundelen som **erklærer sin kostbaseline**, slik at
`run_project(bundle_dir=...)` faktisk forankrer den deterministiske gaten mot prosjektets virkelige
kostlinjer. De eldre bundlene bærer bevisst INGEN `cost-baseline.json` — de er kontrollen som viser
at forankringen er opt-in per bundle (pre-amendment-bundler kjører uendret).
## Innhold (progressiv disclosure)
- [bygg-kontor-baseline.md](bygg-kontor-baseline.md) — `type: project` — bygget og energibaselinen
som lese-kontekst (og `bundle_citations`-kilde).
`validator-input.json` er IR-projeksjonen (kandidatens kost-IR), og `cost-baseline.json` er
prosjektets FAKTISKE kostlinjer (`code -> {quantity, unit_cost}`) som `okf.load_cost_baseline` leser
og validatoren avstemmer `affected_items` mot. De to er bevisst adskilt: IR-projeksjonen er hva
noen *foreslår*, baselinen er hva prosjektet *er*.

View file

@ -0,0 +1,13 @@
{
"_note": "SYNTHETIC repo-lokal IR-projeksjon (S4.0-fixture) — ikke ekte data. Kandidatens kost-IR; project_id matcher det bundle-backede prosjektets id (run._project_from_bundle fail-faster ved mismatch). Kostlinjene her SPEILER cost-baseline.json med vilje: et forslag om denne kandidaten skal avstemme rent.",
"project_id": "BYGG-ENERGI-BASELINE-MIKRO",
"measure": "LED-retrofit av kontorbelysning",
"affected_items": [
{
"code": "ENERGI-TOTAL-EL",
"quantity": 180000,
"unit_cost": 1.0
}
],
"claimed_saving_nok": 18000
}

View file

@ -28,7 +28,7 @@ from agent_framework import BaseChatClient, Message
from pydantic import ValidationError
from portfolio_optimiser.budget import TokenMeter
from portfolio_optimiser.ir import SavingsProposal
from portfolio_optimiser.ir import CostBaseline, SavingsProposal
from portfolio_optimiser.reference_domain import Project
from portfolio_optimiser.validator import (
Rejection,
@ -110,6 +110,7 @@ async def generate_via_llm(
meter: TokenMeter,
*,
max_attempts: int = 3,
baseline: CostBaseline | None = None,
) -> ValidatedProposal | Rejection:
"""Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds,
the meter checked in this loop:
@ -123,7 +124,12 @@ async def generate_via_llm(
The only per-attempt falsifier here is the deterministic validator (the numbers). The
checker is a run-level, one-shot signal (run.py, before generation); seeding generation
with the checker critique is separately scoped and NOT done here. Returns
with the checker critique is separately scoped and NOT done here.
``baseline`` (S4.0) is handed straight to ``validate_proposal``, so a fabricated cost line is
falsified per ATTEMPT like any other rejection and its reason feeds the next attempt's prompt
through the SAME informed-refinement path (Step 5), which is why no new loop appears here.
Returns
``ValidatedProposal | Rejection``; never a malformed proposal; raises ``BudgetExceeded``
when the meter cap is crossed."""
@ -146,7 +152,7 @@ async def generate_via_llm(
# accumulated history (bounded prompt growth).
messages = _build_messages(project, context, prior_rejection=last)
candidate = await _fetch_parsed(messages)
result = validate_proposal(candidate)
result = validate_proposal(candidate, baseline=baseline)
if isinstance(result, ValidatedProposal):
return result
last = result

View file

@ -24,6 +24,31 @@ class AffectedItem(BaseModel):
return self.quantity * self.unit_cost
class CostBaselineLine(BaseModel):
"""One line of a project's ACTUAL cost baseline: the quantity and unit cost a proposal's
``AffectedItem`` for that code must reconcile against (S4.0, F3)."""
quantity: float = Field(ge=0)
unit_cost: float = Field(gt=0)
class CostBaseline(BaseModel):
"""A project's cost baseline, keyed by cost code — the ground truth the deterministic
validator anchors ``affected_items`` to, so the gate cannot be fed hallucinated cost lines.
Deliberately a typed IR contract (not a loader-private shape): both sources project INTO
it an OKF bundle's ``cost-baseline.json`` (``okf.load_cost_baseline``) and the road
reference domain's ``cost_items`` (``validator.baseline_from_project``) — so the validator
sees ONE representation regardless of path, and the Claude-SDK sibling can mirror it (D7).
The projection/tolerance semantics were decided HERE: the commons amendment specifying
``cost-baseline.json`` never arrived, exactly as in S3.2. D7 mirroring stays OPEN.
"""
project_id: str
items: dict[str, CostBaselineLine]
class SavingsProposal(BaseModel):
"""Typed IR for a candidate cost-saving measure (B1)."""

View file

@ -3,7 +3,8 @@
Reads a bundle the way OKF intends (progressive disclosure): start at ``index.md``, follow
intra-bundle cross-links **recursively, depth-first in first-seen link order**, parse each file's
YAML frontmatter, classify by the one required ``type`` field. **NO** ``agent_framework``, **NO**
``mcp`` pure stdlib, so the SAME navigation serves both the MAF and the Claude-SDK
``mcp`` stdlib + ``pydantic`` only (as ``dimension.py``; the typed contracts this module loads
live in ``ir.py``), so the SAME navigation serves both the MAF and the Claude-SDK
implementations unchanged (målbilde §4 vendor-neutrality).
Link resolution follows ``shared/method-spec.md`` §3 Step 1: a leading ``/`` denotes the **bundle
@ -29,10 +30,12 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Any
from portfolio_optimiser.ir import CostBaseline
from portfolio_optimiser.retrieval import PathSecurityError, safe_resolve
_INDEX_NAME = "index.md"
_IR_PROJECTION = "validator-input.json"
_COST_BASELINE = "cost-baseline.json"
# Intra-bundle markdown cross-links: ``](target.md)``. A path separator is NOT a rejection reason —
# ``_resolve_target`` decides in-/out-of-bundle, and only escape is refused (method-spec §3 Step 1).
_LINK_RE = re.compile(r"\]\(([^)]+\.md)\)")
@ -285,6 +288,39 @@ def link_in_index(bundle_dir: str, target_name: str, label: str) -> bool:
return True
def load_cost_baseline(bundle_dir: str, name: str = _COST_BASELINE) -> CostBaseline:
"""Load the bundle's cost baseline (``cost-baseline.json`` by default): the project's ACTUAL
cost lines (``{code: {quantity, unit_cost}}``), which the deterministic validator reconciles a
proposal's ``affected_items`` against (S4.0, F3).
Fail-fast, mirroring ``load_ir_projection`` and ``dimension.load_dimension``: a missing file
raises ``FileNotFoundError`` and malformed content raises ``pydantic.ValidationError``. A cost
baseline is authoritative gate input a tolerantly-degraded one would silently un-anchor the
gate, which is precisely the failure this stage exists to prevent. (The tolerant skip rule
belongs to the RAW verdict-inbox layer, never here.)
Use ``load_optional_cost_baseline`` where the ABSENCE of the file is legitimate."""
resolved = Path(safe_resolve(bundle_dir, name))
if not resolved.is_file():
raise FileNotFoundError(f"cost baseline not found in bundle: {name!r}")
return CostBaseline.model_validate_json(resolved.read_text(encoding="utf-8"))
def load_optional_cost_baseline(bundle_dir: str, name: str = _COST_BASELINE) -> CostBaseline | None:
"""``load_cost_baseline`` where a MISSING file is legitimate: returns ``None`` instead of
raising. This is the run path's loader — a bundle authored before the baseline amendment is
simply un-anchored (``None`` = pre-S4.0 behaviour), not an error, which is what keeps every
existing bundle (including the commons-owned goldens) running byte-identically.
The tolerance stops at absence: a baseline that EXISTS but is malformed still raises. Reading a
corrupt baseline as "no baseline" would hand back an un-anchored gate under the appearance of an
anchored one (the same reasoning as ``budget.read_spend``)."""
try:
return load_cost_baseline(bundle_dir, name)
except FileNotFoundError:
return None
def load_ir_projection(bundle_dir: str, name: str = _IR_PROJECTION) -> dict[str, Any]:
"""Load the bundle's IR projection (``validator-input.json`` by default): the candidate
measure's cost-IR (``measure``, ``affected_items``, ``claimed_saving_nok``) — the

View file

@ -55,7 +55,7 @@ from portfolio_optimiser.generate import generate_via_llm
from portfolio_optimiser.ir import SavingsProposal
from portfolio_optimiser.provenance import ProvenanceStamp
from portfolio_optimiser.reference_domain import Project, load_reference_projects
from portfolio_optimiser.validator import Rejection, ValidatedProposal
from portfolio_optimiser.validator import Rejection, ValidatedProposal, baseline_from_project
from portfolio_optimiser import okf, outbox
from portfolio_optimiser.semretrieval import (
SEMANTIC_WEIGHT_DEFAULT,
@ -373,9 +373,16 @@ async def run_project(
# NOT keyword chunk-stuffing; the road path keeps the chunk-retrieval data source. ``debate_tools``
# is the query-time retrieval surface — empty on the bundle path (navigation already placed the
# curated context in the prompt, and a docs_dir==bundle_dir tool would re-leak the verdict layer).
# S4.0 (F3): the run path SETS the validator's cost baseline, so the deterministic gate is
# anchored to the project's real cost lines instead of the ones the proposal asserts.
# * road path: the reference project's own ``cost_items`` ARE the baseline -> always anchored.
# * bundle path: anchored only when the bundle SHIPS a ``cost-baseline.json``. A bundle written
# before the amendment (every commons-owned golden) is legitimately un-anchored -> None =
# pre-S4.0 behaviour. A baseline that exists but is malformed still raises (fail-closed).
if bundle_dir is not None:
bundle = okf.navigate_bundle(bundle_dir)
project = _project_from_bundle(bundle_dir, project_id, bundle=bundle)
baseline = okf.load_optional_cost_baseline(bundle_dir)
# §4.1a context-scope: agents read ONLY dimension-scoped bundle knowledge (Step-3 filter);
# dimension=None keeps the full context, byte-identical to before.
context = okf.bundle_context(bundle, dimension=dimension.id if dimension else None)
@ -383,6 +390,7 @@ async def run_project(
debate_tools: list[Any] = []
else:
project = _project_by_id(project_id)
baseline = baseline_from_project(project)
chunks = retrieve_chunks("cost saving measure", docs_dir, top_k)
citations = [chunk_dict_to_citation(c) for c in chunks]
context = "\n".join(c["snippet"] for c in chunks)
@ -480,7 +488,9 @@ async def run_project(
# 5. Structured candidate -> blocking validation on the NUMBERS; token bound = the meter.
proposer_client = factory("proposer")
validator_outcome = await generate_via_llm(proposer_client, project, gen_context, meter)
validator_outcome = await generate_via_llm(
proposer_client, project, gen_context, meter, baseline=baseline
)
proposal = validator_outcome.proposal
# 6. First-class provenance stamp (authoritative; independent of MAF Annotation).

View file

@ -24,13 +24,13 @@ from __future__ import annotations
import random
import statistics
import warnings
from collections.abc import Callable
from collections.abc import Callable, Mapping
from contextlib import contextmanager
from dataclasses import dataclass
import pulp
from portfolio_optimiser.ir import AffectedItem, SavingsProposal
from portfolio_optimiser.ir import AffectedItem, CostBaseline, CostBaselineLine, SavingsProposal
from portfolio_optimiser.reference_domain import Project
MAX_SAVING_FRACTION = 0.30
@ -46,6 +46,21 @@ INDEPENDENT gate: it can reject a proposal the generic P90 stage passes (not red
fraction is calibrated against the reference domain; the CONDITION (a method-scoped stricter cap) is
the encoded rule. Returns the same ``Rejection`` type a validator stage, not a new gate."""
METHOD_CAPS: dict[str, float] = {_ENERGY_METHOD_MEASURE: _ENERGY_METHOD_MAX_FRACTION}
"""S4.0 (F8): the method-cap REGISTRY — measure type -> method-scoped max saving fraction. The
rule used to be an ``if proposal.measure == "energy_efficiency"`` branch, so encoding a second
assessment method meant editing the validator. It is now data: a caller passes its own registry
(``validate_proposal(..., method_caps=...)``), keyed by the measure type a dimension admits
(``dimension.allowed_measure_types``), and the built-in entry stays the default so the Step-9
behaviour is unchanged. Deliberately NOT a config file yet the deliverable is the key-by-config
seam (90%-prinsippet), not a settings format."""
BASELINE_TOLERANCE_DEFAULT = 0.05
"""S4.0: the relative deviation a reconciled ``AffectedItem`` may show against its cost-baseline
line (5%). A tolerance is needed at all because a proposer restates magnitudes in prose-derived,
rounded form; it is small because its whole purpose is to leave no room for a FABRICATED magnitude.
Config, not policy: every caller can tighten or loosen it per run (``tolerance=``)."""
_MC_SAMPLES = 512
_MC_SEED = 20260624
@ -122,9 +137,83 @@ def _monte_carlo(
return deciles[0], deciles[4], deciles[8] # P10, P50, P90
def validate_proposal(proposal: SavingsProposal) -> ValidatedProposal | Rejection:
def baseline_from_project(project: Project) -> CostBaseline:
"""Project a road reference-domain ``Project``'s ``cost_items`` into the ``CostBaseline``
contract the road-path counterpart of ``okf.load_cost_baseline`` (S4.0). The road path always
HAS its baseline (the estimate is the project), so this projection is total: no optional
variant, and a run on this path is always anchored."""
return CostBaseline(
project_id=project.id,
items={
ci.code: CostBaselineLine(quantity=ci.quantity, unit_cost=ci.unit_cost)
for ci in project.cost_items
},
)
def _reconcile_against_baseline(
proposal: SavingsProposal, baseline: CostBaseline, tolerance: float
) -> Rejection | None:
"""S4.0 (F3): every affected item must correspond to a REAL line of the project's cost baseline.
Two independent failures, both fail-closed:
* the cost code is absent from the baseline a fabricated line;
* the code is real but its ``quantity``/``unit_cost`` deviates from the baseline line by more
than ``tolerance`` (relative to the BASELINE value, which is the ground truth) a real code
carrying a fabricated magnitude.
Returns the first ``Rejection`` (validator's own type — never a new gate), or ``None`` when the
proposal reconciles. Items are checked in their stated order so the reason is deterministic.
A validation, never a repair: the proposal is rejected, not silently corrected to the baseline."""
for item in proposal.affected_items:
line = baseline.items.get(item.code)
if line is None:
return Rejection(
proposal=proposal,
reason=(
f"unknown cost code {item.code!r}: not in project {baseline.project_id}'s "
f"cost baseline ({len(baseline.items)} known codes)"
),
)
for field, claimed, actual in (
("quantity", item.quantity, line.quantity),
("unit_cost", item.unit_cost, line.unit_cost),
):
if abs(claimed - actual) > tolerance * actual:
return Rejection(
proposal=proposal,
reason=(
f"{field} {claimed:g} for cost code {item.code!r} is outside the "
f"{tolerance:.1%} tolerance around the baseline {field} {actual:g}"
),
)
return None
def validate_proposal(
proposal: SavingsProposal,
*,
baseline: CostBaseline | None = None,
tolerance: float = BASELINE_TOLERANCE_DEFAULT,
method_caps: Mapping[str, float] | None = None,
) -> ValidatedProposal | Rejection:
"""Deterministic blocking validation. Returns a ``ValidatedProposal`` only when the
claim is feasible; otherwise a ``Rejection`` that cannot be consumed as validated."""
claim is feasible; otherwise a ``Rejection`` that cannot be consumed as validated.
``baseline`` (S4.0, F3) anchors the gate to the project's ACTUAL cost lines: without it every
stage reasons only about numbers the proposal supplied itself, so an internally-consistent
hallucination clears the gate. It is OPTIONAL ``None`` is exactly the pre-S4.0 behaviour, so a
caller with no baseline (a bundle authored before the amendment) is unchanged but both run
paths SET it. ``tolerance`` is the reconciliation's config knob; ``method_caps`` overrides the
built-in method-cap registry (F8)."""
# Stage 0 (S4.0): reconcile against the cost baseline BEFORE the solver. It is the cheapest
# stage and the only one that can tell a fabricated line from a real one — spending a CBC solve
# on numbers that do not belong to the project is work on a claim that cannot be validated.
if baseline is not None:
blocked = _reconcile_against_baseline(proposal, baseline, tolerance)
if blocked is not None:
return blocked
# Stage 1 (Pydantic) already ran at construction. Stage 2: real CBC solve.
nominal = _solve_max_feasible(proposal.affected_items, MAX_SAVING_FRACTION)
# Stage 3: Monte Carlo percentiles of the feasible saving.
@ -152,15 +241,18 @@ def validate_proposal(proposal: SavingsProposal) -> ValidatedProposal | Rejectio
# Stage 5 (Step 9, SC7-B): a method-specific rule STRICTER than the generic cap. A proposal in
# the energy method (IPMVP Option A) must clear a lower, method-scoped feasible — an INDEPENDENT
# gate that can reject a proposal the P90 stage passed. Same ``Rejection`` type, not a new gate.
if proposal.measure == _ENERGY_METHOD_MEASURE:
method_feasible = _ENERGY_METHOD_MAX_FRACTION * sum(
it.total for it in proposal.affected_items
)
# F8 (S4.0): the cap is looked up in a REGISTRY keyed by measure type (config), not compared
# against the ``energy_efficiency`` literal — a second assessment method is now data, not an
# edit to this function. The built-in registry keeps the Step-9 behaviour identical.
caps = METHOD_CAPS if method_caps is None else method_caps
method_fraction = caps.get(proposal.measure)
if method_fraction is not None:
method_feasible = method_fraction * sum(it.total for it in proposal.affected_items)
if proposal.claimed_saving_nok > method_feasible:
return Rejection(
proposal=proposal,
reason=(
f"claimed {proposal.claimed_saving_nok:.0f} exceeds the {_ENERGY_METHOD_MEASURE} "
f"claimed {proposal.claimed_saving_nok:.0f} exceeds the {proposal.measure} "
f"method cap {method_feasible:.0f} (stricter than the generic P90)"
),
)