feat(verdicts): key each verdict on its own candidate, not the bundle's one IR projection (S3.2)
seed_store_from_bundle keyed EVERY `type: verdict` file on bundle_candidate_features — the single candidate the bundle's validator-input.json describes. A bundle carrying verdicts about several candidates collapsed them onto one key, so a verdict about candidate B scored a perfect structural match against candidate A's query and could be folded into A's hypothesis prompt. The ExpeL substrate was single-candidate by construction. A verdict file may now carry its own structural key in frontmatter (affected_codes / measure_type / claimed_saving_nok); absent, keying falls back to the bundle candidate, so every pre-S3.2 seed keeps working unchanged. promote_verdict writes the three fields, so a promoted verdict — frequently about a different candidate than the target bundle's projection — does not impersonate that candidate. Semantics decided HERE, not pulled: commons' seeding rule (method-spec §3 Steg 1 + bundle example) has not arrived; we said we would build locally first. D7 mirroring stays open. - ALL THREE fields or none. A partial declaration raises VerdictFrontmatterError rather than merging with the bundle candidate, which would mint a key belonging to NEITHER candidate. Validation, never repair (mirrors write_concept_file); the tolerant-skip rule belongs to the RAW inbox layer. - claimed_saving_nok parses via json.loads — the SAME literal rule the IR projection went through — and is written back with str() of the raw value. _mint_id hashes that value, so 30000 and 30000.0 are different keys; a normalising writer would split one candidate's signal across two ids. - The structural key is signal-free, so it does not weaken the Step-8 no-leak property (Test C green). Load-bearing MEASURED, five mutations all red: detach per-verdict keying · detach the fields promote_verdict writes · make a partial/unparseable key tolerant · normalise the magnitude on write · remove the fallback (control — breaks the step1 suite at collection, proving the fallback bears load). 589 -> 597 tests. Full gate green (pytest, ruff, mypy). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QkjvTTxrg9LTrmghebfiij
This commit is contained in:
parent
e8cec2e2c0
commit
012adc0a3c
10 changed files with 496 additions and 17 deletions
19
CLAUDE.md
19
CLAUDE.md
|
|
@ -105,6 +105,25 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
(`tests/test_step8_promotion_loadbearing.py`): gaten avviser ikke-godkjent dom (RØD uten gate); godkjent
|
||||
dom er navigerbar (RØD når `link_in_index` detaches); promotert signal holdes ute av `bundle_context`
|
||||
(RØD når en beskrivende index-label lekker det inn). Index-RMW er ikke-atomisk (enprosess-MVP).
|
||||
- **En dom nøkles på SIN kandidat, ikke bundelens ene IR-projeksjon (S3.2):** `seed_store_from_bundle`
|
||||
leser hver `type: verdict`-fils EGNE strukturelle felt fra frontmatter (`affected_codes` /
|
||||
`measure_type` / `claimed_saving_nok`); mangler de, faller nøklingen tilbake til
|
||||
`bundle_candidate_features` — så hver pre-S3.2-seed står uendret (fallbacken er BÆRENDE: fjernes
|
||||
den, brekker step1-suiten ved collection). Uten dette kollapset en bundle med dommer om flere
|
||||
kandidater dem på ÉN nøkkel, og en dom om kandidat B scoret perfekt strukturell match mot
|
||||
kandidat As query. **ALLE TRE felt eller ingen:** en delvis erklæring raiser
|
||||
`VerdictFrontmatterError` i stedet for å slås sammen med bundle-kandidaten — sammenslåingen ville
|
||||
myntet en nøkkel som tilhører INGEN av kandidatene. Validering, ALDRI reparasjon (speiler
|
||||
`write_concept_file`); den tolerante hopp-over-regelen hører til RAW-innboks-laget.
|
||||
`claimed_saving_nok` parses med `json.loads` — SAMME literal-regel IR-projeksjonen gikk gjennom —
|
||||
og skrives tilbake som `str()` av råverdien, fordi `_mint_id` hasher den (`30000` ≠ `30000.0`; en
|
||||
normaliserende skriver ville splittet én kandidats signal på to id-er). `promote_verdict` skriver
|
||||
de tre feltene, så en promotert dom ikke utgir seg for målbundelens kandidat; nøkkelen er
|
||||
signal-fri, så Steg-8s no-leak-egenskap står. **Semantikken er bestemt HER** — commons' seeding-regel
|
||||
(`method-spec` §3 Steg 1) hadde ikke kommet; D7-speiling forblir ÅPEN. Load-bearing MÅLT
|
||||
(`tests/test_step32_multicandidate_loadbearing.py` + `test_step8_promotion_loadbearing.py`), fem
|
||||
mutasjoner alle røde: detach per-dom-nøklingen · detach feltene `promote_verdict` skriver · gjør en
|
||||
delvis/uparsebar nøkkel tolerant · normaliser magnituden ved skriving · fjern fallbacken (kontroll).
|
||||
- **Globalt token-tak håndheves FØR kall, aldri bare etterpå (S3.4, F10):** `PortfolioBudget` +
|
||||
`PortfolioMeter` er ÉN ledger over hele porteføljepasset (og — seedet av `read_spend` — på tvers
|
||||
av pass), mens per-run `Budget`/`TokenMeter` er uendret. Taket har tre tenner, med hver sin jobb:
|
||||
|
|
|
|||
|
|
@ -141,6 +141,14 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
|||
silently ignoring the flag. Nothing about a flag-off run changes, and no savings claim depends
|
||||
on it.
|
||||
|
||||
A bundle may hold verdicts about **several candidates**, while its `validator-input.json`
|
||||
describes only one. A `type: verdict` file therefore may declare its own retrieval key in
|
||||
frontmatter — `affected_codes`, `measure_type`, `claimed_saving_nok` — and is keyed on that;
|
||||
omit them and it falls back to the bundle's candidate, exactly as before. The three are **all or
|
||||
nothing**: a partial declaration is refused rather than merged with the bundle candidate, since
|
||||
the merge would produce a key belonging to neither. `promote_verdict` writes all three, so a
|
||||
promoted verdict about one candidate never surfaces for another.
|
||||
|
||||
The **prior-verdict fold — the learning step — happens only on the `--bundle-dir` path**; a
|
||||
plain `--docs-dir`-only run is single-shot (no fold). `--decision`/`--rationale` apply to the
|
||||
single-project path only and are inert in portfolio mode. **`--outbox-dir` must differ from
|
||||
|
|
|
|||
|
|
@ -0,0 +1,34 @@
|
|||
---
|
||||
type: index
|
||||
okf_version: 0.1
|
||||
title: "Multi-kandidat mikro — repo-lokal fixture (S3.2 per-dom-noekling)"
|
||||
description: "Minimal repo-lokal OKF-bundle under data/ med dommer om TO ULIKE kandidater, mens IR-projeksjonen beskriver kandidat A alene. Fixturen gjoer forskjellen mellom per-dom-noekling og bundle-noekling maalbar."
|
||||
tags: [fixture, S3.2, multi-kandidat]
|
||||
timestamp: 2026-08-03
|
||||
---
|
||||
|
||||
# Multi-kandidat mikro (repo-lokal test-fixture)
|
||||
|
||||
En **repo-lokal mini OKF-bundle** under pakkens `data/` (ALDRI `shared/` — subtree er PULL-ONLY).
|
||||
Den finnes for ett formaal: bundelen bærer dommer om **to ulike kandidater**, mens
|
||||
`validator-input.json` (ExpeL-query-noekkelen) beskriver **kandidat A alene**.
|
||||
|
||||
Foer S3.2 noeklet `seed_store_from_bundle` HVER dom paa bundelens ene IR-projeksjon, saa dommen om
|
||||
kandidat B ble uskillelig fra dommen om kandidat A og kunne naa kandidat As hypotese-prompt. Med
|
||||
per-dom-noekling leser hver dom sine EGNE strukturelle felt fra frontmatter.
|
||||
|
||||
Rekkefoelgen under er bevisst: **kandidat B er lenket foerst**, saa en detachet per-dom-noekling
|
||||
(begge dommer faar kandidat As noekkel → uavgjort likhet, uavgjort id) rangerer B-dommen oeverst.
|
||||
|
||||
## Innhold (progressiv disclosure)
|
||||
|
||||
- [verdict-b-asfalt.md](verdict-b-asfalt.md) — `type: verdict` — dom om **kandidat B**
|
||||
(asfalttykkelse, kode `05.2`). Bærer sine egne `affected_codes`/`measure_type`/
|
||||
`claimed_saving_nok`. Skal ALDRI naa kandidat As hypotese-prompt.
|
||||
- [verdict-a-led.md](verdict-a-led.md) — `type: verdict` — dom om **kandidat A**
|
||||
(LED-retrofit, kode `ENERGI-TOTAL-EL`). Dette er dommen kandidat As query skal hente.
|
||||
- [prosjekt-multi.md](prosjekt-multi.md) — `type: project` — bygget/strekningen begge kandidatene
|
||||
hoerer til. Concept-filen OKF-navigasjonen leser inn som lese-kontekst.
|
||||
|
||||
`validator-input.json` er IR-projeksjonen den deterministiske validatoren konsumerer, og kilden for
|
||||
ExpeL-query-noekkelen (`bundle_candidate_features`) — kandidat A.
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
---
|
||||
type: project
|
||||
title: "Kombinert bygg + veistrekning (syntetisk fixture-prosjekt)"
|
||||
description: "Syntetisk prosjekt som baerer to uavhengige kandidat-tiltak — ett energitiltak i kontorfloy A og ett dekketiltak paa veistrekning B."
|
||||
resource: MULTI-KANDIDAT-MIKRO
|
||||
tags: [fixture, project]
|
||||
timestamp: 2026-08-03
|
||||
---
|
||||
|
||||
# Kombinert bygg + veistrekning
|
||||
|
||||
Syntetisk fixture-prosjekt. To uavhengige tiltak er identifisert:
|
||||
|
||||
- **Kandidat A — LED-retrofit kontorfloy A.** Beroerer kostkode `ENERGI-TOTAL-EL`.
|
||||
Modellert besparelse 18 000 NOK/aar.
|
||||
- **Kandidat B — redusert asfalttykkelse veistrekning B.** Beroerer kostkode `05.2`.
|
||||
Modellert besparelse 900 000 NOK.
|
||||
|
||||
Kandidatene deler verken kostkode, tiltakstype eller stoerrelsesorden. Det er med vilje: en dom om
|
||||
den ene sier ingenting om den andre, og skal derfor ikke hentes for den andre.
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"_note": "SYNTHETIC repo-local multi-candidate mini-bundle IR-projeksjon (Fase 2a S3.2-fixture) — ikke ekte data. Bundelen bærer dommer om TO kandidater, men IR-projeksjonen (og dermed bundle_candidate_features) er kandidat A alene — nettopp asymmetrien per-verdict-noekling maa haandtere.",
|
||||
"project_id": "MULTI-KANDIDAT-MIKRO",
|
||||
"measure": "LED-retrofit kontorfloy A",
|
||||
"affected_items": [
|
||||
{
|
||||
"code": "ENERGI-TOTAL-EL",
|
||||
"quantity": 180000,
|
||||
"unit_cost": 1.0
|
||||
}
|
||||
],
|
||||
"claimed_saving_nok": 18000,
|
||||
"assumptions": {
|
||||
"ENERGI-TOTAL-EL": [0.70, 1.40]
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
type: verdict
|
||||
title: "Ekspert-dom (froe) — LED-retrofit kontorfloy A"
|
||||
description: "Froesatt realiseringskorreksjon for KANDIDAT A (energitiltak). Dette er dommen kandidat As hypotese-prompt skal hente."
|
||||
resource: MULTI-KANDIDAT-MIKRO
|
||||
measure_id: LED-RETROFIT-A
|
||||
decision: approved
|
||||
affected_codes: [ENERGI-TOTAL-EL]
|
||||
measure_type: "LED-retrofit kontorfloy A"
|
||||
claimed_saving_nok: 18000
|
||||
realization_rate: 0.80
|
||||
expected_actual_saving_nok: 14400
|
||||
gap_source: hours-of-use-overestimation
|
||||
provenance: "froe — AI-forfattet test-fixture for S3.2; ikke ekte data"
|
||||
tags: [verdict, fixture, kandidat-A]
|
||||
timestamp: 2026-08-03
|
||||
---
|
||||
|
||||
# Ekspert-dom (froe) — kandidat A
|
||||
|
||||
**Beslutning:** godkjent, med realiseringskorreksjon.
|
||||
|
||||
Den modellerte besparelsen paa 18 000 NOK/aar er teknisk korrekt, men i drift realiseres
|
||||
erfaringsvis ~80 % av en timeplan-stipulert LED-besparelse i kontorbygg.
|
||||
|
||||
Dette er dommen som ER relevant for LED-hypotesen — samme kostkode, samme tiltakstype, samme
|
||||
stoerrelsesorden.
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
type: verdict
|
||||
title: "Ekspert-dom (froe) — redusert asfalttykkelse veistrekning B"
|
||||
description: "Froesatt dom om KANDIDAT B (dekketiltak). Baerer sine egne strukturelle felt, saa den noekles paa sin egen kandidat og ikke paa bundelens IR-projeksjon."
|
||||
resource: MULTI-KANDIDAT-MIKRO
|
||||
measure_id: ASFALT-B
|
||||
decision: rejected
|
||||
affected_codes: [05.2]
|
||||
measure_type: "Redusert asfalttykkelse veistrekning B"
|
||||
claimed_saving_nok: 900000
|
||||
realization_rate: 0.31
|
||||
expected_actual_saving_nok: 279000
|
||||
gap_source: levetidsforkortelse-gir-tidligere-reasfaltering
|
||||
provenance: "froe — AI-forfattet test-fixture for S3.2; ikke ekte data"
|
||||
tags: [verdict, fixture, kandidat-B]
|
||||
timestamp: 2026-08-03
|
||||
---
|
||||
|
||||
# Ekspert-dom (froe) — kandidat B
|
||||
|
||||
**Beslutning:** avvist.
|
||||
|
||||
Den modellerte besparelsen paa 900 000 NOK er regnemessig korrekt, men tynnere dekke forkorter
|
||||
levetiden saa mye at reasfaltering kommer tidligere. Livsloepsregnskapet blir negativt.
|
||||
|
||||
Denne dommen handler om **dekketiltaket**, ikke om energitiltaket. Den skal ikke hentes naar
|
||||
hypotesen gjelder LED-retrofit — kostkode, tiltakstype og stoerrelsesorden er alle ulike.
|
||||
|
|
@ -474,6 +474,79 @@ def bundle_candidate_features(bundle_dir: str) -> ProposalFeatures:
|
|||
return _features_from_ir(okf.load_ir_projection(bundle_dir))
|
||||
|
||||
|
||||
# S3.2: a verdict file MAY carry its own structural key. All three fields or none — see
|
||||
# ``_features_from_verdict_frontmatter``.
|
||||
_STRUCTURAL_FRONTMATTER_KEYS = ("affected_codes", "measure_type", "claimed_saving_nok")
|
||||
|
||||
|
||||
class VerdictFrontmatterError(ValueError):
|
||||
"""A ``type: verdict`` file declares its structural key PARTIALLY or unparseably. Fail-fast
|
||||
(validation, never repair — mirroring ``write_concept_file``): the curated context layer is
|
||||
hand-written or written by ``promote_verdict``, and the silent alternative — falling back to the
|
||||
bundle candidate — keys the verdict to the WRONG candidate, which is the exact defect S3.2
|
||||
closes. Contrast the tolerant RAW inbox layer (``load_verdicts_from_dir``), which skips
|
||||
malformed files because anyone may drop anything there."""
|
||||
|
||||
|
||||
def _unquote(raw: str) -> str:
|
||||
"""``parse_frontmatter`` preserves quotes (OKF SPEC §4); the structural fields are compared and
|
||||
hashed against the IR projection's RAW JSON values, so the quotes have to come off here."""
|
||||
return raw.strip().strip('"').strip("'").strip()
|
||||
|
||||
|
||||
def _parse_affected_codes(raw: str) -> frozenset[str]:
|
||||
"""Parse ``affected_codes`` — written by ``promote_verdict`` as ``[A, B]``, and accepted bare
|
||||
(``A, B``) for hand-authored files. Empty is an error: a declared-but-contentless code set
|
||||
Jaccard-matches every other empty set, which is a silent mis-key rather than a key."""
|
||||
codes = {_unquote(part) for part in _unquote(raw).strip("[]").split(",")}
|
||||
codes.discard("")
|
||||
if not codes:
|
||||
raise VerdictFrontmatterError("'affected_codes' is declared but empty")
|
||||
return frozenset(codes)
|
||||
|
||||
|
||||
def _parse_claimed_saving(raw: str) -> float:
|
||||
"""Parse ``claimed_saving_nok`` with ``json.loads`` — deliberately the SAME literal rule the IR
|
||||
projection went through, so ``18000`` stays an int and ``18000.0`` a float. ``_mint_id`` hashes
|
||||
the raw value, so a parser that normalised the type would mint a different id for a promoted
|
||||
verdict than for the bundle-keyed seed describing the same candidate."""
|
||||
try:
|
||||
value = json.loads(_unquote(raw))
|
||||
except ValueError as exc:
|
||||
raise VerdictFrontmatterError(f"'claimed_saving_nok' is not a number: {raw!r}") from exc
|
||||
if not isinstance(value, (int, float)) or isinstance(value, bool):
|
||||
raise VerdictFrontmatterError(f"'claimed_saving_nok' is not a number: {raw!r}")
|
||||
return value
|
||||
|
||||
|
||||
def _features_from_verdict_frontmatter(fm: dict[str, str]) -> ProposalFeatures | None:
|
||||
"""The verdict's OWN structural key, or ``None`` when it declares none (caller falls back to the
|
||||
bundle candidate — how every pre-S3.2 seed keeps working).
|
||||
|
||||
ALL THREE fields or none. A partial declaration is refused rather than merged with the bundle
|
||||
candidate, because the merge would mint a key belonging to NEITHER candidate — a synthetic third
|
||||
proposal that retrieves for nothing. Both fully-present and fully-absent are honest; half is not.
|
||||
"""
|
||||
present = [key for key in _STRUCTURAL_FRONTMATTER_KEYS if fm.get(key, "").strip()]
|
||||
if not present:
|
||||
return None
|
||||
if len(present) != len(_STRUCTURAL_FRONTMATTER_KEYS):
|
||||
missing = [k for k in _STRUCTURAL_FRONTMATTER_KEYS if k not in present]
|
||||
raise VerdictFrontmatterError(
|
||||
f"a verdict file declares {present} but not {missing}; a verdict carries its whole "
|
||||
"structural key or none of it (a partial key belongs to no candidate)"
|
||||
)
|
||||
measure_type = _unquote(fm["measure_type"])
|
||||
if not measure_type:
|
||||
raise VerdictFrontmatterError("'measure_type' is declared but empty")
|
||||
return ProposalFeatures(
|
||||
affected_codes=_parse_affected_codes(fm["affected_codes"]),
|
||||
measure_type=measure_type,
|
||||
claimed_saving_nok=_parse_claimed_saving(fm["claimed_saving_nok"]),
|
||||
description=measure_type,
|
||||
)
|
||||
|
||||
|
||||
def _verdict_rationale(fm: dict[str, str]) -> str:
|
||||
"""Build the few-shot rationale from a ``type: verdict`` file's frontmatter, carrying the
|
||||
learning signal the deterministic validator cannot compute (the realization rate + expected
|
||||
|
|
@ -491,20 +564,34 @@ def _verdict_rationale(fm: dict[str, str]) -> str:
|
|||
|
||||
|
||||
def seed_store_from_bundle(bundle_dir: str) -> VerdictStore:
|
||||
"""Build a ``VerdictStore`` from an OKF bundle's ``type: verdict`` files. Each verdict is keyed
|
||||
on the bundle's candidate features (so it retrieves for that measure) and carries the
|
||||
realization signal in its rationale. The seed verdict stands in for the durable HITL verdict a
|
||||
real expert would supply via the same folder interface (målbilde §3)."""
|
||||
features = bundle_candidate_features(bundle_dir)
|
||||
"""Build a ``VerdictStore`` from an OKF bundle's ``type: verdict`` files. Each verdict carries
|
||||
the realization signal in its rationale, and stands in for the durable HITL verdict a real
|
||||
expert would supply via the same folder interface (målbilde §3).
|
||||
|
||||
KEYING (S3.2): a verdict is keyed on ITS OWN candidate when its frontmatter declares the
|
||||
structural fields (``affected_codes`` / ``measure_type`` / ``claimed_saving_nok``), and on the
|
||||
bundle's IR-projection candidate otherwise. The bundle key alone was single-candidate by
|
||||
construction: a bundle holding verdicts about several candidates collapsed them onto one key, so
|
||||
a verdict about candidate B scored a perfect match against candidate A's query and could be
|
||||
folded into A's hypothesis prompt. The fallback is what keeps every pre-S3.2 seed working
|
||||
unchanged; the fields are OPTIONAL, never required."""
|
||||
bundle = okf.navigate_bundle(bundle_dir)
|
||||
verdicts = [
|
||||
capture_verdict(
|
||||
features,
|
||||
vf.frontmatter.get("decision", "approved"),
|
||||
_verdict_rationale(vf.frontmatter),
|
||||
fallback: ProposalFeatures | None = None
|
||||
verdicts = []
|
||||
for vf in bundle.verdicts:
|
||||
features = _features_from_verdict_frontmatter(vf.frontmatter)
|
||||
if features is None:
|
||||
# Read the IR projection lazily: a bundle whose verdicts all carry their own key does
|
||||
# not need one, and this keeps the fallback path's behaviour byte-identical.
|
||||
fallback = fallback if fallback is not None else bundle_candidate_features(bundle_dir)
|
||||
features = fallback
|
||||
verdicts.append(
|
||||
capture_verdict(
|
||||
features,
|
||||
vf.frontmatter.get("decision", "approved"),
|
||||
_verdict_rationale(vf.frontmatter),
|
||||
)
|
||||
)
|
||||
for vf in bundle.verdicts
|
||||
]
|
||||
return VerdictStore(verdicts=verdicts)
|
||||
|
||||
|
||||
|
|
@ -555,11 +642,19 @@ def promote_verdict(
|
|||
wiki. Provenance-stamped (who/which-experiment/when). ``timestamp`` is a required keyword (no
|
||||
wall-clock default) so promotion is deterministic and the stamp reproducible.
|
||||
|
||||
The promoted file is MINIMAL: it does NOT reproduce the hand-authored seed's structured fields
|
||||
(``realization_rate`` etc.) — the raw ``Verdict`` model carries the learning signal only as
|
||||
``rationale`` prose, which becomes the ``description`` frontmatter ``seed_store_from_bundle``
|
||||
folds into ExpeL. The index link uses a NEUTRAL label (``_PROMOTED_LINK_LABEL``), so the signal
|
||||
reaches a prompt only via the gated fold, never via ``bundle_context`` (§3/§6).
|
||||
The promoted file is MINIMAL as to the LEARNING SIGNAL: it does NOT reproduce the hand-authored
|
||||
seed's structured signal fields (``realization_rate`` etc.) — the raw ``Verdict`` model carries
|
||||
that only as ``rationale`` prose, which becomes the ``description`` frontmatter
|
||||
``seed_store_from_bundle`` folds into ExpeL. It DOES carry its own structural KEY (S3.2:
|
||||
``affected_codes`` / ``measure_type`` / ``claimed_saving_nok``), so the next run keys it on the
|
||||
candidate it is about — a promoted verdict is frequently about a different candidate than the
|
||||
one that bundle's IR projection describes. The index link uses a NEUTRAL label
|
||||
(``_PROMOTED_LINK_LABEL``), so the signal reaches a prompt only via the gated fold, never via
|
||||
``bundle_context`` (§3/§6) — and the structural key is signal-free by construction.
|
||||
|
||||
Round-trip caveat: ``render_frontmatter`` single-lines every value, so a ``measure_type``
|
||||
containing newlines is re-read with those collapsed to spaces and would re-mint a different id.
|
||||
Measure strings are single-line in practice; this is stated rather than defended against.
|
||||
|
||||
Known limitation (mirrors ``write_verdict``): ``_mint_id`` keys on the candidate features, so two
|
||||
approved verdicts about the SAME candidate share an id -> share a filename -> last-write-wins;
|
||||
|
|
@ -575,6 +670,16 @@ def promote_verdict(
|
|||
"type": "verdict",
|
||||
"decision": verdict.decision,
|
||||
"description": verdict.rationale,
|
||||
# S3.2: the promoted file carries its OWN structural key, so the next run's
|
||||
# seed_store_from_bundle keys it on the candidate it is actually about rather than on
|
||||
# whichever candidate that bundle's IR projection happens to describe. Written as the three
|
||||
# fields _features_from_verdict_frontmatter reads back — all three, never a partial key.
|
||||
"affected_codes": "[" + ", ".join(sorted(f.affected_codes)) + "]",
|
||||
"measure_type": f.measure_type,
|
||||
# ``str`` of the raw value, NOT a normalised format: it round-trips through
|
||||
# ``_parse_claimed_saving``'s ``json.loads`` back to the same int/float, and ``_mint_id``
|
||||
# hashes that raw value (``18000`` and ``18000.0`` are different keys).
|
||||
"claimed_saving_nok": str(f.claimed_saving_nok),
|
||||
"verdict_id": verdict.id,
|
||||
"provenance": f"godkjent av {approver}; eksperiment {experiment}; {timestamp}",
|
||||
"timestamp": timestamp,
|
||||
|
|
|
|||
137
tests/test_step32_multicandidate_loadbearing.py
Normal file
137
tests/test_step32_multicandidate_loadbearing.py
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
"""S3.2 load-bearing seam: a verdict is keyed on ITS OWN candidate, not on the bundle's single IR
|
||||
projection (sesjonsplan Fase 2-6 §S3.2; målbilde §2 step 1).
|
||||
|
||||
The gap: ``seed_store_from_bundle`` keyed EVERY ``type: verdict`` file in a bundle on
|
||||
``bundle_candidate_features`` — the one candidate the bundle's ``validator-input.json`` describes.
|
||||
A bundle carrying verdicts about several candidates therefore collapsed them all onto one key: a
|
||||
verdict about candidate B scored a perfect structural match against candidate A's query and could
|
||||
be folded into A's hypothesis prompt. The ExpeL substrate was single-candidate by construction.
|
||||
|
||||
S3.2 lets each verdict file carry its own structural fields in frontmatter
|
||||
(``affected_codes`` / ``measure_type`` / ``claimed_saving_nok``); when they are absent, keying falls
|
||||
back to the bundle candidate — so every hand-authored seed written before S3.2 keeps working
|
||||
unchanged (proven by the untouched step1/step7/step8 suites plus Test B here).
|
||||
|
||||
Three load-bearing tests:
|
||||
- Test A (SEPARATION, the RED point): a bundle with verdicts about two disjoint candidates — a
|
||||
verdict about candidate B must NEVER reach candidate A's hypothesis prompt. Goes RED the moment
|
||||
per-verdict keying is detached: both verdicts then carry candidate A's key, score identically,
|
||||
mint an identical id, and the stable sort hands back whichever the bundle links first — which the
|
||||
fixture deliberately makes the B verdict.
|
||||
- Test B (FALLBACK): a pre-S3.2 seed with no structural frontmatter still keys on the bundle
|
||||
candidate and still retrieves — backward compatibility, stated as a test rather than assumed.
|
||||
- Test C (DISTINCT IDENTITY): the two verdicts mint DIFFERENT ids. Ids hash the structural features,
|
||||
so a shared id would silently collapse the two candidates in ``VerdictStore.add`` (first-write-wins)
|
||||
even where retrieval separated them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
from importlib.resources import files
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser.verdicts import (
|
||||
ExpeLContextProvider,
|
||||
VerdictFrontmatterError,
|
||||
bundle_candidate_features,
|
||||
seed_store_from_bundle,
|
||||
)
|
||||
|
||||
_MULTI_BUNDLE = str(files("portfolio_optimiser").joinpath("data/bundles/multi-kandidat-mikro"))
|
||||
_SINGLE_BUNDLE = str(files("portfolio_optimiser").joinpath("data/bundles/bygg-energi-mikro-a"))
|
||||
|
||||
# Realization markers unique to each verdict file's frontmatter; they reach a prompt only through the
|
||||
# gated ExpeL fold, so their presence/absence in the few-shot IS the retrieval outcome.
|
||||
_MARKER_A = "realiseringsgrad=0.80"
|
||||
_MARKER_B = "realiseringsgrad=0.31"
|
||||
|
||||
|
||||
def _fewshot_for_bundle_candidate(bundle_dir: str, *, k: int = 1) -> str:
|
||||
"""The ExpeL few-shot block Step 1 folds into the hypothesis prompt, for the bundle's OWN
|
||||
candidate — the exact string an agent would read."""
|
||||
store = seed_store_from_bundle(bundle_dir)
|
||||
query = bundle_candidate_features(bundle_dir)
|
||||
return ExpeLContextProvider(store, query, k=k).format_fewshot()
|
||||
|
||||
|
||||
def test_verdict_about_another_candidate_never_reaches_this_candidates_prompt() -> None:
|
||||
"""Test A — the RED point. Candidate A's hypothesis prompt carries A's verdict and NOT B's."""
|
||||
fewshot = _fewshot_for_bundle_candidate(_MULTI_BUNDLE)
|
||||
|
||||
assert _MARKER_A in fewshot, (
|
||||
"candidate A's own verdict must reach A's hypothesis prompt; got:\n" + fewshot
|
||||
)
|
||||
assert _MARKER_B not in fewshot, (
|
||||
"a verdict about candidate B (different cost code, measure and magnitude) reached "
|
||||
"candidate A's hypothesis prompt — the verdicts are keyed on the bundle's single IR "
|
||||
"projection instead of on their own candidate:\n" + fewshot
|
||||
)
|
||||
|
||||
|
||||
def test_verdict_without_structural_frontmatter_still_keys_on_the_bundle_candidate() -> None:
|
||||
"""Test B — fallback. A pre-S3.2 seed (no ``affected_codes``/``measure_type``/
|
||||
``claimed_saving_nok`` in frontmatter) keys on the bundle candidate exactly as before, so it
|
||||
still scores a perfect structural match and still reaches the prompt."""
|
||||
store = seed_store_from_bundle(_SINGLE_BUNDLE)
|
||||
query = bundle_candidate_features(_SINGLE_BUNDLE)
|
||||
|
||||
assert store.verdicts, "the single-candidate fixture must still seed at least one verdict"
|
||||
assert all(v.proposal_features == query for v in store.verdicts), (
|
||||
"verdicts with no structural frontmatter must fall back to the bundle candidate key"
|
||||
)
|
||||
assert "realiseringsgrad=0.80" in ExpeLContextProvider(store, query, k=1).format_fewshot()
|
||||
|
||||
|
||||
def test_the_two_candidates_verdicts_mint_distinct_ids() -> None:
|
||||
"""Test C — distinct identity. Ids hash the structural features, so per-verdict keying must also
|
||||
give the two verdicts different ids; a shared id would collapse them in ``VerdictStore.add``
|
||||
(first-write-wins) even where retrieval kept them apart."""
|
||||
verdicts = seed_store_from_bundle(_MULTI_BUNDLE).verdicts
|
||||
|
||||
assert len(verdicts) == 2, f"expected both candidates' verdicts, got {len(verdicts)}"
|
||||
assert len({v.id for v in verdicts}) == 2, (
|
||||
f"two disjoint candidates minted the same verdict id: {[v.id for v in verdicts]}"
|
||||
)
|
||||
|
||||
|
||||
# --- Test D: a half-declared key is refused, never silently merged -------------------------------
|
||||
|
||||
|
||||
def _bundle_with_patched_verdict(tmp_path: Path, old: str, new: str) -> str:
|
||||
"""A throwaway copy of the multi-candidate bundle with one line of candidate B's verdict
|
||||
frontmatter rewritten — the packaged fixture is never mutated."""
|
||||
dst = tmp_path / "bundle"
|
||||
shutil.copytree(_MULTI_BUNDLE, dst)
|
||||
target = dst / "verdict-b-asfalt.md"
|
||||
text = target.read_text(encoding="utf-8")
|
||||
assert old in text, f"fixture drift: {old!r} not in verdict-b-asfalt.md"
|
||||
target.write_text(text.replace(old, new, 1), encoding="utf-8")
|
||||
return str(dst)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("old", "new", "why"),
|
||||
[
|
||||
("measure_type: ", "measure_type_disabled: ", "two of the three fields declared"),
|
||||
("claimed_saving_nok: 900000", "claimed_saving_nok: nokså mye", "unparseable magnitude"),
|
||||
("affected_codes: [05.2]", "affected_codes: []", "declared but empty code set"),
|
||||
],
|
||||
)
|
||||
def test_a_partial_or_unparseable_structural_key_is_refused(tmp_path, old, new, why) -> None:
|
||||
"""Test D — fail-fast. A verdict file that declares its structural key PARTIALLY or unparseably
|
||||
raises rather than falling back to the bundle candidate.
|
||||
|
||||
The silent alternative is not neutral: it keys the verdict to the WRONG candidate, which is the
|
||||
exact defect S3.2 closes — and a merge of some declared fields with some bundle fields mints a
|
||||
key belonging to NEITHER candidate. The curated context layer is validated, never repaired
|
||||
(mirroring ``write_concept_file``); the tolerant-skip rule belongs to the RAW inbox layer, where
|
||||
anyone may drop anything.
|
||||
|
||||
RED if the refusal is relaxed into a fallback."""
|
||||
bundle_dir = _bundle_with_patched_verdict(tmp_path, old, new)
|
||||
|
||||
with pytest.raises(VerdictFrontmatterError):
|
||||
seed_store_from_bundle(bundle_dir)
|
||||
|
|
@ -33,11 +33,13 @@ import pytest
|
|||
|
||||
from portfolio_optimiser import okf
|
||||
from portfolio_optimiser.verdicts import (
|
||||
ExpeLContextProvider,
|
||||
ProposalFeatures,
|
||||
PromotionRefused,
|
||||
Verdict,
|
||||
bundle_candidate_features,
|
||||
promote_verdict,
|
||||
seed_store_from_bundle,
|
||||
)
|
||||
|
||||
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||
|
|
@ -172,3 +174,87 @@ def test_promoted_signal_stays_out_of_bundle_context(tmp_path) -> None:
|
|||
# though it IS present in the bundle (just excluded from context, like the seed verdict):
|
||||
assert _MARKER in okf.parse_frontmatter(path)["description"]
|
||||
assert path.name in {f.name for f in bundle.verdicts}
|
||||
|
||||
|
||||
# --- Test D (S3.2): the promoted verdict carries its OWN structural key --------------------------
|
||||
|
||||
|
||||
def test_promoted_verdict_about_another_candidate_keeps_its_own_key(tmp_path) -> None:
|
||||
"""S3.2 ROUND-TRIP: a promoted verdict is frequently about a DIFFERENT candidate than the one
|
||||
the target bundle's IR projection describes. ``promote_verdict`` therefore writes the verdict's
|
||||
own structural key, and ``seed_store_from_bundle`` reads it back — so the promoted verdict does
|
||||
not impersonate the bundle candidate in the next run's retrieval.
|
||||
|
||||
RED if ``promote_verdict`` stops writing the three structural fields (the promoted verdict then
|
||||
falls back to the bundle candidate's key and its marker reaches that candidate's prompt)."""
|
||||
bundle_dir = _copy_bundle(tmp_path)
|
||||
other_marker = "realiseringsgrad=0.19"
|
||||
other_candidate = Verdict(
|
||||
id="STEG8-OTHER-CANDIDATE",
|
||||
proposal_features=ProposalFeatures(
|
||||
affected_codes=frozenset({"05.2", "03.1"}),
|
||||
measure_type="Redusert asfalttykkelse",
|
||||
claimed_saving_nok=900000,
|
||||
),
|
||||
decision="approved",
|
||||
rationale=f"asfalttiltak godkjent med kraftig realiseringskorreksjon ({other_marker})",
|
||||
)
|
||||
|
||||
path = promote_verdict(
|
||||
bundle_dir, other_candidate, approver="persona", experiment="exp-D", timestamp="2026-06-30"
|
||||
)
|
||||
|
||||
fm = okf.parse_frontmatter(path)
|
||||
assert fm["affected_codes"] == "[03.1, 05.2]" # sorted -> deterministic bytes
|
||||
assert fm["measure_type"] == "Redusert asfalttykkelse"
|
||||
assert fm["claimed_saving_nok"] == "900000"
|
||||
|
||||
# The round trip: the next run's seed keys it on ITS candidate, so it does not surface for the
|
||||
# bundle's own (LED) candidate.
|
||||
store = seed_store_from_bundle(bundle_dir)
|
||||
query = bundle_candidate_features(bundle_dir)
|
||||
fewshot = ExpeLContextProvider(store, query, k=1).format_fewshot()
|
||||
assert other_marker not in fewshot, (
|
||||
"a promoted verdict about a different candidate reached this candidate's hypothesis "
|
||||
"prompt — the promoted file is not carrying its own structural key:\n" + fewshot
|
||||
)
|
||||
# Keyed on the promoted candidate's STRUCTURAL fields (``description`` is surface text, outside
|
||||
# both the similarity score and the minted id — the seeder fills it from ``measure_type``).
|
||||
keys = {
|
||||
(v.proposal_features.affected_codes, v.proposal_features.measure_type)
|
||||
for v in store.verdicts
|
||||
}
|
||||
assert (frozenset({"05.2", "03.1"}), "Redusert asfalttykkelse") in keys, (
|
||||
f"the promoted verdict was seeded with the wrong structural key; got {keys}"
|
||||
)
|
||||
|
||||
|
||||
def test_promotion_round_trip_preserves_the_key_for_the_bundles_own_candidate(tmp_path) -> None:
|
||||
"""S3.2 IDENTITY: promoting a verdict about the bundle's OWN candidate must re-seed to exactly
|
||||
the key the pre-S3.2 fallback produced — same codes, same measure, same magnitude TYPE.
|
||||
|
||||
Otherwise a candidate's learning signal splits across two keys over time (the promoted verdicts
|
||||
under one, the hand-authored seeds under the fallback), and neither retrieves the other. The
|
||||
magnitude is asserted on its exact value AND type because ``_mint_id`` hashes the raw value:
|
||||
``30000`` and ``30000.0`` are different ids."""
|
||||
bundle_dir = _copy_bundle(tmp_path)
|
||||
candidate = bundle_candidate_features(bundle_dir)
|
||||
promote_verdict(
|
||||
bundle_dir,
|
||||
_approved_verdict(bundle_dir),
|
||||
approver="persona",
|
||||
experiment="exp-E",
|
||||
timestamp="2026-06-30",
|
||||
)
|
||||
|
||||
promoted = [v for v in seed_store_from_bundle(bundle_dir).verdicts if _MARKER in v.rationale]
|
||||
assert len(promoted) == 1, f"expected exactly the promoted verdict, got {len(promoted)}"
|
||||
seeded = promoted[0].proposal_features
|
||||
|
||||
assert seeded.affected_codes == candidate.affected_codes
|
||||
assert seeded.measure_type == candidate.measure_type
|
||||
assert seeded.claimed_saving_nok == candidate.claimed_saving_nok
|
||||
assert type(seeded.claimed_saving_nok) is type(candidate.claimed_saving_nok), (
|
||||
f"magnitude type changed across the round trip: {seeded.claimed_saving_nok!r} vs "
|
||||
f"{candidate.claimed_saving_nok!r} — _mint_id hashes the raw value, so this splits the id"
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue