Kjell Tore Guttormsen
a362504108
The 18.09 re-measurement took row 2 to 3 of 3 GREEN on a tree this product had never
run in: four handwritten outcome.json, four handwritten <run_id>-coverage.json in an
outbox the forger named in those same files, and os.utime for the ordering. "Round 0
must be a named real run" was implemented as "a file with that name exists" — which
touch satisfies. The attack is committed as a red test in b769537.
Three bindings, chosen because each removes one of the forger's three moves:
1. The outbox is DERIVED, never declared. It is <rounds-dir>/<n>/outbox/, and an
outcome.json that names one is refused by name. A path a round file chooses is a
path it can point at a directory the same hand just wrote.
2. "The run exists" now means the run's own artefacts agree WITH EACH OTHER on content
the gate recomputes (verify_run). Every evaluated approach has the proposal/outcome
pair write_outbox actually persists; outcome_type IS the coverage status, reason IS
its detail, a validated row's figure IS the proposal's own claimed_saving_nok, the
provenance stamp follows the same branch, and verdict_id is RE-MINTED here from the
proposal's own IR with the product's one minting rule (A5) rather than read. A
not_evaluated approach wrote neither file, and an artefact naming an approach the
coverage omits belongs to another run. verdicts._features_from_ir is made public for
this: a second private copy in the gate could drift from the rule the run stamped
with, which would turn the binding into a coincidence.
3. mtime decides nothing. The run's time is the round's declared ran_at (ISO-8601 with
zone, required). An mtime is not evidence — it is a filesystem attribute one utime
call sets.
What this does NOT do is prove a run happened. Nothing in a directory can: the outbox
writers are byte-deterministic and carry no clock by contract. So row 2 states its own
limit on every run instead of leaving GREEN to imply it (Row.attests / RUN_ATTESTATION):
that a run was actually made, and when, is the operator's to confirm. The cost of a
forgery moves from touch to reproducing the product's own artefact set, minting rule
included.
Measured, in a scratch copy, never in the work tree — four new mutants in this class,
each one line, each felled by the whole suite:
M7 if "outbox" in data: -> if False: 1 failed (declares its own outbox)
M8 if verdict_id != minted: -> if False: 1 failed (key is not the IR's)
M9 if strays: -> if False: 1 failed (artefact of another run)
M10 ran_at -> coverage mtime 18 failed (incl. the utime test)
Control, same scratch copy, unmutated: 1993 passed, 10 skipped, 5 xfailed.
Work tree, re-run after git add: uv run pytest -q -> 1998 passed, 5 skipped, 5 xfailed.
Gate: uv run python -m portfolio_optimiser.evals.v1_gate -> exit 1, row 2 RED (0 of 3).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
779 lines
35 KiB
Python
779 lines
35 KiB
Python
"""VerdictStore + ExpeL retrieval + Layer-2 out-of-band verdict capture (B2).
|
|
|
|
The learning substrate: a structurally-ranked store of historical expert verdicts, surfaced
|
|
as ExpeL few-shots for the next run. **Similarity is structural, not textual** — a weighted
|
|
score over the affected cost-code set (Jaccard) + measure-type match + a magnitude bucket;
|
|
raw ``description`` text is deliberately excluded, so a true match with different wording
|
|
beats surface-text decoys.
|
|
|
|
Two corrections vs the Fase 1 spike:
|
|
|
|
1. **Two-arg ``extend_instructions``** (the Critical Fase 1 bug): ``before_run`` calls
|
|
``context.extend_instructions(self.source_id, [...])`` — the genuine GA signature
|
|
(``_sessions.py:253``), exercised against a REAL ``SessionContext`` in the tests.
|
|
2. **Layer-2 capture** (``capture_verdict``): mints a stable content-hash ``Verdict.id`` so a
|
|
structurally identical proposal maps to the same id (the learning-loop key). The store is
|
|
**in-memory only** for the MVP — durable persistence is deferred to Fase 3.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
import json
|
|
import logging
|
|
import re
|
|
from dataclasses import dataclass
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
from agent_framework import ContextProvider, SessionContext
|
|
|
|
from portfolio_optimiser import okf, semretrieval
|
|
|
|
_log = logging.getLogger(__name__)
|
|
|
|
# Weights: the affected cost-code overlap dominates, then measure type, then magnitude.
|
|
_W_CODES, _W_MEASURE, _W_MAGNITUDE = 0.60, 0.25, 0.15
|
|
_MAGNITUDE_BUCKETS = [(0.0, 1e5), (1e5, 5e5), (5e5, 1e6), (1e6, float("inf"))]
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class ProposalFeatures:
|
|
"""The *structured* features retrieval ranks on. ``description`` is surface text and is
|
|
deliberately NOT part of the similarity score (nor the minted id)."""
|
|
|
|
affected_codes: frozenset[str]
|
|
measure_type: str
|
|
claimed_saving_nok: float
|
|
description: str = ""
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class Verdict:
|
|
"""One historical expert verdict in the store."""
|
|
|
|
id: str
|
|
proposal_features: ProposalFeatures
|
|
decision: str # "approved" | "rejected"
|
|
rationale: str
|
|
|
|
|
|
def _magnitude_bucket(value: float) -> int:
|
|
for i, (low, high) in enumerate(_MAGNITUDE_BUCKETS):
|
|
if low <= value < high:
|
|
return i
|
|
return len(_MAGNITUDE_BUCKETS) - 1
|
|
|
|
|
|
def _jaccard(a: frozenset[str], b: frozenset[str]) -> float:
|
|
union = a | b
|
|
return len(a & b) / len(union) if union else 1.0
|
|
|
|
|
|
def similarity(query: ProposalFeatures, candidate: ProposalFeatures) -> float:
|
|
"""Weighted structural similarity in [0, 1] — text is ignored by design."""
|
|
codes = _jaccard(query.affected_codes, candidate.affected_codes)
|
|
measure = 1.0 if query.measure_type == candidate.measure_type else 0.0
|
|
magnitude = (
|
|
1.0
|
|
if _magnitude_bucket(query.claimed_saving_nok)
|
|
== _magnitude_bucket(candidate.claimed_saving_nok)
|
|
else 0.0
|
|
)
|
|
return _W_CODES * codes + _W_MEASURE * measure + _W_MAGNITUDE * magnitude
|
|
|
|
|
|
def _mint_id(features: ProposalFeatures) -> str:
|
|
"""Stable content-hash id over the STRUCTURAL fields (not description), so a
|
|
structurally identical proposal maps to the same id."""
|
|
canonical = json.dumps(
|
|
{
|
|
"affected_codes": sorted(features.affected_codes),
|
|
"measure_type": features.measure_type,
|
|
"claimed_saving_nok": features.claimed_saving_nok,
|
|
},
|
|
sort_keys=True,
|
|
separators=(",", ":"),
|
|
)
|
|
return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:16]
|
|
|
|
|
|
def verdict_key(features: ProposalFeatures) -> str:
|
|
"""The id a verdict on a proposal with these features keys under — the learning-loop key.
|
|
|
|
Public because a run must be able to stamp an artefact with the key an expert verdict on THAT
|
|
candidate will arrive under, WITHOUT capturing a decision nobody has made yet (A5: every
|
|
evaluated approach gets its own judgeable artefact, but only one of them is the run's outcome).
|
|
One key, one minting rule: this delegates to ``_mint_id`` rather than restating the hash, so a
|
|
caller can never drift from the id ``capture_verdict`` actually assigns (the ``(p)`` precedent
|
|
— a second private copy of a keying rule is the defect, not the convenience)."""
|
|
return _mint_id(features)
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class VerdictCollision:
|
|
"""Two knowledge bases described ONE candidate, and the second verdict was dropped.
|
|
|
|
``VerdictStore.add`` is first-write-wins per id and ``_mint_id`` excludes the corpus BY
|
|
CONSTRUCTION, so this drop is CORRECT — one candidate, one learning key — but it used to be
|
|
silent, and a base whose finding never entered the store looked exactly like a base that found
|
|
nothing (operator decision D2).
|
|
|
|
A DIAGNOSTIC, never an aggregate over bases: it reports that two bases described one candidate
|
|
and never merges, ranks or sums them, which is why it needs no combination rule. ``add`` itself
|
|
is unchanged — it is called from inside ``run_project``/``run_portfolio``, where a drop can
|
|
equally be against a Step-7 inbox verdict or a bundle seed, so only the dispatcher holds the
|
|
id -> base map that makes "a SECOND base" a statement worth making.
|
|
"""
|
|
|
|
verdict_id: str
|
|
first_bundle_id: str
|
|
second_bundle_id: str
|
|
|
|
|
|
def capture_verdict(features: ProposalFeatures, decision: str, rationale: str) -> Verdict:
|
|
"""Layer-2 out-of-band verdict constructor: mint a stable content-hash id (the
|
|
learning-loop key) and build the ``Verdict`` to persist in the store."""
|
|
return Verdict(
|
|
id=_mint_id(features),
|
|
proposal_features=features,
|
|
decision=decision,
|
|
rationale=rationale,
|
|
)
|
|
|
|
|
|
# --- Async file inbox (Fase 5, Steg 7): plain-JSON verdict serialization + tolerant folder load ---
|
|
# The raw output layer is a plain JSON store (R2), NOT OKF — OKF is reserved for the promoted
|
|
# (Step-8) layer. One verdict per file lets an expert/persona drop verdicts into a folder out of
|
|
# band, and a separate later run pick them up (målbilde §3 long loop).
|
|
|
|
_REQUIRED_VERDICT_KEYS = {"id", "decision", "rationale", "proposal_features"}
|
|
# The binary run-path decision vocabulary (Verdict.decision domain + FeedbackContract). An inbox file
|
|
# with any other decision is SKIPPED by load_verdicts_from_dir — ``approved_with_adjustment`` is
|
|
# deliberately EXCLUDED here (it lives only in bundle-seed frontmatter + the promotion gate).
|
|
_INBOX_DECISION_VOCABULARY = frozenset({"approved", "rejected"})
|
|
|
|
|
|
def verdict_to_dict(verdict: Verdict) -> dict[str, Any]:
|
|
"""Serialize a ``Verdict`` to a JSON-ready dict. ``affected_codes`` (a frozenset) is emitted as
|
|
a SORTED list — matching ``_mint_id``'s canonical form — so the round-trip is lossless and the
|
|
on-disk form is deterministic."""
|
|
f = verdict.proposal_features
|
|
return {
|
|
"id": verdict.id,
|
|
"decision": verdict.decision,
|
|
"rationale": verdict.rationale,
|
|
"proposal_features": {
|
|
"affected_codes": sorted(f.affected_codes),
|
|
"measure_type": f.measure_type,
|
|
"claimed_saving_nok": f.claimed_saving_nok,
|
|
"description": f.description,
|
|
},
|
|
}
|
|
|
|
|
|
def verdict_from_dict(data: dict[str, Any]) -> Verdict:
|
|
"""Reconstruct a ``Verdict`` from its serialized dict. The ``id`` is read VERBATIM — never
|
|
re-minted — so a verdict authored elsewhere keeps its identity and the int/float magnitudes in
|
|
its features survive the JSON round-trip untouched (re-minting would re-hash and could diverge,
|
|
since ``_mint_id`` hashes the raw value, e.g. ``30000`` vs ``30000.0``)."""
|
|
pf = data["proposal_features"]
|
|
return Verdict(
|
|
id=data["id"],
|
|
proposal_features=ProposalFeatures(
|
|
affected_codes=frozenset(pf["affected_codes"]),
|
|
measure_type=pf["measure_type"],
|
|
claimed_saving_nok=pf["claimed_saving_nok"],
|
|
description=pf.get("description", ""),
|
|
),
|
|
decision=data["decision"],
|
|
rationale=data["rationale"],
|
|
)
|
|
|
|
|
|
def write_verdict(directory: str, verdict: Verdict) -> Path:
|
|
"""Write one verdict as ``{id}.json`` into ``directory`` (created if needed) and return the path.
|
|
|
|
This is the public authoring primitive the expert/persona (sim) or human (prod) uses to drop a
|
|
verdict into the async inbox via the SAME folder interface — and the writer a future Step-8
|
|
promotion would reuse. It is deliberately NOT called inside ``run_project``: the system READS the
|
|
inbox, it does not write to it (målbilde §3 role split).
|
|
|
|
Limitation: ``id`` is a content hash of the FEATURES only (``_mint_id``), so two verdicts with
|
|
identical features but different decisions share a filename — last write wins on disk (mirroring
|
|
``VerdictStore.add``'s first-wins in memory). Acceptable for the raw MVP layer."""
|
|
path = Path(directory)
|
|
path.mkdir(parents=True, exist_ok=True)
|
|
target = path / f"{verdict.id}.json"
|
|
target.write_text(
|
|
json.dumps(verdict_to_dict(verdict), sort_keys=True, indent=2), encoding="utf-8"
|
|
)
|
|
return target
|
|
|
|
|
|
def load_verdicts_from_dir(
|
|
directory: str,
|
|
*,
|
|
max_rationale_len: int | None = None,
|
|
max_files: int | None = None,
|
|
) -> list[Verdict]:
|
|
"""Load every well-formed verdict JSON file from an async inbox folder, TOLERANTLY (OKF §4
|
|
spirit): a missing folder yields ``[]``; files that are not ``.json``, fail to parse, lack a
|
|
required key, carry a decision outside the binary vocabulary ``{approved, rejected}``, or exceed
|
|
``max_rationale_len`` are SKIPPED, never raised. The folder is written out of band by an external
|
|
party, so half-written or foreign files (e.g. a bundle's ``golden.json``) are realistic — this is
|
|
the raw layer, not required input (contrast ``okf.load_ir_projection``'s fail-fast). Order is
|
|
deterministic (sorted by filename).
|
|
|
|
Herding (S2.5), two distinct axes by design:
|
|
- ``max_rationale_len`` (per-file, TOLERANT): an over-long rationale is skipped + logged, never
|
|
raised — matching the per-file skip contract the Step-7 loop relies on.
|
|
- ``max_files`` (aggregate, FAIL-FAST): more files than the cap RAISES ``ValueError`` before any
|
|
parsing — an oversized inbox is an aggregate integrity signal, not a per-file anomaly.
|
|
Both default to ``None`` (no cap), so existing callers are byte-for-byte unaffected."""
|
|
path = Path(directory)
|
|
if not path.is_dir():
|
|
return []
|
|
files = sorted(path.glob("*.json"))
|
|
if max_files is not None and len(files) > max_files:
|
|
raise ValueError(
|
|
f"verdict inbox {directory!r} has {len(files)} files, over the max_files cap "
|
|
f"({max_files}) — an oversized inbox is a fail-fast aggregate guard (contrast the "
|
|
"per-file tolerant skips)"
|
|
)
|
|
verdicts: list[Verdict] = []
|
|
for file in files:
|
|
try:
|
|
data = json.loads(file.read_text(encoding="utf-8"))
|
|
except (OSError, UnicodeDecodeError, json.JSONDecodeError):
|
|
# UnicodeDecodeError: a *.json file hand-saved in Latin-1 (Norwegian æ/ø/å) is invalid
|
|
# UTF-8 — a per-file tolerant skip, not a raise (a ValueError subclass, caught by neither
|
|
# OSError nor JSONDecodeError). Kept in lockstep with hitl._load_json_dict.
|
|
continue
|
|
if not isinstance(data, dict) or not _REQUIRED_VERDICT_KEYS <= data.keys():
|
|
continue
|
|
if data.get("decision") not in _INBOX_DECISION_VOCABULARY:
|
|
_log.debug(
|
|
"skipping inbox verdict %s: decision %r outside vocabulary",
|
|
file.name,
|
|
data.get("decision"),
|
|
)
|
|
continue
|
|
if (
|
|
max_rationale_len is not None
|
|
and len(str(data.get("rationale", ""))) > max_rationale_len
|
|
):
|
|
_log.warning(
|
|
"skipping inbox verdict %s: rationale length %d exceeds max_rationale_len %d",
|
|
file.name,
|
|
len(str(data.get("rationale", ""))),
|
|
max_rationale_len,
|
|
)
|
|
continue
|
|
try:
|
|
verdicts.append(verdict_from_dict(data))
|
|
except (KeyError, TypeError):
|
|
continue
|
|
return verdicts
|
|
|
|
|
|
@dataclass
|
|
class VerdictStore:
|
|
"""Minimal store of historical verdicts. In-memory by default; ``from_dir`` / the
|
|
``load_verdicts_from_dir`` merge primitive back it with the async file inbox (Steg 7)."""
|
|
|
|
verdicts: list[Verdict]
|
|
|
|
# S3.1 opt-in seam: ``None`` means the structural default, which is byte-identical to the
|
|
# pre-seam inline sort. Only an explicit caller (``run.py --semantic-retrieval``) installs a
|
|
# different ranker, so default retrieval behaviour is unchanged.
|
|
retriever: semretrieval.Retriever | None = None
|
|
|
|
@classmethod
|
|
def from_dir(cls, directory: str) -> VerdictStore:
|
|
"""Build a store from an async inbox folder (convenience constructor). ``run_project`` uses
|
|
the merge primitive (``add`` per loaded verdict) instead, to preserve a passed store's
|
|
existing verdicts (the cross-project threading in ``run_portfolio``); this is for callers
|
|
that want a fresh store straight from a folder."""
|
|
return cls(verdicts=load_verdicts_from_dir(directory))
|
|
|
|
def retrieve(
|
|
self,
|
|
query: ProposalFeatures,
|
|
k: int,
|
|
*,
|
|
retriever: semretrieval.Retriever | None = None,
|
|
) -> list[Verdict]:
|
|
"""Return the top-``k`` verdicts. Deterministic: ties break by verdict id, so ordering
|
|
is stable across runs.
|
|
|
|
Ranking resolves in three steps: the explicit ``retriever`` argument, then
|
|
``self.retriever``, then ``StructuralRetriever`` — the same weighted structural score and
|
|
``(-similarity, id)`` key as before the seam existed.
|
|
|
|
``retriever`` is PER CALL and keyword-only. It exists because the alternative — assigning
|
|
``store.retriever`` — mutates an object the caller owns, so an opt-in made for one
|
|
retrieval silently governed every later use of that store (including a subsequent run with
|
|
the flag OFF). ``self.retriever`` survives as the store-level default for callers that
|
|
genuinely want a store to rank one way for its whole lifetime; see ``docs/extending.md``."""
|
|
if k <= 0:
|
|
raise ValueError(f"k must be positive, got {k}")
|
|
ranker = retriever if retriever is not None else self.retriever
|
|
if ranker is None:
|
|
ranker = semretrieval.StructuralRetriever(similarity)
|
|
return ranker.rank(query, self.verdicts, k)
|
|
|
|
def add(self, verdict: Verdict) -> None:
|
|
"""Persist a captured verdict in-memory.
|
|
|
|
Conflict semantics (chosen, documented): the store is FIRST-write-wins per ``id`` — a
|
|
later verdict with the same content-hash id is dropped, which makes repeated Step-7 inbox
|
|
merges idempotent. The disk layers are the opposite: ``write_verdict`` (inbox) and
|
|
``promote_verdict`` (wiki) are last-write-wins per file. Because ids hash the candidate
|
|
FEATURES, "same id" means "same candidate measure", not "same verdict event". A full
|
|
verdict-conflict taxonomy (B10: rejection categories + a rule for conflicting expert
|
|
verdicts) is deliberately deferred until real experts produce conflicting verdicts."""
|
|
if all(v.id != verdict.id for v in self.verdicts):
|
|
self.verdicts.append(verdict)
|
|
|
|
|
|
class ExpeLContextProvider(ContextProvider):
|
|
"""Wraps ``VerdictStore.retrieve`` for ExpeL few-shot injection via the GA
|
|
``ContextProvider`` hook."""
|
|
|
|
def __init__(
|
|
self,
|
|
store: VerdictStore,
|
|
query: ProposalFeatures,
|
|
*,
|
|
k: int = 3,
|
|
retriever: semretrieval.Retriever | None = None,
|
|
) -> None:
|
|
super().__init__(source_id="expel-verdictstore")
|
|
self._store = store
|
|
self._query = query
|
|
self._k = k
|
|
# Per-call ranker, threaded straight through to ``retrieve`` — see its docstring for why
|
|
# this is a parameter rather than an assignment on the store.
|
|
self._retriever = retriever
|
|
|
|
def format_fewshot(self) -> str:
|
|
hits = self._store.retrieve(self._query, self._k, retriever=self._retriever)
|
|
body = "\n".join(f"- [{v.id}] {v.decision}: {v.rationale}" for v in hits)
|
|
return f"Relevant prior verdicts (ExpeL few-shot):\n{body}"
|
|
|
|
async def before_run(
|
|
self,
|
|
*,
|
|
agent: object,
|
|
session: object,
|
|
context: SessionContext,
|
|
state: dict,
|
|
) -> None:
|
|
# Two-arg GA signature (_sessions.py:253) — the Fase 1 single-arg bug is fixed.
|
|
context.extend_instructions(self.source_id, [self.format_fewshot()])
|
|
|
|
|
|
def seed_store() -> VerdictStore:
|
|
"""Seed 12 synthetic verdicts spanning the reference domain's cost codes, measure types,
|
|
magnitudes, and decisions (B2 store of 10-20)."""
|
|
rows = [
|
|
(
|
|
"V01",
|
|
{"05.2", "03.1"},
|
|
"scope_reduction",
|
|
180_000,
|
|
"approved",
|
|
"asphalt + base course trimmed within feasible range",
|
|
),
|
|
(
|
|
"V02",
|
|
{"05.2"},
|
|
"rate_renegotiation",
|
|
60_000,
|
|
"approved",
|
|
"renegotiated asphalt unit rate",
|
|
),
|
|
(
|
|
"V03",
|
|
{"07.4"},
|
|
"material_substitution",
|
|
120_000,
|
|
"rejected",
|
|
"granite kerb substitution unsafe",
|
|
),
|
|
(
|
|
"V04",
|
|
{"09.1"},
|
|
"scope_reduction",
|
|
240_000,
|
|
"approved",
|
|
"fewer LED masts on low-traffic stretch",
|
|
),
|
|
(
|
|
"V05",
|
|
{"02.3", "03.1"},
|
|
"scope_reduction",
|
|
350_000,
|
|
"rejected",
|
|
"soil replacement is load-bearing, cannot cut",
|
|
),
|
|
("V06", {"21.2"}, "rate_renegotiation", 90_000, "approved", "blasting rate renegotiated"),
|
|
(
|
|
"V07",
|
|
{"22.4"},
|
|
"material_substitution",
|
|
700_000,
|
|
"rejected",
|
|
"fiber shotcrete spec is mandated",
|
|
),
|
|
(
|
|
"V08",
|
|
{"88.2"},
|
|
"scope_reduction",
|
|
150_000,
|
|
"approved",
|
|
"concrete repair area re-measured smaller",
|
|
),
|
|
(
|
|
"V09",
|
|
{"87.3"},
|
|
"material_substitution",
|
|
130_000,
|
|
"approved",
|
|
"alternative membrane qualified",
|
|
),
|
|
(
|
|
"V10",
|
|
{"05.2", "03.1"},
|
|
"rate_renegotiation",
|
|
200_000,
|
|
"approved",
|
|
"combined paving rate discount",
|
|
),
|
|
(
|
|
"V11",
|
|
{"01.1"},
|
|
"scope_reduction",
|
|
95_000,
|
|
"rejected",
|
|
"rigging is fixed cost, no scope to cut",
|
|
),
|
|
(
|
|
"V12",
|
|
{"31.3"},
|
|
"scope_reduction",
|
|
110_000,
|
|
"approved",
|
|
"drainage length reduced after survey",
|
|
),
|
|
]
|
|
return VerdictStore(
|
|
verdicts=[
|
|
Verdict(
|
|
id=vid,
|
|
proposal_features=ProposalFeatures(
|
|
affected_codes=frozenset(codes),
|
|
measure_type=mtype,
|
|
claimed_saving_nok=saving,
|
|
description=desc,
|
|
),
|
|
decision=decision,
|
|
rationale=desc,
|
|
)
|
|
for vid, codes, mtype, saving, decision, desc in rows
|
|
]
|
|
)
|
|
|
|
|
|
# --- OKF-bundle seeding (Fase 2a): turn a project's bundle into the ExpeL substrate ---
|
|
|
|
|
|
def features_from_ir(ir: dict[str, Any]) -> ProposalFeatures:
|
|
"""Map a proposal IR dict (a bundle's ``validator-input.json`` projection, or an outbox
|
|
artefact's own ``proposal`` payload) to the structural features the store ranks on: the
|
|
affected cost-code set, the measure string, and the claimed magnitude.
|
|
|
|
PUBLIC for the same reason ``verdict_key`` is (A5, one minting rule): the v1 gate re-mints an
|
|
artefact's verdict id from the artefact's OWN IR to check that a run's outbox agrees with
|
|
itself, and a second private copy of this mapping in the gate would be a rule that can drift
|
|
from the one the run actually used — the defect, not the convenience."""
|
|
return ProposalFeatures(
|
|
affected_codes=frozenset(item["code"] for item in ir["affected_items"]),
|
|
measure_type=ir["measure"],
|
|
claimed_saving_nok=ir["claimed_saving_nok"],
|
|
description=ir.get("measure", ""),
|
|
)
|
|
|
|
|
|
def bundle_candidate_features(bundle_dir: str) -> ProposalFeatures:
|
|
"""The pre-hypothesis ExpeL query key: the candidate measure's structural features, read from
|
|
the OKF bundle's IR projection. Available BEFORE any proposal is generated — which is what lets
|
|
Step-1 retrieve prior verdicts and fold them into the hypothesis prompt (målbilde §2 step 1).
|
|
|
|
Fail-fast. Use ``optional_bundle_candidate_features`` where a base without a projection is
|
|
legitimate — and note that its two consumers answer that absence DIFFERENTLY, on purpose."""
|
|
return features_from_ir(okf.load_ir_projection(bundle_dir))
|
|
|
|
|
|
def optional_bundle_candidate_features(bundle_dir: str) -> ProposalFeatures | None:
|
|
"""``bundle_candidate_features`` where a base with no IR projection is legitimate: ``None``
|
|
instead of raising (S7b søm 1). An ingested corpus has no hand-written projection, so it has no
|
|
pre-hypothesis candidate to key retrieval against — a fact about the base, not an error.
|
|
|
|
Tolerance stops at absence, as everywhere: a projection that exists but is malformed still
|
|
raises (``okf.load_optional_ir_projection``'s rule)."""
|
|
ir = okf.load_optional_ir_projection(bundle_dir)
|
|
return None if ir is None else features_from_ir(ir)
|
|
|
|
|
|
class VerdictKeyUnavailable(ValueError):
|
|
"""A ``type: verdict`` concept declares no structural key AND its bundle has no IR projection to
|
|
fall back on (S7b søm 1).
|
|
|
|
Fail-closed, and the ONE place in this seam where absence is not tolerable. The pre-S3.2
|
|
fallback keys such a verdict on the bundle's projection candidate; with no projection there is
|
|
no candidate, and minting a key anyway would attach the verdict to a candidate it is not about —
|
|
precisely the defect S3.2 closes. Validation, never repair (``write_concept_file``'s rule): the
|
|
three fields are the author's to declare, and ``promote_verdict`` writes them, so a base grown
|
|
by the loop itself is never affected."""
|
|
|
|
|
|
# S3.2: a verdict file MAY carry its own structural key. All three fields or none — see
|
|
# ``_features_from_verdict_frontmatter``.
|
|
_STRUCTURAL_FRONTMATTER_KEYS = ("affected_codes", "measure_type", "claimed_saving_nok")
|
|
|
|
|
|
class VerdictFrontmatterError(ValueError):
|
|
"""A ``type: verdict`` file declares its structural key PARTIALLY or unparseably. Fail-fast
|
|
(validation, never repair — mirroring ``write_concept_file``): the curated context layer is
|
|
hand-written or written by ``promote_verdict``, and the silent alternative — falling back to the
|
|
bundle candidate — keys the verdict to the WRONG candidate, which is the exact defect S3.2
|
|
closes. Contrast the tolerant RAW inbox layer (``load_verdicts_from_dir``), which skips
|
|
malformed files because anyone may drop anything there."""
|
|
|
|
|
|
# ``parse_frontmatter`` preserves quotes (OKF SPEC §4); the structural fields are compared and
|
|
# hashed against the IR projection's RAW JSON values, so the quotes have to come off. DELEGATED,
|
|
# not copied: this was a private implementation while ``okf.bundle_context`` stripped only ``"``,
|
|
# and a duplicated conversion drifts (the (p) precedent). ``okf`` owns ``parse_frontmatter``, so it
|
|
# owns the unquoting rule. Gated by ``tests/test_frontmatter_unquote_loadbearing.py``.
|
|
_unquote = okf.unquote_scalar
|
|
|
|
|
|
def _parse_affected_codes(raw: str) -> frozenset[str]:
|
|
"""Parse ``affected_codes`` — written by ``promote_verdict`` as ``[A, B]``, and accepted bare
|
|
(``A, B``) for hand-authored files. Empty is an error: a declared-but-contentless code set
|
|
Jaccard-matches every other empty set, which is a silent mis-key rather than a key."""
|
|
codes = {_unquote(part) for part in _unquote(raw).strip("[]").split(",")}
|
|
codes.discard("")
|
|
if not codes:
|
|
raise VerdictFrontmatterError("'affected_codes' is declared but empty")
|
|
return frozenset(codes)
|
|
|
|
|
|
def _parse_claimed_saving(raw: str) -> float:
|
|
"""Parse ``claimed_saving_nok`` with ``json.loads`` — deliberately the SAME literal rule the IR
|
|
projection went through, so ``18000`` stays an int and ``18000.0`` a float. ``_mint_id`` hashes
|
|
the raw value, so a parser that normalised the type would mint a different id for a promoted
|
|
verdict than for the bundle-keyed seed describing the same candidate."""
|
|
try:
|
|
value = json.loads(_unquote(raw))
|
|
except ValueError as exc:
|
|
raise VerdictFrontmatterError(f"'claimed_saving_nok' is not a number: {raw!r}") from exc
|
|
if not isinstance(value, (int, float)) or isinstance(value, bool):
|
|
raise VerdictFrontmatterError(f"'claimed_saving_nok' is not a number: {raw!r}")
|
|
return value
|
|
|
|
|
|
def _features_from_verdict_frontmatter(fm: dict[str, str]) -> ProposalFeatures | None:
|
|
"""The verdict's OWN structural key, or ``None`` when it declares none (caller falls back to the
|
|
bundle candidate — how every pre-S3.2 seed keeps working).
|
|
|
|
ALL THREE fields or none. A partial declaration is refused rather than merged with the bundle
|
|
candidate, because the merge would mint a key belonging to NEITHER candidate — a synthetic third
|
|
proposal that retrieves for nothing. Both fully-present and fully-absent are honest; half is not.
|
|
"""
|
|
present = [key for key in _STRUCTURAL_FRONTMATTER_KEYS if fm.get(key, "").strip()]
|
|
if not present:
|
|
return None
|
|
if len(present) != len(_STRUCTURAL_FRONTMATTER_KEYS):
|
|
missing = [k for k in _STRUCTURAL_FRONTMATTER_KEYS if k not in present]
|
|
raise VerdictFrontmatterError(
|
|
f"a verdict file declares {present} but not {missing}; a verdict carries its whole "
|
|
"structural key or none of it (a partial key belongs to no candidate)"
|
|
)
|
|
measure_type = _unquote(fm["measure_type"])
|
|
if not measure_type:
|
|
raise VerdictFrontmatterError("'measure_type' is declared but empty")
|
|
return ProposalFeatures(
|
|
affected_codes=_parse_affected_codes(fm["affected_codes"]),
|
|
measure_type=measure_type,
|
|
claimed_saving_nok=_parse_claimed_saving(fm["claimed_saving_nok"]),
|
|
description=measure_type,
|
|
)
|
|
|
|
|
|
def _verdict_rationale(fm: dict[str, str]) -> str:
|
|
"""Build the few-shot rationale from a ``type: verdict`` file's frontmatter, carrying the
|
|
learning signal the deterministic validator cannot compute (the realization rate + expected
|
|
actual). This is the ExpeL signal that must reach the next hypothesis."""
|
|
base = fm.get("description", "")
|
|
signal = [
|
|
f"{label}={fm[key]}"
|
|
for key, label in (
|
|
("realization_rate", "realiseringsgrad"),
|
|
("expected_actual_saving_nok", "forventet_faktisk_NOK"),
|
|
)
|
|
if fm.get(key)
|
|
]
|
|
return f"{base} [{'; '.join(signal)}]" if signal else base
|
|
|
|
|
|
def seed_store_from_bundle(bundle_dir: str) -> VerdictStore:
|
|
"""Build a ``VerdictStore`` from an OKF bundle's ``type: verdict`` files. Each verdict carries
|
|
the realization signal in its rationale, and stands in for the durable HITL verdict a real
|
|
expert would supply via the same folder interface (målbilde §3).
|
|
|
|
KEYING (S3.2): a verdict is keyed on ITS OWN candidate when its frontmatter declares the
|
|
structural fields (``affected_codes`` / ``measure_type`` / ``claimed_saving_nok``), and on the
|
|
bundle's IR-projection candidate otherwise. The bundle key alone was single-candidate by
|
|
construction: a bundle holding verdicts about several candidates collapsed them onto one key, so
|
|
a verdict about candidate B scored a perfect match against candidate A's query and could be
|
|
folded into A's hypothesis prompt. The fallback is what keeps every pre-S3.2 seed working
|
|
unchanged; the fields are OPTIONAL, never required."""
|
|
bundle = okf.navigate_bundle(bundle_dir)
|
|
fallback: ProposalFeatures | None = None
|
|
verdicts = []
|
|
for vf in bundle.verdicts:
|
|
features = _features_from_verdict_frontmatter(vf.frontmatter)
|
|
if features is None:
|
|
# Read the IR projection lazily: a bundle whose verdicts all carry their own key does
|
|
# not need one, and this keeps the fallback path's behaviour byte-identical.
|
|
fallback = (
|
|
fallback if fallback is not None else optional_bundle_candidate_features(bundle_dir)
|
|
)
|
|
if fallback is None:
|
|
raise VerdictKeyUnavailable(
|
|
f"{vf.name}: this verdict declares no structural key "
|
|
f"({', '.join(_STRUCTURAL_FRONTMATTER_KEYS)}) and the knowledge base carries no "
|
|
"IR projection to fall back on, so there is no candidate to key it to"
|
|
)
|
|
features = fallback
|
|
verdicts.append(
|
|
capture_verdict(
|
|
features,
|
|
vf.frontmatter.get("decision", "approved"),
|
|
_verdict_rationale(vf.frontmatter),
|
|
)
|
|
)
|
|
return VerdictStore(verdicts=verdicts)
|
|
|
|
|
|
# --- Gated wiki-promotion (Fase 6, Steg 8): output layer -> context layer, HITL-gated ------------
|
|
# målbilde §3 (promoterings-gate) / §6 (kun godkjent kunnskap, aldri rå agent-output; provenance) /
|
|
# §7 (load-bearing: a non-approved verdict must NOT reach the wiki). R4 = optional+gated: this is a
|
|
# PUBLIC opt-in primitive, deliberately NOT wired into run_project — the system reads context; the
|
|
# gate/persona promotes (mirrors write_verdict's role split).
|
|
|
|
_APPROVED_DECISIONS = frozenset({"approved", "approved_with_adjustment"})
|
|
# A FIXED neutral index label carrying NO verdict signal. link_in_index folds it into index.md ->
|
|
# index_summary -> bundle_context verbatim, so passing the rationale here would leak the realization
|
|
# signal into the read-context on a path that bypasses the gate (målbilde §3/§6). Load-bearing:
|
|
# test_step8 Test C goes red if this is replaced by the rationale.
|
|
_PROMOTED_LINK_LABEL = "Promotert ekspert-vurdering (gated)"
|
|
|
|
|
|
class PromotionRefused(Exception):
|
|
"""The gate (målbilde §6): a non-approved verdict was offered for promotion. Fail-closed — the
|
|
wiki receives ONLY human/persona-approved knowledge, never raw agent output (self-contamination).
|
|
"""
|
|
|
|
|
|
def _safe_filename_token(verdict_id: str) -> str:
|
|
"""Turn a verbatim ``Verdict.id`` (arbitrary author string — sentinels, hashes, anything) into a
|
|
filename/link-safe token: keep ``[A-Za-z0-9._-]``, replace the rest with ``-``. A token that is
|
|
only separators/dots (degenerate, e.g. ``".."``) falls back to a content hash. This prevents an
|
|
id with ``/`` (an unnavigable link) or ``..`` from steering the written path — defense beside
|
|
``write_concept_file``'s fail-closed ``safe_resolve``. The original id is kept in frontmatter."""
|
|
token = re.sub(r"[^A-Za-z0-9._-]", "-", verdict_id)
|
|
if not token.strip(".-_"):
|
|
return hashlib.sha256(verdict_id.encode("utf-8")).hexdigest()[:16]
|
|
return token
|
|
|
|
|
|
def promote_verdict(
|
|
bundle_dir: str,
|
|
verdict: Verdict,
|
|
*,
|
|
approver: str,
|
|
experiment: str,
|
|
timestamp: str,
|
|
) -> Path:
|
|
"""Promote an APPROVED verdict from the raw output layer into the OKF context layer (the wiki) as
|
|
a ``type: verdict`` concept file, navigable by the next run's ``seed_store_from_bundle`` (Steg 8,
|
|
målbilde §3/§6/§7). GATE (fail-closed): a verdict whose ``decision`` is not an approval raises
|
|
``PromotionRefused`` and writes/links NOTHING — only human/persona-approved knowledge enters the
|
|
wiki. Provenance-stamped (who/which-experiment/when). ``timestamp`` is a required keyword (no
|
|
wall-clock default) so promotion is deterministic and the stamp reproducible.
|
|
|
|
The promoted file is MINIMAL as to the LEARNING SIGNAL: it does NOT reproduce the hand-authored
|
|
seed's structured signal fields (``realization_rate`` etc.) — the raw ``Verdict`` model carries
|
|
that only as ``rationale`` prose, which becomes the ``description`` frontmatter
|
|
``seed_store_from_bundle`` folds into ExpeL. It DOES carry its own structural KEY (S3.2:
|
|
``affected_codes`` / ``measure_type`` / ``claimed_saving_nok``), so the next run keys it on the
|
|
candidate it is about — a promoted verdict is frequently about a different candidate than the
|
|
one that bundle's IR projection describes. The index link uses a NEUTRAL label
|
|
(``_PROMOTED_LINK_LABEL``), so the signal reaches a prompt only via the gated fold, never via
|
|
``bundle_context`` (§3/§6) — and the structural key is signal-free by construction.
|
|
|
|
Round-trip caveat: ``render_frontmatter`` single-lines every value, so a ``measure_type``
|
|
containing newlines is re-read with those collapsed to spaces and would re-mint a different id.
|
|
Measure strings are single-line in practice; this is stated rather than defended against.
|
|
|
|
Known limitation (mirrors ``write_verdict``): ``_mint_id`` keys on the candidate features, so two
|
|
approved verdicts about the SAME candidate share an id -> share a filename -> last-write-wins;
|
|
the wiki grows one curated verdict file per distinct candidate measure, not per verdict event.
|
|
Returns the written path."""
|
|
if verdict.decision not in _APPROVED_DECISIONS:
|
|
raise PromotionRefused(
|
|
f"refusing to promote a non-approved verdict (decision={verdict.decision!r}); "
|
|
"only human/persona-approved knowledge enters the wiki (målbilde §6)"
|
|
)
|
|
f = verdict.proposal_features
|
|
frontmatter = {
|
|
"type": "verdict",
|
|
"decision": verdict.decision,
|
|
"description": verdict.rationale,
|
|
# S3.2: the promoted file carries its OWN structural key, so the next run's
|
|
# seed_store_from_bundle keys it on the candidate it is actually about rather than on
|
|
# whichever candidate that bundle's IR projection happens to describe. Written as the three
|
|
# fields _features_from_verdict_frontmatter reads back — all three, never a partial key.
|
|
"affected_codes": "[" + ", ".join(sorted(f.affected_codes)) + "]",
|
|
"measure_type": f.measure_type,
|
|
# ``str`` of the raw value, NOT a normalised format: it round-trips through
|
|
# ``_parse_claimed_saving``'s ``json.loads`` back to the same int/float, and ``_mint_id``
|
|
# hashes that raw value (``18000`` and ``18000.0`` are different keys).
|
|
"claimed_saving_nok": str(f.claimed_saving_nok),
|
|
"verdict_id": verdict.id,
|
|
"provenance": f"godkjent av {approver}; eksperiment {experiment}; {timestamp}",
|
|
"timestamp": timestamp,
|
|
# SPEC §5.2 provenance, reusing the EXISTING required ``timestamp`` keyword so no
|
|
# wall-clock default sneaks in. The approver is written verbatim: a scripted stand-in lands
|
|
# in ``machine-confirmed``, which is true of it.
|
|
"verified": okf.verified_field(approver, timestamp),
|
|
"tags": "[verdict, promoted, HITL]",
|
|
}
|
|
codes = ", ".join(sorted(f.affected_codes))
|
|
body = (
|
|
"# Promotert ekspert-vurdering\n\n"
|
|
f"{verdict.rationale}\n\n"
|
|
f"- Tiltak: {f.measure_type}\n"
|
|
f"- Berørte koder: {codes}\n"
|
|
f"- Beslutning: {verdict.decision}\n"
|
|
f"- Provenance: {frontmatter['provenance']}\n"
|
|
)
|
|
filename = f"promoted-verdict-{_safe_filename_token(verdict.id)}.md"
|
|
path = okf.write_concept_file(bundle_dir, filename, frontmatter, body)
|
|
okf.link_in_index(bundle_dir, filename, _PROMOTED_LINK_LABEL)
|
|
return path
|