feat(fase5): add the long/async verdict file inbox (Steg 7 resumable feedback)

The short loop captured the expert verdict inline into an in-memory store, so a
verdict arriving days/weeks later in a separate run could not influence any future
hypothesis (målbilde §5 row 7). Steg 7 adds the long timescale: run_project gains an
opt-in verdict_dir async inbox that load_verdicts_from_dir -> store.add MERGES into the
store BEFORE the Step-1 ExpeL fold, so a verdict dropped after an earlier run reaches a
separate, later run's hypothesis — fully resumable across runs separated in time.

- verdicts.py: verdict_to_dict / verdict_from_dict (id read verbatim, never re-minted),
  write_verdict (public authoring primitive, NOT wired into run_project — system reads
  the folder, expert/persona writes it, §3 role split), tolerant load_verdicts_from_dir
  (missing/foreign/half-written files skipped, not raised — RAW layer per §10 R2),
  VerdictStore.from_dir.
- run.py: verdict_dir kwarg; ingest-merge block after load_contracts (merge not replace
  keeps run_portfolio's cross-project threading; store.add idempotent on content-hash id;
  no change to the fold). CLI --bundle-dir/--verdict-dir thread the long loop to the
  console entry. No auto-persist of the run's own captured verdict (outbox/Steg 8).
- Load-bearing PAIR (test_step7_async_loop_loadbearing.py): a verdict dropped after run A
  must reach run B's prompt (run B uses a FRESH store -> the transfer is the file loop,
  not in-memory carryover); empty-inbox control proves causality. Marker = a realization
  value absent from the bundle (not the seed's 0.82). Proven RED on ingest detach.

Suite 138 -> 140 passed, 4 skipped; mypy + ruff check clean. Målbilde treated as frozen
(no §3/§5/§7 edit). Step 8 (gated wiki promotion) remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MHR8iKxJRxDiDfNw8HZmWE
This commit is contained in:
Kjell Tore Guttormsen 2026-06-30 09:54:23 +02:00
commit e2861cac0c
5 changed files with 297 additions and 6 deletions

View file

@ -56,6 +56,19 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
her) — så koden påstår ikke mer enn den gjør. Load-bearing: her) — så koden påstår ikke mer enn den gjør. Load-bearing:
`tests/test_step5_refine_loadbearing.py` blir rød når reason-injeksjonen detaches (utfallet `tests/test_step5_refine_loadbearing.py` blir rød når reason-injeksjonen detaches (utfallet
flipper aldri + reason-verbatim-asserten faller); kontrollen beviser at løkka forblir bundet. flipper aldri + reason-verbatim-asserten faller); kontrollen beviser at løkka forblir bundet.
- **Lang/async fil-løkke (Steg 7, målbilde §3/§7):** `run_project(verdict_dir=...)` er den lange
tilbakemeldings-tidsskalaen — en ekspert/persona dropper en verdict-fil (vanlig JSON, RAW-laget
per §10 R2) i en inbox-mappe ETTER en kjøring, og en separat, senere kjøring `load_verdicts_from_dir`
`store.add` **merger** den inn FØR Steg-1-folden (ingen endring i folden), så dommen når neste
hypotese. **Rolledeling (§3, ufravikelig):** systemet LESER mappa; eksperten/personaen SKRIVER den
`run_project` persisterer ALDRI sin egen fangede dom tilbake (det er outbox/Steg 8). Merge, aldri
erstatt (`run_portfolio`-tråding intakt); tolerant last (manglende mappe / fremmede / halvskrevne
filer hoppes over, ikke raises — RAW-lag, kontrast `okf.load_ir_projection`s fail-fast); `id` leses
verbatim, re-mintes aldri. `write_verdict` er den offentlige authoring-primitiven (persona/test +
framtidig Steg 8), men wires IKKE inn i `run_project`. Load-bearing:
`tests/test_step7_async_loop_loadbearing.py` — en dom droppet etter Run A MÅ nå Run B's prompt
(Run B bruker FERSK store → overføringen er fil-løkka, ikke in-memory-carryover); tom-inbox-kontroll
beviser kausalitet. Markør = realiseringsverdi som finnes ingen steder i bundelen (ikke frøets 0.82).
- **Kostnadsdisiplin:** utvikle primært på lokal profil (gratis); Foundry/Azure (privat tenant finnes) kun til målrettet, minimal verifisering; billigste modeller + små syntetiske data + harde token-tak. Ingen tunge test-kjøringer. - **Kostnadsdisiplin:** utvikle primært på lokal profil (gratis); Foundry/Azure (privat tenant finnes) kun til målrettet, minimal verifisering; billigste modeller + små syntetiske data + harde token-tak. Ingen tunge test-kjøringer.
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet. - **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase. - Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.

View file

@ -2,7 +2,7 @@
Generic, open framework on **Microsoft Agent Framework (MAF)** for finding cost-savings / efficiency proposals *within* each project of a portfolio of independent projects. Multiple agents collaborate to generate candidate proposals; a mandatory deterministic validator (solver + Monte Carlo) decides the numbers; domain experts review via human-in-the-loop, and the system learns from their verdicts. Generic, open framework on **Microsoft Agent Framework (MAF)** for finding cost-savings / efficiency proposals *within* each project of a portfolio of independent projects. Multiple agents collaborate to generate candidate proposals; a mandatory deterministic validator (solver + Monte Carlo) decides the numbers; domain experts review via human-in-the-loop, and the system learns from their verdicts.
> **Status:** Early development. The deterministic backbone is solid; the agentic learning loop is being wired one load-bearing seam at a time — **Step 1 (OKF-navigated context → hypothesis) is wired**, with the verdict layer kept out of the read-context (see below). Not yet end-to-end usable. > **Status:** Early development. The deterministic backbone is solid; the agentic learning loop is being wired one load-bearing seam at a time — **Steps 1, 3/4, 5 and 7 are wired** (OKF-navigated context, checker gate, informed refinement, and the async file feedback loop; see below). Step 8 (gated wiki promotion) remains. Not yet end-to-end usable.
> **Disclaimer — technical framework only.** This project is a *technical framework*. Organizations that deploy it are themselves responsible for ensuring a valid processing purpose and for any required assessments (DPIA, risk/ROS, security reviews, etc.). The framework ships technical affordances (local-only mode, provenance/audit logging, no silent data egress) to *enable* compliant use, but makes no compliance guarantees. > **Disclaimer — technical framework only.** This project is a *technical framework*. Organizations that deploy it are themselves responsible for ensuring a valid processing purpose and for any required assessments (DPIA, risk/ROS, security reviews, etc.). The framework ships technical affordances (local-only mode, provenance/audit logging, no silent data egress) to *enable* compliant use, but makes no compliance guarantees.
@ -20,7 +20,8 @@ The mandatory deterministic backbone (validator + budget meter + provenance) is
- Two load-bearing tests fail when the seam is detached: the realization signal must reach the prompt via the fold (`tests/test_step1_expel_loadbearing.py`), and must never leak via context (`tests/test_okf.py::test_bundle_context_excludes_verdict_layer`). - Two load-bearing tests fail when the seam is detached: the realization signal must reach the prompt via the fold (`tests/test_step1_expel_loadbearing.py`), and must never leak via context (`tests/test_okf.py::test_bundle_context_excludes_verdict_layer`).
- **Steps 3/4 — checker gates the reasoning (wired).** Two falsifiers now act on the same candidate: the deterministic validator gates the **numbers** (blocking, as before), and the maker-checker's checker gates the **reasoning** (target picture §2/§6). The checker ends its turn with a `VERDICT: APPROVE` / `VERDICT: REJECT — <reason>` line; an explicit reject blocks an otherwise-validated proposal (`run_project` surfaces both debate participants via `output_from=agents` and overrides the outcome to a checker-sourced `Rejection`). The gate is opt-in-reject (fail-open on a missing marker), and `provenance.validator_decision` stays honest — it reflects the validator only, never the checker. Load-bearing: `tests/test_checker_gate_loadbearing.py` goes red on *either* detach (revert `output_from`, or drop the override). - **Steps 3/4 — checker gates the reasoning (wired).** Two falsifiers now act on the same candidate: the deterministic validator gates the **numbers** (blocking, as before), and the maker-checker's checker gates the **reasoning** (target picture §2/§6). The checker ends its turn with a `VERDICT: APPROVE` / `VERDICT: REJECT — <reason>` line; an explicit reject blocks an otherwise-validated proposal (`run_project` surfaces both debate participants via `output_from=agents` and overrides the outcome to a checker-sourced `Rejection`). The gate is opt-in-reject (fail-open on a missing marker), and `provenance.validator_decision` stays honest — it reflects the validator only, never the checker. Load-bearing: `tests/test_checker_gate_loadbearing.py` goes red on *either* detach (revert `output_from`, or drop the override).
- **Step 5 — informed refinement (wired).** The proposer's bounded retry is no longer blind: the validator's *previous* `Rejection.reason` is fed into the next attempt's prompt (`generate.py` `generate_via_llm``_build_messages(prior_rejection=...)`), so the model corrects against the falsification instead of re-answering identically (target picture §5/§7). It carries only the most-recent reason (never an accumulated history, never the rejected proposal JSON) and runs under the *existing* `max_attempts` + token-meter cap — no new loop, so "improve until good enough" without a ceiling stays impossible. The only per-attempt falsifier here is the validator; seeding generation with the checker's critique is a run-level, separately-scoped concern and is deliberately not done here. Load-bearing: `tests/test_step5_refine_loadbearing.py` goes red when the reason is detached from the prompt (the outcome never flips *and* the verbatim-reason assertion fails), with a bounded control proving the loop still stops at `max_attempts`. - **Step 5 — informed refinement (wired).** The proposer's bounded retry is no longer blind: the validator's *previous* `Rejection.reason` is fed into the next attempt's prompt (`generate.py` `generate_via_llm``_build_messages(prior_rejection=...)`), so the model corrects against the falsification instead of re-answering identically (target picture §5/§7). It carries only the most-recent reason (never an accumulated history, never the rejected proposal JSON) and runs under the *existing* `max_attempts` + token-meter cap — no new loop, so "improve until good enough" without a ceiling stays impossible. The only per-attempt falsifier here is the validator; seeding generation with the checker's critique is a run-level, separately-scoped concern and is deliberately not done here. Load-bearing: `tests/test_step5_refine_loadbearing.py` goes red when the reason is detached from the prompt (the outcome never flips *and* the verbatim-reason assertion fails), with a bounded control proving the loop still stops at `max_attempts`.
- **Steps 78** (async file feedback, gated wiki promotion) — not yet wired. - **Step 7 — async file feedback loop (wired).** `run_project(..., verdict_dir=...)` adds the long feedback timescale (target picture §3/§7): an expert/persona drops a verdict file (plain JSON — the raw output layer, §10 R2) into an inbox folder *after* a run, and a separate, later run ingests it — merged into the store *before* the Step-1 fold — so a verdict that landed out of band reaches the next hypothesis. The loop is fully resumable across runs separated in time; no live session is assumed. The system *reads* the folder, the expert/persona *writes* it (§3 role split), so `run_project` deliberately does **not** persist its own captured verdict back (that is the outbox / Step-8 concern). Ingestion is tolerant (a missing folder, foreign or half-written files are skipped, not raised) and **merges** (never replaces), preserving `run_portfolio`'s cross-project store. Reachable from the CLI via `--bundle-dir --verdict-dir`. Load-bearing: `tests/test_step7_async_loop_loadbearing.py` — a verdict dropped after run A must reach run B's prompt (run B uses a *fresh* store, so the transfer is the file loop, not in-memory carryover), with an empty-inbox control proving causality.
- **Step 8** (gated wiki promotion) — not yet wired.
## Docs ## Docs

View file

@ -53,6 +53,7 @@ from portfolio_optimiser.verdicts import (
VerdictStore, VerdictStore,
bundle_candidate_features, bundle_candidate_features,
capture_verdict, capture_verdict,
load_verdicts_from_dir,
) )
from portfolio_optimiser.workflow import fresh_workflow from portfolio_optimiser.workflow import fresh_workflow
@ -198,6 +199,7 @@ async def run_project(
verdict_input: dict[str, str], verdict_input: dict[str, str],
bundle_dir: str | None = None, bundle_dir: str | None = None,
store: VerdictStore | None = None, store: VerdictStore | None = None,
verdict_dir: str | None = None,
client_factory: Callable[[str], BaseChatClient] | None = None, client_factory: Callable[[str], BaseChatClient] | None = None,
max_rounds: int = 3, max_rounds: int = 3,
max_tokens: int = 100_000, max_tokens: int = 100_000,
@ -210,9 +212,13 @@ async def run_project(
(defaults to the real backend). ``verdict_input`` carries the expert decision/rationale (defaults to the real backend). ``verdict_input`` carries the expert decision/rationale
(Layer-2). ``bundle_dir`` (Fase 2a) makes the run OKF-bundle-driven: the project is derived (Layer-2). ``bundle_dir`` (Fase 2a) makes the run OKF-bundle-driven: the project is derived
from the bundle and, before generation, the candidate's prior verdicts in ``store`` are folded from the bundle and, before generation, the candidate's prior verdicts in ``store`` are folded
into the hypothesis prompt (Step-1 ExpeL wiring, målbilde §5/§7). Raises into the hypothesis prompt (Step-1 ExpeL wiring, målbilde §5/§7). ``verdict_dir`` (Fase 5,
``pydantic.ValidationError`` on a bad contract and ``BudgetExceeded`` when the token/round cap Steg 7, målbilde §3/§7) is the async file inbox: a folder of expert/persona-authored verdict
is crossed.""" files (plain JSON, R2 raw layer) MERGED into the store BEFORE the Step-1 fold, so a verdict
dropped after an earlier run is consumed by this separate, later run the long feedback loop,
fully resumable across runs separated in time. The system READS this folder; it does not write
to it (the expert/persona writes, målbilde §3). Raises ``pydantic.ValidationError`` on a bad
contract and ``BudgetExceeded`` when the token/round cap is crossed."""
# 1. Fail-fast: validate ALL contracts (incl. the verdict-feedback shape) before any client. # 1. Fail-fast: validate ALL contracts (incl. the verdict-feedback shape) before any client.
load_contracts( load_contracts(
{"docs_dir": docs_dir, "top_k": top_k}, {"docs_dir": docs_dir, "top_k": top_k},
@ -220,6 +226,15 @@ async def run_project(
verdict_input, verdict_input,
) )
# 1b. Long loop (Steg 7): ingest the async verdict inbox INTO the store before the Step-1 fold.
# Merge (not replace) into the passed store so run_portfolio's cross-project threading stays
# intact; store.add is idempotent on the content-hash id. A verdict that landed after an earlier
# run thus reaches THIS run's hypothesis via the existing fold below — no change to the fold.
if verdict_dir is not None:
store = store if store is not None else VerdictStore(verdicts=[])
for dropped in load_verdicts_from_dir(verdict_dir):
store.add(dropped)
# 2-3. Project + agent read-context + first-class citations. A bundle run derives ALL THREE from # 2-3. Project + agent read-context + first-class citations. A bundle run derives ALL THREE from
# the navigated OKF bundle via progressive disclosure (verdict layer EXCLUDED — målbilde §2/§4), # the navigated OKF bundle via progressive disclosure (verdict layer EXCLUDED — målbilde §2/§4),
# NOT keyword chunk-stuffing; the road path keeps the chunk-retrieval data source. ``debate_tools`` # NOT keyword chunk-stuffing; the road path keeps the chunk-retrieval data source. ``debate_tools``
@ -406,6 +421,15 @@ def main(argv: list[str] | None = None) -> int:
parser.add_argument("project_id") parser.add_argument("project_id")
parser.add_argument("--profile", default="local") parser.add_argument("--profile", default="local")
parser.add_argument("--docs-dir", required=True) parser.add_argument("--docs-dir", required=True)
parser.add_argument(
"--bundle-dir", default=None, help="OKF bundle dir (enables the Step-1 fold)"
)
parser.add_argument(
"--verdict-dir",
default=None,
help="async verdict inbox: a folder of dropped expert verdicts, ingested before generation "
"(the long loop — a verdict that landed after an earlier run is consumed by this run)",
)
parser.add_argument("--decision", default="approved", choices=["approved", "rejected"]) parser.add_argument("--decision", default="approved", choices=["approved", "rejected"])
parser.add_argument("--rationale", default="reviewed by expert") parser.add_argument("--rationale", default="reviewed by expert")
args = parser.parse_args(argv) args = parser.parse_args(argv)
@ -415,6 +439,8 @@ def main(argv: list[str] | None = None) -> int:
args.project_id, args.project_id,
args.profile, args.profile,
docs_dir=args.docs_dir, docs_dir=args.docs_dir,
bundle_dir=args.bundle_dir,
verdict_dir=args.verdict_dir,
verdict_input={"decision": args.decision, "rationale": args.rationale}, verdict_input={"decision": args.decision, "rationale": args.rationale},
) )
) )

View file

@ -21,6 +21,7 @@ from __future__ import annotations
import hashlib import hashlib
import json import json
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path
from typing import Any from typing import Any
from agent_framework import ContextProvider, SessionContext from agent_framework import ContextProvider, SessionContext
@ -104,12 +105,111 @@ def capture_verdict(features: ProposalFeatures, decision: str, rationale: str) -
) )
# --- Async file inbox (Fase 5, Steg 7): plain-JSON verdict serialization + tolerant folder load ---
# The raw output layer is a plain JSON store (R2), NOT OKF — OKF is reserved for the promoted
# (Step-8) layer. One verdict per file lets an expert/persona drop verdicts into a folder out of
# band, and a separate later run pick them up (målbilde §3 long loop).
_REQUIRED_VERDICT_KEYS = {"id", "decision", "rationale", "proposal_features"}
def verdict_to_dict(verdict: Verdict) -> dict[str, Any]:
"""Serialize a ``Verdict`` to a JSON-ready dict. ``affected_codes`` (a frozenset) is emitted as
a SORTED list matching ``_mint_id``'s canonical form — so the round-trip is lossless and the
on-disk form is deterministic."""
f = verdict.proposal_features
return {
"id": verdict.id,
"decision": verdict.decision,
"rationale": verdict.rationale,
"proposal_features": {
"affected_codes": sorted(f.affected_codes),
"measure_type": f.measure_type,
"claimed_saving_nok": f.claimed_saving_nok,
"description": f.description,
},
}
def verdict_from_dict(data: dict[str, Any]) -> Verdict:
"""Reconstruct a ``Verdict`` from its serialized dict. The ``id`` is read VERBATIM — never
re-minted so a verdict authored elsewhere keeps its identity and the int/float magnitudes in
its features survive the JSON round-trip untouched (re-minting would re-hash and could diverge,
since ``_mint_id`` hashes the raw value, e.g. ``30000`` vs ``30000.0``)."""
pf = data["proposal_features"]
return Verdict(
id=data["id"],
proposal_features=ProposalFeatures(
affected_codes=frozenset(pf["affected_codes"]),
measure_type=pf["measure_type"],
claimed_saving_nok=pf["claimed_saving_nok"],
description=pf.get("description", ""),
),
decision=data["decision"],
rationale=data["rationale"],
)
def write_verdict(directory: str, verdict: Verdict) -> Path:
"""Write one verdict as ``{id}.json`` into ``directory`` (created if needed) and return the path.
This is the public authoring primitive the expert/persona (sim) or human (prod) uses to drop a
verdict into the async inbox via the SAME folder interface and the writer a future Step-8
promotion would reuse. It is deliberately NOT called inside ``run_project``: the system READS the
inbox, it does not write to it (målbilde §3 role split).
Limitation: ``id`` is a content hash of the FEATURES only (``_mint_id``), so two verdicts with
identical features but different decisions share a filename last write wins on disk (mirroring
``VerdictStore.add``'s first-wins in memory). Acceptable for the raw MVP layer."""
path = Path(directory)
path.mkdir(parents=True, exist_ok=True)
target = path / f"{verdict.id}.json"
target.write_text(
json.dumps(verdict_to_dict(verdict), sort_keys=True, indent=2), encoding="utf-8"
)
return target
def load_verdicts_from_dir(directory: str) -> list[Verdict]:
"""Load every well-formed verdict JSON file from an async inbox folder, TOLERANTLY (OKF §4
spirit): a missing folder yields ``[]``; files that are not ``.json``, fail to parse, or lack a
required key are SKIPPED, never raised. The folder is written out of band by an external party,
so half-written or foreign files (e.g. a bundle's ``golden.json``) are realistic — this is the
raw layer, not required input (contrast ``okf.load_ir_projection``'s fail-fast). Order is
deterministic (sorted by filename)."""
path = Path(directory)
if not path.is_dir():
return []
verdicts: list[Verdict] = []
for file in sorted(path.glob("*.json")):
try:
data = json.loads(file.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
continue
if not isinstance(data, dict) or not _REQUIRED_VERDICT_KEYS <= data.keys():
continue
try:
verdicts.append(verdict_from_dict(data))
except (KeyError, TypeError):
continue
return verdicts
@dataclass @dataclass
class VerdictStore: class VerdictStore:
"""Minimal in-memory store of historical verdicts (MVP — no file persistence).""" """Minimal store of historical verdicts. In-memory by default; ``from_dir`` / the
``load_verdicts_from_dir`` merge primitive back it with the async file inbox (Steg 7)."""
verdicts: list[Verdict] verdicts: list[Verdict]
@classmethod
def from_dir(cls, directory: str) -> VerdictStore:
"""Build a store from an async inbox folder (convenience constructor). ``run_project`` uses
the merge primitive (``add`` per loaded verdict) instead, to preserve a passed store's
existing verdicts (the cross-project threading in ``run_portfolio``); this is for callers
that want a fresh store straight from a folder."""
return cls(verdicts=load_verdicts_from_dir(directory))
def retrieve(self, query: ProposalFeatures, k: int) -> list[Verdict]: def retrieve(self, query: ProposalFeatures, k: int) -> list[Verdict]:
"""Return the top-``k`` verdicts by structural similarity. Deterministic: ties break """Return the top-``k`` verdicts by structural similarity. Deterministic: ties break
by verdict id, so ordering is stable across runs.""" by verdict id, so ordering is stable across runs."""

View file

@ -0,0 +1,151 @@
"""Step-7 load-bearing seam (målbilde §3 / §5 row 7 / §7): the LONG/async feedback loop — a
verdict file that lands in a folder AFTER one run MUST be consumed by a separate, later run.
The gap (fot-i-bakken, verified in code): the short loop captured the expert verdict inline
(``verdict_input``) into an IN-MEMORY ``VerdictStore`` so a verdict that arrives days/weeks
later, in a separate run, could not influence any future hypothesis. Fase 5 adds a file inbox:
``run_project(verdict_dir=...)`` ingests expert/persona-authored verdict files (plain JSON, R2 raw
layer) and MERGES them into the store BEFORE the Step-1 ExpeL fold, so the dropped verdict reaches
the next run's hypothesis prompt through the existing fold.
These two tests are a load-bearing PAIR (per the Fase-2 green-but-dead trap):
- the positive test keys on a realization marker that occurs NOWHERE in the bundle (0.57, not the
bundle seed's 0.82), carried in the dropped verdict's rationale proving the learning PAYLOAD
round-tripped end to end (write -> file -> ingest -> retrieve -> fold -> prompt), not merely that
an object was retrieved. It goes RED the moment the ingest is detached.
- the control proves causality: the SAME run with an EMPTY inbox carries no marker so the signal
is caused by the file loop, not incidental to the bundle. ``bundle_context`` excludes
``type: verdict`` files (okf.py), so the marker can ONLY arrive via ingest -> store -> fold.
Critical invariant: run B uses a FRESH store (never run A's object) — otherwise the transfer would
be in-memory carryover, not the file loop.
"""
from __future__ import annotations
from pathlib import Path
from portfolio_optimiser.run import run_project
from portfolio_optimiser.validator import ValidatedProposal
from portfolio_optimiser.verdicts import (
Verdict,
VerdictStore,
bundle_candidate_features,
write_verdict,
)
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
# A VALID BYGG-KONTOR-NORD proposal (same as the Step-1 test): P90 = 0.30 x 300000 = 90000 >=
# claimed 30000 -> validates on the first attempt.
_ENERGY_REPLY = (
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
)
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
# Two load-bearing markers carried by the DROPPED verdict, by design:
# - a realization value absent from EVERY bundle file (the bundle seed is 0.82). It can reach a
# prompt only via the ingest->fold path -> isolates the file loop AND would catch a regression
# where bundle navigation starts leaking verdict files (a 0.82 marker would pass for the wrong
# reason and mask that).
# - a sentinel verdict id (cannot appear in the bundle's raw text) for object identity.
_MARKER = "realiseringsgrad=0.57"
_DROP_ID = "STEG7-DROP"
def _generation_prompts(sink: list[str]) -> list[str]:
"""The generation-call prompts (``generate._build_messages`` embeds 'SavingsProposal'),
isolated from the debate-round prompts also captured in the sink."""
return [p for p in sink if "SavingsProposal" in p]
def _drop_expert_verdict(inbox: Path) -> None:
"""Simulate the long loop: an expert/persona drops a verdict file into the inbox AFTER a run.
Keyed on the bundle candidate's features (the same key the Step-1 fold queries on), carrying the
realization marker in its rationale. ``write_verdict`` is the public authoring primitive a
persona (sim) or human (prod) uses against the SAME folder interface."""
verdict = Verdict(
id=_DROP_ID,
proposal_features=bundle_candidate_features(str(BUNDLE_DIR)),
decision="approved",
rationale=f"LED-retrofit godkjent med realiseringskorreksjon ({_MARKER})",
)
write_verdict(str(inbox), verdict)
async def test_dropped_verdict_reaches_later_run(make_recording_client_factory, tmp_path) -> None:
"""LOAD-BEARING: a verdict dropped into the inbox AFTER run A reaches run B's hypothesis prompt.
Goes RED if ``run_project`` stops ingesting ``verdict_dir`` before the Step-1 fold (the marker
never arrives -> no fold -> the assertion fails)."""
inbox = tmp_path / "verdict-inbox"
inbox.mkdir()
# Run A: an earlier run with an EMPTY store and an EMPTY inbox -> nothing folds in, and (no
# auto-persist) run A does NOT write its own captured verdict back to the inbox.
factory_a, recorded_a = make_recording_client_factory(_ENERGY_REPLY)
await run_project(
"BYGG-KONTOR-NORD",
"local",
docs_dir=str(BUNDLE_DIR),
bundle_dir=str(BUNDLE_DIR),
verdict_input=_VERDICT_INPUT,
store=VerdictStore(verdicts=[]),
verdict_dir=str(inbox),
client_factory=factory_a,
)
assert all(_MARKER not in p for p in recorded_a), "precondition: run A carries no marker"
# Days/weeks later, out of band: the expert drops a verdict into the inbox folder.
_drop_expert_verdict(inbox)
# Run B: a SEPARATE, later run with a FRESH store (never run A's object) -> the dropped verdict
# must reach the hypothesis prompt purely via the file loop.
factory_b, recorded_b = make_recording_client_factory(_ENERGY_REPLY)
result = await run_project(
"BYGG-KONTOR-NORD",
"local",
docs_dir=str(BUNDLE_DIR),
bundle_dir=str(BUNDLE_DIR),
verdict_input=_VERDICT_INPUT,
store=VerdictStore(verdicts=[]),
verdict_dir=str(inbox),
client_factory=factory_b,
)
assert isinstance(result.outcome, ValidatedProposal)
gen_prompts = _generation_prompts(recorded_b)
assert gen_prompts, "the generation call must have happened"
assert any(_MARKER in p for p in gen_prompts), (
"the dropped verdict's realization signal did not reach run B's hypothesis prompt — the "
"async file inbox is not ingested before the Step-1 fold"
)
assert any(_DROP_ID in p for p in gen_prompts), (
"the dropped verdict's id did not reach the prompt"
)
async def test_empty_inbox_leaves_no_signal(make_recording_client_factory, tmp_path) -> None:
"""CAUSALITY CONTROL: the same run with an EMPTY inbox carries no marker into any prompt —
proving the positive test's signal is caused by the file loop, not incidental to the bundle
(``bundle_context`` excludes ``type: verdict`` files, so the marker has no other path)."""
inbox = tmp_path / "verdict-inbox"
inbox.mkdir()
factory, recorded = make_recording_client_factory(_ENERGY_REPLY)
await run_project(
"BYGG-KONTOR-NORD",
"local",
docs_dir=str(BUNDLE_DIR),
bundle_dir=str(BUNDLE_DIR),
verdict_input=_VERDICT_INPUT,
store=VerdictStore(verdicts=[]),
verdict_dir=str(inbox),
client_factory=factory,
)
assert all(_MARKER not in p for p in recorded), (
"the realization marker reached a prompt with an EMPTY inbox — the positive test is not "
"load-bearing (the signal is incidental, not caused by the ingest seam)"
)
assert all(_DROP_ID not in p for p in recorded), "the sentinel id leaked without a dropped file"