feat(fase2): wire Step-1 ExpeL retrieval into the hypothesis prompt
Closes maalbilde §5 gap #1 (the one missing "feedback-into-prompt" dataflow) for the OKF-bundle path. Before, ExpeL was computed AFTER generation into a discarded SessionContext, so a prior verdict could not influence any hypothesis (context_providers=0). - New okf.py: framework-neutral OKF bundle navigation (index + frontmatter + cross-links), pure stdlib, no agent_framework/mcp (D7-portable), enforced by test_okf_is_maf_free. - verdicts.py: seed_store_from_bundle + bundle_candidate_features build the ExpeL substrate + the pre-hypothesis query key from a bundle. - run_project(bundle_dir=...): folds the candidate's prior verdicts into the generation context BEFORE generate_via_llm; the road path is unchanged. Load-bearing (maalbilde §7): test_step1_expel_loadbearing proves a prior verdict reaches the hypothesis prompt and goes RED when the fold is detached (shown via TDD red->green). The marker is the minted verdict id (content hash) because docs_dir==bundle_dir lets keyword chunk-stuffing leak the realization rate; clean layer separation is Fase 2b. Suite 121->133 passed; mypy + ruff check clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MHR8iKxJRxDiDfNw8HZmWE
This commit is contained in:
parent
2b1a1832ef
commit
d6d83d42b5
9 changed files with 529 additions and 7 deletions
|
|
@ -25,6 +25,10 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
- **Rent teknisk rammeverk:** deployer eier DPIA/ROS/behandlingsformål. Bygg IKKE compliance-funksjoner — kun tekniske forutsetninger (lokal-only, provenance, ingen stille egress) + disclaimer.
|
||||
- **90%-prinsipp:** bygg den generiske kjernen + tydelige extension points; jakt IKKE de siste 10 %.
|
||||
- **Deterministisk validator er obligatorisk og blokkerende** — aldri valgfri plugin.
|
||||
- **Framework-nøytral kontekst-søm:** OKF-bundle-navigasjon (`okf.py`) og den delte
|
||||
`shared/`-kjernen er ren stdlib — null `agent_framework`/`mcp`-import, så samme bundles
|
||||
konsumeres uendret av begge stacker (D7-portabel). Håndhevet av
|
||||
`tests/test_okf.py::test_okf_is_maf_free`; importér aldri MAF inn i kontekst-laget.
|
||||
- **Stoppkriterier + budsjett-tak påkrevd ved oppstart** (fail-fast, aldri ubegrenset loop).
|
||||
- **Group Chat maker-checker** som debatt-default (IKKE Magentic, som er eksperimentell).
|
||||
- **Kostnadsdisiplin:** utvikle primært på lokal profil (gratis); Foundry/Azure (privat tenant finnes) kun til målrettet, minimal verifisering; billigste modeller + små syntetiske data + harde token-tak. Ingen tunge test-kjøringer.
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Generic, open framework on **Microsoft Agent Framework (MAF)** for finding cost-savings / efficiency proposals *within* each project of a portfolio of independent projects. Multiple agents collaborate to generate candidate proposals; a mandatory deterministic validator (solver + Monte Carlo) decides the numbers; domain experts review via human-in-the-loop, and the system learns from their verdicts.
|
||||
|
||||
> **Status:** Early development (plan phase). Not yet usable.
|
||||
> **Status:** Early development. The deterministic backbone is solid; the agentic learning loop is being wired one load-bearing seam at a time — **Step 1 (OKF context → hypothesis) is wired** (see below). Not yet end-to-end usable.
|
||||
|
||||
> **Disclaimer — technical framework only.** This project is a *technical framework*. Organizations that deploy it are themselves responsible for ensuring a valid processing purpose and for any required assessments (DPIA, risk/ROS, security reviews, etc.). The framework ships technical affordances (local-only mode, provenance/audit logging, no silent data egress) to *enable* compliant use, but makes no compliance guarantees.
|
||||
|
||||
|
|
@ -10,6 +10,13 @@ Generic, open framework on **Microsoft Agent Framework (MAF)** for finding cost-
|
|||
|
||||
The result will never fit any single customer 100%. The goal is a **~90% genuinely generic core plus clear extension points**, so competent people can configure the last mile per customer. We deliberately do not chase the final 10%.
|
||||
|
||||
## Agentic loop — wiring status
|
||||
|
||||
The mandatory deterministic backbone (validator + budget meter + provenance) is solid and load-bearing. The agentic learning loop (see the [target picture](docs/plan/2026-06-26-maalbilde-agentic-loop.md) §11) is wired one seam at a time:
|
||||
|
||||
- **Step 1 — OKF context → hypothesis (wired).** `run_project(..., bundle_dir=...)` navigates a project's [OKF bundle](shared/examples/bygg-energi-mikro/) and folds the candidate's prior expert verdicts (ExpeL retrieval) into the hypothesis prompt *before* generation — so a prior verdict provably influences the next hypothesis. Guarded by a load-bearing test that fails when the seam is detached (`tests/test_step1_expel_loadbearing.py`).
|
||||
- **Steps 3–8** (checker gating, informed refinement, async file feedback, gated wiki promotion) — not yet wired.
|
||||
|
||||
## Docs
|
||||
|
||||
- [`docs/plan/2026-06-26-maalbilde-agentic-loop.md`](docs/plan/2026-06-26-maalbilde-agentic-loop.md) — target picture: the agentic cost-saving loop + OKF knowledge architecture (north star).
|
||||
|
|
|
|||
135
src/portfolio_optimiser/okf.py
Normal file
135
src/portfolio_optimiser/okf.py
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
"""OKF (Open Knowledge Format) bundle navigation — framework-neutral, D7-portable context seam.
|
||||
|
||||
Reads a bundle the way OKF intends (progressive disclosure): start at ``index.md``, follow
|
||||
intra-bundle cross-links, parse each file's YAML frontmatter, classify by the one required
|
||||
``type`` field. **NO** ``agent_framework``, **NO** ``mcp`` — pure stdlib, so the SAME navigation
|
||||
serves both the MAF and the Claude-SDK implementations unchanged (målbilde §4 vendor-neutrality).
|
||||
|
||||
Robustness is part of the spec (OKF SPEC §4): consumers MUST tolerate broken links and unknown
|
||||
fields. A link to a missing file — or one escaping the bundle — is silently skipped, never raised.
|
||||
Path-safety reuses ``retrieval.safe_resolve`` (also pure stdlib): each cross-link is canonicalised
|
||||
and boundary-checked against the bundle dir, fail-closed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from portfolio_optimiser.retrieval import PathSecurityError, safe_resolve
|
||||
|
||||
_INDEX_NAME = "index.md"
|
||||
_IR_PROJECTION = "validator-input.json"
|
||||
# Intra-bundle markdown cross-links: ``](target.md)``. Targets with a path separator (``/``) are
|
||||
# treated as out-of-bundle and skipped (only same-dir bundle files are navigated).
|
||||
_LINK_RE = re.compile(r"\]\(([^)]+\.md)\)")
|
||||
|
||||
|
||||
def parse_frontmatter(path: str | Path) -> dict[str, str]:
|
||||
"""Read the leading ``---``-delimited YAML frontmatter block as key:value strings.
|
||||
|
||||
Minimal by design (no ``yaml`` dependency): enough for the one required ``type`` field and the
|
||||
verdict's scalar fields. List values (``tags: [...]``) are kept verbatim; unknown fields are
|
||||
preserved (OKF SPEC §4). Returns ``{}`` when there is no frontmatter block."""
|
||||
lines = Path(path).read_text(encoding="utf-8").splitlines()
|
||||
if not lines or lines[0].strip() != "---":
|
||||
return {}
|
||||
fm: dict[str, str] = {}
|
||||
for line in lines[1:]:
|
||||
if line.strip() == "---":
|
||||
break
|
||||
key, sep, val = line.partition(":")
|
||||
if sep:
|
||||
fm[key.strip()] = val.strip()
|
||||
return fm
|
||||
|
||||
|
||||
def _read_body(path: Path) -> str:
|
||||
"""The markdown body after the frontmatter block (or the whole file if there is none)."""
|
||||
lines = path.read_text(encoding="utf-8").splitlines(keepends=True)
|
||||
if lines and lines[0].strip() == "---":
|
||||
for i in range(1, len(lines)):
|
||||
if lines[i].strip() == "---":
|
||||
return "".join(lines[i + 1 :]).lstrip("\n")
|
||||
return "".join(lines)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BundleFile:
|
||||
"""One OKF file: its name, declared ``type`` (``""`` if absent), frontmatter, and body."""
|
||||
|
||||
name: str
|
||||
type: str
|
||||
frontmatter: dict[str, str]
|
||||
body: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Bundle:
|
||||
"""A navigated OKF bundle: ``index.md`` plus every cross-linked file that resolves."""
|
||||
|
||||
dir: str
|
||||
files: tuple[BundleFile, ...]
|
||||
|
||||
@property
|
||||
def index_summary(self) -> str:
|
||||
"""The index body — the progressive-disclosure entry point (not whole-bundle stuffing)."""
|
||||
return next((f.body for f in self.files if f.name == _INDEX_NAME), "")
|
||||
|
||||
@property
|
||||
def verdicts(self) -> list[BundleFile]:
|
||||
"""Every ``type: verdict`` file (the ExpeL seeds the Step-1 wiring retrieves)."""
|
||||
return [f for f in self.files if f.type == "verdict"]
|
||||
|
||||
@property
|
||||
def hypothesis(self) -> BundleFile | None:
|
||||
"""The candidate ``type: hypothesis`` file, if present."""
|
||||
return next((f for f in self.files if f.type == "hypothesis"), None)
|
||||
|
||||
|
||||
def _load_file(bundle_dir: str, name: str) -> BundleFile | None:
|
||||
"""Resolve ``name`` within ``bundle_dir`` and read it, or ``None`` if missing / escaping the
|
||||
bundle (OKF §4 broken-link tolerance + fail-closed path-safety)."""
|
||||
try:
|
||||
resolved = Path(safe_resolve(bundle_dir, name))
|
||||
except PathSecurityError:
|
||||
return None
|
||||
if not resolved.is_file():
|
||||
return None
|
||||
fm = parse_frontmatter(resolved)
|
||||
return BundleFile(name=name, type=fm.get("type", ""), frontmatter=fm, body=_read_body(resolved))
|
||||
|
||||
|
||||
def navigate_bundle(bundle_dir: str) -> Bundle:
|
||||
"""Navigate the OKF bundle from ``index.md``: parse the index, follow its intra-bundle ``.md``
|
||||
cross-links, and read each linked file's frontmatter + body. Deterministic: index first, then
|
||||
links in first-seen order, de-duplicated. Broken / escaping links are skipped (§4). Raises
|
||||
``ValueError`` only when ``index.md`` itself is unreadable (a bundle has no entry point)."""
|
||||
index = _load_file(bundle_dir, _INDEX_NAME)
|
||||
if index is None:
|
||||
raise ValueError(f"OKF bundle has no readable {_INDEX_NAME}: {bundle_dir!r}")
|
||||
files: list[BundleFile] = [index]
|
||||
seen = {_INDEX_NAME}
|
||||
for target in _LINK_RE.findall(index.body):
|
||||
if "/" in target or target in seen:
|
||||
continue # only same-dir bundle files; de-dup repeated links
|
||||
seen.add(target)
|
||||
linked = _load_file(bundle_dir, target)
|
||||
if linked is not None:
|
||||
files.append(linked)
|
||||
return Bundle(dir=bundle_dir, files=tuple(files))
|
||||
|
||||
|
||||
def load_ir_projection(bundle_dir: str, name: str = _IR_PROJECTION) -> dict[str, Any]:
|
||||
"""Load the bundle's IR projection (``validator-input.json`` by default): the candidate
|
||||
measure's cost-IR (``measure``, ``affected_items``, ``claimed_saving_nok``) — the
|
||||
pre-hypothesis ExpeL query-key source. Raises if missing / escaping the bundle (fail-fast: it
|
||||
is required input, not an optional cross-link)."""
|
||||
resolved = Path(safe_resolve(bundle_dir, name))
|
||||
if not resolved.is_file():
|
||||
raise FileNotFoundError(f"IR projection not found in bundle: {name!r}")
|
||||
data: dict[str, Any] = json.loads(resolved.read_text(encoding="utf-8"))
|
||||
return data
|
||||
|
|
@ -44,11 +44,13 @@ from portfolio_optimiser.ir import SavingsProposal
|
|||
from portfolio_optimiser.provenance import ProvenanceStamp
|
||||
from portfolio_optimiser.reference_domain import Project, load_reference_projects
|
||||
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
||||
from portfolio_optimiser import okf
|
||||
from portfolio_optimiser.verdicts import (
|
||||
ExpeLContextProvider,
|
||||
ProposalFeatures,
|
||||
Verdict,
|
||||
VerdictStore,
|
||||
bundle_candidate_features,
|
||||
capture_verdict,
|
||||
)
|
||||
from portfolio_optimiser.workflow import fresh_workflow
|
||||
|
|
@ -114,6 +116,34 @@ def _project_by_id(project_id: str) -> Project:
|
|||
raise ValueError(f"unknown project_id: {project_id!r}")
|
||||
|
||||
|
||||
def _project_from_bundle(bundle_dir: str, project_id: str) -> Project:
|
||||
"""Derive a minimal ``Project`` from an OKF bundle (so a bundle the loop runs need NOT be a
|
||||
road reference-domain project). Only ``id`` + ``name`` reach the generation prompt
|
||||
(``generate._build_messages``), so ``cost_items`` is empty and ``verdict_input`` is unused here
|
||||
(the Layer-2 decision flows via the ``verdict_input`` argument). Fail-fast: the bundle's IR
|
||||
``project_id`` must match the requested id."""
|
||||
ir = okf.load_ir_projection(bundle_dir)
|
||||
if ir["project_id"] != project_id:
|
||||
raise ValueError(f"bundle project_id {ir['project_id']!r} != requested {project_id!r}")
|
||||
project_file = next(
|
||||
(f for f in okf.navigate_bundle(bundle_dir).files if f.type == "project"), None
|
||||
)
|
||||
name = (
|
||||
project_file.frontmatter.get("title", project_id).strip('"')
|
||||
if project_file is not None
|
||||
else project_id
|
||||
)
|
||||
return Project(
|
||||
id=project_id,
|
||||
name=name,
|
||||
description="",
|
||||
currency="NOK",
|
||||
cost_items=(),
|
||||
docs_dir=bundle_dir,
|
||||
verdict_input={},
|
||||
)
|
||||
|
||||
|
||||
def _features_of(proposal: SavingsProposal) -> ProposalFeatures:
|
||||
return ProposalFeatures(
|
||||
affected_codes=frozenset(item.code for item in proposal.affected_items),
|
||||
|
|
@ -136,6 +166,7 @@ async def run_project(
|
|||
*,
|
||||
docs_dir: str,
|
||||
verdict_input: dict[str, str],
|
||||
bundle_dir: str | None = None,
|
||||
store: VerdictStore | None = None,
|
||||
client_factory: Callable[[str], BaseChatClient] | None = None,
|
||||
max_rounds: int = 3,
|
||||
|
|
@ -147,8 +178,11 @@ async def run_project(
|
|||
) -> RunResult:
|
||||
"""Run the vertical slice for ONE project. ``client_factory`` is the test-injection seam
|
||||
(defaults to the real backend). ``verdict_input`` carries the expert decision/rationale
|
||||
(Layer-2). Raises ``pydantic.ValidationError`` on a bad contract and ``BudgetExceeded``
|
||||
when the token/round cap is crossed."""
|
||||
(Layer-2). ``bundle_dir`` (Fase 2a) makes the run OKF-bundle-driven: the project is derived
|
||||
from the bundle and, before generation, the candidate's prior verdicts in ``store`` are folded
|
||||
into the hypothesis prompt (Step-1 ExpeL wiring, målbilde §5/§7). Raises
|
||||
``pydantic.ValidationError`` on a bad contract and ``BudgetExceeded`` when the token/round cap
|
||||
is crossed."""
|
||||
# 1. Fail-fast: validate ALL contracts (incl. the verdict-feedback shape) before any client.
|
||||
load_contracts(
|
||||
{"docs_dir": docs_dir, "top_k": top_k},
|
||||
|
|
@ -156,8 +190,13 @@ async def run_project(
|
|||
verdict_input,
|
||||
)
|
||||
|
||||
# 2-3. Project + cited chunks (first-class provenance citations).
|
||||
project = _project_by_id(project_id)
|
||||
# 2-3. Project + cited chunks (first-class provenance citations). An OKF-bundle run derives its
|
||||
# project from the bundle (need not be a road reference-domain project).
|
||||
project = (
|
||||
_project_from_bundle(bundle_dir, project_id)
|
||||
if bundle_dir is not None
|
||||
else _project_by_id(project_id)
|
||||
)
|
||||
chunks = retrieve_chunks("cost saving measure", docs_dir, top_k)
|
||||
citations = [chunk_dict_to_citation(c) for c in chunks]
|
||||
if not citations:
|
||||
|
|
@ -189,6 +228,17 @@ async def run_project(
|
|||
debate_output = _debate_text(result)
|
||||
gen_context = debate_output or context
|
||||
|
||||
# Step-1 ExpeL wiring (Fase 2a, målbilde §5/§7): fold the candidate's prior verdicts INTO the
|
||||
# hypothesis context BEFORE generation, keyed on the OKF bundle's candidate features (available
|
||||
# pre-hypothesis). THIS is the one missing dataflow — previously ExpeL was computed
|
||||
# post-generation into a discarded SessionContext (step 7 below), so a prior verdict could not
|
||||
# reach the next hypothesis. Bundle-driven path with a populated store only; the road path is
|
||||
# untouched (its post-hoc, proposal-keyed retrieval below is unchanged).
|
||||
if bundle_dir is not None and store is not None and store.verdicts:
|
||||
expel_query = bundle_candidate_features(bundle_dir)
|
||||
fewshot = ExpeLContextProvider(store, expel_query, k=top_k).format_fewshot()
|
||||
gen_context = f"{fewshot}\n\n{gen_context}"
|
||||
|
||||
# 5. Structured candidate -> blocking validation; token bound = the meter in this loop.
|
||||
proposer_client = factory("proposer")
|
||||
outcome = await generate_via_llm(proposer_client, project, gen_context, meter)
|
||||
|
|
@ -211,8 +261,11 @@ async def run_project(
|
|||
token_usage=meter.tokens,
|
||||
)
|
||||
|
||||
# 7. ExpeL: surface prior verdicts for this proposal (exercises the two-arg
|
||||
# extend_instructions injection on a real SessionContext — the learning loop).
|
||||
# 7. ExpeL (regression guard + traceability): exercises the two-arg extend_instructions
|
||||
# injection on a REAL SessionContext (the Critical Fase-1 GA-signature guard), and surfaces
|
||||
# the proposal-keyed retrieval for RunResult.retrieved. On the bundle path the load-bearing
|
||||
# ExpeL->prompt dataflow already happened pre-generation (above); this block's SessionContext
|
||||
# is NOT what reaches the prompt.
|
||||
store = store if store is not None else VerdictStore(verdicts=[])
|
||||
features = _features_of(proposal)
|
||||
provider = ExpeLContextProvider(store, features, k=top_k)
|
||||
|
|
|
|||
|
|
@ -21,9 +21,12 @@ from __future__ import annotations
|
|||
import hashlib
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from agent_framework import ContextProvider, SessionContext
|
||||
|
||||
from portfolio_optimiser import okf
|
||||
|
||||
# Weights: the affected cost-code overlap dominates, then measure type, then magnitude.
|
||||
_W_CODES, _W_MEASURE, _W_MAGNITUDE = 0.60, 0.25, 0.15
|
||||
_MAGNITUDE_BUCKETS = [(0.0, 1e5), (1e5, 5e5), (5e5, 1e6), (1e6, float("inf"))]
|
||||
|
|
@ -196,3 +199,58 @@ def seed_store() -> VerdictStore:
|
|||
for vid, codes, mtype, saving, decision, desc in rows
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
# --- OKF-bundle seeding (Fase 2a): turn a project's bundle into the ExpeL substrate ---
|
||||
|
||||
|
||||
def _features_from_ir(ir: dict[str, Any]) -> ProposalFeatures:
|
||||
"""Map a bundle's IR projection (``validator-input.json``) to the structural features the
|
||||
store ranks on: the affected cost-code set, the measure string, and the claimed magnitude."""
|
||||
return ProposalFeatures(
|
||||
affected_codes=frozenset(item["code"] for item in ir["affected_items"]),
|
||||
measure_type=ir["measure"],
|
||||
claimed_saving_nok=ir["claimed_saving_nok"],
|
||||
description=ir.get("measure", ""),
|
||||
)
|
||||
|
||||
|
||||
def bundle_candidate_features(bundle_dir: str) -> ProposalFeatures:
|
||||
"""The pre-hypothesis ExpeL query key: the candidate measure's structural features, read from
|
||||
the OKF bundle's IR projection. Available BEFORE any proposal is generated — which is what lets
|
||||
Step-1 retrieve prior verdicts and fold them into the hypothesis prompt (målbilde §2 step 1)."""
|
||||
return _features_from_ir(okf.load_ir_projection(bundle_dir))
|
||||
|
||||
|
||||
def _verdict_rationale(fm: dict[str, str]) -> str:
|
||||
"""Build the few-shot rationale from a ``type: verdict`` file's frontmatter, carrying the
|
||||
learning signal the deterministic validator cannot compute (the realization rate + expected
|
||||
actual). This is the ExpeL signal that must reach the next hypothesis."""
|
||||
base = fm.get("description", "")
|
||||
signal = [
|
||||
f"{label}={fm[key]}"
|
||||
for key, label in (
|
||||
("realization_rate", "realiseringsgrad"),
|
||||
("expected_actual_saving_nok", "forventet_faktisk_NOK"),
|
||||
)
|
||||
if fm.get(key)
|
||||
]
|
||||
return f"{base} [{'; '.join(signal)}]" if signal else base
|
||||
|
||||
|
||||
def seed_store_from_bundle(bundle_dir: str) -> VerdictStore:
|
||||
"""Build a ``VerdictStore`` from an OKF bundle's ``type: verdict`` files. Each verdict is keyed
|
||||
on the bundle's candidate features (so it retrieves for that measure) and carries the
|
||||
realization signal in its rationale. The seed verdict stands in for the durable HITL verdict a
|
||||
real expert would supply via the same folder interface (målbilde §3)."""
|
||||
features = bundle_candidate_features(bundle_dir)
|
||||
bundle = okf.navigate_bundle(bundle_dir)
|
||||
verdicts = [
|
||||
capture_verdict(
|
||||
features,
|
||||
vf.frontmatter.get("decision", "approved"),
|
||||
_verdict_rationale(vf.frontmatter),
|
||||
)
|
||||
for vf in bundle.verdicts
|
||||
]
|
||||
return VerdictStore(verdicts=verdicts)
|
||||
|
|
|
|||
|
|
@ -163,6 +163,43 @@ def make_portfolio_client_factory() -> Callable[..., Callable[[str], BaseChatCli
|
|||
return _make
|
||||
|
||||
|
||||
class _RecordingChatClient(SyntheticUsageChatClient):
|
||||
"""Records the incoming prompt blob per call into a SHARED sink, then returns a fixed valid
|
||||
reply. Lets a test assert exactly what text reached the prompt — the probe the Step-1 ExpeL
|
||||
wiring is made load-bearing against (does a prior verdict reach the hypothesis prompt?)."""
|
||||
|
||||
def __init__(self, sink: list[str], reply: str, *, tokens_per_reply: int = 8) -> None:
|
||||
super().__init__(default_reply=reply, tokens_per_reply=tokens_per_reply)
|
||||
self._sink = sink
|
||||
|
||||
def _inner_get_response(
|
||||
self, *, messages: Sequence[Message], stream: bool, options: Any, **kwargs: Any
|
||||
) -> Any:
|
||||
self._sink.append(" ".join(getattr(m, "text", "") or "" for m in messages))
|
||||
return super()._inner_get_response(
|
||||
messages=messages, stream=stream, options=options, **kwargs
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def make_recording_client_factory() -> Callable[
|
||||
[str], tuple[Callable[[str], BaseChatClient], list[str]]
|
||||
]:
|
||||
"""Return a maker that builds a per-role client factory recording every prompt blob into a
|
||||
shared list. Returns ``(factory, recorded_prompts)`` so the test inspects what reached the
|
||||
prompt across the whole run (debate rounds + generation)."""
|
||||
|
||||
def _make(reply: str) -> tuple[Callable[[str], BaseChatClient], list[str]]:
|
||||
sink: list[str] = []
|
||||
|
||||
def factory(role: str) -> BaseChatClient:
|
||||
return _RecordingChatClient(sink, reply)
|
||||
|
||||
return factory, sink
|
||||
|
||||
return _make
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def fresh_store() -> VerdictStore:
|
||||
return VerdictStore(verdicts=[])
|
||||
|
|
|
|||
96
tests/test_okf.py
Normal file
96
tests/test_okf.py
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
"""OKF bundle navigation (okf.py) — the framework-neutral, D7-portable context seam.
|
||||
|
||||
These tests pin the minimal navigation the Step-1 ExpeL wiring depends on: read ``index.md``,
|
||||
follow intra-bundle cross-links, parse each file's frontmatter, classify by ``type``, and locate
|
||||
the candidate IR projection. Robustness (OKF SPEC §4): broken links are tolerated, never raised.
|
||||
|
||||
No ``agent_framework``/``mcp`` import is allowed in ``okf`` — guarded by ``test_okf_is_maf_free``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from portfolio_optimiser import okf
|
||||
|
||||
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||
|
||||
|
||||
def test_navigate_classifies_every_typed_file() -> None:
|
||||
"""The energi bundle resolves to its six typed OKF files (index + project + hypothesis +
|
||||
methodology + reference + verdict), each carrying its declared ``type``."""
|
||||
bundle = okf.navigate_bundle(str(BUNDLE_DIR))
|
||||
types = {f.name: f.type for f in bundle.files}
|
||||
assert types == {
|
||||
"index.md": "index",
|
||||
"bygg-kontor-nord.md": "project",
|
||||
"tiltak-led-retrofit.md": "hypothesis",
|
||||
"metode-ipmvp-a.md": "methodology",
|
||||
"kilder-realiseringsgap.md": "reference",
|
||||
"verdict-led-fro.md": "verdict",
|
||||
}
|
||||
|
||||
|
||||
def test_navigate_exposes_verdicts_and_hypothesis() -> None:
|
||||
"""The navigation surfaces exactly one ``type: verdict`` file (the ExpeL seed) and the
|
||||
candidate hypothesis — the two the Step-1 wiring keys on."""
|
||||
bundle = okf.navigate_bundle(str(BUNDLE_DIR))
|
||||
assert [f.name for f in bundle.verdicts] == ["verdict-led-fro.md"]
|
||||
assert bundle.verdicts[0].frontmatter["realization_rate"] == "0.82"
|
||||
assert bundle.hypothesis is not None
|
||||
assert bundle.hypothesis.name == "tiltak-led-retrofit.md"
|
||||
|
||||
|
||||
def test_navigate_index_summary_is_progressive_disclosure() -> None:
|
||||
"""``index_summary`` is the index body (the progressive-disclosure entry point), not stuffing
|
||||
the whole bundle."""
|
||||
bundle = okf.navigate_bundle(str(BUNDLE_DIR))
|
||||
assert "progressiv disclosure" in bundle.index_summary.lower()
|
||||
|
||||
|
||||
def test_navigate_tolerates_broken_links(tmp_path) -> None:
|
||||
"""OKF SPEC §4: a consumer MUST tolerate broken links. An index linking a missing file
|
||||
navigates without raising, simply omitting the absent target."""
|
||||
(tmp_path / "index.md").write_text(
|
||||
"---\ntype: index\n---\n\nSee [gone](missing.md) and [here](real.md).\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(tmp_path / "real.md").write_text("---\ntype: project\n---\n\nbody\n", encoding="utf-8")
|
||||
bundle = okf.navigate_bundle(str(tmp_path))
|
||||
names = {f.name for f in bundle.files}
|
||||
assert names == {"index.md", "real.md"} # missing.md silently skipped, no raise
|
||||
|
||||
|
||||
def test_load_ir_projection_returns_candidate_ir() -> None:
|
||||
"""The bundle's IR projection (``validator-input.json``) is the candidate measure's cost-IR —
|
||||
the pre-hypothesis ExpeL query key source."""
|
||||
ir = okf.load_ir_projection(str(BUNDLE_DIR))
|
||||
assert ir["project_id"] == "BYGG-KONTOR-NORD"
|
||||
assert [a["code"] for a in ir["affected_items"]] == ["ENERGI-TOTAL-EL"]
|
||||
assert ir["claimed_saving_nok"] == 30000
|
||||
|
||||
|
||||
def test_parse_frontmatter_reads_scalar_fields() -> None:
|
||||
"""The minimal frontmatter reader returns the leading ``---`` block as key:value strings."""
|
||||
fm = okf.parse_frontmatter(BUNDLE_DIR / "verdict-led-fro.md")
|
||||
assert fm["type"] == "verdict"
|
||||
assert fm["decision"] == "approved_with_adjustment"
|
||||
|
||||
|
||||
def test_okf_is_maf_free() -> None:
|
||||
"""D7 portability: ``okf.py`` IMPORTS no ``agent_framework`` / ``mcp`` (the docstring may name
|
||||
them to document the constraint, exactly as ``retrieval.py`` does) — checked via the AST, not a
|
||||
raw substring, so the prose claim doesn't trip the guard."""
|
||||
import ast
|
||||
|
||||
src = (
|
||||
Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser" / "okf.py"
|
||||
).read_text(encoding="utf-8")
|
||||
imported: list[str] = []
|
||||
for node in ast.walk(ast.parse(src)):
|
||||
if isinstance(node, ast.Import):
|
||||
imported += [a.name for a in node.names]
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
imported.append(node.module or "")
|
||||
forbidden = [m for m in imported if m.split(".")[0] in {"agent_framework", "mcp"}]
|
||||
assert forbidden == [], f"okf.py must not import MAF/mcp, found: {forbidden}"
|
||||
93
tests/test_step1_expel_loadbearing.py
Normal file
93
tests/test_step1_expel_loadbearing.py
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
"""Step-1 load-bearing seam (målbilde §5 / §7): a prior expert verdict MUST reach the next
|
||||
hypothesis prompt.
|
||||
|
||||
The diagnosis (fot-i-bakken, verified in code): ExpeL retrieval was computed AFTER generation and
|
||||
written into a discarded ``SessionContext`` — so the seed verdict could not influence any
|
||||
hypothesis (``context_providers`` = 0). Fase 2a wires it: before ``generate_via_llm``, the
|
||||
candidate's prior verdicts are folded into the generation context.
|
||||
|
||||
These two tests are a load-bearing PAIR, not a smoke test (per the Fase-2 green-but-dead trap):
|
||||
- the positive test asserts the seed verdict's realization signal ("0.82") IS in the generation
|
||||
prompt — it goes RED the moment the fold is detached;
|
||||
- the empty-store test asserts the signal is ABSENT without a seed — proving the signal's presence
|
||||
is CAUSED by the seam, not incidental to the fixture.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from portfolio_optimiser.run import run_project
|
||||
from portfolio_optimiser.validator import ValidatedProposal
|
||||
from portfolio_optimiser.verdicts import VerdictStore, seed_store_from_bundle
|
||||
|
||||
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||
|
||||
# A VALID BYGG-KONTOR-NORD proposal: affected total = 300000 x 1.0; no assumptions -> degenerate
|
||||
# Monte Carlo P90 = 0.30 x 300000 = 90000 >= claimed 30000 -> validates on the first attempt.
|
||||
_ENERGY_REPLY = (
|
||||
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
|
||||
)
|
||||
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
|
||||
|
||||
# The load-bearing marker is the seed verdict's MINTED id (a 16-char content hash that
|
||||
# ``ExpeLContextProvider.format_fewshot`` emits as ``- [<id>] ...``). It cannot appear in the
|
||||
# bundle's raw text, so its presence in the prompt isolates the STRUCTURAL ExpeL path. (The
|
||||
# realization rate "0.82" is NOT a usable marker here: ``docs_dir == bundle_dir``, so keyword
|
||||
# chunk-stuffing reads the verdict file and would leak "0.82" regardless of the fold. Layer
|
||||
# separation — verdicts out of the keyword docs folder — is Fase 2b.)
|
||||
_SEED_ID = seed_store_from_bundle(str(BUNDLE_DIR)).verdicts[0].id
|
||||
|
||||
|
||||
def _generation_prompts(sink: list[str]) -> list[str]:
|
||||
"""The generation-call prompts (``generate._build_messages`` embeds 'SavingsProposal'),
|
||||
isolated from the debate-round prompts also captured in the sink."""
|
||||
return [p for p in sink if "SavingsProposal" in p]
|
||||
|
||||
|
||||
async def test_prior_verdict_reaches_hypothesis_prompt(make_recording_client_factory) -> None:
|
||||
"""LOAD-BEARING: with the energi bundle's seed verdict in the store, the realization signal
|
||||
reaches the hypothesis-generation prompt. Detach the fold in ``run.py`` and this goes red."""
|
||||
store = seed_store_from_bundle(str(BUNDLE_DIR))
|
||||
assert store.verdicts, "precondition: the bundle seeds exactly one verdict"
|
||||
factory, recorded = make_recording_client_factory(_ENERGY_REPLY)
|
||||
|
||||
result = await run_project(
|
||||
"BYGG-KONTOR-NORD",
|
||||
"local",
|
||||
docs_dir=str(BUNDLE_DIR),
|
||||
bundle_dir=str(BUNDLE_DIR),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
store=store,
|
||||
client_factory=factory,
|
||||
)
|
||||
|
||||
assert isinstance(result.outcome, ValidatedProposal)
|
||||
gen_prompts = _generation_prompts(recorded)
|
||||
assert gen_prompts, "the generation call must have happened"
|
||||
assert any(_SEED_ID in p for p in gen_prompts), (
|
||||
"the seed verdict did not reach the hypothesis prompt — the ExpeL->prompt fold is detached"
|
||||
)
|
||||
|
||||
|
||||
async def test_empty_store_leaves_no_signal_in_prompt(make_recording_client_factory) -> None:
|
||||
"""CAUSALITY CONTROL: the same run with an EMPTY store carries no realization signal into the
|
||||
prompt — proving the signal above is caused by the seam, making the positive test load-bearing
|
||||
(not merely green against the fixture's mere presence)."""
|
||||
factory, recorded = make_recording_client_factory(_ENERGY_REPLY)
|
||||
|
||||
await run_project(
|
||||
"BYGG-KONTOR-NORD",
|
||||
"local",
|
||||
docs_dir=str(BUNDLE_DIR),
|
||||
bundle_dir=str(BUNDLE_DIR),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
store=VerdictStore(verdicts=[]),
|
||||
client_factory=factory,
|
||||
)
|
||||
|
||||
assert all(_SEED_ID not in p for p in recorded), (
|
||||
"no seed verdict in the store, yet its id appeared in a prompt — "
|
||||
"the assertion is not actually load-bearing"
|
||||
)
|
||||
|
|
@ -7,6 +7,8 @@ genuine two-arg ``extend_instructions(source_id, instructions)`` GA signature
|
|||
Critical Fase 1 risk. Pattern: tests/spikes/test_d_verdictstore.py + real SessionContext.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from agent_framework import SessionContext
|
||||
|
||||
|
|
@ -15,9 +17,13 @@ from portfolio_optimiser.verdicts import (
|
|||
ProposalFeatures,
|
||||
Verdict,
|
||||
VerdictStore,
|
||||
bundle_candidate_features,
|
||||
capture_verdict,
|
||||
seed_store_from_bundle,
|
||||
)
|
||||
|
||||
_BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||
|
||||
_QUERY = ProposalFeatures(
|
||||
affected_codes=frozenset({"05.2", "03.1"}),
|
||||
measure_type="scope_reduction",
|
||||
|
|
@ -106,3 +112,36 @@ async def test_before_run_populates_real_sessioncontext_two_arg() -> None:
|
|||
ctx = SessionContext(input_messages=[], instructions=[])
|
||||
await provider.before_run(agent=None, session=None, context=ctx, state={})
|
||||
assert any(true_id in instr for instr in ctx.instructions)
|
||||
|
||||
|
||||
# --- OKF-bundle seeding (Fase 2a): the pre-hypothesis ExpeL query key + the seed store ---
|
||||
|
||||
|
||||
def test_bundle_candidate_features_keys_on_the_ir_projection() -> None:
|
||||
"""The pre-hypothesis ExpeL query is the candidate measure's cost-IR features (from the
|
||||
bundle's ``validator-input.json``) — available BEFORE any proposal is generated."""
|
||||
features = bundle_candidate_features(str(_BUNDLE_DIR))
|
||||
assert features.affected_codes == frozenset({"ENERGI-TOTAL-EL"})
|
||||
assert "LED-retrofit" in features.measure_type
|
||||
assert features.claimed_saving_nok == 30000
|
||||
|
||||
|
||||
def test_seed_store_from_bundle_carries_the_realization_signal() -> None:
|
||||
"""Each ``type: verdict`` file becomes a structurally-keyed ``Verdict`` whose rationale carries
|
||||
the learning signal the validator cannot compute (the realization rate 0.82)."""
|
||||
store = seed_store_from_bundle(str(_BUNDLE_DIR))
|
||||
assert len(store.verdicts) == 1
|
||||
seed = store.verdicts[0]
|
||||
assert seed.proposal_features.affected_codes == frozenset({"ENERGI-TOTAL-EL"})
|
||||
assert "approved" in seed.decision
|
||||
assert "0.82" in seed.rationale
|
||||
|
||||
|
||||
def test_seed_store_retrieval_matches_the_candidate() -> None:
|
||||
"""A3: seed and query derive from the SAME IR -> similarity 1.0 -> the lone seed is retrieved
|
||||
for the candidate (the structural match the Step-1 wiring relies on)."""
|
||||
store = seed_store_from_bundle(str(_BUNDLE_DIR))
|
||||
query = bundle_candidate_features(str(_BUNDLE_DIR))
|
||||
hits = store.retrieve(query, k=3)
|
||||
assert len(hits) == 1
|
||||
assert "0.82" in hits[0].rationale
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue