From 8814a698c217e62d17e658b3153c5ad6ce4b7457 Mon Sep 17 00:00:00 2001 From: Kjell Tore Guttormsen Date: Tue, 30 Jun 2026 06:42:19 +0200 Subject: [PATCH] feat(fase2b): OKF-navigated bundle context replaces chunk-stuffing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the honest Fase 2a limitation: docs_dir==bundle_dir let keyword chunk-stuffing leak the verdict's realization rate ("0.82") into the debate / generation prompt regardless of the ExpeL fold (it surfaced from both verdict-led-fro.md AND golden.json). The realization signal now reaches the hypothesis prompt ONLY via the gated ExpeL fold. - okf.py: bundle_context() + Bundle.context_files render the navigated bundle (index + frontmatter + cross-links) as the agent read-context, EXCLUDING type: verdict (maalbilde §2/§4). Pure stdlib, still MAF-free. - datasource.py: bundle_citations() derives first-class citations from the navigated non-verdict files. - run_project: on the bundle path context + citations + debate tools come from navigation (tools=[]; navigation replaces query-time RAG); the road path keeps chunk-stuffing unchanged. Load-bearing (maalbilde §7): the marker is upgraded from the minted verdict id to the realization signal itself. The empty-store control now asserts "0.82" reaches NO prompt — RED against the pre-2b chunk-stuffing path, green after navigation (TDD red->green). New okf-level test_bundle_context_excludes_verdict_layer guards the seam directly. Suite 133->134 passed, 4 skipped; mypy + ruff check clean. Reverted unrelated ruff-format drift (backends/budget/verdicts/test_contracts) to keep the diff surgical. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01MHR8iKxJRxDiDfNw8HZmWE --- CLAUDE.md | 6 ++++ README.md | 7 ++-- src/portfolio_optimiser/datasource.py | 16 +++++++++ src/portfolio_optimiser/okf.py | 20 +++++++++++ src/portfolio_optimiser/run.py | 50 ++++++++++++++++----------- tests/test_okf.py | 20 +++++++++++ tests/test_step1_expel_loadbearing.py | 27 +++++++++++---- 7 files changed, 117 insertions(+), 29 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d80b45b..efb110f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,6 +29,12 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b `shared/`-kjernen er ren stdlib — null `agent_framework`/`mcp`-import, så samme bundles konsumeres uendret av begge stacker (D7-portabel). Håndhevet av `tests/test_okf.py::test_okf_is_maf_free`; importér aldri MAF inn i kontekst-laget. +- **OKF-navigert bundle-kontekst (ikke stuffing):** på bundle-stien bygges agent-lese-konteksten + ved å NAVIGERE bundelen (`okf.bundle_context`: index + frontmatter + cross-links, progressiv + disclosure) — aldri keyword-chunk-stuffing (målbilde §2/§4). `type: verdict`-laget ekskluderes + fra denne konteksten: tidligere dommer når hypotese-prompten KUN via den gatede ExpeL-folden. + Load-bearing: `test_bundle_context_excludes_verdict_layer` + empty-store-kontrollen i + `test_step1_expel_loadbearing.py` (realiseringssignalet lekker aldri inn via kontekst). - **Stoppkriterier + budsjett-tak påkrevd ved oppstart** (fail-fast, aldri ubegrenset loop). - **Group Chat maker-checker** som debatt-default (IKKE Magentic, som er eksperimentell). - **Kostnadsdisiplin:** utvikle primært på lokal profil (gratis); Foundry/Azure (privat tenant finnes) kun til målrettet, minimal verifisering; billigste modeller + små syntetiske data + harde token-tak. Ingen tunge test-kjøringer. diff --git a/README.md b/README.md index 99849bc..6d9c8df 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Generic, open framework on **Microsoft Agent Framework (MAF)** for finding cost-savings / efficiency proposals *within* each project of a portfolio of independent projects. Multiple agents collaborate to generate candidate proposals; a mandatory deterministic validator (solver + Monte Carlo) decides the numbers; domain experts review via human-in-the-loop, and the system learns from their verdicts. -> **Status:** Early development. The deterministic backbone is solid; the agentic learning loop is being wired one load-bearing seam at a time — **Step 1 (OKF context → hypothesis) is wired** (see below). Not yet end-to-end usable. +> **Status:** Early development. The deterministic backbone is solid; the agentic learning loop is being wired one load-bearing seam at a time — **Step 1 (OKF-navigated context → hypothesis) is wired**, with the verdict layer kept out of the read-context (see below). Not yet end-to-end usable. > **Disclaimer — technical framework only.** This project is a *technical framework*. Organizations that deploy it are themselves responsible for ensuring a valid processing purpose and for any required assessments (DPIA, risk/ROS, security reviews, etc.). The framework ships technical affordances (local-only mode, provenance/audit logging, no silent data egress) to *enable* compliant use, but makes no compliance guarantees. @@ -14,7 +14,10 @@ The result will never fit any single customer 100%. The goal is a **~90% genuine The mandatory deterministic backbone (validator + budget meter + provenance) is solid and load-bearing. The agentic learning loop (see the [target picture](docs/plan/2026-06-26-maalbilde-agentic-loop.md) §11) is wired one seam at a time: -- **Step 1 — OKF context → hypothesis (wired).** `run_project(..., bundle_dir=...)` navigates a project's [OKF bundle](shared/examples/bygg-energi-mikro/) and folds the candidate's prior expert verdicts (ExpeL retrieval) into the hypothesis prompt *before* generation — so a prior verdict provably influences the next hypothesis. Guarded by a load-bearing test that fails when the seam is detached (`tests/test_step1_expel_loadbearing.py`). +- **Step 1 — OKF context → hypothesis (wired).** `run_project(..., bundle_dir=...)` wires the first agentic seam in two halves: + - **Context by navigation, not stuffing.** The agent read-context is built by **navigating** the project's [OKF bundle](shared/examples/bygg-energi-mikro/) (`index.md` + frontmatter + cross-links, progressive disclosure) — never keyword chunk-stuffing (target picture §2/§4). + - **Verdict layer gated out of context.** The `type: verdict` layer is excluded from that context; the candidate's prior expert verdicts reach the hypothesis prompt *only* through the gated ExpeL fold (folded in *before* generation), so a prior verdict provably — and exclusively — influences the next hypothesis. + - Two load-bearing tests fail when the seam is detached: the realization signal must reach the prompt via the fold (`tests/test_step1_expel_loadbearing.py`), and must never leak via context (`tests/test_okf.py::test_bundle_context_excludes_verdict_layer`). - **Steps 3–8** (checker gating, informed refinement, async file feedback, gated wiki promotion) — not yet wired. ## Docs diff --git a/src/portfolio_optimiser/datasource.py b/src/portfolio_optimiser/datasource.py index c145f5f..29caee8 100644 --- a/src/portfolio_optimiser/datasource.py +++ b/src/portfolio_optimiser/datasource.py @@ -17,6 +17,7 @@ from typing import Any from agent_framework import FunctionTool, tool +from portfolio_optimiser.okf import Bundle from portfolio_optimiser.provenance import Citation from portfolio_optimiser.retrieval import RetrievedChunk, TextSpan, retrieve @@ -36,6 +37,21 @@ def retrieve_chunks(query: str, docs_dir: str, top_k: int = 3) -> list[dict[str, return [_chunk_to_dict(c) for c in retrieve(query, docs_dir, top_k)] +def bundle_citations(bundle: Bundle) -> list[Citation]: + """First-class citations for the OKF-navigated bundle context: one ``Citation`` per non-verdict + concept file (exactly the files ``okf.bundle_context`` renders). Whole-file locators, exact by + construction (``snippet == body[start:end]``). The verdict layer is deliberately uncited — it is + not cost documentation; it enters only via the gated ExpeL fold.""" + return [ + Citation( + file=f.name, + locator=TextSpan(start_index=0, end_index=len(f.body)), + snippet=f.body, + ) + for f in bundle.context_files + ] + + def chunk_dict_to_citation(d: dict[str, Any]) -> Citation: """Map a structuredContent chunk dict into a first-class ``provenance.Citation``.""" loc = d["locator"] diff --git a/src/portfolio_optimiser/okf.py b/src/portfolio_optimiser/okf.py index a873436..d0c3b41 100644 --- a/src/portfolio_optimiser/okf.py +++ b/src/portfolio_optimiser/okf.py @@ -84,6 +84,13 @@ class Bundle: """Every ``type: verdict`` file (the ExpeL seeds the Step-1 wiring retrieves).""" return [f for f in self.files if f.type == "verdict"] + @property + def context_files(self) -> list[BundleFile]: + """The non-index, non-``verdict`` concept files — the bodies that form the agent context. + The verdict layer is deliberately EXCLUDED: prior verdicts reach the hypothesis prompt only + through the gated ExpeL fold, never by stuffing them into the read-context (målbilde §4).""" + return [f for f in self.files if f.name != _INDEX_NAME and f.type != "verdict"] + @property def hypothesis(self) -> BundleFile | None: """The candidate ``type: hypothesis`` file, if present.""" @@ -123,6 +130,19 @@ def navigate_bundle(bundle_dir: str) -> Bundle: return Bundle(dir=bundle_dir, files=tuple(files)) +def bundle_context(bundle: Bundle) -> str: + """Render a navigated bundle as agent read-context via progressive disclosure: the ``index.md`` + summary, then each concept file as ``## {type}: {title}\\n{body}``. ``type: verdict`` files are + EXCLUDED (målbilde §2 step 1 / §4: navigation, not chunk-stuffing — the verdict layer folds in + only via the gated ExpeL retrieval). Deterministic: index first, then context files in + navigation order; empty sections are dropped.""" + sections = [bundle.index_summary] + for f in bundle.context_files: + title = f.frontmatter.get("title", f.name).strip('"') + sections.append(f"## {f.type or 'document'}: {title}\n{f.body}") + return "\n\n".join(s for s in sections if s.strip()) + + def load_ir_projection(bundle_dir: str, name: str = _IR_PROJECTION) -> dict[str, Any]: """Load the bundle's IR projection (``validator-input.json`` by default): the candidate measure's cost-IR (``measure``, ``affected_items``, ``claimed_saving_nok``) — the diff --git a/src/portfolio_optimiser/run.py b/src/portfolio_optimiser/run.py index 2d8698f..d8ee9d3 100644 --- a/src/portfolio_optimiser/run.py +++ b/src/portfolio_optimiser/run.py @@ -35,6 +35,7 @@ from portfolio_optimiser.backends import Profile, get_backend, resolve_model from portfolio_optimiser.budget import Budget, BudgetMiddleware, TokenMeter from portfolio_optimiser.contracts import load_contracts from portfolio_optimiser.datasource import ( + bundle_citations, chunk_dict_to_citation, make_retrieval_tool, retrieve_chunks, @@ -116,18 +117,20 @@ def _project_by_id(project_id: str) -> Project: raise ValueError(f"unknown project_id: {project_id!r}") -def _project_from_bundle(bundle_dir: str, project_id: str) -> Project: +def _project_from_bundle( + bundle_dir: str, project_id: str, *, bundle: okf.Bundle | None = None +) -> Project: """Derive a minimal ``Project`` from an OKF bundle (so a bundle the loop runs need NOT be a road reference-domain project). Only ``id`` + ``name`` reach the generation prompt (``generate._build_messages``), so ``cost_items`` is empty and ``verdict_input`` is unused here (the Layer-2 decision flows via the ``verdict_input`` argument). Fail-fast: the bundle's IR - ``project_id`` must match the requested id.""" + ``project_id`` must match the requested id. ``bundle`` reuses an already-navigated bundle to + avoid a second navigation.""" ir = okf.load_ir_projection(bundle_dir) if ir["project_id"] != project_id: raise ValueError(f"bundle project_id {ir['project_id']!r} != requested {project_id!r}") - project_file = next( - (f for f in okf.navigate_bundle(bundle_dir).files if f.type == "project"), None - ) + nav = bundle if bundle is not None else okf.navigate_bundle(bundle_dir) + project_file = next((f for f in nav.files if f.type == "project"), None) name = ( project_file.frontmatter.get("title", project_id).strip('"') if project_file is not None @@ -190,23 +193,29 @@ async def run_project( verdict_input, ) - # 2-3. Project + cited chunks (first-class provenance citations). An OKF-bundle run derives its - # project from the bundle (need not be a road reference-domain project). - project = ( - _project_from_bundle(bundle_dir, project_id) - if bundle_dir is not None - else _project_by_id(project_id) - ) - chunks = retrieve_chunks("cost saving measure", docs_dir, top_k) - citations = [chunk_dict_to_citation(c) for c in chunks] + # 2-3. Project + agent read-context + first-class citations. A bundle run derives ALL THREE from + # the navigated OKF bundle via progressive disclosure (verdict layer EXCLUDED — målbilde §2/§4), + # NOT keyword chunk-stuffing; the road path keeps the chunk-retrieval data source. ``debate_tools`` + # is the query-time retrieval surface — empty on the bundle path (navigation already placed the + # curated context in the prompt, and a docs_dir==bundle_dir tool would re-leak the verdict layer). + if bundle_dir is not None: + bundle = okf.navigate_bundle(bundle_dir) + project = _project_from_bundle(bundle_dir, project_id, bundle=bundle) + context = okf.bundle_context(bundle) + citations = bundle_citations(bundle) + debate_tools: list[Any] = [] + else: + project = _project_by_id(project_id) + chunks = retrieve_chunks("cost saving measure", docs_dir, top_k) + citations = [chunk_dict_to_citation(c) for c in chunks] + context = "\n".join(c["snippet"] for c in chunks) + debate_tools = [make_retrieval_tool(docs_dir, top_k=top_k)] if not citations: raise ValueError(f"no citable content in docs_dir: {docs_dir!r}") - context = "\n".join(c["snippet"] for c in chunks) - # 4. Budget + maker-checker debate (round-capped; Layer-1 HITL optional). - # The shared meter is driven on the debate's chat calls by the BudgetMiddleware - # (the brief's named short-circuit mechanism); the retrieval tool exposes the - # citation-bearing data source to the agents (no longer string-stuffed out-of-band). + # 4. Budget + maker-checker debate (round-capped; Layer-1 HITL optional). The shared meter is + # driven on the debate's chat calls by the BudgetMiddleware (the brief's named short-circuit + # mechanism); ``debate_tools`` exposes the citation-bearing data source on the road path. meter = ( meter if meter is not None @@ -214,12 +223,11 @@ async def run_project( ) factory = client_factory if client_factory is not None else _default_factory(profile) budget_mw = BudgetMiddleware(meter) - retrieval_tool = make_retrieval_tool(docs_dir, top_k=top_k) debate = fresh_workflow( factory, max_rounds=max_rounds, enable_layer1_hitl=enable_layer1_hitl, - tools=[retrieval_tool], + tools=debate_tools, middleware=[budget_mw], ) result = await debate.run(f"Find a cost-saving measure for {project.id}.\nContext:\n{context}") diff --git a/tests/test_okf.py b/tests/test_okf.py index 6397a80..dddade8 100644 --- a/tests/test_okf.py +++ b/tests/test_okf.py @@ -48,6 +48,26 @@ def test_navigate_index_summary_is_progressive_disclosure() -> None: assert "progressiv disclosure" in bundle.index_summary.lower() +def test_bundle_context_excludes_verdict_layer() -> None: + """Fase 2b LOAD-BEARING (okf level): ``bundle_context`` renders the concept files for the agent + prompt but EXCLUDES the ``type: verdict`` file, so the realization signal reaches a prompt only + via the gated ExpeL fold — never by stuffing it into the read-context (målbilde §2/§4).""" + bundle = okf.navigate_bundle(str(BUNDLE_DIR)) + assert {f.type for f in bundle.context_files} == { + "project", + "hypothesis", + "methodology", + "reference", + } + context = okf.bundle_context(bundle) + assert "0.82" not in context # the verdict's realization signal is NOT stuffed in + assert ( + bundle.verdicts[0].frontmatter["realization_rate"] == "0.82" + ) # though it IS in the bundle + assert "## hypothesis:" in context # concept files ARE rendered (progressive disclosure) + assert "progressiv disclosure" in context.lower() # the index summary is the entry point + + def test_navigate_tolerates_broken_links(tmp_path) -> None: """OKF SPEC §4: a consumer MUST tolerate broken links. An index linking a missing file navigates without raising, simply omitting the absent target.""" diff --git a/tests/test_step1_expel_loadbearing.py b/tests/test_step1_expel_loadbearing.py index 4abb58b..d442411 100644 --- a/tests/test_step1_expel_loadbearing.py +++ b/tests/test_step1_expel_loadbearing.py @@ -31,13 +31,16 @@ _ENERGY_REPLY = ( ) _VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"} -# The load-bearing marker is the seed verdict's MINTED id (a 16-char content hash that -# ``ExpeLContextProvider.format_fewshot`` emits as ``- [] ...``). It cannot appear in the -# bundle's raw text, so its presence in the prompt isolates the STRUCTURAL ExpeL path. (The -# realization rate "0.82" is NOT a usable marker here: ``docs_dir == bundle_dir``, so keyword -# chunk-stuffing reads the verdict file and would leak "0.82" regardless of the fold. Layer -# separation — verdicts out of the keyword docs folder — is Fase 2b.) +# Two load-bearing markers, by design: +# - the seed verdict's MINTED id (16-char content hash ``format_fewshot`` emits as ``- [] ...``); +# it cannot appear in the bundle's raw text, so it isolates the STRUCTURAL ExpeL path. +# - the realization rate "0.82". Fase 2b replaced keyword chunk-stuffing with OKF-navigation that +# EXCLUDES ``type: verdict`` from the bundle context, so the signal now reaches a prompt ONLY via +# the gated ExpeL fold. Pre-2b (``docs_dir == bundle_dir``) chunk-stuffing leaked "0.82" from the +# verdict + golden files into the debate prompt regardless of the fold — which the empty-store +# control below asserts is no longer true (it goes RED against the pre-2b chunk-stuffing path). _SEED_ID = seed_store_from_bundle(str(BUNDLE_DIR)).verdicts[0].id +_REALIZATION_SIGNAL = "realiseringsgrad=0.82" # the exact ``_verdict_rationale`` fewshot string def _generation_prompts(sink: list[str]) -> list[str]: @@ -69,6 +72,11 @@ async def test_prior_verdict_reaches_hypothesis_prompt(make_recording_client_fac assert any(_SEED_ID in p for p in gen_prompts), ( "the seed verdict did not reach the hypothesis prompt — the ExpeL->prompt fold is detached" ) + # Fase 2b: the realization SIGNAL itself (not just the id) is now a clean marker — it can only + # reach the prompt via the gated fold, since OKF-navigation excludes the verdict from context. + assert any(_REALIZATION_SIGNAL in p for p in gen_prompts), ( + "the realization signal did not reach the hypothesis prompt via the ExpeL fold" + ) async def test_empty_store_leaves_no_signal_in_prompt(make_recording_client_factory) -> None: @@ -91,3 +99,10 @@ async def test_empty_store_leaves_no_signal_in_prompt(make_recording_client_fact "no seed verdict in the store, yet its id appeared in a prompt — " "the assertion is not actually load-bearing" ) + # Fase 2b LOAD-BEARING: with no fold, the realization rate must appear in NO prompt. This goes + # RED against the pre-2b chunk-stuffing path (which leaked "0.82" from the verdict + golden files + # into the debate prompt) and only passes once OKF-navigation excludes the verdict from context. + assert all("0.82" not in p for p in recorded), ( + "the realization rate leaked into a prompt without a seed verdict — bundle context is still " + "chunk-stuffing the verdict/golden files (Fase 2b OKF-navigation not in effect)" + )