open/portfolio-optimiser-claude
1
0
Fork 0
Code Issues Releases 1 Activity

feat(portfolio): K3 — portfolio learning loop (shared verdict store, parity row 5) [skip-docs]

Browse source 
A single VerdictStore threaded through run_portfolio: a verdict available when
project k composes survives into project k+1's fold (method-spec §5 cross-project
threading). The optional verdict_dir is the portfolio-level expert inbox, read
before each fold (role split §3 Step 7 — the portfolio never writes a run's own
verdict back; §1/§6 — no self-contamination, only expert/seed verdicts cross).
compose_run_context gains an optional passed-in store (None = fresh; every
existing caller composes exactly as before).

Load-bearing (tests/test_portfolio_learning_loadbearing.py), 2 detach proofs +
control + §4.2 idempotency:
- cross-project threading: project 1's bundle seed survives into project 2's
  prompt via the shared store; detach (compose ignores the passed-in store,
  always fresh) -> red.
- portfolio inbox fold: a verdict_dir marker reaches the project's fold; detach
  (drop the run_portfolio merge) -> red; control (no verdict_dir) -> marker absent.
- double-merge idempotency: a verdict merged before every project folds exactly
  once (first-write-wins on id).

437->442 green, golden byte-exact, full gate clean (ruff + format + mypy strict).
run_s10.py and runs/ byte-untouched. README synced (test count, portfolio block,
load-bearing list). K2 re-entrancy test stays green — the shared store threads
verdict fold lines only, never bundle context markers.

[skip-docs]: no invariant changed (CLAUDE.md untouched); the run_portfolio and
compose_run_context docstrings + README carry the doc need.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RiTwaKLesgcwXx2mDviqpt
This commit is contained in:
Kjell Tore Guttormsen 2026-07-23 22:08:02 +02:00
commit 9bae4fb563
4 changed files with 275 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

21
README.md
View file

@ -13,7 +13,7 @@ human-in-the-loop, and the system learns from the verdicts.
> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
> seam, each proven by load-bearing tests (437 tests, all running offline without an API
> seam, each proven by load-bearing tests (442 tests, all running offline without an API
> key). The programme's single budgeted **live model run has been executed and validated** —
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
@ -93,11 +93,14 @@ description, never from its code)
budget stop included. The model client is injected, so the offline suite proves the
same orchestration with a scripted client; only the CLI's default constructs the SDK
client.
- `portfolio.py` — the sequential multi-project run: `run_portfolio` drives N projects from
a schema-validated reference config, composing each project's context afresh (re-entrant,
fresh debate state per run) and collecting one result per project in config order. The
§8 budget meter is the one explicitly shared, portfolio-wide cap; the default failure
policy raises (a stack-local choice until D-D flips it to collect-and-continue).
- `portfolio.py` — the sequential multi-project run and learning loop: `run_portfolio` drives
N projects from a schema-validated reference config, composing each project's context afresh
(re-entrant, fresh debate state per run) and collecting one result per project in config
order. Two things are deliberately shared portfolio-wide: the §8 budget meter (the cap) and a
single learning `VerdictStore` — a verdict available when project k composes survives into
project k+1's fold (cross-project threading, §5), and an optional `verdict_dir` is the
portfolio-level expert inbox the system reads before each fold. The default failure policy
raises (a stack-local choice until D-D flips it to collect-and-continue).
- `run_s10.py` — the programme's ONE live run (cost discipline D6); run-path only.
### Load-bearing tests (§11)
@ -110,7 +113,9 @@ proposal), `test_step5_refine_loadbearing.py` (the rejection reason verifiably r
retry prompt, and the loop still stops at the cap), `test_step7_async_loop_loadbearing.py`
(a verdict dropped after run A reaches run B's prompt through the file loop, with an
empty-inbox control), `test_step8_promotion_loadbearing.py` (the gate refuses non-approved
verdicts; the promoted signal stays out of the read-context), and
verdicts; the promoted signal stays out of the read-context),
`test_portfolio_learning_loadbearing.py` (a verdict available at project k survives into
project k+1's fold via the shared store, with a marker-absent control), and
`test_sdk_isolation.py` (local config cannot capture the checker).
## The ingest layer — CSV and SQL, in front of the loop
@ -173,7 +178,7 @@ Python ≥3.10 · [`claude-agent-sdk`](https://pypi.org/project/claude-agent-sdk
```bash
uv sync # install dependencies
uv run pytest # 437 tests — run without any API key and without network
uv run pytest # 442 tests — run without any API key and without network
uv run ruff check . && uv run ruff format --check .
uv run mypy src # strict
```

37
src/portfolio_optimiser_claude/portfolio.py
View file

@ -1,4 +1,4 @@
"""Sequential multi-project portfolio run (method-spec §3, §5, §8; paritetsrad 4).
"""Sequential multi-project portfolio run + learning loop (method-spec §3, §5, §8; paritetsrad 4–5).
``run_portfolio`` drives N projects SEQUENTIALLY from a schema-validated reference
config. For each project it composes the §5 read-context (merge inbox → seed →
@ -8,15 +8,18 @@ run path MAF got in its Fase 1 and D7 never had (the prior entrances,
``run_s10.py`` and ``run.py``, drive a single bundle).
Re-entrancy (§3 Step 3): the loop core keeps all debate state local to a run, so
nothing survives one project into the next EXCEPT the explicitly shared mutable
state — the §8 budget ``meter``, a PORTFOLIO-WIDE cap threaded through every run.
Each project composes its OWN context inside the loop, never a hoisted, shared one.
nothing survives one project into the next EXCEPT the deliberately shared state —
the §8 budget ``meter`` (a PORTFOLIO-WIDE cap) and, for the learning loop (K3,
paritetsrad 5; §3 Step 1, §5), a single ``VerdictStore``: a verdict available when
project k composes survives into project k+1's fold (cross-project threading). Each
project still composes its OWN context inside the loop, never a hoisted, shared one.
Failure policy is a STACK-LOCAL choice until D-D: the default RAISES (today
everything is thrown — a budget stop or any run error propagates and the portfolio
stops). K18 flips this to a collect-and-continue wave model when the D-D fasit
lands. Goals/ledger are out of scope here beyond being accepted as optional
arguments in a later session (K3/K11); this module only wires the run path.
arguments in a later session (K11); this module wires the run path and the
portfolio learning store, nothing more.
"""
from __future__ import annotations
@ -26,6 +29,8 @@ from pathlib import Path
from portfolio_optimiser_claude.budget import BudgetMeter
from portfolio_optimiser_claude.contracts import ReferenceProjectsContract
from portfolio_optimiser_claude.experience import VerdictStore
from portfolio_optimiser_claude.inbox import merge_inbox_into_store
from portfolio_optimiser_claude.loop import ModelClient, RunResult, run_project
from portfolio_optimiser_claude.run import compose_run_context
@ -53,18 +58,36 @@ def run_portfolio(
top_k: int,
max_debate_rounds: int,
max_attempts: int,
verdict_dir: Path | None = None,
) -> PortfolioResult:
"""Run each configured project through the loop, collecting results in config order.
The ``projects`` config is ALREADY schema-validated (``load_reference_projects``,
§10) — an invalid entry never reaches here. The shared ``meter`` is the
portfolio-wide §8 cap; the default failure policy RAISES on the first run error.
The portfolio learning loop (K3, paritetsrad 5; §3 Step 1, §5): a SINGLE
``VerdictStore`` is threaded through every project, so a verdict available
when project k composes survives into project k+1's fold (§5 cross-project
threading). ``verdict_dir`` is the OPTIONAL portfolio-level expert inbox —
the system READS it before each run's fold (role split §3 Step 7: the
portfolio never writes a run's own verdict back). Composition still happens
INSIDE the loop (re-entrancy §3 Step 3) — only the learning store and the
§8 meter are the deliberately shared, portfolio-wide state.
"""
shared_store = VerdictStore()
results: list[ProjectRunResult] = []
for project in projects.projects:
# The portfolio inbox is merged BEFORE each fold (§5); repeated merges are
# idempotent (first-write-wins on id), so re-merging every project is safe.
if verdict_dir is not None:
merge_inbox_into_store(shared_store, verdict_dir)
inbox_dir = Path(project.inbox_dir) if project.inbox_dir is not None else None
# Fresh composition per project (re-entrancy §3 Step 3) — never hoisted.
composed = compose_run_context(Path(project.bundle_dir), inbox_dir, k=top_k)
# Fresh composition per project (re-entrancy §3 Step 3) — never hoisted;
# the shared store is the ONLY verdict state carried across projects.
composed = compose_run_context(
Path(project.bundle_dir), inbox_dir, k=top_k, store=shared_store
)
run = run_project(
client,
composed.context,

11
src/portfolio_optimiser_claude/run.py
View file

@ -61,7 +61,7 @@ class ComposedRunContext:
def compose_run_context(
bundle_dir: Path, inbox_dir: Path | None = None, *, k: int
bundle_dir: Path, inbox_dir: Path | None = None, *, k: int, store: VerdictStore | None = None
) -> ComposedRunContext:
"""Compose the run context per §5: merge inbox → seed → fold — read-only.
@ -69,10 +69,17 @@ def compose_run_context(
fast ahead of any spend (§9). A missing/empty ``inbox_dir`` (or ``None``)
leaves the composition identical to the no-inbox base. Nothing is ever
written — the system reads the inbox, the expert writes it (§3 Step 7).
A passed-in ``store`` is used AS-IS (its existing verdicts survive the
merge, first-write-wins) — the §5 cross-project threading a portfolio pass
(K3) relies on: a verdict available at project k reaches project k+1's
fold. ``None`` (the single-run default) builds a fresh store, so every
existing caller composes exactly as before.
"""
citations = build_citations(navigate_bundle(bundle_dir))
ir_projection = load_validator_input(bundle_dir)
store = VerdictStore()
if store is None:
store = VerdictStore()
inbox_merged = merge_inbox_into_store(store, inbox_dir) if inbox_dir is not None else 0
seeded = seed_store_from_bundle(store, bundle_dir)
context = fold_experience(

223
tests/test_portfolio_learning_loadbearing.py Normal file
View file

@ -0,0 +1,223 @@
"""Portfolio learning loop — LOAD-BEARING (K3; method-spec §3 Step 1, §5, §11; paritetsrad 5).
The seam this file keeps alive: within ONE portfolio pass, a verdict available
when project k is composed SURVIVES into project k+1's hypothesis prompt. The
carrier is a SINGLE ``VerdictStore`` threaded through every project (§5: "a
passed-in store's existing verdicts survive — cross-project threading") plus a
portfolio-level ``verdict_dir`` (the expert inbox, merged read-only before each
run's fold). Honesty rule (§1): the crossing carries EXPERT / bundle-seed
verdicts only — never a run's own captured output. The role split is unwaivable
(§3 Step 7): the portfolio READS ``verdict_dir``; it NEVER writes a run's verdict
back into it, and the promotion gate / authoring primitive remain the only
writers.
Detach proofs (each restored from a copy of the implemented version, never
``git checkout`` — that would go to HEAD and erase the uncommitted src):
* Shared store (§5, cross-project threading): make ``compose_run_context`` ignore
its passed-in ``store`` and always build a fresh one (``store = VerdictStore()``
unconditionally) → project 2 seeds only its OWN bundle → project 1's seed id no
longer reaches project 2's prompt → ``test_earlier_project_seed_survives_into_
later_project`` goes red.
* Inbox threading (§5): drop the ``verdict_dir`` merge in ``run_portfolio`` (or stop
passing the shared store into ``compose_run_context``) → the portfolio inbox
verdict never reaches any fold → ``test_portfolio_inbox_verdict_folds_into_
project_prompt`` goes red.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from _scripted import ScriptedClient, reply
from portfolio_optimiser_claude.budget import BudgetMeter
from portfolio_optimiser_claude.contracts import (
ReferenceProjectsContract,
TerminationContract,
load_reference_projects,
)
from portfolio_optimiser_claude.experience import (
CandidateFeatures,
mint_verdict_id,
)
from portfolio_optimiser_claude.inbox import VerdictDocument, write_verdict
from portfolio_optimiser_claude.ir import load_validator_input
from portfolio_optimiser_claude.portfolio import run_portfolio
_REPO = Path(__file__).resolve().parents[1]
LED_BUNDLE = _REPO / "shared" / "examples" / "bygg-energi-mikro"
VFD_BUNDLE = _REPO / "tests" / "data" / "mini-bundle"
VFD_MARKER = "K2-VFD-CONTEXT-MARKER" # in the repo-local fixture's rendered context
# A token that lives NOWHERE in either bundle — its presence in a prompt can only
# come from the portfolio ``verdict_dir`` fold, never from context or seeding.
INBOX_MARKER = "K3-PORTFOLIO-LEARNING-MARKER"
def _meter(*, max_rounds: int = 100, max_tokens: int = 10_000) -> BudgetMeter:
return BudgetMeter(TerminationContract(max_rounds=max_rounds, max_tokens=max_tokens))
def _happy_replies(bundles: list[Path]) -> list[object]:
"""Three scripted replies per project: debate reasoning → APPROVE → candidate JSON."""
replies: list[object] = []
for bundle in bundles:
replies += [
reply("debate reasoning"),
reply("VERDICT: APPROVE"),
reply(json.dumps(load_validator_input(bundle).model_dump())),
]
return replies
def _config(entries: list[tuple[str, Path]]) -> ReferenceProjectsContract:
return load_reference_projects(
{"projects": [{"project_id": pid, "bundle_dir": str(path)} for pid, path in entries]}
)
def _seed_id(bundle: Path) -> str:
"""The id the bundle's ``type: verdict`` seed is keyed on (§3 Step 1 / §4.2)."""
return mint_verdict_id(CandidateFeatures.from_proposal(load_validator_input(bundle)))
def _author_inbox_verdict(verdict_dir: Path, marker: str) -> VerdictDocument:
"""Drop an EXPERT verdict into the portfolio inbox (§5 write side, expert-authored).
Keyed on features similar to the VFD candidate (same code + measure type, a
distinct saving so its id does not collide with the VFD seed), so it ranks
high for the VFD project. The marker rides in the rationale — the fold's
learning-signal carrier (§3 Step 1).
"""
vfd = CandidateFeatures.from_proposal(load_validator_input(VFD_BUNDLE))
features = CandidateFeatures(
affected_codes=vfd.affected_codes,
measure_type=vfd.measure_type,
claimed_saving_nok=vfd.claimed_saving_nok + 3000.0, # same bucket, distinct id
)
verdict = VerdictDocument.from_candidate(
features,
decision="approved",
rationale=f"prior portfolio verdict — realiseringskorreksjon [{marker}]",
description="K3 portfolio inbox fixture (surface text, excluded from ranking)",
)
write_verdict(verdict_dir, verdict)
return verdict
class TestCrossProjectThreading:
"""§5 cross-project threading: a passed-in store's verdicts survive into the next run.
The shared ``VerdictStore`` ``run_portfolio`` owns carries project 1's bundle
seed forward, so it is still retrievable when project 2 composes — the
portfolio learns across projects in a single pass (S2.0, paritetsrad 5).
"""
def test_earlier_project_seed_survives_into_later_project(self) -> None:
projects = _config([("BYGG-KONTOR-NORD", LED_BUNDLE), ("PUMPE-SOR", VFD_BUNDLE)])
client = ScriptedClient(replies=_happy_replies([LED_BUNDLE, VFD_BUNDLE]))
run_portfolio(projects, client, _meter(), top_k=3, max_debate_rounds=3, max_attempts=3)
led_seed_id = _seed_id(LED_BUNDLE)
vfd_prompts = [p for p in client.prompts("proposer") if VFD_MARKER in p]
assert vfd_prompts, "project 2's own context must reach its prompts"
# The shared store carried project 1's seed into project 2's fold.
assert any(led_seed_id in p for p in vfd_prompts), (
"project 1's seed must survive into project 2's prompt via the shared store"
)
def test_threading_is_order_directional(self) -> None:
# Reverse the config: now VFD is project 1 → its seed threads into LED's prompt,
# and LED's seed (project 2, composed last) is absent from project 1's prompt.
led_seed_id = _seed_id(LED_BUNDLE)
vfd_seed_id = _seed_id(VFD_BUNDLE)
projects = _config([("PUMPE-SOR", VFD_BUNDLE), ("BYGG-KONTOR-NORD", LED_BUNDLE)])
client = ScriptedClient(replies=_happy_replies([VFD_BUNDLE, LED_BUNDLE]))
run_portfolio(projects, client, _meter(), top_k=3, max_debate_rounds=3, max_attempts=3)
vfd_prompts = [p for p in client.prompts("proposer") if VFD_MARKER in p]
assert vfd_prompts
# Project 1 (VFD) ran before LED was ever seeded → LED's seed cannot appear.
assert not any(led_seed_id in p for p in vfd_prompts), (
"a later project's seed must not reach an earlier project's prompt"
)
# VFD's own seed is of course present in its own fold.
assert any(vfd_seed_id in p for p in vfd_prompts)
class TestPortfolioInboxFold:
"""A portfolio-level ``verdict_dir`` (expert inbox) folds into each project (§5, §3 Step 1)."""
def test_portfolio_inbox_verdict_folds_into_project_prompt(self, tmp_path: Path) -> None:
verdict_dir = tmp_path / "portfolio-inbox"
inbox_verdict = _author_inbox_verdict(verdict_dir, INBOX_MARKER)
projects = _config([("BYGG-KONTOR-NORD", LED_BUNDLE), ("PUMPE-SOR", VFD_BUNDLE)])
client = ScriptedClient(replies=_happy_replies([LED_BUNDLE, VFD_BUNDLE]))
run_portfolio(
projects,
client,
_meter(),
top_k=3,
max_debate_rounds=3,
max_attempts=3,
verdict_dir=verdict_dir,
)
vfd_prompts = [p for p in client.prompts("proposer") if VFD_MARKER in p]
assert vfd_prompts
assert any(INBOX_MARKER in p and inbox_verdict.id in p for p in vfd_prompts), (
"the portfolio inbox verdict (id + marker) must reach the project's fold"
)
def test_no_verdict_dir_leaves_the_marker_absent(self) -> None:
# Control: without a portfolio inbox, the marker appears in NO prompt — it
# has no other channel into the context (seeding and bundle context never
# carry it).
projects = _config([("BYGG-KONTOR-NORD", LED_BUNDLE), ("PUMPE-SOR", VFD_BUNDLE)])
client = ScriptedClient(replies=_happy_replies([LED_BUNDLE, VFD_BUNDLE]))
run_portfolio(projects, client, _meter(), top_k=3, max_debate_rounds=3, max_attempts=3)
assert not any(INBOX_MARKER in prompt for _role, prompt in client.calls)
class TestPortfolioMergeIdempotency:
"""§4.2 first-write-wins: repeated portfolio merges are idempotent (double-merge, §5).
``verdict_dir`` is merged before EVERY project's fold. Across a three-project
pass the same inbox verdict is merged three times, yet the shared store holds
it exactly ONCE — its fold line is not duplicated.
"""
def test_repeated_merges_keep_a_single_store_entry(self, tmp_path: Path) -> None:
verdict_dir = tmp_path / "portfolio-inbox"
inbox_verdict = _author_inbox_verdict(verdict_dir, INBOX_MARKER)
projects = _config(
[
("BYGG-KONTOR-NORD", LED_BUNDLE),
("PUMPE-SOR", VFD_BUNDLE),
("BYGG-KONTOR-NORD-2", LED_BUNDLE),
]
)
client = ScriptedClient(replies=_happy_replies([LED_BUNDLE, VFD_BUNDLE, LED_BUNDLE]))
run_portfolio(
projects,
client,
_meter(),
top_k=3,
max_debate_rounds=3,
max_attempts=3,
verdict_dir=verdict_dir,
)
vfd_prompts = [p for p in client.prompts("proposer") if VFD_MARKER in p]
assert vfd_prompts
# The verdict was merged three times (once before each project) but appears
# once per fold — first-write-wins deduplicates on id.
assert all(p.count(inbox_verdict.id) == 1 for p in vfd_prompts), (
"a verdict merged N times must fold exactly once (first-write-wins, §4.2)"
)
if __name__ == "__main__": # pragma: no cover
raise SystemExit(pytest.main([__file__, "-q"]))