feat(portfolio): K2 — sequential multi-project run (parity row 4)
New portfolio.py: run_portfolio drives N projects sequentially from a schema-validated reference config, composing each project's §5 context (merge inbox -> seed -> fold) and running the loop core UNCHANGED per project, collecting one typed result per project IN CONFIG ORDER. This is the run path MAF got in its Fase 1 and D7 never had — the prior entrances (run.py, run_s10.py) drive a single bundle. PortfolioResult holds per-project results tagged with the config project_id. Re-entrancy (§3 Step 3): each project composes its OWN context inside the loop, never a hoisted shared one, so nothing survives one project into the next except the explicitly shared mutable state — the §8 budget meter, a portfolio-wide cap. Failure policy is a STACK-LOCAL choice until D-D: the default RAISES (today everything is thrown); K18 flips it to collect-and-continue when the D-D wave model lands. New config contract in contracts.py: ReferenceProjectContract (project_id + required non-empty bundle_dir + optional inbox_dir) + ReferenceProjectsContract, loaded fail-fast by load_reference_projects (§10) — a project without a bundle path is refused before any run. New data/reference_projects.json example (shape-validated, never executed by the suite). New repo-local mini-bundle fixture under tests/data/ (a distinct second project, VFD-retrofit — ALDRI in shared/). Two detach proofs delivered: drop the bundle_dir Field requirement -> a run starts on the invalid config and only crashes mid-run -> the fail-fast test goes red; hoist the per-project composition out of the loop -> project 2 runs on project 1's context and the VFD marker never reaches its prompt -> the re-entrancy test goes red. 11 new tests (test_portfolio.py 5 + test_contracts.py TestReferenceProjects 6). 426 -> 437 tests, golden byte-exact, full gate clean (ruff + format + mypy strict). README synced (test count + a Run layer module block). [skip-docs] — README documents the new module; CLAUDE.md holds invariants (rules/commands) only, and K2 adds no new invariant, command, or convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RiTwaKLesgcwXx2mDviqpt
This commit is contained in:
parent
a7e8ffecb8
commit
f2c64da9ee
10 changed files with 406 additions and 3 deletions
|
|
@ -13,7 +13,7 @@ human-in-the-loop, and the system learns from the verdicts.
|
||||||
> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
|
> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
|
||||||
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
|
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
|
||||||
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
|
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
|
||||||
> seam, each proven by load-bearing tests (426 tests, all running offline without an API
|
> seam, each proven by load-bearing tests (437 tests, all running offline without an API
|
||||||
> key). The programme's single budgeted **live model run has been executed and validated** —
|
> key). The programme's single budgeted **live model run has been executed and validated** —
|
||||||
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
|
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
|
||||||
|
|
||||||
|
|
@ -93,6 +93,11 @@ description, never from its code)
|
||||||
budget stop included. The model client is injected, so the offline suite proves the
|
budget stop included. The model client is injected, so the offline suite proves the
|
||||||
same orchestration with a scripted client; only the CLI's default constructs the SDK
|
same orchestration with a scripted client; only the CLI's default constructs the SDK
|
||||||
client.
|
client.
|
||||||
|
- `portfolio.py` — the sequential multi-project run: `run_portfolio` drives N projects from
|
||||||
|
a schema-validated reference config, composing each project's context afresh (re-entrant,
|
||||||
|
fresh debate state per run) and collecting one result per project in config order. The
|
||||||
|
§8 budget meter is the one explicitly shared, portfolio-wide cap; the default failure
|
||||||
|
policy raises (a stack-local choice until D-D flips it to collect-and-continue).
|
||||||
- `run_s10.py` — the programme's ONE live run (cost discipline D6); run-path only.
|
- `run_s10.py` — the programme's ONE live run (cost discipline D6); run-path only.
|
||||||
|
|
||||||
### Load-bearing tests (§11)
|
### Load-bearing tests (§11)
|
||||||
|
|
@ -168,7 +173,7 @@ Python ≥3.10 · [`claude-agent-sdk`](https://pypi.org/project/claude-agent-sdk
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv sync # install dependencies
|
uv sync # install dependencies
|
||||||
uv run pytest # 426 tests — run without any API key and without network
|
uv run pytest # 437 tests — run without any API key and without network
|
||||||
uv run ruff check . && uv run ruff format --check .
|
uv run ruff check . && uv run ruff format --check .
|
||||||
uv run mypy src # strict
|
uv run mypy src # strict
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -28,6 +28,7 @@ from typing import Annotated, Any, Literal
|
||||||
from pydantic import BaseModel, Field, model_validator
|
from pydantic import BaseModel, Field, model_validator
|
||||||
|
|
||||||
_MODEL_MAP_RESOURCE = "data/model_map.json"
|
_MODEL_MAP_RESOURCE = "data/model_map.json"
|
||||||
|
_REFERENCE_PROJECTS_RESOURCE = "data/reference_projects.json"
|
||||||
|
|
||||||
# C2.6: an empty-string model id is a startup schema error, never a client-layer one.
|
# C2.6: an empty-string model id is a startup schema error, never a client-layer one.
|
||||||
_ModelId = Annotated[str, Field(min_length=1)]
|
_ModelId = Annotated[str, Field(min_length=1)]
|
||||||
|
|
@ -67,6 +68,25 @@ class FeedbackContract(BaseModel):
|
||||||
rationale: str = Field(min_length=1)
|
rationale: str = Field(min_length=1)
|
||||||
|
|
||||||
|
|
||||||
|
class ReferenceProjectContract(BaseModel):
|
||||||
|
"""One portfolio entry: a project id + its OKF bundle dir (+ an optional inbox).
|
||||||
|
|
||||||
|
The ``bundle_dir`` is REQUIRED and non-empty — an entry without a bundle path
|
||||||
|
is a startup schema error (§10), never a run that starts and then crashes for
|
||||||
|
lack of a data source. ``inbox_dir`` is optional (the tolerant §5 inbox).
|
||||||
|
"""
|
||||||
|
|
||||||
|
project_id: str = Field(min_length=1)
|
||||||
|
bundle_dir: str = Field(min_length=1)
|
||||||
|
inbox_dir: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class ReferenceProjectsContract(BaseModel):
|
||||||
|
"""The schema-validated portfolio config (§10: fail-fast before any run)."""
|
||||||
|
|
||||||
|
projects: list[ReferenceProjectContract] = Field(min_length=1)
|
||||||
|
|
||||||
|
|
||||||
class Contracts(BaseModel):
|
class Contracts(BaseModel):
|
||||||
"""The validated bundle of all startup contracts."""
|
"""The validated bundle of all startup contracts."""
|
||||||
|
|
||||||
|
|
@ -98,6 +118,27 @@ def _bundled_model_map() -> dict[str, Any]:
|
||||||
return raw
|
return raw
|
||||||
|
|
||||||
|
|
||||||
|
def _bundled_reference_projects() -> dict[str, Any]:
|
||||||
|
raw: dict[str, Any] = json.loads(
|
||||||
|
files("portfolio_optimiser_claude")
|
||||||
|
.joinpath(_REFERENCE_PROJECTS_RESOURCE)
|
||||||
|
.read_text(encoding="utf-8")
|
||||||
|
)
|
||||||
|
return raw
|
||||||
|
|
||||||
|
|
||||||
|
def load_reference_projects(raw: dict[str, Any] | None = None) -> ReferenceProjectsContract:
|
||||||
|
"""Validate the portfolio config at startup (§10, fail-fast before any run).
|
||||||
|
|
||||||
|
Raises ``pydantic.ValidationError`` on the first malformed entry — a project
|
||||||
|
without a bundle path never reaches ``run_portfolio``. ``raw`` defaults to the
|
||||||
|
bundled ``data/reference_projects.json`` EXAMPLE, whose shape (not its paths)
|
||||||
|
is validated; the suite never executes that example (offline invariant).
|
||||||
|
"""
|
||||||
|
data = _bundled_reference_projects() if raw is None else raw
|
||||||
|
return ReferenceProjectsContract(**data)
|
||||||
|
|
||||||
|
|
||||||
def load_contracts(
|
def load_contracts(
|
||||||
data_source: dict[str, Any],
|
data_source: dict[str, Any],
|
||||||
termination: dict[str, Any],
|
termination: dict[str, Any],
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,9 @@
|
||||||
|
{
|
||||||
|
"_note": "EXAMPLE portfolio config (contracts.ReferenceProjectsContract). Schema-validated fail-fast at startup (§10); the suite validates only its SHAPE and never executes it (offline invariant). bundle_dir/inbox_dir are illustrative paths relative to the repo root — operators point them at their own OKF bundles. The one shipped example bundle is shared/examples/bygg-energi-mikro; a real portfolio lists several.",
|
||||||
|
"projects": [
|
||||||
|
{
|
||||||
|
"project_id": "BYGG-KONTOR-NORD",
|
||||||
|
"bundle_dir": "shared/examples/bygg-energi-mikro"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
77
src/portfolio_optimiser_claude/portfolio.py
Normal file
77
src/portfolio_optimiser_claude/portfolio.py
Normal file
|
|
@ -0,0 +1,77 @@
|
||||||
|
"""Sequential multi-project portfolio run (method-spec §3, §5, §8; paritetsrad 4).
|
||||||
|
|
||||||
|
``run_portfolio`` drives N projects SEQUENTIALLY from a schema-validated reference
|
||||||
|
config. For each project it composes the §5 read-context (merge inbox → seed →
|
||||||
|
fold, via ``compose_run_context``) and runs the loop core UNCHANGED
|
||||||
|
(``run_project``), collecting one typed result per project IN CONFIG ORDER — the
|
||||||
|
run path MAF got in its Fase 1 and D7 never had (the prior entrances,
|
||||||
|
``run_s10.py`` and ``run.py``, drive a single bundle).
|
||||||
|
|
||||||
|
Re-entrancy (§3 Step 3): the loop core keeps all debate state local to a run, so
|
||||||
|
nothing survives one project into the next EXCEPT the explicitly shared mutable
|
||||||
|
state — the §8 budget ``meter``, a PORTFOLIO-WIDE cap threaded through every run.
|
||||||
|
Each project composes its OWN context inside the loop, never a hoisted, shared one.
|
||||||
|
|
||||||
|
Failure policy is a STACK-LOCAL choice until D-D: the default RAISES (today
|
||||||
|
everything is thrown — a budget stop or any run error propagates and the portfolio
|
||||||
|
stops). K18 flips this to a collect-and-continue wave model when the D-D fasit
|
||||||
|
lands. Goals/ledger are out of scope here beyond being accepted as optional
|
||||||
|
arguments in a later session (K3/K11); this module only wires the run path.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from portfolio_optimiser_claude.budget import BudgetMeter
|
||||||
|
from portfolio_optimiser_claude.contracts import ReferenceProjectsContract
|
||||||
|
from portfolio_optimiser_claude.loop import ModelClient, RunResult, run_project
|
||||||
|
from portfolio_optimiser_claude.run import compose_run_context
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ProjectRunResult:
|
||||||
|
"""One project's outcome in the portfolio, tagged with its config ``project_id``."""
|
||||||
|
|
||||||
|
project_id: str
|
||||||
|
run: RunResult
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class PortfolioResult:
|
||||||
|
"""The portfolio's per-project results, in config order."""
|
||||||
|
|
||||||
|
results: list[ProjectRunResult]
|
||||||
|
|
||||||
|
|
||||||
|
def run_portfolio(
|
||||||
|
projects: ReferenceProjectsContract,
|
||||||
|
client: ModelClient,
|
||||||
|
meter: BudgetMeter,
|
||||||
|
*,
|
||||||
|
top_k: int,
|
||||||
|
max_debate_rounds: int,
|
||||||
|
max_attempts: int,
|
||||||
|
) -> PortfolioResult:
|
||||||
|
"""Run each configured project through the loop, collecting results in config order.
|
||||||
|
|
||||||
|
The ``projects`` config is ALREADY schema-validated (``load_reference_projects``,
|
||||||
|
§10) — an invalid entry never reaches here. The shared ``meter`` is the
|
||||||
|
portfolio-wide §8 cap; the default failure policy RAISES on the first run error.
|
||||||
|
"""
|
||||||
|
results: list[ProjectRunResult] = []
|
||||||
|
for project in projects.projects:
|
||||||
|
inbox_dir = Path(project.inbox_dir) if project.inbox_dir is not None else None
|
||||||
|
# Fresh composition per project (re-entrancy §3 Step 3) — never hoisted.
|
||||||
|
composed = compose_run_context(Path(project.bundle_dir), inbox_dir, k=top_k)
|
||||||
|
run = run_project(
|
||||||
|
client,
|
||||||
|
composed.context,
|
||||||
|
meter=meter,
|
||||||
|
max_debate_rounds=max_debate_rounds,
|
||||||
|
max_attempts=max_attempts,
|
||||||
|
default_project_id=composed.ir_projection.project_id,
|
||||||
|
)
|
||||||
|
results.append(ProjectRunResult(project_id=project.project_id, run=run))
|
||||||
|
return PortfolioResult(results=results)
|
||||||
21
tests/data/mini-bundle/index.md
Normal file
21
tests/data/mini-bundle/index.md
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
---
|
||||||
|
type: index
|
||||||
|
title: "Mini-bundle (K2-fixture) — pumpe-VFD, ett tiltak"
|
||||||
|
description: "Repo-lokal mini OKF-bundle for portefølje-testing (K2): ett bygg, ett VFD-retrofit-tiltak. ALDRI i shared/ — dette er en test-fixture, ikke en delt fasit."
|
||||||
|
tags: [fixture, portefølje, K2]
|
||||||
|
timestamp: 2026-07-16
|
||||||
|
---
|
||||||
|
|
||||||
|
# Mini-bundle (K2-fixture) — K2-VFD-CONTEXT-MARKER
|
||||||
|
|
||||||
|
En repo-lokal **mini OKF-bundle** brukt KUN av porteføljetesten (`test_portfolio.py`): ett
|
||||||
|
bygg, ett kandidat-tiltak (VFD-retrofit på sirkulasjonspumper). Den ligger under `tests/data/`
|
||||||
|
med vilje — den er en test-fixture og deles ALDRI som fasit i `shared/`.
|
||||||
|
|
||||||
|
Formålet er et andre, distinkt prosjekt ved siden av `shared/examples/bygg-energi-mikro`, så
|
||||||
|
`run_portfolio` kan kjøre N prosjekter sekvensielt med per-prosjekt kontekst.
|
||||||
|
|
||||||
|
## Innhold
|
||||||
|
|
||||||
|
- [tiltak-vfd.md](tiltak-vfd.md) — `type: hypothesis` — VFD-retrofit-kandidaten.
|
||||||
|
- [verdict-vfd-fro.md](verdict-vfd-fro.md) — `type: verdict` — frøsatt ekspert-dom (ExpeL-frø).
|
||||||
21
tests/data/mini-bundle/tiltak-vfd.md
Normal file
21
tests/data/mini-bundle/tiltak-vfd.md
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
---
|
||||||
|
type: hypothesis
|
||||||
|
title: "VFD-retrofit av sirkulasjonspumper"
|
||||||
|
description: "Turtallsregulering (variable frequency drive) på 12 sirkulasjonspumper som i dag går på konstant hastighet."
|
||||||
|
measure_id: VFD-RETROFIT-01
|
||||||
|
resource: PUMPE-SOR
|
||||||
|
tags: [pumper, VFD, energieffektivisering]
|
||||||
|
timestamp: 2026-07-16
|
||||||
|
---
|
||||||
|
|
||||||
|
# VFD-retrofit av sirkulasjonspumper
|
||||||
|
|
||||||
|
Kandidat-tiltaket: montere frekvensomformere (VFD) på 12 sirkulasjonspumper i et
|
||||||
|
sørvendt næringsbygg. Pumpene går i dag på konstant turtall; med turtallsregulering
|
||||||
|
følger de faktisk lastbehov, og pumpeeffekten faller tilnærmet kubisk med turtallet.
|
||||||
|
|
||||||
|
## Mapping til validatoren
|
||||||
|
|
||||||
|
`validator-input.json` projiserer tiltaket inn i kost-IR-en: `affected_items` er byggets
|
||||||
|
årlige pumpe-energikostnad (`ENERGI-PUMPE-EL`), `claimed_saving_nok` er den modellerte
|
||||||
|
VFD-besparelsen, og `assumptions` bærer energipris-båndet for Monte Carlo-en.
|
||||||
16
tests/data/mini-bundle/validator-input.json
Normal file
16
tests/data/mini-bundle/validator-input.json
Normal file
|
|
@ -0,0 +1,16 @@
|
||||||
|
{
|
||||||
|
"_note": "IR-projeksjon (ir.SavingsProposal) for K2-fixturen. Distinkt fra bygg-energi-mikro: annet prosjekt, annet tiltak, annen kost-kode. claimed_saving_nok ligger godt under feasibel-grensen (0.30 x 120000 = 36000) slik at validate_proposal deterministisk gir ValidatedProposal uansett MC-band.",
|
||||||
|
"project_id": "PUMPE-SOR",
|
||||||
|
"measure": "VFD-retrofit av 12 sirkulasjonspumper (konstant -> turtallsregulert drift)",
|
||||||
|
"affected_items": [
|
||||||
|
{
|
||||||
|
"code": "ENERGI-PUMPE-EL",
|
||||||
|
"quantity": 120000,
|
||||||
|
"unit_cost": 1.0
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"claimed_saving_nok": 12000,
|
||||||
|
"assumptions": {
|
||||||
|
"ENERGI-PUMPE-EL": [0.70, 1.40]
|
||||||
|
}
|
||||||
|
}
|
||||||
26
tests/data/mini-bundle/verdict-vfd-fro.md
Normal file
26
tests/data/mini-bundle/verdict-vfd-fro.md
Normal file
|
|
@ -0,0 +1,26 @@
|
||||||
|
---
|
||||||
|
type: verdict
|
||||||
|
title: "Ekspert-dom (frø): VFD-retrofit — godkjent med realiseringskorreksjon"
|
||||||
|
description: "Frøsatt ekspert-dom for VFD-tiltaket. Koder et realiseringsgap distinkt fra LED-frøet (0.75, ikke 0.82) så porteføljetesten kan skille prosjektenes kontekst."
|
||||||
|
resource: PUMPE-SOR
|
||||||
|
measure_id: VFD-RETROFIT-01
|
||||||
|
decision: approved_with_adjustment
|
||||||
|
realization_rate: 0.75
|
||||||
|
modelled_saving_nok: 12000
|
||||||
|
expected_actual_saving_nok: 9000
|
||||||
|
gap_source: pump-affinity-oversimplification
|
||||||
|
context_key: "naeringsbygg; pumpe-last=stipulert"
|
||||||
|
provenance: "frø — AI-forfattet test-fixture; erstattes av ekte HITL i produksjon"
|
||||||
|
tags: [verdict, realization-rate, ExpeL-seed, fixture]
|
||||||
|
timestamp: 2026-07-16
|
||||||
|
---
|
||||||
|
|
||||||
|
# Ekspert-dom (frø): VFD-retrofit
|
||||||
|
|
||||||
|
> **Dette er et frø**, ikke en ekte dom — en test-fixture for porteføljekjøringen.
|
||||||
|
|
||||||
|
## Dommen
|
||||||
|
|
||||||
|
**Beslutning:** godkjent — med realiseringskorreksjon. Den modellerte besparelsen er
|
||||||
|
teknisk plausibel, men i drift realiseres erfaringsvis ~75 % fordi den antatte
|
||||||
|
affinitets-forenklingen overvurderer hvor mye tid pumpene faktisk kjører på redusert turtall.
|
||||||
|
|
@ -12,7 +12,7 @@ from typing import Any
|
||||||
import pytest
|
import pytest
|
||||||
from pydantic import ValidationError
|
from pydantic import ValidationError
|
||||||
|
|
||||||
from portfolio_optimiser_claude.contracts import load_contracts
|
from portfolio_optimiser_claude.contracts import load_contracts, load_reference_projects
|
||||||
|
|
||||||
VALID_DATA_SOURCE: dict[str, Any] = {"docs_dir": "docs", "top_k": 3}
|
VALID_DATA_SOURCE: dict[str, Any] = {"docs_dir": "docs", "top_k": 3}
|
||||||
VALID_TERMINATION: dict[str, Any] = {"max_rounds": 4, "max_tokens": 20_000}
|
VALID_TERMINATION: dict[str, Any] = {"max_rounds": 4, "max_tokens": 20_000}
|
||||||
|
|
@ -118,6 +118,42 @@ class TestModelMap:
|
||||||
assert "default" in contracts.model_map.profiles["anthropic"]
|
assert "default" in contracts.model_map.profiles["anthropic"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestReferenceProjects:
|
||||||
|
"""§10 (K2): the portfolio config — each entry needs a non-empty bundle path."""
|
||||||
|
|
||||||
|
def test_valid_config_accepted(self) -> None:
|
||||||
|
config = load_reference_projects(
|
||||||
|
{"projects": [{"project_id": "P1", "bundle_dir": "bundles/p1"}]}
|
||||||
|
)
|
||||||
|
assert config.projects[0].project_id == "P1"
|
||||||
|
assert config.projects[0].inbox_dir is None
|
||||||
|
|
||||||
|
def test_missing_bundle_dir_rejected(self) -> None:
|
||||||
|
# The fail-fast seam K2's detach proof removes: no bundle path is a
|
||||||
|
# startup schema error, never a run that starts then crashes for lack of data.
|
||||||
|
with pytest.raises(ValidationError):
|
||||||
|
load_reference_projects({"projects": [{"project_id": "P1"}]})
|
||||||
|
|
||||||
|
def test_empty_bundle_dir_rejected(self) -> None:
|
||||||
|
with pytest.raises(ValidationError):
|
||||||
|
load_reference_projects({"projects": [{"project_id": "P1", "bundle_dir": ""}]})
|
||||||
|
|
||||||
|
def test_empty_project_id_rejected(self) -> None:
|
||||||
|
with pytest.raises(ValidationError):
|
||||||
|
load_reference_projects({"projects": [{"project_id": "", "bundle_dir": "b"}]})
|
||||||
|
|
||||||
|
def test_empty_projects_list_rejected(self) -> None:
|
||||||
|
with pytest.raises(ValidationError):
|
||||||
|
load_reference_projects({"projects": []})
|
||||||
|
|
||||||
|
def test_bundled_reference_projects_is_valid(self) -> None:
|
||||||
|
# raw=None falls back to the bundled data/reference_projects.json EXAMPLE,
|
||||||
|
# whose shape must itself satisfy the contract (fail-fast on shipped config).
|
||||||
|
config = load_reference_projects()
|
||||||
|
assert config.projects
|
||||||
|
assert all(p.bundle_dir for p in config.projects)
|
||||||
|
|
||||||
|
|
||||||
def test_happy_path_validates_all_four() -> None:
|
def test_happy_path_validates_all_four() -> None:
|
||||||
contracts = load()
|
contracts = load()
|
||||||
assert contracts.data_source.top_k == 3
|
assert contracts.data_source.top_k == 3
|
||||||
|
|
|
||||||
151
tests/test_portfolio.py
Normal file
151
tests/test_portfolio.py
Normal file
|
|
@ -0,0 +1,151 @@
|
||||||
|
"""Portfolio run — LOAD-BEARING (K2; method-spec §3, §5, §8, §10, §11; paritetsrad 4).
|
||||||
|
|
||||||
|
The seam this file keeps alive: ``run_portfolio`` drives N projects SEQUENTIALLY
|
||||||
|
from a schema-validated reference config, composing each project's §5 context
|
||||||
|
(merge inbox → seed → fold) and running the loop core UNCHANGED per project,
|
||||||
|
collecting per-project results IN CONFIG ORDER. This is the run path MAF got in
|
||||||
|
its Fase 1 and D7 never had — the only prior run entrance (``run_s10.py``,
|
||||||
|
``run.py``) drives a single bundle.
|
||||||
|
|
||||||
|
Detach proofs (each restored from a copy, never ``git checkout``):
|
||||||
|
|
||||||
|
* Fail-fast (§10): drop the ``bundle_dir`` Field requirement in
|
||||||
|
``ReferenceProjectContract`` (e.g. ``bundle_dir: str = ""``) → a config missing
|
||||||
|
its bundle path no longer raises at load → ``run_portfolio`` STARTS on the
|
||||||
|
invalid config and only crashes mid-run → ``test_malformed_config_never_reaches
|
||||||
|
_a_client`` goes red (``FileNotFoundError``, not ``ValidationError``).
|
||||||
|
* Re-entrancy (§3 Step 3, the key assumption): hoist the per-project
|
||||||
|
``compose_run_context`` call OUT of the loop (compose project 1 once, reuse it
|
||||||
|
for every project) → project 2 runs on project 1's context → the VFD marker
|
||||||
|
never reaches any prompt → ``test_each_project_composes_its_own_context`` goes red.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from pydantic import ValidationError
|
||||||
|
|
||||||
|
from _scripted import ScriptedClient, reply
|
||||||
|
|
||||||
|
from portfolio_optimiser_claude.budget import BudgetExceeded, BudgetMeter
|
||||||
|
from portfolio_optimiser_claude.contracts import (
|
||||||
|
ReferenceProjectsContract,
|
||||||
|
TerminationContract,
|
||||||
|
load_reference_projects,
|
||||||
|
)
|
||||||
|
from portfolio_optimiser_claude.ir import load_validator_input
|
||||||
|
from portfolio_optimiser_claude.portfolio import PortfolioResult, run_portfolio
|
||||||
|
from portfolio_optimiser_claude.validator import ValidatedProposal
|
||||||
|
|
||||||
|
_REPO = Path(__file__).resolve().parents[1]
|
||||||
|
LED_BUNDLE = _REPO / "shared" / "examples" / "bygg-energi-mikro"
|
||||||
|
VFD_BUNDLE = _REPO / "tests" / "data" / "mini-bundle"
|
||||||
|
|
||||||
|
# Distinctive tokens that reach the composed context of ONE bundle only.
|
||||||
|
LED_MARKER = "LED-retrofit" # in the shared bundle's rendered context
|
||||||
|
VFD_MARKER = "K2-VFD-CONTEXT-MARKER" # embedded in the repo-local fixture's index
|
||||||
|
|
||||||
|
|
||||||
|
def _meter(*, max_rounds: int = 100, max_tokens: int = 10_000) -> BudgetMeter:
|
||||||
|
return BudgetMeter(TerminationContract(max_rounds=max_rounds, max_tokens=max_tokens))
|
||||||
|
|
||||||
|
|
||||||
|
def _happy_replies(bundles: list[Path]) -> list[object]:
|
||||||
|
"""Three scripted replies per project: debate reasoning → APPROVE → candidate JSON.
|
||||||
|
|
||||||
|
The candidate JSON is each bundle's own validated IR projection, so the
|
||||||
|
deterministic validator returns ``ValidatedProposal`` first try (§3 Steps 2–4).
|
||||||
|
"""
|
||||||
|
replies: list[object] = []
|
||||||
|
for bundle in bundles:
|
||||||
|
replies += [
|
||||||
|
reply("debate reasoning"),
|
||||||
|
reply("VERDICT: APPROVE"),
|
||||||
|
reply(json.dumps(load_validator_input(bundle).model_dump())),
|
||||||
|
]
|
||||||
|
return replies
|
||||||
|
|
||||||
|
|
||||||
|
def _config(entries: list[tuple[str, Path]]) -> ReferenceProjectsContract:
|
||||||
|
return load_reference_projects(
|
||||||
|
{"projects": [{"project_id": pid, "bundle_dir": str(path)} for pid, path in entries]}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestSequentialConfigOrder:
|
||||||
|
"""Two-project config → both run, results collected in config order (§3, paritetsrad 4)."""
|
||||||
|
|
||||||
|
def test_two_project_config_runs_both_in_config_order(self) -> None:
|
||||||
|
projects = _config([("BYGG-KONTOR-NORD", LED_BUNDLE), ("PUMPE-SOR", VFD_BUNDLE)])
|
||||||
|
client = ScriptedClient(replies=_happy_replies([LED_BUNDLE, VFD_BUNDLE]))
|
||||||
|
result = run_portfolio(
|
||||||
|
projects, client, _meter(), top_k=3, max_debate_rounds=3, max_attempts=3
|
||||||
|
)
|
||||||
|
assert isinstance(result, PortfolioResult)
|
||||||
|
assert [r.project_id for r in result.results] == ["BYGG-KONTOR-NORD", "PUMPE-SOR"]
|
||||||
|
assert all(isinstance(r.run.outcome, ValidatedProposal) for r in result.results)
|
||||||
|
|
||||||
|
def test_result_order_follows_config_order_not_incidentally(self) -> None:
|
||||||
|
# Reverse the config → the results reverse with it: order is the config's.
|
||||||
|
projects = _config([("PUMPE-SOR", VFD_BUNDLE), ("BYGG-KONTOR-NORD", LED_BUNDLE)])
|
||||||
|
client = ScriptedClient(replies=_happy_replies([VFD_BUNDLE, LED_BUNDLE]))
|
||||||
|
result = run_portfolio(
|
||||||
|
projects, client, _meter(), top_k=3, max_debate_rounds=3, max_attempts=3
|
||||||
|
)
|
||||||
|
assert [r.project_id for r in result.results] == ["PUMPE-SOR", "BYGG-KONTOR-NORD"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestReEntrancy:
|
||||||
|
"""Key assumption (§3 Step 3): the loop core is re-entrant — fresh state per project.
|
||||||
|
|
||||||
|
Two sequential runs share no mutable state beyond the explicitly shared meter:
|
||||||
|
each project composes its OWN context, so the two projects' distinctive markers
|
||||||
|
never appear together in a single prompt.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_each_project_composes_its_own_context(self) -> None:
|
||||||
|
projects = _config([("BYGG-KONTOR-NORD", LED_BUNDLE), ("PUMPE-SOR", VFD_BUNDLE)])
|
||||||
|
client = ScriptedClient(replies=_happy_replies([LED_BUNDLE, VFD_BUNDLE]))
|
||||||
|
run_portfolio(projects, client, _meter(), top_k=3, max_debate_rounds=3, max_attempts=3)
|
||||||
|
|
||||||
|
proposer_prompts = client.prompts("proposer")
|
||||||
|
led = [p for p in proposer_prompts if LED_MARKER in p]
|
||||||
|
vfd = [p for p in proposer_prompts if VFD_MARKER in p]
|
||||||
|
assert led, "project 1's own context must reach its prompts"
|
||||||
|
assert vfd, "project 2's own context must reach its prompts (no reuse of project 1's)"
|
||||||
|
# No single prompt ever mixes the two projects' contexts.
|
||||||
|
assert not any(LED_MARKER in p and VFD_MARKER in p for p in proposer_prompts)
|
||||||
|
|
||||||
|
|
||||||
|
class TestFailurePolicyRaise:
|
||||||
|
"""Failure policy is a stack-local choice until D-D: the default RAISES (§8).
|
||||||
|
|
||||||
|
Today everything is thrown; K18 flips this to collect-and-continue when the
|
||||||
|
D-D wave model lands. A budget stop in project 1 propagates as the typed
|
||||||
|
``BudgetExceeded`` and the portfolio stops — project 2 is never touched.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_project_error_propagates_and_stops_the_portfolio(self) -> None:
|
||||||
|
projects = _config([("BYGG-KONTOR-NORD", LED_BUNDLE), ("PUMPE-SOR", VFD_BUNDLE)])
|
||||||
|
client = ScriptedClient(replies=_happy_replies([LED_BUNDLE, VFD_BUNDLE]))
|
||||||
|
# max_tokens=5: project 1's first proposer reply (10 tokens) breaches the cap.
|
||||||
|
meter = _meter(max_tokens=5)
|
||||||
|
with pytest.raises(BudgetExceeded):
|
||||||
|
run_portfolio(projects, client, meter, top_k=3, max_debate_rounds=3, max_attempts=3)
|
||||||
|
# The stop fired inside project 1's first call; project 2 never ran.
|
||||||
|
assert len(client.calls) == 1
|
||||||
|
assert not any(VFD_MARKER in prompt for _role, prompt in client.calls)
|
||||||
|
|
||||||
|
|
||||||
|
class TestFailFastSchemaValidation:
|
||||||
|
"""§10: a malformed config is refused at load, BEFORE any model client is touched."""
|
||||||
|
|
||||||
|
def test_malformed_config_never_reaches_a_client(self) -> None:
|
||||||
|
spy = ScriptedClient(replies=[reply("must never be used")])
|
||||||
|
with pytest.raises(ValidationError):
|
||||||
|
projects = load_reference_projects({"projects": [{"project_id": "NO-BUNDLE"}]})
|
||||||
|
run_portfolio(projects, spy, _meter(), top_k=3, max_debate_rounds=3, max_attempts=3)
|
||||||
|
assert spy.calls == []
|
||||||
Loading…
Add table
Add a link
Reference in a new issue