feat(portfolio): K8 — live-run drill, pre-call artifact capture (parity row 21) [skip-docs]

A future operator-gated live run (the M2-analog) is fully rigged and rehearsed
OFFLINE — without one model call, without a key (S4.2-analog, parity row 21;
buildable after K5 + K7). `--live-dry-run` builds everything a real run would
(contracts fail-fast §10 → compose §5 → SDK-client construction → preflight)
and captures the run-config + preflight artifacts, then STOPS before the first
model call. The stop IS the boundary: the loop is never entered, so nothing is
spent (strictly offline, no D6 gate).

- run.py --live-dry-run: requires --outbox + --run-id (the drill's artifacts are
  run_id-named), rejected fail-fast before any build. Writes a run_id-named PAIR
  to the outbox:
  * {run_id}-runconfig.json — comparison-protocol §4 pt 3: model-id per role the
    loop calls (proposer/checker, THROUGH resolve_model — the run's own path),
    profile, and every cap/parameter. Deliberately NO wall-clock date, so the
    bytes stay deterministic (the run's date is stamped at report time, §4 pt 3).
  * {run_id}-preflight.json — the captured preflight verdict (clear + refusals).
    The drill CAPTURES the preflight result rather than gating the build on it:
    exit 0 when clear (rig go-live-ready), non-zero when refused — artifacts
    captured and ZERO model calls in EITHER case.
- The client is constructed (the verified key-free SDK premise) but never called;
  a call-counting stand-in proves 0 calls. Bytes reuse the deterministic house
  JSON writer; run_s10.py/runs/ byte-untouched.

- test_dry_run_loadbearing.py: 7 tests. TWO seams detach-proven RED — the
  0-calls stop seam (neutralise the branch → falls to execute_run → the counting
  client fires → red) and the capture seam (drop the writes → outbox lacks the
  pair → red). Env monkeypatched so the preflight verdict is deterministic
  regardless of the operator's ambient shell.
- 514→521 green, golden byte-exact, full gate clean (ruff+format+mypy strict,
  24 src files). README: test-count sync ×2 + run.py drill note + load-bearing
  mention. IKKE-scope (held): the actual live run (M2-analog, operator) and any
  change to preflight/outbox.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RiTwaKLesgcwXx2mDviqpt
This commit is contained in:
Kjell Tore Guttormsen 2026-07-24 06:54:58 +02:00
commit 08ffddbbb1
3 changed files with 397 additions and 8 deletions

View file

@ -13,7 +13,7 @@ human-in-the-loop, and the system learns from the verdicts.
> **Status:** the D7 build (S5S10) is complete, and the deterministic **ingest layer**
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
> seam, each proven by load-bearing tests (514 tests, all running offline without an API
> seam, each proven by load-bearing tests (521 tests, all running offline without an API
> key). The programme's single budgeted **live model run has been executed and validated**
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
@ -100,7 +100,13 @@ description, never from its code)
the loop under the budget meter, persisting artifacts on both outcomes — a structured
budget stop included. The model client is injected, so the offline suite proves the
same orchestration with a scripted client; only the CLI's default constructs the SDK
client.
client. `--live-dry-run` is the **live-run drill** (K8): it builds everything a real run
would (contracts → compose → client construction → preflight) and captures a `run_id`-named
`runconfig` + `preflight` pair to the outbox (model-id, parameters, caps — no wall-clock, so
the bytes stay deterministic), then **stops before the first model call**. It exits 0 when
the preflight is clear and non-zero when it refused, but captures the artifacts and makes
zero model calls either way — a future operator-gated live run is rigged and rehearsed
offline, with no spend.
- `portfolio.py` — the sequential multi-project run and learning loop: `run_portfolio` drives
N projects from a schema-validated reference config, composing each project's context afresh
(re-entrant, fresh debate state per run) and collecting one result per project in config
@ -144,7 +150,10 @@ project k+1's fold via the shared store, with a marker-absent control),
`test_outbox_loadbearing.py` (a completed run's `run_id`-named outbox pair is written on the
entrance path, with a no-outbox control, and the outcome carries the inbox join key),
`test_preflight.py` (a missing credential and a placeholder model id are each refused before
any spend, and the preflight carries no network path of its own), and
any spend, and the preflight carries no network path of its own),
`test_dry_run_loadbearing.py` (the live-run drill captures its `runconfig` + `preflight`
artifacts and stops before the first model call — a call-counting client proves zero calls,
red the moment the stop seam is detached), and
`test_sdk_isolation.py` (local config cannot capture the checker).
## The ingest layer — CSV and SQL, in front of the loop
@ -207,7 +216,7 @@ Python ≥3.10 · [`claude-agent-sdk`](https://pypi.org/project/claude-agent-sdk
```bash
uv sync # install dependencies
uv run pytest # 514 tests — run without any API key and without network
uv run pytest # 521 tests — run without any API key and without network
uv run ruff check . && uv run ruff format --check .
uv run mypy src # strict
```

View file

@ -14,6 +14,9 @@ executed by the suite — honesty rule §1); the navigated docs dir comes from
the validated startup contract, never straight from the raw argument (§10).
Run: uv run python -m portfolio_optimiser_claude.run --bundle <dir> [--inbox <dir>]
# K8 live-run drill (builds all, captures artifacts, STOPS before the first call):
uv run python -m portfolio_optimiser_claude.run --bundle <dir> \\
--outbox <dir> --run-id <id> --live-dry-run
"""
from __future__ import annotations
@ -21,16 +24,18 @@ from __future__ import annotations
import argparse
from dataclasses import dataclass
from pathlib import Path
from typing import Callable
from typing import Any, Callable
from portfolio_optimiser_claude.artifacts import (
_dump_json,
build_citations,
persist_run_artifacts,
persist_stop_artifacts,
)
from portfolio_optimiser_claude.outbox import persist_outbox
from portfolio_optimiser_claude.budget import BudgetExceeded, BudgetMeter
from portfolio_optimiser_claude.contracts import Contracts, load_contracts
from portfolio_optimiser_claude.contracts import Contracts, load_contracts, resolve_model
from portfolio_optimiser_claude.preflight import Refusal, run_preflight
from portfolio_optimiser_claude.experience import (
CandidateFeatures,
VerdictStore,
@ -45,6 +50,9 @@ from portfolio_optimiser_claude.loop import ModelClient, run_project
from portfolio_optimiser_claude.validator import Rejection
_PROPOSER_ROLE = "proposer"
_CHECKER_ROLE = "checker"
# The one backend profile the run resolves against (mirrors SdkModelClient's default).
_DEFAULT_PROFILE = "anthropic"
# The injected client seam of the entrance: (contracts, max_budget_usd_per_call).
ClientFactory = Callable[[Contracts, float], ModelClient]
@ -195,6 +203,97 @@ def execute_run(
return 0
def build_dry_run_config(
contracts: Contracts,
*,
profile: str,
bundle_name: str,
run_id: str,
max_rounds: int,
max_tokens: int,
max_budget_usd_per_call: float,
max_debate_rounds: int,
max_attempts: int,
top_k: int,
) -> dict[str, Any]:
"""The run-config log (comparison protocol §4 pt 3): model-id, parameters, caps.
Records the model id each role the loop calls resolves to THROUGH
``resolve_model`` (the run's own resolution path), never a raw dict read — the
profile, and every cap/parameter a live run would carry. Deliberately carries
NO wall-clock date: the outbox promises byte-determinism (same input + run_id
identical file), and the run's date is stamped at report time (§4 pt 3),
never into the deterministic log.
"""
return {
"run_id": run_id,
"profile": profile,
"bundle": bundle_name,
"models": {
role: resolve_model(contracts.model_map, role, profile=profile)
for role in (_PROPOSER_ROLE, _CHECKER_ROLE)
},
"caps": {
"max_rounds": max_rounds,
"max_tokens": max_tokens,
"max_budget_usd_per_call": max_budget_usd_per_call,
"max_debate_rounds": max_debate_rounds,
"max_attempts": max_attempts,
"top_k": top_k,
},
}
def execute_dry_run(
*,
outbox_dir: Path,
run_id: str,
profile: str,
run_config: dict[str, Any],
refusals: list[Refusal],
) -> int:
"""Capture the run-config + preflight artifacts; STOP before any model call (K8).
Writes the run_id-named PAIR ``{run_id}-runconfig.json`` and
``{run_id}-preflight.json`` to the outbox as deterministic house JSON, then
returns WITHOUT ever driving the loop: the drill rehearses the whole build and
artifact capture offline, so a future operator-gated live run (the M2-analog)
is fully rigged. Exit 0 when the preflight is clear (rig go-live-ready); exit 1
when it refused the artifacts are captured EITHER way (the refusal is itself
one of them), and no model call is made in either case.
"""
outbox_dir.mkdir(parents=True, exist_ok=True)
paths = {
"runconfig": outbox_dir / f"{run_id}-runconfig.json",
"preflight": outbox_dir / f"{run_id}-preflight.json",
}
_dump_json(paths["runconfig"], run_config)
_dump_json(
paths["preflight"],
{
"run_id": run_id,
"profile": profile,
"clear": not refusals,
"refusals": [{"check": r.check, "detail": r.detail} for r in refusals],
},
)
for name, path in sorted(paths.items()):
print(f"artifact: {name} -> {path}")
if refusals:
print(
f"DRILL: preflight REFUSED ({len(refusals)}) — rig NOT clear to go live "
"(artifacts captured, no model call was made):"
)
for refusal in refusals:
print(f" [{refusal.check}] {refusal.detail}")
return 1
print(
"DRILL OK — built all, captured artifacts, stopped before the first model call "
"(0 model calls); rig clear to go live."
)
return 0
def main(argv: list[str] | None = None, *, client_factory: ClientFactory | None = None) -> int:
"""The thin CLI: contracts fail-fast (§10) → compose (§5) → execute (§3, §8)."""
parser = argparse.ArgumentParser(
@ -211,12 +310,25 @@ def main(argv: list[str] | None = None, *, client_factory: ClientFactory | None
parser.add_argument("--max-debate-rounds", type=int, default=3)
parser.add_argument("--max-attempts", type=int, default=3)
parser.add_argument("--top-k", type=int, default=3)
parser.add_argument(
"--live-dry-run",
action="store_true",
help="build all, capture run-config + preflight to the outbox, STOP before "
"the first model call (K8 live-run drill; no spend, no model call).",
)
args = parser.parse_args(argv)
# fail-fast (§10 spirit): a run persisted to the outbox MUST carry an explicit
# run_id — reject BEFORE composing or constructing a client, so no spend rides
# on a run that cannot be filed (no wall-clock default fills the gap).
if args.outbox is not None and not (args.run_id or "").strip():
# on a run that cannot be filed (no wall-clock default fills the gap). The dry
# run's artifacts are ALSO run_id-named, so it requires both an outbox and an id.
if args.live_dry_run:
if args.outbox is None or not (args.run_id or "").strip():
parser.error(
"--live-dry-run requires --outbox and --run-id "
"(the drill's artifacts are run_id-named in the outbox)"
)
elif args.outbox is not None and not (args.run_id or "").strip():
parser.error("--outbox requires --run-id (no wall-clock default)")
# §10: ALL startup contracts schema-validated BEFORE any model client exists.
@ -237,6 +349,40 @@ def main(argv: list[str] | None = None, *, client_factory: ClientFactory | None
)
factory = default_client_factory if client_factory is None else client_factory
client = factory(contracts, args.max_budget_usd_per_call)
# K8: the live-run drill builds the client (the key-free SDK construction
# premise) but never calls it — it captures the run-config + preflight
# artifacts and STOPS before the first model call. A future operator-gated
# live run is thus rigged and rehearsed offline, with zero spend.
if args.live_dry_run:
assert args.outbox is not None # narrowed by the fail-fast above
run_id = args.run_id or ""
refusals = run_preflight(
profile=_DEFAULT_PROFILE,
max_rounds=args.max_rounds,
max_tokens=args.max_tokens,
max_budget_usd_per_call=args.max_budget_usd_per_call,
)
run_config = build_dry_run_config(
contracts,
profile=_DEFAULT_PROFILE,
bundle_name=args.bundle.name,
run_id=run_id,
max_rounds=args.max_rounds,
max_tokens=args.max_tokens,
max_budget_usd_per_call=args.max_budget_usd_per_call,
max_debate_rounds=args.max_debate_rounds,
max_attempts=args.max_attempts,
top_k=args.top_k,
)
return execute_dry_run(
outbox_dir=args.outbox,
run_id=run_id,
profile=_DEFAULT_PROFILE,
run_config=run_config,
refusals=refusals,
)
return execute_run(
client,
composed,

View file

@ -0,0 +1,234 @@
"""Live-run drill — LOAD-BEARING (K8; method-spec §8; comparison protocol §4 pt 3).
The seam this file keeps alive: ``--live-dry-run`` BUILDS everything a real
live run would (contracts fail-fast compose client construction preflight)
and CAPTURES the run-config + preflight artifacts to the outbox, then STOPS
before the first model call. A future operator-gated live run (the M2-analog) is
thus fully rigged and rehearsed offline without one model call, without a key.
Detach proof (the 0-calls seam): remove the dry-run branch from ``main`` so it
falls through to ``execute_run`` the injected call-counting client's
``complete`` fires ``calls`` is non-empty (and the empty-reply stand-in raises)
red. Detach proof (the capture seam): drop the artifact write the outbox
lacks the run_id-named pair red.
No credential and no network are needed: the drill constructs the client (the
verified key-free SDK premise) and the call-counting stand-in guarantees the
boundary. The env is monkeypatched so the preflight verdict is deterministic
regardless of the operator's ambient shell.
"""
from __future__ import annotations
import json
from pathlib import Path
from typing import Callable
import pytest
from _scripted import ScriptedClient
from portfolio_optimiser_claude.contracts import Contracts, load_contracts
from portfolio_optimiser_claude.loop import ModelClient
from portfolio_optimiser_claude.run import main
BUNDLE = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
RUN_ID = "dryrun-001"
ClientFactory = Callable[[Contracts, float], ModelClient]
def _counting_factory() -> tuple[ClientFactory, list[ScriptedClient]]:
"""A factory whose clients record every call and carry NO replies.
An empty reply list means any ``complete`` both records the call and raises
so a detached dry-run (one that reaches the loop) fails loudly, and a correct
dry-run leaves ``calls`` empty.
"""
created: list[ScriptedClient] = []
def factory(contracts: Contracts, max_budget_usd_per_call: float) -> ModelClient:
client = ScriptedClient(replies=[])
created.append(client)
return client
return factory, created
def _clear_credentials(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
monkeypatch.delenv("CLAUDE_CODE_OAUTH_TOKEN", raising=False)
def _set_credential(monkeypatch: pytest.MonkeyPatch) -> None:
# A non-placeholder form; the preflight never validates it online, so this is
# not a real key and never leaves the process (the counting client blocks any
# call). It only exercises the clear-preflight branch.
_clear_credentials(monkeypatch)
monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-drill-not-a-real-key")
class TestDryRunStopsBeforeFirstCall:
"""The boundary: the drill builds everything but never calls the model."""
def test_zero_model_calls(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
_clear_credentials(monkeypatch)
factory, created = _counting_factory()
main(
[
"--bundle",
str(BUNDLE),
"--outbox",
str(tmp_path / "outbox"),
"--run-id",
RUN_ID,
"--live-dry-run",
],
client_factory=factory,
)
# The client was constructed (the drill builds the client), but never called.
(client,) = created
assert client.calls == []
def test_requires_outbox_and_run_id(self, tmp_path: Path) -> None:
factory, _ = _counting_factory()
# No --outbox / --run-id: the run_id-named artifacts have nowhere to go.
with pytest.raises(SystemExit):
main(
["--bundle", str(BUNDLE), "--live-dry-run"],
client_factory=factory,
)
class TestDryRunArtifactCapture:
"""The captured set (run-config + preflight) is complete and deterministic."""
def test_captures_runconfig_and_preflight(
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
_clear_credentials(monkeypatch)
outbox = tmp_path / "outbox"
factory, _ = _counting_factory()
main(
[
"--bundle",
str(BUNDLE),
"--outbox",
str(outbox),
"--run-id",
RUN_ID,
"--live-dry-run",
],
client_factory=factory,
)
runconfig = json.loads((outbox / f"{RUN_ID}-runconfig.json").read_text("utf-8"))
preflight = json.loads((outbox / f"{RUN_ID}-preflight.json").read_text("utf-8"))
# §4 pt 3: model-id per role the loop calls, parameters, caps — no wall-clock.
assert runconfig["run_id"] == RUN_ID
assert runconfig["profile"] == "anthropic"
assert runconfig["models"]["proposer"] == "claude-haiku-4-5-20251001"
assert runconfig["models"]["checker"] == "claude-haiku-4-5-20251001"
assert runconfig["caps"]["max_rounds"] == 12
assert runconfig["caps"]["max_tokens"] == 150_000
assert runconfig["caps"]["max_budget_usd_per_call"] == 0.25
assert "date" not in runconfig # determinism: date is stamped at report time
# Preflight result captured (no credential here → credential refusal recorded).
assert preflight["run_id"] == RUN_ID
assert preflight["clear"] is False
assert any(r["check"] == "credential" for r in preflight["refusals"])
def test_artifacts_are_byte_deterministic(
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
_clear_credentials(monkeypatch)
first = tmp_path / "a"
second = tmp_path / "b"
for outbox in (first, second):
factory, _ = _counting_factory()
main(
[
"--bundle",
str(BUNDLE),
"--outbox",
str(outbox),
"--run-id",
RUN_ID,
"--live-dry-run",
],
client_factory=factory,
)
for name in (f"{RUN_ID}-runconfig.json", f"{RUN_ID}-preflight.json"):
assert (first / name).read_bytes() == (second / name).read_bytes()
class TestDryRunPreflightGate:
"""Exit code reflects go-live readiness; capture happens either way."""
def test_clear_preflight_exits_zero(
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
_set_credential(monkeypatch)
outbox = tmp_path / "outbox"
factory, created = _counting_factory()
code = main(
[
"--bundle",
str(BUNDLE),
"--outbox",
str(outbox),
"--run-id",
RUN_ID,
"--live-dry-run",
],
client_factory=factory,
)
assert code == 0
preflight = json.loads((outbox / f"{RUN_ID}-preflight.json").read_text("utf-8"))
assert preflight["clear"] is True
assert preflight["refusals"] == []
(client,) = created
assert client.calls == [] # still zero calls
def test_refused_preflight_captures_but_exits_nonzero(
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
_clear_credentials(monkeypatch)
outbox = tmp_path / "outbox"
factory, created = _counting_factory()
code = main(
[
"--bundle",
str(BUNDLE),
"--outbox",
str(outbox),
"--run-id",
RUN_ID,
"--live-dry-run",
],
client_factory=factory,
)
assert code != 0 # refused: the rig is not clear to go live
# ...yet the artifacts are captured and no model call was made.
assert (outbox / f"{RUN_ID}-runconfig.json").is_file()
assert (outbox / f"{RUN_ID}-preflight.json").is_file()
(client,) = created
assert client.calls == []
class TestKeyFreeConstruction:
"""K8 key premise: the SDK client constructs with no credential (no call)."""
def test_default_factory_constructs_without_credential(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
_clear_credentials(monkeypatch)
from portfolio_optimiser_claude.run import default_client_factory
from portfolio_optimiser_claude.sdk_client import SdkModelClient
contracts = load_contracts(
data_source={"docs_dir": str(BUNDLE), "top_k": 3},
termination={"max_rounds": 1, "max_tokens": 1},
feedback={"decision": "approved", "rationale": "startup shape check (§10)"},
)
client = default_client_factory(contracts, 0.25)
assert isinstance(client, SdkModelClient)