feat(portfolio): K8 — live-run drill, pre-call artifact capture (parity row 21) [skip-docs]
A future operator-gated live run (the M2-analog) is fully rigged and rehearsed
OFFLINE — without one model call, without a key (S4.2-analog, parity row 21;
buildable after K5 + K7). `--live-dry-run` builds everything a real run would
(contracts fail-fast §10 → compose §5 → SDK-client construction → preflight)
and captures the run-config + preflight artifacts, then STOPS before the first
model call. The stop IS the boundary: the loop is never entered, so nothing is
spent (strictly offline, no D6 gate).
- run.py --live-dry-run: requires --outbox + --run-id (the drill's artifacts are
run_id-named), rejected fail-fast before any build. Writes a run_id-named PAIR
to the outbox:
* {run_id}-runconfig.json — comparison-protocol §4 pt 3: model-id per role the
loop calls (proposer/checker, THROUGH resolve_model — the run's own path),
profile, and every cap/parameter. Deliberately NO wall-clock date, so the
bytes stay deterministic (the run's date is stamped at report time, §4 pt 3).
* {run_id}-preflight.json — the captured preflight verdict (clear + refusals).
The drill CAPTURES the preflight result rather than gating the build on it:
exit 0 when clear (rig go-live-ready), non-zero when refused — artifacts
captured and ZERO model calls in EITHER case.
- The client is constructed (the verified key-free SDK premise) but never called;
a call-counting stand-in proves 0 calls. Bytes reuse the deterministic house
JSON writer; run_s10.py/runs/ byte-untouched.
- test_dry_run_loadbearing.py: 7 tests. TWO seams detach-proven RED — the
0-calls stop seam (neutralise the branch → falls to execute_run → the counting
client fires → red) and the capture seam (drop the writes → outbox lacks the
pair → red). Env monkeypatched so the preflight verdict is deterministic
regardless of the operator's ambient shell.
- 514→521 green, golden byte-exact, full gate clean (ruff+format+mypy strict,
24 src files). README: test-count sync ×2 + run.py drill note + load-bearing
mention. IKKE-scope (held): the actual live run (M2-analog, operator) and any
change to preflight/outbox.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RiTwaKLesgcwXx2mDviqpt
This commit is contained in:
parent
c08d92a358
commit
08ffddbbb1
3 changed files with 397 additions and 8 deletions
17
README.md
17
README.md
|
|
@ -13,7 +13,7 @@ human-in-the-loop, and the system learns from the verdicts.
|
||||||
> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
|
> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
|
||||||
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
|
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
|
||||||
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
|
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
|
||||||
> seam, each proven by load-bearing tests (514 tests, all running offline without an API
|
> seam, each proven by load-bearing tests (521 tests, all running offline without an API
|
||||||
> key). The programme's single budgeted **live model run has been executed and validated** —
|
> key). The programme's single budgeted **live model run has been executed and validated** —
|
||||||
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
|
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
|
||||||
|
|
||||||
|
|
@ -100,7 +100,13 @@ description, never from its code)
|
||||||
the loop under the budget meter, persisting artifacts on both outcomes — a structured
|
the loop under the budget meter, persisting artifacts on both outcomes — a structured
|
||||||
budget stop included. The model client is injected, so the offline suite proves the
|
budget stop included. The model client is injected, so the offline suite proves the
|
||||||
same orchestration with a scripted client; only the CLI's default constructs the SDK
|
same orchestration with a scripted client; only the CLI's default constructs the SDK
|
||||||
client.
|
client. `--live-dry-run` is the **live-run drill** (K8): it builds everything a real run
|
||||||
|
would (contracts → compose → client construction → preflight) and captures a `run_id`-named
|
||||||
|
`runconfig` + `preflight` pair to the outbox (model-id, parameters, caps — no wall-clock, so
|
||||||
|
the bytes stay deterministic), then **stops before the first model call**. It exits 0 when
|
||||||
|
the preflight is clear and non-zero when it refused, but captures the artifacts and makes
|
||||||
|
zero model calls either way — a future operator-gated live run is rigged and rehearsed
|
||||||
|
offline, with no spend.
|
||||||
- `portfolio.py` — the sequential multi-project run and learning loop: `run_portfolio` drives
|
- `portfolio.py` — the sequential multi-project run and learning loop: `run_portfolio` drives
|
||||||
N projects from a schema-validated reference config, composing each project's context afresh
|
N projects from a schema-validated reference config, composing each project's context afresh
|
||||||
(re-entrant, fresh debate state per run) and collecting one result per project in config
|
(re-entrant, fresh debate state per run) and collecting one result per project in config
|
||||||
|
|
@ -144,7 +150,10 @@ project k+1's fold via the shared store, with a marker-absent control),
|
||||||
`test_outbox_loadbearing.py` (a completed run's `run_id`-named outbox pair is written on the
|
`test_outbox_loadbearing.py` (a completed run's `run_id`-named outbox pair is written on the
|
||||||
entrance path, with a no-outbox control, and the outcome carries the inbox join key),
|
entrance path, with a no-outbox control, and the outcome carries the inbox join key),
|
||||||
`test_preflight.py` (a missing credential and a placeholder model id are each refused before
|
`test_preflight.py` (a missing credential and a placeholder model id are each refused before
|
||||||
any spend, and the preflight carries no network path of its own), and
|
any spend, and the preflight carries no network path of its own),
|
||||||
|
`test_dry_run_loadbearing.py` (the live-run drill captures its `runconfig` + `preflight`
|
||||||
|
artifacts and stops before the first model call — a call-counting client proves zero calls,
|
||||||
|
red the moment the stop seam is detached), and
|
||||||
`test_sdk_isolation.py` (local config cannot capture the checker).
|
`test_sdk_isolation.py` (local config cannot capture the checker).
|
||||||
|
|
||||||
## The ingest layer — CSV and SQL, in front of the loop
|
## The ingest layer — CSV and SQL, in front of the loop
|
||||||
|
|
@ -207,7 +216,7 @@ Python ≥3.10 · [`claude-agent-sdk`](https://pypi.org/project/claude-agent-sdk
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv sync # install dependencies
|
uv sync # install dependencies
|
||||||
uv run pytest # 514 tests — run without any API key and without network
|
uv run pytest # 521 tests — run without any API key and without network
|
||||||
uv run ruff check . && uv run ruff format --check .
|
uv run ruff check . && uv run ruff format --check .
|
||||||
uv run mypy src # strict
|
uv run mypy src # strict
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -14,6 +14,9 @@ executed by the suite — honesty rule §1); the navigated docs dir comes from
|
||||||
the validated startup contract, never straight from the raw argument (§10).
|
the validated startup contract, never straight from the raw argument (§10).
|
||||||
|
|
||||||
Run: uv run python -m portfolio_optimiser_claude.run --bundle <dir> [--inbox <dir>]
|
Run: uv run python -m portfolio_optimiser_claude.run --bundle <dir> [--inbox <dir>]
|
||||||
|
# K8 live-run drill (builds all, captures artifacts, STOPS before the first call):
|
||||||
|
uv run python -m portfolio_optimiser_claude.run --bundle <dir> \\
|
||||||
|
--outbox <dir> --run-id <id> --live-dry-run
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
@ -21,16 +24,18 @@ from __future__ import annotations
|
||||||
import argparse
|
import argparse
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Callable
|
from typing import Any, Callable
|
||||||
|
|
||||||
from portfolio_optimiser_claude.artifacts import (
|
from portfolio_optimiser_claude.artifacts import (
|
||||||
|
_dump_json,
|
||||||
build_citations,
|
build_citations,
|
||||||
persist_run_artifacts,
|
persist_run_artifacts,
|
||||||
persist_stop_artifacts,
|
persist_stop_artifacts,
|
||||||
)
|
)
|
||||||
from portfolio_optimiser_claude.outbox import persist_outbox
|
from portfolio_optimiser_claude.outbox import persist_outbox
|
||||||
from portfolio_optimiser_claude.budget import BudgetExceeded, BudgetMeter
|
from portfolio_optimiser_claude.budget import BudgetExceeded, BudgetMeter
|
||||||
from portfolio_optimiser_claude.contracts import Contracts, load_contracts
|
from portfolio_optimiser_claude.contracts import Contracts, load_contracts, resolve_model
|
||||||
|
from portfolio_optimiser_claude.preflight import Refusal, run_preflight
|
||||||
from portfolio_optimiser_claude.experience import (
|
from portfolio_optimiser_claude.experience import (
|
||||||
CandidateFeatures,
|
CandidateFeatures,
|
||||||
VerdictStore,
|
VerdictStore,
|
||||||
|
|
@ -45,6 +50,9 @@ from portfolio_optimiser_claude.loop import ModelClient, run_project
|
||||||
from portfolio_optimiser_claude.validator import Rejection
|
from portfolio_optimiser_claude.validator import Rejection
|
||||||
|
|
||||||
_PROPOSER_ROLE = "proposer"
|
_PROPOSER_ROLE = "proposer"
|
||||||
|
_CHECKER_ROLE = "checker"
|
||||||
|
# The one backend profile the run resolves against (mirrors SdkModelClient's default).
|
||||||
|
_DEFAULT_PROFILE = "anthropic"
|
||||||
|
|
||||||
# The injected client seam of the entrance: (contracts, max_budget_usd_per_call).
|
# The injected client seam of the entrance: (contracts, max_budget_usd_per_call).
|
||||||
ClientFactory = Callable[[Contracts, float], ModelClient]
|
ClientFactory = Callable[[Contracts, float], ModelClient]
|
||||||
|
|
@ -195,6 +203,97 @@ def execute_run(
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def build_dry_run_config(
|
||||||
|
contracts: Contracts,
|
||||||
|
*,
|
||||||
|
profile: str,
|
||||||
|
bundle_name: str,
|
||||||
|
run_id: str,
|
||||||
|
max_rounds: int,
|
||||||
|
max_tokens: int,
|
||||||
|
max_budget_usd_per_call: float,
|
||||||
|
max_debate_rounds: int,
|
||||||
|
max_attempts: int,
|
||||||
|
top_k: int,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""The run-config log (comparison protocol §4 pt 3): model-id, parameters, caps.
|
||||||
|
|
||||||
|
Records the model id each role the loop calls resolves to — THROUGH
|
||||||
|
``resolve_model`` (the run's own resolution path), never a raw dict read — the
|
||||||
|
profile, and every cap/parameter a live run would carry. Deliberately carries
|
||||||
|
NO wall-clock date: the outbox promises byte-determinism (same input + run_id
|
||||||
|
⇒ identical file), and the run's date is stamped at report time (§4 pt 3),
|
||||||
|
never into the deterministic log.
|
||||||
|
"""
|
||||||
|
return {
|
||||||
|
"run_id": run_id,
|
||||||
|
"profile": profile,
|
||||||
|
"bundle": bundle_name,
|
||||||
|
"models": {
|
||||||
|
role: resolve_model(contracts.model_map, role, profile=profile)
|
||||||
|
for role in (_PROPOSER_ROLE, _CHECKER_ROLE)
|
||||||
|
},
|
||||||
|
"caps": {
|
||||||
|
"max_rounds": max_rounds,
|
||||||
|
"max_tokens": max_tokens,
|
||||||
|
"max_budget_usd_per_call": max_budget_usd_per_call,
|
||||||
|
"max_debate_rounds": max_debate_rounds,
|
||||||
|
"max_attempts": max_attempts,
|
||||||
|
"top_k": top_k,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def execute_dry_run(
|
||||||
|
*,
|
||||||
|
outbox_dir: Path,
|
||||||
|
run_id: str,
|
||||||
|
profile: str,
|
||||||
|
run_config: dict[str, Any],
|
||||||
|
refusals: list[Refusal],
|
||||||
|
) -> int:
|
||||||
|
"""Capture the run-config + preflight artifacts; STOP before any model call (K8).
|
||||||
|
|
||||||
|
Writes the run_id-named PAIR — ``{run_id}-runconfig.json`` and
|
||||||
|
``{run_id}-preflight.json`` — to the outbox as deterministic house JSON, then
|
||||||
|
returns WITHOUT ever driving the loop: the drill rehearses the whole build and
|
||||||
|
artifact capture offline, so a future operator-gated live run (the M2-analog)
|
||||||
|
is fully rigged. Exit 0 when the preflight is clear (rig go-live-ready); exit 1
|
||||||
|
when it refused — the artifacts are captured EITHER way (the refusal is itself
|
||||||
|
one of them), and no model call is made in either case.
|
||||||
|
"""
|
||||||
|
outbox_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
paths = {
|
||||||
|
"runconfig": outbox_dir / f"{run_id}-runconfig.json",
|
||||||
|
"preflight": outbox_dir / f"{run_id}-preflight.json",
|
||||||
|
}
|
||||||
|
_dump_json(paths["runconfig"], run_config)
|
||||||
|
_dump_json(
|
||||||
|
paths["preflight"],
|
||||||
|
{
|
||||||
|
"run_id": run_id,
|
||||||
|
"profile": profile,
|
||||||
|
"clear": not refusals,
|
||||||
|
"refusals": [{"check": r.check, "detail": r.detail} for r in refusals],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
for name, path in sorted(paths.items()):
|
||||||
|
print(f"artifact: {name} -> {path}")
|
||||||
|
if refusals:
|
||||||
|
print(
|
||||||
|
f"DRILL: preflight REFUSED ({len(refusals)}) — rig NOT clear to go live "
|
||||||
|
"(artifacts captured, no model call was made):"
|
||||||
|
)
|
||||||
|
for refusal in refusals:
|
||||||
|
print(f" [{refusal.check}] {refusal.detail}")
|
||||||
|
return 1
|
||||||
|
print(
|
||||||
|
"DRILL OK — built all, captured artifacts, stopped before the first model call "
|
||||||
|
"(0 model calls); rig clear to go live."
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def main(argv: list[str] | None = None, *, client_factory: ClientFactory | None = None) -> int:
|
def main(argv: list[str] | None = None, *, client_factory: ClientFactory | None = None) -> int:
|
||||||
"""The thin CLI: contracts fail-fast (§10) → compose (§5) → execute (§3, §8)."""
|
"""The thin CLI: contracts fail-fast (§10) → compose (§5) → execute (§3, §8)."""
|
||||||
parser = argparse.ArgumentParser(
|
parser = argparse.ArgumentParser(
|
||||||
|
|
@ -211,12 +310,25 @@ def main(argv: list[str] | None = None, *, client_factory: ClientFactory | None
|
||||||
parser.add_argument("--max-debate-rounds", type=int, default=3)
|
parser.add_argument("--max-debate-rounds", type=int, default=3)
|
||||||
parser.add_argument("--max-attempts", type=int, default=3)
|
parser.add_argument("--max-attempts", type=int, default=3)
|
||||||
parser.add_argument("--top-k", type=int, default=3)
|
parser.add_argument("--top-k", type=int, default=3)
|
||||||
|
parser.add_argument(
|
||||||
|
"--live-dry-run",
|
||||||
|
action="store_true",
|
||||||
|
help="build all, capture run-config + preflight to the outbox, STOP before "
|
||||||
|
"the first model call (K8 live-run drill; no spend, no model call).",
|
||||||
|
)
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
# fail-fast (§10 spirit): a run persisted to the outbox MUST carry an explicit
|
# fail-fast (§10 spirit): a run persisted to the outbox MUST carry an explicit
|
||||||
# run_id — reject BEFORE composing or constructing a client, so no spend rides
|
# run_id — reject BEFORE composing or constructing a client, so no spend rides
|
||||||
# on a run that cannot be filed (no wall-clock default fills the gap).
|
# on a run that cannot be filed (no wall-clock default fills the gap). The dry
|
||||||
if args.outbox is not None and not (args.run_id or "").strip():
|
# run's artifacts are ALSO run_id-named, so it requires both an outbox and an id.
|
||||||
|
if args.live_dry_run:
|
||||||
|
if args.outbox is None or not (args.run_id or "").strip():
|
||||||
|
parser.error(
|
||||||
|
"--live-dry-run requires --outbox and --run-id "
|
||||||
|
"(the drill's artifacts are run_id-named in the outbox)"
|
||||||
|
)
|
||||||
|
elif args.outbox is not None and not (args.run_id or "").strip():
|
||||||
parser.error("--outbox requires --run-id (no wall-clock default)")
|
parser.error("--outbox requires --run-id (no wall-clock default)")
|
||||||
|
|
||||||
# §10: ALL startup contracts schema-validated BEFORE any model client exists.
|
# §10: ALL startup contracts schema-validated BEFORE any model client exists.
|
||||||
|
|
@ -237,6 +349,40 @@ def main(argv: list[str] | None = None, *, client_factory: ClientFactory | None
|
||||||
)
|
)
|
||||||
factory = default_client_factory if client_factory is None else client_factory
|
factory = default_client_factory if client_factory is None else client_factory
|
||||||
client = factory(contracts, args.max_budget_usd_per_call)
|
client = factory(contracts, args.max_budget_usd_per_call)
|
||||||
|
|
||||||
|
# K8: the live-run drill builds the client (the key-free SDK construction
|
||||||
|
# premise) but never calls it — it captures the run-config + preflight
|
||||||
|
# artifacts and STOPS before the first model call. A future operator-gated
|
||||||
|
# live run is thus rigged and rehearsed offline, with zero spend.
|
||||||
|
if args.live_dry_run:
|
||||||
|
assert args.outbox is not None # narrowed by the fail-fast above
|
||||||
|
run_id = args.run_id or ""
|
||||||
|
refusals = run_preflight(
|
||||||
|
profile=_DEFAULT_PROFILE,
|
||||||
|
max_rounds=args.max_rounds,
|
||||||
|
max_tokens=args.max_tokens,
|
||||||
|
max_budget_usd_per_call=args.max_budget_usd_per_call,
|
||||||
|
)
|
||||||
|
run_config = build_dry_run_config(
|
||||||
|
contracts,
|
||||||
|
profile=_DEFAULT_PROFILE,
|
||||||
|
bundle_name=args.bundle.name,
|
||||||
|
run_id=run_id,
|
||||||
|
max_rounds=args.max_rounds,
|
||||||
|
max_tokens=args.max_tokens,
|
||||||
|
max_budget_usd_per_call=args.max_budget_usd_per_call,
|
||||||
|
max_debate_rounds=args.max_debate_rounds,
|
||||||
|
max_attempts=args.max_attempts,
|
||||||
|
top_k=args.top_k,
|
||||||
|
)
|
||||||
|
return execute_dry_run(
|
||||||
|
outbox_dir=args.outbox,
|
||||||
|
run_id=run_id,
|
||||||
|
profile=_DEFAULT_PROFILE,
|
||||||
|
run_config=run_config,
|
||||||
|
refusals=refusals,
|
||||||
|
)
|
||||||
|
|
||||||
return execute_run(
|
return execute_run(
|
||||||
client,
|
client,
|
||||||
composed,
|
composed,
|
||||||
|
|
|
||||||
234
tests/test_dry_run_loadbearing.py
Normal file
234
tests/test_dry_run_loadbearing.py
Normal file
|
|
@ -0,0 +1,234 @@
|
||||||
|
"""Live-run drill — LOAD-BEARING (K8; method-spec §8; comparison protocol §4 pt 3).
|
||||||
|
|
||||||
|
The seam this file keeps alive: ``--live-dry-run`` BUILDS everything a real
|
||||||
|
live run would (contracts fail-fast → compose → client construction → preflight)
|
||||||
|
and CAPTURES the run-config + preflight artifacts to the outbox, then STOPS
|
||||||
|
before the first model call. A future operator-gated live run (the M2-analog) is
|
||||||
|
thus fully rigged and rehearsed offline — without one model call, without a key.
|
||||||
|
|
||||||
|
Detach proof (the 0-calls seam): remove the dry-run branch from ``main`` so it
|
||||||
|
falls through to ``execute_run`` → the injected call-counting client's
|
||||||
|
``complete`` fires → ``calls`` is non-empty (and the empty-reply stand-in raises)
|
||||||
|
→ red. Detach proof (the capture seam): drop the artifact write → the outbox
|
||||||
|
lacks the run_id-named pair → red.
|
||||||
|
|
||||||
|
No credential and no network are needed: the drill constructs the client (the
|
||||||
|
verified key-free SDK premise) and the call-counting stand-in guarantees the
|
||||||
|
boundary. The env is monkeypatched so the preflight verdict is deterministic
|
||||||
|
regardless of the operator's ambient shell.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Callable
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from _scripted import ScriptedClient
|
||||||
|
|
||||||
|
from portfolio_optimiser_claude.contracts import Contracts, load_contracts
|
||||||
|
from portfolio_optimiser_claude.loop import ModelClient
|
||||||
|
from portfolio_optimiser_claude.run import main
|
||||||
|
|
||||||
|
BUNDLE = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||||
|
RUN_ID = "dryrun-001"
|
||||||
|
|
||||||
|
ClientFactory = Callable[[Contracts, float], ModelClient]
|
||||||
|
|
||||||
|
|
||||||
|
def _counting_factory() -> tuple[ClientFactory, list[ScriptedClient]]:
|
||||||
|
"""A factory whose clients record every call and carry NO replies.
|
||||||
|
|
||||||
|
An empty reply list means any ``complete`` both records the call and raises —
|
||||||
|
so a detached dry-run (one that reaches the loop) fails loudly, and a correct
|
||||||
|
dry-run leaves ``calls`` empty.
|
||||||
|
"""
|
||||||
|
created: list[ScriptedClient] = []
|
||||||
|
|
||||||
|
def factory(contracts: Contracts, max_budget_usd_per_call: float) -> ModelClient:
|
||||||
|
client = ScriptedClient(replies=[])
|
||||||
|
created.append(client)
|
||||||
|
return client
|
||||||
|
|
||||||
|
return factory, created
|
||||||
|
|
||||||
|
|
||||||
|
def _clear_credentials(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||||||
|
monkeypatch.delenv("CLAUDE_CODE_OAUTH_TOKEN", raising=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _set_credential(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
# A non-placeholder form; the preflight never validates it online, so this is
|
||||||
|
# not a real key and never leaves the process (the counting client blocks any
|
||||||
|
# call). It only exercises the clear-preflight branch.
|
||||||
|
_clear_credentials(monkeypatch)
|
||||||
|
monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-drill-not-a-real-key")
|
||||||
|
|
||||||
|
|
||||||
|
class TestDryRunStopsBeforeFirstCall:
|
||||||
|
"""The boundary: the drill builds everything but never calls the model."""
|
||||||
|
|
||||||
|
def test_zero_model_calls(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
_clear_credentials(monkeypatch)
|
||||||
|
factory, created = _counting_factory()
|
||||||
|
main(
|
||||||
|
[
|
||||||
|
"--bundle",
|
||||||
|
str(BUNDLE),
|
||||||
|
"--outbox",
|
||||||
|
str(tmp_path / "outbox"),
|
||||||
|
"--run-id",
|
||||||
|
RUN_ID,
|
||||||
|
"--live-dry-run",
|
||||||
|
],
|
||||||
|
client_factory=factory,
|
||||||
|
)
|
||||||
|
# The client was constructed (the drill builds the client), but never called.
|
||||||
|
(client,) = created
|
||||||
|
assert client.calls == []
|
||||||
|
|
||||||
|
def test_requires_outbox_and_run_id(self, tmp_path: Path) -> None:
|
||||||
|
factory, _ = _counting_factory()
|
||||||
|
# No --outbox / --run-id: the run_id-named artifacts have nowhere to go.
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
main(
|
||||||
|
["--bundle", str(BUNDLE), "--live-dry-run"],
|
||||||
|
client_factory=factory,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestDryRunArtifactCapture:
|
||||||
|
"""The captured set (run-config + preflight) is complete and deterministic."""
|
||||||
|
|
||||||
|
def test_captures_runconfig_and_preflight(
|
||||||
|
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
_clear_credentials(monkeypatch)
|
||||||
|
outbox = tmp_path / "outbox"
|
||||||
|
factory, _ = _counting_factory()
|
||||||
|
main(
|
||||||
|
[
|
||||||
|
"--bundle",
|
||||||
|
str(BUNDLE),
|
||||||
|
"--outbox",
|
||||||
|
str(outbox),
|
||||||
|
"--run-id",
|
||||||
|
RUN_ID,
|
||||||
|
"--live-dry-run",
|
||||||
|
],
|
||||||
|
client_factory=factory,
|
||||||
|
)
|
||||||
|
runconfig = json.loads((outbox / f"{RUN_ID}-runconfig.json").read_text("utf-8"))
|
||||||
|
preflight = json.loads((outbox / f"{RUN_ID}-preflight.json").read_text("utf-8"))
|
||||||
|
|
||||||
|
# §4 pt 3: model-id per role the loop calls, parameters, caps — no wall-clock.
|
||||||
|
assert runconfig["run_id"] == RUN_ID
|
||||||
|
assert runconfig["profile"] == "anthropic"
|
||||||
|
assert runconfig["models"]["proposer"] == "claude-haiku-4-5-20251001"
|
||||||
|
assert runconfig["models"]["checker"] == "claude-haiku-4-5-20251001"
|
||||||
|
assert runconfig["caps"]["max_rounds"] == 12
|
||||||
|
assert runconfig["caps"]["max_tokens"] == 150_000
|
||||||
|
assert runconfig["caps"]["max_budget_usd_per_call"] == 0.25
|
||||||
|
assert "date" not in runconfig # determinism: date is stamped at report time
|
||||||
|
# Preflight result captured (no credential here → credential refusal recorded).
|
||||||
|
assert preflight["run_id"] == RUN_ID
|
||||||
|
assert preflight["clear"] is False
|
||||||
|
assert any(r["check"] == "credential" for r in preflight["refusals"])
|
||||||
|
|
||||||
|
def test_artifacts_are_byte_deterministic(
|
||||||
|
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
_clear_credentials(monkeypatch)
|
||||||
|
first = tmp_path / "a"
|
||||||
|
second = tmp_path / "b"
|
||||||
|
for outbox in (first, second):
|
||||||
|
factory, _ = _counting_factory()
|
||||||
|
main(
|
||||||
|
[
|
||||||
|
"--bundle",
|
||||||
|
str(BUNDLE),
|
||||||
|
"--outbox",
|
||||||
|
str(outbox),
|
||||||
|
"--run-id",
|
||||||
|
RUN_ID,
|
||||||
|
"--live-dry-run",
|
||||||
|
],
|
||||||
|
client_factory=factory,
|
||||||
|
)
|
||||||
|
for name in (f"{RUN_ID}-runconfig.json", f"{RUN_ID}-preflight.json"):
|
||||||
|
assert (first / name).read_bytes() == (second / name).read_bytes()
|
||||||
|
|
||||||
|
|
||||||
|
class TestDryRunPreflightGate:
|
||||||
|
"""Exit code reflects go-live readiness; capture happens either way."""
|
||||||
|
|
||||||
|
def test_clear_preflight_exits_zero(
|
||||||
|
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
_set_credential(monkeypatch)
|
||||||
|
outbox = tmp_path / "outbox"
|
||||||
|
factory, created = _counting_factory()
|
||||||
|
code = main(
|
||||||
|
[
|
||||||
|
"--bundle",
|
||||||
|
str(BUNDLE),
|
||||||
|
"--outbox",
|
||||||
|
str(outbox),
|
||||||
|
"--run-id",
|
||||||
|
RUN_ID,
|
||||||
|
"--live-dry-run",
|
||||||
|
],
|
||||||
|
client_factory=factory,
|
||||||
|
)
|
||||||
|
assert code == 0
|
||||||
|
preflight = json.loads((outbox / f"{RUN_ID}-preflight.json").read_text("utf-8"))
|
||||||
|
assert preflight["clear"] is True
|
||||||
|
assert preflight["refusals"] == []
|
||||||
|
(client,) = created
|
||||||
|
assert client.calls == [] # still zero calls
|
||||||
|
|
||||||
|
def test_refused_preflight_captures_but_exits_nonzero(
|
||||||
|
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
_clear_credentials(monkeypatch)
|
||||||
|
outbox = tmp_path / "outbox"
|
||||||
|
factory, created = _counting_factory()
|
||||||
|
code = main(
|
||||||
|
[
|
||||||
|
"--bundle",
|
||||||
|
str(BUNDLE),
|
||||||
|
"--outbox",
|
||||||
|
str(outbox),
|
||||||
|
"--run-id",
|
||||||
|
RUN_ID,
|
||||||
|
"--live-dry-run",
|
||||||
|
],
|
||||||
|
client_factory=factory,
|
||||||
|
)
|
||||||
|
assert code != 0 # refused: the rig is not clear to go live
|
||||||
|
# ...yet the artifacts are captured and no model call was made.
|
||||||
|
assert (outbox / f"{RUN_ID}-runconfig.json").is_file()
|
||||||
|
assert (outbox / f"{RUN_ID}-preflight.json").is_file()
|
||||||
|
(client,) = created
|
||||||
|
assert client.calls == []
|
||||||
|
|
||||||
|
|
||||||
|
class TestKeyFreeConstruction:
|
||||||
|
"""K8 key premise: the SDK client constructs with no credential (no call)."""
|
||||||
|
|
||||||
|
def test_default_factory_constructs_without_credential(
|
||||||
|
self, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
_clear_credentials(monkeypatch)
|
||||||
|
from portfolio_optimiser_claude.run import default_client_factory
|
||||||
|
from portfolio_optimiser_claude.sdk_client import SdkModelClient
|
||||||
|
|
||||||
|
contracts = load_contracts(
|
||||||
|
data_source={"docs_dir": str(BUNDLE), "top_k": 3},
|
||||||
|
termination={"max_rounds": 1, "max_tokens": 1},
|
||||||
|
feedback={"decision": "approved", "rationale": "startup shape check (§10)"},
|
||||||
|
)
|
||||||
|
client = default_client_factory(contracts, 0.25)
|
||||||
|
assert isinstance(client, SdkModelClient)
|
||||||
Loading…
Add table
Add a link
Reference in a new issue