Kjell Tore Guttormsen
1600c188b4
Before ANY spend the operator sees a deterministic UPPER-BOUND USD estimate for a (portfolio-)run — a what-if over the models in model_map.json (Claude models) × effort levels (S3.6-analog, D-I pkt. 3 MUST-krav). No network, no model call, no key: pure config arithmetic (bound by an import-purity test, mirroring okf.py). - contracts.py: ModelPriceContract (usd_per_mtok > 0 + REQUIRED source + source_date so a stale rate is visible, never silent, §1) + PricingContract (non-empty; no hardcoded fallback rate) + load_pricing/_bundled_pricing. - data/pricing.example.json: per-Mtok rate per model id, each with source+date. Example rates are Anthropic's OUTPUT price (the higher rate) so the whole cap billed at that single rate can only overstate — the figure is marked ESTIMAT. Covers the model model_map configures, so the default path runs green. - costsim.py: estimate_costs (n_projects × cap × effort_factor tokens at the per-Mtok rate; a model with no price fails fast "missing price for <id>", never a guess) + render_estimate + `python -m …costsim`. Effort factors are a coarse modeling weight (not prices) — max effort = full cap = the true upper bound. No price literal anywhere (grep-guard proves it). - tests/test_costsim.py: schema fail-fast, missing-price fail-fast, scales with model × effort + reproducible, grep-guard, import purity, bundled-example + CLI offline smoke. Three seams detach-proven RED (effort factor, price guard, price literal). 462→478 green, golden byte-exact, full gate clean (ruff+format+mypy strict, 23 src files), run_s10.py/runs/ byte-untouched. README test-count sync ×2 + costsim.py module note. CLI run-total-cap wiring stays out of scope (planen lists 4 files); the mechanism is complete and proven load-bearing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RiTwaKLesgcwXx2mDviqpt
206 lines
13 KiB
Markdown
206 lines
13 KiB
Markdown
# portfolio-optimiser-claude
|
||
|
||
[](LICENSE)
|
||
[](pyproject.toml)
|
||
[](https://github.com/anthropics/claude-agent-sdk-python)
|
||
|
||
Sibling implementation of the portfolio-optimiser method on the **Claude Agent SDK**
|
||
(decision D7). An open, generic Python framework that finds cost savings *inside* each
|
||
project in a portfolio of independent projects: agents generate candidate measures, a
|
||
mandatory deterministic validator gates the numbers, domain experts judge via
|
||
human-in-the-loop, and the system learns from the verdicts.
|
||
|
||
> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
|
||
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
|
||
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
|
||
> seam, each proven by load-bearing tests (478 tests, all running offline without an API
|
||
> key). The programme's single budgeted **live model run has been executed and validated** —
|
||
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
|
||
|
||
> **Disclaimer — technical framework only.** The deployer owns DPIA, risk assessment, and
|
||
> the legal basis for any processing. The framework ships only the technical
|
||
> preconditions: local-only operation, first-class provenance, no silent data egress.
|
||
|
||
## Built from the spec, not the sibling
|
||
|
||
The method itself is framework-neutral and lives in
|
||
[`portfolio-optimiser-commons`](https://git.fromaitochitta.com/ktg/portfolio-optimiser-commons)
|
||
(consumed here as a git subtree under [`shared/`](shared/)): the normative method spec
|
||
(RFC 2119), the OKF bundle-navigation contract, the golden/conformance suite — the *only*
|
||
oracle for the validator — and the shared expert-reviewer persona skill. This repo
|
||
implements that spec on the Claude Agent SDK; it deliberately does **not**
|
||
reverse-engineer the MAF sibling
|
||
([`open/portfolio-optimiser`](https://git.fromaitochitta.com/open/portfolio-optimiser)).
|
||
Two independent implementations of one spec, compared afterwards, is the point of D7.
|
||
|
||
## Architecture — the seams
|
||
|
||
Everything below the run layer is pure config/file logic and runs deterministically,
|
||
offline. Module by module:
|
||
|
||
**Deterministic backbone** (method-spec §3 step 4, §7–§10)
|
||
- `ir.py` — the typed cost-IR of a candidate measure (§7.1).
|
||
- `validator.py` — the deterministic validator; blocking, and frozen by the shared golden
|
||
suite (§7.2), which is the only fasit it answers to.
|
||
- `provenance.py` — the first-class provenance stamp (§9); authoritative data, not
|
||
after-the-fact logging.
|
||
- `contracts.py` — fail-fast startup contracts (§10): stop criteria and budget caps are
|
||
required at startup, and the model map (`data/model_map.json`, role → Claude model id
|
||
per backend profile) is validated before anything runs.
|
||
|
||
**Context seam** (§3 step 1)
|
||
- `okf.py` — read-context built by **navigating** the project's OKF bundle (`index.md` +
|
||
frontmatter + cross-links, progressive disclosure) — never keyword chunk-stuffing. The
|
||
`type: verdict` layer is excluded from the read-context.
|
||
- `experience.py` — the ExpeL-style experience seam: store, structural retrieval, and the
|
||
gated fold. A prior expert verdict reaches the next hypothesis *only* through the fold,
|
||
never by leaking through context.
|
||
|
||
**Agentic loop** (§3 steps 2–5, §8)
|
||
- `budget.py` — the budget meter: no unbounded loop exists anywhere in the framework. On
|
||
top of the post-charge token/round caps sits an optional pre-call run-total USD belt that
|
||
refuses the next model call once the run has crossed its USD budget.
|
||
- `loop.py` — generate, maker–checker debate, gate, and informed refinement: the
|
||
validator's previous rejection reason is fed into the next bounded attempt, so the
|
||
model corrects against the falsification instead of re-answering identically.
|
||
|
||
**Learning loop** (§3 steps 7–8, §4–§6)
|
||
- `inbox.py` — the async verdict-file contract: an expert drops a plain-JSON verdict into
|
||
an inbox folder after a run; a later run ingests it tolerantly and merges it before the
|
||
fold. The shippable entrance for that later run is `run.py`:
|
||
`uv run python -m portfolio_optimiser_claude.run --bundle <dir> --inbox <dir>`.
|
||
- `promotion.py` — the promotion gate, **fail-closed**: only an approved verdict is
|
||
lifted into the OKF context layer; anything else raises and writes nothing.
|
||
- `persona.py` — the expert-reviewer persona sourced from the shared artifact in
|
||
[`shared/skills/expert-reviewer/`](shared/skills/expert-reviewer/) at call time, so the
|
||
shared persona is genuinely consumed and cannot rot silently.
|
||
|
||
**Value layer** (stack-local contract — mirrored from the sibling plan's capability
|
||
description, never from its code)
|
||
- `ledger.py` — the typed savings ledger: realized savings enter the book **only** through
|
||
the fail-closed expert gate (an approved verdict + a named expert + an explicit
|
||
timestamp), and the sum key is dimension-free, so the same realized saving surfaced via
|
||
two dimensions is never double-counted. Persistence is deterministic JSON.
|
||
- `goals.py` — the goal contract: absolute savings target, hard/soft. A hard goal reached
|
||
raises a structured stop event, never a silent stop; the percent-goal baseline is
|
||
D-E-gated and refused explicitly.
|
||
|
||
**Run layer** (the only part that touches the network)
|
||
- `sdk_client.py` — the Claude Agent SDK client, isolated from local configuration
|
||
(`setting_sources=[]`) so no user/project config can leak into a run.
|
||
- `artifacts.py` — §9 citations plus deterministic run-artifact persistence, including on
|
||
structured stops (a budget stop still leaves artifacts behind).
|
||
- `outbox.py` — the outbox output layer (S2.1): each completed run persists a `run_id`-named
|
||
proposal/outcome pair — the system's own output, which it writes freely (the role split
|
||
governs the inbox and wiki, not this). The outcome carries the inbox join key (`verdict_id`,
|
||
minted the same way the inbox mints an expert verdict's id) so outstanding verdicts can be
|
||
tracked and live artifacts captured; bytes reuse the deterministic house JSON writer, and
|
||
the S10 artifact formats are untouched.
|
||
- `run.py` — the generic run entrance: composes merge-inbox → seed → fold (§5) and drives
|
||
the loop under the budget meter, persisting artifacts on both outcomes — a structured
|
||
budget stop included. The model client is injected, so the offline suite proves the
|
||
same orchestration with a scripted client; only the CLI's default constructs the SDK
|
||
client.
|
||
- `portfolio.py` — the sequential multi-project run and learning loop: `run_portfolio` drives
|
||
N projects from a schema-validated reference config, composing each project's context afresh
|
||
(re-entrant, fresh debate state per run) and collecting one result per project in config
|
||
order. Two things are deliberately shared portfolio-wide: the §8 budget meter (the cap) and a
|
||
single learning `VerdictStore` — a verdict available when project k composes survives into
|
||
project k+1's fold (cross-project threading, §5), and an optional `verdict_dir` is the
|
||
portfolio-level expert inbox the system reads before each fold. The default failure policy
|
||
raises (a stack-local choice until D-D flips it to collect-and-continue).
|
||
- `run_s10.py` — the programme's ONE live run (cost discipline D6); run-path only.
|
||
- `costsim.py` — pre-run cost simulation (**offline** — the one Run-layer module that never
|
||
touches the network): a deterministic UPPER-BOUND USD estimate for a (portfolio-)run
|
||
*before* any spend, a what-if over the models in `model_map.json` × effort levels. Pricing
|
||
is schema-validated config (`data/pricing.example.json`): a per-Mtok rate per model, each
|
||
with a required source + date so a stale rate is visible, never silent. A model configured
|
||
with no price fails fast — there is no hardcoded rate anywhere (a grep-guard proves it), and
|
||
the figure is marked `ESTIMAT` (the whole cap billed at the rate is an upper bound; real runs
|
||
cost less). `uv run python -m portfolio_optimiser_claude.costsim`.
|
||
|
||
### Load-bearing tests (§11)
|
||
|
||
Every seam is proven by a test that goes **red when the seam is detached** — green-but-dead
|
||
is the failure mode the rule exists for. Among them: `test_step1_expel_loadbearing.py`
|
||
(the verdict signal reaches the prompt via the fold, and only via the fold),
|
||
`test_checker_gate_loadbearing.py` (an explicit checker reject blocks a validated
|
||
proposal), `test_step5_refine_loadbearing.py` (the rejection reason verifiably reaches the
|
||
retry prompt, and the loop still stops at the cap), `test_step7_async_loop_loadbearing.py`
|
||
(a verdict dropped after run A reaches run B's prompt through the file loop, with an
|
||
empty-inbox control), `test_step8_promotion_loadbearing.py` (the gate refuses non-approved
|
||
verdicts; the promoted signal stays out of the read-context),
|
||
`test_portfolio_learning_loadbearing.py` (a verdict available at project k survives into
|
||
project k+1's fold via the shared store, with a marker-absent control),
|
||
`test_outbox_loadbearing.py` (a completed run's `run_id`-named outbox pair is written on the
|
||
entrance path, with a no-outbox control, and the outcome carries the inbox join key), and
|
||
`test_sdk_isolation.py` (local config cannot capture the checker).
|
||
|
||
## The ingest layer — CSV and SQL, in front of the loop
|
||
|
||
The method spec forbids query-time retrieval against the bundle (§3 Step 1), so **data
|
||
reaches the model only via OKF bundles.** The ingest layer is the deterministic step that
|
||
satisfies that: a connector reads a real source, and the extract is materialized as an OKF
|
||
bundle the existing 8-step loop then consumes **unchanged**. It makes **zero model calls**,
|
||
touches no network, and `ingest.py` imports nothing from the SDK — it is pure standard
|
||
library. Built from the shared [`ingest-spec.md`](shared/ingest-spec.md) alone.
|
||
|
||
D7 implements the two conformance-required source types:
|
||
|
||
- **`file`** — a local CSV catalogue; extraction paths are boundary-checked fail-closed
|
||
against the source `root` (the OKF path rule).
|
||
- **`sql`** — a local SQLite database, opened **read-only** (`mode=ro`), one SELECT per
|
||
extraction; the connection location is resolved at run time from a named environment
|
||
variable (`connection_ref`), never stored in the manifest.
|
||
|
||
Both are frozen by byte-identical golden extractions
|
||
([`examples/ingest-golden-file/`](examples/ingest-golden-file/),
|
||
[`examples/ingest-golden-sql/`](examples/ingest-golden-sql/)) and by the load-bearing seam
|
||
tests (provenance stamping, navigability through the unchanged `okf.py`, the reserved
|
||
verdict layer, and re-ingest safety over a promoted verdict).
|
||
|
||
**Honesty rule (§1):** the `http` source type is an **optional extension point, not built in
|
||
D7** — a manifest naming it is rejected fail-fast at validation
|
||
(`test_malformed_manifest_is_rejected`), never silently accepted. The HTTP/MCP extension
|
||
point is demonstrated only in the MAF sibling (against a local mock, behind an opt-in
|
||
network flag); this repo ships no network connector and no live-source integration. How the
|
||
layer works and how one would extend it is documented in [`docs/extending.md`](docs/extending.md).
|
||
|
||
## The live run — S10, executed and validated
|
||
|
||
The loop's closure is proven offline by a scripted two-run simulation
|
||
(`simulation.py`: run A → persona verdict → §6 promotion gate → run B on a fresh store —
|
||
the marker crosses runs via the gate, and never without it). On top of that offline proof
|
||
sits the programme's single budgeted **real** run (D6: exactly one live API run in the
|
||
whole programme), executed 2026-07-03 against the micro bundle
|
||
[`shared/examples/bygg-energi-mikro/`](shared/examples/bygg-energi-mikro/):
|
||
|
||
- exit 0 · validator `validated` · checker `approve` on the first attempt · 2 of 12
|
||
rounds · 36 791 of 150 000 budgeted tokens · **cost $0.127514** (Haiku 4.5, per the
|
||
model map), under a first-class `max_budget_usd` cap.
|
||
- The proposal claimed a deliberately conservative 30 000 NOK saving against the
|
||
bundle's p10–p90 band of 68.5k–121k — and validates.
|
||
- All four artifacts are committed as fixed reference output in [`runs/s10/`](runs/s10/):
|
||
`proposal.json`, `provenance.json` (with §9 citations), `run_result.json`, `usage.json`.
|
||
|
||
Honesty rule (§1): everything else in the repo is deterministic and offline; nothing here
|
||
claims more live behaviour than that one documented run.
|
||
|
||
## Stack
|
||
|
||
Python ≥3.10 · [`claude-agent-sdk`](https://pypi.org/project/claude-agent-sdk/) ≥0.2
|
||
(bundles the Claude Code CLI; an API key is needed only at actual `query()` time) ·
|
||
`uv` · Pydantic for contract validation.
|
||
|
||
## Development
|
||
|
||
```bash
|
||
uv sync # install dependencies
|
||
uv run pytest # 478 tests — run without any API key and without network
|
||
uv run ruff check . && uv run ruff format --check .
|
||
uv run mypy src # strict
|
||
```
|
||
|
||
The offline invariant is deliberate: everything below the run layer is pure config/file
|
||
logic, so the full suite (including every load-bearing seam proof) runs with no key and
|
||
no network.
|