Kjell Tore Guttormsen
b9dd479865
The operator's view of the long feedback loop (S5.1-analog, parity row 22;
buildable after K5): which proposals still AWAIT an expert verdict, and who
should judge each — a pure file-based id-join across the three layers hitl
READS and NEVER writes (role split §3 Step 7: the expert writes the inbox, the
system reads it; notification is K10's job, never this).
- hitl.py:
* pending_proposals — the id-join. An outbox proposal (K5) is pending unless
its persisted verdict_id (read verbatim from {run_id}-outcome.json, minted
the SAME way the inbox mints a verdict id — the K5 assumption) is in the
settled set. settled = §4.2-valid inbox verdicts (THROUGH load_inbox, so a
skipped/unknown decision never settles anything) ∪ promoted verdicts (§6,
optional bundle_dirs, so the core join is exactly outbox↔inbox).
* RoutingContract — nøkkel→ekspert, schema-validated fail-fast (§10): non-empty
table, non-empty keys/expert ids, optional default_expert. route_pending maps
a proposal's measure (a config-string key NOW; K13 formalizes the dimension
catalog) to an expert; an unmatched measure → default, else UNROUTED.
* CLI python -m …hitl pending|route — pending is a pure report (exit 0); route
loads the routing config fail-fast (a malformed/missing config exits non-zero
WITHOUT touching any layer). Neither subcommand writes anything.
- test_hitl_loadbearing.py: 23 tests. TWO seams detach-proven RED — the id-join
seam (drop the `not in settled` filter → a judged proposal is STILL listed →
red) and the read-only seam (any read path that writes a byte → the before/
after outbox+inbox snapshot diverges → red). Covers: undecided → pending,
inbox/promoted verdict settles, exact-id join (no coincidental match), skipped
decision does not settle, deterministic order, malformed routing fail-fast,
measure→expert / default / UNROUTED, and the CLI subcommands.
- 521→544 green, golden byte-exact, full gate clean (ruff+format+mypy strict,
25 src files). README: test-count sync ×2 + hitl module note + load-bearing
mention. IKKE-scope (held): notification (K10), web-UI, writing the inbox.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RiTwaKLesgcwXx2mDviqpt
237 lines
16 KiB
Markdown
237 lines
16 KiB
Markdown
# portfolio-optimiser-claude
|
||
|
||
[](LICENSE)
|
||
[](pyproject.toml)
|
||
[](https://github.com/anthropics/claude-agent-sdk-python)
|
||
|
||
Sibling implementation of the portfolio-optimiser method on the **Claude Agent SDK**
|
||
(decision D7). An open, generic Python framework that finds cost savings *inside* each
|
||
project in a portfolio of independent projects: agents generate candidate measures, a
|
||
mandatory deterministic validator gates the numbers, domain experts judge via
|
||
human-in-the-loop, and the system learns from the verdicts.
|
||
|
||
> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
|
||
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
|
||
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
|
||
> seam, each proven by load-bearing tests (544 tests, all running offline without an API
|
||
> key). The programme's single budgeted **live model run has been executed and validated** —
|
||
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
|
||
|
||
> **Disclaimer — technical framework only.** The deployer owns DPIA, risk assessment, and
|
||
> the legal basis for any processing. The framework ships only the technical
|
||
> preconditions: local-only operation, first-class provenance, no silent data egress.
|
||
|
||
## Built from the spec, not the sibling
|
||
|
||
The method itself is framework-neutral and lives in
|
||
[`portfolio-optimiser-commons`](https://git.fromaitochitta.com/ktg/portfolio-optimiser-commons)
|
||
(consumed here as a git subtree under [`shared/`](shared/)): the normative method spec
|
||
(RFC 2119), the OKF bundle-navigation contract, the golden/conformance suite — the *only*
|
||
oracle for the validator — and the shared expert-reviewer persona skill. This repo
|
||
implements that spec on the Claude Agent SDK; it deliberately does **not**
|
||
reverse-engineer the MAF sibling
|
||
([`open/portfolio-optimiser`](https://git.fromaitochitta.com/open/portfolio-optimiser)).
|
||
Two independent implementations of one spec, compared afterwards, is the point of D7.
|
||
|
||
## Architecture — the seams
|
||
|
||
Everything below the run layer is pure config/file logic and runs deterministically,
|
||
offline. Module by module:
|
||
|
||
**Deterministic backbone** (method-spec §3 step 4, §7–§10)
|
||
- `ir.py` — the typed cost-IR of a candidate measure (§7.1).
|
||
- `validator.py` — the deterministic validator; blocking, and frozen by the shared golden
|
||
suite (§7.2), which is the only fasit it answers to.
|
||
- `provenance.py` — the first-class provenance stamp (§9); authoritative data, not
|
||
after-the-fact logging.
|
||
- `contracts.py` — fail-fast startup contracts (§10): stop criteria and budget caps are
|
||
required at startup, and the model map (`data/model_map.json`, role → Claude model id
|
||
per backend profile) is validated before anything runs.
|
||
|
||
**Context seam** (§3 step 1)
|
||
- `okf.py` — read-context built by **navigating** the project's OKF bundle (`index.md` +
|
||
frontmatter + cross-links, progressive disclosure) — never keyword chunk-stuffing. The
|
||
`type: verdict` layer is excluded from the read-context.
|
||
- `experience.py` — the ExpeL-style experience seam: store, structural retrieval, and the
|
||
gated fold. A prior expert verdict reaches the next hypothesis *only* through the fold,
|
||
never by leaking through context.
|
||
|
||
**Agentic loop** (§3 steps 2–5, §8)
|
||
- `budget.py` — the budget meter: no unbounded loop exists anywhere in the framework. On
|
||
top of the post-charge token/round caps sits an optional pre-call run-total USD belt that
|
||
refuses the next model call once the run has crossed its USD budget.
|
||
- `loop.py` — generate, maker–checker debate, gate, and informed refinement: the
|
||
validator's previous rejection reason is fed into the next bounded attempt, so the
|
||
model corrects against the falsification instead of re-answering identically.
|
||
|
||
**Learning loop** (§3 steps 7–8, §4–§6)
|
||
- `inbox.py` — the async verdict-file contract: an expert drops a plain-JSON verdict into
|
||
an inbox folder after a run; a later run ingests it tolerantly and merges it before the
|
||
fold. The shippable entrance for that later run is `run.py`:
|
||
`uv run python -m portfolio_optimiser_claude.run --bundle <dir> --inbox <dir>`.
|
||
- `promotion.py` — the promotion gate, **fail-closed**: only an approved verdict is
|
||
lifted into the OKF context layer; anything else raises and writes nothing.
|
||
- `persona.py` — the expert-reviewer persona sourced from the shared artifact in
|
||
[`shared/skills/expert-reviewer/`](shared/skills/expert-reviewer/) at call time, so the
|
||
shared persona is genuinely consumed and cannot rot silently.
|
||
|
||
**Value layer** (stack-local contract — mirrored from the sibling plan's capability
|
||
description, never from its code)
|
||
- `ledger.py` — the typed savings ledger: realized savings enter the book **only** through
|
||
the fail-closed expert gate (an approved verdict + a named expert + an explicit
|
||
timestamp), and the sum key is dimension-free, so the same realized saving surfaced via
|
||
two dimensions is never double-counted. Persistence is deterministic JSON.
|
||
- `goals.py` — the goal contract: absolute savings target, hard/soft. A hard goal reached
|
||
raises a structured stop event, never a silent stop; the percent-goal baseline is
|
||
D-E-gated and refused explicitly.
|
||
|
||
**Run layer** (the only part that touches the network)
|
||
- `sdk_client.py` — the Claude Agent SDK client, isolated from local configuration
|
||
(`setting_sources=[]`) so no user/project config can leak into a run.
|
||
- `artifacts.py` — §9 citations plus deterministic run-artifact persistence, including on
|
||
structured stops (a budget stop still leaves artifacts behind).
|
||
- `outbox.py` — the outbox output layer (S2.1): each completed run persists a `run_id`-named
|
||
proposal/outcome pair — the system's own output, which it writes freely (the role split
|
||
governs the inbox and wiki, not this). The outcome carries the inbox join key (`verdict_id`,
|
||
minted the same way the inbox mints an expert verdict's id) so outstanding verdicts can be
|
||
tracked and live artifacts captured; bytes reuse the deterministic house JSON writer, and
|
||
the S10 artifact formats are untouched.
|
||
- `run.py` — the generic run entrance: composes merge-inbox → seed → fold (§5) and drives
|
||
the loop under the budget meter, persisting artifacts on both outcomes — a structured
|
||
budget stop included. The model client is injected, so the offline suite proves the
|
||
same orchestration with a scripted client; only the CLI's default constructs the SDK
|
||
client. `--live-dry-run` is the **live-run drill** (K8): it builds everything a real run
|
||
would (contracts → compose → client construction → preflight) and captures a `run_id`-named
|
||
`runconfig` + `preflight` pair to the outbox (model-id, parameters, caps — no wall-clock, so
|
||
the bytes stay deterministic), then **stops before the first model call**. It exits 0 when
|
||
the preflight is clear and non-zero when it refused, but captures the artifacts and makes
|
||
zero model calls either way — a future operator-gated live run is rigged and rehearsed
|
||
offline, with no spend.
|
||
- `portfolio.py` — the sequential multi-project run and learning loop: `run_portfolio` drives
|
||
N projects from a schema-validated reference config, composing each project's context afresh
|
||
(re-entrant, fresh debate state per run) and collecting one result per project in config
|
||
order. Two things are deliberately shared portfolio-wide: the §8 budget meter (the cap) and a
|
||
single learning `VerdictStore` — a verdict available when project k composes survives into
|
||
project k+1's fold (cross-project threading, §5), and an optional `verdict_dir` is the
|
||
portfolio-level expert inbox the system reads before each fold. The default failure policy
|
||
raises (a stack-local choice until D-D flips it to collect-and-continue).
|
||
- `run_s10.py` — the programme's ONE live run (cost discipline D6); run-path only.
|
||
- `costsim.py` — pre-run cost simulation (**offline** — the one Run-layer module that never
|
||
touches the network): a deterministic UPPER-BOUND USD estimate for a (portfolio-)run
|
||
*before* any spend, a what-if over the models in `model_map.json` × effort levels. Pricing
|
||
is schema-validated config (`data/pricing.example.json`): a per-Mtok rate per model, each
|
||
with a required source + date so a stale rate is visible, never silent. A model configured
|
||
with no price fails fast — there is no hardcoded rate anywhere (a grep-guard proves it), and
|
||
the figure is marked `ESTIMAT` (the whole cap billed at the rate is an upper bound; real runs
|
||
cost less). `uv run python -m portfolio_optimiser_claude.costsim`.
|
||
- `preflight.py` — the SDK/API preflight (**offline** — the boundary the operator crosses
|
||
*before* any spend): everything that can be validated without a model call is checked here,
|
||
so a broken config stops cheaply instead of on the first billed call. Four checks — a
|
||
credential is present and not a placeholder form (`ANTHROPIC_API_KEY`, or the bundled CLI's
|
||
own `CLAUDE_CODE_OAUTH_TOKEN`; the value itself is *never* validated online), the model_map
|
||
profile exists and every id it resolves to is real, `claude_agent_sdk` imports and its
|
||
bundled CLI is present on disk, and the §8 stop/budget contract is set. It never calls the
|
||
API — a green preflight implies no more than that (§1). Each deficiency is a structured,
|
||
actionable refusal. `uv run python -m portfolio_optimiser_claude.preflight`.
|
||
- `hitl.py` — the operator's view of the long feedback loop (**offline, read-only**): which
|
||
proposals still *await* an expert verdict, and who should judge each. A pure file-based
|
||
id-join across the three layers it READS and never writes — the outbox (K5, the persisted
|
||
`verdict_id` join key), the inbox (a §4.2-valid verdict settles a proposal), and optionally a
|
||
bundle's promoted verdicts (§6). Routing maps a proposal's `measure` (a config-string key
|
||
now; K13 formalizes the dimension catalog) to an expert via a schema-validated table
|
||
(`nøkkel→ekspert`, fail-fast) with an optional default; an unmatched measure is UNROUTED.
|
||
`uv run python -m portfolio_optimiser_claude.hitl pending|route`.
|
||
|
||
### Load-bearing tests (§11)
|
||
|
||
Every seam is proven by a test that goes **red when the seam is detached** — green-but-dead
|
||
is the failure mode the rule exists for. Among them: `test_step1_expel_loadbearing.py`
|
||
(the verdict signal reaches the prompt via the fold, and only via the fold),
|
||
`test_checker_gate_loadbearing.py` (an explicit checker reject blocks a validated
|
||
proposal), `test_step5_refine_loadbearing.py` (the rejection reason verifiably reaches the
|
||
retry prompt, and the loop still stops at the cap), `test_step7_async_loop_loadbearing.py`
|
||
(a verdict dropped after run A reaches run B's prompt through the file loop, with an
|
||
empty-inbox control), `test_step8_promotion_loadbearing.py` (the gate refuses non-approved
|
||
verdicts; the promoted signal stays out of the read-context),
|
||
`test_portfolio_learning_loadbearing.py` (a verdict available at project k survives into
|
||
project k+1's fold via the shared store, with a marker-absent control),
|
||
`test_outbox_loadbearing.py` (a completed run's `run_id`-named outbox pair is written on the
|
||
entrance path, with a no-outbox control, and the outcome carries the inbox join key),
|
||
`test_preflight.py` (a missing credential and a placeholder model id are each refused before
|
||
any spend, and the preflight carries no network path of its own),
|
||
`test_dry_run_loadbearing.py` (the live-run drill captures its `runconfig` + `preflight`
|
||
artifacts and stops before the first model call — a call-counting client proves zero calls,
|
||
red the moment the stop seam is detached),
|
||
`test_hitl_loadbearing.py` (a proposal with no verdict is listed pending and disappears once
|
||
an inbox or promoted verdict shares its id — red the moment the id-join filter is detached —
|
||
and hitl never writes any layer, proven by a before/after byte snapshot), and
|
||
`test_sdk_isolation.py` (local config cannot capture the checker).
|
||
|
||
## The ingest layer — CSV and SQL, in front of the loop
|
||
|
||
The method spec forbids query-time retrieval against the bundle (§3 Step 1), so **data
|
||
reaches the model only via OKF bundles.** The ingest layer is the deterministic step that
|
||
satisfies that: a connector reads a real source, and the extract is materialized as an OKF
|
||
bundle the existing 8-step loop then consumes **unchanged**. It makes **zero model calls**,
|
||
touches no network, and `ingest.py` imports nothing from the SDK — it is pure standard
|
||
library. Built from the shared [`ingest-spec.md`](shared/ingest-spec.md) alone.
|
||
|
||
D7 implements the two conformance-required source types:
|
||
|
||
- **`file`** — a local CSV catalogue; extraction paths are boundary-checked fail-closed
|
||
against the source `root` (the OKF path rule).
|
||
- **`sql`** — a local SQLite database, opened **read-only** (`mode=ro`), one SELECT per
|
||
extraction; the connection location is resolved at run time from a named environment
|
||
variable (`connection_ref`), never stored in the manifest.
|
||
|
||
Both are frozen by byte-identical golden extractions
|
||
([`examples/ingest-golden-file/`](examples/ingest-golden-file/),
|
||
[`examples/ingest-golden-sql/`](examples/ingest-golden-sql/)) and by the load-bearing seam
|
||
tests (provenance stamping, navigability through the unchanged `okf.py`, the reserved
|
||
verdict layer, and re-ingest safety over a promoted verdict).
|
||
|
||
**Honesty rule (§1):** the `http` source type is an **optional extension point, not built in
|
||
D7** — a manifest naming it is rejected fail-fast at validation
|
||
(`test_malformed_manifest_is_rejected`), never silently accepted. The HTTP/MCP extension
|
||
point is demonstrated only in the MAF sibling (against a local mock, behind an opt-in
|
||
network flag); this repo ships no network connector and no live-source integration. How the
|
||
layer works and how one would extend it is documented in [`docs/extending.md`](docs/extending.md).
|
||
|
||
## The live run — S10, executed and validated
|
||
|
||
The loop's closure is proven offline by a scripted two-run simulation
|
||
(`simulation.py`: run A → persona verdict → §6 promotion gate → run B on a fresh store —
|
||
the marker crosses runs via the gate, and never without it). On top of that offline proof
|
||
sits the programme's single budgeted **real** run (D6: exactly one live API run in the
|
||
whole programme), executed 2026-07-03 against the micro bundle
|
||
[`shared/examples/bygg-energi-mikro/`](shared/examples/bygg-energi-mikro/):
|
||
|
||
- exit 0 · validator `validated` · checker `approve` on the first attempt · 2 of 12
|
||
rounds · 36 791 of 150 000 budgeted tokens · **cost $0.127514** (Haiku 4.5, per the
|
||
model map), under a first-class `max_budget_usd` cap.
|
||
- The proposal claimed a deliberately conservative 30 000 NOK saving against the
|
||
bundle's p10–p90 band of 68.5k–121k — and validates.
|
||
- All four artifacts are committed as fixed reference output in [`runs/s10/`](runs/s10/):
|
||
`proposal.json`, `provenance.json` (with §9 citations), `run_result.json`, `usage.json`.
|
||
|
||
Honesty rule (§1): everything else in the repo is deterministic and offline; nothing here
|
||
claims more live behaviour than that one documented run.
|
||
|
||
## Stack
|
||
|
||
Python ≥3.10 · [`claude-agent-sdk`](https://pypi.org/project/claude-agent-sdk/) ≥0.2
|
||
(bundles the Claude Code CLI; an API key is needed only at actual `query()` time) ·
|
||
`uv` · Pydantic for contract validation.
|
||
|
||
## Development
|
||
|
||
```bash
|
||
uv sync # install dependencies
|
||
uv run pytest # 544 tests — run without any API key and without network
|
||
uv run ruff check . && uv run ruff format --check .
|
||
uv run mypy src # strict
|
||
```
|
||
|
||
The offline invariant is deliberate: everything below the run layer is pure config/file
|
||
logic, so the full suite (including every load-bearing seam proof) runs with no key and
|
||
no network.
|