Kjell Tore Guttormsen
a831aa1e3b
PortfolioBudget + PortfolioMeter carry ONE token ledger over a whole portfolio pass -- and, seeded from a persisted spend file, across passes -- while the per-run Budget/TokenMeter pair is untouched. Three enforcement points, each doing a different job: - startup: a remainder that cannot fund one run raises BudgetRefused before anything loads (a pass that can afford zero projects is a caller mistake, not a result); - wave assembly: an unfundable project is NEVER STARTED and the pass stops structurally (budget_stop + stopped_early, completed runs preserved). Because every member of a wave is funded against the SAME pre-wave remainder, admission RESERVES each member's requirement -- otherwise a wave of k over-commits the cap by up to k runs; - pre-call: BudgetMiddleware refuses a call the remainder cannot pay for instead of making it. The post-charge check stays: real usage is only knowable after the response, so the guard stops the NEXT call, never the one in flight. budget_stop is its own field rather than a widened stop_reason -- a goal-stop is success, this is resource exhaustion, and fusing them would make "we stopped" unreadable. PortfolioMeter splits record/check so tokens the provider already billed reach the ledger even when the same charge breaks the run's own cap. read_spend raises on corrupt content (our own accounting state, unlike the tolerant RAW inbox layer); write_spend takes a REQUIRED stamp with no wall-clock default, mirroring promote_verdict. Load-bearing MEASURED, not asserted -- 6 mutations, all red: detach the wave check; detach the pre-call guard; detach the wave reservation; check the run cap before crediting the global ledger; detach the startup refusal; make read_spend tolerant. Files restored from shasum-verified copies after each. Two findings worth keeping: the pre-call guard MASKS a detached wave check if the test asserts on overspend (spend stays under the cap either way), so the load-bearing assertion had to become failures == () plus never-started; and the token arithmetic is probed (32 tokens/run at tokens=8), not guessed. 537 -> 553 tests, ruff + mypy green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EaxFnaDAbMQkmTeX4u7sd
202 lines
13 KiB
Markdown
202 lines
13 KiB
Markdown
# portfolio-optimiser
|
||
|
||
[](LICENSE)
|
||
[](pyproject.toml)
|
||
[](https://github.com/microsoft/agent-framework)
|
||
|
||
A generic, open framework — built on **Microsoft Agent Framework (MAF)** — that finds cost
|
||
savings *inside* each project of a portfolio of independent projects. A swarm of agents
|
||
generates candidate measures; a **mandatory deterministic validator** (solver + Monte Carlo)
|
||
decides the numbers; domain experts judge the outcomes (human-in-the-loop); and the system
|
||
**learns from their verdicts** across runs.
|
||
|
||
> **Status:** the full 8-step agentic loop is wired and proven with load-bearing tests, and the
|
||
> end-to-end proof is an **offline simulation** with a scripted stand-in client — no live-model
|
||
> run yet. The **ingest layer** (real data sources) is implemented — file/CSV and SQL on both
|
||
> stacks with bit-identical golden extractions from the shared spec, plus HTTP as a MAF-only
|
||
> demonstrated extension point against a local mock — but exercised only against committed
|
||
> fixtures: no bundle has yet been materialized from a live source. A sibling implementation of
|
||
> the same method on the **Claude Agents SDK** is built in parallel from the same shared spec.
|
||
|
||
> **Disclaimer — technical framework only.** Deploying organizations own their processing
|
||
> purposes and assessments (DPIA, risk/ROS, security review). The framework ships the technical
|
||
> prerequisites — local-only mode, provenance, no silent data egress — but makes no compliance
|
||
> guarantees.
|
||
|
||
## Built on an LLM wiki: Karpathy's idea, Google's format
|
||
|
||
The knowledge architecture is the heart of the project, and it is deliberately not ours:
|
||
|
||
- **The idea** is Andrej Karpathy's **"LLM wiki"**: instead of pointing a model at documents
|
||
written for people, you curate a small, versioned body of knowledge written *for the model
|
||
to read* — concept files, explicit structure, explicit links.
|
||
- **The format** is Google Cloud's **[Open Knowledge Format (OKF)](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)**
|
||
(open spec, v0.1), which formalizes that pattern: a knowledge **bundle** is a directory of
|
||
markdown files with YAML frontmatter (one required field, `type`), a reserved `index.md`
|
||
entry point, and intra-bundle cross-links forming an emergent graph. Custom frontmatter
|
||
fields are allowed and must be preserved — which is exactly where this project's own layers
|
||
(expert verdicts, ingest provenance) live.
|
||
|
||
Because OKF is open and vendor-neutral, the *same* bundles are consumed unchanged by both
|
||
reference implementations (MAF and the Claude Agents SDK sibling) — the knowledge outlives any
|
||
particular agent stack.
|
||
|
||
**Not RAG.** Agents read a bundle by **navigating** it — `index.md` first, then its
|
||
cross-links, with progressive disclosure — never by keyword retrieval or stuffing the whole
|
||
bundle into a prompt. Query-time retrieval against the bundle is explicitly forbidden by the
|
||
method spec: it would leak the verdict layer around the learning gate.
|
||
|
||
## AI-first, humans on top
|
||
|
||
A traditional wiki is built for *people* — optimized for humans finding and reading
|
||
information, with machine access bolted on afterwards. This project inverts that order, and is
|
||
a concrete example of what that looks like:
|
||
|
||
- The wiki (the OKF bundle) is written **for the model**: it is the agent's working memory and
|
||
the substrate the learning loop reads from and promotes into.
|
||
- The **human affordances are layers on top**: experts judge outcomes by dropping a plain JSON
|
||
verdict file in an inbox folder; an explicit, fail-closed **promotion gate** is the only path
|
||
by which an approved verdict becomes wiki knowledge; reports and reviews are rendered *from*
|
||
the machine-readable layers.
|
||
|
||
Humans stay decisive — nothing enters the wiki without an approval — but the primary reader of
|
||
every file is the model, not a person browsing.
|
||
|
||
## How it works
|
||
|
||
One run, one project, eight steps — with the learning loop closing across runs:
|
||
|
||
1. **Understand** — navigate the project's OKF bundle; fold the candidate's *prior expert
|
||
verdicts* into the hypothesis prompt (ExpeL-style, retrieved structurally, never by text).
|
||
2. **Hypothesise** — one typed candidate measure (strict IR, fail-fast schema).
|
||
3. **Debate** — a maker-checker pair argues the reasoning (round-capped).
|
||
4. **Validate** — two falsifiers on the same candidate: the **deterministic validator** gates
|
||
the numbers (blocking, never optional) and the **checker** gates the reasoning.
|
||
5. **Refine** — a rejected attempt retries *informed* by the rejection reason, under hard
|
||
attempt and token caps. Unbounded loops are forbidden everywhere.
|
||
6. **Propose or discard** — a validated proposal with risk percentiles, or a typed rejection.
|
||
7. **Expert feedback** — days later, an expert drops a verdict file in an inbox folder; a
|
||
later run picks it up. Fully resumable; no live session assumed.
|
||
8. **Promote** — an *approved* verdict is lifted into the wiki as a `type: verdict` concept
|
||
file, navigable by the next run. The gate is fail-closed: raw agent output never
|
||
self-promotes.
|
||
|
||
Every proposal carries provenance (citations into the bundle, model, validator decision, token
|
||
usage). Every seam above is protected by a **load-bearing test** — a test designed to *fail*
|
||
when the seam is detached, so the loop cannot silently degrade into theater.
|
||
|
||
## How it is set up
|
||
|
||
- **One shared, framework-neutral core** ([`shared/`](shared/), a git subtree of
|
||
[`portfolio-optimiser-commons`](https://git.fromaitochitta.com/ktg/portfolio-optimiser-commons)):
|
||
the business concept, the normative [method spec](shared/method-spec.md) and
|
||
[ingest spec](shared/ingest-spec.md), the expert-reviewer persona as an Agent Skill, and an
|
||
example bundle with a golden suite as the only ground truth. Both stacks implement from the
|
||
spec alone.
|
||
- **Per project: one OKF bundle** — the bundled examples are hand-curated; the ingest layer that
|
||
materializes a bundle from a source (file catalogues/CSV + SQL, HTTP as a MAF-only demonstrated
|
||
extension point) via a deterministic, schema-validated manifest that runs *before* the loop is
|
||
implemented and exercised against committed fixtures — no bundle has yet been materialized from
|
||
a live source.
|
||
- **Run:** the `run.py` CLI has **three modes** — a documented partition, since one invocation
|
||
cannot exercise every flag:
|
||
- **Single-project** — `PROJECT_ID --docs-dir <dir>`, plus optional `--bundle-dir`,
|
||
`--verdict-dir`, `--outbox-dir` (which requires `--run-id`), `--dimension-config`,
|
||
`--semantic-retrieval`, `--decision`/`--rationale`, and `--live-dry-run`.
|
||
- **Portfolio** — `--portfolio`, plus optional `--goals`, `--ledger`, `--dimension-config`,
|
||
`--semantic-retrieval`; it stops early and prints a `goal reached: …` line when the
|
||
accumulated ledger meets a goal.
|
||
- **Value report (S5.4, read-only)** — `--report --ledger <file>` rolls up the ledger's realized
|
||
savings to stdout: per-project totals, the portfolio total, flagged cross-dimension overlaps
|
||
(each counted once), and per-entry provenance. Add `--json` for deterministic JSON instead of
|
||
the human table. It makes **no model calls** and is mode-exclusive — only `--ledger`/`--json`
|
||
are permitted alongside `--report`; `--report` requires `--ledger`, and a stray `--json`
|
||
without `--report` is refused (rc 1, never silently ignored).
|
||
|
||
```bash
|
||
# Single-project, offline drill (builds contracts + clients, stops before the first model call):
|
||
uv run python -m portfolio_optimiser.run FV42-GSV-E1 --docs-dir <docs> --bundle-dir <bundle> --live-dry-run
|
||
# Portfolio run with a savings goal checked against an accumulated ledger:
|
||
uv run python -m portfolio_optimiser.run --portfolio --goals goals.json --ledger ledger.json
|
||
# Read-only value report over an accumulated ledger (human table; add --json for JSON):
|
||
uv run python -m portfolio_optimiser.run --report --ledger ledger.json
|
||
```
|
||
|
||
`--semantic-retrieval` (S3.1) is an **opt-in** ranking change, **off by default**. Off, prior
|
||
verdicts are ranked exactly as before: a structural score over the affected cost-code set,
|
||
measure type and magnitude bucket, with surface text deliberately excluded. On, that score is
|
||
blended with a cosine term over the *same* structural triple, which lets a prior verdict on a
|
||
*different* cost-code set outrank one that ties structurally.
|
||
|
||
**What this ships is the seam, not better retrieval.** The bundled `FakeEmbedder` is a
|
||
deterministic sha256 projection carrying no semantics, so over a structural tie the resulting
|
||
order is deterministic but arbitrary. Retrieval *quality* depends entirely on injecting a real
|
||
embedder — `--embedder-config` selects one from a closed registry (never an import path; a
|
||
config file can never name arbitrary code to load), and `docs/extending.md` documents the
|
||
`Embedder` protocol. The embedding excludes `description`, matching the structural score and the
|
||
verdict-id hash, so a flag-on run reads no surface text either.
|
||
|
||
The flag is accepted in both run modes, but in single-project mode it **requires** `--bundle-dir`
|
||
and `--verdict-dir`: without them it cannot take effect, and the run is refused rather than
|
||
silently ignoring the flag. Nothing about a flag-off run changes, and no savings claim depends
|
||
on it.
|
||
|
||
The **prior-verdict fold — the learning step — happens only on the `--bundle-dir` path**; a
|
||
plain `--docs-dir`-only run is single-shot (no fold). `--decision`/`--rationale` apply to the
|
||
single-project path only and are inert in portfolio mode. **`--outbox-dir` must differ from
|
||
`--verdict-dir`**: writing the raw outbox into a folder later read as an inbox would re-ingest
|
||
raw agent output past the promotion gate (self-contamination) — documented here, deliberately
|
||
not CLI-enforced. Stop criteria and budget caps are required at startup. Try the offline
|
||
end-to-end proof (no model, no network): `uv run python -m portfolio_optimiser.simulation`.
|
||
|
||
- **A global token cap across the whole portfolio, enforced before the call.** Per-run caps alone
|
||
let N projects cost N times that with no ceiling over the pass. Pass a
|
||
`PortfolioMeter(PortfolioBudget(max_total_tokens=…, max_tokens_per_run=…))` to `run_portfolio`
|
||
and one ledger bounds the entire pass — and, seeded from `budget.read_spend`, a *series* of
|
||
passes. It bites in three places: a remainder that cannot fund one run refuses the pass at
|
||
startup (`BudgetRefused`); a project that cannot be funded is **never started**, stopping the
|
||
pass structurally (`budget_stop`, completed runs preserved); and a chat call the remainder
|
||
cannot pay for is **refused rather than made** (the post-charge check remains, since real usage
|
||
is only knowable after the response). Spend persists via `budget.write_spend`, which takes an
|
||
explicit stamp and no wall-clock default, so the file is byte-deterministic. Python API only —
|
||
not yet exposed on the CLI.
|
||
|
||
## What this enables
|
||
|
||
The reference case is portfolio cost review (the example bundle is a building-energy measure),
|
||
but the architecture is designed to generalize to any setting with the same shape — candidate
|
||
measures inside independent projects, numbers a deterministic tool can check, and judgement
|
||
only an expert has:
|
||
|
||
- **Portfolio reviews** — cost savings, energy efficiency, maintenance and procurement
|
||
measures, proposed per project and validated against the project's own data.
|
||
- **Compounding organizational memory** — approved expert verdicts become navigable knowledge;
|
||
the next run's hypotheses start from what experts actually decided, including realization
|
||
gaps no solver can compute.
|
||
- **Auditable AI** — an unbroken provenance chain from expert decision back through proposal,
|
||
bundle file and text span, and (with ingest) to the source system, query, and timestamp.
|
||
- **Vendor-neutral knowledge** — the same bundles drive two different agent stacks; switching
|
||
frameworks does not orphan the organization's curated knowledge.
|
||
|
||
## Docs
|
||
|
||
- [Building a knowledge base](docs/knowledge-base-recipe.md) — the team recipe (technical +
|
||
domain expert) for curating a bundle, with the honest expectation that a good base takes 1–2
|
||
weeks of dedicated work.
|
||
- [Target picture](docs/plan/2026-06-26-maalbilde-agentic-loop.md) — the agentic loop + OKF
|
||
knowledge architecture (north star).
|
||
- [Prior-art & platform research](docs/research/2026-06-23-prior-art-platform.md) (incl.
|
||
implementation register §15).
|
||
- [Ingest target picture](docs/plan/2026-07-03-maalbilde-ingest-lag.md) — connectors and the
|
||
ingest layer (frozen 2026-07-03).
|
||
|
||
## Stack & develop
|
||
|
||
Python ≥3.10 · MAF via the split GA packages (see `pyproject.toml`) · `uv`. Backend profiles:
|
||
Azure/Foundry (full) + local (fallback).
|
||
|
||
```bash
|
||
uv sync
|
||
uv run pytest
|
||
uv run ruff check .
|
||
```
|