portfolio-optimiser/README.md

# portfolio-optimiser

Generic, open framework on Microsoft Agent Framework (MAF): multi-agent cost-saving proposals gated by a mandatory deterministic validator, with HITL learning.

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-%E2%89%A53.10-blue.svg)](pyproject.toml)
[![Built on Microsoft Agent Framework](https://img.shields.io/badge/built%20on-Microsoft%20Agent%20Framework-0078D4.svg)](https://github.com/microsoft/agent-framework)

A generic, open framework — built on **Microsoft Agent Framework (MAF)** — that finds cost
savings *inside* each project of a portfolio of independent projects. A swarm of agents
generates candidate measures; a **mandatory deterministic validator** (solver + Monte Carlo)
decides the numbers; domain experts judge the outcomes (human-in-the-loop); and the system
**learns from their verdicts** across runs.

## Install

Python ≥3.10, with [`uv`](https://docs.astral.sh/uv/). The package is not published to a package
index — install it from source:

```bash
git clone https://git.fromaitochitta.com/open/portfolio-optimiser.git
cd portfolio-optimiser
uv sync
```

The clone is the documented path because the walkthrough below points at files in the tree. It is
no longer a technical requirement: a built wheel carries the shared spec, the persona skill and the
example bundles under [`shared/`](shared/README.md) as packaged data, and the resolver prefers a
working tree when one is present (`PORTFOLIO_SHARED_ROOT` overrides both) — so an installed
distribution works without a checkout.

`uv sync` installs two commands: `portfolio-optimiser` (the CLI) and `portfolio-optimiser-demo`
(the offline end-to-end proof). They are equivalent to the `python -m portfolio_optimiser.run` and
`python -m portfolio_optimiser.simulation` forms used throughout this README, which keep working —
the module form is spelled out below so a reader can see which module answers a given command.

Verify the install by running the whole suite from the clean clone:

```bash
uv run pytest
```

There is no CI runner in this organization, so nothing runs that suite automatically — the command
above is the verification.

## Walk the whole chain offline

Seven commands, no API key, no network, no cost. They exercise the real loop — context navigation
over the knowledge base, the maker/checker debate, the deterministic validator, the verdict — with
**scripted stand-ins for the agents' answers**. Every scripted invocation prints a banner saying so,
because a scripted run that reads like a model run would be worse than having no offline mode at
all. What this shows is that the loop closes and the gate bites; it does not show how well a given
model would propose or judge.

**1 — Look at the knowledge base.** It is curated markdown, not a black box:

```bash
ls shared/examples/bygg-energi-mikro/
```

**2 — Watch the learning loop close.** Two runs separated by an expert approval, with the second
demonstrably informed by the first:

```bash
uv run python -m portfolio_optimiser.simulation   # or: uv run portfolio-optimiser-demo
```

Each run shows the refinement step: the proposer's first claim is falsified by the deterministic
validator, and the corrected claim validates. Between the runs, knowledge travels back on **both
feedback timescales, each carrying its own marker** so neither can stand in for the other: the
persona's approved verdict is promoted into the file-backed wiki (promote → re-seed → fold), and a
second verdict — an operations note an expert drops into an inbox folder *after* the run, the way a
reviewer would days later — is merged from disk into the next run's store before its hypothesis is
formed (write file → merge → fold). Both markers are present in Run B's prompt and absent from
Run A's. Nothing crosses in memory.

The run is **anchored**: the demo copies the example knowledge base and adds a `cost-baseline.json`
— the project's actual cost lines — so the validator's first stage reconciles every cost line a
proposal claims against them, before the solver runs. The declared baseline is printed at the top,
because it is the one property the rest of the output looks identical with or without. Those
numbers are synthetic, derived from the scripted replies themselves; a knowledge base that ships
its own baseline replaces them through the same seam, with no code change.

**3 — Run the loop over a knowledge base, with answers you supply.** Write the stand-in replies,
then point the CLI at the bundle:

```bash
cat > replies.json <<'JSON'
{
  "proposer": "{\"measure\":\"LED-retrofit\",\"affected_items\":[{\"code\":\"ENERGI-TOTAL-EL\",\"quantity\":300000,\"unit_cost\":1.0}],\"claimed_saving_nok\":30000}",
  "checker": "The numbers are within a feasible range. VERDICT: APPROVE"
}
JSON

uv run python -m portfolio_optimiser.run BYGG-KONTOR-NORD \
  --docs-dir shared/examples/bygg-energi-mikro \
  --bundle-dir shared/examples/bygg-energi-mikro \
  --scripted-replies replies.json
```

Ends in `ValidatedProposal`. Swap `--bundle-dir`/`--docs-dir` for your own bundle to run it over
your own data — that is the point of this door, and the reason it is not the same thing as step 2.
Your bundle needs one file the ingest layer does not write for you: `validator-input.json`, the
candidate the deterministic validator judges (a bundle without it is refused, by design — see
[`docs/extending.md`](docs/extending.md)). Copy the shape from
`shared/examples/bygg-energi-mikro/`.

**4 — Watch it say no.** Raise `claimed_saving_nok` to `250000` in `replies.json` and run the same
command again. The outcome becomes `Rejection`: the deterministic validator refuses a saving the
project's own numbers cannot support, no matter how confidently the proposer asserted it. This is
the part of the method that carries the weight — the agents propose, and something that cannot be
argued with decides.

Read that summary line carefully: `Rejection (verdict id=…, decision=approved)` is not a
contradiction. `Rejection` is the **validator's** outcome, while `decision=` echoes the
**human's** recorded verdict — here the `--decision` default, since nobody reviewed this run.
The two are deliberately separate: a machine gate that blocks, and a human judgement that
approves, are different questions and are never collapsed into one field.

**5 — See what it would cost with a real model**, before spending anything:

```bash
uv run python -m portfolio_optimiser.costsim --projects 4 --profile local
```

Modelled upper bounds per role and model, with the source of each price quoted. `--profile local`
prices the free local backend; the estimate is a ceiling, not a bill.

**6 — Run the whole portfolio, and watch the gate anchor to each project separately.** The same
flag works across every bundled reference project at once:

```bash
cat > replies.json <<'JSON'
{
  "proposer": "{\"measure\":\"scope_reduction\",\"affected_items\":[{\"code\":\"01.1\",\"quantity\":1,\"unit_cost\":850000}],\"claimed_saving_nok\":40000}",
  "checker": "Rigging and site operations can absorb this reduction. VERDICT: APPROVE"
}
JSON

uv run python -m portfolio_optimiser.run --portfolio --scripted-replies replies.json
```

One `ValidatedProposal`, three `Rejection`. All four reference projects carry a cost line `01.1`,
but at four different amounts — so a claim stated against one project's estimate is refused for the
other three. Nothing about the proposal changed between them; what changed is the project's own
numbers, which is the whole point of anchoring the gate to a cost baseline rather than to the
proposal's internal arithmetic.

The four lines quote the same `verdict id=`. That is not a bug: a verdict is keyed on the
*candidate* it judges, not on the project it was judged in, so an identical proposal mints an
identical id by design — that key is how a later run finds the earlier judgement.

A portfolio pass reports what happened to every project. Projects that raised are printed to
stderr with their error, and the command exits non-zero; the projects that completed still print
their outcome, because one dead project must not discard the rest of the pass. A pass stopped
because a savings goal was reached says so too.

The global token cap is reported separately from a goal stop — running out of budget and hitting
your target are not the same event — but note that the cap itself has no command-line flag yet:
only a library caller can install one, so that line is unreachable from the CLI today.

**7 — Report what has actually been realized:**

```bash
uv run python -m portfolio_optimiser.run --report --ledger savings-ledger.json
```

This reads a savings ledger and prints per-project and portfolio totals with each entry's
provenance. It makes no model calls and changes nothing.

**The ledger is an input, and the framework will not write it for you.** It records savings that
were *actually realized* — a contract was changed, an invoice came in lower — which is a fact about
the world, not a conclusion the system is entitled to draw from its own proposals. A validated
proposal is a claim; a ledger entry is a result. Keeping them apart is deliberate, and it is why no
command here produces a ledger as a side effect. Run `--report` before creating one and it says so
plainly (`run report refused: savings ledger not found`).

You write entries when the saving materializes:

```python
from portfolio_optimiser.ledger import LedgerEntry, SavingsLedger, to_ore

ledger = SavingsLedger()
ledger.add_realized(
    LedgerEntry(
        project_id="FV42-GSV-E1",
        dimension="rigg",
        candidate_identity="33fba649cade8529",
        amount_ore=to_ore(40000),
        verdict_id="33fba649cade8529",
        provenance="expert Kari Nordmann, 2026-08-05, realized via contract amendment",
    )
)
ledger.save("savings-ledger.json")
```

Amounts are held in øre as integers, and `to_ore` is the only conversion — money is quantized once,
per amount, before anything is summed.

> `--live-dry-run` is a different, narrower drill: it builds contracts, clients and budget against
> your own configuration and **stops before the first model call**. It verifies the setup; it does
> not run the loop. `--scripted-replies` runs the whole loop. The two are mutually exclusive and
> passing both is refused rather than one silently winning.

## Non-goals

- **Not a compliance product.** It ships the technical prerequisites — local-only operation,
  provenance on every proposal, no silent data egress — and stops there. Processing purpose, DPIA
  and risk assessment stay with the deploying organization.
- **Not a portfolio-level reallocator.** It finds savings *inside* each project. Moving budget
  between projects, ranking projects against one another and portfolio governance sit above the
  method and are out of scope.
- **Not autonomous decision-making.** The deterministic validator can only block; approving a
  measure is a domain expert's call (human-in-the-loop), and the framework implements nothing on
  the agents' say-so.
- **Not a turnkey vertical solution.** The aim is a generic core with explicit extension points
  (data sources, cost models, personas) — not the last 10% of any one domain.
- **Not a model benchmark.** The end-to-end proof runs offline against a scripted stand-in client:
  it shows that the loop closes, not how well a given LLM proposes or judges.

> **Status:** the full 8-step agentic loop is wired and proven with load-bearing tests, and the
> end-to-end proof is an **offline simulation** with a scripted stand-in client — no live-model
> run yet. The **ingest layer** (real data sources) is implemented — file/CSV and SQL on both
> stacks with bit-identical golden extractions from the shared spec, plus HTTP as a MAF-only
> demonstrated extension point against a local mock — but exercised only against committed
> fixtures: no bundle has yet been materialized from a live source. Ingested content passes a
> **content gate** (`materialize_gated`) that scans every generated concept with
> [`llm-ingestion-guard`](https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security)
> before any of it reaches the bundle; a refused run writes nothing. A sibling implementation of
> the same method on the **Claude Agents SDK** is built in parallel from the same shared spec.

> **Disclaimer — technical framework only.** Deploying organizations own their processing
> purposes and assessments (DPIA, risk/ROS, security review). The framework ships the technical
> prerequisites — local-only mode, provenance, no silent data egress — but makes no compliance
> guarantees.

## Built on an LLM wiki: Karpathy's idea, Google's format

The knowledge architecture is the heart of the project, and it is deliberately not ours:

- **The idea** is Andrej Karpathy's **"LLM wiki"**: instead of pointing a model at documents
  written for people, you curate a small, versioned body of knowledge written *for the model
  to read* — concept files, explicit structure, explicit links.
- **The format** is Google Cloud's **[Open Knowledge Format (OKF)](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)**
  (open spec, v0.1), which formalizes that pattern: a knowledge **bundle** is a directory of
  markdown files with YAML frontmatter (one required field, `type`), a reserved `index.md`
  entry point, and intra-bundle cross-links forming an emergent graph. Custom frontmatter
  fields are allowed and must be preserved — which is exactly where this project's own layers
  (expert verdicts, ingest provenance) live.

Because OKF is open and vendor-neutral, the *same* bundles are consumed unchanged by both
reference implementations (MAF and the Claude Agents SDK sibling) — the knowledge outlives any
particular agent stack.

**Not RAG.** Agents read a bundle by **navigating** it — `index.md` first, then its
cross-links, with progressive disclosure — never by keyword retrieval or stuffing the whole
bundle into a prompt. Query-time retrieval against the bundle is explicitly forbidden by the
method spec: it would leak the verdict layer around the learning gate.

## AI-first, humans on top

A traditional wiki is built for *people* — optimized for humans finding and reading
information, with machine access bolted on afterwards. This project inverts that order, and is
a concrete example of what that looks like:

- The wiki (the OKF bundle) is written **for the model**: it is the agent's working memory and
  the substrate the learning loop reads from and promotes into.
- The **human affordances are layers on top**: experts judge outcomes by dropping a plain JSON
  verdict file in an inbox folder; an explicit, fail-closed **promotion gate** is the only path
  by which an approved verdict becomes wiki knowledge; reports and reviews are rendered *from*
  the machine-readable layers.

Humans stay decisive — nothing enters the wiki without an approval — but the primary reader of
every file is the model, not a person browsing.

## How it works

One run, one project, eight steps — with the learning loop closing across runs:

1. **Understand** — navigate the project's OKF bundle; fold the candidate's *prior expert
   verdicts* into the hypothesis prompt (ExpeL-style, retrieved structurally, never by text).
2. **Hypothesise** — one typed candidate measure (strict IR, fail-fast schema).
3. **Debate** — a maker-checker pair argues the reasoning (round-capped).
4. **Validate** — two falsifiers on the same candidate: the **deterministic validator** gates
   the numbers (blocking, never optional) and the **checker** gates the reasoning. The validator
   is anchored to the project's declared cost baseline, so a proposal cannot invent the cost
   lines it claims to save against.
5. **Refine** — a rejected attempt retries *informed* by the rejection reason, under hard
   attempt and token caps. Unbounded loops are forbidden everywhere. The falsifications that
   informed a later attempt are surfaced on the result (`RunResult.refinements`), so what the
   run corrected in response to is inspectable, not just what it ended up with.
6. **Propose or discard** — a validated proposal with risk percentiles, or a typed rejection.
7. **Expert feedback** — days later, an expert drops a verdict file in an inbox folder; a
   later run picks it up. Fully resumable; no live session assumed.
8. **Promote** — an *approved* verdict is lifted into the wiki as a `type: verdict` concept
   file, navigable by the next run. The gate is fail-closed: raw agent output never
   self-promotes.

Every proposal carries provenance (citations into the bundle, model, validator decision, token
usage). Every seam above is protected by a **load-bearing test** — a test designed to *fail*
when the seam is detached, so the loop cannot silently degrade into theater.

## How it is set up

- **One shared, framework-neutral core** ([`shared/`](shared/README.md), a git subtree of
  [`portfolio-optimiser-commons`](https://git.fromaitochitta.com/open/portfolio-optimiser-commons)):
  the business concept, the normative [method spec](shared/method-spec.md) and
  [ingest spec](shared/ingest-spec.md), the expert-reviewer persona as an Agent Skill, and an
  example bundle with a golden suite as the only ground truth. Both stacks implement from the
  spec alone.
- **Per project: one OKF bundle** — the bundled examples are hand-curated; the ingest layer that
  materializes a bundle from a source (file catalogues/CSV + SQL, HTTP as a MAF-only demonstrated
  extension point) via a deterministic, schema-validated manifest that runs *before* the loop is
  implemented and exercised against committed fixtures — no bundle has yet been materialized from
  a live source.
- **Run:** the `run.py` CLI has **three modes** — a documented partition, since one invocation
  cannot exercise every flag:
  - **Single-project** — `PROJECT_ID --docs-dir <dir>`, plus optional `--bundle-dir`,
    `--verdict-dir`, `--outbox-dir` (which requires `--run-id`), `--dimension-config`,
    `--semantic-retrieval`, `--decision`/`--rationale`, `--live-dry-run`, and
    `--scripted-replies <file>` (the offline whole-loop door — see
    [Walk the whole chain offline](#walk-the-whole-chain-offline); mutually exclusive with
    `--live-dry-run`, which stops before the first model call rather than answering it).
  - **Portfolio** — `--portfolio`, plus optional `--goals`, `--ledger`, `--dimension-config`,
    `--semantic-retrieval`; it stops early and prints a `goal reached: …` line when the
    accumulated ledger meets a goal.
  - **Value report (S5.4, read-only)** — `--report --ledger <file>` rolls up the ledger's realized
    savings to stdout: per-project totals, the portfolio total, flagged cross-dimension overlaps
    (each counted once), and per-entry provenance. Add `--json` for deterministic JSON instead of
    the human table. It makes **no model calls** and is mode-exclusive — only `--ledger`/`--json`
    are permitted alongside `--report`; `--report` requires `--ledger`, and a stray `--json`
    without `--report` is refused (rc 1, never silently ignored).

  ```bash
  # Single-project, offline drill (builds contracts + clients, stops before the first model call):
  uv run python -m portfolio_optimiser.run FV42-GSV-E1 --docs-dir <docs> --bundle-dir <bundle> --live-dry-run
  # Portfolio run with a savings goal checked against an accumulated ledger:
  uv run python -m portfolio_optimiser.run --portfolio --goals goals.json --ledger ledger.json
  # Read-only value report over an accumulated ledger (human table; add --json for JSON):
  uv run python -m portfolio_optimiser.run --report --ledger ledger.json
  ```

  `--semantic-retrieval` (S3.1) is an **opt-in** ranking change, **off by default**. Off, prior
  verdicts are ranked exactly as before: a structural score over the affected cost-code set,
  measure type and magnitude bucket, with surface text deliberately excluded. On, that score is
  blended with a cosine term over the *same* structural triple, which lets a prior verdict on a
  *different* cost-code set outrank one that ties structurally.

  **What this ships is the seam, not better retrieval.** The bundled `FakeEmbedder` is a
  deterministic sha256 projection carrying no semantics, so over a structural tie the resulting
  order is deterministic but arbitrary. Retrieval *quality* depends entirely on injecting a real
  embedder — `--embedder-config` selects one from a closed registry (never an import path; a
  config file can never name arbitrary code to load), and `docs/extending.md` documents the
  `Embedder` protocol. The embedding excludes `description`, matching the structural score and the
  verdict-id hash, so a flag-on run reads no surface text either.

  The flag is accepted in both run modes, but in single-project mode it **requires** `--bundle-dir`
  and `--verdict-dir`: without them it cannot take effect, and the run is refused rather than
  silently ignoring the flag. Nothing about a flag-off run changes, and no savings claim depends
  on it.

  A bundle may hold verdicts about **several candidates**, while its `validator-input.json`
  describes only one. A `type: verdict` file therefore may declare its own retrieval key in
  frontmatter — `affected_codes`, `measure_type`, `claimed_saving_nok` — and is keyed on that;
  omit them and it falls back to the bundle's candidate, exactly as before. The three are **all or
  nothing**: a partial declaration is refused rather than merged with the bundle candidate, since
  the merge would produce a key belonging to neither. `promote_verdict` writes all three, so a
  promoted verdict about one candidate never surfaces for another.

  A bundle may also ship a **`cost-baseline.json`** — the project's actual cost lines,
  `{code: {quantity, unit_cost}}` — and when it does, the deterministic validator reconciles every
  affected item of a proposal against it before anything else runs. A cost code the project does not
  have is rejected, and so is a real code carrying a quantity or unit cost outside the configured
  tolerance (5% by default, relative to the baseline value). Without it, every stage of the gate
  reasons only about numbers the proposal supplied itself, so an internally consistent hallucination
  passes. The reconciliation validates; it never repairs a proposal into the baseline. A bundle that
  ships no baseline is simply un-anchored and runs exactly as before, while a baseline that is
  present but malformed is an error rather than a silent fall-back to un-anchored. On the
  reference-domain (non-bundle) path the project's own cost items are the baseline, so those runs
  are always anchored.

  The **prior-verdict fold — the learning step — happens only on the `--bundle-dir` path**; a
  plain `--docs-dir`-only run is single-shot (no fold). `--decision`/`--rationale` apply to the
  single-project path only and are inert in portfolio mode. **`--outbox-dir` must differ from
  `--verdict-dir`**: writing the raw outbox into a folder later read as an inbox would re-ingest
  raw agent output past the promotion gate (self-contamination) — documented here, deliberately
  not CLI-enforced. Stop criteria and budget caps are required at startup. Try the offline
  end-to-end proof (no model, no network): `uv run python -m portfolio_optimiser.simulation`.

- **A global token cap across the whole portfolio, enforced before the call.** Per-run caps alone
  let N projects cost N times that with no ceiling over the pass. Pass a
  `PortfolioMeter(PortfolioBudget(max_total_tokens=…, max_tokens_per_run=…))` to `run_portfolio`
  and one ledger bounds the entire pass — and, seeded from `budget.read_spend`, a *series* of
  passes. It bites in three places: a remainder that cannot fund one run refuses the pass at
  startup (`BudgetRefused`); a project that cannot be funded is **never started**, stopping the
  pass structurally (`budget_stop`, completed runs preserved); and a chat call the remainder
  cannot pay for is **refused rather than made** (the post-charge check remains, since real usage
  is only knowable after the response). Spend persists via `budget.write_spend`, which takes an
  explicit stamp and no wall-clock default, so the file is byte-deterministic. Python API only —
  not yet exposed on the CLI.

## What this enables

The reference case is portfolio cost review (the example bundle is a building-energy measure),
but the architecture is designed to generalize to any setting with the same shape — candidate
measures inside independent projects, numbers a deterministic tool can check, and judgement
only an expert has:

- **Portfolio reviews** — cost savings, energy efficiency, maintenance and procurement
  measures, proposed per project and validated against the project's own data.
- **Compounding organizational memory** — approved expert verdicts become navigable knowledge;
  the next run's hypotheses start from what experts actually decided, including realization
  gaps no solver can compute.
- **Auditable AI** — an unbroken provenance chain from expert decision back through proposal,
  bundle file and text span, and (with ingest) to the source system, query, and timestamp.
- **Vendor-neutral knowledge** — the same bundles drive two different agent stacks; switching
  frameworks does not orphan the organization's curated knowledge.

## Docs

- [Bestille en kjøring](docs/bestille-en-kjoring.md) *(norsk)* — for the domain expert who
  COMMISSIONS a run: naming the approaches the run must evaluate (and/or asking the system for its
  own), stating what the run is for, and reading the announcement it prints before spending
  anything and the settlement it prints afterwards. The commission directs what is *evaluated*,
  never what is *approved*.
- [Ekspert-svar](docs/ekspert-svar.md) *(norsk)* — for the domain expert who has to deliver the
  verdict: where a judgement goes, what an approval, an approval-with-correction and a rejection
  actually look like, and paste-ready examples of each. Marked throughout as AI-authored and not
  verified professional judgement.
- [Building a knowledge base](docs/knowledge-base-recipe.md) — the team recipe (technical +
  domain expert) for curating a bundle, with the honest expectation that a good base takes 1–2
  weeks of dedicated work.
- [Target picture](docs/plan/2026-06-26-maalbilde-agentic-loop.md) — the agentic loop + OKF
  knowledge architecture (north star).
- [Prior-art & platform research](docs/research/2026-06-23-prior-art-platform.md) (incl.
  implementation register §15).
- [Ingest target picture](docs/plan/2026-07-03-maalbilde-ingest-lag.md) — connectors and the
  ingest layer (frozen 2026-07-03).

## Stack & develop

Python ≥3.10 · MAF via the split GA packages (see `pyproject.toml`) · `uv`. Backend profiles:
Azure/Foundry (full) + local (fallback).

```bash
uv sync
uv run pytest
uv run ruff check .
```