open/portfolio-optimiser
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity Actions

docs(readme): refocus on LLM-wiki foundation, AI-first framing, and use cases

Browse source 
Rewritten around what the project is built on — Karpathy's LLM-wiki idea
as formalized by Google Cloud's Open Knowledge Format (spec-linked) — how
the 8-step loop works and is set up (shared commons core, per-project
bundles, specified ingest layer), and what the architecture enables.
Adds the AI-first framing: the wiki is written for the model, human
affordances (verdict inbox, promotion gate) are layers on top — the
inverse of a human-first wiki. Honest status kept: offline proof only,
ingest specified but not implemented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AaQCFnfsh3tfq1VfzdJpoi
This commit is contained in:
Kjell Tore Guttormsen 2026-07-03 14:45:05 +02:00
commit c33795db3f
1 changed files with 119 additions and 68 deletions
Download patch file Download diff file Expand all files Collapse all files

159
README.md
View file

@ -1,78 +1,129 @@
# portfolio-optimiser
Generic, open framework on **Microsoft Agent Framework (MAF)** for finding cost-savings / efficiency proposals *within* each project of a portfolio of independent projects. Multiple agents collaborate to generate candidate proposals; a mandatory deterministic validator (solver + Monte Carlo) decides the numbers; domain experts review via human-in-the-loop, and the system learns from their verdicts.
A generic, open framework — built on **Microsoft Agent Framework (MAF)** — that finds cost
savings *inside* each project of a portfolio of independent projects. A swarm of agents
generates candidate measures; a **mandatory deterministic validator** (solver + Monte Carlo)
decides the numbers; domain experts judge the outcomes (human-in-the-loop); and the system
**learns from their verdicts** across runs.
> **Status:** Early development. The deterministic backbone is solid; the agentic learning loop is being wired one load-bearing seam at a time — **Steps 1, 3/4, 5, 7 and 8 are wired** (OKF-navigated context, checker gate, informed refinement, the async file feedback loop, and gated wiki promotion; see below). Not yet end-to-end usable.
> **Status:** the full 8-step agentic loop is wired and proven with load-bearing tests, and the
> end-to-end proof is an **offline simulation** with a scripted stand-in client — no live-model
> run yet. The ingest layer (real data sources) is specified but not yet implemented. A sibling
> implementation of the same method on the **Claude Agents SDK** is built in parallel from the
> same shared spec.
> **Disclaimer — technical framework only.** This project is a *technical framework*. Organizations that deploy it are themselves responsible for ensuring a valid processing purpose and for any required assessments (DPIA, risk/ROS, security reviews, etc.). The framework ships technical affordances (local-only mode, provenance/audit logging, no silent data egress) to *enable* compliant use, but makes no compliance guarantees.
> **Disclaimer — technical framework only.** Deploying organizations own their processing
> purposes and assessments (DPIA, risk/ROS, security review). The framework ships the technical
> prerequisites — local-only mode, provenance, no silent data egress — but makes no compliance
> guarantees.
## Design philosophy
## Built on an LLM wiki: Karpathy's idea, Google's format
The result will never fit any single customer 100%. The goal is a **~90% genuinely generic core plus clear extension points**, so competent people can configure the last mile per customer. We deliberately do not chase the final 10%.
The knowledge architecture is the heart of the project, and it is deliberately not ours:
## Agentic loop — wiring status
- **The idea** is Andrej Karpathy's **"LLM wiki"**: instead of pointing a model at documents
written for people, you curate a small, versioned body of knowledge written *for the model
to read* — concept files, explicit structure, explicit links.
- **The format** is Google Cloud's **[Open Knowledge Format (OKF)](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)**
(open spec, v0.1), which formalizes that pattern: a knowledge **bundle** is a directory of
markdown files with YAML frontmatter (one required field, `type`), a reserved `index.md`
entry point, and intra-bundle cross-links forming an emergent graph. Custom frontmatter
fields are allowed and must be preserved — which is exactly where this project's own layers
(expert verdicts, ingest provenance) live.
The mandatory deterministic backbone (validator + budget meter + provenance) is solid and load-bearing. The agentic learning loop (see the [target picture](docs/plan/2026-06-26-maalbilde-agentic-loop.md) §11) is wired one seam at a time:
Because OKF is open and vendor-neutral, the *same* bundles are consumed unchanged by both
reference implementations (MAF and the Claude Agents SDK sibling) — the knowledge outlives any
particular agent stack.
- **Step 1 — OKF context → hypothesis (wired).** `run_project(..., bundle_dir=...)` wires the first agentic seam in two halves:
- **Context by navigation, not stuffing.** The agent read-context is built by **navigating** the project's [OKF bundle](shared/examples/bygg-energi-mikro/) (`index.md` + frontmatter + cross-links, progressive disclosure) — never keyword chunk-stuffing (target picture §2/§4).
- **Verdict layer gated out of context.** The `type: verdict` layer is excluded from that context; the candidate's prior expert verdicts reach the hypothesis prompt *only* through the gated ExpeL fold (folded in *before* generation), so a prior verdict provably — and exclusively — influences the next hypothesis.
- Two load-bearing tests fail when the seam is detached: the realization signal must reach the prompt via the fold (`tests/test_step1_expel_loadbearing.py`), and must never leak via context (`tests/test_okf.py::test_bundle_context_excludes_verdict_layer`).
- **Steps 3/4 — checker gates the reasoning (wired).** Two falsifiers now act on the same candidate: the deterministic validator gates the **numbers** (blocking, as before), and the maker-checker's checker gates the **reasoning** (target picture §2/§6). The checker ends its turn with a `VERDICT: APPROVE` / `VERDICT: REJECT — <reason>` line; an explicit reject blocks an otherwise-validated proposal (`run_project` surfaces both debate participants via `output_from=agents` and overrides the outcome to a checker-sourced `Rejection`). The gate is opt-in-reject (fail-open on a missing marker), and `provenance.validator_decision` stays honest — it reflects the validator only, never the checker. Load-bearing: `tests/test_checker_gate_loadbearing.py` goes red on *either* detach (revert `output_from`, or drop the override).
- **Step 5 — informed refinement (wired).** The proposer's bounded retry is no longer blind: the validator's *previous* `Rejection.reason` is fed into the next attempt's prompt (`generate.py` `generate_via_llm` → `_build_messages(prior_rejection=...)`), so the model corrects against the falsification instead of re-answering identically (target picture §5/§7). It carries only the most-recent reason (never an accumulated history, never the rejected proposal JSON) and runs under the *existing* `max_attempts` + token-meter cap — no new loop, so "improve until good enough" without a ceiling stays impossible. The only per-attempt falsifier here is the validator; seeding generation with the checker's critique is a run-level, separately-scoped concern and is deliberately not done here. Load-bearing: `tests/test_step5_refine_loadbearing.py` goes red when the reason is detached from the prompt (the outcome never flips *and* the verbatim-reason assertion fails), with a bounded control proving the loop still stops at `max_attempts`.
- **Step 7 — async file feedback loop (wired).** `run_project(..., verdict_dir=...)` adds the long feedback timescale (target picture §3/§7): an expert/persona drops a verdict file (plain JSON — the raw output layer, §10 R2) into an inbox folder *after* a run, and a separate, later run ingests it — merged into the store *before* the Step-1 fold — so a verdict that landed out of band reaches the next hypothesis. The loop is fully resumable across runs separated in time; no live session is assumed. The system *reads* the folder, the expert/persona *writes* it (§3 role split), so `run_project` deliberately does **not** persist its own captured verdict back (that is the outbox / Step-8 concern). Ingestion is tolerant (a missing folder, foreign or half-written files are skipped, not raised) and **merges** (never replaces), preserving `run_portfolio`'s cross-project store. Reachable from the CLI via `--bundle-dir --verdict-dir`. Load-bearing: `tests/test_step7_async_loop_loadbearing.py` — a verdict dropped after run A must reach run B's prompt (run B uses a *fresh* store, so the transfer is the file loop, not in-memory carryover), with an empty-inbox control proving causality.
- **Step 8 — gated wiki promotion (wired).** When an expert/persona **approves** an outcome, `verdicts.promote_verdict` lifts it from the raw output layer into the context layer (the OKF bundle) as a `type: verdict` concept file, navigable by the next run's `seed_store_from_bundle` (target picture §3/§6/§7). The **gate** is fail-closed: a verdict whose decision is not an approval raises `PromotionRefused` and writes/links nothing — only human/persona-approved knowledge enters the wiki, never raw agent output (self-contamination). The promotion is provenance-stamped (who approved / which experiment / when — `timestamp` is a required keyword, no wall-clock default). The OKF writer lives in `okf.py` and stays pure stdlib (D7-portable, MAF-free). **R4 = optional + gated:** `promote_verdict` is a public opt-in primitive, deliberately **not** wired into `run_project` (mirrors `write_verdict` — the system reads context; the gate/persona promotes). Two honesty limits: the promoted file is *minimal* (it carries the learning signal only as `description`/body prose — it does not reproduce the hand-authored seed's structured `realization_rate` etc.), and because the verdict id is the learning key, two approvals about the same candidate share a filename (last-write-wins, like `write_verdict`) — the wiki grows one curated file per distinct candidate, not per verdict event. Load-bearing trio (`tests/test_step8_promotion_loadbearing.py`): the gate refuses a non-approved verdict (red if the gate is removed), the approved verdict is navigable (red if `link_in_index` is detached), and the promoted signal stays out of `bundle_context` — reaching a prompt only via the gated ExpeL fold (red if a descriptive index label leaks it into the read-context).
**Not RAG.** Agents read a bundle by **navigating** it — `index.md` first, then its
cross-links, with progressive disclosure — never by keyword retrieval or stuffing the whole
bundle into a prompt. Query-time retrieval against the bundle is explicitly forbidden by the
method spec: it would leak the verdict layer around the learning gate.
**Verdict conflict semantics (chosen, minimal).** The in-memory `VerdictStore` is **first-write-wins per verdict id** (a re-ingested duplicate is dropped, so repeated inbox merges are idempotent), while the disk layers — `write_verdict` (inbox authoring) and `promote_verdict` (wiki) — are **last-write-wins per file**. Ids are content-hashes of the candidate *features*, so "same id" means "same candidate measure", not "same verdict event". A full verdict-conflict taxonomy (B10: rejection categories + a rule for conflicting expert verdicts) is deliberately deferred until real domain experts produce conflicting verdicts.
## AI-first, humans on top
## Offline simulation — the end-to-end proof
A traditional wiki is built for *people* — optimized for humans finding and reading
information, with machine access bolted on afterwards. This project inverts that order, and is
a concrete example of what that looks like:
The loop is proven end to end **offline**, with no real model. `portfolio_optimiser.simulation`
drives `run_project` with a **scripted** synthetic chat client (network-free) across two runs
separated by a promotion, and shows the learning loop close: Run A produces a validated,
persona-approved verdict carrying a realization marker absent from the bundle; `promote_verdict`
lifts it into the OKF wiki; a re-seed picks it up; **Run B's hypothesis prompt then carries the
marker** (an empty-wiki control on Run A proves causality). Run it:
- The wiki (the OKF bundle) is written **for the model**: it is the agent's working memory and
the substrate the learning loop reads from and promotes into.
- The **human affordances are layers on top**: experts judge outcomes by dropping a plain JSON
verdict file in an inbox folder; an explicit, fail-closed **promotion gate** is the only path
by which an approved verdict becomes wiki knowledge; reports and reviews are rendered *from*
the machine-readable layers.
```
uv run python -m portfolio_optimiser.simulation
```
Humans stay decisive — nothing enters the wiki without an approval — but the primary reader of
every file is the model, not a person browsing.
This is a deliberate, cost-driven substitution for a real-model run (target picture §11 step 8):
it proves the plumbing, the deterministic spine, and that the **learning dataflow closes**. It does
**not** prove that a live LLM would *produce* the proposal or verdict — those are scripted stand-ins
for the swarm and the expert persona (honesty per §1). The scripted client is MAF-side scaffolding,
not part of the framework-neutral `shared/` core. Load-bearing: `tests/test_simulation_loadbearing.py`
goes red the moment promotion is detached (the marker never crosses into Run B).
## How it works
## Shared expert-reviewer persona (§8)
One run, one project, eight steps — with the learning loop closing across runs:
The expert reviewer is a **framework-neutral Agent Skill** in [`shared/skills/expert-reviewer/`](shared/skills/expert-reviewer/)
— a `SKILL.md` persona prompt (the energy-advisor / M&V role, the realization-gap methodology the
validator cannot compute) plus a canonical `references/example-verdict.json`. It is the shared
artifact both reference implementations consume to instantiate the reviewer; `shared/` stays pure
data, so the MAF side reads it via `portfolio_optimiser.persona.load_persona_example` and the
Claude-SDK sibling reads the same JSON with its own loader. This **de-stubs the simulation**: its
persona judgement (decision + rationale + traced marker) is now sourced from the artifact at call
time, not a hardcoded literal — so the shared persona is genuinely consumed and cannot rot silently.
The persona's `decision` is binary (`approved` / `rejected`, the feedback contract the run path
accepts); the realization correction lives in the rationale prose. Load-bearing trio
(`tests/test_persona_skill_loadbearing.py`): structure + framework-neutrality (red on a framework
import), the example is valid pipeline input (red on schema/contract drift), and the simulation's
marker follows the artifact file (red the moment the persona is re-inlined).
1. **Understand** — navigate the project's OKF bundle; fold the candidate's *prior expert
verdicts* into the hypothesis prompt (ExpeL-style, retrieved structurally, never by text).
2. **Hypothesise** — one typed candidate measure (strict IR, fail-fast schema).
3. **Debate** — a maker-checker pair argues the reasoning (round-capped).
4. **Validate** — two falsifiers on the same candidate: the **deterministic validator** gates
the numbers (blocking, never optional) and the **checker** gates the reasoning.
5. **Refine** — a rejected attempt retries *informed* by the rejection reason, under hard
attempt and token caps. Unbounded loops are forbidden everywhere.
6. **Propose or discard** — a validated proposal with risk percentiles, or a typed rejection.
7. **Expert feedback** — days later, an expert drops a verdict file in an inbox folder; a
later run picks it up. Fully resumable; no live session assumed.
8. **Promote** — an *approved* verdict is lifted into the wiki as a `type: verdict` concept
file, navigable by the next run. The gate is fail-closed: raw agent output never
self-promotes.
Every proposal carries provenance (citations into the bundle, model, validator decision, token
usage). Every seam above is protected by a **load-bearing test** — a test designed to *fail*
when the seam is detached, so the loop cannot silently degrade into theater.
## How it is set up
- **One shared, framework-neutral core** ([`shared/`](shared/), a git subtree of
[`portfolio-optimiser-commons`](https://git.fromaitochitta.com/ktg/portfolio-optimiser-commons)):
the business concept, the normative [method spec](shared/method-spec.md) and
[ingest spec](shared/ingest-spec.md), the expert-reviewer persona as an Agent Skill, and an
example bundle with a golden suite as the only ground truth. Both stacks implement from the
spec alone.
- **Per project: one OKF bundle** — hand-curated today; the specified ingest layer will
materialize bundles from real sources (file catalogues/CSV, SQL, HTTP as an extension point)
via a deterministic, schema-validated manifest that runs *before* the loop.
- **Run:** the CLI takes a bundle directory and a verdict inbox directory; stop criteria and
budget caps are required at startup. Try the offline end-to-end proof (no model, no network):
`uv run python -m portfolio_optimiser.simulation`
## What this enables
The reference case is portfolio cost review (the example bundle is a building-energy measure),
but the architecture is designed to generalize to any setting with the same shape — candidate
measures inside independent projects, numbers a deterministic tool can check, and judgement
only an expert has:
- **Portfolio reviews** — cost savings, energy efficiency, maintenance and procurement
measures, proposed per project and validated against the project's own data.
- **Compounding organizational memory** — approved expert verdicts become navigable knowledge;
the next run's hypotheses start from what experts actually decided, including realization
gaps no solver can compute.
- **Auditable AI** — an unbroken provenance chain from expert decision back through proposal,
bundle file and text span, and (with ingest) to the source system, query, and timestamp.
- **Vendor-neutral knowledge** — the same bundles drive two different agent stacks; switching
frameworks does not orphan the organization's curated knowledge.
## Docs
- [`docs/plan/2026-06-26-maalbilde-agentic-loop.md`](docs/plan/2026-06-26-maalbilde-agentic-loop.md) — target picture: the agentic cost-saving loop + OKF knowledge architecture (north star).
- [`docs/research/2026-06-23-prior-art-platform.md`](docs/research/2026-06-23-prior-art-platform.md) — prior-art & platform research (incl. implementation register §15).
- [`docs/plan/2026-06-23-incremental-plan.md`](docs/plan/2026-06-23-incremental-plan.md) — incremental delivery plan (deterministic backbone).
- [`shared/`](shared/) — framework-neutral shared core (concept + example OKF knowledge bundles + the expert-reviewer persona skill), reused unchanged by both reference implementations.
- [Target picture](docs/plan/2026-06-26-maalbilde-agentic-loop.md) — the agentic loop + OKF
knowledge architecture (north star).
- [Prior-art & platform research](docs/research/2026-06-23-prior-art-platform.md) (incl.
implementation register §15).
- [Ingest target picture](docs/plan/2026-07-03-maalbilde-ingest-lag.md) — connectors and the
ingest layer (frozen 2026-07-03).
## Stack
## Stack & develop
Python ≥3.10 · MAF via the split GA packages (`agent-framework-core`, `agent-framework-foundry`, `agent-framework-openai`, `agent-framework-orchestrations` — deliberately **not** the `agent-framework` meta-package; see `pyproject.toml`) · `uv`. Backend profiles: Azure/Foundry (full) + local (fallback).
## Develop
Python ≥3.10 · MAF via the split GA packages (see `pyproject.toml`) · `uv`. Backend profiles:
Azure/Foundry (full) + local (fallback).
```bash
uv sync