portfolio-optimiser/shared/method-spec.md

Method specification — portfolio cost-saving loop (framework-neutral)

Status: normative. This document specifies the method both reference implementations build: an agentic loop that finds cost savings inside each project of a portfolio, with a mandatory deterministic validator, expert judgement in the loop, and learning from the verdicts. It is written so the method can be implemented from this spec alone — without reverse-engineering any existing implementation. The prose is framework-neutral by rule: it never names a concrete agent toolkit or vendor stack, and a guard test keeps it that way.

The key words MUST, MUST NOT, SHOULD, and MAY are to be interpreted as in RFC 2119. Requirements are labelled normative; anything marked (reference) documents the reference implementation's concrete choice and is informative, not binding — except where the golden suite (§7) freezes it.

1. Scope and conformance

The method is the product: a swarm of agents generates candidate cost-saving measures for one project at a time; a deterministic validator decides the numbers (mandatory, blocking); domain experts judge the outcomes (human-in-the-loop); the system learns from the verdicts across runs. A conforming implementation:

1. implements the 8-step loop of §3 with the contracts of §4–§10;
2. reproduces the shared golden suite's decided outcomes (§7) on the shared example bundle;
3. proves every load-bearing seam with a test that FAILS when that seam is detached (§11).

Honesty rule (unwaivable): no artifact — code, docstring, README, or report — may claim more than the implementation does. Scripted stand-ins (synthetic clients, seeded verdicts) MUST be labelled as such wherever their output is presented.

Boundary: this is a purely technical framework. The deploying organisation owns all processing purposes and impact assessments; implementations provide only the technical prerequisites (local-only operation, provenance, no silent egress) and a disclaimer.

2. Terms and architecture

Three layers, strictly separated (the separation is load-bearing — see §3 Step 1 and §6):

• Context layer — one curated, version-controlled knowledge bundle per project (an "LLM wiki") in the open OKF format: a directory of markdown files with YAML frontmatter, one required field type, a reserved index.md entry point, and intra-bundle cross-links. This layer holds project documents, methodology, verified literature, constraints, and approved verdicts (type: verdict files). It is what runtime reading and experience retrieval draw from.
• Output layer — a run-scoped folder structure of raw results: proposals pending verdict, rejections with reasons, and raw verdict files (plain JSON, one per file). Append-heavy, never part of the wiki.
• Promotion gate — the only path from the output layer into the context layer (§6). Only expert-approved knowledge crosses it.

Other terms: the IR is the typed intermediate representation of a candidate measure (§7.1); the store is the in-memory collection of historical verdicts retrieval ranks over (§4.2); the fold is the injection of retrieved prior verdicts into the hypothesis prompt (§3 Step 1); the two falsifiers are the deterministic validator (numbers) and the debate checker (reasoning) — they judge the same candidate and are never conflated (§3 Step 4, §9).

3. The loop (normative)

Eight steps. Steps 1–6 happen within one run; steps 7–8 close the learning loop across runs separated in time.

Step 1 — Understand the context (navigate, never stuff)

The agent read-context for a project MUST be built by navigating its OKF bundle with progressive disclosure — never by stuffing the whole bundle (or keyword-retrieved chunks of it) into the prompt:

• Navigation starts at index.md and follows its intra-bundle markdown cross-links (](target.md)). Targets containing a path separator are out-of-bundle and MUST be skipped. Repeated links are de-duplicated; order is deterministic (index first, then links in first-seen order). (reference: the link pattern is \]\(([^)]+\.md)\))
• A missing index.md is an error (a bundle has no entry point without it). A broken or bundle-escaping cross-link MUST be tolerated — skipped, never raised (OKF robustness rule); path resolution MUST be boundary-checked against the bundle directory, fail-closed.
• Frontmatter is the leading ----delimited block, parsed line-oriented as key: value strings; the single required field is type; unknown fields MUST be preserved.
• The rendered read-context is the index body (the summary) followed by each non-index concept file as a ## {type}: {title} section; empty sections are dropped.
• type: verdict files MUST be excluded from the read-context. Prior verdicts reach the hypothesis prompt ONLY via the gated experience fold below — never via context rendering, and never via a query-time retrieval tool pointed at the bundle (which would re-leak the verdict layer).

Experience fold (ExpeL-style, the learning seam): before generation, the candidate's prior verdicts are retrieved from the store and folded into the hypothesis prompt:

• The retrieval query key is the bundle's candidate features, read from the IR projection (§7.1) — available before any proposal exists.
• Seeding: every type: verdict file in the bundle becomes a store entry keyed on those candidate features, with decision from frontmatter (default approved) and a rationale built from the description frontmatter plus, when present, the structured learning fields rendered as [realiseringsgrad={realization_rate}; forventet_faktisk_NOK= {expected_actual_saving_nok}].
• Ranking is structural, never textual — surface text MUST NOT contribute to similarity. Similarity is the weighted sum 0.60 × Jaccard(affected-code sets) + 0.25 × [measure-type equality] + 0.15 × [same magnitude bucket], with magnitude buckets [0, 1e5), [1e5, 5e5), [5e5, 1e6), [1e6, ∞) over the claimed saving and Jaccard of two empty sets defined as 1. Retrieval returns the top-k by similarity, ties broken by verdict id ascending (deterministic); k MUST be positive.
• The fold prepends the retrieved verdicts (id, decision, rationale per line) to the generation context. The rationale is the carrier of the learning signal — the fold is what lets an expert's realization-rate correction reach the next hypothesis.

Step 2 — Hypothesise (structured candidate generation)

The proposer model is asked for exactly one candidate measure as a JSON object for the IR (§7.1): project_id, measure, affected_items (list of {code, quantity, unit_cost}), claimed_saving_nok, optional assumptions. A reply that fails to parse into the typed IR MUST be retried (a blind parse-retry), never silently accepted or repaired downstream — bounded by the budget meter (§8). project_id MAY be defaulted from the project when the model omits it. IR schema invariants (§7.1) are enforced at construction, so a malformed proposal can never exist as a value.

Step 3 — Debate (maker-checker)

Candidate reasoning is debated by a two-role maker-checker pair — a proposer and a checker — alternating turns, with the debate's converged proposer output feeding generation (Step 2's context). Requirements:

• The debate MUST be round-capped (a hard maximum-rounds bound) with an additional turn-count termination safety net above it; an unbounded debate is forbidden (§8).
• Debate state MUST be fresh per run — no conversation state may survive from one project run into the next.
• The checker MUST be instructed to end its reply with exactly one verdict line: VERDICT: APPROVE if the reasoning holds, or VERDICT: REJECT - <short reason> if not.
• An optional synchronous in-run review gate on the checker MAY be enabled (the short feedback timescale, §3 Step 7); durable in-run checkpointing is NOT required.

Step 4 — Validate / falsify (two falsifiers, same candidate)

The deterministic validator gates the numbers — mandatory, blocking, never an optional plugin. It is the one endpoint-free judge that anchors the loop against swarm self-confirmation. Semantics (frozen by the golden suite, §7.2):

1. Schema invariants already hold (IR construction, §7.1).
2. Feasibility bound: the maximum feasible saving is capped at a policy fraction (0.30) of the affected items' total cost. (reference: computed with an LP solve whose closed form here is 0.30 × Σ quantity·unit_cost; a missing solver MUST escalate, never silently fall back.)
3. Risk simulation: seeded Monte Carlo over uncertain unit costs yields p10/p50/p90 percentiles of the feasible saving (§7.2 fixes the procedure).
4. Structural block: a claim above the optimistic feasible bound (p90) yields a rejection that is a distinct type from a validated proposal — carrying the claimed and feasible figures in its reason, and no percentiles — so it can never be consumed as validated.

The checker gates the reasoning (the second falsifier), parsed from the checker's LAST surfaced debate output, case-insensitively; the reject marker takes precedence and its trailing text is the reason. The gate is opt-in-reject (fail-open): VERDICT: APPROVE or a missing/unparseable marker never blocks — the validator remains the sole gate on such runs. An explicit checker REJECT MUST override an otherwise-validated outcome into a rejection (reason prefixed with the checker's reason). A validator rejection stands regardless of the checker. The two falsifiers MUST be recorded separately (§9): the provenance field mirrors ONLY the validator; the checker's decision (approve / reject / absent) is reported as its own result field. Either the checker actually gates, or the debate must not be called maker-checker.

Step 5 — Refine, informed and bounded

When the validator rejects, the next attempt MUST be informed by the falsification: the previous attempt's rejection reason is fed verbatim into the next attempt's prompt as a revision instruction. Constraints (all normative):

• Only the most recent rejection reason is carried — never an accumulated history (bounded prompt growth).
• Only the reason is carried — never the prior proposal JSON (the model must address the falsification, not parrot the rejected candidate).
• The refinement loop runs under the EXISTING caps — the attempt bound (max_attempts, reference default 3) and the token/round meter (§8). "Refine until good enough" without a cap is forbidden; no new loop may be introduced.
• The only per-attempt falsifier in this loop is the deterministic validator. Seeding generation with the checker's critique is a run-level concern outside this loop's scope.
• Attempt 1 MUST use the unchanged base prompt (the informed block appears only after a rejection).

Step 6 — Discard or propose

The run's outcome is either the validated proposal (with its percentiles) or a typed rejection with its reason — never a bare failure. Raw results (proposals pending verdict, rejections, captured verdict files) belong to the output layer (§2) as plain JSON — the raw layer deliberately does NOT use the wiki format (that is reserved for promoted knowledge).

Step 7 — Respond to feedback (two timescales)

• Short loop: an expert may review synchronously in-run (the optional Step-3 gate).
• Long loop (the async file inbox): days or weeks later, an expert (or, in simulation, a persona) drops a verdict file into an inbox folder; a separate, later run picks it up whenever it lands. The system MUST be fully resumable across runs separated in time — no live-session assumption. Contract in §5. Role split (unwaivable): the system READS the inbox; the expert writes it. A run MUST NOT persist its own captured verdict back into the inbox (writing is the authoring primitive's and the promotion gate's job).

In simulation a dedicated expert persona plays the human; in production a human uses the SAME folder interface. The persona is defined once, as a shared skill artifact (§4.3).

Step 8 — Promote approved knowledge (optional + gated)

When an expert APPROVES an outcome, it may be promoted from the raw output layer into the context layer as a type: verdict concept file, navigable by the next run's seeding (closing the learning loop through the file system). Promotion is an opt-in public primitive — it MUST NOT be wired into the run itself (the system reads context; the gate/persona promotes). Full gate semantics in §6.

4. The verdict contract

Three machine-readable shapes carry expert judgement. All three share the decision vocabulary rule: the run-path decision is binary — approved or rejected. The adjusted-approval case (the signature case in practice: the measure is worth doing but the modelled saving overstates the expected actual) is an approved decision whose rationale records the correction. approved_with_adjustment exists ONLY in bundle-seed frontmatter and in the promotion gate's accepted set (§6) — a run-path feedback contract MUST reject it.

4.1 Run-path feedback

The expert decision consumed by a run: decision ∈ {approved, rejected} (exactly two values) and a non-empty rationale string. Validated fail-fast at startup (§10).

4.2 The verdict file (raw output layer / inbox)

One verdict per JSON file, named {id}.json. Top-level fields (all required):

Field	Meaning
id	The learning-loop key (see minting, below). Read VERBATIM on load — never re-minted.
decision	The expert decision (§4 vocabulary).
rationale	Prose carrying the knowledge the validator cannot compute.
proposal_features	The structural features of the judged candidate (below).

proposal_features fields: affected_codes (emitted as a SORTED list), measure_type (string), claimed_saving_nok (number), description (string; surface text — deliberately excluded from both similarity ranking and id minting).

Id minting (normative): id is the first 16 hex characters of the SHA-256 of the canonical JSON {"affected_codes": <sorted list>, "claimed_saving_nok": <number>, "measure_type": <string>} with keys sorted and separators ,/: (no whitespace). The id therefore keys on the candidate measure, not the verdict event: a structurally identical proposal maps to the same id. Because raw JSON number formatting participates in the hash (30000 vs 30000.0 differ), a loaded verdict's id MUST be kept verbatim — re-minting could diverge.

Conflict semantics (chosen, documented): the in-memory store is FIRST-write-wins per id (repeated inbox merges are idempotent); the disk layers — inbox files and promoted wiki files — are LAST-write-wins per file. Two verdicts about the same candidate share an id and hence a filename. A full verdict-conflict taxonomy is deliberately deferred until real experts produce conflicting verdicts.

4.3 The persona example artifact

The expert-reviewer persona is a shared skill: a persona prompt plus one canonical example verdict JSON with fields decision, marker, and rationale. Requirements:

• decision MUST be a run-path value (§4.1); the canonical example is approved.
• marker MUST be a substring of rationale — it is the traceable payload (the realization rate) a simulation follows from the persona's judgement into a later run's prompt.
• Implementations MUST source the persona judgement from this artifact at call time (a loader, fail-fast on a missing/malformed file — it is required input), never from an inlined copy. The persona prompt's prose never names a concrete agent toolkit.

5. The inbox/outbox folder contract

The long feedback loop's folder interface (§3 Step 7). Normative:

• One verdict per file, {id}.json, shape per §4.2. The authoring primitive creates the directory if needed and writes deterministically (reference: sorted keys, 2-space indent).
• Tolerant load (the raw layer is written out of band; half-written or foreign files are realistic): a missing folder yields zero verdicts; files that are not .json, fail to parse, or lack a required top-level key (id, decision, rationale, proposal_features) are SKIPPED, never raised. Contrast: required inputs (the IR projection §7.1, the persona example §4.3) are fail-fast.
• Deterministic order: files are processed sorted by filename.
• Merge, never replace: a run ingests the inbox INTO its store (per-verdict add, first-write-wins per id) BEFORE the Step-1 fold, so a passed-in store's existing verdicts survive (cross-project threading) and repeated merges are idempotent.
• Role split: the system reads; the expert/persona writes (§3 Step 7).

6. The promotion gate

promote lifts one APPROVED verdict from the raw output layer into the OKF context layer. Normative semantics:

• Fail-closed: a verdict whose decision is not in {approved, approved_with_adjustment} MUST be refused with an error, writing and linking NOTHING. Only human/persona-approved knowledge enters the wiki — never raw agent output (self-contamination).
• Provenance-stamped: the promoted file records who approved, which experiment, and when. The timestamp MUST be an explicit required argument — no wall-clock default — so promotion is deterministic and reproducible.
• Minimal promoted file: frontmatter type: verdict, the decision, the verdict's rationale as the description field (the learning signal as prose), the verbatim verdict id, the provenance stamp, and tags. The promoted file MUST NOT reproduce a hand-authored seed's structured learning fields (realization_rate etc.) — the raw verdict model carries the signal only as rationale prose, and seeding (§3 Step 1) folds the description in.
• Navigability: the file is written into the bundle (path-safe, fail-closed against escaping names; filename promoted-verdict-{token}.md where the token is the id sanitised to [A-Za-z0-9._-], a degenerate token falling back to a content hash) and linked from index.md — navigation follows only index cross-links, so an unlinked file is unreachable. Linking MUST be idempotent (re-promotion never double-links).
• Neutral label: the index link label is FIXED and carries NO verdict signal. The index body flows verbatim into the rendered read-context, so a descriptive label (e.g. the rationale) would leak the learning signal around the gated fold.
• Per-candidate growth: ids key on candidate features (§4.2), so two approvals of the same candidate share a filename — last-write-wins; the wiki grows one curated verdict file per distinct candidate, not one per verdict event.
• (reference limitation) the index read-modify-write is not atomic — single-process use is assumed for the MVP.

7. Ground truth: IR projection and golden suite

The shared example bundle ships two JSON files that are the only ground truth ("fasit") for cross-implementation equivalence. Implementations MUST consume them unchanged.

7.1 The IR projection (validator-input.json)

The candidate measure projected into the typed cost-IR the validator consumes:

• project_id (string), measure (string), affected_items — a non-empty list of {code: string, quantity: number ≥ 0, unit_cost: number > 0} — claimed_saving_nok (number > 0), and assumptions: a map code → [low_unit_cost, high_unit_cost] giving the uncertainty band per cost code for the risk simulation (empty = degenerate, no spread).
• Construction-time invariant: the claimed saving MUST NOT exceed the affected items' own total (Σ quantity·unit_cost); violation is a schema error, not a validator rejection.
• Loading the IR projection from a bundle is FAIL-FAST: a missing file raises (required input — contrast the tolerant inbox, §5).

7.2 The golden suite (golden.json)

Two parts, both normative on their decided fields:

• validator — the frozen deterministic outcome of validating the IR projection: outcome (the validated type's name), validates (true — the claim sits within the feasible range), claimed_saving_nok, nominal_feasible, and the percentiles p10, p50, p90. A conforming implementation MUST reproduce these values (approx-equality on floats) — either with the reference procedure below or an equivalent deterministic method that reproduces the golden outcomes. The meaningful assertion is validates = true (claimed ≤ p90); the frozen numbers are the regression net.

  Reference procedure (what generated the golden): nominal_feasible = 0.30 × Σ quantity·unit_cost; Monte Carlo with a Mersenne-Twister PRNG seeded 20260624, 512 samples; per sample, iterate affected_items in order and draw the unit cost uniformly from the item's assumptions band (fixed cost when no band), the sample's feasible saving being 0.30 × the sampled total; percentiles are the 1st, 5th and 9th cut points of the 10-quantiles (inclusive method) over the 512 feasible values.

• learning_surface — what the validator CANNOT compute, encoded by the seed verdict: modelled_saving_nok, realization_rate (strictly between 0 and 1 — a realization gap), expected_actual_saving_nok (= realization_rate × modelled_saving_nok, internal consistency required), gap_source, and context_key (the context the correction holds for). This is the ExpeL seed's anchor: the signal Step 1's fold must carry into the next hypothesis, and the reason the learning loop exists at all.

8. Budget and stop criteria

Never an unbounded loop, anywhere. Normative:

• Required at startup: positive max_rounds and max_tokens caps (the termination contract, §10). Cap objects MUST refuse construction with non-positive values.
• Real usage, never a proxy: token accounting MUST come from the provider-reported usage (total token count) after each model call — never a word-count or character proxy. On counting paths, a response missing usage MUST fail closed (an error), not silently stop counting.
• Structured stop: crossing a cap raises a structured stop event carrying the breached kind (tokens or rounds), the limit, and the observed value — never a silent hang.
• Every retry loop is attempt-bounded (Steps 2 and 5); the debate is round-capped (Step 3); round ticks are charged between attempts so the meter also bounds parse-retries.

9. Provenance

Every proposal carries a first-class provenance stamp — authoritative data, not display metadata:

• At least one citation into the source documents (file + exact text span + snippet); a run whose context yields no citable content MUST fail fast.
• The model and role that produced the proposal. An injected test client's real model id is stamped when available; a neutral unknown is the fallback — never a fabricated name.
• validator_decision ∈ {validated, rejected} — mirrors the DETERMINISTIC VALIDATOR only, stamped from the validator's outcome BEFORE any checker override, so a checker-gated proposal whose numbers passed is never mislabelled as validator-rejected. The checker's decision is a separate result field (§3 Step 4); the two falsifiers are never conflated.
• The run's token usage (from the meter, §8).
• Promotion provenance is §6 (who/experiment/when, explicit timestamp).

10. Startup contracts

ALL configuration MUST be schema-validated fail-fast at startup, BEFORE any model client is constructed: the data source (a docs directory + a positive top-k), the model map (role → model id per backend profile, each profile REQUIRING a default entry), the termination contract (§8), and the feedback shape (§4.1). The first malformed contract raises; a run never starts on a bad config.

11. Load-bearing conformance tests

A conforming implementation MUST prove each seam with a test that FAILS when the seam is detached ("green-but-dead" tests are the failure mode this rule exists to prevent). The required red-conditions, mirroring the reference suite (test names cited for cross-reference):

Seam	The test MUST fail when…	Reference test
Step-1 fold	a prior verdict no longer reaches the next hypothesis prompt; control: an empty store changes the outcome signal	test_step1_expel_loadbearing.py
Verdict-layer exclusion	the realization signal appears in the rendered read-context	test_okf.py (bundle-context exclusion)
Checker gate	the checker's surfaced output is detached OR its REJECT no longer overrides a validated outcome	test_checker_gate_loadbearing.py
Informed refinement	the prior rejection reason no longer appears verbatim in the next prompt / the outcome never flips	test_step5_refine_loadbearing.py
Async file loop	a verdict dropped after Run A fails to reach Run B's prompt via a FRESH store; control: an empty inbox	test_step7_async_loop_loadbearing.py
Promotion gate	a non-approved verdict reaches the wiki; an approved one is not navigable; the index label leaks the signal	test_step8_promotion_loadbearing.py
Persona artifact	the example drifts from the pipeline schema, or the judgement is re-inlined instead of artifact-sourced	test_persona_skill_loadbearing.py
Closed loop	the two-run simulation's marker crosses runs without the promotion (or fails to cross with it)	test_simulation_loadbearing.py
Golden regression	the validator's decided fields diverge from golden.json	test_bygg_energi_mikro.py
Context-seam purity	the navigation/context module imports an agent toolkit	test_okf.py (import guard)
Spec integrity	this spec goes missing, names a framework, or stops documenting a consumed contract field	test_method_spec_loadbearing.py

12. Cross-check table

Every field of the machine-readable contracts, mapped to its normative section (completeness is enforced by the spec-integrity test):

Field	Contract	Section
decision	persona example / verdict file / run-path feedback	§4, §4.1–§4.3
marker	persona example	§4.3
rationale	persona example / verdict file / run-path feedback	§4.1–§4.3
id	verdict file	§4.2
proposal_features	verdict file	§4.2
affected_codes	verdict file (features)	§4.2
measure_type	verdict file (features)	§4.2
claimed_saving_nok	verdict file (features) / IR projection / golden	§4.2, §7.1, §7.2
description	verdict file (features) / promoted frontmatter	§4.2, §6
project_id	IR projection	§7.1
measure	IR projection	§7.1
affected_items	IR projection	§7.1
code, quantity, unit_cost	IR projection (affected item)	§7.1
assumptions	IR projection	§7.1
outcome, validates	golden (validator)	§7.2
nominal_feasible, p10, p50, p90	golden (validator)	§7.2
modelled_saving_nok, realization_rate, expected_actual_saving_nok	golden (learning surface)	§7.2
gap_source, context_key	golden (learning surface)	§7.2
approved, rejected	decision vocabulary (run path, binary)	§4, §4.1
approved_with_adjustment	decision vocabulary (seed frontmatter + gate only)	§4, §6
type	OKF frontmatter (required field)	§2, §3 Step 1
realization_rate (frontmatter)	bundle seed (structured learning fields)	§3 Step 1, §6