# Method specification — portfolio cost-saving loop (framework-neutral) > **Status:** normative. This document specifies the method both reference implementations > build: an agentic loop that finds cost savings *inside* each project of a portfolio, with a > mandatory deterministic validator, expert judgement in the loop, and learning from the > verdicts. It is written so the method can be implemented **from this spec alone** — without > reverse-engineering any existing implementation. The prose is framework-neutral by rule: it > never names a concrete agent toolkit or vendor stack, and a guard test keeps it that way. > > The key words MUST, MUST NOT, SHOULD, and MAY are to be interpreted as in RFC 2119. > Requirements are labelled normative; anything marked *(reference)* documents the reference > implementation's concrete choice and is informative, not binding — except where the golden > suite (§7) freezes it. ## 1. Scope and conformance The method is the product: a swarm of agents generates candidate cost-saving measures for one project at a time; a **deterministic validator decides the numbers** (mandatory, blocking); domain experts judge the outcomes (human-in-the-loop); the system **learns from the verdicts** across runs. A conforming implementation: 1. implements the 8-step loop of §3 with the contracts of §4–§10; 2. reproduces the shared golden suite's decided outcomes (§7) on the shared example bundle; 3. proves every load-bearing seam with a test that FAILS when that seam is detached (§11). **Honesty rule (unwaivable):** no artifact — code, docstring, README, or report — may claim more than the implementation does. Scripted stand-ins (synthetic clients, seeded verdicts) MUST be labelled as such wherever their output is presented. **Boundary:** this is a purely technical framework. The deploying organisation owns all processing purposes and impact assessments; implementations provide only the technical prerequisites (local-only operation, provenance, no silent egress) and a disclaimer. ## 2. Terms and architecture Three layers, strictly separated (the separation is load-bearing — see §3 Step 1 and §6): - **Context layer** — one curated, version-controlled knowledge bundle per project (an "LLM wiki") in the open OKF format: a directory of markdown files with YAML frontmatter, one required field `type`, a reserved `index.md` entry point, and intra-bundle cross-links. This layer holds project documents, methodology, verified literature, constraints, **and approved verdicts** (`type: verdict` files). It is what runtime reading and experience retrieval draw from. - **Output layer** — a run-scoped folder structure of raw results: proposals pending verdict, rejections with reasons, and raw verdict files (plain JSON, one per file). Append-heavy, **never part of the wiki**. - **Promotion gate** — the only path from the output layer into the context layer (§6). Only expert-approved knowledge crosses it. Other terms: the **IR** is the typed intermediate representation of a candidate measure (§7.1); the **store** is the in-memory collection of historical verdicts retrieval ranks over (§4.2); the **fold** is the injection of retrieved prior verdicts into the hypothesis prompt (§3 Step 1); the **two falsifiers** are the deterministic validator (numbers) and the debate checker (reasoning) — they judge the same candidate and are never conflated (§3 Step 4, §9). ## 3. The loop (normative) Eight steps. Steps 1–6 happen within one run; steps 7–8 close the learning loop across runs separated in time. ### Step 1 — Understand the context (navigate, never stuff) The agent read-context for a project MUST be built by **navigating** its OKF bundle with progressive disclosure — never by stuffing the whole bundle (or keyword-retrieved chunks of it) into the prompt: - Navigation starts at `index.md` and follows its intra-bundle markdown cross-links (`](target.md)`). Targets containing a path separator are out-of-bundle and MUST be skipped. Repeated links are de-duplicated; order is deterministic (index first, then links in first-seen order). *(reference: the link pattern is `\]\(([^)]+\.md)\)`)* - A missing `index.md` is an error (a bundle has no entry point without it). A broken or bundle-escaping cross-link MUST be tolerated — skipped, never raised (OKF robustness rule); path resolution MUST be boundary-checked against the bundle directory, fail-closed. - Frontmatter is the leading `---`-delimited block, parsed line-oriented as `key: value` strings; the single required field is `type`; unknown fields MUST be preserved. - The rendered read-context is the index body (the summary) followed by each non-index concept file as a `## {type}: {title}` section; empty sections are dropped. - **`type: verdict` files MUST be excluded from the read-context.** Prior verdicts reach the hypothesis prompt ONLY via the gated experience fold below — never via context rendering, and never via a query-time retrieval tool pointed at the bundle (which would re-leak the verdict layer). **Experience fold (ExpeL-style, the learning seam):** before generation, the candidate's prior verdicts are retrieved from the store and folded into the hypothesis prompt: - The retrieval query key is the bundle's candidate features, read from the IR projection (§7.1) — available *before* any proposal exists. - Seeding: every `type: verdict` file in the bundle becomes a store entry keyed on those candidate features, with `decision` from frontmatter (default `approved`) and a rationale built from the `description` frontmatter plus, when present, the structured learning fields rendered as `[realiseringsgrad={realization_rate}; forventet_faktisk_NOK= {expected_actual_saving_nok}]`. - Ranking is **structural, never textual** — surface text MUST NOT contribute to similarity. Similarity is the weighted sum `0.60 × Jaccard(affected-code sets) + 0.25 × [measure-type equality] + 0.15 × [same magnitude bucket]`, with magnitude buckets `[0, 1e5), [1e5, 5e5), [5e5, 1e6), [1e6, ∞)` over the claimed saving and Jaccard of two empty sets defined as 1. Retrieval returns the top-k by similarity, ties broken by verdict id ascending (deterministic); k MUST be positive. - The fold prepends the retrieved verdicts (id, decision, rationale per line) to the generation context. The rationale is the carrier of the learning signal — the fold is what lets an expert's realization-rate correction reach the next hypothesis. ### Step 2 — Hypothesise (structured candidate generation) The proposer model is asked for exactly one candidate measure as a JSON object for the IR (§7.1): `project_id`, `measure`, `affected_items` (list of `{code, quantity, unit_cost}`), `claimed_saving_nok`, optional `assumptions`. A reply that fails to parse into the typed IR MUST be retried (a blind parse-retry), never silently accepted or repaired downstream — bounded by the budget meter (§8). `project_id` MAY be defaulted from the project when the model omits it. IR schema invariants (§7.1) are enforced at construction, so a malformed proposal can never exist as a value. ### Step 3 — Debate (maker-checker) Candidate reasoning is debated by a two-role maker-checker pair — a `proposer` and a `checker` — alternating turns, with the debate's converged proposer output feeding generation (Step 2's context). Requirements: - The debate MUST be round-capped (a hard maximum-rounds bound) with an additional turn-count termination safety net above it; an unbounded debate is forbidden (§8). - Debate state MUST be fresh per run — no conversation state may survive from one project run into the next. - The checker MUST be instructed to end its reply with exactly one verdict line: `VERDICT: APPROVE` if the reasoning holds, or `VERDICT: REJECT - ` if not. - An optional synchronous in-run review gate on the checker MAY be enabled (the short feedback timescale, §3 Step 7); durable in-run checkpointing is NOT required. ### Step 4 — Validate / falsify (two falsifiers, same candidate) **The deterministic validator gates the numbers — mandatory, blocking, never an optional plugin.** It is the one endpoint-free judge that anchors the loop against swarm self-confirmation. Semantics (frozen by the golden suite, §7.2): 1. Schema invariants already hold (IR construction, §7.1). 2. **Feasibility bound:** the maximum feasible saving is capped at a policy fraction (0.30) of the affected items' total cost. *(reference: computed with an LP solve whose closed form here is `0.30 × Σ quantity·unit_cost`; a missing solver MUST escalate, never silently fall back.)* 3. **Risk simulation:** seeded Monte Carlo over uncertain unit costs yields `p10`/`p50`/`p90` percentiles of the feasible saving (§7.2 fixes the procedure). 4. **Structural block:** a claim above the optimistic feasible bound (`p90`) yields a **rejection that is a distinct type from a validated proposal** — carrying the claimed and feasible figures in its reason, and no percentiles — so it can never be consumed as validated. **The checker gates the reasoning** (the second falsifier), parsed from the checker's LAST surfaced debate output, case-insensitively; the reject marker takes precedence and its trailing text is the reason. The gate is **opt-in-reject (fail-open)**: `VERDICT: APPROVE` or a missing/unparseable marker never blocks — the validator remains the sole gate on such runs. An explicit checker REJECT MUST override an otherwise-validated outcome into a rejection (reason prefixed with the checker's reason). A validator rejection stands regardless of the checker. The two falsifiers MUST be recorded separately (§9): the provenance field mirrors ONLY the validator; the checker's decision (`approve` / `reject` / `absent`) is reported as its own result field. Either the checker actually gates, or the debate must not be called maker-checker. ### Step 5 — Refine, informed and bounded When the validator rejects, the next attempt MUST be informed by the falsification: the previous attempt's rejection **reason** is fed verbatim into the next attempt's prompt as a revision instruction. Constraints (all normative): - Only the *most recent* rejection reason is carried — never an accumulated history (bounded prompt growth). - Only the *reason* is carried — never the prior proposal JSON (the model must address the falsification, not parrot the rejected candidate). - The refinement loop runs under the EXISTING caps — the attempt bound (`max_attempts`, reference default 3) and the token/round meter (§8). "Refine until good enough" without a cap is forbidden; no new loop may be introduced. - The only *per-attempt* falsifier in this loop is the deterministic validator. Seeding generation with the checker's critique is a run-level concern outside this loop's scope. - Attempt 1 MUST use the unchanged base prompt (the informed block appears only after a rejection). ### Step 6 — Discard or propose The run's outcome is either the validated proposal (with its percentiles) or a typed rejection with its reason — never a bare failure. Raw results (proposals pending verdict, rejections, captured verdict files) belong to the output layer (§2) as plain JSON — the raw layer deliberately does NOT use the wiki format (that is reserved for promoted knowledge). ### Step 7 — Respond to feedback (two timescales) - **Short loop:** an expert may review synchronously in-run (the optional Step-3 gate). - **Long loop (the async file inbox):** days or weeks later, an expert (or, in simulation, a persona) drops a verdict file into an inbox **folder**; a separate, later run picks it up whenever it lands. The system MUST be fully resumable across runs separated in time — no live-session assumption. Contract in §5. **Role split (unwaivable): the system READS the inbox; the expert writes it.** A run MUST NOT persist its own captured verdict back into the inbox (writing is the authoring primitive's and the promotion gate's job). In simulation a dedicated expert persona plays the human; in production a human uses the SAME folder interface. The persona is defined once, as a shared skill artifact (§4.3). ### Step 8 — Promote approved knowledge (optional + gated) When an expert APPROVES an outcome, it may be promoted from the raw output layer into the context layer as a `type: verdict` concept file, navigable by the next run's seeding (closing the learning loop through the file system). Promotion is an **opt-in public primitive** — it MUST NOT be wired into the run itself (the system reads context; the gate/persona promotes). Full gate semantics in §6. ## 4. The verdict contract Three machine-readable shapes carry expert judgement. All three share the decision vocabulary rule: **the run-path decision is binary** — `approved` or `rejected`. The adjusted-approval case (the signature case in practice: the measure is worth doing but the modelled saving overstates the expected actual) is an `approved` decision whose `rationale` records the correction. `approved_with_adjustment` exists ONLY in bundle-seed frontmatter and in the promotion gate's accepted set (§6) — a run-path feedback contract MUST reject it. ### 4.1 Run-path feedback The expert decision consumed by a run: `decision` ∈ {`approved`, `rejected`} (exactly two values) and a non-empty `rationale` string. Validated fail-fast at startup (§10). ### 4.2 The verdict file (raw output layer / inbox) One verdict per JSON file, named `{id}.json`. Top-level fields (all required): | Field | Meaning | |---|---| | `id` | The learning-loop key (see minting, below). Read VERBATIM on load — never re-minted. | | `decision` | The expert decision (§4 vocabulary). | | `rationale` | Prose carrying the knowledge the validator cannot compute. | | `proposal_features` | The structural features of the judged candidate (below). | `proposal_features` fields: `affected_codes` (emitted as a SORTED list), `measure_type` (string), `claimed_saving_nok` (number), `description` (string; surface text — deliberately excluded from both similarity ranking and id minting). **Id minting (normative):** `id` is the first 16 hex characters of the SHA-256 of the canonical JSON `{"affected_codes": , "claimed_saving_nok": , "measure_type": }` with keys sorted and separators `,`/`:` (no whitespace). The id therefore keys on the **candidate measure**, not the verdict event: a structurally identical proposal maps to the same id. Because raw JSON number formatting participates in the hash (`30000` vs `30000.0` differ), a loaded verdict's `id` MUST be kept verbatim — re-minting could diverge. **Conflict semantics (chosen, documented):** the in-memory store is FIRST-write-wins per `id` (repeated inbox merges are idempotent); the disk layers — inbox files and promoted wiki files — are LAST-write-wins per file. Two verdicts about the same candidate share an id and hence a filename. A full verdict-conflict taxonomy is deliberately deferred until real experts produce conflicting verdicts. ### 4.3 The persona example artifact The expert-reviewer persona is a shared skill: a persona prompt plus one canonical example verdict JSON with fields `decision`, `marker`, and `rationale`. Requirements: - `decision` MUST be a run-path value (§4.1); the canonical example is `approved`. - `marker` MUST be a substring of `rationale` — it is the traceable payload (the realization rate) a simulation follows from the persona's judgement into a later run's prompt. - Implementations MUST source the persona judgement from this artifact at call time (a loader, fail-fast on a missing/malformed file — it is required input), never from an inlined copy. The persona prompt's prose never names a concrete agent toolkit. ## 5. The inbox/outbox folder contract The long feedback loop's folder interface (§3 Step 7). Normative: - **One verdict per file**, `{id}.json`, shape per §4.2. The authoring primitive creates the directory if needed and writes deterministically *(reference: sorted keys, 2-space indent)*. - **Tolerant load** (the raw layer is written out of band; half-written or foreign files are realistic): a missing folder yields zero verdicts; files that are not `.json`, fail to parse, or lack a required top-level key (`id`, `decision`, `rationale`, `proposal_features`) are SKIPPED, never raised. Contrast: required inputs (the IR projection §7.1, the persona example §4.3) are fail-fast. - **Deterministic order:** files are processed sorted by filename. - **Merge, never replace:** a run ingests the inbox INTO its store (per-verdict add, first-write-wins per id) BEFORE the Step-1 fold, so a passed-in store's existing verdicts survive (cross-project threading) and repeated merges are idempotent. - **Role split:** the system reads; the expert/persona writes (§3 Step 7). ## 6. The promotion gate `promote` lifts one APPROVED verdict from the raw output layer into the OKF context layer. Normative semantics: - **Fail-closed:** a verdict whose `decision` is not in {`approved`, `approved_with_adjustment`} MUST be refused with an error, writing and linking NOTHING. Only human/persona-approved knowledge enters the wiki — never raw agent output (self-contamination). - **Provenance-stamped:** the promoted file records who approved, which experiment, and when. The timestamp MUST be an explicit required argument — no wall-clock default — so promotion is deterministic and reproducible. - **Minimal promoted file:** frontmatter `type: verdict`, the `decision`, the verdict's `rationale` as the `description` field (the learning signal as prose), the verbatim verdict id, the provenance stamp, and tags. The promoted file MUST NOT reproduce a hand-authored seed's structured learning fields (`realization_rate` etc.) — the raw verdict model carries the signal only as rationale prose, and seeding (§3 Step 1) folds the `description` in. - **Navigability:** the file is written into the bundle (path-safe, fail-closed against escaping names; filename `promoted-verdict-{token}.md` where the token is the id sanitised to `[A-Za-z0-9._-]`, a degenerate token falling back to a content hash) and linked from `index.md` — navigation follows only index cross-links, so an unlinked file is unreachable. Linking MUST be idempotent (re-promotion never double-links). - **Neutral label:** the index link label is FIXED and carries NO verdict signal. The index body flows verbatim into the rendered read-context, so a descriptive label (e.g. the rationale) would leak the learning signal around the gated fold. - **Per-candidate growth:** ids key on candidate features (§4.2), so two approvals of the same candidate share a filename — last-write-wins; the wiki grows one curated verdict file per distinct candidate, not one per verdict event. - *(reference limitation)* the index read-modify-write is not atomic — single-process use is assumed for the MVP. ## 7. Ground truth: IR projection and golden suite The shared example bundle ships two JSON files that are **the only ground truth** ("fasit") for cross-implementation equivalence. Implementations MUST consume them unchanged. ### 7.1 The IR projection (`validator-input.json`) The candidate measure projected into the typed cost-IR the validator consumes: - `project_id` (string), `measure` (string), `affected_items` — a non-empty list of `{code: string, quantity: number ≥ 0, unit_cost: number > 0}` — `claimed_saving_nok` (number > 0), and `assumptions`: a map `code → [low_unit_cost, high_unit_cost]` giving the uncertainty band per cost code for the risk simulation (empty = degenerate, no spread). - Construction-time invariant: the claimed saving MUST NOT exceed the affected items' own total (`Σ quantity·unit_cost`); violation is a schema error, not a validator rejection. - Loading the IR projection from a bundle is FAIL-FAST: a missing file raises (required input — contrast the tolerant inbox, §5). ### 7.2 The golden suite (`golden.json`) Two parts, both normative on their decided fields: - **`validator`** — the frozen deterministic outcome of validating the IR projection: `outcome` (the validated type's name), `validates` (true — the claim sits within the feasible range), `claimed_saving_nok`, `nominal_feasible`, and the percentiles `p10`, `p50`, `p90`. A conforming implementation MUST reproduce these values (approx-equality on floats) — either with the reference procedure below or an equivalent deterministic method that reproduces the golden outcomes. The meaningful assertion is `validates` = true (claimed ≤ `p90`); the frozen numbers are the regression net. Reference procedure (what generated the golden): `nominal_feasible = 0.30 × Σ quantity·unit_cost`; Monte Carlo with a Mersenne-Twister PRNG seeded `20260624`, 512 samples; per sample, iterate `affected_items` in order and draw the unit cost uniformly from the item's `assumptions` band (fixed cost when no band), the sample's feasible saving being `0.30 ×` the sampled total; percentiles are the 1st, 5th and 9th cut points of the 10-quantiles (inclusive method) over the 512 feasible values. - **`learning_surface`** — what the validator CANNOT compute, encoded by the seed verdict: `modelled_saving_nok`, `realization_rate` (strictly between 0 and 1 — a realization gap), `expected_actual_saving_nok` (= `realization_rate` × `modelled_saving_nok`, internal consistency required), `gap_source`, and `context_key` (the context the correction holds for). This is the ExpeL seed's anchor: the signal Step 1's fold must carry into the next hypothesis, and the reason the learning loop exists at all. ## 8. Budget and stop criteria Never an unbounded loop, anywhere. Normative: - **Required at startup:** positive `max_rounds` and `max_tokens` caps (the termination contract, §10). Cap objects MUST refuse construction with non-positive values. - **Real usage, never a proxy:** token accounting MUST come from the provider-reported usage (total token count) after each model call — never a word-count or character proxy. On counting paths, a response missing usage MUST fail closed (an error), not silently stop counting. - **Structured stop:** crossing a cap raises a structured stop event carrying the breached kind (tokens or rounds), the limit, and the observed value — never a silent hang. - Every retry loop is attempt-bounded (Steps 2 and 5); the debate is round-capped (Step 3); round ticks are charged between attempts so the meter also bounds parse-retries. ## 9. Provenance Every proposal carries a first-class provenance stamp — authoritative data, not display metadata: - **At least one citation** into the source documents (file + exact text span + snippet); a run whose context yields no citable content MUST fail fast. - The `model` and `role` that produced the proposal. An injected test client's real model id is stamped when available; a neutral `unknown` is the fallback — never a fabricated name. - **`validator_decision`** ∈ {`validated`, `rejected`} — mirrors the DETERMINISTIC VALIDATOR only, stamped from the validator's outcome BEFORE any checker override, so a checker-gated proposal whose numbers passed is never mislabelled as validator-rejected. The checker's decision is a separate result field (§3 Step 4); the two falsifiers are never conflated. - The run's token usage (from the meter, §8). - Promotion provenance is §6 (who/experiment/when, explicit timestamp). ## 10. Startup contracts ALL configuration MUST be schema-validated fail-fast at startup, BEFORE any model client is constructed: the data source (a docs directory + a positive top-k), the model map (role → model id per backend profile, each profile REQUIRING a `default` entry), the termination contract (§8), and the feedback shape (§4.1). The first malformed contract raises; a run never starts on a bad config. ## 11. Load-bearing conformance tests A conforming implementation MUST prove each seam with a test that FAILS when the seam is detached ("green-but-dead" tests are the failure mode this rule exists to prevent). The required red-conditions, mirroring the reference suite (test names cited for cross-reference): | Seam | The test MUST fail when… | Reference test | |---|---|---| | Step-1 fold | a prior verdict no longer reaches the next hypothesis prompt; control: an empty store changes the outcome signal | `test_step1_expel_loadbearing.py` | | Verdict-layer exclusion | the realization signal appears in the rendered read-context | `test_okf.py` (bundle-context exclusion) | | Checker gate | the checker's surfaced output is detached OR its REJECT no longer overrides a validated outcome | `test_checker_gate_loadbearing.py` | | Informed refinement | the prior rejection reason no longer appears verbatim in the next prompt / the outcome never flips | `test_step5_refine_loadbearing.py` | | Async file loop | a verdict dropped after Run A fails to reach Run B's prompt via a FRESH store; control: an empty inbox | `test_step7_async_loop_loadbearing.py` | | Promotion gate | a non-approved verdict reaches the wiki; an approved one is not navigable; the index label leaks the signal | `test_step8_promotion_loadbearing.py` | | Persona artifact | the example drifts from the pipeline schema, or the judgement is re-inlined instead of artifact-sourced | `test_persona_skill_loadbearing.py` | | Closed loop | the two-run simulation's marker crosses runs without the promotion (or fails to cross with it) | `test_simulation_loadbearing.py` | | Golden regression | the validator's decided fields diverge from `golden.json` | `test_bygg_energi_mikro.py` | | Context-seam purity | the navigation/context module imports an agent toolkit | `test_okf.py` (import guard) | | Spec integrity | this spec goes missing, names a framework, or stops documenting a consumed contract field | `test_method_spec_loadbearing.py` | ## 12. Cross-check table Every field of the machine-readable contracts, mapped to its normative section (completeness is enforced by the spec-integrity test): | Field | Contract | Section | |---|---|---| | `decision` | persona example / verdict file / run-path feedback | §4, §4.1–§4.3 | | `marker` | persona example | §4.3 | | `rationale` | persona example / verdict file / run-path feedback | §4.1–§4.3 | | `id` | verdict file | §4.2 | | `proposal_features` | verdict file | §4.2 | | `affected_codes` | verdict file (features) | §4.2 | | `measure_type` | verdict file (features) | §4.2 | | `claimed_saving_nok` | verdict file (features) / IR projection / golden | §4.2, §7.1, §7.2 | | `description` | verdict file (features) / promoted frontmatter | §4.2, §6 | | `project_id` | IR projection | §7.1 | | `measure` | IR projection | §7.1 | | `affected_items` | IR projection | §7.1 | | `code`, `quantity`, `unit_cost` | IR projection (affected item) | §7.1 | | `assumptions` | IR projection | §7.1 | | `outcome`, `validates` | golden (validator) | §7.2 | | `nominal_feasible`, `p10`, `p50`, `p90` | golden (validator) | §7.2 | | `modelled_saving_nok`, `realization_rate`, `expected_actual_saving_nok` | golden (learning surface) | §7.2 | | `gap_source`, `context_key` | golden (learning surface) | §7.2 | | `approved`, `rejected` | decision vocabulary (run path, binary) | §4, §4.1 | | `approved_with_adjustment` | decision vocabulary (seed frontmatter + gate only) | §4, §6 | | `type` | OKF frontmatter (required field) | §2, §3 Step 1 | | `realization_rate` (frontmatter) | bundle seed (structured learning fields) | §3 Step 1, §6 |