From 5de1c93b69af94625067786b214e19cd78a588f6 Mon Sep 17 00:00:00 2001 From: Kjell Tore Guttormsen Date: Fri, 3 Jul 2026 00:50:42 +0200 Subject: [PATCH] =?UTF-8?q?feat(spec):=20S2=20=E2=80=94=20normative=20fram?= =?UTF-8?q?ework-neutral=20method=20spec=20as=20the=20fourth=20shared=20ar?= =?UTF-8?q?tifact?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Author shared/method-spec.md: the 8-step loop (normative, RFC-2119), the verdict JSON contract incl. the id-minting algorithm and the chosen conflict semantics, the inbox/outbox folder contract, the fail-closed promotion-gate semantics, the IR projection + golden suite as the only ground truth (incl. the reproducible Monte Carlo procedure), and the budget/stop, provenance and startup-contract requirements — every normative claim cross-checked against the load-bearing tests/code. The sibling implementation builds from this spec alone. Load-bearing trio (tests/test_method_spec_loadbearing.py, persona-trio style): required structure, a name-shaped framework-neutrality guard over the spec + the persona skill tree, and a cross-check-completeness test driven from the REAL artifacts and the REAL verdict serializer (red on code drift). All three detach points proven RED (missing file / framework name / dropped field). shared/README.md: the "(planned)" line replaced with the real entry. Suite 152 -> 155 passed / 4 skipped; ruff check+format clean; mypy src clean. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01AaQCFnfsh3tfq1VfzdJpoi --- shared/README.md | 7 +- shared/method-spec.md | 441 ++++++++++++++++++++++++++ tests/test_method_spec_loadbearing.py | 131 ++++++++ 3 files changed, 578 insertions(+), 1 deletion(-) create mode 100644 shared/method-spec.md create mode 100644 tests/test_method_spec_loadbearing.py diff --git a/shared/README.md b/shared/README.md index c24704e..7671a1f 100644 --- a/shared/README.md +++ b/shared/README.md @@ -25,7 +25,12 @@ so the only thing that differs is the agent framework itself. realization-gap methodology the validator cannot compute) and a canonical `references/example-verdict.json`. Both reference implementations instantiate the reviewer from this one artifact; `shared/` stays pure data (each stack reads the JSON with its own loader). -- *(planned)* the method specification. +- [`method-spec.md`](method-spec.md) — the **normative method specification**, framework-neutral + (the prose never names a concrete agent toolkit — enforced by a guard test): the 8-step loop, + the verdict JSON contract, the inbox/outbox folder contract, the promotion-gate semantics, the + IR projection + golden suite as the only ground truth, and the budget/provenance requirements. + The sibling implementation is built from this spec alone, without reverse-engineering the + reference code. ## Rules diff --git a/shared/method-spec.md b/shared/method-spec.md new file mode 100644 index 0000000..bcb40a9 --- /dev/null +++ b/shared/method-spec.md @@ -0,0 +1,441 @@ +# Method specification — portfolio cost-saving loop (framework-neutral) + +> **Status:** normative. This document specifies the method both reference implementations +> build: an agentic loop that finds cost savings *inside* each project of a portfolio, with a +> mandatory deterministic validator, expert judgement in the loop, and learning from the +> verdicts. It is written so the method can be implemented **from this spec alone** — without +> reverse-engineering any existing implementation. The prose is framework-neutral by rule: it +> never names a concrete agent toolkit or vendor stack, and a guard test keeps it that way. +> +> The key words MUST, MUST NOT, SHOULD, and MAY are to be interpreted as in RFC 2119. +> Requirements are labelled normative; anything marked *(reference)* documents the reference +> implementation's concrete choice and is informative, not binding — except where the golden +> suite (§7) freezes it. + +## 1. Scope and conformance + +The method is the product: a swarm of agents generates candidate cost-saving measures for one +project at a time; a **deterministic validator decides the numbers** (mandatory, blocking); +domain experts judge the outcomes (human-in-the-loop); the system **learns from the verdicts** +across runs. A conforming implementation: + +1. implements the 8-step loop of §3 with the contracts of §4–§10; +2. reproduces the shared golden suite's decided outcomes (§7) on the shared example bundle; +3. proves every load-bearing seam with a test that FAILS when that seam is detached (§11). + +**Honesty rule (unwaivable):** no artifact — code, docstring, README, or report — may claim +more than the implementation does. Scripted stand-ins (synthetic clients, seeded verdicts) +MUST be labelled as such wherever their output is presented. + +**Boundary:** this is a purely technical framework. The deploying organisation owns all +processing purposes and impact assessments; implementations provide only the technical +prerequisites (local-only operation, provenance, no silent egress) and a disclaimer. + +## 2. Terms and architecture + +Three layers, strictly separated (the separation is load-bearing — see §3 Step 1 and §6): + +- **Context layer** — one curated, version-controlled knowledge bundle per project (an "LLM + wiki") in the open OKF format: a directory of markdown files with YAML frontmatter, one + required field `type`, a reserved `index.md` entry point, and intra-bundle cross-links. + This layer holds project documents, methodology, verified literature, constraints, **and + approved verdicts** (`type: verdict` files). It is what runtime reading and experience + retrieval draw from. +- **Output layer** — a run-scoped folder structure of raw results: proposals pending verdict, + rejections with reasons, and raw verdict files (plain JSON, one per file). Append-heavy, + **never part of the wiki**. +- **Promotion gate** — the only path from the output layer into the context layer (§6). Only + expert-approved knowledge crosses it. + +Other terms: the **IR** is the typed intermediate representation of a candidate measure +(§7.1); the **store** is the in-memory collection of historical verdicts retrieval ranks over +(§4.2); the **fold** is the injection of retrieved prior verdicts into the hypothesis prompt +(§3 Step 1); the **two falsifiers** are the deterministic validator (numbers) and the debate +checker (reasoning) — they judge the same candidate and are never conflated (§3 Step 4, §9). + +## 3. The loop (normative) + +Eight steps. Steps 1–6 happen within one run; steps 7–8 close the learning loop across runs +separated in time. + +### Step 1 — Understand the context (navigate, never stuff) + +The agent read-context for a project MUST be built by **navigating** its OKF bundle with +progressive disclosure — never by stuffing the whole bundle (or keyword-retrieved chunks of +it) into the prompt: + +- Navigation starts at `index.md` and follows its intra-bundle markdown cross-links + (`](target.md)`). Targets containing a path separator are out-of-bundle and MUST be + skipped. Repeated links are de-duplicated; order is deterministic (index first, then links + in first-seen order). *(reference: the link pattern is `\]\(([^)]+\.md)\)`)* +- A missing `index.md` is an error (a bundle has no entry point without it). A broken or + bundle-escaping cross-link MUST be tolerated — skipped, never raised (OKF robustness rule); + path resolution MUST be boundary-checked against the bundle directory, fail-closed. +- Frontmatter is the leading `---`-delimited block, parsed line-oriented as `key: value` + strings; the single required field is `type`; unknown fields MUST be preserved. +- The rendered read-context is the index body (the summary) followed by each non-index + concept file as a `## {type}: {title}` section; empty sections are dropped. +- **`type: verdict` files MUST be excluded from the read-context.** Prior verdicts reach the + hypothesis prompt ONLY via the gated experience fold below — never via context rendering, + and never via a query-time retrieval tool pointed at the bundle (which would re-leak the + verdict layer). + +**Experience fold (ExpeL-style, the learning seam):** before generation, the candidate's +prior verdicts are retrieved from the store and folded into the hypothesis prompt: + +- The retrieval query key is the bundle's candidate features, read from the IR projection + (§7.1) — available *before* any proposal exists. +- Seeding: every `type: verdict` file in the bundle becomes a store entry keyed on those + candidate features, with `decision` from frontmatter (default `approved`) and a rationale + built from the `description` frontmatter plus, when present, the structured learning + fields rendered as `[realiseringsgrad={realization_rate}; forventet_faktisk_NOK= + {expected_actual_saving_nok}]`. +- Ranking is **structural, never textual** — surface text MUST NOT contribute to similarity. + Similarity is the weighted sum `0.60 × Jaccard(affected-code sets) + 0.25 × + [measure-type equality] + 0.15 × [same magnitude bucket]`, with magnitude buckets + `[0, 1e5), [1e5, 5e5), [5e5, 1e6), [1e6, ∞)` over the claimed saving and Jaccard of two + empty sets defined as 1. Retrieval returns the top-k by similarity, ties broken by verdict + id ascending (deterministic); k MUST be positive. +- The fold prepends the retrieved verdicts (id, decision, rationale per line) to the + generation context. The rationale is the carrier of the learning signal — the fold is what + lets an expert's realization-rate correction reach the next hypothesis. + +### Step 2 — Hypothesise (structured candidate generation) + +The proposer model is asked for exactly one candidate measure as a JSON object for the IR +(§7.1): `project_id`, `measure`, `affected_items` (list of `{code, quantity, unit_cost}`), +`claimed_saving_nok`, optional `assumptions`. A reply that fails to parse into the typed IR +MUST be retried (a blind parse-retry), never silently accepted or repaired downstream — +bounded by the budget meter (§8). `project_id` MAY be defaulted from the project when the +model omits it. IR schema invariants (§7.1) are enforced at construction, so a malformed +proposal can never exist as a value. + +### Step 3 — Debate (maker-checker) + +Candidate reasoning is debated by a two-role maker-checker pair — a `proposer` and a +`checker` — alternating turns, with the debate's converged proposer output feeding generation +(Step 2's context). Requirements: + +- The debate MUST be round-capped (a hard maximum-rounds bound) with an additional + turn-count termination safety net above it; an unbounded debate is forbidden (§8). +- Debate state MUST be fresh per run — no conversation state may survive from one project + run into the next. +- The checker MUST be instructed to end its reply with exactly one verdict line: + `VERDICT: APPROVE` if the reasoning holds, or `VERDICT: REJECT - ` if not. +- An optional synchronous in-run review gate on the checker MAY be enabled (the short + feedback timescale, §3 Step 7); durable in-run checkpointing is NOT required. + +### Step 4 — Validate / falsify (two falsifiers, same candidate) + +**The deterministic validator gates the numbers — mandatory, blocking, never an optional +plugin.** It is the one endpoint-free judge that anchors the loop against swarm +self-confirmation. Semantics (frozen by the golden suite, §7.2): + +1. Schema invariants already hold (IR construction, §7.1). +2. **Feasibility bound:** the maximum feasible saving is capped at a policy fraction + (0.30) of the affected items' total cost. *(reference: computed with an LP solve whose + closed form here is `0.30 × Σ quantity·unit_cost`; a missing solver MUST escalate, never + silently fall back.)* +3. **Risk simulation:** seeded Monte Carlo over uncertain unit costs yields `p10`/`p50`/`p90` + percentiles of the feasible saving (§7.2 fixes the procedure). +4. **Structural block:** a claim above the optimistic feasible bound (`p90`) yields a + **rejection that is a distinct type from a validated proposal** — carrying the claimed and + feasible figures in its reason, and no percentiles — so it can never be consumed as + validated. + +**The checker gates the reasoning** (the second falsifier), parsed from the checker's LAST +surfaced debate output, case-insensitively; the reject marker takes precedence and its +trailing text is the reason. The gate is **opt-in-reject (fail-open)**: `VERDICT: APPROVE` or +a missing/unparseable marker never blocks — the validator remains the sole gate on such runs. +An explicit checker REJECT MUST override an otherwise-validated outcome into a rejection +(reason prefixed with the checker's reason). A validator rejection stands regardless of the +checker. The two falsifiers MUST be recorded separately (§9): the provenance field mirrors +ONLY the validator; the checker's decision (`approve` / `reject` / `absent`) is reported as +its own result field. Either the checker actually gates, or the debate must not be called +maker-checker. + +### Step 5 — Refine, informed and bounded + +When the validator rejects, the next attempt MUST be informed by the falsification: the +previous attempt's rejection **reason** is fed verbatim into the next attempt's prompt as a +revision instruction. Constraints (all normative): + +- Only the *most recent* rejection reason is carried — never an accumulated history + (bounded prompt growth). +- Only the *reason* is carried — never the prior proposal JSON (the model must address the + falsification, not parrot the rejected candidate). +- The refinement loop runs under the EXISTING caps — the attempt bound (`max_attempts`, + reference default 3) and the token/round meter (§8). "Refine until good enough" without a + cap is forbidden; no new loop may be introduced. +- The only *per-attempt* falsifier in this loop is the deterministic validator. Seeding + generation with the checker's critique is a run-level concern outside this loop's scope. +- Attempt 1 MUST use the unchanged base prompt (the informed block appears only after a + rejection). + +### Step 6 — Discard or propose + +The run's outcome is either the validated proposal (with its percentiles) or a typed +rejection with its reason — never a bare failure. Raw results (proposals pending verdict, +rejections, captured verdict files) belong to the output layer (§2) as plain JSON — the raw +layer deliberately does NOT use the wiki format (that is reserved for promoted knowledge). + +### Step 7 — Respond to feedback (two timescales) + +- **Short loop:** an expert may review synchronously in-run (the optional Step-3 gate). +- **Long loop (the async file inbox):** days or weeks later, an expert (or, in simulation, a + persona) drops a verdict file into an inbox **folder**; a separate, later run picks it up + whenever it lands. The system MUST be fully resumable across runs separated in time — no + live-session assumption. Contract in §5. **Role split (unwaivable): the system READS the + inbox; the expert writes it.** A run MUST NOT persist its own captured verdict back into + the inbox (writing is the authoring primitive's and the promotion gate's job). + +In simulation a dedicated expert persona plays the human; in production a human uses the +SAME folder interface. The persona is defined once, as a shared skill artifact (§4.3). + +### Step 8 — Promote approved knowledge (optional + gated) + +When an expert APPROVES an outcome, it may be promoted from the raw output layer into the +context layer as a `type: verdict` concept file, navigable by the next run's seeding +(closing the learning loop through the file system). Promotion is an **opt-in public +primitive** — it MUST NOT be wired into the run itself (the system reads context; the +gate/persona promotes). Full gate semantics in §6. + +## 4. The verdict contract + +Three machine-readable shapes carry expert judgement. All three share the decision +vocabulary rule: **the run-path decision is binary** — `approved` or `rejected`. The +adjusted-approval case (the signature case in practice: the measure is worth doing but the +modelled saving overstates the expected actual) is an `approved` decision whose `rationale` +records the correction. `approved_with_adjustment` exists ONLY in bundle-seed frontmatter +and in the promotion gate's accepted set (§6) — a run-path feedback contract MUST reject it. + +### 4.1 Run-path feedback + +The expert decision consumed by a run: `decision` ∈ {`approved`, `rejected`} (exactly two +values) and a non-empty `rationale` string. Validated fail-fast at startup (§10). + +### 4.2 The verdict file (raw output layer / inbox) + +One verdict per JSON file, named `{id}.json`. Top-level fields (all required): + +| Field | Meaning | +|---|---| +| `id` | The learning-loop key (see minting, below). Read VERBATIM on load — never re-minted. | +| `decision` | The expert decision (§4 vocabulary). | +| `rationale` | Prose carrying the knowledge the validator cannot compute. | +| `proposal_features` | The structural features of the judged candidate (below). | + +`proposal_features` fields: `affected_codes` (emitted as a SORTED list), `measure_type` +(string), `claimed_saving_nok` (number), `description` (string; surface text — deliberately +excluded from both similarity ranking and id minting). + +**Id minting (normative):** `id` is the first 16 hex characters of the SHA-256 of the +canonical JSON `{"affected_codes": , "claimed_saving_nok": , +"measure_type": }` with keys sorted and separators `,`/`:` (no whitespace). The id +therefore keys on the **candidate measure**, not the verdict event: a structurally identical +proposal maps to the same id. Because raw JSON number formatting participates in the hash +(`30000` vs `30000.0` differ), a loaded verdict's `id` MUST be kept verbatim — re-minting +could diverge. + +**Conflict semantics (chosen, documented):** the in-memory store is FIRST-write-wins per +`id` (repeated inbox merges are idempotent); the disk layers — inbox files and promoted wiki +files — are LAST-write-wins per file. Two verdicts about the same candidate share an id and +hence a filename. A full verdict-conflict taxonomy is deliberately deferred until real +experts produce conflicting verdicts. + +### 4.3 The persona example artifact + +The expert-reviewer persona is a shared skill: a persona prompt plus one canonical example +verdict JSON with fields `decision`, `marker`, and `rationale`. Requirements: + +- `decision` MUST be a run-path value (§4.1); the canonical example is `approved`. +- `marker` MUST be a substring of `rationale` — it is the traceable payload (the realization + rate) a simulation follows from the persona's judgement into a later run's prompt. +- Implementations MUST source the persona judgement from this artifact at call time (a + loader, fail-fast on a missing/malformed file — it is required input), never from an + inlined copy. The persona prompt's prose never names a concrete agent toolkit. + +## 5. The inbox/outbox folder contract + +The long feedback loop's folder interface (§3 Step 7). Normative: + +- **One verdict per file**, `{id}.json`, shape per §4.2. The authoring primitive creates the + directory if needed and writes deterministically *(reference: sorted keys, 2-space + indent)*. +- **Tolerant load** (the raw layer is written out of band; half-written or foreign files are + realistic): a missing folder yields zero verdicts; files that are not `.json`, fail to + parse, or lack a required top-level key (`id`, `decision`, `rationale`, + `proposal_features`) are SKIPPED, never raised. Contrast: required inputs (the IR + projection §7.1, the persona example §4.3) are fail-fast. +- **Deterministic order:** files are processed sorted by filename. +- **Merge, never replace:** a run ingests the inbox INTO its store (per-verdict add, + first-write-wins per id) BEFORE the Step-1 fold, so a passed-in store's existing verdicts + survive (cross-project threading) and repeated merges are idempotent. +- **Role split:** the system reads; the expert/persona writes (§3 Step 7). + +## 6. The promotion gate + +`promote` lifts one APPROVED verdict from the raw output layer into the OKF context layer. +Normative semantics: + +- **Fail-closed:** a verdict whose `decision` is not in {`approved`, + `approved_with_adjustment`} MUST be refused with an error, writing and linking NOTHING. + Only human/persona-approved knowledge enters the wiki — never raw agent output + (self-contamination). +- **Provenance-stamped:** the promoted file records who approved, which experiment, and + when. The timestamp MUST be an explicit required argument — no wall-clock default — so + promotion is deterministic and reproducible. +- **Minimal promoted file:** frontmatter `type: verdict`, the `decision`, the verdict's + `rationale` as the `description` field (the learning signal as prose), the verbatim + verdict id, the provenance stamp, and tags. The promoted file MUST NOT reproduce a + hand-authored seed's structured learning fields (`realization_rate` etc.) — the raw + verdict model carries the signal only as rationale prose, and seeding (§3 Step 1) folds + the `description` in. +- **Navigability:** the file is written into the bundle (path-safe, fail-closed against + escaping names; filename `promoted-verdict-{token}.md` where the token is the id + sanitised to `[A-Za-z0-9._-]`, a degenerate token falling back to a content hash) and + linked from `index.md` — navigation follows only index cross-links, so an unlinked file + is unreachable. Linking MUST be idempotent (re-promotion never double-links). +- **Neutral label:** the index link label is FIXED and carries NO verdict signal. The index + body flows verbatim into the rendered read-context, so a descriptive label (e.g. the + rationale) would leak the learning signal around the gated fold. +- **Per-candidate growth:** ids key on candidate features (§4.2), so two approvals of the + same candidate share a filename — last-write-wins; the wiki grows one curated verdict + file per distinct candidate, not one per verdict event. +- *(reference limitation)* the index read-modify-write is not atomic — single-process use + is assumed for the MVP. + +## 7. Ground truth: IR projection and golden suite + +The shared example bundle ships two JSON files that are **the only ground truth** ("fasit") +for cross-implementation equivalence. Implementations MUST consume them unchanged. + +### 7.1 The IR projection (`validator-input.json`) + +The candidate measure projected into the typed cost-IR the validator consumes: + +- `project_id` (string), `measure` (string), `affected_items` — a non-empty list of + `{code: string, quantity: number ≥ 0, unit_cost: number > 0}` — `claimed_saving_nok` + (number > 0), and `assumptions`: a map `code → [low_unit_cost, high_unit_cost]` giving the + uncertainty band per cost code for the risk simulation (empty = degenerate, no spread). +- Construction-time invariant: the claimed saving MUST NOT exceed the affected items' own + total (`Σ quantity·unit_cost`); violation is a schema error, not a validator rejection. +- Loading the IR projection from a bundle is FAIL-FAST: a missing file raises (required + input — contrast the tolerant inbox, §5). + +### 7.2 The golden suite (`golden.json`) + +Two parts, both normative on their decided fields: + +- **`validator`** — the frozen deterministic outcome of validating the IR projection: + `outcome` (the validated type's name), `validates` (true — the claim sits within the + feasible range), `claimed_saving_nok`, `nominal_feasible`, and the percentiles `p10`, + `p50`, `p90`. A conforming implementation MUST reproduce these values (approx-equality on + floats) — either with the reference procedure below or an equivalent deterministic method + that reproduces the golden outcomes. The meaningful assertion is `validates` = true + (claimed ≤ `p90`); the frozen numbers are the regression net. + + Reference procedure (what generated the golden): `nominal_feasible = 0.30 × + Σ quantity·unit_cost`; Monte Carlo with a Mersenne-Twister PRNG seeded `20260624`, 512 + samples; per sample, iterate `affected_items` in order and draw the unit cost uniformly + from the item's `assumptions` band (fixed cost when no band), the sample's feasible saving + being `0.30 ×` the sampled total; percentiles are the 1st, 5th and 9th cut points of the + 10-quantiles (inclusive method) over the 512 feasible values. + +- **`learning_surface`** — what the validator CANNOT compute, encoded by the seed verdict: + `modelled_saving_nok`, `realization_rate` (strictly between 0 and 1 — a realization gap), + `expected_actual_saving_nok` (= `realization_rate` × `modelled_saving_nok`, internal + consistency required), `gap_source`, and `context_key` (the context the correction holds + for). This is the ExpeL seed's anchor: the signal Step 1's fold must carry into the next + hypothesis, and the reason the learning loop exists at all. + +## 8. Budget and stop criteria + +Never an unbounded loop, anywhere. Normative: + +- **Required at startup:** positive `max_rounds` and `max_tokens` caps (the termination + contract, §10). Cap objects MUST refuse construction with non-positive values. +- **Real usage, never a proxy:** token accounting MUST come from the provider-reported + usage (total token count) after each model call — never a word-count or character proxy. + On counting paths, a response missing usage MUST fail closed (an error), not silently + stop counting. +- **Structured stop:** crossing a cap raises a structured stop event carrying the breached + kind (tokens or rounds), the limit, and the observed value — never a silent hang. +- Every retry loop is attempt-bounded (Steps 2 and 5); the debate is round-capped (Step 3); + round ticks are charged between attempts so the meter also bounds parse-retries. + +## 9. Provenance + +Every proposal carries a first-class provenance stamp — authoritative data, not display +metadata: + +- **At least one citation** into the source documents (file + exact text span + snippet); a + run whose context yields no citable content MUST fail fast. +- The `model` and `role` that produced the proposal. An injected test client's real model id + is stamped when available; a neutral `unknown` is the fallback — never a fabricated name. +- **`validator_decision`** ∈ {`validated`, `rejected`} — mirrors the DETERMINISTIC VALIDATOR + only, stamped from the validator's outcome BEFORE any checker override, so a + checker-gated proposal whose numbers passed is never mislabelled as validator-rejected. + The checker's decision is a separate result field (§3 Step 4); the two falsifiers are + never conflated. +- The run's token usage (from the meter, §8). +- Promotion provenance is §6 (who/experiment/when, explicit timestamp). + +## 10. Startup contracts + +ALL configuration MUST be schema-validated fail-fast at startup, BEFORE any model client is +constructed: the data source (a docs directory + a positive top-k), the model map (role → +model id per backend profile, each profile REQUIRING a `default` entry), the termination +contract (§8), and the feedback shape (§4.1). The first malformed contract raises; a run +never starts on a bad config. + +## 11. Load-bearing conformance tests + +A conforming implementation MUST prove each seam with a test that FAILS when the seam is +detached ("green-but-dead" tests are the failure mode this rule exists to prevent). The +required red-conditions, mirroring the reference suite (test names cited for +cross-reference): + +| Seam | The test MUST fail when… | Reference test | +|---|---|---| +| Step-1 fold | a prior verdict no longer reaches the next hypothesis prompt; control: an empty store changes the outcome signal | `test_step1_expel_loadbearing.py` | +| Verdict-layer exclusion | the realization signal appears in the rendered read-context | `test_okf.py` (bundle-context exclusion) | +| Checker gate | the checker's surfaced output is detached OR its REJECT no longer overrides a validated outcome | `test_checker_gate_loadbearing.py` | +| Informed refinement | the prior rejection reason no longer appears verbatim in the next prompt / the outcome never flips | `test_step5_refine_loadbearing.py` | +| Async file loop | a verdict dropped after Run A fails to reach Run B's prompt via a FRESH store; control: an empty inbox | `test_step7_async_loop_loadbearing.py` | +| Promotion gate | a non-approved verdict reaches the wiki; an approved one is not navigable; the index label leaks the signal | `test_step8_promotion_loadbearing.py` | +| Persona artifact | the example drifts from the pipeline schema, or the judgement is re-inlined instead of artifact-sourced | `test_persona_skill_loadbearing.py` | +| Closed loop | the two-run simulation's marker crosses runs without the promotion (or fails to cross with it) | `test_simulation_loadbearing.py` | +| Golden regression | the validator's decided fields diverge from `golden.json` | `test_bygg_energi_mikro.py` | +| Context-seam purity | the navigation/context module imports an agent toolkit | `test_okf.py` (import guard) | +| Spec integrity | this spec goes missing, names a framework, or stops documenting a consumed contract field | `test_method_spec_loadbearing.py` | + +## 12. Cross-check table + +Every field of the machine-readable contracts, mapped to its normative section (completeness +is enforced by the spec-integrity test): + +| Field | Contract | Section | +|---|---|---| +| `decision` | persona example / verdict file / run-path feedback | §4, §4.1–§4.3 | +| `marker` | persona example | §4.3 | +| `rationale` | persona example / verdict file / run-path feedback | §4.1–§4.3 | +| `id` | verdict file | §4.2 | +| `proposal_features` | verdict file | §4.2 | +| `affected_codes` | verdict file (features) | §4.2 | +| `measure_type` | verdict file (features) | §4.2 | +| `claimed_saving_nok` | verdict file (features) / IR projection / golden | §4.2, §7.1, §7.2 | +| `description` | verdict file (features) / promoted frontmatter | §4.2, §6 | +| `project_id` | IR projection | §7.1 | +| `measure` | IR projection | §7.1 | +| `affected_items` | IR projection | §7.1 | +| `code`, `quantity`, `unit_cost` | IR projection (affected item) | §7.1 | +| `assumptions` | IR projection | §7.1 | +| `outcome`, `validates` | golden (validator) | §7.2 | +| `nominal_feasible`, `p10`, `p50`, `p90` | golden (validator) | §7.2 | +| `modelled_saving_nok`, `realization_rate`, `expected_actual_saving_nok` | golden (learning surface) | §7.2 | +| `gap_source`, `context_key` | golden (learning surface) | §7.2 | +| `approved`, `rejected` | decision vocabulary (run path, binary) | §4, §4.1 | +| `approved_with_adjustment` | decision vocabulary (seed frontmatter + gate only) | §4, §6 | +| `type` | OKF frontmatter (required field) | §2, §3 Step 1 | +| `realization_rate` (frontmatter) | bundle seed (structured learning fields) | §3 Step 1, §6 | diff --git a/tests/test_method_spec_loadbearing.py b/tests/test_method_spec_loadbearing.py new file mode 100644 index 0000000..904463f --- /dev/null +++ b/tests/test_method_spec_loadbearing.py @@ -0,0 +1,131 @@ +"""Load-bearing tests for the shared method specification (S2, målbilde §8 — the fourth shared +artifact). + +The method spec is the normative, framework-neutral document the sibling implementation is built +FROM — it must be implementable from the spec alone, without reverse-engineering this repo's code. +These tests make the artifact load-bearing in the persona-trio's style: structure + framework +neutrality (grep-shaped), and a cross-check that every field of the machine-readable contracts the +pipeline actually consumes is documented in the spec. Each test goes RED when its seam is detached: +the spec goes missing, prose starts naming a concrete framework, or the code's verdict contract +drifts away from what the spec documents. +""" + +from __future__ import annotations + +import json +import re +from pathlib import Path + +from portfolio_optimiser.verdicts import ProposalFeatures, capture_verdict, verdict_to_dict + +REPO_ROOT = Path(__file__).resolve().parents[1] +SPEC_PATH = REPO_ROOT / "shared" / "method-spec.md" +SKILL_DIR = REPO_ROOT / "shared" / "skills" / "expert-reviewer" +EXAMPLE_VERDICT = SKILL_DIR / "references" / "example-verdict.json" +BUNDLE_DIR = REPO_ROOT / "shared" / "examples" / "bygg-energi-mikro" + +# Name-shaped framework guard (stricter than the persona test's import-shaped guard): the spec's +# PROSE must never name a concrete agent framework or vendor stack — the same rule the persona +# SKILL.md follows (CLAUDE.md §delt ekspert-persona). Scoped to the spec + the persona skill tree; +# shared/README.md and the bundle legitimately NAME the two implementations and are exempt. +_FRAMEWORK_NAMES = re.compile( + r"\bagent[_ -]framework\b|\bMAF\b|\bClaude\b|\bAnthropic\b|\bMicrosoft\b|\bAzure\b" + r"|\bFoundry\b|\bOpenAI\b|\bOllama\b|\bLangChain\b|\bAutoGen\b|\bSemantic Kernel\b" + r"|\bCrewAI\b|\bMagentic\b", + re.IGNORECASE, +) + +# The spec's required skeleton: the 8-step loop normative + one section per contract the session +# plan names (verdict JSON, inbox/outbox, promotion gate, IR+golden as fasit, budget, provenance) +# + the cross-check table the verification criterion requires. +_REQUIRED_MARKERS = [ + "# Method specification", + "## 1. Scope", + "## 3. The loop", + "## 4. The verdict contract", + "## 5. The inbox/outbox folder contract", + "## 6. The promotion gate", + "## 7. Ground truth", + "## 8. Budget and stop criteria", + "## 9. Provenance", + "## 12. Cross-check table", +] + + +def _spec_text() -> str: + assert SPEC_PATH.is_file(), "shared/method-spec.md missing (the fourth shared artifact)" + return SPEC_PATH.read_text(encoding="utf-8") + + +def test_method_spec_exists_with_required_structure() -> None: + """Test 1: the spec exists and carries the normative skeleton — the 8 steps plus every + contract section the session plan names. RED before the artifact exists, or when a required + section is dropped.""" + text = _spec_text() + for marker in _REQUIRED_MARKERS: + assert marker in text, f"method-spec.md is missing required section marker: {marker!r}" + for step in range(1, 9): + assert f"### Step {step} — " in text, f"method-spec.md must specify loop step {step}" + # Normative, not descriptive: RFC-2119-style requirement language must be present. + assert "MUST" in text, "the spec must be normative (RFC-2119 MUST language)" + + +def test_method_spec_is_framework_neutral() -> None: + """Test 2 (grep-shaped guard): the spec — and the persona skill tree it sits beside — never + NAME a concrete agent framework or vendor stack. Mirrors the persona SKILL.md rule; stricter + than the import-shaped guard because the spec is pure prose (no AST to parse). RED the moment + a framework name leaks into the shared method prose.""" + for f in [SPEC_PATH, *sorted(p for p in SKILL_DIR.rglob("*") if p.is_file())]: + assert f.is_file(), f"expected shared artifact missing: {f}" + hit = _FRAMEWORK_NAMES.search(f.read_text(encoding="utf-8")) + assert hit is None, f"framework name {hit.group(0)!r} in shared method prose: {f}" + + +def test_spec_documents_every_contract_field() -> None: + """Test 3 (cross-check completeness): every field of the machine-readable contracts the + pipeline consumes is documented in the spec — driven from the REAL artifacts and the REAL + serializer, so the test goes RED when either the persona example, the golden suite, the IR + projection, or the code's verdict serialization drifts away from the spec.""" + text = _spec_text() + + def documented(field: str, source: str) -> None: + assert f"`{field}`" in text, f"spec does not document {source} field `{field}`" + + # Persona-artifact contract: the canonical example verdict's fields, read from the artifact. + example = json.loads(EXAMPLE_VERDICT.read_text(encoding="utf-8")) + for field in example: + documented(field, "example-verdict.json") + + # Serialized verdict contract: emitted by the REAL serializer (red on code drift). + verdict = capture_verdict( + ProposalFeatures( + affected_codes=frozenset({"X.1"}), + measure_type="m", + claimed_saving_nok=1.0, + description="d", + ), + "approved", + "r", + ) + payload = verdict_to_dict(verdict) + for field in payload: + documented(field, "verdict file") + for field in payload["proposal_features"]: + documented(field, "verdict proposal_features") + + # IR projection + golden suite: the ground-truth contracts, read from the shared fixture + # ("_"-prefixed keys are annotations, not contract). + ir = json.loads((BUNDLE_DIR / "validator-input.json").read_text(encoding="utf-8")) + for field in (k for k in ir if not k.startswith("_")): + documented(field, "validator-input.json") + for field in ir["affected_items"][0]: + documented(field, "affected_items") + golden = json.loads((BUNDLE_DIR / "golden.json").read_text(encoding="utf-8")) + for section in ("validator", "learning_surface"): + for field in (k for k in golden[section] if not k.startswith("_")): + documented(field, f"golden.json {section}") + + # The decision vocabulary: binary on the run path; the gate additionally admits the seed's + # approved_with_adjustment (and nothing else may claim otherwise). + for value in ("approved", "rejected", "approved_with_adjustment"): + documented(value, "decision vocabulary")