Kjell Tore Guttormsen
8e9f385d17
The spec arrives by subtree pull from commons, so drift is something we RECEIVE rather than author. The existing guard only asserted the section HEADINGS were present: a pull could empty a section of its normative content and stay green. Measured, not assumed — with `escape, not depth` deleted from Step 1, the old skeleton test still passes (M5 below). Two new guards, one file: - Rule level (test 6): 25 normative rules, each bound to the section that OWNS it, so a phrase surviving elsewhere in the document does not count as the spec still stating the rule. Selection is principled rather than taste — every entry anchors a §11 conformance seam or a CLAUDE.md invariant. Matching is whitespace- and emphasis-insensitive: re-wrapping a paragraph upstream is an editorial change, not a rule change, and must not cost a false red. - §11 table (test 7): the conformance table still names the seams this suite implements, and every test it cites exists here. One-directional by design — the repo carries 26 load-bearing tests, the spec anchors 10, because the spec governs the METHOD, not this repo. Five measurements, all as expected: M1 delete a rule → RED · M2 RELOCATE a rule out of its section while leaving it in the document → RED (this is what the section binding buys) · M3 drop a seam row from §11 → RED · M4 harmless re-wrap + de-bolding → GREEN · M5 the old skeleton test does NOT catch M1 → GREEN. Spec restored from a byte copy after each mutation, shasum verified. Known gap, deliberately not patched here: §11 has no row for the portfolio-wide budget seam (S3.4/F10) — the pulled spec predates it. That is a commons amendment; a consumer editing shared/ is exactly the drift this guards against. 553 → 555 tests, ruff + mypy green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Aw9TECznT5b5H6374CBTvV
344 lines
18 KiB
Python
344 lines
18 KiB
Python
"""Load-bearing tests for the shared method specification (S2, målbilde §8 — the fourth shared
|
||
artifact).
|
||
|
||
The method spec is the normative, framework-neutral document the sibling implementation is built
|
||
FROM — it must be implementable from the spec alone, without reverse-engineering this repo's code.
|
||
These tests make the artifact load-bearing in the persona-trio's style: structure + framework
|
||
neutrality (grep-shaped), a cross-check that every field of the machine-readable contracts the
|
||
pipeline actually consumes is documented in the spec, and — because the spec arrives here by
|
||
subtree pull rather than by local edit — a rule-level guard that the sections still STATE the
|
||
normative rules this repo is built against. Each test goes RED when its seam is detached: the spec
|
||
goes missing, prose starts naming a concrete framework, the code's verdict contract drifts away
|
||
from what the spec documents, a normative rule is deleted or relocated out of its owning section,
|
||
or the §11 conformance table stops naming the seams this suite implements.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import dataclasses
|
||
import json
|
||
import re
|
||
from pathlib import Path
|
||
|
||
from portfolio_optimiser.ingest import Extraction, FileSource, HttpSource, ManifestV1, SqlSource
|
||
from portfolio_optimiser.verdicts import ProposalFeatures, capture_verdict, verdict_to_dict
|
||
|
||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||
SPEC_PATH = REPO_ROOT / "shared" / "method-spec.md"
|
||
INGEST_SPEC_PATH = REPO_ROOT / "shared" / "ingest-spec.md"
|
||
SKILL_DIR = REPO_ROOT / "shared" / "skills" / "expert-reviewer"
|
||
EXAMPLE_VERDICT = SKILL_DIR / "references" / "example-verdict.json"
|
||
BUNDLE_DIR = REPO_ROOT / "shared" / "examples" / "bygg-energi-mikro"
|
||
|
||
# Name-shaped framework guard (stricter than the persona test's import-shaped guard): the spec's
|
||
# PROSE must never name a concrete agent framework or vendor stack — the same rule the persona
|
||
# SKILL.md follows (CLAUDE.md §delt ekspert-persona). Scoped to the specs + the persona skill tree;
|
||
# shared/README.md and the bundle legitimately NAME the two implementations and are exempt.
|
||
_FRAMEWORK_NAMES = re.compile(
|
||
r"\bagent[_ -]framework\b|\bMAF\b|\bClaude\b|\bAnthropic\b|\bMicrosoft\b|\bAzure\b"
|
||
r"|\bFoundry\b|\bOpenAI\b|\bOllama\b|\bLangChain\b|\bAutoGen\b|\bSemantic Kernel\b"
|
||
r"|\bCrewAI\b|\bMagentic\b",
|
||
re.IGNORECASE,
|
||
)
|
||
|
||
# The spec's required skeleton: the 8-step loop normative + one section per contract the session
|
||
# plan names (verdict JSON, inbox/outbox, promotion gate, IR+golden as fasit, budget, provenance)
|
||
# + the cross-check table the verification criterion requires.
|
||
_REQUIRED_MARKERS = [
|
||
"# Method specification",
|
||
"## 1. Scope",
|
||
"## 3. The loop",
|
||
"## 4. The verdict contract",
|
||
"## 5. The inbox/outbox folder contract",
|
||
"## 6. The promotion gate",
|
||
"## 7. Ground truth",
|
||
"## 8. Budget and stop criteria",
|
||
"## 9. Provenance",
|
||
"## 12. Cross-check table",
|
||
]
|
||
|
||
|
||
# The ingest spec's required skeleton (I1, ingest-målbilde §2–§11): the contract sections both
|
||
# stacks implement from — manifest, materialization, index generation, provenance, layer
|
||
# separation, golden format — plus the cross-check table.
|
||
_INGEST_REQUIRED_MARKERS = [
|
||
"# Ingest specification",
|
||
"## 1. Scope",
|
||
"## 2. Architecture",
|
||
"## 3. Layer separation",
|
||
"## 4. The ingest manifest",
|
||
"## 5. Materialization",
|
||
"## 6. Index generation",
|
||
"## 7. Provenance",
|
||
"## 8. Security",
|
||
"## 9. The HITL gate",
|
||
"## 10. Re-ingest",
|
||
"## 11. Determinism",
|
||
"## 12. Cross-check table",
|
||
]
|
||
|
||
|
||
# The rules this repo's invariants stand on, each bound to the section that OWNS it. The skeleton
|
||
# markers above only prove the HEADINGS survive: a commons pull could empty a section of its
|
||
# normative content and stay green, which is exactly how the shared contract drifts away from the
|
||
# code built against it. Selection is not taste — every entry anchors either a §11 conformance seam
|
||
# or a CLAUDE.md invariant, and the section binding means a phrase that merely survives elsewhere in
|
||
# the document does not count as the spec still stating the rule.
|
||
#
|
||
# A RED here means "read the commons diff", not "the code is broken": re-wrapping and emphasis are
|
||
# normalised away (below), so only a changed or deleted RULE trips it.
|
||
_REQUIRED_RULES = [
|
||
# (section prefix, rule phrase)
|
||
("## 1. Scope", "Honesty rule (unwaivable)"),
|
||
# §11 "Navigation boundary" — the hierarchical contract F4 re-grounded (CLAUDE.md).
|
||
("### Step 1 — ", "bundle root"),
|
||
("### Step 1 — ", "escape, not depth"),
|
||
("### Step 1 — ", "depth-first in first-seen link order"),
|
||
("### Step 1 — ", "de-duplicated on the resolved path"),
|
||
("### Step 1 — ", "SOLE in-/out-of-bundle test"),
|
||
("### Step 1 — ", "flat regardless of nesting depth"),
|
||
# §11 "Verdict-layer exclusion" — a per-file type check, never a link-graph property.
|
||
("### Step 1 — ", "type check on each file as it is reached"),
|
||
# §11 "Step-1 fold" — structural ranking is what keeps surface text out of similarity.
|
||
("### Step 1 — ", "structural, never textual"),
|
||
# §11 "Checker gate" — the marker, the mandatory validator, and the two-falsifier separation.
|
||
("### Step 3 — ", "VERDICT: REJECT"),
|
||
("### Step 4 — ", "mandatory, blocking, never an optional plugin"),
|
||
("### Step 4 — ", "opt-in-reject (fail-open)"),
|
||
("### Step 4 — ", "mirrors ONLY the validator"),
|
||
# §11 "Informed refinement" — most-recent-only, reason-only, under the existing caps.
|
||
("### Step 5 — ", "Only the most recent rejection reason"),
|
||
("### Step 5 — ", "never the prior proposal JSON"),
|
||
("### Step 5 — ", "no new loop may be introduced"),
|
||
# §11 "Async file loop" — the role split, merge semantics, and the tolerant raw layer.
|
||
("### Step 7 — ", "the system READS the inbox; the expert writes it"),
|
||
("## 5. The inbox/outbox", "Merge, never replace"),
|
||
("## 5. The inbox/outbox", "SKIPPED, never raised"),
|
||
# §11 "Promotion gate" — fail-closed, neutral label, explicit timestamp.
|
||
("## 6. The promotion gate", "Fail-closed:"),
|
||
("## 6. The promotion gate", "index link label is FIXED and carries NO verdict signal"),
|
||
("## 6. The promotion gate", "no wall-clock default"),
|
||
# §8 — the normative anchor under the budget invariant (S3.4/F10).
|
||
("## 8. Budget", "never a word-count or character proxy"),
|
||
("## 8. Budget", "structured stop event"),
|
||
# §7.1 — the fail-fast required input, the deliberate contrast to the tolerant inbox above.
|
||
("### 7.1", "FAIL-FAST"),
|
||
]
|
||
|
||
# The §11 rows this repo's suite is anchored in. A bare count would be arbitrary AND satisfiable by
|
||
# the wrong rows; naming the seams is what makes a dropped row RED. Deliberately one-directional:
|
||
# the repo carries far more load-bearing tests than the spec anchors (the spec governs the METHOD,
|
||
# not this repo), so only these are required to appear.
|
||
_REQUIRED_CONFORMANCE_TESTS = [
|
||
"test_bygg_energi_mikro.py",
|
||
"test_checker_gate_loadbearing.py",
|
||
"test_okf.py",
|
||
"test_persona_skill_loadbearing.py",
|
||
"test_simulation_loadbearing.py",
|
||
"test_step1_expel_loadbearing.py",
|
||
"test_step5_refine_loadbearing.py",
|
||
"test_step7_async_loop_loadbearing.py",
|
||
"test_step8_promotion_loadbearing.py",
|
||
]
|
||
|
||
|
||
def _spec_text() -> str:
|
||
assert SPEC_PATH.is_file(), "shared/method-spec.md missing (the fourth shared artifact)"
|
||
return SPEC_PATH.read_text(encoding="utf-8")
|
||
|
||
|
||
def _flat(text: str) -> str:
|
||
"""Whitespace- and emphasis-insensitive view of the prose: re-wrapping a paragraph or bolding a
|
||
clause is an editorial change upstream, not a rule change, and must not turn the guard red."""
|
||
return re.sub(r"\s+", " ", text.replace("*", ""))
|
||
|
||
|
||
def _section_body(text: str, prefix: str) -> str:
|
||
"""The body owned by the heading starting with ``prefix`` (``##`` and ``###`` alike), so a rule
|
||
can be anchored to its own section. Asserting exactly one match also catches a duplicated
|
||
heading, which would make "which section states this rule?" ambiguous."""
|
||
bodies: dict[str, list[str]] = {}
|
||
current: str | None = None
|
||
for line in text.splitlines():
|
||
if line.startswith(("## ", "### ")):
|
||
current = line.strip()
|
||
bodies.setdefault(current, [])
|
||
elif current is not None:
|
||
bodies[current].append(line)
|
||
matches = [lines for head, lines in bodies.items() if head.startswith(prefix)]
|
||
assert len(matches) == 1, (
|
||
f"expected exactly one section starting {prefix!r}, got {len(matches)}"
|
||
)
|
||
return "\n".join(matches[0])
|
||
|
||
|
||
def test_method_spec_exists_with_required_structure() -> None:
|
||
"""Test 1: the spec exists and carries the normative skeleton — the 8 steps plus every
|
||
contract section the session plan names. RED before the artifact exists, or when a required
|
||
section is dropped."""
|
||
text = _spec_text()
|
||
for marker in _REQUIRED_MARKERS:
|
||
assert marker in text, f"method-spec.md is missing required section marker: {marker!r}"
|
||
for step in range(1, 9):
|
||
assert f"### Step {step} — " in text, f"method-spec.md must specify loop step {step}"
|
||
# Normative, not descriptive: RFC-2119-style requirement language must be present.
|
||
assert "MUST" in text, "the spec must be normative (RFC-2119 MUST language)"
|
||
|
||
|
||
def test_method_spec_is_framework_neutral() -> None:
|
||
"""Test 2 (grep-shaped guard): the specs — and the persona skill tree they sit beside — never
|
||
NAME a concrete agent framework or vendor stack. Mirrors the persona SKILL.md rule; stricter
|
||
than the import-shaped guard because the specs are pure prose (no AST to parse). RED the moment
|
||
a framework name leaks into the shared method prose. Covers BOTH shared specs: the method spec
|
||
and the ingest spec (I1) — a new spec file is guarded explicitly, never implicitly."""
|
||
for f in [SPEC_PATH, INGEST_SPEC_PATH, *sorted(p for p in SKILL_DIR.rglob("*") if p.is_file())]:
|
||
assert f.is_file(), f"expected shared artifact missing: {f}"
|
||
hit = _FRAMEWORK_NAMES.search(f.read_text(encoding="utf-8"))
|
||
assert hit is None, f"framework name {hit.group(0)!r} in shared method prose: {f}"
|
||
|
||
|
||
def test_spec_documents_every_contract_field() -> None:
|
||
"""Test 3 (cross-check completeness): every field of the machine-readable contracts the
|
||
pipeline consumes is documented in the spec — driven from the REAL artifacts and the REAL
|
||
serializer, so the test goes RED when either the persona example, the golden suite, the IR
|
||
projection, or the code's verdict serialization drifts away from the spec."""
|
||
text = _spec_text()
|
||
|
||
def documented(field: str, source: str) -> None:
|
||
assert f"`{field}`" in text, f"spec does not document {source} field `{field}`"
|
||
|
||
# Persona-artifact contract: the canonical example verdict's fields, read from the artifact.
|
||
example = json.loads(EXAMPLE_VERDICT.read_text(encoding="utf-8"))
|
||
for field in example:
|
||
documented(field, "example-verdict.json")
|
||
|
||
# Serialized verdict contract: emitted by the REAL serializer (red on code drift).
|
||
verdict = capture_verdict(
|
||
ProposalFeatures(
|
||
affected_codes=frozenset({"X.1"}),
|
||
measure_type="m",
|
||
claimed_saving_nok=1.0,
|
||
description="d",
|
||
),
|
||
"approved",
|
||
"r",
|
||
)
|
||
payload = verdict_to_dict(verdict)
|
||
for field in payload:
|
||
documented(field, "verdict file")
|
||
for field in payload["proposal_features"]:
|
||
documented(field, "verdict proposal_features")
|
||
|
||
# IR projection + golden suite: the ground-truth contracts, read from the shared fixture
|
||
# ("_"-prefixed keys are annotations, not contract).
|
||
ir = json.loads((BUNDLE_DIR / "validator-input.json").read_text(encoding="utf-8"))
|
||
for field in (k for k in ir if not k.startswith("_")):
|
||
documented(field, "validator-input.json")
|
||
for field in ir["affected_items"][0]:
|
||
documented(field, "affected_items")
|
||
golden = json.loads((BUNDLE_DIR / "golden.json").read_text(encoding="utf-8"))
|
||
for section in ("validator", "learning_surface"):
|
||
for field in (k for k in golden[section] if not k.startswith("_")):
|
||
documented(field, f"golden.json {section}")
|
||
|
||
# The decision vocabulary: binary on the run path; the gate additionally admits the seed's
|
||
# approved_with_adjustment (and nothing else may claim otherwise).
|
||
for value in ("approved", "rejected", "approved_with_adjustment"):
|
||
documented(value, "decision vocabulary")
|
||
|
||
|
||
def test_method_spec_states_its_load_bearing_rules() -> None:
|
||
"""Test 6 (rule-level integrity — closes test 1's gap): the spec still STATES each normative
|
||
rule this repo is built against, inside the section that owns it. Test 1 proves only that the
|
||
headings survive, so a commons pull could empty a section of its rules and stay green. RED when
|
||
a rule is deleted, reworded away, or relocated out of its own section — the three ways the
|
||
pulled contract silently stops saying what the code assumes."""
|
||
text = _spec_text()
|
||
for section, rule in _REQUIRED_RULES:
|
||
body = _flat(_section_body(text, section))
|
||
assert rule in body, (
|
||
f"method-spec.md section {section!r} no longer states the rule: {rule!r}"
|
||
)
|
||
|
||
|
||
def test_method_spec_conformance_table_cites_the_real_suite() -> None:
|
||
"""Test 7 (§11 integrity): the load-bearing conformance table still names the seams this repo's
|
||
suite implements, and every test it cites actually exists here. RED when a seam row is dropped
|
||
upstream, and RED when a cited test is deleted or renamed here — the table is the shared
|
||
contract for WHICH seams must be provable, so both directions of drift are findings.
|
||
|
||
(Known, deliberate gap: the portfolio-wide budget seam (S3.4/F10) has no §11 row yet — the
|
||
pulled spec predates it. That is a commons amendment, not something this consumer may patch.)
|
||
"""
|
||
section = _section_body(_spec_text(), "## 11. Load-bearing conformance tests")
|
||
cited = set(re.findall(r"`(test_[a-z0-9_]+\.py)`", section))
|
||
for name in _REQUIRED_CONFORMANCE_TESTS:
|
||
assert name in cited, f"method-spec.md §11 no longer cites the seam test {name!r}"
|
||
for name in sorted(cited):
|
||
assert (REPO_ROOT / "tests" / name).is_file(), f"§11 cites a test this repo lacks: {name}"
|
||
|
||
|
||
def test_ingest_spec_exists_with_required_structure() -> None:
|
||
"""Test 4 (I1): the ingest spec exists in shared/ and carries its normative skeleton — the
|
||
contract sections both stacks implement the ingest layer from, incl. the verdict-layer
|
||
reservation (ingest-målbilde §3: manifest mapping may NEVER produce the promotion gate's
|
||
layer). RED before the artifact exists, when a required section is dropped, or when the
|
||
reservation stops being stated. The field-level cross-check (spec ↔ manifest contract code)
|
||
arrives with the I2 contracts, mirroring test 3."""
|
||
assert INGEST_SPEC_PATH.is_file(), "shared/ingest-spec.md missing (the I1 shared artifact)"
|
||
text = INGEST_SPEC_PATH.read_text(encoding="utf-8")
|
||
for marker in _INGEST_REQUIRED_MARKERS:
|
||
assert marker in text, f"ingest-spec.md is missing required section marker: {marker!r}"
|
||
# Normative, not descriptive: RFC-2119-style requirement language must be present.
|
||
assert "MUST" in text, "the ingest spec must be normative (RFC-2119 MUST language)"
|
||
# The verdict-layer reservation is load-bearing prose: both the reserved OKF type and the
|
||
# reserved filename namespace must be stated verbatim.
|
||
assert "`type: verdict`" in text, "ingest spec must state the reserved verdict OKF type"
|
||
assert "promoted-verdict-" in text, "ingest spec must state the reserved filename namespace"
|
||
# The determinism anchor: the explicit-timestamp rule must be visible at field level.
|
||
assert "`ingested_at`" in text, "ingest spec must document the explicit timestamp field"
|
||
|
||
|
||
def test_ingest_spec_documents_every_contract_field() -> None:
|
||
"""Test 5 (I2 — discharges test 4's deferred note): every field of the REAL ingest manifest
|
||
contract is documented in the ingest spec — driven from the pydantic models' ``model_fields``
|
||
(all three polymorphic source variants + the extraction shape), the seven §5/§7 provenance
|
||
frontmatter keys, and the §11 golden-case entries. Mirrors test 3: the test goes RED the
|
||
moment the contract code and the spec's §12 cross-check table drift apart (a new model field
|
||
without a spec entry, a renamed frontmatter key, a changed golden layout)."""
|
||
text = INGEST_SPEC_PATH.read_text(encoding="utf-8")
|
||
|
||
def documented(field: str, source: str) -> None:
|
||
assert f"`{field}`" in text, f"ingest spec does not document {source} field `{field}`"
|
||
|
||
for field in (f.name for f in dataclasses.fields(ManifestV1)):
|
||
documented(field, "manifest top-level")
|
||
for model, source in (
|
||
(FileSource, "file source"),
|
||
(SqlSource, "sql source"),
|
||
(HttpSource, "http source"),
|
||
(Extraction, "extraction"),
|
||
):
|
||
for field in (f.name for f in dataclasses.fields(model)):
|
||
documented(field, source)
|
||
# The library's source models consume the `type` discriminator during validation dispatch
|
||
# instead of storing it as a field, so it is no longer reachable by introspection. It is a
|
||
# REAL §4 contract field, so it is asserted explicitly — without this line the swap from
|
||
# `model_fields` to `dataclasses.fields` would silently drop it from the cross-check.
|
||
documented("type", "source discriminator")
|
||
|
||
# The §5/§7 provenance layer — exactly the keys the materializer stamps.
|
||
for key in (
|
||
"type",
|
||
"title",
|
||
"source_system",
|
||
"source_query",
|
||
"ingested_at",
|
||
"ingest_manifest",
|
||
"generated",
|
||
):
|
||
documented(key, "provenance frontmatter")
|
||
|
||
# The §11 golden extraction case layout.
|
||
for entry in ("manifest.json", "fixture/", "ingested-at.txt", "expected-bundle/"):
|
||
documented(entry, "golden extraction case")
|