feat(simulation): K4 — closed-loop two-run simulation binds §11 'Closed loop' (closes R-1)
Scripted two-run driver over the run.py composition: run A -> persona verdict (shared skill artifact) -> §6 promotion gate -> run B on a fresh store. The marker crosses runs via the promoted wiki layer ONLY - run B reads no inbox, a rejected verdict is refused fail-closed and its marker never crosses. Two detach proofs delivered (promotion step removed -> red; verdict exclusion in bundle_context removed -> red via the '## verdict' section anchor). Known-limitation note (C-F5, deferred to C3.2): a persona verdict over the bundle seed's own candidate mints the seed's §4.2 id and is silently shadowed by first-write-wins; the test has run A propose a distinct candidate. 395 -> 400 tests; README synced (test count + the S10 section now reflects that D7 has its own scripted closed-loop proof). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
3587854074
commit
d4efdd9a35
3 changed files with 341 additions and 5 deletions
12
README.md
12
README.md
|
|
@ -13,7 +13,7 @@ human-in-the-loop, and the system learns from the verdicts.
|
||||||
> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
|
> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
|
||||||
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
|
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
|
||||||
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
|
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
|
||||||
> seam, each proven by load-bearing tests (395 tests, all running offline without an API
|
> seam, each proven by load-bearing tests (400 tests, all running offline without an API
|
||||||
> key). The programme's single budgeted **live model run has been executed and validated** —
|
> key). The programme's single budgeted **live model run has been executed and validated** —
|
||||||
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
|
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
|
||||||
|
|
||||||
|
|
@ -130,9 +130,11 @@ layer works and how one would extend it is documented in [`docs/extending.md`](d
|
||||||
|
|
||||||
## The live run — S10, executed and validated
|
## The live run — S10, executed and validated
|
||||||
|
|
||||||
Where the MAF sibling proves its loop end-to-end with a scripted offline simulation, this
|
The loop's closure is proven offline by a scripted two-run simulation
|
||||||
repo's end-to-end proof is the programme's single budgeted **real** run (D6: exactly one
|
(`simulation.py`: run A → persona verdict → §6 promotion gate → run B on a fresh store —
|
||||||
live API run in the whole programme), executed 2026-07-03 against the micro bundle
|
the marker crosses runs via the gate, and never without it). On top of that offline proof
|
||||||
|
sits the programme's single budgeted **real** run (D6: exactly one live API run in the
|
||||||
|
whole programme), executed 2026-07-03 against the micro bundle
|
||||||
[`shared/examples/bygg-energi-mikro/`](shared/examples/bygg-energi-mikro/):
|
[`shared/examples/bygg-energi-mikro/`](shared/examples/bygg-energi-mikro/):
|
||||||
|
|
||||||
- exit 0 · validator `validated` · checker `approve` on the first attempt · 2 of 12
|
- exit 0 · validator `validated` · checker `approve` on the first attempt · 2 of 12
|
||||||
|
|
@ -156,7 +158,7 @@ Python ≥3.10 · [`claude-agent-sdk`](https://pypi.org/project/claude-agent-sdk
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv sync # install dependencies
|
uv sync # install dependencies
|
||||||
uv run pytest # 395 tests — run without any API key and without network
|
uv run pytest # 400 tests — run without any API key and without network
|
||||||
uv run ruff check . && uv run ruff format --check .
|
uv run ruff check . && uv run ruff format --check .
|
||||||
uv run mypy src # strict
|
uv run mypy src # strict
|
||||||
```
|
```
|
||||||
|
|
|
||||||
152
src/portfolio_optimiser_claude/simulation.py
Normal file
152
src/portfolio_optimiser_claude/simulation.py
Normal file
|
|
@ -0,0 +1,152 @@
|
||||||
|
"""Closed-loop two-run simulation driver (method-spec §11 «Closed loop»; §3 Steps 7–8, §6).
|
||||||
|
|
||||||
|
Honesty rule (§1): this is a SCRIPTED stand-in, not a live experiment. The
|
||||||
|
driver makes no model calls of its own — both runs use injected ``ModelClient``
|
||||||
|
instances (scripted in the suite), and the persona plays the human expert from
|
||||||
|
the shared skill artifact. What it proves is the loop's closure: run A →
|
||||||
|
persona judgement → §6 promotion gate → run B with a FRESH store, where the
|
||||||
|
marker reaches run B's hypothesis prompt through the promoted wiki layer and
|
||||||
|
the gated fold ONLY. Run B deliberately reads NO inbox: the persona's raw
|
||||||
|
verdict is authored into the inbox (§3 Step 7 write side), but the sole
|
||||||
|
sanctioned crossing channel between runs is the gate — a rejected verdict is
|
||||||
|
refused there (fail-closed, recorded, nothing written) and its marker must
|
||||||
|
never cross. ``timestamp`` is an explicit required argument (no wall-clock
|
||||||
|
default) — the whole simulation is deterministic and reproducible.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from portfolio_optimiser_claude.budget import BudgetMeter
|
||||||
|
from portfolio_optimiser_claude.contracts import Contracts, load_contracts
|
||||||
|
from portfolio_optimiser_claude.experience import CandidateFeatures
|
||||||
|
from portfolio_optimiser_claude.inbox import VerdictDocument
|
||||||
|
from portfolio_optimiser_claude.loop import ModelClient, RunResult, run_project
|
||||||
|
from portfolio_optimiser_claude.persona import drop_persona_verdict
|
||||||
|
from portfolio_optimiser_claude.promotion import PromotionError, promote
|
||||||
|
from portfolio_optimiser_claude.run import ComposedRunContext, compose_run_context
|
||||||
|
|
||||||
|
_PERSONA_APPROVER = "persona:expert-reviewer"
|
||||||
|
_VERDICT_DESCRIPTION = "closed-loop simulation: persona judgement of run A's candidate"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SimulationRun:
|
||||||
|
"""One simulated run: the §5 composition that fed it plus the §3 result."""
|
||||||
|
|
||||||
|
composed: ComposedRunContext
|
||||||
|
result: RunResult
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ClosedLoopResult:
|
||||||
|
"""The two-run outcome: what run A produced, what the gate did, what run B saw.
|
||||||
|
|
||||||
|
Exactly one of ``promoted_path`` / ``promotion_refusal`` is set: the §6
|
||||||
|
gate either promoted the persona's verdict into the bundle or refused it
|
||||||
|
fail-closed (the refusal reason recorded, nothing written).
|
||||||
|
"""
|
||||||
|
|
||||||
|
run_a: SimulationRun
|
||||||
|
verdict: VerdictDocument
|
||||||
|
promoted_path: Path | None
|
||||||
|
promotion_refusal: str | None
|
||||||
|
run_b: SimulationRun
|
||||||
|
|
||||||
|
|
||||||
|
def _run_once(
|
||||||
|
client: ModelClient,
|
||||||
|
bundle_dir: Path,
|
||||||
|
contracts: Contracts,
|
||||||
|
*,
|
||||||
|
max_debate_rounds: int,
|
||||||
|
max_attempts: int,
|
||||||
|
) -> SimulationRun:
|
||||||
|
# Fresh store (inside compose) and fresh §8 meter per run — nothing crosses
|
||||||
|
# runs except what the promotion gate wrote into the bundle.
|
||||||
|
composed = compose_run_context(bundle_dir, None, k=contracts.data_source.top_k)
|
||||||
|
result = run_project(
|
||||||
|
client,
|
||||||
|
composed.context,
|
||||||
|
meter=BudgetMeter(contracts.termination),
|
||||||
|
max_debate_rounds=max_debate_rounds,
|
||||||
|
max_attempts=max_attempts,
|
||||||
|
default_project_id=composed.ir_projection.project_id,
|
||||||
|
)
|
||||||
|
return SimulationRun(composed=composed, result=result)
|
||||||
|
|
||||||
|
|
||||||
|
def simulate_closed_loop(
|
||||||
|
bundle_dir: Path,
|
||||||
|
persona_artifact: Path,
|
||||||
|
inbox_dir: Path,
|
||||||
|
*,
|
||||||
|
client_a: ModelClient,
|
||||||
|
client_b: ModelClient,
|
||||||
|
timestamp: str,
|
||||||
|
experiment: str,
|
||||||
|
k: int = 3,
|
||||||
|
max_rounds: int = 12,
|
||||||
|
max_tokens: int = 150_000,
|
||||||
|
max_debate_rounds: int = 3,
|
||||||
|
max_attempts: int = 3,
|
||||||
|
) -> ClosedLoopResult:
|
||||||
|
"""Drive the closed loop: run A → persona verdict → §6 gate → run B (fresh store).
|
||||||
|
|
||||||
|
The persona judges run A's actual candidate (§4.2: the verdict id is
|
||||||
|
minted from the produced proposal's features) and authors the verdict
|
||||||
|
through the inbox primitive. The gate then decides: an approved verdict is
|
||||||
|
promoted into the bundle's wiki layer; anything else is refused fail-closed
|
||||||
|
and recorded. Run B recomposes from the bundle alone — the promoted layer
|
||||||
|
is the ONLY channel a judgement may cross runs on.
|
||||||
|
"""
|
||||||
|
contracts = load_contracts(
|
||||||
|
data_source={"docs_dir": str(bundle_dir), "top_k": k},
|
||||||
|
termination={"max_rounds": max_rounds, "max_tokens": max_tokens},
|
||||||
|
feedback={"decision": "approved", "rationale": "startup shape check (§10)"},
|
||||||
|
)
|
||||||
|
# The navigated dir is the CONTRACT's, so the validated config is load-bearing.
|
||||||
|
bundle = Path(contracts.data_source.docs_dir)
|
||||||
|
run_a = _run_once(
|
||||||
|
client_a,
|
||||||
|
bundle,
|
||||||
|
contracts,
|
||||||
|
max_debate_rounds=max_debate_rounds,
|
||||||
|
max_attempts=max_attempts,
|
||||||
|
)
|
||||||
|
verdict = drop_persona_verdict(
|
||||||
|
inbox_dir,
|
||||||
|
persona_artifact,
|
||||||
|
CandidateFeatures.from_proposal(run_a.result.proposal),
|
||||||
|
description=_VERDICT_DESCRIPTION,
|
||||||
|
)
|
||||||
|
promoted_path: Path | None
|
||||||
|
promotion_refusal: str | None
|
||||||
|
try:
|
||||||
|
promoted_path = promote(
|
||||||
|
verdict,
|
||||||
|
bundle,
|
||||||
|
approved_by=_PERSONA_APPROVER,
|
||||||
|
experiment=experiment,
|
||||||
|
timestamp=timestamp,
|
||||||
|
)
|
||||||
|
promotion_refusal = None
|
||||||
|
except PromotionError as refusal:
|
||||||
|
promoted_path = None
|
||||||
|
promotion_refusal = str(refusal)
|
||||||
|
run_b = _run_once(
|
||||||
|
client_b,
|
||||||
|
bundle,
|
||||||
|
contracts,
|
||||||
|
max_debate_rounds=max_debate_rounds,
|
||||||
|
max_attempts=max_attempts,
|
||||||
|
)
|
||||||
|
return ClosedLoopResult(
|
||||||
|
run_a=run_a,
|
||||||
|
verdict=verdict,
|
||||||
|
promoted_path=promoted_path,
|
||||||
|
promotion_refusal=promotion_refusal,
|
||||||
|
run_b=run_b,
|
||||||
|
)
|
||||||
182
tests/test_simulation_loadbearing.py
Normal file
182
tests/test_simulation_loadbearing.py
Normal file
|
|
@ -0,0 +1,182 @@
|
||||||
|
"""Closed-loop two-run simulation — LOAD-BEARING (K4/R-1; method-spec §11 «Closed loop»).
|
||||||
|
|
||||||
|
The seam this file keeps alive: the persona's marker crosses from run A to
|
||||||
|
run B's hypothesis prompt VIA the §6 promotion gate — and NEVER crosses
|
||||||
|
without it. Both directions are bound: detach the promotion step from the
|
||||||
|
driver → the marker is absent from run B → red; render the verdict layer
|
||||||
|
into the read-context → the marker leaks around the gated fold → red (the
|
||||||
|
verdict-exclusion invariant, re-proven at simulation level).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from _scripted import ScriptedClient, reply
|
||||||
|
|
||||||
|
from portfolio_optimiser_claude.ir import load_validator_input
|
||||||
|
from portfolio_optimiser_claude.okf import bundle_context
|
||||||
|
from portfolio_optimiser_claude.persona import load_persona_example
|
||||||
|
from portfolio_optimiser_claude.simulation import ClosedLoopResult, simulate_closed_loop
|
||||||
|
|
||||||
|
SHARED_BUNDLE = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||||
|
SHARED_ARTIFACT = (
|
||||||
|
Path(__file__).resolve().parents[1]
|
||||||
|
/ "shared"
|
||||||
|
/ "skills"
|
||||||
|
/ "expert-reviewer"
|
||||||
|
/ "references"
|
||||||
|
/ "example-verdict.json"
|
||||||
|
)
|
||||||
|
# The shared artifact's own marker (realiseringsgrad=0.79) — distinct from the
|
||||||
|
# bundle seed's 0.82, so its presence in run B proves the PROMOTION crossing,
|
||||||
|
# not the pre-existing seeding path.
|
||||||
|
MARKER = load_persona_example(SHARED_ARTIFACT).marker
|
||||||
|
|
||||||
|
TIMESTAMP = "2026-07-17T04:00:00Z"
|
||||||
|
EXPERIMENT = "K4-closed-loop-sim"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def bundle(tmp_path: Path) -> Path:
|
||||||
|
import shutil
|
||||||
|
|
||||||
|
target = tmp_path / "bundle"
|
||||||
|
shutil.copytree(SHARED_BUNDLE, target)
|
||||||
|
return target
|
||||||
|
|
||||||
|
|
||||||
|
def _proposal_json(bundle: Path, *, measure_suffix: str = "") -> str:
|
||||||
|
proposal = load_validator_input(bundle).model_dump()
|
||||||
|
if measure_suffix:
|
||||||
|
proposal["measure"] = f"{proposal['measure']}{measure_suffix}"
|
||||||
|
return json.dumps(proposal)
|
||||||
|
|
||||||
|
|
||||||
|
def _scripted_client(proposal_json: str) -> ScriptedClient:
|
||||||
|
# One full run: debate round (proposer + approving checker) + generation.
|
||||||
|
return ScriptedClient(
|
||||||
|
replies=[reply("debate reasoning"), reply("VERDICT: APPROVE"), reply(proposal_json)]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _simulate(
|
||||||
|
bundle: Path, artifact: Path, inbox: Path
|
||||||
|
) -> tuple[ClosedLoopResult, ScriptedClient, ScriptedClient]:
|
||||||
|
# Run A proposes a DISTINCT candidate (measure variant): the persona's
|
||||||
|
# verdict then gets its own §4.2 id. Judging the bundle seed's own
|
||||||
|
# candidate would mint the seed's id, and the store's first-write-wins
|
||||||
|
# would silently shadow the promoted rationale behind the seed (the known
|
||||||
|
# C-F5 feature-keying limitation — deferred to C3.2, not worked around
|
||||||
|
# in src).
|
||||||
|
client_a = _scripted_client(_proposal_json(bundle, measure_suffix="-simulert-variant"))
|
||||||
|
client_b = _scripted_client(_proposal_json(bundle))
|
||||||
|
result = simulate_closed_loop(
|
||||||
|
bundle,
|
||||||
|
artifact,
|
||||||
|
inbox,
|
||||||
|
client_a=client_a,
|
||||||
|
client_b=client_b,
|
||||||
|
timestamp=TIMESTAMP,
|
||||||
|
experiment=EXPERIMENT,
|
||||||
|
)
|
||||||
|
return result, client_a, client_b
|
||||||
|
|
||||||
|
|
||||||
|
def _rejecting_artifact(tmp_path: Path, marker: str) -> Path:
|
||||||
|
path = tmp_path / "rejecting-example-verdict.json"
|
||||||
|
path.write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"decision": "rejected",
|
||||||
|
"marker": marker,
|
||||||
|
"rationale": f"Avvist: driftstimene er overestimert ({marker}).",
|
||||||
|
}
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
class TestMarkerCrossesViaPromotion:
|
||||||
|
"""§11 «Closed loop», direction 1: WITH promotion the marker MUST cross."""
|
||||||
|
|
||||||
|
def test_the_persona_marker_reaches_run_b_via_promotion(
|
||||||
|
self, bundle: Path, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
result, _, client_b = _simulate(bundle, SHARED_ARTIFACT, tmp_path / "inbox")
|
||||||
|
# The marker did not exist anywhere before the persona judged run A.
|
||||||
|
assert MARKER not in result.run_a.composed.context
|
||||||
|
# The gate promoted the approved verdict into the bundle...
|
||||||
|
assert result.promoted_path is not None and result.promoted_path.is_file()
|
||||||
|
assert result.promotion_refusal is None
|
||||||
|
# ...and run B's FRESH store picked it up through seeding into the fold.
|
||||||
|
assert result.run_b.composed.seeded == result.run_a.composed.seeded + 1
|
||||||
|
assert MARKER in result.run_b.composed.context
|
||||||
|
# Detach proof 1 anchor: the marker reaches run B's actual prompts.
|
||||||
|
assert any(MARKER in prompt for prompt in client_b.prompts("proposer"))
|
||||||
|
|
||||||
|
def test_both_runs_complete_against_the_scripted_clients(
|
||||||
|
self, bundle: Path, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
result, _, _ = _simulate(bundle, SHARED_ARTIFACT, tmp_path / "inbox")
|
||||||
|
assert result.run_a.result.validator_decision == "validated"
|
||||||
|
assert result.run_b.result.validator_decision == "validated"
|
||||||
|
# The persona judged run A's candidate — the verdict carries its rationale.
|
||||||
|
assert MARKER in result.verdict.rationale
|
||||||
|
|
||||||
|
|
||||||
|
class TestMarkerNeverCrossesWithoutPromotion:
|
||||||
|
"""§11 «Closed loop», direction 2: WITHOUT promotion the marker NEVER crosses.
|
||||||
|
|
||||||
|
The rejected persona verdict IS authored into the inbox — if run B read the
|
||||||
|
inbox (or any channel other than the promoted wiki layer), the marker would
|
||||||
|
cross anyway and this control goes red.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_a_rejected_verdict_is_refused_and_the_marker_stays_out(
|
||||||
|
self, bundle: Path, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
marker = "realiseringsgrad=0.11"
|
||||||
|
artifact = _rejecting_artifact(tmp_path, marker)
|
||||||
|
result, _, client_b = _simulate(bundle, artifact, tmp_path / "inbox")
|
||||||
|
assert result.promoted_path is None
|
||||||
|
assert result.promotion_refusal is not None
|
||||||
|
assert "rejected" in result.promotion_refusal
|
||||||
|
assert marker not in result.run_b.composed.context
|
||||||
|
assert all(marker not in prompt for prompt in client_b.prompts("proposer"))
|
||||||
|
|
||||||
|
def test_a_refused_promotion_leaves_the_bundle_untouched(
|
||||||
|
self, bundle: Path, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
before = {p.name: p.read_bytes() for p in bundle.iterdir()}
|
||||||
|
_simulate(
|
||||||
|
bundle, _rejecting_artifact(tmp_path, "realiseringsgrad=0.11"), tmp_path / "inbox"
|
||||||
|
)
|
||||||
|
after = {p.name: p.read_bytes() for p in bundle.iterdir()}
|
||||||
|
assert after == before
|
||||||
|
|
||||||
|
|
||||||
|
class TestGatedFoldIsTheOnlyChannel:
|
||||||
|
"""LOAD-BEARING (§11): the marker crosses ONLY via the gated experience fold.
|
||||||
|
|
||||||
|
Detach proof 2 anchor: render the verdict layer into the read-context and
|
||||||
|
the marker leaks into the base context around the fold — red.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_the_verdict_layer_is_never_rendered_into_either_context(
|
||||||
|
self, bundle: Path, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
result, _, _ = _simulate(bundle, SHARED_ARTIFACT, tmp_path / "inbox")
|
||||||
|
assert MARKER in result.run_b.composed.context # crossing happened...
|
||||||
|
# ...but NO verdict-typed concept is rendered into the read-context —
|
||||||
|
# a `## verdict` section in either run's context means the seed's (or
|
||||||
|
# a promoted file's) body leaked around the gated fold (§3 Step 1).
|
||||||
|
assert "## verdict" not in result.run_a.composed.context
|
||||||
|
assert "## verdict" not in result.run_b.composed.context
|
||||||
|
# Belt and braces: the promoted marker itself never surfaces via
|
||||||
|
# rendering (promoted files are frontmatter-only by design, §6).
|
||||||
|
assert MARKER not in bundle_context(bundle)
|
||||||
Loading…
Add table
Add a link
Reference in a new issue