Kjell Tore Guttormsen
a9144cb9bb
The primary method proof, offline — a deliberate, cost-driven substitution for målbilde §11.8's real-model run (the operator runs MAF against no real model; API for both repos is too costly privately). `portfolio_optimiser.simulation` drives `run_project` with a scripted synthetic chat client across two runs separated by a promotion, and shows the learning loop close end to end: - ScriptedChatClient subclasses the LAYERED OpenAIChatCompletionClient (not bare BaseChatClient — else the always-attached BudgetMiddleware no-ops), constructs offline (loopback url + dummy key), role-keys proposer/checker replies, and records every prompt into a shared sink. - simulate_learning_loop: Run A (fresh wiki) -> validated, persona-approved verdict carrying a realization marker absent from the bundle -> promote_verdict into the OKF wiki -> seed_store_from_bundle re-reads it -> Run B's hypothesis prompt carries the marker. An empty-wiki control on Run A proves causality. - `python -m portfolio_optimiser.simulation` prints an honest trace. Honesty (§1): this proves the plumbing, the deterministic spine, and that the learning dataflow closes — NOT that a live LLM would produce the proposal or verdict (scripted stand-ins). The genuine model-behaviour comparison lives on the Claude-SDK side (a minimal API run); the scripted client is MAF-side scaffolding, not part of the framework-neutral shared/ core. Load-bearing: tests/test_simulation_loadbearing.py goes red when promotion is detached (the marker never crosses into Run B). Suite 148->149. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MHR8iKxJRxDiDfNw8HZmWE
53 lines
3 KiB
Python
53 lines
3 KiB
Python
"""Offline simulation — the end-to-end method proof (replaces målbilde §11.8's real-model run).
|
|
|
|
Operator decision: MAF is NOT run against a real model (Azure/Foundry or local Ollama) — API for
|
|
both repos is too costly privately. Instead this offline simulation, driven by SCRIPTED synthetic
|
|
agent replies, is the primary proof that the agentic loop's dataflow closes end to end:
|
|
context -> hypothesis -> maker/checker debate -> validator -> verdict -> PROMOTION -> next run's
|
|
hypothesis. It proves the plumbing + the deterministic spine + that the learning loop closes; it
|
|
does NOT prove a live LLM would produce the proposal (that is scripted) — honesty per målbilde §1.
|
|
|
|
This load-bearing test runs the actual two-run cycle: Run A (fresh wiki) produces a validated,
|
|
persona-approved verdict carrying a realization marker absent from the bundle; `promote_verdict`
|
|
lifts it into the OKF wiki; a re-seed picks it up; Run B's hypothesis prompt then carries the
|
|
marker. The empty-store control (Run A carries no marker) proves causality — the signal reaches
|
|
Run B only via promotion. RED the moment promotion is detached.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
from portfolio_optimiser.simulation import simulate_learning_loop
|
|
from portfolio_optimiser.validator import ValidatedProposal
|
|
|
|
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
|
|
|
|
|
async def test_simulation_closes_the_learning_loop(tmp_path) -> None:
|
|
"""LOAD-BEARING: a persona verdict approved in Run A reaches Run B's hypothesis prompt purely
|
|
via the file-backed OKF wiki (promote -> re-seed -> ExpeL fold). Goes RED if ``promote_verdict``
|
|
is detached from ``simulate_learning_loop`` (Run B's store then lacks the marker)."""
|
|
result = await simulate_learning_loop(str(BUNDLE_DIR), str(tmp_path))
|
|
|
|
# The loop ran end to end on both runs (scripted proposal validates: P90=90000 >= 30000).
|
|
assert isinstance(result.run_a.outcome, ValidatedProposal)
|
|
assert isinstance(result.run_b.outcome, ValidatedProposal)
|
|
# A1: the synthetic client is layered, so the always-attached BudgetMiddleware metered real-shaped
|
|
# token usage (a bare BaseChatClient would silently no-op).
|
|
assert result.run_a.provenance.token_usage > 0
|
|
|
|
# The promoted verdict landed in the wiki.
|
|
assert result.promoted_path.exists()
|
|
assert result.promoted_path.name.startswith("promoted-verdict-")
|
|
|
|
# Causality control: Run A (empty wiki) carries no marker into its hypothesis prompt.
|
|
assert not result.marker_in_run_a_prompt, (
|
|
"Run A carried the marker with an empty wiki — the positive result would not be caused by "
|
|
"promotion"
|
|
)
|
|
# The learning loop closed: the promoted persona knowledge reached Run B's hypothesis.
|
|
assert result.marker_in_run_b_prompt, (
|
|
"the persona verdict approved in Run A did not reach Run B's hypothesis prompt — the "
|
|
"promote -> re-seed -> fold learning loop is not closed"
|
|
)
|