feat(explore): U4 kallsted 3 - demo-scenarioet, naabart ved NAVN og bare der (ORDRE 20260823T204216Z) [skip-docs]
Kallsted (3), siste av oerkt 57s fire. simulate_exploration er et TREDJE scenario ved siden av simulate_learning_loop, og bevisst ikke en del av det: demoens stdout og stderr er begge byte-pinnede fasiter, og laeringsgjennomgangens paastand (en dom krysser to kjoeringer) er en ANNEN paastand enn denne (en prompt + en kunnskapsbase blir et mandat pipelinen evaluerer). Aa slaa dem sammen ville flyttet et pinnet transkript av en grunn som ikke har noe med det transkriptet pinner aa gjoere. main() kaller det ikke - og at golden-transkriptet er byte-uendret ETTER at scenarioet ble lagt til er selve maalingen av det (ea8c534773acdbe41ae68f2c55724d69aaf8be4f). VAKUITETS-VAKTEN er den baerende delen: en label kunnskapsbasen ALLEREDE oppgir ville naadd hypotese-prompten som ordinaer navigert kontekst enten utforskningen kjoerte eller ei, saa scenarioets egen assert ville holdt mot en implementasjon som aldri wiret mandatet. Refusert, ikke demonstrert - noeyaktig samme vakt simulate_learning_loop raiser paa naar de to markoerene faller sammen. Manager-manuset noekles paa PROMPT-STADIET, ikke paa prosjekt-ID-en, og det er ikke et unntak fra scripted_proposer-regelen: manageren faar FEM ulike spoersmaal (fakta, plan, progress ledger, replan, sluttsvar), og hvilket det er er det eneste et svar KAN noekles paa - prosjekt-ID-en er konstant over alle fem og ville valgt ingenting. Aerlighets-grense, samme som resten av demoen (maalbilde §1): hvert svar er skriptet, saa det som vises er at roerleggingen lukkes - at en formet retning faktisk blir en Approach proposeren blir spurt om - aldri at en levende modell ville formet en god en. Load-bearing MAALT (tests/test_explore_callsites_loadbearing.py, 2 nye tester), to mutasjoner begge roede mot HELE suiten + groenn kontroll 998/5: detach mandate= fra scenarioets run_project (1 roed) - detach vakuitets-vakten (1 roed). mypy + ruff rene. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YRZhBJcxqTcqWyMW6hBttx
This commit is contained in:
parent
234c8d138a
commit
118eabf9db
2 changed files with 247 additions and 1 deletions
|
|
@ -39,6 +39,16 @@ from agent_framework import (
|
|||
from agent_framework_openai import OpenAIChatCompletionClient
|
||||
|
||||
from portfolio_optimiser import okf
|
||||
from portfolio_optimiser.explore import (
|
||||
HYPOTHESIS_MARKER,
|
||||
HYPOTHESISER_ROLE,
|
||||
MANAGER_ROLE,
|
||||
NAVIGATOR_ROLE,
|
||||
ExplorationContract,
|
||||
ExplorationResult,
|
||||
ExplorationTrace,
|
||||
explore,
|
||||
)
|
||||
from portfolio_optimiser.ir import AffectedItem, CostBaseline, CostBaselineLine, SavingsProposal
|
||||
from portfolio_optimiser.persona import load_persona_example
|
||||
from portfolio_optimiser.run import RunResult, run_project
|
||||
|
|
@ -632,6 +642,198 @@ async def simulate_learning_loop(
|
|||
)
|
||||
|
||||
|
||||
#: The U4 walkthrough's prompt and the direction its scripted hypothesiser commits to. The LABEL
|
||||
#: must be absent from the knowledge base — checked at call time, refused rather than assumed —
|
||||
#: because a label the base already states would reach the hypothesis prompt as ordinary navigated
|
||||
#: context, and the scenario would demonstrate nothing (``simulate_learning_loop``'s two-marker
|
||||
#: guard, in the one form this scenario can go vacuous).
|
||||
_EXPLORE_PROMPT = "Finn den rimeligste besparelsen som kan testes i denne kunnskapsbasen."
|
||||
_EXPLORE_LABEL = "styring etter tilstedevaerelse"
|
||||
_EXPLORE_RATIONALE = (
|
||||
"kunnskapsbasen beskriver armaturer som staar paa hele driftsdoegnet, saa en styringsgevinst "
|
||||
"kan testes mot de samme kostlinjene som selve retrofitten"
|
||||
)
|
||||
#: The walkthrough's stated bounds. Written out rather than defaulted for the reason
|
||||
#: ``ExplorationContract`` has no defaults at all: the demo is the one place a reader looks to see
|
||||
#: what bounding an exploration actually looks like.
|
||||
_EXPLORE_CONTRACT = ExplorationContract(
|
||||
max_rounds=4,
|
||||
max_tokens=100_000,
|
||||
max_stall_count=1,
|
||||
max_reset_count=1,
|
||||
max_plan_revisions=0,
|
||||
enable_plan_review=False,
|
||||
)
|
||||
|
||||
|
||||
def _exploration_manager_reply(ledgers: Sequence[str]) -> Callable[[str, str], str]:
|
||||
"""Route a manager prompt to its scripted reply by STAGE, consuming ``ledgers`` in order.
|
||||
|
||||
Keyed on the orchestrator's own prompt text rather than on the project id (the
|
||||
``scripted_proposer`` rule), and that is not an exception to it: the manager is asked five
|
||||
DIFFERENT questions — facts, plan, progress ledger, replan, final answer — and which one it is
|
||||
being asked is the only thing a reply can be keyed on. The project id is constant across all
|
||||
five and would select nothing."""
|
||||
remaining = list(ledgers)
|
||||
|
||||
def _select(blob: str, _role: str) -> str:
|
||||
if "provide the final answer" in blob:
|
||||
return "FINAL: utforskningen er ferdig."
|
||||
if "pure JSON format" in blob:
|
||||
return remaining.pop(0) if remaining else _exploration_ledger(satisfied=True)
|
||||
if "went wrong on this last run" in blob:
|
||||
return "PLAN-UPDATE: revidert plan."
|
||||
if "rewrite the following fact sheet" in blob:
|
||||
return "FACTS-UPDATE: reviderte fakta."
|
||||
if "bullet-point plan" in blob:
|
||||
return "PLAN: - la hypotesiseren forme en retning"
|
||||
if "pre-survey" in blob:
|
||||
return "FACTS: kunnskapsbasen er forankret."
|
||||
return "{}"
|
||||
|
||||
return _select
|
||||
|
||||
|
||||
def _exploration_ledger(*, satisfied: bool) -> str:
|
||||
"""One progress ledger naming a REAL participant. The name is what makes the run non-vacuous:
|
||||
a ``next_speaker`` matching nobody makes the orchestrator answer having asked no one."""
|
||||
return json.dumps(
|
||||
{
|
||||
"is_request_satisfied": {"reason": "r", "answer": satisfied},
|
||||
"is_in_loop": {"reason": "r", "answer": False},
|
||||
"is_progress_being_made": {"reason": "r", "answer": True},
|
||||
"next_speaker": {"reason": "r", "answer": HYPOTHESISER_ROLE},
|
||||
"instruction_or_question": {"reason": "r", "answer": "Form én retning."},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def scripted_exploration_factory(
|
||||
sink: list[str], *, label: str, rationale: str, ledgers: Sequence[str]
|
||||
) -> Callable[[str], BaseChatClient]:
|
||||
"""A role-keyed client factory covering BOTH the exploration's roles and the pipeline's.
|
||||
|
||||
One factory, because the walkthrough is one continuous story: the same call that shapes the
|
||||
mandate hands it to the run that evaluates it. The pipeline's roles fall through to the
|
||||
existing ``_proposer_reply`` / ``_CHECKER_APPROVE`` scaffolding, so nothing about the debate
|
||||
changes."""
|
||||
pipeline = scripted_factory({"proposer": _proposer_reply, "checker": _CHECKER_APPROVE}, sink)
|
||||
hypothesis_line = f"{HYPOTHESIS_MARKER} " + json.dumps({"label": label, "rationale": rationale})
|
||||
|
||||
def factory(role: str) -> BaseChatClient:
|
||||
if role == MANAGER_ROLE:
|
||||
return ScriptedChatClient(
|
||||
sink=sink, role=role, reply_selector=_exploration_manager_reply(ledgers)
|
||||
)
|
||||
if role == HYPOTHESISER_ROLE:
|
||||
return ScriptedChatClient(hypothesis_line, sink, role=role)
|
||||
if role == NAVIGATOR_ROLE:
|
||||
return ScriptedChatClient("NAVIGATOR: leste indeksen.", sink, role=role)
|
||||
return pipeline(role)
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExplorationSimulationResult:
|
||||
"""The trace of one U4 walkthrough: what the loop shaped, and whether the pipeline used it.
|
||||
|
||||
``label_in_bundle`` is the causality control carried in the result, exactly as
|
||||
``marker_in_run_a_prompt`` is: without it, a label the base already states would look like a
|
||||
shaped direction and the whole scenario would prove nothing."""
|
||||
|
||||
exploration: ExplorationResult
|
||||
trace: ExplorationTrace
|
||||
run: RunResult
|
||||
label: str
|
||||
label_in_bundle: bool
|
||||
label_in_generation_prompt: bool
|
||||
generation_prompts: list[str]
|
||||
|
||||
|
||||
async def simulate_exploration(
|
||||
bundle_dir: str,
|
||||
work_dir: str,
|
||||
*,
|
||||
project_id: str = _PROJECT_ID,
|
||||
prompt: str = _EXPLORE_PROMPT,
|
||||
label: str = _EXPLORE_LABEL,
|
||||
rationale: str = _EXPLORE_RATIONALE,
|
||||
contract: ExplorationContract | None = None,
|
||||
max_rounds: int = 3,
|
||||
) -> ExplorationSimulationResult:
|
||||
"""Walk U4 offline: prompt + knowledge base -> ``Mandate`` -> the pipeline that evaluates it.
|
||||
|
||||
A THIRD scenario beside ``simulate_learning_loop``, and deliberately not part of it. The
|
||||
demo's stdout and stderr are both byte-pinned fixtures (``tests/golden/demo-transcript.*``),
|
||||
and the learning walkthrough's claim — that a verdict crosses two runs — is a different claim
|
||||
from this one. Folding them together would make each harder to read and would move a pinned
|
||||
transcript for a reason unrelated to what it pins. This scenario is reachable by NAME only;
|
||||
``main()`` does not call it.
|
||||
|
||||
Honesty limit, the same one §1 states for the rest of the demo: every reply is scripted, so
|
||||
what is shown is that the plumbing closes — that a shaped direction really becomes an
|
||||
``Approach`` the proposer is asked about — never that a live model would shape a good one.
|
||||
"""
|
||||
copy = Path(work_dir) / "explore-bundle"
|
||||
shutil.copytree(bundle_dir, copy)
|
||||
copy_s = str(copy)
|
||||
|
||||
# The vacuity guard. A label the base already states would reach the hypothesis prompt as
|
||||
# navigated context whether or not the exploration ran, so the scenario's own assertion would
|
||||
# hold against an implementation that never wired the mandate at all.
|
||||
context = okf.bundle_context(okf.navigate_bundle(copy_s))
|
||||
if label in context:
|
||||
raise ValueError(
|
||||
f"label {label!r} already appears in the knowledge base, so it would reach the "
|
||||
"hypothesis prompt as ordinary context and the walkthrough would demonstrate nothing; "
|
||||
"choose a direction the base does not already state"
|
||||
)
|
||||
|
||||
sink: list[str] = []
|
||||
trace = ExplorationTrace()
|
||||
exploration = await explore(
|
||||
prompt,
|
||||
contract=contract or _EXPLORE_CONTRACT,
|
||||
bundle_dirs=(copy_s,),
|
||||
client_factory=scripted_exploration_factory(
|
||||
sink,
|
||||
label=label,
|
||||
rationale=rationale,
|
||||
ledgers=[_exploration_ledger(satisfied=False), _exploration_ledger(satisfied=True)],
|
||||
),
|
||||
trace=trace,
|
||||
)
|
||||
|
||||
run = cast(
|
||||
RunResult,
|
||||
await run_project(
|
||||
project_id,
|
||||
"local",
|
||||
docs_dir=copy_s,
|
||||
bundle_dir=copy_s,
|
||||
verdict_input={"decision": "approved", "rationale": "ekspert-persona (sim)"},
|
||||
store=VerdictStore(verdicts=[]),
|
||||
client_factory=scripted_exploration_factory(
|
||||
sink, label=label, rationale=rationale, ledgers=[]
|
||||
),
|
||||
mandate=exploration.mandate,
|
||||
max_rounds=max_rounds,
|
||||
),
|
||||
)
|
||||
|
||||
prompts = _generation_prompts(sink)
|
||||
return ExplorationSimulationResult(
|
||||
exploration=exploration,
|
||||
trace=trace,
|
||||
run=run,
|
||||
label=label,
|
||||
label_in_bundle=label in context,
|
||||
label_in_generation_prompt=any(label in p for p in prompts),
|
||||
generation_prompts=prompts,
|
||||
)
|
||||
|
||||
|
||||
def _outcome_line(result: RunResult) -> str:
|
||||
o = result.outcome
|
||||
if isinstance(o, ValidatedProposal):
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ import pytest
|
|||
from agent_framework import BaseChatClient
|
||||
|
||||
import portfolio_optimiser
|
||||
from portfolio_optimiser import explore, hosting, okf, run
|
||||
from portfolio_optimiser import explore, hosting, okf, run, simulation
|
||||
from portfolio_optimiser.budget import BudgetExceeded
|
||||
from portfolio_optimiser.explore import ExplorationContract, ExplorationTrace
|
||||
from portfolio_optimiser.mandate import Approach, Mandate
|
||||
|
|
@ -685,3 +685,47 @@ async def test_the_hosted_door_refuses_by_name_on_the_callers_channel(payload, e
|
|||
with pytest.raises(ValueError) as excinfo:
|
||||
await hosting.invoke(_hosted_payload(**payload))
|
||||
assert expected in str(excinfo.value)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 5. The demo scenario — a THIRD entry, reachable only by name
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_demo_scenario_lets_a_shaped_direction_reach_the_hypothesis(tmp_path) -> None:
|
||||
"""S1: the offline walkthrough of U4 — a prompt and a knowledge base become a mandate, and the
|
||||
direction the loop shaped reaches the proposer VERBATIM.
|
||||
|
||||
The same honesty limit the rest of the demo carries applies here and is worth restating: this
|
||||
proves the plumbing and that the data flow closes, NOT that a live model would shape a good
|
||||
direction. Every reply is scripted.
|
||||
|
||||
Detach point: drop ``mandate=`` from the scenario's ``run_project`` call → RED.
|
||||
"""
|
||||
result = await simulation.simulate_exploration(str(_BUNDLE_DIR), str(tmp_path), max_rounds=3)
|
||||
|
||||
assert result.label_in_bundle is False, "the control the scenario refuses on"
|
||||
assert [a.label for a in result.exploration.mandate.approaches] == [result.label]
|
||||
assert result.label_in_generation_prompt, (
|
||||
"the shaped direction never reached the hypothesis prompt — the demo would show a mandate "
|
||||
"the pipeline ignored"
|
||||
)
|
||||
assert result.trace.ledger, "the exploration recorded no rounds"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_direction_the_base_already_states_is_refused_as_vacuous(tmp_path) -> None:
|
||||
"""S2: a label the knowledge base ALREADY contains is refused, not demonstrated.
|
||||
|
||||
Exactly the guard ``simulate_learning_loop`` raises on when its two markers coincide: the
|
||||
scenario's whole claim is that the direction came from the LOOP, and a label the bundle states
|
||||
on its own would reach the prompt as ordinary context — a demonstration that demonstrates
|
||||
nothing, which is this repo's vacuous-gate class in demo form.
|
||||
|
||||
Detach point: drop the guard → RED.
|
||||
"""
|
||||
stated = "LED-retrofit" # present in the bundle's own text
|
||||
with pytest.raises(ValueError) as excinfo:
|
||||
await simulation.simulate_exploration(str(_BUNDLE_DIR), str(tmp_path), label=stated)
|
||||
assert stated in str(excinfo.value)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue