portfolio-optimiser/tests/test_exploration_artefact_approaches_loadbearing.py
Kjell Tore Guttormsen c84e8bf6f1 feat(p19): a direction must NAME the requirement that binds it, and have READ it
Two paid rounds scored 0 of 26 fasit concepts opened -- the same number twice.
P18 closed the navigation side (a listing is a window, an invented path is
refused by name) and it did not move, which makes it a ROLE question: nothing
in the loop ever asked the model to say what requirement binds the direction it
committed to, so opening one was never on the critical path to an answer.

A PREMISE OF THE ORDER WAS FELLED BEFORE ANYTHING WAS BUILT ON IT. A1 places
the demand in _INSTRUCTIONS[HYPOTHESISER_ROLE] alone. Measured: the stress
command sends --mandate and NOT --explore, the two are refused together by
name, and none of the nine round-1/2 outboxes holds a {run_id}-exploration.json
-- the hypothesiser never runs in a stress round, so A3 would have been
unreachable in exactly the paid runs this order commissions.

A2's own sentence resolves it: the refusal goes to the model "som en tur den
kan rette (samme mekanisme som quick_validate's nekt), ikke som en raise" --
and quick_validate IS a tool. declare_requirement therefore lives in
navigator_tools, held by BOTH roles that navigate (the exploration, and since
S2c the debate). It EXISTS only when the caller offers both sinks, which keeps
every pre-P19 call site byte-identical; one sink without the other is refused
at construction. 'opened' is the SAME list ExplorationToolRecorder fills, so
the refusal reads the run's own read trace.

The marked hypothesis carries 'requirement' as a REQUIRED key: omitted is a
hard error, explicit null is legal and needs 'why_none', a half-named one is
refused. A minted approach carries it; a seed never acquires one. The proposer
prompt names it only when the field exists, and the judge counts a hit against
THIS approach's fasit concepts, never against the base.

Load-bearing measured (12 arms), four mutations all red against the whole
suite, green control 1711/5 and demo-transcript.stdout byte-unchanged.
A-iii's predicted signature was FALSIFIED: the golden stays green because the
demo runs without a mandate, so _build_messages' approach branch is never
taken there. A-iv was GREEN first -- the repo's vacuous-gate class, 24th time:
the arm drove _attributable while the hit is computed at the call site.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-15 01:21:47 +02:00

179 lines
6.9 KiB
Python

"""Fiks-ordre 20260904T070930Z, punkt 2 — ``{run_id}-exploration.json`` must carry the approaches
the loop actually SHAPED, not just the mandate announcement on stdout.
**Measured** on K2's syretest (``docs/2026-09-04-syretest-s7b-k2.md § 3.6``): the artefact had
``rounds``/``tool_calls``/``plan_reviews``/``quick_validations``, while which approaches the loop
minted stood only in ``mandate.announce``'s stdout block — printed by ``run.py``, never written.
An operator reading the artefact days later (the whole point of the async U12 door) could not
recover what the run had decided to evaluate without also having kept the terminal's stdout.
The fix widens ``explore.trace_payload`` with a required ``mandate`` keyword: it renders
``mandate.approaches`` under an ``"approaches"`` key, using the SAME fields ``mandate.announce``
reads off ``Approach`` (id, label). This file proves the artefact and the stdout announcement
agree — not just that a key exists.
"""
from __future__ import annotations
import json
from collections.abc import Callable
from pathlib import Path
from typing import Any
from agent_framework import BaseChatClient
from portfolio_optimiser import explore, run
from portfolio_optimiser.explore import ExplorationTrace
from portfolio_optimiser.simulation import ScriptedChatClient
_BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
_PID = "BYGG-KONTOR-NORD"
_LABEL = "SENTINEL-APPROACHES-9f2a"
_RATIONALE = "because the fittings are old (sentinel rationale)"
_ENERGY_REPLY = (
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
)
_CONTRACT_JSON: dict[str, Any] = {
"max_rounds": 4,
"max_tokens": 100_000,
"max_stall_count": 2,
"max_reset_count": 1,
"max_plan_revisions": 0,
"enable_plan_review": False,
}
def _ledger_json(*, satisfied: bool, speaker: str = "hypothesiser") -> str:
return json.dumps(
{
"is_request_satisfied": {"reason": "r", "answer": satisfied},
"is_in_loop": {"reason": "r", "answer": False},
"is_progress_being_made": {"reason": "r", "answer": True},
"next_speaker": {"reason": "r", "answer": speaker},
"instruction_or_question": {"reason": "r", "answer": "Shape one hypothesis."},
}
)
def _manager_script(ledgers: list[str]) -> Callable[[str, str], str]:
def _select(blob: str, _role: str) -> str:
if "provide the final answer" in blob:
return "FINAL: exploration done."
if "pure JSON format" in blob:
return ledgers.pop(0) if ledgers else _ledger_json(satisfied=True)
if "went wrong on this last run" in blob:
return "PLAN-UPDATE: revised plan."
if "rewrite the following fact sheet" in blob:
return "FACTS-UPDATE: revised facts."
if "bullet-point plan" in blob:
return "PLAN: - ask the hypothesiser"
if "pre-survey" in blob:
return "FACTS: the bundle is anchored."
return "{}"
return _select
def _hypothesis_line(label: str, rationale: str) -> str:
return f"{explore.HYPOTHESIS_MARKER} " + json.dumps(
{
"label": label,
"rationale": rationale,
"requirement": None,
"why_none": "scripted rehearsal: no document was read",
}
)
def _factory(*, ledgers: list[str], hypothesiser: list[str]) -> Callable[[str], BaseChatClient]:
def factory(role: str) -> BaseChatClient:
if role == explore.MANAGER_ROLE:
return ScriptedChatClient(reply_selector=_manager_script(ledgers), role=role)
if role == explore.HYPOTHESISER_ROLE:
replies = list(hypothesiser)
def _hyp(_blob: str, _role: str) -> str:
return replies.pop(0) if replies else "nothing further."
return ScriptedChatClient(reply_selector=_hyp, role=role)
if role == explore.NAVIGATOR_ROLE:
return ScriptedChatClient("NAVIGATOR: index read.", role=role)
return ScriptedChatClient(_ENERGY_REPLY, role=role)
return factory
def _config_file(tmp_path: Path) -> str:
path = tmp_path / "exploration.json"
path.write_text(json.dumps(_CONTRACT_JSON), encoding="utf-8")
return str(path)
def _base_argv(tmp_path: Path) -> list[str]:
return [
_PID,
"--docs-dir",
str(_BUNDLE_DIR),
"--bundle-dir",
str(_BUNDLE_DIR),
"--explore",
"Find the cheapest saving.",
"--explore-config",
_config_file(tmp_path),
]
def test_the_artefact_names_the_same_approach_the_announcement_did(
tmp_path, capsys, monkeypatch
) -> None:
"""Goal test: the label ``mandate.announce`` printed on stdout is the SAME label the artefact
carries, under the SAME minted id — read back from disk, not scraped off stdout.
Detach point: drop the ``"approaches"`` key from ``trace_payload`` → ``KeyError`` here.
Detach point: render an empty list regardless of ``mandate`` → the id/label assertions go RED.
"""
factory = _factory(
ledgers=[_ledger_json(satisfied=False), _ledger_json(satisfied=True)],
hypothesiser=[_hypothesis_line(_LABEL, _RATIONALE)],
)
monkeypatch.setattr("portfolio_optimiser.run._default_factory", lambda profile: factory)
outbox = tmp_path / "outbox"
rc = run.main(_base_argv(tmp_path) + ["--outbox-dir", str(outbox), "--run-id", "r3"])
assert rc == 0, capsys.readouterr().err
out = capsys.readouterr().out
assert f"1. hypothesis-1 — {_LABEL}" in out, "the stdout announcement is the fixture's oracle"
payload = json.loads((outbox / "r3-exploration.json").read_text(encoding="utf-8"))
assert payload["approaches"] == [
{
"id": "hypothesis-1",
"label": _LABEL,
"description": _RATIONALE,
"affected_codes": [],
"claimed_saving_nok": None,
"bundle_id": "bygg-energi-mikro",
}
], (
"the artefact must name the same approach the terminal announced, or a reader without the terminal learns nothing"
)
def test_a_mandate_with_no_commissioned_approaches_renders_an_empty_but_present_list() -> None:
"""Control: ``mandate.approaches == ()`` (pure own-proposals) is a legitimate state and must
render as ``[]``, not be confused with "no mandate was ever formed" — that absence is what
``completed=False`` already says (``trace_payload``'s own docstring)."""
trace = ExplorationTrace()
from portfolio_optimiser.mandate import Mandate
empty_mandate = Mandate(objective="find something", approaches=())
payload = explore.trace_payload(trace, stop=None, completed=True, mandate=empty_mandate)
assert payload["approaches"] == []
payload_no_mandate = explore.trace_payload(trace, stop=None, completed=False, mandate=None)
assert payload_no_mandate["approaches"] == []