open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(explore): den offline generalproeven kan AAPNE en base (MAJOR-1 b, ORDRE 20260902T151931Z)

Browse source 
`--scripted-replies` tok EN konstant streng per rolle, saa ingen skriptet rolle
kunne emittere et `function_call`: maalt 0 verktoeykall / 0 approaches / 1 runde
paa 4/4 baser, mens hvert andre felt i artefaktet saa ut som en kjoering som
hadde virket. En operatoer som foelger dette repoets egen maalestige — «bevis saa
mye som mulig gratis foer det dyre trinnet» — kunne ikke bevise at navigatoeren
aapner noe.

En UTFORSKNINGSROLLE kan naa faa en LISTE av trinn, der et trinn er en tekst
eller ETT verktoeykall (`{"call": "<tool>", "args": {...}}`). Formen utvider den
ENE kanoniske `_inner_get_response`-kroppen (S2.5-konsolideringen) — ikke en
andre klient: rapportens vedlegg-A-`RecordingClient` var maaleinstrument, og en
kopi av kroppen ville drevet fra de tre conftest-dublettene som rir paa den.
Manuset ligger VED SIDEN AV selektoren, ikke inni: en selektor returnerer `str`
ved kontrakt, og et verktoeykall er ikke tekst. Naar manuset er tomt svarer
selektoren som foer, saa et utloept manus degraderer til konstantformen, ikke til
stillhet.

Listeformen NEKTES for debattens roller ved navn (proposeren svarer `generate`s
eget kall, ikke en agent-loekke som kan kalle et verktoey mellom turer), og hvert
malformet trinn nektes ved navn — validering, ALDRI reparasjon
(`write_concept_file`-regelen). Et trinn som slipper gjennom naar
`step["call"]` og krasjer midt i kjoeringen, som er nettopp MAJOR-2-klassen
filas foerste halvdel lukket.

MAALT: ni mutasjoner, alle roede mot HELE suiten, hver med sitt eget vitne +
groenn kontroll 2 failed (F15-diffen, KJENT) / 1087 passed / 5 skipped:
M1 detach recorderen fra explore() OG resume (1 roed) · M2 dropp `tool_calls`
fra `trace_payload` (3) · M3 sorter+dedupliser sinken (2) · M4 registrer navnet
uten basen (3) · M5 nekt listeformen igjen (2) · M6 aksepter lista men emitter
tekst (1) · M7 toler et malformet trinn (1) · M8 la en debattrolle ta
listeformen (1) · M9 dropp ukjent-noekkel-whitelisten (1). Golden
`demo-transcript.stdout` BYTE-UENDRET (ea8c534773acdbe41ae68f2c55724d69aaf8be4f).

M7 FALSIFISERTE TESTEN FOERST (repoets vakuoes-gate-klasse, TOLVTE gang):
nekt-testen brukte `{"invoke": "list_bundles"}` og sto GROENN med forms-sjekken
detached — UKJENT-NOEKKEL-grenen fanget den i stedet, saa den nekten som var
under test hadde intet vitne. De to grenene oeves naa av to armer med hver sin
mutasjon (M7 og M9).

Aerlighetsgrense, uttalt: at en LEVENDE modell kaller verktoeyene er fortsatt
ikke bevist (samme klasse som structured-output-grensen) — dette gjoer den
GRATIS halvdelen av stigen ekte, ikke den betalte.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-02 18:45:53 +02:00
commit 2654219e8c
3 changed files with 267 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

181
tests/test_scripted_explore_door_loadbearing.py
View file

@ -281,3 +281,184 @@ def test_the_record_leaves_the_run_in_the_artefact_beside_the_advisory_verdicts(
"the artefact must say which bases were opened, or a paid run leaves no record that any were"
)
assert json.dumps(payload), "the payload must stay plain data — the RAW layer is MAF-free"
# ---------------------------------------------------------------------------------------------
# MAJOR-1 (b) — THE SCRIPT CAN NOW CALL A TOOL, AND THE END-TO-END GATE OVER BOTH HALVES
# ---------------------------------------------------------------------------------------------
_BUNDLE_ID = _BUNDLE_DIR.name
def _ledger(*, satisfied: bool, speaker: str = "navigator") -> str:
return json.dumps(
{
"is_request_satisfied": {"reason": "r", "answer": satisfied},
"is_in_loop": {"reason": "r", "answer": False},
"is_progress_being_made": {"reason": "r", "answer": True},
"next_speaker": {"reason": "r", "answer": speaker},
"instruction_or_question": {"reason": "r", "answer": "open a base"},
}
)
#: The manager answered as a STEP LIST — one entry per stage, in the order the orchestrator asks
#: (facts, plan, ledger, ledger, final answer). This is the same measured reason
#: ``simulation._exploration_manager_reply`` keys on the PROMPT rather than the project id: the
#: manager is asked five DIFFERENT questions and a single constant answers all of them with the
#: first round's ledger — which is why the pre-existing constant-reply scenario concludes after one
#: round having given no participant a turn (measured here). Text steps, no calls: the manager
#: carries no tools.
_MANAGER_STAGES = [
"FACTS: the base is anchored.",
"PLAN: - let the navigator open a base",
_ledger(satisfied=False),
_ledger(satisfied=True),
"FINAL: the exploration is done.",
]
_HYPOTHESIS = "HYPOTHESIS: " + json.dumps({"label": "x", "rationale": "y"})
def _artefact(tmp_path: Path, run_id: str) -> dict[str, Any]:
path = tmp_path / "outbox" / f"{run_id}-exploration.json"
assert path.exists(), "the exploration artefact must be written even when the run failed"
return json.loads(path.read_text(encoding="utf-8"))
def _explore_argv(tmp_path: Path, replies_path: str, run_id: str) -> list[str]:
return [
*_base_argv(tmp_path, replies_path),
"--outbox-dir",
str(tmp_path / "outbox"),
"--run-id",
run_id,
]
def test_a_scripted_navigator_can_open_a_base_and_the_artefact_says_which(tmp_path) -> None:
"""THE GOAL, end to end and over BOTH halves: a script of
``list_bundles -> read_bundle -> text`` makes the navigator actually call the tools, and the
artefact an operator reads afterwards names which base was opened.
RED without (b): a step LIST is refused at the door, so the run never starts. RED without (a):
the tools run and ``tool_calls`` is empty — indistinguishable from the vacuous rehearsal this
whole section exists to make visible.
"""
replies = _replies_file(
tmp_path,
{
"proposer": _PROPOSER_REPLY,
"checker": "VERDICT: APPROVE",
"manager": list(_MANAGER_STAGES),
"navigator": [
{"call": "list_bundles"},
{"call": "read_bundle", "args": {"bundle_id": _BUNDLE_ID}},
"NAVIGATOR: read the index.",
],
"hypothesiser": _HYPOTHESIS,
},
)
rc = run.main(_explore_argv(tmp_path, replies, "scripted-tools"))
assert rc in (0, 1), rc
calls = _artefact(tmp_path, "scripted-tools")["tool_calls"]
assert [c["name"] for c in calls][:2] == ["list_bundles", "read_bundle"], (
"an offline rehearsal must be able to prove the navigator OPENS something — that is the "
f"whole of the free half of the measurement ladder; got {calls!r}"
)
assert calls[1]["bundle_id"] == _BUNDLE_ID, (
"the name alone answers 'a tool ran'; WHICH base it was asked for is what tells an "
"operator the navigator went where it was pointed"
)
def test_a_constant_string_script_records_no_tool_calls(tmp_path) -> None:
"""The CONTROL, and the measurement MAJOR-1 reported: one constant string per role cannot
emit a ``function_call``, so the rehearsal calls nothing. The recorder must SAY so rather than
invent entries — a record that filled itself in would be worth less than none.
"""
replies = _replies_file(
tmp_path,
{
"proposer": _PROPOSER_REPLY,
"checker": "VERDICT: APPROVE",
"manager": list(_MANAGER_STAGES),
"navigator": "NAVIGATOR: read the index.",
"hypothesiser": _HYPOTHESIS,
},
)
rc = run.main(_explore_argv(tmp_path, replies, "scripted-constant"))
assert rc in (0, 1), rc
assert _artefact(tmp_path, "scripted-constant")["tool_calls"] == [], (
"a constant-reply rehearsal opens nothing, and the artefact must be able to say that"
)
def _refuse(tmp_path, capsys, step: Any, run_id: str) -> str:
replies = _replies_file(
tmp_path,
{
"proposer": _PROPOSER_REPLY,
"checker": "VERDICT: APPROVE",
"manager": list(_MANAGER_STAGES),
"navigator": [step],
"hypothesiser": _HYPOTHESIS,
},
)
rc = run.main(_explore_argv(tmp_path, replies, run_id))
err = capsys.readouterr().err
assert rc == 1, err
assert "run refused" in err and "navigator" in err, err
return err
def test_a_step_that_names_no_tool_is_refused_by_name(tmp_path, capsys) -> None:
"""Validation, NEVER repair (the ``write_concept_file`` rule). A step that forgot to say WHAT
to call is refused at the door — letting it through would reach ``step["call"]`` inside the
client and crash mid-run with a raw ``KeyError``, which is precisely the MAJOR-2 failure the
first half of this file exists to have closed.
**This arm was FALSIFIED first** (this repo's vacuous-gate class, twelfth time). It originally
used ``{"invoke": "list_bundles"}``, and stayed GREEN with the shape check detached — the
UNKNOWN-KEY branch below caught it instead, so the refusal under test had no witness at all.
The two branches are now exercised by two arms with two mutations.
"""
_refuse(tmp_path, capsys, {"args": {"bundle_id": _BUNDLE_ID}}, "scripted-no-call")
def test_a_step_naming_an_unknown_key_is_refused_by_name(tmp_path, capsys) -> None:
"""The whitelist half (the hosted surface's ``_ALLOWED_FIELDS`` precedent, one layer down): a
key a step does not carry is refused BY NAME, never dropped in silence. An operator who wrote
``result`` expecting the rehearsal to assert something would otherwise be told nothing.
"""
err = _refuse(
tmp_path, capsys, {"call": "list_bundles", "result": "[]"}, "scripted-unknown-key"
)
assert "result" in err, err
def test_a_step_list_is_refused_for_a_debate_role(tmp_path, capsys) -> None:
"""The list form is the EXPLORATION's, and is refused elsewhere by name rather than accepted
and quietly ignored. The debate's proposer is driven by ``generate``'s own call, not by an
agent loop that would invoke a tool between turns, so a script of calls there would describe a
rehearsal that cannot happen.
"""
replies = _replies_file(
tmp_path,
{
"proposer": [{"call": "retrieve_cost_docs", "args": {"query": "x"}}],
"checker": "VERDICT: APPROVE",
"manager": list(_MANAGER_STAGES),
"navigator": "NAVIGATOR: read the index.",
"hypothesiser": _HYPOTHESIS,
},
)
rc = run.main(_explore_argv(tmp_path, replies, "scripted-debate-list"))
assert rc == 1
err = capsys.readouterr().err
assert "run refused" in err and "proposer" in err, err