feat(explore): den offline generalproeven kan AAPNE en base (MAJOR-1 b, ORDRE 20260902T151931Z)
`--scripted-replies` tok EN konstant streng per rolle, saa ingen skriptet rolle
kunne emittere et `function_call`: maalt 0 verktoeykall / 0 approaches / 1 runde
paa 4/4 baser, mens hvert andre felt i artefaktet saa ut som en kjoering som
hadde virket. En operatoer som foelger dette repoets egen maalestige — «bevis saa
mye som mulig gratis foer det dyre trinnet» — kunne ikke bevise at navigatoeren
aapner noe.
En UTFORSKNINGSROLLE kan naa faa en LISTE av trinn, der et trinn er en tekst
eller ETT verktoeykall (`{"call": "<tool>", "args": {...}}`). Formen utvider den
ENE kanoniske `_inner_get_response`-kroppen (S2.5-konsolideringen) — ikke en
andre klient: rapportens vedlegg-A-`RecordingClient` var maaleinstrument, og en
kopi av kroppen ville drevet fra de tre conftest-dublettene som rir paa den.
Manuset ligger VED SIDEN AV selektoren, ikke inni: en selektor returnerer `str`
ved kontrakt, og et verktoeykall er ikke tekst. Naar manuset er tomt svarer
selektoren som foer, saa et utloept manus degraderer til konstantformen, ikke til
stillhet.
Listeformen NEKTES for debattens roller ved navn (proposeren svarer `generate`s
eget kall, ikke en agent-loekke som kan kalle et verktoey mellom turer), og hvert
malformet trinn nektes ved navn — validering, ALDRI reparasjon
(`write_concept_file`-regelen). Et trinn som slipper gjennom naar
`step["call"]` og krasjer midt i kjoeringen, som er nettopp MAJOR-2-klassen
filas foerste halvdel lukket.
MAALT: ni mutasjoner, alle roede mot HELE suiten, hver med sitt eget vitne +
groenn kontroll 2 failed (F15-diffen, KJENT) / 1087 passed / 5 skipped:
M1 detach recorderen fra explore() OG resume (1 roed) · M2 dropp `tool_calls`
fra `trace_payload` (3) · M3 sorter+dedupliser sinken (2) · M4 registrer navnet
uten basen (3) · M5 nekt listeformen igjen (2) · M6 aksepter lista men emitter
tekst (1) · M7 toler et malformet trinn (1) · M8 la en debattrolle ta
listeformen (1) · M9 dropp ukjent-noekkel-whitelisten (1). Golden
`demo-transcript.stdout` BYTE-UENDRET (ea8c534773acdbe41ae68f2c55724d69aaf8be4f).
M7 FALSIFISERTE TESTEN FOERST (repoets vakuoes-gate-klasse, TOLVTE gang):
nekt-testen brukte `{"invoke": "list_bundles"}` og sto GROENN med forms-sjekken
detached — UKJENT-NOEKKEL-grenen fanget den i stedet, saa den nekten som var
under test hadde intet vitne. De to grenene oeves naa av to armer med hver sin
mutasjon (M7 og M9).
Aerlighetsgrense, uttalt: at en LEVENDE modell kaller verktoeyene er fortsatt
ikke bevist (samme klasse som structured-output-grensen) — dette gjoer den
GRATIS halvdelen av stigen ekte, ikke den betalte.
Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
parent
4aa4f9c429
commit
2654219e8c
3 changed files with 267 additions and 14 deletions
|
|
@ -1618,13 +1618,52 @@ _SCRIPTED_BANNER = (
|
|||
)
|
||||
|
||||
|
||||
def _validate_script(role: str, steps: Any, path: str) -> None:
|
||||
"""Refuse a malformed step script BY NAME (MAJOR-1 b). Validation, NEVER repair.
|
||||
|
||||
A step is either a plain reply (``str``) or one tool call
|
||||
(``{"call": "<tool>", "args": {...}}``). Skipping a step nobody could parse would run a
|
||||
rehearsal that proves less than the operator wrote while looking exactly like one that proved
|
||||
it — the failure mode this whole seam exists to make visible. An EMPTY list is refused for the
|
||||
same reason: it is the constant form written obscurely, and it would fall through to the
|
||||
default reply.
|
||||
"""
|
||||
if not isinstance(steps, list) or not steps:
|
||||
raise ValueError(f"--scripted-replies[{role!r}] must be a non-empty list of steps ({path})")
|
||||
for index, step in enumerate(steps):
|
||||
if isinstance(step, str):
|
||||
continue
|
||||
if not isinstance(step, dict) or not isinstance(step.get("call"), str):
|
||||
raise ValueError(
|
||||
f"--scripted-replies[{role!r}] step {index} is neither a text reply nor a tool "
|
||||
f'call {{"call": "<tool>", "args": {{...}}}} ({path})'
|
||||
)
|
||||
unknown = sorted(set(step) - {"call", "args"})
|
||||
if unknown:
|
||||
raise ValueError(
|
||||
f"--scripted-replies[{role!r}] step {index} names unknown key(s) "
|
||||
f"{', '.join(unknown)}; a step carries only 'call' and 'args' ({path})"
|
||||
)
|
||||
if step.get("args") is not None and not isinstance(step["args"], dict):
|
||||
raise ValueError(
|
||||
f"--scripted-replies[{role!r}] step {index}: 'args' must be an object ({path})"
|
||||
)
|
||||
|
||||
|
||||
def _load_scripted_replies(
|
||||
path: str, required_roles: Sequence[str] = _SCRIPTED_ROLES
|
||||
) -> dict[str, str]:
|
||||
) -> dict[str, Any]:
|
||||
"""Load the caller's scripted answers, fail-fast. Every role ``required_roles`` names must be
|
||||
present AND a string: a missing role would otherwise surface as a ``KeyError`` deep inside
|
||||
present AND readable: a missing role would otherwise surface as a ``KeyError`` deep inside
|
||||
``scripted_factory``'s lookup, mid-run, long after the run appeared to start cleanly (MAJOR-2:
|
||||
measured for the three ``explore()`` adds on top of the debate's own two)."""
|
||||
measured for the three ``explore()`` adds on top of the debate's own two).
|
||||
|
||||
**An EXPLORATION role may also be given a step LIST** (MAJOR-1 b), because a single constant
|
||||
string can never emit a ``function_call`` — measured: 0 tool calls / 0 approaches / 1 round on
|
||||
4/4 bases, an offline rehearsal that was vacuous by construction. The list form is refused for
|
||||
the debate's roles BY NAME rather than accepted and ignored: the proposer answers
|
||||
``generate``'s own call, not an agent loop that would invoke a tool between turns, so a script
|
||||
of calls there describes a rehearsal that cannot happen."""
|
||||
try:
|
||||
raw = json.loads(Path(path).read_text(encoding="utf-8"))
|
||||
except FileNotFoundError as exc:
|
||||
|
|
@ -1633,12 +1672,22 @@ def _load_scripted_replies(
|
|||
raise ValueError(f"--scripted-replies is not valid JSON ({path}): {exc}") from exc
|
||||
if not isinstance(raw, dict):
|
||||
raise ValueError(f"--scripted-replies must be a JSON object of role -> reply ({path})")
|
||||
missing = [r for r in required_roles if not isinstance(raw.get(r), str)]
|
||||
missing = [r for r in required_roles if not isinstance(raw.get(r), (str, list))]
|
||||
if missing:
|
||||
raise ValueError(
|
||||
f"--scripted-replies needs a string reply for each of {', '.join(required_roles)}; "
|
||||
f"missing or non-string: {', '.join(missing)} ({path})"
|
||||
f"--scripted-replies needs a reply for each of {', '.join(required_roles)}; "
|
||||
f"missing or of an unusable type: {', '.join(missing)} ({path})"
|
||||
)
|
||||
for role in required_roles:
|
||||
if isinstance(raw[role], str):
|
||||
continue
|
||||
if role not in _EXPLORATION_SCRIPTED_ROLES:
|
||||
raise ValueError(
|
||||
f"--scripted-replies[{role!r}] is a step list, but that form is the exploration's: "
|
||||
f"only {', '.join(_EXPLORATION_SCRIPTED_ROLES)} run inside an agent loop that can "
|
||||
f"invoke a tool between turns ({path})"
|
||||
)
|
||||
_validate_script(role, raw[role], path)
|
||||
return {role: raw[role] for role in required_roles}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -32,6 +32,7 @@ from agent_framework import (
|
|||
BaseChatClient,
|
||||
ChatResponse,
|
||||
ChatResponseUpdate,
|
||||
Content,
|
||||
Message,
|
||||
ResponseStream,
|
||||
UsageDetails,
|
||||
|
|
@ -399,6 +400,7 @@ class ScriptedChatClient(OpenAIChatCompletionClient):
|
|||
sink: list[str] | None = None,
|
||||
*,
|
||||
reply_selector: Callable[[str, str], str] | None = None,
|
||||
script: Sequence[str | Mapping[str, Any]] | None = None,
|
||||
role: str = "",
|
||||
default_reply: str = "ok",
|
||||
tokens_per_reply: int = 8,
|
||||
|
|
@ -415,6 +417,11 @@ class ScriptedChatClient(OpenAIChatCompletionClient):
|
|||
self._select = lambda _prompt, _role: reply
|
||||
else:
|
||||
self._select = lambda _prompt, _role: self._default
|
||||
# The step SCRIPT (MAJOR-1 b), consumed one entry per call and then exhausted — after
|
||||
# which the selector answers as before, so a script that runs out degrades to the constant
|
||||
# form rather than to silence. It lives BESIDE the selector, not inside it: a selector
|
||||
# returns ``str`` by contract, and a tool call is not text.
|
||||
self._script: list[str | Mapping[str, Any]] = list(script or ())
|
||||
self._tokens = tokens_per_reply
|
||||
self.call_count = 0
|
||||
|
||||
|
|
@ -430,22 +437,33 @@ class ScriptedChatClient(OpenAIChatCompletionClient):
|
|||
if self._sink is not None:
|
||||
self._sink.append(blob)
|
||||
self.call_count += 1
|
||||
reply = self._select(blob, self._role)
|
||||
usage = UsageDetails(total_token_count=self._tokens)
|
||||
step = self._script.pop(0) if self._script else None
|
||||
if step is None or isinstance(step, str):
|
||||
reply = self._select(blob, self._role) if step is None else step
|
||||
# The framework accepts a {"type": "text", ...} dict in an update (its types
|
||||
# under-specify it); a plain string is what the non-stream message has always carried.
|
||||
contents: list[Any] = [reply]
|
||||
update_contents: list[Any] = [{"type": "text", "text": reply}]
|
||||
else:
|
||||
call = Content(
|
||||
"function_call",
|
||||
call_id=f"scripted-{self.call_count}",
|
||||
name=str(step["call"]),
|
||||
arguments=dict(step.get("args") or {}),
|
||||
)
|
||||
contents = [call]
|
||||
update_contents = [call]
|
||||
if stream:
|
||||
|
||||
async def _agen() -> Any:
|
||||
# The framework accepts a {"type": "text", ...} dict here (its types under-specify it).
|
||||
yield ChatResponseUpdate(
|
||||
role="assistant",
|
||||
contents=[{"type": "text", "text": reply}], # type: ignore[list-item]
|
||||
)
|
||||
yield ChatResponseUpdate(role="assistant", contents=update_contents)
|
||||
|
||||
return self._build_response_stream(_agen())
|
||||
|
||||
async def _coro() -> ChatResponse:
|
||||
return ChatResponse(
|
||||
messages=[Message(role="assistant", contents=[reply])],
|
||||
messages=[Message(role="assistant", contents=contents)],
|
||||
response_id="synthetic",
|
||||
usage_details=usage,
|
||||
)
|
||||
|
|
@ -454,7 +472,7 @@ class ScriptedChatClient(OpenAIChatCompletionClient):
|
|||
|
||||
|
||||
def scripted_factory(
|
||||
replies: Mapping[str, str | Callable[[str, str], str]], sink: list[str]
|
||||
replies: Mapping[str, str | Sequence[Any] | Callable[[str, str], str]], sink: list[str]
|
||||
) -> Callable[[str], BaseChatClient]:
|
||||
"""A role-keyed client factory: ``factory("proposer")`` and ``factory("checker")`` each return a
|
||||
fresh ``ScriptedChatClient`` with that role's reply, all sharing ONE ``sink``. MAF stamps the
|
||||
|
|
@ -470,6 +488,11 @@ def scripted_factory(
|
|||
reply = replies[role]
|
||||
if callable(reply):
|
||||
return ScriptedChatClient(sink=sink, role=role, reply_selector=reply)
|
||||
if not isinstance(reply, str):
|
||||
# A step LIST (MAJOR-1 b): the only form that can make a scripted role emit a tool
|
||||
# call, and therefore the only form under which an offline rehearsal can prove the
|
||||
# navigator opens anything.
|
||||
return ScriptedChatClient(sink=sink, role=role, script=list(reply))
|
||||
return ScriptedChatClient(reply, sink, role=role)
|
||||
|
||||
return factory
|
||||
|
|
|
|||
|
|
@ -281,3 +281,184 @@ def test_the_record_leaves_the_run_in_the_artefact_beside_the_advisory_verdicts(
|
|||
"the artefact must say which bases were opened, or a paid run leaves no record that any were"
|
||||
)
|
||||
assert json.dumps(payload), "the payload must stay plain data — the RAW layer is MAF-free"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# MAJOR-1 (b) — THE SCRIPT CAN NOW CALL A TOOL, AND THE END-TO-END GATE OVER BOTH HALVES
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
_BUNDLE_ID = _BUNDLE_DIR.name
|
||||
|
||||
|
||||
def _ledger(*, satisfied: bool, speaker: str = "navigator") -> str:
|
||||
return json.dumps(
|
||||
{
|
||||
"is_request_satisfied": {"reason": "r", "answer": satisfied},
|
||||
"is_in_loop": {"reason": "r", "answer": False},
|
||||
"is_progress_being_made": {"reason": "r", "answer": True},
|
||||
"next_speaker": {"reason": "r", "answer": speaker},
|
||||
"instruction_or_question": {"reason": "r", "answer": "open a base"},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
#: The manager answered as a STEP LIST — one entry per stage, in the order the orchestrator asks
|
||||
#: (facts, plan, ledger, ledger, final answer). This is the same measured reason
|
||||
#: ``simulation._exploration_manager_reply`` keys on the PROMPT rather than the project id: the
|
||||
#: manager is asked five DIFFERENT questions and a single constant answers all of them with the
|
||||
#: first round's ledger — which is why the pre-existing constant-reply scenario concludes after one
|
||||
#: round having given no participant a turn (measured here). Text steps, no calls: the manager
|
||||
#: carries no tools.
|
||||
_MANAGER_STAGES = [
|
||||
"FACTS: the base is anchored.",
|
||||
"PLAN: - let the navigator open a base",
|
||||
_ledger(satisfied=False),
|
||||
_ledger(satisfied=True),
|
||||
"FINAL: the exploration is done.",
|
||||
]
|
||||
_HYPOTHESIS = "HYPOTHESIS: " + json.dumps({"label": "x", "rationale": "y"})
|
||||
|
||||
|
||||
def _artefact(tmp_path: Path, run_id: str) -> dict[str, Any]:
|
||||
path = tmp_path / "outbox" / f"{run_id}-exploration.json"
|
||||
assert path.exists(), "the exploration artefact must be written even when the run failed"
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def _explore_argv(tmp_path: Path, replies_path: str, run_id: str) -> list[str]:
|
||||
return [
|
||||
*_base_argv(tmp_path, replies_path),
|
||||
"--outbox-dir",
|
||||
str(tmp_path / "outbox"),
|
||||
"--run-id",
|
||||
run_id,
|
||||
]
|
||||
|
||||
|
||||
def test_a_scripted_navigator_can_open_a_base_and_the_artefact_says_which(tmp_path) -> None:
|
||||
"""THE GOAL, end to end and over BOTH halves: a script of
|
||||
``list_bundles -> read_bundle -> text`` makes the navigator actually call the tools, and the
|
||||
artefact an operator reads afterwards names which base was opened.
|
||||
|
||||
RED without (b): a step LIST is refused at the door, so the run never starts. RED without (a):
|
||||
the tools run and ``tool_calls`` is empty — indistinguishable from the vacuous rehearsal this
|
||||
whole section exists to make visible.
|
||||
"""
|
||||
replies = _replies_file(
|
||||
tmp_path,
|
||||
{
|
||||
"proposer": _PROPOSER_REPLY,
|
||||
"checker": "VERDICT: APPROVE",
|
||||
"manager": list(_MANAGER_STAGES),
|
||||
"navigator": [
|
||||
{"call": "list_bundles"},
|
||||
{"call": "read_bundle", "args": {"bundle_id": _BUNDLE_ID}},
|
||||
"NAVIGATOR: read the index.",
|
||||
],
|
||||
"hypothesiser": _HYPOTHESIS,
|
||||
},
|
||||
)
|
||||
|
||||
rc = run.main(_explore_argv(tmp_path, replies, "scripted-tools"))
|
||||
assert rc in (0, 1), rc
|
||||
|
||||
calls = _artefact(tmp_path, "scripted-tools")["tool_calls"]
|
||||
assert [c["name"] for c in calls][:2] == ["list_bundles", "read_bundle"], (
|
||||
"an offline rehearsal must be able to prove the navigator OPENS something — that is the "
|
||||
f"whole of the free half of the measurement ladder; got {calls!r}"
|
||||
)
|
||||
assert calls[1]["bundle_id"] == _BUNDLE_ID, (
|
||||
"the name alone answers 'a tool ran'; WHICH base it was asked for is what tells an "
|
||||
"operator the navigator went where it was pointed"
|
||||
)
|
||||
|
||||
|
||||
def test_a_constant_string_script_records_no_tool_calls(tmp_path) -> None:
|
||||
"""The CONTROL, and the measurement MAJOR-1 reported: one constant string per role cannot
|
||||
emit a ``function_call``, so the rehearsal calls nothing. The recorder must SAY so rather than
|
||||
invent entries — a record that filled itself in would be worth less than none.
|
||||
"""
|
||||
replies = _replies_file(
|
||||
tmp_path,
|
||||
{
|
||||
"proposer": _PROPOSER_REPLY,
|
||||
"checker": "VERDICT: APPROVE",
|
||||
"manager": list(_MANAGER_STAGES),
|
||||
"navigator": "NAVIGATOR: read the index.",
|
||||
"hypothesiser": _HYPOTHESIS,
|
||||
},
|
||||
)
|
||||
|
||||
rc = run.main(_explore_argv(tmp_path, replies, "scripted-constant"))
|
||||
assert rc in (0, 1), rc
|
||||
|
||||
assert _artefact(tmp_path, "scripted-constant")["tool_calls"] == [], (
|
||||
"a constant-reply rehearsal opens nothing, and the artefact must be able to say that"
|
||||
)
|
||||
|
||||
|
||||
def _refuse(tmp_path, capsys, step: Any, run_id: str) -> str:
|
||||
replies = _replies_file(
|
||||
tmp_path,
|
||||
{
|
||||
"proposer": _PROPOSER_REPLY,
|
||||
"checker": "VERDICT: APPROVE",
|
||||
"manager": list(_MANAGER_STAGES),
|
||||
"navigator": [step],
|
||||
"hypothesiser": _HYPOTHESIS,
|
||||
},
|
||||
)
|
||||
rc = run.main(_explore_argv(tmp_path, replies, run_id))
|
||||
err = capsys.readouterr().err
|
||||
assert rc == 1, err
|
||||
assert "run refused" in err and "navigator" in err, err
|
||||
return err
|
||||
|
||||
|
||||
def test_a_step_that_names_no_tool_is_refused_by_name(tmp_path, capsys) -> None:
|
||||
"""Validation, NEVER repair (the ``write_concept_file`` rule). A step that forgot to say WHAT
|
||||
to call is refused at the door — letting it through would reach ``step["call"]`` inside the
|
||||
client and crash mid-run with a raw ``KeyError``, which is precisely the MAJOR-2 failure the
|
||||
first half of this file exists to have closed.
|
||||
|
||||
**This arm was FALSIFIED first** (this repo's vacuous-gate class, twelfth time). It originally
|
||||
used ``{"invoke": "list_bundles"}``, and stayed GREEN with the shape check detached — the
|
||||
UNKNOWN-KEY branch below caught it instead, so the refusal under test had no witness at all.
|
||||
The two branches are now exercised by two arms with two mutations.
|
||||
"""
|
||||
_refuse(tmp_path, capsys, {"args": {"bundle_id": _BUNDLE_ID}}, "scripted-no-call")
|
||||
|
||||
|
||||
def test_a_step_naming_an_unknown_key_is_refused_by_name(tmp_path, capsys) -> None:
|
||||
"""The whitelist half (the hosted surface's ``_ALLOWED_FIELDS`` precedent, one layer down): a
|
||||
key a step does not carry is refused BY NAME, never dropped in silence. An operator who wrote
|
||||
``result`` expecting the rehearsal to assert something would otherwise be told nothing.
|
||||
"""
|
||||
err = _refuse(
|
||||
tmp_path, capsys, {"call": "list_bundles", "result": "[]"}, "scripted-unknown-key"
|
||||
)
|
||||
assert "result" in err, err
|
||||
|
||||
|
||||
def test_a_step_list_is_refused_for_a_debate_role(tmp_path, capsys) -> None:
|
||||
"""The list form is the EXPLORATION's, and is refused elsewhere by name rather than accepted
|
||||
and quietly ignored. The debate's proposer is driven by ``generate``'s own call, not by an
|
||||
agent loop that would invoke a tool between turns, so a script of calls there would describe a
|
||||
rehearsal that cannot happen.
|
||||
"""
|
||||
replies = _replies_file(
|
||||
tmp_path,
|
||||
{
|
||||
"proposer": [{"call": "retrieve_cost_docs", "args": {"query": "x"}}],
|
||||
"checker": "VERDICT: APPROVE",
|
||||
"manager": list(_MANAGER_STAGES),
|
||||
"navigator": "NAVIGATOR: read the index.",
|
||||
"hypothesiser": _HYPOTHESIS,
|
||||
},
|
||||
)
|
||||
|
||||
rc = run.main(_explore_argv(tmp_path, replies, "scripted-debate-list"))
|
||||
|
||||
assert rc == 1
|
||||
err = capsys.readouterr().err
|
||||
assert "run refused" in err and "proposer" in err, err
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue