fix(cli): a provider failure leaves the CLI as one line, and a guessed base id is correctable

Funn 99, measured offline against the artefacts the paid Q5=B run left behind — no paid
run here.

ROOT, verbatim from the records: the three failing quick_validate calls all sent
bundle_id="renholdstekniske_funksjonskrav" — a CONCEPT name guessed out of the seeded cut,
while the base's id is k2-trinn1-20260903. Both arguments parsed against the signature, so
it was _resolve_bundle's raise MAF counted, proven by quick_validations being EMPTY while
all three stand in tool_calls. Denominator: 12 tool calls, and those three came BEFORE
list_bundles.

The order's causal chain is FELLED: the quick_validate triple is records 4-6 and the run
continued for 13 more model calls; the triple immediately before the 400 is the navigator's
three read_file refusals on del-ii-bilag-7-prisskjema*. The limit fired TWICE.

(A) ChatClientException is caught on BOTH seams — the exploration dispatch and the full-run
dispatch — because the debate's own model calls go through the same provider. The line is
"run stopped:", not "run refused:" (a stated divergence from the order): the argv was fine
and tokens were already spent, which is the MAJOR-2 arm's own reason, verbatim. Caught
INSIDE the try/finally so the exploration artefact still lands.

(B) quick_validate answers an unknown base id with {"decision": "refused", ...} naming the
configured ids, and records it in the sink. MAF turns a tool raise into the opaque
"Error: Function failed." (_tools.py:1426), so the one thing the refusal knew and the model
did not never reached it — the replies show it guessing at the JSON format instead.
read_file/read_dir/read_bundle still raise: measured, reported, out of scope.

Seven mutations all red against the whole suite, green control 1529/5, golden ea8c534
unchanged. One existing gate REWRITTEN, not deleted; its second half is what keeps (B)
scoped. The test double raises from the reply_selector seam rather than a new
_inner_get_response body, so the S2.5 consolidation guard stays untouched.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 21:03:18 +02:00
commit 078a099898
5 changed files with 543 additions and 5 deletions

View file

@ -752,10 +752,23 @@ def test_an_unknown_knowledge_base_is_refused_by_name() -> None:
Model-chosen arguments are untrusted input. Answering an unknown id with an empty result would
let the manager conclude the base is empty rather than absent the fourth face of the
verification law, arrived at through a tool rather than a query.
REWRITTEN, not weakened (funn 99): ``quick_validate`` now answers with a ``refused`` VERDICT
rather than a raise, because MAF converts a tool raise into the opaque ``"Error: Function
failed."`` and counts three of them as grounds to stop function calling for the whole request —
so the one thing this refusal knows and the model does not (which ids exist) never reached it.
The property T20 was written for is untouched and asserted here in both halves: the answer is
still a refusal, and it still names what IS configured. The ``navigator_tools`` half is what
keeps the change SCOPED those three tools still raise.
"""
validate = explore.quick_validate_tool(("/tmp/base-a",))
verdict = validate.func(bundle_id="base-b", proposal_json="{}")
assert verdict["decision"] == "refused"
assert "base-b" in verdict["reason"] and "base-a" in verdict["reason"]
read_file = next(t for t in explore.navigator_tools(("/tmp/base-a",)) if t.name == "read_file")
with pytest.raises(explore.ExplorationError) as excinfo:
validate.func(bundle_id="base-b", proposal_json="{}")
read_file.func(bundle_id="base-b", path="x.md")
assert "base-a" in str(excinfo.value)