fix(cli): a provider failure leaves the CLI as one line, and a guessed base id is correctable

Funn 99, measured offline against the artefacts the paid Q5=B run left behind — no paid
run here.

ROOT, verbatim from the records: the three failing quick_validate calls all sent
bundle_id="renholdstekniske_funksjonskrav" — a CONCEPT name guessed out of the seeded cut,
while the base's id is k2-trinn1-20260903. Both arguments parsed against the signature, so
it was _resolve_bundle's raise MAF counted, proven by quick_validations being EMPTY while
all three stand in tool_calls. Denominator: 12 tool calls, and those three came BEFORE
list_bundles.

The order's causal chain is FELLED: the quick_validate triple is records 4-6 and the run
continued for 13 more model calls; the triple immediately before the 400 is the navigator's
three read_file refusals on del-ii-bilag-7-prisskjema*. The limit fired TWICE.

(A) ChatClientException is caught on BOTH seams — the exploration dispatch and the full-run
dispatch — because the debate's own model calls go through the same provider. The line is
"run stopped:", not "run refused:" (a stated divergence from the order): the argv was fine
and tokens were already spent, which is the MAJOR-2 arm's own reason, verbatim. Caught
INSIDE the try/finally so the exploration artefact still lands.

(B) quick_validate answers an unknown base id with {"decision": "refused", ...} naming the
configured ids, and records it in the sink. MAF turns a tool raise into the opaque
"Error: Function failed." (_tools.py:1426), so the one thing the refusal knew and the model
did not never reached it — the replies show it guessing at the JSON format instead.
read_file/read_dir/read_bundle still raise: measured, reported, out of scope.

Seven mutations all red against the whole suite, green control 1529/5, golden ea8c534
unchanged. One existing gate REWRITTEN, not deleted; its second half is what keeps (B)
scoped. The test double raises from the reply_selector seam rather than a new
_inner_get_response body, so the S2.5 consolidation guard stays untouched.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 21:03:18 +02:00
commit 078a099898
5 changed files with 543 additions and 5 deletions

View file

@ -1114,6 +1114,27 @@ def navigator_tools(
return [list_bundles, read_bundle, read_dir, read_file]
def _refused(
reason: str,
*,
sink: list[QuickValidation] | None,
bundle_id: str,
proposal_json: str,
) -> dict[str, Any]:
"""The refused verdict, recorded on the same rule every other branch is (see below).
``anchored`` is ``False`` and that is a statement of fact, not a default: no base was
resolved, so no ``cost-baseline.json`` was read and this verdict was reached without the
project's own cost lines — exactly what the field says everywhere else it appears.
"""
verdict: dict[str, Any] = {"decision": "refused", "reason": reason, "anchored": False}
if sink is not None:
sink.append(
QuickValidation(bundle_id=bundle_id, proposal_json=proposal_json, verdict=verdict)
)
return verdict
def quick_validate_tool(
bundle_dirs: Sequence[str], *, sink: list[QuickValidation] | None = None
) -> FunctionTool:
@ -1148,9 +1169,24 @@ def quick_validate_tool(
),
)
def quick_validate(bundle_id: str, proposal_json: str) -> dict[str, Any]:
bundle_dir = _resolve_bundle(index, bundle_id)
baseline = okf.load_optional_cost_baseline(bundle_dir)
verdict: dict[str, Any]
try:
bundle_dir = _resolve_bundle(index, bundle_id)
except ExplorationError as exc:
# A base id the model guessed wrong is a thing it can CORRECT — so it comes back as a
# verdict naming the configured ids, never as a raise. MEASURED (funn 99, Q5=B on K2):
# three consecutive calls carried ``bundle_id="renholdstekniske_funksjonskrav"``, a
# concept name guessed out of the seeded cut, and MAF turned each raise into the opaque
# ``"Error: Function failed."`` (``_tools.py:1426`` — the detail is suppressed unless
# ``include_detailed_errors``), so the ONE thing this refusal knows and the model did
# not — which ids exist — never reached it. The replies show the consequence: it went
# on guessing at the JSON format. Three in a row is
# ``DEFAULT_MAX_CONSECUTIVE_ERRORS_PER_REQUEST`` (``_tools.py:96``), after which MAF
# stops all function calling for the request. This is NOT a general softening of the
# navigator's refusals: ``read_file``/``read_dir``/``read_bundle`` still raise, and
# their raises are counted the same way (measured, reported, out of this order's scope).
return _refused(str(exc), sink=sink, bundle_id=bundle_id, proposal_json=proposal_json)
baseline = okf.load_optional_cost_baseline(bundle_dir)
try:
proposal = SavingsProposal.model_validate_json(proposal_json)
except ValidationError as exc:
@ -1177,8 +1213,13 @@ def quick_validate_tool(
"p90": outcome.p90,
}
# Recorded AFTER the verdict is decided and on EVERY branch — an unparseable candidate is
# as much a thing the hypothesiser asked about as a validated one. A refused bundle id
# raises above and is deliberately not recorded: nothing was validated.
# as much a thing the hypothesiser asked about as a validated one. A refused bundle id is
# recorded too, on that same rule: the comment that used to stand here ("nothing was
# validated") was written for a RAISE, which left no verdict at all. Now that there IS one,
# keeping it out would make a refused call the single quick_validate outcome invisible in
# ``quick_validations`` — and an operator reading that list could not tell "never called"
# from "called three times with an id that does not exist", which is exactly the read this
# defect needed two artefacts to reconstruct.
if sink is not None:
sink.append(
QuickValidation(bundle_id=bundle_id, proposal_json=proposal_json, verdict=verdict)