fix(cli): a provider failure leaves the CLI as one line, and a guessed base id is correctable

Funn 99, measured offline against the artefacts the paid Q5=B run left behind — no paid
run here.

ROOT, verbatim from the records: the three failing quick_validate calls all sent
bundle_id="renholdstekniske_funksjonskrav" — a CONCEPT name guessed out of the seeded cut,
while the base's id is k2-trinn1-20260903. Both arguments parsed against the signature, so
it was _resolve_bundle's raise MAF counted, proven by quick_validations being EMPTY while
all three stand in tool_calls. Denominator: 12 tool calls, and those three came BEFORE
list_bundles.

The order's causal chain is FELLED: the quick_validate triple is records 4-6 and the run
continued for 13 more model calls; the triple immediately before the 400 is the navigator's
three read_file refusals on del-ii-bilag-7-prisskjema*. The limit fired TWICE.

(A) ChatClientException is caught on BOTH seams — the exploration dispatch and the full-run
dispatch — because the debate's own model calls go through the same provider. The line is
"run stopped:", not "run refused:" (a stated divergence from the order): the argv was fine
and tokens were already spent, which is the MAJOR-2 arm's own reason, verbatim. Caught
INSIDE the try/finally so the exploration artefact still lands.

(B) quick_validate answers an unknown base id with {"decision": "refused", ...} naming the
configured ids, and records it in the sink. MAF turns a tool raise into the opaque
"Error: Function failed." (_tools.py:1426), so the one thing the refusal knew and the model
did not never reached it — the replies show it guessing at the JSON format instead.
read_file/read_dir/read_bundle still raise: measured, reported, out of scope.

Seven mutations all red against the whole suite, green control 1529/5, golden ea8c534
unchanged. One existing gate REWRITTEN, not deleted; its second half is what keeps (B)
scoped. The test double raises from the reply_selector seam rather than a new
_inner_get_response body, so the S2.5 consolidation guard stays untouched.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 21:03:18 +02:00
commit 078a099898
5 changed files with 543 additions and 5 deletions

View file

@ -1114,6 +1114,27 @@ def navigator_tools(
return [list_bundles, read_bundle, read_dir, read_file]
def _refused(
reason: str,
*,
sink: list[QuickValidation] | None,
bundle_id: str,
proposal_json: str,
) -> dict[str, Any]:
"""The refused verdict, recorded on the same rule every other branch is (see below).
``anchored`` is ``False`` and that is a statement of fact, not a default: no base was
resolved, so no ``cost-baseline.json`` was read and this verdict was reached without the
project's own cost lines — exactly what the field says everywhere else it appears.
"""
verdict: dict[str, Any] = {"decision": "refused", "reason": reason, "anchored": False}
if sink is not None:
sink.append(
QuickValidation(bundle_id=bundle_id, proposal_json=proposal_json, verdict=verdict)
)
return verdict
def quick_validate_tool(
bundle_dirs: Sequence[str], *, sink: list[QuickValidation] | None = None
) -> FunctionTool:
@ -1148,9 +1169,24 @@ def quick_validate_tool(
),
)
def quick_validate(bundle_id: str, proposal_json: str) -> dict[str, Any]:
bundle_dir = _resolve_bundle(index, bundle_id)
baseline = okf.load_optional_cost_baseline(bundle_dir)
verdict: dict[str, Any]
try:
bundle_dir = _resolve_bundle(index, bundle_id)
except ExplorationError as exc:
# A base id the model guessed wrong is a thing it can CORRECT — so it comes back as a
# verdict naming the configured ids, never as a raise. MEASURED (funn 99, Q5=B on K2):
# three consecutive calls carried ``bundle_id="renholdstekniske_funksjonskrav"``, a
# concept name guessed out of the seeded cut, and MAF turned each raise into the opaque
# ``"Error: Function failed."`` (``_tools.py:1426`` — the detail is suppressed unless
# ``include_detailed_errors``), so the ONE thing this refusal knows and the model did
# not — which ids exist — never reached it. The replies show the consequence: it went
# on guessing at the JSON format. Three in a row is
# ``DEFAULT_MAX_CONSECUTIVE_ERRORS_PER_REQUEST`` (``_tools.py:96``), after which MAF
# stops all function calling for the request. This is NOT a general softening of the
# navigator's refusals: ``read_file``/``read_dir``/``read_bundle`` still raise, and
# their raises are counted the same way (measured, reported, out of this order's scope).
return _refused(str(exc), sink=sink, bundle_id=bundle_id, proposal_json=proposal_json)
baseline = okf.load_optional_cost_baseline(bundle_dir)
try:
proposal = SavingsProposal.model_validate_json(proposal_json)
except ValidationError as exc:
@ -1177,8 +1213,13 @@ def quick_validate_tool(
"p90": outcome.p90,
}
# Recorded AFTER the verdict is decided and on EVERY branch — an unparseable candidate is
# as much a thing the hypothesiser asked about as a validated one. A refused bundle id
# raises above and is deliberately not recorded: nothing was validated.
# as much a thing the hypothesiser asked about as a validated one. A refused bundle id is
# recorded too, on that same rule: the comment that used to stand here ("nothing was
# validated") was written for a RAISE, which left no verdict at all. Now that there IS one,
# keeping it out would make a refused call the single quick_validate outcome invisible in
# ``quick_validations`` — and an operator reading that list could not tell "never called"
# from "called three times with an id that does not exist", which is exactly the read this
# defect needed two artefacts to reconstruct.
if sink is not None:
sink.append(
QuickValidation(bundle_id=bundle_id, proposal_json=proposal_json, verdict=verdict)

View file

@ -34,6 +34,7 @@ from pathlib import Path
from typing import Any, Literal, cast
from agent_framework import BaseChatClient, SessionContext
from agent_framework.exceptions import ChatClientException
from pydantic import ValidationError
from portfolio_optimiser.backends import Profile, get_backend, resolve_model
@ -3496,6 +3497,7 @@ def main(argv: list[str] | None = None) -> int:
exploration: ExplorationResult | None = None
parked_now: PlanReviewParked | None = None
budget_now: BudgetExceeded | None = None
provider_now: ChatClientException | None = None
try:
if resumed is not None:
# The parked state, not argv, is what rebuilds the workflow: the graph has to match
@ -3543,6 +3545,17 @@ def main(argv: list[str] | None = None) -> int:
# than left to escape, because parking is what the operator ASKED for by giving
# --checkpoint-dir; the artefact is where a machine reads that it happened.
parked_now = parked_exc
except ChatClientException as provider_exc:
# Funn 99. The model endpoint rejected the request, so the run ENDED — it was never
# refused. ``run stopped:``, not ``run refused:``, and the class is what routes it here
# (the MAJOR-2 precedent, verbatim: "the argv was fine and the run had already spent
# tokens, so 'refused' would mislabel it"). MEASURED on the paid Q5=B run: an Azure 400
# ("No tool call found for function call output with call_id ...") left ``main()`` as a
# traceback, because ``ChatClientException`` is in none of this function's refusal
# tuples — the same gap ``BudgetExceeded`` was added to this very block for. Caught
# INSIDE the try/finally so the ``finally`` still writes ``{run_id}-exploration.json``:
# the run that most needs the evidence is the one a provider cut short.
provider_now = provider_exc
except BudgetExceeded as budget_exc:
# A cap that fired is a refusal at this door, not a programming error — the CLI's
# existing contract for every other loader mistake below (stderr + rc 1, never a
@ -3574,6 +3587,11 @@ def main(argv: list[str] | None = None) -> int:
),
),
)
if provider_now is not None:
# One line, rc 1, no traceback — and reported BEFORE the budget arm only because the
# two are mutually exclusive by construction (a single exception left the block).
print(f"run stopped: {provider_now}", file=sys.stderr)
return 1
if budget_now is not None:
# Same shape as every other refusal in this function: one line on stderr, rc 1, no
# traceback. The artefact was already written by the ``finally`` above (``completed``
@ -3858,6 +3876,14 @@ def main(argv: list[str] | None = None) -> int:
)
),
)
except ChatClientException as exc:
# Funn 99, the SECOND seam: the debate's own model calls go through the same provider, so
# an arm on the exploration block alone leaves an ordinary ``run.main([...])`` tracebacking.
# Same channel and same reason as ``ProposalReviewInputError`` below — the argv was fine and
# tokens were already spent — and a distinct class from every ``ValueError``-shaped refusal,
# so a reader can tell "the request was wrong" from "the endpoint rejected it".
print(f"run stopped: {exc}", file=sys.stderr)
return 1
except ProposalReviewInputError as exc:
# A DISTINCT channel from ``run refused:`` below, and the class is what routes it here.
# The argv was fine and the run had already spent tokens, so "refused" would mislabel it;