fix(cli): a provider failure leaves the CLI as one line, and a guessed base id is correctable
Funn 99, measured offline against the artefacts the paid Q5=B run left behind — no paid
run here.
ROOT, verbatim from the records: the three failing quick_validate calls all sent
bundle_id="renholdstekniske_funksjonskrav" — a CONCEPT name guessed out of the seeded cut,
while the base's id is k2-trinn1-20260903. Both arguments parsed against the signature, so
it was _resolve_bundle's raise MAF counted, proven by quick_validations being EMPTY while
all three stand in tool_calls. Denominator: 12 tool calls, and those three came BEFORE
list_bundles.
The order's causal chain is FELLED: the quick_validate triple is records 4-6 and the run
continued for 13 more model calls; the triple immediately before the 400 is the navigator's
three read_file refusals on del-ii-bilag-7-prisskjema*. The limit fired TWICE.
(A) ChatClientException is caught on BOTH seams — the exploration dispatch and the full-run
dispatch — because the debate's own model calls go through the same provider. The line is
"run stopped:", not "run refused:" (a stated divergence from the order): the argv was fine
and tokens were already spent, which is the MAJOR-2 arm's own reason, verbatim. Caught
INSIDE the try/finally so the exploration artefact still lands.
(B) quick_validate answers an unknown base id with {"decision": "refused", ...} naming the
configured ids, and records it in the sink. MAF turns a tool raise into the opaque
"Error: Function failed." (_tools.py:1426), so the one thing the refusal knew and the model
did not never reached it — the replies show it guessing at the JSON format instead.
read_file/read_dir/read_bundle still raise: measured, reported, out of scope.
Seven mutations all red against the whole suite, green control 1529/5, golden ea8c534
unchanged. One existing gate REWRITTEN, not deleted; its second half is what keeps (B)
scoped. The test double raises from the reply_selector seam rather than a new
_inner_get_response body, so the S2.5 consolidation guard stays untouched.
Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
parent
4f23fa2a70
commit
078a099898
5 changed files with 543 additions and 5 deletions
|
|
@ -1114,6 +1114,27 @@ def navigator_tools(
|
|||
return [list_bundles, read_bundle, read_dir, read_file]
|
||||
|
||||
|
||||
def _refused(
|
||||
reason: str,
|
||||
*,
|
||||
sink: list[QuickValidation] | None,
|
||||
bundle_id: str,
|
||||
proposal_json: str,
|
||||
) -> dict[str, Any]:
|
||||
"""The refused verdict, recorded on the same rule every other branch is (see below).
|
||||
|
||||
``anchored`` is ``False`` and that is a statement of fact, not a default: no base was
|
||||
resolved, so no ``cost-baseline.json`` was read and this verdict was reached without the
|
||||
project's own cost lines — exactly what the field says everywhere else it appears.
|
||||
"""
|
||||
verdict: dict[str, Any] = {"decision": "refused", "reason": reason, "anchored": False}
|
||||
if sink is not None:
|
||||
sink.append(
|
||||
QuickValidation(bundle_id=bundle_id, proposal_json=proposal_json, verdict=verdict)
|
||||
)
|
||||
return verdict
|
||||
|
||||
|
||||
def quick_validate_tool(
|
||||
bundle_dirs: Sequence[str], *, sink: list[QuickValidation] | None = None
|
||||
) -> FunctionTool:
|
||||
|
|
@ -1148,9 +1169,24 @@ def quick_validate_tool(
|
|||
),
|
||||
)
|
||||
def quick_validate(bundle_id: str, proposal_json: str) -> dict[str, Any]:
|
||||
bundle_dir = _resolve_bundle(index, bundle_id)
|
||||
baseline = okf.load_optional_cost_baseline(bundle_dir)
|
||||
verdict: dict[str, Any]
|
||||
try:
|
||||
bundle_dir = _resolve_bundle(index, bundle_id)
|
||||
except ExplorationError as exc:
|
||||
# A base id the model guessed wrong is a thing it can CORRECT — so it comes back as a
|
||||
# verdict naming the configured ids, never as a raise. MEASURED (funn 99, Q5=B on K2):
|
||||
# three consecutive calls carried ``bundle_id="renholdstekniske_funksjonskrav"``, a
|
||||
# concept name guessed out of the seeded cut, and MAF turned each raise into the opaque
|
||||
# ``"Error: Function failed."`` (``_tools.py:1426`` — the detail is suppressed unless
|
||||
# ``include_detailed_errors``), so the ONE thing this refusal knows and the model did
|
||||
# not — which ids exist — never reached it. The replies show the consequence: it went
|
||||
# on guessing at the JSON format. Three in a row is
|
||||
# ``DEFAULT_MAX_CONSECUTIVE_ERRORS_PER_REQUEST`` (``_tools.py:96``), after which MAF
|
||||
# stops all function calling for the request. This is NOT a general softening of the
|
||||
# navigator's refusals: ``read_file``/``read_dir``/``read_bundle`` still raise, and
|
||||
# their raises are counted the same way (measured, reported, out of this order's scope).
|
||||
return _refused(str(exc), sink=sink, bundle_id=bundle_id, proposal_json=proposal_json)
|
||||
baseline = okf.load_optional_cost_baseline(bundle_dir)
|
||||
try:
|
||||
proposal = SavingsProposal.model_validate_json(proposal_json)
|
||||
except ValidationError as exc:
|
||||
|
|
@ -1177,8 +1213,13 @@ def quick_validate_tool(
|
|||
"p90": outcome.p90,
|
||||
}
|
||||
# Recorded AFTER the verdict is decided and on EVERY branch — an unparseable candidate is
|
||||
# as much a thing the hypothesiser asked about as a validated one. A refused bundle id
|
||||
# raises above and is deliberately not recorded: nothing was validated.
|
||||
# as much a thing the hypothesiser asked about as a validated one. A refused bundle id is
|
||||
# recorded too, on that same rule: the comment that used to stand here ("nothing was
|
||||
# validated") was written for a RAISE, which left no verdict at all. Now that there IS one,
|
||||
# keeping it out would make a refused call the single quick_validate outcome invisible in
|
||||
# ``quick_validations`` — and an operator reading that list could not tell "never called"
|
||||
# from "called three times with an id that does not exist", which is exactly the read this
|
||||
# defect needed two artefacts to reconstruct.
|
||||
if sink is not None:
|
||||
sink.append(
|
||||
QuickValidation(bundle_id=bundle_id, proposal_json=proposal_json, verdict=verdict)
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ from pathlib import Path
|
|||
from typing import Any, Literal, cast
|
||||
|
||||
from agent_framework import BaseChatClient, SessionContext
|
||||
from agent_framework.exceptions import ChatClientException
|
||||
from pydantic import ValidationError
|
||||
|
||||
from portfolio_optimiser.backends import Profile, get_backend, resolve_model
|
||||
|
|
@ -3496,6 +3497,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
exploration: ExplorationResult | None = None
|
||||
parked_now: PlanReviewParked | None = None
|
||||
budget_now: BudgetExceeded | None = None
|
||||
provider_now: ChatClientException | None = None
|
||||
try:
|
||||
if resumed is not None:
|
||||
# The parked state, not argv, is what rebuilds the workflow: the graph has to match
|
||||
|
|
@ -3543,6 +3545,17 @@ def main(argv: list[str] | None = None) -> int:
|
|||
# than left to escape, because parking is what the operator ASKED for by giving
|
||||
# --checkpoint-dir; the artefact is where a machine reads that it happened.
|
||||
parked_now = parked_exc
|
||||
except ChatClientException as provider_exc:
|
||||
# Funn 99. The model endpoint rejected the request, so the run ENDED — it was never
|
||||
# refused. ``run stopped:``, not ``run refused:``, and the class is what routes it here
|
||||
# (the MAJOR-2 precedent, verbatim: "the argv was fine and the run had already spent
|
||||
# tokens, so 'refused' would mislabel it"). MEASURED on the paid Q5=B run: an Azure 400
|
||||
# ("No tool call found for function call output with call_id ...") left ``main()`` as a
|
||||
# traceback, because ``ChatClientException`` is in none of this function's refusal
|
||||
# tuples — the same gap ``BudgetExceeded`` was added to this very block for. Caught
|
||||
# INSIDE the try/finally so the ``finally`` still writes ``{run_id}-exploration.json``:
|
||||
# the run that most needs the evidence is the one a provider cut short.
|
||||
provider_now = provider_exc
|
||||
except BudgetExceeded as budget_exc:
|
||||
# A cap that fired is a refusal at this door, not a programming error — the CLI's
|
||||
# existing contract for every other loader mistake below (stderr + rc 1, never a
|
||||
|
|
@ -3574,6 +3587,11 @@ def main(argv: list[str] | None = None) -> int:
|
|||
),
|
||||
),
|
||||
)
|
||||
if provider_now is not None:
|
||||
# One line, rc 1, no traceback — and reported BEFORE the budget arm only because the
|
||||
# two are mutually exclusive by construction (a single exception left the block).
|
||||
print(f"run stopped: {provider_now}", file=sys.stderr)
|
||||
return 1
|
||||
if budget_now is not None:
|
||||
# Same shape as every other refusal in this function: one line on stderr, rc 1, no
|
||||
# traceback. The artefact was already written by the ``finally`` above (``completed``
|
||||
|
|
@ -3858,6 +3876,14 @@ def main(argv: list[str] | None = None) -> int:
|
|||
)
|
||||
),
|
||||
)
|
||||
except ChatClientException as exc:
|
||||
# Funn 99, the SECOND seam: the debate's own model calls go through the same provider, so
|
||||
# an arm on the exploration block alone leaves an ordinary ``run.main([...])`` tracebacking.
|
||||
# Same channel and same reason as ``ProposalReviewInputError`` below — the argv was fine and
|
||||
# tokens were already spent — and a distinct class from every ``ValueError``-shaped refusal,
|
||||
# so a reader can tell "the request was wrong" from "the endpoint rejected it".
|
||||
print(f"run stopped: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
except ProposalReviewInputError as exc:
|
||||
# A DISTINCT channel from ``run refused:`` below, and the class is what routes it here.
|
||||
# The argv was fine and the run had already spent tokens, so "refused" would mislabel it;
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue