fix(cli): a provider failure leaves the CLI as one line, and a guessed base id is correctable
Funn 99, measured offline against the artefacts the paid Q5=B run left behind — no paid
run here.
ROOT, verbatim from the records: the three failing quick_validate calls all sent
bundle_id="renholdstekniske_funksjonskrav" — a CONCEPT name guessed out of the seeded cut,
while the base's id is k2-trinn1-20260903. Both arguments parsed against the signature, so
it was _resolve_bundle's raise MAF counted, proven by quick_validations being EMPTY while
all three stand in tool_calls. Denominator: 12 tool calls, and those three came BEFORE
list_bundles.
The order's causal chain is FELLED: the quick_validate triple is records 4-6 and the run
continued for 13 more model calls; the triple immediately before the 400 is the navigator's
three read_file refusals on del-ii-bilag-7-prisskjema*. The limit fired TWICE.
(A) ChatClientException is caught on BOTH seams — the exploration dispatch and the full-run
dispatch — because the debate's own model calls go through the same provider. The line is
"run stopped:", not "run refused:" (a stated divergence from the order): the argv was fine
and tokens were already spent, which is the MAJOR-2 arm's own reason, verbatim. Caught
INSIDE the try/finally so the exploration artefact still lands.
(B) quick_validate answers an unknown base id with {"decision": "refused", ...} naming the
configured ids, and records it in the sink. MAF turns a tool raise into the opaque
"Error: Function failed." (_tools.py:1426), so the one thing the refusal knew and the model
did not never reached it — the replies show it guessing at the JSON format instead.
read_file/read_dir/read_bundle still raise: measured, reported, out of scope.
Seven mutations all red against the whole suite, green control 1529/5, golden ea8c534
unchanged. One existing gate REWRITTEN, not deleted; its second half is what keeps (B)
scoped. The test double raises from the reply_selector seam rather than a new
_inner_get_response body, so the S2.5 consolidation guard stays untouched.
Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
parent
4f23fa2a70
commit
078a099898
5 changed files with 543 additions and 5 deletions
|
|
@ -1114,6 +1114,27 @@ def navigator_tools(
|
|||
return [list_bundles, read_bundle, read_dir, read_file]
|
||||
|
||||
|
||||
def _refused(
|
||||
reason: str,
|
||||
*,
|
||||
sink: list[QuickValidation] | None,
|
||||
bundle_id: str,
|
||||
proposal_json: str,
|
||||
) -> dict[str, Any]:
|
||||
"""The refused verdict, recorded on the same rule every other branch is (see below).
|
||||
|
||||
``anchored`` is ``False`` and that is a statement of fact, not a default: no base was
|
||||
resolved, so no ``cost-baseline.json`` was read and this verdict was reached without the
|
||||
project's own cost lines — exactly what the field says everywhere else it appears.
|
||||
"""
|
||||
verdict: dict[str, Any] = {"decision": "refused", "reason": reason, "anchored": False}
|
||||
if sink is not None:
|
||||
sink.append(
|
||||
QuickValidation(bundle_id=bundle_id, proposal_json=proposal_json, verdict=verdict)
|
||||
)
|
||||
return verdict
|
||||
|
||||
|
||||
def quick_validate_tool(
|
||||
bundle_dirs: Sequence[str], *, sink: list[QuickValidation] | None = None
|
||||
) -> FunctionTool:
|
||||
|
|
@ -1148,9 +1169,24 @@ def quick_validate_tool(
|
|||
),
|
||||
)
|
||||
def quick_validate(bundle_id: str, proposal_json: str) -> dict[str, Any]:
|
||||
bundle_dir = _resolve_bundle(index, bundle_id)
|
||||
baseline = okf.load_optional_cost_baseline(bundle_dir)
|
||||
verdict: dict[str, Any]
|
||||
try:
|
||||
bundle_dir = _resolve_bundle(index, bundle_id)
|
||||
except ExplorationError as exc:
|
||||
# A base id the model guessed wrong is a thing it can CORRECT — so it comes back as a
|
||||
# verdict naming the configured ids, never as a raise. MEASURED (funn 99, Q5=B on K2):
|
||||
# three consecutive calls carried ``bundle_id="renholdstekniske_funksjonskrav"``, a
|
||||
# concept name guessed out of the seeded cut, and MAF turned each raise into the opaque
|
||||
# ``"Error: Function failed."`` (``_tools.py:1426`` — the detail is suppressed unless
|
||||
# ``include_detailed_errors``), so the ONE thing this refusal knows and the model did
|
||||
# not — which ids exist — never reached it. The replies show the consequence: it went
|
||||
# on guessing at the JSON format. Three in a row is
|
||||
# ``DEFAULT_MAX_CONSECUTIVE_ERRORS_PER_REQUEST`` (``_tools.py:96``), after which MAF
|
||||
# stops all function calling for the request. This is NOT a general softening of the
|
||||
# navigator's refusals: ``read_file``/``read_dir``/``read_bundle`` still raise, and
|
||||
# their raises are counted the same way (measured, reported, out of this order's scope).
|
||||
return _refused(str(exc), sink=sink, bundle_id=bundle_id, proposal_json=proposal_json)
|
||||
baseline = okf.load_optional_cost_baseline(bundle_dir)
|
||||
try:
|
||||
proposal = SavingsProposal.model_validate_json(proposal_json)
|
||||
except ValidationError as exc:
|
||||
|
|
@ -1177,8 +1213,13 @@ def quick_validate_tool(
|
|||
"p90": outcome.p90,
|
||||
}
|
||||
# Recorded AFTER the verdict is decided and on EVERY branch — an unparseable candidate is
|
||||
# as much a thing the hypothesiser asked about as a validated one. A refused bundle id
|
||||
# raises above and is deliberately not recorded: nothing was validated.
|
||||
# as much a thing the hypothesiser asked about as a validated one. A refused bundle id is
|
||||
# recorded too, on that same rule: the comment that used to stand here ("nothing was
|
||||
# validated") was written for a RAISE, which left no verdict at all. Now that there IS one,
|
||||
# keeping it out would make a refused call the single quick_validate outcome invisible in
|
||||
# ``quick_validations`` — and an operator reading that list could not tell "never called"
|
||||
# from "called three times with an id that does not exist", which is exactly the read this
|
||||
# defect needed two artefacts to reconstruct.
|
||||
if sink is not None:
|
||||
sink.append(
|
||||
QuickValidation(bundle_id=bundle_id, proposal_json=proposal_json, verdict=verdict)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue