open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

fix(cli): a provider failure leaves the CLI as one line, and a guessed base id is correctable

Browse source 
Funn 99, measured offline against the artefacts the paid Q5=B run left behind — no paid
run here.

ROOT, verbatim from the records: the three failing quick_validate calls all sent
bundle_id="renholdstekniske_funksjonskrav" — a CONCEPT name guessed out of the seeded cut,
while the base's id is k2-trinn1-20260903. Both arguments parsed against the signature, so
it was _resolve_bundle's raise MAF counted, proven by quick_validations being EMPTY while
all three stand in tool_calls. Denominator: 12 tool calls, and those three came BEFORE
list_bundles.

The order's causal chain is FELLED: the quick_validate triple is records 4-6 and the run
continued for 13 more model calls; the triple immediately before the 400 is the navigator's
three read_file refusals on del-ii-bilag-7-prisskjema*. The limit fired TWICE.

(A) ChatClientException is caught on BOTH seams — the exploration dispatch and the full-run
dispatch — because the debate's own model calls go through the same provider. The line is
"run stopped:", not "run refused:" (a stated divergence from the order): the argv was fine
and tokens were already spent, which is the MAJOR-2 arm's own reason, verbatim. Caught
INSIDE the try/finally so the exploration artefact still lands.

(B) quick_validate answers an unknown base id with {"decision": "refused", ...} naming the
configured ids, and records it in the sink. MAF turns a tool raise into the opaque
"Error: Function failed." (_tools.py:1426), so the one thing the refusal knew and the model
did not never reached it — the replies show it guessing at the JSON format instead.
read_file/read_dir/read_bundle still raise: measured, reported, out of scope.

Seven mutations all red against the whole suite, green control 1529/5, golden ea8c534
unchanged. One existing gate REWRITTEN, not deleted; its second half is what keeps (B)
scoped. The test double raises from the reply_selector seam rather than a new
_inner_get_response body, so the S2.5 consolidation guard stays untouched.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 21:03:18 +02:00
commit 078a099898
5 changed files with 543 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

190
docs/2026-09-08-funn-99-chatclient-refusal.md Normal file
View file

@ -0,0 +1,190 @@
# Funn 99 — `ChatClientException` forlot CLI-en som traceback
**Dato:** 2026-09-08 · **Ordre:** `20260908T180612Z-6878816140-from-.claude` · **HEAD før:** `4f23fa2`
Ingen betalt kjøring. Alt under er målt offline mot artefaktene den betalte Q5=B-kjøringen
etterlot (`scratchpad/q5b/live/`, økt 99) og mot MAF-kilden i `.venv/`.
MAF: `agent-framework-core` **1.16.0**, `agent-framework-orchestrations` **1.1.1** (`uv.lock`).
Remote målt: `git ls-remote origin main` = `4f23fa2` = HEAD → **UPUSHET = 0** (STATE-ens «3» var
foreldet og er rettet).
---
## 1. Målingen
### 1.1 De tre feilende `quick_validate`-kallene, ORDRETT
Fra `scratchpad/q5b/live/outbox-Bseed/q5b-Bseed-exploration.json` (`tool_calls`, i kall-rekkefølge)
og fra argument-blobbene i `Bseed-records.json` record 7:
| # | `bundle_id` | `proposal_json` | Hva MAF la tilbake |
|---|---|---|---|
| 1 | `renholdstekniske_funksjonskrav` | `{"cost_saving_direction":"Implement a comprehensive system integration and coordination strategy under a dedicated Responsible for IT and Building (RITB) role …"}` | `Error: Function failed.` |
| 2 | `renholdstekniske_funksjonskrav` | `{"cost_saving_direction":"Implement a dedicated Responsible for IT and Building (RITB) role …","rationale":"General technical requirements highlight …"}` | `Error: Function failed.` |
| 3 | `renholdstekniske_funksjonskrav` | `{"cost_saving_direction":"Assign a dedicated RITB role to coordinate system integration …","rationale":"General technical requirements advise dedicated roles (RITB) …"}` | `Error: Function failed.` |
**Roten: ukjent base-id.** Basens ekte id er `k2-trinn1-20260903`;
`renholdstekniske_funksjonskrav` er et KONSEPT-navn modellen gjettet ut av det seedede kuttet.
Begge argumentene var strenger og parset mot signaturen `quick_validate(bundle_id: str,
proposal_json: str)` — MAF feilet altså **ikke** på argumentformen. Det som feilet var
`_resolve_bundle`s raise (`src/portfolio_optimiser/explore.py:898-900`, `ExplorationError`).
**Beviset på at det var raisen og ikke et verdikt:** `quick_validations` i artefaktet er **tom**,
mens alle tre kallene står i `tool_calls`. Sinken appendes på HVER verdikt-gren
(`unparseable` / `rejected` / `validated`), og bare raisen gikk utenom. Ordrens premiss (ii)
er dermed bekreftet.
**Nevner:** 12 verktøykall totalt i kjøringen —
`read_file` 5 · **`quick_validate` 3** · `read_dir` 2 · `list_bundles` 1 · `read_bundle` 1.
Alle tre `quick_validate`-kallene er kall **0, 1 og 2**, altså FØR `list_bundles` (kall 3):
modellen gjettet en base-id før den noen gang spurte hvilke som fantes.
### 1.2 MAF-tellingen
- `.venv/…/agent_framework/_tools.py:1410-1432` — `_function_execution_error_result`: en raise i
verktøyet blir `Content.from_function_result(call_id=…, result="Error: Function failed.",
exception=str(exception))`. Detaljen er **undertrykt** med mindre `include_detailed_errors`
(linje 1427). **Et raise TELLER altså som «error» — svar på ordrens spørsmål.**
- `_tools.py:1895-1900` — `had_errors` er sann når et `function_result` bærer `exception`.
- `_tools.py:2718-2731` — `_update_consecutive_error_count`; grensen er
`DEFAULT_MAX_CONSECUTIVE_ERRORS_PER_REQUEST = 3` (`_tools.py:96`).
- `_tools.py:2856-2869` — ved grensen appendes `Message(role="tool", contents=execution_results)`
og `action="stop"`.
- `_tools.py:3303-3305` — «stop» betyr **`options["tool_choice"] = "none"` og så ett kall til**;
det er dét MAF sender etter grensen.
### 1.3 Ordrens ÅRSAKSKJEDE er FELT av målingen
Ordren leste kjeden som «tre `quick_validate`-feil på rad → grensen → resultat uten kall → 400».
**Records viser noe annet.** `quick_validate`-trippelen er records 4-6; kjøringen fortsatte
etterpå gjennom 13 modellkall og 9 verktøykall til. Trippelen som ligger UMIDDELBART foran 400-en
er en helt annen — navigatørens tre siste kall, ordrett fra record 19s prompt:
```
ok {"bundle_id":"k2-trinn1-20260903","path":"…/1-1"}
ok {"bundle_id":"k2-trinn1-20260903","path":"…/1-1/03-01-2023-vask-av-layout.md"}
ERR {"bundle_id":"k2-trinn1-20260903","path":"inbox-del-ii-bilag-7-prisskjema.md"}
ERR {"bundle_id":"k2-trinn1-20260903","path":"del-ii-bilag-7-prisskjema"}
ERR {"bundle_id":"k2-trinn1-20260903","path":"del-ii-bilag-7-prisskjema.md"}
```
Grensen fyrte altså **to ganger** i kjøringen, og 400-en kom på det NESTE agent-kallet
(record 19, hypothesiser), ikke på kallet rett etter `quick_validate`-trippelen.
### 1.4 «Resultat uten kall» — MAF-defekt eller po?
**IKKE VERIFISERT, rapportert som hypotese med linjer.** Feilen er
`400 … 'No tool call found for function call output with call_id call_AyeuYJmqvmmej9utu361Phtt.'`
— Responses-APIet fant et `function_call_output` uten sitt `function_call` i input.
Den ene mekanismen jeg KAN peke på i kilden er
`agent_framework_openai/_chat_client.py:1557-1559`: under service-side storage DROPPES
`function_call`-items fra inline input (`case "function_call": if request_uses_service_side_storage:
continue`), mens `function_result` sendes videre (kommentar :1540-1542, «plain function_call_output
pairs by call_id and is safe under storage»). Den parringen holder bare så lenge serveren HAR det
matchende kallet.
Hvorvidt det er dét som brast her er **ikke målt** — det ville krevd en ny betalt kjøring, som
ordren forbyr. Ingenting i `po` konstruerer meldinger på denne stien; po leverer et verktøy som
raiser, og MAF eier både konverteringen og transporten. **Fikset ALDRI i MAF.** Rapportert her.
---
## 2. Beslutningen
### (A) CLI-døren — JA, og på TO sømmer
`ChatClientException` (fra `agent_framework.exceptions`, ALDRI bar `Exception`) fanges nå
1. i utforsknings-dispatchen (`run.py`, samme blokk `BudgetExceeded` ble lagt til for), og
2. i fullkjørings-dispatchen (`run_project`) — debattens egne modellkall går gjennom samme
leverandør, så en fiks på bare den ene lar en helt vanlig `run.main([...])` tracebacke.
**AVVIK fra ordren, uttalt:** ordren ber om linjen `run refused:`. Levert er **`run stopped:`**.
Grunnen er repoets egen etablerte kontrakt, sitert ordrett fra `run.py`s MAJOR-2-arm:
«the argv was fine and the run had already spent tokens, so "refused" would mislabel it».
En leverandør-400 midt i en utforskning er ikke en argv-feil; `run refused:` ville sagt at po
avviste bestillingen. Klassen er dét som ruter den, akkurat som for `ProposalReviewInputError`.
Er PM uenig, er endringen én streng på to steder.
Artefaktet overlever: armen står **INNE i** try/finally, så `finally`-en fortsatt skriver
`{run_id}-exploration.json` med `completed: false`. Målt, ikke antatt (T3).
### (B) Verktøyet — JA, og kun for `quick_validate`
Målingen i 1.1 viser at det VAR `_resolve_bundle`s raise som ble telt, altså er ordrens betingelse
oppfylt. `quick_validate` returnerer nå
`{"decision": "refused", "reason": "unknown knowledge base 'X'; configured: Y", "anchored": false}`.
Hvorfor: MAF gjør raisen om til den ugjennomsiktige `"Error: Function failed."`, så **den ene
tingen refusalen visste og modellen ikke — hvilke id-er som finnes — nådde den aldri.** Følgen står
i modellens egne ord (record 5): «It appears the quick validation function is failing, possibly due
to format or content expectations» — den gjettet på JSON-formatet, som var riktig hele veien.
`anchored: false` er et faktum, ikke en default: ingen base ble resolvert, så ingen
`cost-baseline.json` ble lest.
**«Ikke i sinken» er OPPHEVET, med begrunnelse.** Kommentaren i koden var skrevet for en RAISE, som
etterlot null verdikt. Nå FINNES det et verdikt, og sinkens egen regel er «hver gren». Å holde det
ute ville gjort en refusert forespørsel til det ene `quick_validate`-utfallet som er usynlig i
`quick_validations` — og det er nøyaktig hvorfor denne økta måtte lese TO artefakter for å finne
roten.
**Scopet er smalt med vilje:** `read_file` / `read_dir` / `read_bundle` raiser fortsatt, og deres
raises telles på nøyaktig samme måte (se 1.3 — det var DEN trippelen som lå foran 400-en). Å endre
dem er en egen beslutning, ikke en jeg tar under en ordre som scoper (B) til `quick_validate`.
**MÅLT, RAPPORTERT, IKKE FIKSET.**
### Ikke gjort, og hvorfor
- **Ingen retry-logikk** (ordren forbyr).
- **Ingen ny `max_consecutive_errors_per_request`** — jeg har ingen måling som viser at den hjelper.
- **Ingen skjuling av andre unntakstyper** — begge armene er nøklet på klassen, og hver av dem har
en kjent-negativ arm som blir rød når den utvides til `Exception`.
- **`include_detailed_errors`** er IKKE skrudd på. Det ville sendt po sine feiltekster videre til
modellen på alle fire verktøy og er en flate-endring, ikke en fiks.
- **Hostet flate BEVISST urørt** — feltet er ikke i noen av hostings tre sett, og Fase 4es to
halvdeler står (MAJOR-4/S7bs eget valg gjentatt).
---
## 3. Load-bearing MÅLT
Ny fil: `tests/test_run_cli_chatclient_refusal.py`, 8 armer.
Grønn kontroll **1529 passed / 5 skipped**, golden `demo-transcript.stdout` BYTE-UENDRET
(`shasum -a 1` av INNHOLDET = `ea8c534773acdbe41ae68f2c55724d69aaf8be4f`, aldri git-blob-id-en).
Sju mutasjoner, ALLE RØDE mot HELE suiten:
| # | Mutasjon | Røde |
|---|---|---|
| M1 | detach utforsknings-armen | 2 (T1 + T3) |
| M2 | utvid utforsknings-armen til `Exception` | 5 — hvorav **4 i tester eldre enn dette arbeidet** |
| M3 | detach fullkjørings-armen | 1 (T4 ALENE — T1 grønn, altså er sømmene uavhengig gatet) |
| M4 | utvid fullkjørings-armen til `Exception` | 4 — hvorav 3 eksisterende, uavhengige vitner |
| M5 | gjeninnfør raisen i `quick_validate` | 3 |
| M6 | refuser ubetinget | 3 — hvorav 2 eksisterende (kontrollen er ekte) |
| M7 | dropp sink-appenden på den refuserte grenen | 1 (T8 ALENE) |
**T3 har ingen egen mutasjon, og det står som en ærlighets-grense.** Den pinner ordrens krav om at
allerede skrevne records overlever nekten, men `finally` kjører også ved `return` fra en `except`,
så enhver plassering av armen inne i funksjonen ville bestått den. Den er rød sammen med T1 under
M1 og ellers ikke separabel.
**En eksisterende gate ble SKREVET OM, ikke slettet:**
`test_explore_loadbearing::test_an_unknown_knowledge_base_is_refused_by_name` pinnet raisen.
Egenskapen den ble skrevet for — «refuserer, og sier hva som ER konfigurert» — er urørt og
assertert i begge halvdeler; den nye andre halvdelen driver `read_file` og er dét som holder
(B) SCOPET til `quick_validate`.
**Testdoblen fikk INGEN ny `_inner_get_response`-kropp.** Første utkast gjorde det og felte
S2.5-konsolideringsvakten (`test_scripted_client_consolidation`). Raisen bor nå i
`reply_selector`-sømmen, som den kanoniske kroppen kaller SYNKRONT (`simulation.py:443`) før den
bygger noen koroutine — samme punkt en ekte leverandørfeil ville truffet, og vakten forblir
urørt i stedet for utvidet.
---
## 4. Kommandoen som viser nekten (gratis)
```
uv run pytest tests/test_run_cli_chatclient_refusal.py -q
```

49
src/portfolio_optimiser/explore.py
View file

@ -1114,6 +1114,27 @@ def navigator_tools(
return [list_bundles, read_bundle, read_dir, read_file] return [list_bundles, read_bundle, read_dir, read_file]
def _refused(
reason: str,
*,
sink: list[QuickValidation] | None,
bundle_id: str,
proposal_json: str,
) -> dict[str, Any]:
"""The refused verdict, recorded on the same rule every other branch is (see below).
``anchored`` is ``False`` and that is a statement of fact, not a default: no base was
resolved, so no ``cost-baseline.json`` was read and this verdict was reached without the
project's own cost lines — exactly what the field says everywhere else it appears.
"""
verdict: dict[str, Any] = {"decision": "refused", "reason": reason, "anchored": False}
if sink is not None:
sink.append(
QuickValidation(bundle_id=bundle_id, proposal_json=proposal_json, verdict=verdict)
)
return verdict
def quick_validate_tool( def quick_validate_tool(
bundle_dirs: Sequence[str], *, sink: list[QuickValidation] | None = None bundle_dirs: Sequence[str], *, sink: list[QuickValidation] | None = None
) -> FunctionTool: ) -> FunctionTool:
@ -1148,9 +1169,24 @@ def quick_validate_tool(
), ),
) )
def quick_validate(bundle_id: str, proposal_json: str) -> dict[str, Any]: def quick_validate(bundle_id: str, proposal_json: str) -> dict[str, Any]:
bundle_dir = _resolve_bundle(index, bundle_id)
baseline = okf.load_optional_cost_baseline(bundle_dir)
verdict: dict[str, Any] verdict: dict[str, Any]
try:
bundle_dir = _resolve_bundle(index, bundle_id)
except ExplorationError as exc:
# A base id the model guessed wrong is a thing it can CORRECT — so it comes back as a
# verdict naming the configured ids, never as a raise. MEASURED (funn 99, Q5=B on K2):
# three consecutive calls carried ``bundle_id="renholdstekniske_funksjonskrav"``, a
# concept name guessed out of the seeded cut, and MAF turned each raise into the opaque
# ``"Error: Function failed."`` (``_tools.py:1426`` — the detail is suppressed unless
# ``include_detailed_errors``), so the ONE thing this refusal knows and the model did
# not — which ids exist — never reached it. The replies show the consequence: it went
# on guessing at the JSON format. Three in a row is
# ``DEFAULT_MAX_CONSECUTIVE_ERRORS_PER_REQUEST`` (``_tools.py:96``), after which MAF
# stops all function calling for the request. This is NOT a general softening of the
# navigator's refusals: ``read_file``/``read_dir``/``read_bundle`` still raise, and
# their raises are counted the same way (measured, reported, out of this order's scope).
return _refused(str(exc), sink=sink, bundle_id=bundle_id, proposal_json=proposal_json)
baseline = okf.load_optional_cost_baseline(bundle_dir)
try: try:
proposal = SavingsProposal.model_validate_json(proposal_json) proposal = SavingsProposal.model_validate_json(proposal_json)
except ValidationError as exc: except ValidationError as exc:
@ -1177,8 +1213,13 @@ def quick_validate_tool(
"p90": outcome.p90, "p90": outcome.p90,
} }
# Recorded AFTER the verdict is decided and on EVERY branch — an unparseable candidate is # Recorded AFTER the verdict is decided and on EVERY branch — an unparseable candidate is
# as much a thing the hypothesiser asked about as a validated one. A refused bundle id # as much a thing the hypothesiser asked about as a validated one. A refused bundle id is
# raises above and is deliberately not recorded: nothing was validated. # recorded too, on that same rule: the comment that used to stand here ("nothing was
# validated") was written for a RAISE, which left no verdict at all. Now that there IS one,
# keeping it out would make a refused call the single quick_validate outcome invisible in
# ``quick_validations`` — and an operator reading that list could not tell "never called"
# from "called three times with an id that does not exist", which is exactly the read this
# defect needed two artefacts to reconstruct.
if sink is not None: if sink is not None:
sink.append( sink.append(
QuickValidation(bundle_id=bundle_id, proposal_json=proposal_json, verdict=verdict) QuickValidation(bundle_id=bundle_id, proposal_json=proposal_json, verdict=verdict)

26
src/portfolio_optimiser/run.py
View file

@ -34,6 +34,7 @@ from pathlib import Path
from typing import Any, Literal, cast from typing import Any, Literal, cast
from agent_framework import BaseChatClient, SessionContext from agent_framework import BaseChatClient, SessionContext
from agent_framework.exceptions import ChatClientException
from pydantic import ValidationError from pydantic import ValidationError
from portfolio_optimiser.backends import Profile, get_backend, resolve_model from portfolio_optimiser.backends import Profile, get_backend, resolve_model
@ -3496,6 +3497,7 @@ def main(argv: list[str] | None = None) -> int:
exploration: ExplorationResult | None = None exploration: ExplorationResult | None = None
parked_now: PlanReviewParked | None = None parked_now: PlanReviewParked | None = None
budget_now: BudgetExceeded | None = None budget_now: BudgetExceeded | None = None
provider_now: ChatClientException | None = None
try: try:
if resumed is not None: if resumed is not None:
# The parked state, not argv, is what rebuilds the workflow: the graph has to match # The parked state, not argv, is what rebuilds the workflow: the graph has to match
@ -3543,6 +3545,17 @@ def main(argv: list[str] | None = None) -> int:
# than left to escape, because parking is what the operator ASKED for by giving # than left to escape, because parking is what the operator ASKED for by giving
# --checkpoint-dir; the artefact is where a machine reads that it happened. # --checkpoint-dir; the artefact is where a machine reads that it happened.
parked_now = parked_exc parked_now = parked_exc
except ChatClientException as provider_exc:
# Funn 99. The model endpoint rejected the request, so the run ENDED — it was never
# refused. ``run stopped:``, not ``run refused:``, and the class is what routes it here
# (the MAJOR-2 precedent, verbatim: "the argv was fine and the run had already spent
# tokens, so 'refused' would mislabel it"). MEASURED on the paid Q5=B run: an Azure 400
# ("No tool call found for function call output with call_id ...") left ``main()`` as a
# traceback, because ``ChatClientException`` is in none of this function's refusal
# tuples — the same gap ``BudgetExceeded`` was added to this very block for. Caught
# INSIDE the try/finally so the ``finally`` still writes ``{run_id}-exploration.json``:
# the run that most needs the evidence is the one a provider cut short.
provider_now = provider_exc
except BudgetExceeded as budget_exc: except BudgetExceeded as budget_exc:
# A cap that fired is a refusal at this door, not a programming error — the CLI's # A cap that fired is a refusal at this door, not a programming error — the CLI's
# existing contract for every other loader mistake below (stderr + rc 1, never a # existing contract for every other loader mistake below (stderr + rc 1, never a
@ -3574,6 +3587,11 @@ def main(argv: list[str] | None = None) -> int:
), ),
), ),
) )
if provider_now is not None:
# One line, rc 1, no traceback — and reported BEFORE the budget arm only because the
# two are mutually exclusive by construction (a single exception left the block).
print(f"run stopped: {provider_now}", file=sys.stderr)
return 1
if budget_now is not None: if budget_now is not None:
# Same shape as every other refusal in this function: one line on stderr, rc 1, no # Same shape as every other refusal in this function: one line on stderr, rc 1, no
# traceback. The artefact was already written by the ``finally`` above (``completed`` # traceback. The artefact was already written by the ``finally`` above (``completed``
@ -3858,6 +3876,14 @@ def main(argv: list[str] | None = None) -> int:
) )
), ),
) )
except ChatClientException as exc:
# Funn 99, the SECOND seam: the debate's own model calls go through the same provider, so
# an arm on the exploration block alone leaves an ordinary ``run.main([...])`` tracebacking.
# Same channel and same reason as ``ProposalReviewInputError`` below — the argv was fine and
# tokens were already spent — and a distinct class from every ``ValueError``-shaped refusal,
# so a reader can tell "the request was wrong" from "the endpoint rejected it".
print(f"run stopped: {exc}", file=sys.stderr)
return 1
except ProposalReviewInputError as exc: except ProposalReviewInputError as exc:
# A DISTINCT channel from ``run refused:`` below, and the class is what routes it here. # A DISTINCT channel from ``run refused:`` below, and the class is what routes it here.
# The argv was fine and the run had already spent tokens, so "refused" would mislabel it; # The argv was fine and the run had already spent tokens, so "refused" would mislabel it;

15
tests/test_explore_loadbearing.py
View file

@ -752,10 +752,23 @@ def test_an_unknown_knowledge_base_is_refused_by_name() -> None:
Model-chosen arguments are untrusted input. Answering an unknown id with an empty result would Model-chosen arguments are untrusted input. Answering an unknown id with an empty result would
let the manager conclude the base is empty rather than absent — the fourth face of the let the manager conclude the base is empty rather than absent — the fourth face of the
verification law, arrived at through a tool rather than a query. verification law, arrived at through a tool rather than a query.
REWRITTEN, not weakened (funn 99): ``quick_validate`` now answers with a ``refused`` VERDICT
rather than a raise, because MAF converts a tool raise into the opaque ``"Error: Function
failed."`` and counts three of them as grounds to stop function calling for the whole request —
so the one thing this refusal knows and the model does not (which ids exist) never reached it.
The property T20 was written for is untouched and asserted here in both halves: the answer is
still a refusal, and it still names what IS configured. The ``navigator_tools`` half is what
keeps the change SCOPED — those three tools still raise.
""" """
validate = explore.quick_validate_tool(("/tmp/base-a",)) validate = explore.quick_validate_tool(("/tmp/base-a",))
verdict = validate.func(bundle_id="base-b", proposal_json="{}")
assert verdict["decision"] == "refused"
assert "base-b" in verdict["reason"] and "base-a" in verdict["reason"]
read_file = next(t for t in explore.navigator_tools(("/tmp/base-a",)) if t.name == "read_file")
with pytest.raises(explore.ExplorationError) as excinfo: with pytest.raises(explore.ExplorationError) as excinfo:
validate.func(bundle_id="base-b", proposal_json="{}") read_file.func(bundle_id="base-b", path="x.md")
assert "base-a" in str(excinfo.value) assert "base-a" in str(excinfo.value)

268
tests/test_run_cli_chatclient_refusal.py Normal file
View file

@ -0,0 +1,268 @@
"""Funn 99 — a provider failure must LEAVE the CLI as one line, not as a traceback.
MEASURED FIRST, on the artefacts the paid Q5=B run left behind (``scratchpad/q5b/live/``,
never re-run here): the exploration died on
agent_framework.exceptions.ChatClientException:
<class 'FoundryChatClient'> service failed to complete the prompt: Error code: 400 -
{'error': {'message': 'No tool call found for function call output with call_id
call_AyeuYJmqvmmej9utu361Phtt.', ...}}
and that class is in NONE of ``main()``'s refusal tuples (``run.py`` catches
``FileNotFoundError, ValidationError, ValueError`` plus ``BudgetExceeded``/``PlanReviewParked``),
so the process tracebacked — the same defect class ``BudgetExceeded`` was added to that block for.
TWO seams are closed here and each one has its OWN arm, because each can regress alone:
(A) the EXPLORATION dispatch (``run.py``'s ``explore()``/``resume_exploration()`` block), which
is where the measured failure happened; and
(A2) the FULL-RUN dispatch (``run_project``), which the debate's own model calls go through —
a fix on only one of the two leaves the other tracebacking.
The KNOWN-NEGATIVE is the point of the arm, not decoration: a ``RuntimeError`` from the same seam
must still propagate. We are closing ONE named provider channel, never hiding unknown failures.
Arm (B) is the tool side, and it is authorised by the measurement rather than by symmetry: the
three failing ``quick_validate`` calls in ``Bseed-records.json`` were NOT verdicts. All three sent
``bundle_id="renholdstekniske_funksjonskrav"`` — a concept name, guessed out of the seeded cut,
never a base id — with a well-formed ``proposal_json``, so the arguments parsed against the
signature and it was ``_resolve_bundle``'s raise that MAF counted. Proof it was the raise and not
a verdict: ``q5b-Bseed-exploration.json`` records all three in ``tool_calls`` while
``quick_validations`` is EMPTY, and the sink is appended on every verdict branch.
"""
from __future__ import annotations
import json
from collections.abc import Callable
from pathlib import Path
from typing import Any
import pytest
from agent_framework import BaseChatClient
from agent_framework.exceptions import ChatClientException
from portfolio_optimiser import explore, run
from portfolio_optimiser.simulation import ScriptedChatClient
_BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
_PID = "BYGG-KONTOR-NORD"
#: The provider text VERBATIM from the measured run — truncated only where the records truncate it.
_PROVIDER_TEXT = (
"<class 'agent_framework_foundry._chat_client.FoundryChatClient'> service failed to complete "
"the prompt: Error code: 400 - {'error': {'message': 'No tool call found for function call "
"output with call_id call_AyeuYJmqvmmej9utu361Phtt.', 'type': 'invalid_request_error'}}"
)
_CONTRACT_JSON: dict[str, Any] = {
"max_rounds": 2,
"max_tokens": 50_000,
"max_stall_count": 2,
"max_reset_count": 1,
"max_plan_revisions": 0,
"enable_plan_review": False,
}
@pytest.fixture(autouse=True)
def _isolate_model_env(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
monkeypatch.delenv("PORTFOLIO_OTEL", raising=False)
def _raising_factory(exc: BaseException) -> Callable[[str], Callable[[str], BaseChatClient]]:
"""A factory whose clients raise ``exc`` on the FIRST model call, before any reply exists.
The raise lives in the ``reply_selector`` seam rather than in an ``_inner_get_response``
override, and that is deliberate: the canonical body calls the selector SYNCHRONOUSLY
(``simulation.py:443``) before it builds any coroutine, so a raise there leaves the client at
exactly the point a provider's would — and the S2.5 consolidation guard keeps its property
that there is no second copy of the scripted body anywhere in the tree.
"""
def _raise(_prompt: str, _role: str) -> str:
raise exc
def outer(_profile: Any) -> Callable[[str], BaseChatClient]:
def factory(role: str) -> BaseChatClient:
return ScriptedChatClient(reply_selector=_raise, role=role)
return factory
return outer
def _config_file(tmp_path: Path) -> str:
path = tmp_path / "exploration.json"
path.write_text(json.dumps(_CONTRACT_JSON), encoding="utf-8")
return str(path)
def _explore_argv(tmp_path: Path, *extra: str) -> list[str]:
return [
_PID,
"--docs-dir",
str(_BUNDLE_DIR),
"--bundle-dir",
str(_BUNDLE_DIR),
"--explore",
"Find the cheapest saving.",
"--explore-config",
_config_file(tmp_path),
*extra,
]
# ---------------------------------------------------------------------------------------------
# (A) the exploration dispatch
# ---------------------------------------------------------------------------------------------
def test_a_provider_failure_in_the_exploration_leaves_the_cli_as_one_line(
tmp_path, monkeypatch, capsys
) -> None:
"""T1 — POSITIVE. ``--explore`` against a client that raises ``ChatClientException`` must give
rc 1 and ONE stderr line, never a traceback.
Detach point: remove the ``except ChatClientException`` arm from the exploration block → the
exception escapes ``main()`` and pytest reports the raise instead of an rc → RED.
"""
monkeypatch.setattr(
"portfolio_optimiser.run._default_factory",
_raising_factory(ChatClientException(_PROVIDER_TEXT)),
)
rc = run.main(_explore_argv(tmp_path))
err = capsys.readouterr().err
assert rc == 1
assert "Traceback" not in err
assert len([line for line in err.splitlines() if line.strip()]) == 1
assert err.startswith("run stopped:")
assert "No tool call found for function call output" in err
def test_an_unknown_failure_from_the_same_seam_still_propagates(tmp_path, monkeypatch) -> None:
"""T2 — KNOWN-NEGATIVE. The arm is ONE named provider channel, not a blanket ``except``.
Without this, a fix written as ``except Exception`` would pass T1 and hide every programming
error the exploration can make. Detach point: widen the arm to ``Exception`` → RED.
"""
monkeypatch.setattr(
"portfolio_optimiser.run._default_factory",
_raising_factory(RuntimeError("a defect, not a provider")),
)
with pytest.raises(RuntimeError, match="a defect, not a provider"):
run.main(_explore_argv(tmp_path))
def test_the_exploration_artefact_survives_the_refusal(tmp_path, monkeypatch, capsys) -> None:
"""T3 — the evidence written BEFORE the failure must not disappear with it.
``run.py``'s ``finally`` writes ``{run_id}-exploration.json`` whatever ended the block; an arm
placed so the ``finally`` is skipped (or one that returns before it) would take the one record
of what the run had already spent with it. Detach point: catch the exception OUTSIDE the
try/finally → RED.
"""
outbox = tmp_path / "outbox"
monkeypatch.setattr(
"portfolio_optimiser.run._default_factory",
_raising_factory(ChatClientException(_PROVIDER_TEXT)),
)
rc = run.main(_explore_argv(tmp_path, "--outbox-dir", str(outbox), "--run-id", "funn99"))
assert rc == 1
assert capsys.readouterr().err.startswith("run stopped:")
artefact = outbox / "funn99-exploration.json"
assert artefact.exists()
payload = json.loads(artefact.read_text(encoding="utf-8"))
assert payload["completed"] is False
# ---------------------------------------------------------------------------------------------
# (A2) the full-run dispatch — the debate's own model calls
# ---------------------------------------------------------------------------------------------
def test_a_provider_failure_in_the_full_run_leaves_the_cli_as_one_line(monkeypatch, capsys) -> None:
"""T4 — the SECOND seam. The debate calls the same provider, and a fix on the exploration
block alone leaves an ordinary ``run.main([...])`` tracebacking.
Detach point: remove the ``except ChatClientException`` arm from the full-run dispatch → RED,
and T1 stays green — which is exactly why this is its own arm.
"""
monkeypatch.setattr(
"portfolio_optimiser.run._default_factory",
_raising_factory(ChatClientException(_PROVIDER_TEXT)),
)
rc = run.main([_PID, "--docs-dir", str(_BUNDLE_DIR), "--bundle-dir", str(_BUNDLE_DIR)])
err = capsys.readouterr().err
assert rc == 1
assert "Traceback" not in err
assert len([line for line in err.splitlines() if line.strip()]) == 1
assert err.startswith("run stopped:")
def test_an_unknown_failure_in_the_full_run_still_propagates(monkeypatch) -> None:
"""T5 — KNOWN-NEGATIVE for the second seam, for T2's reason."""
monkeypatch.setattr(
"portfolio_optimiser.run._default_factory",
_raising_factory(RuntimeError("a defect, not a provider")),
)
with pytest.raises(RuntimeError, match="a defect, not a provider"):
run.main([_PID, "--docs-dir", str(_BUNDLE_DIR), "--bundle-dir", str(_BUNDLE_DIR)])
# ---------------------------------------------------------------------------------------------
# (B) the tool — a base id the model guessed wrong is a thing it can CORRECT, not a run-ender
# ---------------------------------------------------------------------------------------------
def test_an_unknown_base_id_comes_back_as_a_refused_verdict() -> None:
"""T6 — POSITIVE. ``quick_validate`` against an id no configured base carries must RETURN a
``refused`` verdict naming the configured ids, not raise.
MEASURED root, verbatim from ``Bseed-records.json``: three calls with
``bundle_id="renholdstekniske_funksjonskrav"``. Each raise became MAF's opaque
``"Error: Function failed."`` (``_tools.py:1426``; details are suppressed unless
``include_detailed_errors``), so the reason po had written — *which* ids exist — never reached
the model, and the replies show it guessing at the JSON format instead. Three in a row hit
``DEFAULT_MAX_CONSECUTIVE_ERRORS_PER_REQUEST`` (``_tools.py:96``, value 3).
Detach point: restore the raise → RED.
"""
sink: list[explore.QuickValidation] = []
tool = explore.quick_validate_tool((str(_BUNDLE_DIR),), sink=sink)
verdict = tool.func(bundle_id="renholdstekniske_funksjonskrav", proposal_json="{}")
assert verdict["decision"] == "refused"
assert "renholdstekniske_funksjonskrav" in verdict["reason"]
assert _BUNDLE_DIR.name in verdict["reason"] or "bygg" in verdict["reason"]
assert verdict["anchored"] is False
def test_a_known_base_id_still_reaches_the_validator() -> None:
"""T7 — CONTROL. Without it, an implementation that answered ``refused`` to EVERY call would
pass T6 while destroying the tool. Detach point: return ``refused`` unconditionally → RED."""
sink: list[explore.QuickValidation] = []
tool = explore.quick_validate_tool((str(_BUNDLE_DIR),), sink=sink)
verdict = tool.func(bundle_id=_bundle_id(), proposal_json="{}")
assert verdict["decision"] == "unparseable"
def test_the_refusal_is_recorded_in_the_sink() -> None:
"""T8 — a refused call is a thing the hypothesiser ASKED for, and the artefact is where an
operator reads that it happened.
While it raised, the call left ``quick_validations`` empty and was visible only in
``tool_calls`` — which is precisely why this session had to read two artefacts to find the
root. Detach point: skip the sink append on the refused branch → RED.
"""
sink: list[explore.QuickValidation] = []
tool = explore.quick_validate_tool((str(_BUNDLE_DIR),), sink=sink)
tool.func(bundle_id="renholdstekniske_funksjonskrav", proposal_json="{}")
assert [entry.verdict["decision"] for entry in sink] == ["refused"]
assert sink[0].bundle_id == "renholdstekniske_funksjonskrav"
def _bundle_id() -> str:
from portfolio_optimiser import okf
return okf.reconcile_bundle_id(str(_BUNDLE_DIR)).id