feat(1b): proposeren får en grammatikk — strict structured output [skip-docs]

Fase 1b, funn 1b. Den første levende kjøringen brant tolv runder på svar som
ikke lot seg parse til IR-formen; e371890 gjorde teksten synlig, dette fjerner
årsaken. generate_via_llm sender nå
options={"response_format": proposal_response_format()} på hvert
genererings-kall.

Formen er MÅLT, ikke valgt. ChatOptions.response_format tar
type[BaseModel] | Mapping, og begge profiler ærer den: LOCAL sender en Mapping
ordrett til Chat Completions, AZURE (FoundryChatClient -> RawFoundryChatClient
-> RawOpenAIChatClient) konverterer samme envelope til Responses-APIets
text.format. Klassen — det korteste svaret — er avvist på bevis: gitt en klasse
konverterer klienten med type_to_response_format_param, som emitterer
minimum/exclusiveMinimum/minItems/prefixItems og et assumptions-node hvis
additionalProperties er et skjema. Azures publiserte subset utelukker alle fire.

assumptions kan ikke bare droppes, og det er også en måling: validator
._monte_carlo faller tilbake på item.unit_cost for hver kode uten bånd, så uten
bånd er alle 512 samples identiske og P10 == P50 == P90. Den stokastiske
falsifisereren ville gått inert mens den fortsatt rapporterte persentiler.
Wire-en bærer derfor et array av navngitte entries som _parse_ir folder tilbake
til IR-ens map — additivt, aldri erstatning. Skjemaet deriveres fra
SavingsProposal; sanitiseren er fail-closed (StructuredOutputUnsupported).

Load-bearing målt mot hele suiten, seks mutasjoner alle røde, grønn kontroll
864/4: detach wiringen (1) · detach sanitiseren (3) · dropp assumptions fra
skjemaet (1) · fail-closed -> stille reparasjon (1) · detach normaliseringen
(3) · erstatning i stedet for tillegg (2, inkl. golden-transkriptet).

T3 ble skrevet vakuøs først og felt av sin egen mutasjon: den påsto å bli rød
når assumptions forsvant fra skjemaet, men den scriptede klienten ignorerer
skjemaet. Testen fikk en direkte assert på skjemaet.

Ærlighets-grense: ingen betalt kjøring gjort. Testene beviser konformitet med
det dokumenterte subsettet, ikke aksept fra det levende endepunktet.

859 -> 864 passed / 4 skipped. ruff + format + mypy rene.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013EQNU4tfAhsBvdefT1jUhk

tests/test_structured_output_loadbearing.py

@@ -0,0 +1,338 @@
+"""Load-bearing: the proposer call must carry a STRICT structured-output schema (Fase 1b, funn 1b).
+
+The gap, measured on the project's first live run
+(``docs/2026-08-14-fase1b-forste-levende-kjoring.md``): ``generate_via_llm`` asked for the typed IR
+in PROSE ("Respond with ONLY a JSON object ...") and nothing else. Twelve rounds burned on replies
+that did not parse, the run died with ``BudgetExceeded``, and the validator was never reached with a
+parseable candidate. ``e371890`` made the raw text visible; this seam removes the cause: the model is
+handed a GRAMMAR, not a request.
+
+**The wire form is decided by measurement, not from memory** (MAF 1.9.0, agent-framework-openai
+1.8.2, agent-framework-foundry 1.8.2):
+
+* ``ChatOptions`` carries ``response_format: type[BaseModel] | Mapping[str, Any] | None``, and BOTH
+  wired profiles honour it. LOCAL (``OpenAIChatCompletionClient``) passes a Mapping through verbatim
+  to Chat Completions; AZURE (``FoundryChatClient`` -> ``RawFoundryChatClient`` ->
+  ``RawOpenAIChatClient``) converts the SAME mapping into the Responses API's ``text.format``. One
+  mapping, both profiles.
+* A ``type[BaseModel]`` would be the shorter spelling, and it is REJECTED here on evidence. The
+  client then converts it with ``openai.lib._parsing._completions.type_to_response_format_param``,
+  which (measured) emits ``minimum`` / ``exclusiveMinimum`` / ``minItems`` / ``prefixItems`` and an
+  ``assumptions`` node whose ``additionalProperties`` is a SCHEMA. Azure's documented structured-
+  output subset lists every one of those as unsupported and requires ``additionalProperties: false``
+  in every object
+  (https://learn.microsoft.com/azure/foundry/openai/how-to/structured-outputs). Passing our own
+  mapping is the only way to control what actually goes on the wire.
+
+**Stripping the constraints loses nothing**, and that is the point of the split: the schema's job is
+SHAPE, the deterministic validator's job is VALUES. ``minItems``/``gt=0`` are re-applied by pydantic
+in ``_parse_ir`` and by ``validate_proposal`` — which the module docstring already names as the
+reliability mechanism.
+
+**``assumptions`` may NOT simply be dropped, and that is a measurement too.** It is the one IR field
+inexpressible in the strict subset (a free-form map of 2-tuples). Dropping it looks harmless because
+the field is optional — but ``validator._monte_carlo`` samples ``item.unit_cost`` unchanged when a
+code has no band, so with no bands at all the 512 samples are IDENTICAL and P10 == P50 == P90. The
+stochastic falsifier would go inert while still reporting percentiles: a gate that can only be green,
+which is this repo's cardinal defect class. So the schema carries the bands in a strict-legal
+ARRAY-of-entries form and ``_parse_ir`` normalises them back to the IR's map — the IR itself is
+untouched.
+
+Five tests, load-bearing as a set. Each detach point is RED on its own:
+
+* T1 — the WIRING: the response_format reaches the client's ``options`` on the generation call
+  (RED when ``options=`` is dropped, or when a bare pydantic class is passed instead of the mapping);
+* T2 — the schema is inside the DOCUMENTED subset, recursively, with a paired CONTROL proving the
+  same walker finds those keywords in the RAW pydantic schema (RED when the sanitiser is detached —
+  and non-vacuous, because the control proves the walker reaches nested ``$defs`` at all);
+* T3 — the Monte Carlo falsifier SURVIVES: a wire reply with bands yields P90 > P10, with a control
+  proving a band-less reply collapses to P90 == P10 (RED when ``assumptions`` is dropped from the
+  schema or the bands do not reach the IR);
+* T4 — FAIL-CLOSED: a model carrying an inexpressible node with no declared override RAISES rather
+  than emitting an illegal schema (RED on a sanitiser that silently skips what it cannot express);
+* T5 — the round trip is VERBATIM and ADDITIVE: array-form bands reach the IR with their exact
+  values, and the IR's own map form is still accepted unchanged.
+"""
+
+from __future__ import annotations
+
+import json
+from collections.abc import Iterator, Mapping, Sequence
+from typing import Any
+
+import pytest
+from agent_framework import ChatResponse, ChatResponseUpdate, Message
+from agent_framework._types import ResponseStream
+from pydantic import BaseModel
+
+from portfolio_optimiser.budget import Budget, TokenMeter
+from portfolio_optimiser.generate import (
+    UNSUPPORTED_SCHEMA_KEYWORDS,
+    StructuredOutputUnsupported,
+    generate_via_llm,
+    proposal_response_format,
+    strict_json_schema,
+)
+from portfolio_optimiser.ir import SavingsProposal
+from portfolio_optimiser.reference_domain import load_reference_projects
+from portfolio_optimiser.simulation import ScriptedChatClient
+from portfolio_optimiser.validator import ValidatedProposal
+
+#: The instruction line ``generate._build_messages`` puts in EVERY generation prompt and nowhere
+#: else — the one identifier that separates a generation call from a debate turn.
+_GENERATION_MARK = "Respond with ONLY a JSON object"
+
+#: One affected line: 100 x 10 = 1000 total -> nominal feasible = 0.30 * 1000 = 300.
+#: A band of (8, 12) encloses the unit_cost (the IR's own model_validator requires that) and makes
+#: the sampled totals range over 800..1200, i.e. feasible 240..360. A claim of 200 therefore clears
+#: BOTH the P90 stage and the nominal stage whether or not bands are present — so the ONLY thing
+#: that differs between T3's positive and its control is the band itself.
+_CODE = "STRUCTURED-OUT-LINE"
+_QUANTITY = 100.0
+_UNIT_COST = 10.0
+_BAND_LOW = 8.0
+_BAND_HIGH = 12.0
+_CLAIM = 200.0
+#: NOT ``energy_efficiency`` — that measure would additionally hit the method cap (0.15 * 1000 = 150)
+#: and reject a claim of 200 for a reason that has nothing to do with this seam.
+_MEASURE = "behovsstyrt_drift"
+
+
+def _wire_reply(*, with_band: bool) -> str:
+    """A reply in exactly the shape the strict schema commissions: ``assumptions`` is an ARRAY of
+    entries, never the IR's map. Both arms are byte-identical apart from that array."""
+    bands = (
+        [{"code": _CODE, "low_unit_cost": _BAND_LOW, "high_unit_cost": _BAND_HIGH}]
+        if with_band
+        else []
+    )
+    return json.dumps(
+        {
+            "project_id": "FV42-GSV-E1",
+            "measure": _MEASURE,
+            "affected_items": [{"code": _CODE, "quantity": _QUANTITY, "unit_cost": _UNIT_COST}],
+            "claimed_saving_nok": _CLAIM,
+            "assumptions": bands,
+        }
+    )
+
+
+def _meter() -> TokenMeter:
+    # Caps well above what one attempt needs, so max_attempts -- not the budget -- is the bound.
+    return TokenMeter(Budget(max_tokens=10**9, max_rounds=20))
+
+
+class _OptionsRecordingChatClient(ScriptedChatClient):
+    """Records the ``options`` mapping of every call, then DELEGATES to the canonical scripted body.
+
+    This double cannot live in the ``reply_selector`` seam: that seam is handed
+    ``(prompt_blob, role)`` and returns a reply string, and ``options`` — this double's entire
+    subject — never reaches it. Registered in ``test_scripted_client_consolidation``'s
+    ``_DELEGATING_OVERRIDES`` for exactly that reason.
+    """
+
+    def __init__(self, reply: str) -> None:
+        super().__init__(reply)
+        self.seen_options: list[Mapping[str, Any]] = []
+
+    def _inner_get_response(
+        self,
+        *,
+        messages: Sequence[Message],
+        options: Mapping[str, Any],
+        stream: bool = False,
+        **kwargs: Any,
+    ) -> Any | ResponseStream[ChatResponseUpdate, ChatResponse]:
+        blob = " ".join(getattr(m, "text", "") or "" for m in messages)
+        if _GENERATION_MARK in blob:
+            self.seen_options.append(dict(options))
+        return super()._inner_get_response(
+            messages=messages, options=options, stream=stream, **kwargs
+        )
+
+
+def _sub_schemas(node: Any) -> Iterator[Mapping[str, Any]]:
+    """Every mapping in the schema tree, including those under ``$defs`` — the walker both T2 and
+    its control run, so the control genuinely proves the walker's reach."""
+    if isinstance(node, Mapping):
+        yield node
+        for value in node.values():
+            yield from _sub_schemas(value)
+    elif isinstance(node, list):
+        for value in node:
+            yield from _sub_schemas(value)
+
+
+def _keywords_found(schema: Any) -> set[str]:
+    return {
+        key for node in _sub_schemas(schema) for key in node if key in UNSUPPORTED_SCHEMA_KEYWORDS
+    }
+
+
+async def test_generation_call_carries_the_strict_schema() -> None:
+    """T1 LOAD-BEARING: the strict response_format reaches the client on the generation call.
+
+    RED when ``generate_via_llm`` stops passing ``options=``, and RED when it passes a bare
+    ``type[BaseModel]`` instead of the sanitised mapping (the assert is on the mapping's identity,
+    not merely on the key's presence)."""
+    project = load_reference_projects()[0]
+    client = _OptionsRecordingChatClient(_wire_reply(with_band=True))
+
+    await generate_via_llm(client, project, "", _meter(), max_attempts=1)
+
+    # Control FIRST: a positive assert over an empty list would pass vacuously.
+    assert client.seen_options, "no generation call was observed — the assert below proves nothing"
+    assert client.seen_options[0].get("response_format") == proposal_response_format(), (
+        "the generation call did not carry the strict structured-output schema"
+    )
+
+
+def test_schema_stays_inside_the_documented_strict_subset() -> None:
+    """T2 LOAD-BEARING: the emitted schema uses only what Azure's structured-output subset allows.
+
+    RED when the sanitiser is detached (the raw pydantic schema's keywords come straight through).
+    The CONTROL is what makes it non-vacuous: the SAME walker must FIND those keywords in the raw
+    schema — including inside ``$defs``, where ``AffectedItem``'s constraints live. Without it, a
+    walker that silently visits nothing would make every assert below green."""
+    raw = SavingsProposal.model_json_schema()
+    raw_found = _keywords_found(raw)
+    assert raw_found, (
+        "control failed: the walker found NO unsupported keyword in the raw pydantic schema, so "
+        "the positive assert below cannot distinguish a working sanitiser from a dead walker"
+    )
+    assert "$defs" in raw and _keywords_found(raw["$defs"]), (
+        "control failed: the walker does not reach nested $defs, where AffectedItem's constraints "
+        "live — a sanitiser that skipped $defs would still pass"
+    )
+
+    schema = proposal_response_format()["json_schema"]["schema"]
+    assert _keywords_found(schema) == set(), (
+        f"emitted schema carries keywords outside the documented strict subset: "
+        f"{sorted(_keywords_found(schema))}"
+    )
+    for node in _sub_schemas(schema):
+        if "properties" not in node:
+            continue
+        assert node.get("additionalProperties") is False, (
+            f"object node without ``additionalProperties: false``: {sorted(node.get('properties'))}"
+        )
+        assert set(node.get("required", [])) == set(node["properties"]), (
+            f"strict mode requires EVERY property listed as required; node "
+            f"{sorted(node['properties'])} lists {sorted(node.get('required', []))}"
+        )
+
+
+async def test_assumption_bands_keep_the_monte_carlo_falsifier_alive() -> None:
+    """T3 LOAD-BEARING: the commissioned schema can still carry uncertainty bands, and they reach
+    the Monte Carlo.
+
+    Two halves, because a scripted reply cannot be constrained by the schema the way a live model
+    is — so the schema half must be asserted DIRECTLY or this test could not tell a schema that
+    commissions bands from one that does not:
+
+    * the schema still COMMISSIONS the bands (RED when ``assumptions`` is dropped from it — which
+      would leave a live model unable to supply a band at all);
+    * a band that does arrive REACHES the Monte Carlo (RED when the array form is not folded into
+      the IR).
+
+    The CONTROL is the band-less arm — it proves the spread asserted below is caused by the band
+    and not by the Monte Carlo being noisy in general."""
+    project = load_reference_projects()[0]
+
+    schema = proposal_response_format()["json_schema"]["schema"]
+    assert "assumptions" in schema["properties"], (
+        "the schema no longer commissions uncertainty bands — a live model constrained by it could "
+        "not supply one, and the Monte Carlo would be degenerate on every generated proposal"
+    )
+    assert "assumptions" in schema["required"]
+
+    banded = await generate_via_llm(
+        _OptionsRecordingChatClient(_wire_reply(with_band=True)),
+        project,
+        "",
+        _meter(),
+        max_attempts=1,
+    )
+    bandless = await generate_via_llm(
+        _OptionsRecordingChatClient(_wire_reply(with_band=False)),
+        project,
+        "",
+        _meter(),
+        max_attempts=1,
+    )
+
+    assert isinstance(banded.outcome, ValidatedProposal), f"fixture: {banded.outcome}"
+    assert isinstance(bandless.outcome, ValidatedProposal), f"fixture: {bandless.outcome}"
+
+    # The control: with no band the falsifier is degenerate — this is the state the seam must NOT
+    # silently ship.
+    assert bandless.outcome.p90 == bandless.outcome.p10, (
+        "control failed: a band-less proposal already spreads, so the spread asserted below would "
+        "not prove the band arrived"
+    )
+    assert banded.outcome.p90 > banded.outcome.p10, (
+        "the uncertainty band did not reach the Monte Carlo — the stochastic falsifier is inert"
+    )
+
+
+def test_sanitiser_is_fail_closed_on_an_inexpressible_node() -> None:
+    """T4 LOAD-BEARING: a node the strict subset cannot express RAISES rather than being emitted or
+    silently skipped.
+
+    Without this, a field added to the IR later would quietly produce an illegal schema and turn
+    every live generation call into a 400 — or, worse, be dropped so the model is never asked for
+    it. Validation, never repair (mirrors ``write_concept_file`` / ``promote_verdict``)."""
+
+    class _FreeFormMap(BaseModel):
+        label: str
+        bands: dict[str, float]  # free-form map: additionalProperties is a SCHEMA, not ``false``
+
+    with pytest.raises(StructuredOutputUnsupported):
+        strict_json_schema(_FreeFormMap)
+
+    class _Expressible(BaseModel):
+        label: str
+        count: int
+
+    # Control: the raise above is caused by the inexpressible node, not by the sanitiser rejecting
+    # everything it is handed.
+    assert strict_json_schema(_Expressible)["properties"].keys() == {"label", "count"}
+
+
+async def test_band_round_trip_is_verbatim_and_the_ir_map_form_still_parses() -> None:
+    """T5 LOAD-BEARING: the array form reaches the IR with its EXACT values, and the change is
+    ADDITIVE — the IR's own map form is still accepted.
+
+    RED when the normalisation is detached (the array never becomes a map), and RED when it is
+    written as a REPLACEMENT rather than an addition (the map form would stop parsing, which would
+    break every existing scripted reply in the suite)."""
+    project = load_reference_projects()[0]
+
+    from_array = await generate_via_llm(
+        _OptionsRecordingChatClient(_wire_reply(with_band=True)),
+        project,
+        "",
+        _meter(),
+        max_attempts=1,
+    )
+    assert isinstance(from_array.outcome, ValidatedProposal)
+    assert from_array.outcome.proposal.assumptions == {_CODE: (_BAND_LOW, _BAND_HIGH)}, (
+        "the array-form band did not reach the IR verbatim"
+    )
+
+    map_form = json.dumps(
+        {
+            "project_id": "FV42-GSV-E1",
+            "measure": _MEASURE,
+            "affected_items": [{"code": _CODE, "quantity": _QUANTITY, "unit_cost": _UNIT_COST}],
+            "claimed_saving_nok": _CLAIM,
+            "assumptions": {_CODE: [_BAND_LOW, _BAND_HIGH]},
+        }
+    )
+    from_map = await generate_via_llm(
+        _OptionsRecordingChatClient(map_form), project, "", _meter(), max_attempts=1
+    )
+    assert isinstance(from_map.outcome, ValidatedProposal), (
+        "the IR's own map form stopped parsing — the normalisation replaced rather than extended"
+    )
+    assert from_map.outcome.proposal.assumptions == {_CODE: (_BAND_LOW, _BAND_HIGH)}