open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(1b): proposeren får en grammatikk — strict structured output [skip-docs]

Browse source 
Fase 1b, funn 1b. Den første levende kjøringen brant tolv runder på svar som
ikke lot seg parse til IR-formen; e371890 gjorde teksten synlig, dette fjerner
årsaken. generate_via_llm sender nå
options={"response_format": proposal_response_format()} på hvert
genererings-kall.

Formen er MÅLT, ikke valgt. ChatOptions.response_format tar
type[BaseModel] | Mapping, og begge profiler ærer den: LOCAL sender en Mapping
ordrett til Chat Completions, AZURE (FoundryChatClient -> RawFoundryChatClient
-> RawOpenAIChatClient) konverterer samme envelope til Responses-APIets
text.format. Klassen — det korteste svaret — er avvist på bevis: gitt en klasse
konverterer klienten med type_to_response_format_param, som emitterer
minimum/exclusiveMinimum/minItems/prefixItems og et assumptions-node hvis
additionalProperties er et skjema. Azures publiserte subset utelukker alle fire.

assumptions kan ikke bare droppes, og det er også en måling: validator
._monte_carlo faller tilbake på item.unit_cost for hver kode uten bånd, så uten
bånd er alle 512 samples identiske og P10 == P50 == P90. Den stokastiske
falsifisereren ville gått inert mens den fortsatt rapporterte persentiler.
Wire-en bærer derfor et array av navngitte entries som _parse_ir folder tilbake
til IR-ens map — additivt, aldri erstatning. Skjemaet deriveres fra
SavingsProposal; sanitiseren er fail-closed (StructuredOutputUnsupported).

Load-bearing målt mot hele suiten, seks mutasjoner alle røde, grønn kontroll
864/4: detach wiringen (1) · detach sanitiseren (3) · dropp assumptions fra
skjemaet (1) · fail-closed -> stille reparasjon (1) · detach normaliseringen
(3) · erstatning i stedet for tillegg (2, inkl. golden-transkriptet).

T3 ble skrevet vakuøs først og felt av sin egen mutasjon: den påsto å bli rød
når assumptions forsvant fra skjemaet, men den scriptede klienten ignorerer
skjemaet. Testen fikk en direkte assert på skjemaet.

Ærlighets-grense: ingen betalt kjøring gjort. Testene beviser konformitet med
det dokumenterte subsettet, ikke aksept fra det levende endepunktet.

859 -> 864 passed / 4 skipped. ruff + format + mypy rene.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013EQNU4tfAhsBvdefT1jUhk
This commit is contained in:
Kjell Tore Guttormsen 2026-08-14 14:00:32 +02:00
commit 642ce8ae9a
4 changed files with 612 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

234
src/portfolio_optimiser/generate.py
View file

@ -22,11 +22,12 @@ Two entry points, because the LLM call is async while ``validator.self_repair``
from __future__ import annotations
import json
from collections.abc import Callable
from collections.abc import Callable, Mapping
from dataclasses import dataclass, field
from typing import Any
from agent_framework import BaseChatClient, Message
from pydantic import ValidationError
from pydantic import BaseModel, ValidationError
from portfolio_optimiser.budget import TokenMeter
from portfolio_optimiser.ir import CostBaseline, SavingsProposal
@ -44,6 +45,195 @@ class GenerationError(RuntimeError):
"""No parseable proposal could be produced within the attempt budget."""
class StructuredOutputUnsupported(TypeError):
"""A schema node cannot be expressed in the provider's strict structured-output subset.
Fail-closed, and deliberately so (mirrors ``write_concept_file`` / ``promote_verdict``:
validation, never repair). The alternative — silently dropping what cannot be expressed — would
stop commissioning a field without saying so, and the field it would have dropped first is
``assumptions``, whose absence makes the Monte Carlo falsifier inert while it still reports
percentiles. A schema this module cannot express is a decision for a human, not a default.
"""
#: Type-specific JSON Schema keywords the provider's structured-output subset does NOT support,
#: transcribed from Azure's published table (Structured outputs -> "Unsupported type-specific
#: keywords", https://learn.microsoft.com/azure/foundry/openai/how-to/structured-outputs), which
#: states it is the same subset OpenAI accepts.
#:
#: ``exclusiveMinimum``/``exclusiveMaximum`` are NOT literally in that table — it names
#: ``minimum maximum multipleOf`` — but they are the same family, and pydantic emits them for
#: ``Field(gt=...)``/``Field(lt=...)``, which is exactly how this repo's IR spells its bounds. Being
#: stricter than the table costs nothing here: every constraint stripped is re-applied by pydantic in
#: ``_parse_ir`` and by ``validate_proposal``. The schema's job is SHAPE; the validator's job is
#: VALUES. ``default`` is stripped for a different reason — strict mode requires every property to be
#: required, so a default can never apply.
UNSUPPORTED_SCHEMA_KEYWORDS = frozenset(
{
# String
"minLength",
"maxLength",
"pattern",
"format",
# Number
"minimum",
"maximum",
"multipleOf",
"exclusiveMinimum",
"exclusiveMaximum",
# Objects
"patternProperties",
"unevaluatedProperties",
"propertyNames",
"minProperties",
"maxProperties",
# Arrays
"unevaluatedItems",
"contains",
"minContains",
"maxContains",
"minItems",
"maxItems",
"uniqueItems",
# Meaningless once every property is required
"default",
}
)
#: The strict-legal stand-in for ``SavingsProposal.assumptions``.
#:
#: The IR spells the uncertainty bands as ``dict[str, tuple[float, float]]`` — a free-form map whose
#: values are tuples. Neither half is expressible: strict mode requires ``additionalProperties:
#: false`` in every object (so a map with arbitrary keys cannot be described), and tuples arrive as
#: ``prefixItems``, which is outside the supported type list. Dropping the field instead would be
#: silent damage: ``validator._monte_carlo`` falls back to the item's stated ``unit_cost`` for every
#: code with no band, so with no bands at all the samples are identical and P10 == P50 == P90 — the
#: stochastic falsifier goes inert while still reporting percentiles.
#:
#: So the WIRE carries an array of named entries and ``_parse_ir`` folds it back into the IR's map.
#: The IR itself is untouched; the entry names spell out what the tuple positions mean, which the
#: model would otherwise have to guess.
_ASSUMPTIONS_WIRE_NODE: dict[str, Any] = {
"type": "array",
"description": (
"Uncertainty band per affected cost line: the low and high unit cost the true price is "
"expected to fall between. The band MUST enclose that item's own unit_cost. Omit an entry "
"for a line whose unit cost is certain; an empty list means no uncertainty is claimed."
),
"items": {
"type": "object",
"properties": {
"code": {"type": "string"},
"low_unit_cost": {"type": "number"},
"high_unit_cost": {"type": "number"},
},
},
}
#: Dotted paths (from the root model's own properties) whose node is replaced before sanitising.
_PROPOSAL_SCHEMA_OVERRIDES: Mapping[str, dict[str, Any]] = {"assumptions": _ASSUMPTIONS_WIRE_NODE}
def _sanitise_schema_node(node: Any, *, path: str, overrides: Mapping[str, dict[str, Any]]) -> Any:
"""Rewrite one JSON Schema node into the strict subset, or raise ``StructuredOutputUnsupported``.
An override is applied FIRST, so a declared replacement is what gets checked and emitted — that
is how the one inexpressible node in this repo's IR (``assumptions``) is expressed rather than
excused. The replacement is then sanitised by the same code as everything else, so an override
cannot smuggle in an illegal node.
"""
if not isinstance(node, Mapping):
return node
if path in overrides:
node = overrides[path]
if "prefixItems" in node:
raise StructuredOutputUnsupported(
f"{path or '<root>'}: tuple types (prefixItems) are outside the strict subset"
)
for combinator in ("oneOf", "allOf"):
if combinator in node:
raise StructuredOutputUnsupported(
f"{path or '<root>'}: {combinator} is outside the strict subset (anyOf is the "
"only supported combinator)"
)
if isinstance(node.get("additionalProperties"), Mapping):
raise StructuredOutputUnsupported(
f"{path or '<root>'}: a free-form map cannot be expressed — strict mode requires "
"additionalProperties: false in every object. Declare an override that spells the "
"entries out as an array."
)
out: dict[str, Any] = {}
for key, value in node.items():
if key in UNSUPPORTED_SCHEMA_KEYWORDS:
continue
if key == "properties" and isinstance(value, Mapping):
out[key] = {
name: _sanitise_schema_node(
sub, path=f"{path}.{name}" if path else name, overrides=overrides
)
for name, sub in value.items()
}
elif key == "$defs" and isinstance(value, Mapping):
out[key] = {
name: _sanitise_schema_node(sub, path=f"$defs.{name}", overrides=overrides)
for name, sub in value.items()
}
elif key == "items":
out[key] = _sanitise_schema_node(value, path=f"{path}[]", overrides=overrides)
elif key == "anyOf" and isinstance(value, list):
out[key] = [_sanitise_schema_node(sub, path=path, overrides=overrides) for sub in value]
else:
out[key] = value
if "properties" in out:
# Strict mode's two structural demands, applied to EVERY object rather than the root only:
# no undeclared keys, and every declared key required.
out["additionalProperties"] = False
out["required"] = sorted(out["properties"])
return out
def strict_json_schema(
model: type[BaseModel], *, overrides: Mapping[str, dict[str, Any]] | None = None
) -> dict[str, Any]:
"""Derive a strict-structured-output schema from ``model``'s own pydantic schema.
DERIVED rather than hand-written on purpose: a hand-written copy of a shape that already exists
in ``ir.py`` is the second copy that drifts (kø-(p)), and it drifts silently — the model would
keep being commissioned for the old shape. ``$defs``/``$ref`` are kept (the published subset
supports definitions), so nested models need no inlining.
"""
schema = _sanitise_schema_node(model.model_json_schema(), path="", overrides=overrides or {})
assert isinstance(schema, dict) # a model's root schema is always an object
return schema
def proposal_response_format() -> dict[str, Any]:
"""The ``response_format`` mapping commissioning a ``SavingsProposal`` from the proposer.
A MAPPING, not the ``type[BaseModel]`` the option also accepts, and the reason is measured: given
a class, the client converts it with ``type_to_response_format_param``, which emits ``minimum`` /
``exclusiveMinimum`` / ``minItems`` / ``prefixItems`` and an ``assumptions`` node whose
``additionalProperties`` is a schema — four things the published subset rules out. Our own
mapping is the only way to control what reaches the wire.
ONE mapping serves both wired profiles (measured against agent-framework-openai 1.8.2 /
agent-framework-foundry 1.8.2): the Chat Completions client passes it through verbatim, and the
Responses client — which ``FoundryChatClient`` delegates to — converts this exact envelope into
``text.format``.
"""
return {
"type": "json_schema",
"json_schema": {
"name": SavingsProposal.__name__,
"strict": True,
"schema": strict_json_schema(SavingsProposal, overrides=_PROPOSAL_SCHEMA_OVERRIDES),
},
}
@dataclass(frozen=True)
class ParseFailure:
"""One model reply that did NOT parse into the typed IR, kept VERBATIM (Fase 1b, funn 1).
@ -136,6 +326,37 @@ def _build_messages(
return [Message(role="user", contents=[prompt])]
def _normalise_assumptions(data: dict[str, Any]) -> None:
"""Fold the WIRE's array-of-entries assumption bands back into the IR's ``code -> (low, high)``
map, in place.
ADDITIVE, never a replacement: a reply that already uses the IR's map form (every scripted reply
in the suite, and any model that answers without honouring the schema) is left untouched. A
malformed entry is raised as ``ValueError`` rather than ``KeyError`` on purpose — ``ValueError``
is what ``_fetch_parsed`` catches, so a bad band is captured as the parse failure it is instead
of escaping the loop and killing the run.
"""
entries = data.get("assumptions")
if not isinstance(entries, list):
return
bands: dict[str, tuple[Any, Any]] = {}
for entry in entries:
if (
not isinstance(entry, Mapping)
or not {
"code",
"low_unit_cost",
"high_unit_cost",
}
<= entry.keys()
):
raise ValueError(
f"each assumption entry needs code, low_unit_cost and high_unit_cost; got {entry!r}"
)
bands[entry["code"]] = (entry["low_unit_cost"], entry["high_unit_cost"])
data["assumptions"] = bands
def _parse_ir(text: str, project: Project) -> SavingsProposal:
"""Parse the model's structured reply into the typed IR. Raises on malformed/text-leaked
output (JSON error or Pydantic ``ValidationError``)."""
@ -143,6 +364,7 @@ def _parse_ir(text: str, project: Project) -> SavingsProposal:
if not isinstance(data, dict):
raise ValueError("reply is not a JSON object")
data.setdefault("project_id", project.id)
_normalise_assumptions(data)
return SavingsProposal(**data)
@ -231,7 +453,13 @@ async def generate_via_llm(
# Parse-robust: a malformed/text-leaked reply is retried; the meter caps total work.
while True:
meter.tick_round() # between-attempt bound (BudgetExceeded over cap)
reply = await chat_client.get_response(messages) # non-streaming
# Fase 1b, funn 1b: hand the model a GRAMMAR, not a prose request. The prompt's
# "Respond with ONLY a JSON object" line stays — a provider that ignores
# ``response_format`` (or a local model that does not implement it) must still be told
# what is wanted, and the parse-retry below remains the backstop either way.
reply = await chat_client.get_response( # non-streaming
messages, options={"response_format": proposal_response_format()}
)
_charge_usage(meter, reply)
try:
return _parse_ir(reply.text, project)