open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(mandate): a domain expert can commission WHICH approaches a run evaluates

Browse source 
Operator feedback: fagpersoner must be able to name the approaches a run shall
evaluate for a project, and/or ask the system for its own. Today the hypothesis
prompt is hardcoded ("Propose ONE concrete cost-saving measure") and the only
expert-facing lever, --dimension-config, FILTERS what may pass the scoping gate
rather than DIRECTING what is spent attempts on. This is the input that was
missing.

`mandate.py` is the typed commission + a fail-fast loader (mirrors
`load_dimension`/`load_goal_config`): missing or malformed refuses, because a run
must never proceed on a silently degraded commission — the coverage report would
then describe work nobody ordered. Stdlib + pydantic only, so it joins
`_MAF_FREE_MODULES` and can be mirrored to the D7 sibling.

Two refusals carry real defect classes: an EMPTY commission (no approaches and no
own proposals) is a caller error, not a result; and a duplicate approach id — or
one claiming the reserved OWN_PROPOSAL_ID — would collapse two coverage rows onto
one key (the S3.2 key-collision class), which is exactly the silence the coverage
report exists to prevent.

The numeric target is deliberately NOT duplicated here: it already lives in
GoalContract, and two copies of one number drift apart ((p) precedent). The
mandate carries intent; `announce` merely restates the figure.

`_build_messages(approach=...)` switches the opening instruction from *find one*
to *quantify THIS one*, carrying the expert's label and description VERBATIM —
the description is the reason the approach is worth trying, the one part the model
cannot infer from cost data. `approach=None` is byte-identical to the previous
prompt, so every existing run and golden is untouched.

The gate is unmoved: `validate_proposal` is called exactly as before. A
commissioned approach gets no discount — the expert directs what is EVALUATED,
never what is APPROVED.

Load-bearing MEASURED against the whole 695-test suite, four mutations all red:
detach the approach injection (2 red, control stayed green) · let a commissioned
approach bypass the validator · make the mandate loader tolerant · drop the
empty-commission refusal.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULCqjLF61rehj5cZmdUoR3
This commit is contained in:
Kjell Tore Guttormsen 2026-08-05 15:43:02 +02:00
commit fa1fa5aafd
5 changed files with 526 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

44
src/portfolio_optimiser/generate.py
View file

@ -29,6 +29,7 @@ from pydantic import ValidationError
from portfolio_optimiser.budget import TokenMeter
from portfolio_optimiser.ir import CostBaseline, SavingsProposal
from portfolio_optimiser.mandate import Approach
from portfolio_optimiser.reference_domain import Project
from portfolio_optimiser.validator import (
Rejection,
@ -43,16 +44,43 @@ class GenerationError(RuntimeError):
def _build_messages(
project: Project, context: str, prior_rejection: Rejection | None = None
project: Project,
context: str,
prior_rejection: Rejection | None = None,
*,
approach: Approach | None = None,
) -> list[Message]:
"""Build the hypothesis prompt. When ``prior_rejection`` is set (Step 5, målbilde §5/§7),
append a revision block carrying ONLY the falsification *reason* verbatim — never the prior
proposal JSON (minimal honest payload: the model must address the falsification, not parrot
the rejected candidate back). ``None`` -> the byte-identical base prompt, so attempt 1 is
unchanged. The reason carries only the rejected claim/feasible figures, which deliberately
do not collide with other load-bearing prompt markers."""
do not collide with other load-bearing prompt markers.
When ``approach`` is set (Trekk A3, krav 1) the opening instruction switches from *find one*
to *quantify THIS one*: the domain expert has already decided what shall be evaluated, and the
model's job is the numbers, not the direction. The expert's ``label`` and ``description`` are
carried VERBATIM — the description is the reason the approach is worth trying, which is
exactly the part the model cannot infer from the cost data. ``None`` -> the byte-identical
base prompt, so an un-commissioned run is untouched (mirrors ``prior_rejection``).
The two are composable: a commissioned approach that the validator rejects is refined through
the SAME informed-refinement block, still bound to that approach.
"""
if approach is None:
head = "Propose ONE concrete cost-saving measure for this project.\n"
else:
head = (
"A domain expert has commissioned ONE specific approach for this project. "
"Quantify THAT approach as a concrete cost-saving measure — do not substitute a "
"different measure. If it does not apply to this project, say so through the "
"numbers rather than proposing something else.\n"
f"Approach: {approach.label}\n"
)
if approach.description:
head += f"Why the expert wants it evaluated: {approach.description}\n"
prompt = (
"Propose ONE concrete cost-saving measure for this project.\n"
f"{head}"
f"Project: {project.id} - {project.name}\n"
f"Context (prior verdicts / cited cost docs):\n{context}\n\n"
"Respond with ONLY a JSON object for a SavingsProposal with keys: project_id, "
@ -111,6 +139,7 @@ async def generate_via_llm(
*,
max_attempts: int = 3,
baseline: CostBaseline | None = None,
approach: Approach | None = None,
) -> ValidatedProposal | Rejection:
"""Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds,
the meter checked in this loop:
@ -126,6 +155,13 @@ async def generate_via_llm(
checker is a run-level, one-shot signal (run.py, before generation); seeding generation
with the checker critique is separately scoped and NOT done here.
``approach`` (Trekk A3, krav 1) binds every attempt of this call to ONE expert-commissioned
approach. It changes only the prompt: ``validate_proposal`` is called exactly as before, so a
commissioned approach gets **no discount at the deterministic gate** — the expert directs what
is evaluated, never what is approved. A commissioned proposal that fails is refined through the
same informed-refinement path, still bound to that approach, and returns a typed ``Rejection``
when the attempt budget runs out.
``baseline`` (S4.0) is handed straight to ``validate_proposal``, so a fabricated cost line is
falsified per ATTEMPT like any other rejection — and its reason feeds the next attempt's prompt
through the SAME informed-refinement path (Step 5), which is why no new loop appears here.
@ -150,7 +186,7 @@ async def generate_via_llm(
# attempt's prompt. ``last`` is None on attempt 1 -> the unchanged base prompt; it is
# overwritten each round -> only the most-recent falsification ("forrige"), never an
# accumulated history (bounded prompt growth).
messages = _build_messages(project, context, prior_rejection=last)
messages = _build_messages(project, context, prior_rejection=last, approach=approach)
candidate = await _fetch_parsed(messages)
result = validate_proposal(candidate, baseline=baseline)
if isinstance(result, ValidatedProposal):

161
src/portfolio_optimiser/mandate.py Normal file
View file

@ -0,0 +1,161 @@
"""Typed IR for a run MANDATE — what a domain expert commissions a run to evaluate, and what
that run is meant to achieve (Trekk A1).
Pure module — imports **only** ``pydantic`` + stdlib. No ``agent_framework``, and deliberately
no ``contracts`` (which pulls MAF in transitively via ``backends``), so this module stays
D7-portable and is guarded by ``tests/test_okf.py::test_okf_is_maf_free`` alongside ``okf.py``
and ``dimension.py``.
**A mandate is not a dimension.** ``dimension.admits`` FILTERS what may pass the scoping gate;
a mandate DIRECTS what the run shall actually spend its attempts on. The two compose: a run may
be commissioned to evaluate three approaches *and* be scoped to one cost axis.
**A mandate is not a goal, either.** The numeric target has exactly one home already
(``contracts.GoalContract`` / ``--goals``); duplicating it here would put the same number in two
places, and two copies of one number drift apart (the ``(p)`` precedent — one quantisation order,
one source). The mandate carries the run's *intent* in plain language; the figure is read from
the goal config and merely RESTATED in the run announcement.
Two refusals are load-bearing, both at construction time:
* an **empty commission** (no approaches and no own proposals) — a run with nothing to do is a
caller error, not a result (mirrors ``budget.BudgetRefused``'s startup refusal: a pass that can
afford zero projects is refused before anything loads);
* a **duplicate approach id**, including one claiming the reserved ``OWN_PROPOSAL_ID`` — ``id`` is
the key each row of the coverage report is written under, so two rows sharing one key would
silently collapse into one. That is the S3.2 key-collision class, and silence is exactly what
the coverage report exists to prevent.
"""
from __future__ import annotations
from pathlib import Path
from pydantic import BaseModel, Field, model_validator
#: Coverage-report id for the run's OWN (non-commissioned) proposal. Reserved: an expert approach
#: may not claim it, because the two rows would collapse onto one key.
OWN_PROPOSAL_ID = "own-proposal"
class Approach(BaseModel):
"""One approach a domain expert wants evaluated for a project.
``description`` is the expert's own prose reason for wanting it tried; it is fed to the
proposer VERBATIM, because the reason is the part the model cannot infer from the cost data.
"""
id: str = Field(min_length=1)
label: str = Field(min_length=1)
description: str = ""
class Mandate(BaseModel):
"""The commission for one run: what to evaluate, and what the run is for.
``allow_own_proposals`` defaults to ``True`` so that naming approaches never silently forbids
the system from adding its own — the operator's requirement is "these **and/or** your own",
and the permissive half is the one that matches today's behaviour.
"""
objective: str = Field(min_length=1)
approaches: tuple[Approach, ...] = ()
allow_own_proposals: bool = True
success_criteria: str = ""
@model_validator(mode="after")
def _commission_is_not_empty(self) -> Mandate:
if not self.approaches and not self.allow_own_proposals:
raise ValueError(
"empty commission: a mandate with no approaches and allow_own_proposals=false "
"gives the run nothing to evaluate"
)
return self
@model_validator(mode="after")
def _approach_ids_are_unique_and_unreserved(self) -> Mandate:
seen: set[str] = set()
for approach in self.approaches:
if approach.id == OWN_PROPOSAL_ID:
raise ValueError(
f"approach id {OWN_PROPOSAL_ID!r} is reserved for the run's own proposal"
)
if approach.id in seen:
raise ValueError(f"duplicate approach id: {approach.id!r}")
seen.add(approach.id)
return self
def load_mandate(path: str | Path) -> Mandate:
"""Fail-fast standalone loader for a run mandate (mirrors ``dimension.load_dimension``).
A mandate is *authoritative startup input*, so loading is fail-fast: a missing file raises
``FileNotFoundError`` and malformed/invalid content raises ``pydantic.ValidationError``. This
is the deliberate contrast to the tolerant verdict-inbox RAW layer
(``verdicts.load_verdicts_from_dir``), which skips bad files rather than raising — a run must
never proceed on a *silently degraded* commission, because the coverage report would then
describe work nobody ordered.
:raises FileNotFoundError: ``path`` does not point at an existing file.
:raises pydantic.ValidationError: the content is not JSON, or violates the ``Mandate`` schema.
"""
p = Path(path)
if not p.is_file():
raise FileNotFoundError(f"mandate not found: {str(path)!r}")
return Mandate.model_validate_json(p.read_text(encoding="utf-8"))
def announce(
mandate: Mandate,
*,
project_id: str,
max_rounds: int,
max_tokens: int,
dimension_label: str | None = None,
goal_nok: float | None = None,
external_services: tuple[str, ...] = (),
) -> str:
"""Render the run announcement: what this run will do, BEFORE the first (paid) model call.
Deterministic and byte-stable — no wall clock, and nothing ordered by set iteration — so it
can be golden-tested. English, like every other line this CLI prints; the Norwegian
explanation of what the block means belongs in ``docs/bestille-en-kjoring.md``, next to the
domain expert who reads it.
Optional lines are OMITTED rather than rendered empty: an announcement that printed
``Scoped to: -`` would imply a scope decision nobody made.
``external_services`` is the egress declaration. It is empty until a run is given MCP servers
(Trekk B); when it is non-empty, every server named here may be contacted — and a run never
reaches a service it did not announce. The "no external services" wording is deliberate: an
omitted line reads the same as an unchecked one.
"""
lines = [
f"Run mandate for {project_id}",
f" Objective: {mandate.objective}",
]
if mandate.approaches:
suffix = " + the system's own proposals" if mandate.allow_own_proposals else " (only these)"
lines.append(
f" Evaluates: {len(mandate.approaches)} expert-proposed approach(es){suffix}"
)
lines.extend(
f" {n}. {a.id} — {a.label}"
for n, a in enumerate(mandate.approaches, start=1)
)
else:
lines.append(" Evaluates: the system's own proposals (none were commissioned)")
if dimension_label is not None:
lines.append(f" Scoped to: {dimension_label}")
if goal_nok is not None:
lines.append(f" Target: >= {goal_nok:.0f} NOK (restated from the goal config)")
lines.append(f" Stops at: {max_rounds} rounds / {max_tokens} tokens")
lines.append(
" Contacts: "
+ (", ".join(external_services) if external_services else "no external services")
)
if mandate.success_criteria:
lines.append(f" Success: {mandate.success_criteria}")
return "\n".join(lines)