feat(mandate): a domain expert can commission WHICH approaches a run evaluates
Operator feedback: fagpersoner must be able to name the approaches a run shall
evaluate for a project, and/or ask the system for its own. Today the hypothesis
prompt is hardcoded ("Propose ONE concrete cost-saving measure") and the only
expert-facing lever, --dimension-config, FILTERS what may pass the scoping gate
rather than DIRECTING what is spent attempts on. This is the input that was
missing.
`mandate.py` is the typed commission + a fail-fast loader (mirrors
`load_dimension`/`load_goal_config`): missing or malformed refuses, because a run
must never proceed on a silently degraded commission — the coverage report would
then describe work nobody ordered. Stdlib + pydantic only, so it joins
`_MAF_FREE_MODULES` and can be mirrored to the D7 sibling.
Two refusals carry real defect classes: an EMPTY commission (no approaches and no
own proposals) is a caller error, not a result; and a duplicate approach id — or
one claiming the reserved OWN_PROPOSAL_ID — would collapse two coverage rows onto
one key (the S3.2 key-collision class), which is exactly the silence the coverage
report exists to prevent.
The numeric target is deliberately NOT duplicated here: it already lives in
GoalContract, and two copies of one number drift apart ((p) precedent). The
mandate carries intent; `announce` merely restates the figure.
`_build_messages(approach=...)` switches the opening instruction from *find one*
to *quantify THIS one*, carrying the expert's label and description VERBATIM —
the description is the reason the approach is worth trying, the one part the model
cannot infer from cost data. `approach=None` is byte-identical to the previous
prompt, so every existing run and golden is untouched.
The gate is unmoved: `validate_proposal` is called exactly as before. A
commissioned approach gets no discount — the expert directs what is EVALUATED,
never what is APPROVED.
Load-bearing MEASURED against the whole 695-test suite, four mutations all red:
detach the approach injection (2 red, control stayed green) · let a commissioned
approach bypass the validator · make the mandate loader tolerant · drop the
empty-commission refusal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULCqjLF61rehj5cZmdUoR3
This commit is contained in:
parent
44526254d0
commit
fa1fa5aafd
5 changed files with 526 additions and 4 deletions
|
|
@ -29,6 +29,7 @@ from pydantic import ValidationError
|
|||
|
||||
from portfolio_optimiser.budget import TokenMeter
|
||||
from portfolio_optimiser.ir import CostBaseline, SavingsProposal
|
||||
from portfolio_optimiser.mandate import Approach
|
||||
from portfolio_optimiser.reference_domain import Project
|
||||
from portfolio_optimiser.validator import (
|
||||
Rejection,
|
||||
|
|
@ -43,16 +44,43 @@ class GenerationError(RuntimeError):
|
|||
|
||||
|
||||
def _build_messages(
|
||||
project: Project, context: str, prior_rejection: Rejection | None = None
|
||||
project: Project,
|
||||
context: str,
|
||||
prior_rejection: Rejection | None = None,
|
||||
*,
|
||||
approach: Approach | None = None,
|
||||
) -> list[Message]:
|
||||
"""Build the hypothesis prompt. When ``prior_rejection`` is set (Step 5, målbilde §5/§7),
|
||||
append a revision block carrying ONLY the falsification *reason* verbatim — never the prior
|
||||
proposal JSON (minimal honest payload: the model must address the falsification, not parrot
|
||||
the rejected candidate back). ``None`` -> the byte-identical base prompt, so attempt 1 is
|
||||
unchanged. The reason carries only the rejected claim/feasible figures, which deliberately
|
||||
do not collide with other load-bearing prompt markers."""
|
||||
do not collide with other load-bearing prompt markers.
|
||||
|
||||
When ``approach`` is set (Trekk A3, krav 1) the opening instruction switches from *find one*
|
||||
to *quantify THIS one*: the domain expert has already decided what shall be evaluated, and the
|
||||
model's job is the numbers, not the direction. The expert's ``label`` and ``description`` are
|
||||
carried VERBATIM — the description is the reason the approach is worth trying, which is
|
||||
exactly the part the model cannot infer from the cost data. ``None`` -> the byte-identical
|
||||
base prompt, so an un-commissioned run is untouched (mirrors ``prior_rejection``).
|
||||
|
||||
The two are composable: a commissioned approach that the validator rejects is refined through
|
||||
the SAME informed-refinement block, still bound to that approach.
|
||||
"""
|
||||
if approach is None:
|
||||
head = "Propose ONE concrete cost-saving measure for this project.\n"
|
||||
else:
|
||||
head = (
|
||||
"A domain expert has commissioned ONE specific approach for this project. "
|
||||
"Quantify THAT approach as a concrete cost-saving measure — do not substitute a "
|
||||
"different measure. If it does not apply to this project, say so through the "
|
||||
"numbers rather than proposing something else.\n"
|
||||
f"Approach: {approach.label}\n"
|
||||
)
|
||||
if approach.description:
|
||||
head += f"Why the expert wants it evaluated: {approach.description}\n"
|
||||
prompt = (
|
||||
"Propose ONE concrete cost-saving measure for this project.\n"
|
||||
f"{head}"
|
||||
f"Project: {project.id} - {project.name}\n"
|
||||
f"Context (prior verdicts / cited cost docs):\n{context}\n\n"
|
||||
"Respond with ONLY a JSON object for a SavingsProposal with keys: project_id, "
|
||||
|
|
@ -111,6 +139,7 @@ async def generate_via_llm(
|
|||
*,
|
||||
max_attempts: int = 3,
|
||||
baseline: CostBaseline | None = None,
|
||||
approach: Approach | None = None,
|
||||
) -> ValidatedProposal | Rejection:
|
||||
"""Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds,
|
||||
the meter checked in this loop:
|
||||
|
|
@ -126,6 +155,13 @@ async def generate_via_llm(
|
|||
checker is a run-level, one-shot signal (run.py, before generation); seeding generation
|
||||
with the checker critique is separately scoped and NOT done here.
|
||||
|
||||
``approach`` (Trekk A3, krav 1) binds every attempt of this call to ONE expert-commissioned
|
||||
approach. It changes only the prompt: ``validate_proposal`` is called exactly as before, so a
|
||||
commissioned approach gets **no discount at the deterministic gate** — the expert directs what
|
||||
is evaluated, never what is approved. A commissioned proposal that fails is refined through the
|
||||
same informed-refinement path, still bound to that approach, and returns a typed ``Rejection``
|
||||
when the attempt budget runs out.
|
||||
|
||||
``baseline`` (S4.0) is handed straight to ``validate_proposal``, so a fabricated cost line is
|
||||
falsified per ATTEMPT like any other rejection — and its reason feeds the next attempt's prompt
|
||||
through the SAME informed-refinement path (Step 5), which is why no new loop appears here.
|
||||
|
|
@ -150,7 +186,7 @@ async def generate_via_llm(
|
|||
# attempt's prompt. ``last`` is None on attempt 1 -> the unchanged base prompt; it is
|
||||
# overwritten each round -> only the most-recent falsification ("forrige"), never an
|
||||
# accumulated history (bounded prompt growth).
|
||||
messages = _build_messages(project, context, prior_rejection=last)
|
||||
messages = _build_messages(project, context, prior_rejection=last, approach=approach)
|
||||
candidate = await _fetch_parsed(messages)
|
||||
result = validate_proposal(candidate, baseline=baseline)
|
||||
if isinstance(result, ValidatedProposal):
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue