Kjell Tore Guttormsen
fa1fa5aafd
Operator feedback: fagpersoner must be able to name the approaches a run shall
evaluate for a project, and/or ask the system for its own. Today the hypothesis
prompt is hardcoded ("Propose ONE concrete cost-saving measure") and the only
expert-facing lever, --dimension-config, FILTERS what may pass the scoping gate
rather than DIRECTING what is spent attempts on. This is the input that was
missing.
`mandate.py` is the typed commission + a fail-fast loader (mirrors
`load_dimension`/`load_goal_config`): missing or malformed refuses, because a run
must never proceed on a silently degraded commission — the coverage report would
then describe work nobody ordered. Stdlib + pydantic only, so it joins
`_MAF_FREE_MODULES` and can be mirrored to the D7 sibling.
Two refusals carry real defect classes: an EMPTY commission (no approaches and no
own proposals) is a caller error, not a result; and a duplicate approach id — or
one claiming the reserved OWN_PROPOSAL_ID — would collapse two coverage rows onto
one key (the S3.2 key-collision class), which is exactly the silence the coverage
report exists to prevent.
The numeric target is deliberately NOT duplicated here: it already lives in
GoalContract, and two copies of one number drift apart ((p) precedent). The
mandate carries intent; `announce` merely restates the figure.
`_build_messages(approach=...)` switches the opening instruction from *find one*
to *quantify THIS one*, carrying the expert's label and description VERBATIM —
the description is the reason the approach is worth trying, the one part the model
cannot infer from cost data. `approach=None` is byte-identical to the previous
prompt, so every existing run and golden is untouched.
The gate is unmoved: `validate_proposal` is called exactly as before. A
commissioned approach gets no discount — the expert directs what is EVALUATED,
never what is APPROVED.
Load-bearing MEASURED against the whole 695-test suite, four mutations all red:
detach the approach injection (2 red, control stayed green) · let a commissioned
approach bypass the validator · make the mandate loader tolerant · drop the
empty-commission refusal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULCqjLF61rehj5cZmdUoR3
100 lines
4.6 KiB
Python
100 lines
4.6 KiB
Python
"""Load-bearing: a COMMISSIONED approach reaches the proposer, and gets no discount from the
|
|
deterministic validator (Trekk A3).
|
|
|
|
The seam this pins is the whole point of krav 1: a domain expert names an approach, and the
|
|
hypothesis prompt must actually carry it — otherwise the run merely *claims* to evaluate what was
|
|
ordered. Two detach points, and each must go RED on its own:
|
|
|
|
* drop the approach block from ``_build_messages`` -> the expert's label/description never reaches
|
|
the model (``test_commissioned_approach_reaches_the_proposer``);
|
|
* let a commissioned approach bypass ``validate_proposal`` -> an expert's wish would outrank the
|
|
deterministic gate (``test_commissioned_approach_gets_no_validator_discount``).
|
|
|
|
The control (``test_no_approach_keeps_the_base_prompt_byte_identical``) proves the addition is
|
|
inert when no mandate is given: a test that can only go green proves nothing.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
from spikes._harness import FakeChatClient, message_texts
|
|
|
|
from portfolio_optimiser.budget import Budget, TokenMeter
|
|
from portfolio_optimiser.generate import _build_messages, generate_via_llm
|
|
from portfolio_optimiser.mandate import Approach
|
|
from portfolio_optimiser.reference_domain import load_reference_projects
|
|
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
|
|
|
_VALID = (
|
|
'{"project_id":"FV42-GSV-E1","measure":"Reduce scope",'
|
|
'"affected_items":[{"code":"05.2","quantity":4300,"unit_cost":215},'
|
|
'{"code":"03.1","quantity":1800,"unit_cost":310}],"claimed_saving_nok":200000}'
|
|
)
|
|
#: Same shape and a value the IR itself accepts, but above the method cap (30% of the ~1.48M
|
|
#: affected total = ~445k) -> the DETERMINISTIC VALIDATOR is what must reject it, not the parser.
|
|
_INFEASIBLE = (
|
|
'{"project_id":"FV42-GSV-E1","measure":"Reduce scope",'
|
|
'"affected_items":[{"code":"05.2","quantity":4300,"unit_cost":215},'
|
|
'{"code":"03.1","quantity":1800,"unit_cost":310}],"claimed_saving_nok":800000}'
|
|
)
|
|
|
|
_APPROACH = Approach(
|
|
id="led-retrofit",
|
|
label="LED retrofit of office lighting",
|
|
description="Operations believes the fixtures are original and run far past their rated life.",
|
|
)
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def project():
|
|
return load_reference_projects()[0] # FV42-GSV-E1
|
|
|
|
|
|
def _meter() -> TokenMeter:
|
|
return TokenMeter(Budget(max_tokens=10**9, max_rounds=20))
|
|
|
|
|
|
def test_commissioned_approach_reaches_the_proposer(project) -> None:
|
|
"""The expert's own words — label AND description — reach the hypothesis prompt VERBATIM.
|
|
|
|
The description is the part the model cannot infer from cost data (it is the expert's reason
|
|
for wanting this tried), so paraphrasing it away would quietly discard the domain knowledge.
|
|
"""
|
|
text = message_texts(_build_messages(project, "ctx", approach=_APPROACH))[0]
|
|
assert _APPROACH.label in text
|
|
assert _APPROACH.description in text
|
|
|
|
|
|
def test_no_approach_keeps_the_base_prompt_byte_identical(project) -> None:
|
|
"""CONTROL: without a mandate the prompt is byte-identical to the pre-Trekk-A one, so every
|
|
existing run and golden is untouched (mirrors ``prior_rejection=None``)."""
|
|
base = message_texts(_build_messages(project, "ctx"))[0]
|
|
assert base == message_texts(_build_messages(project, "ctx", approach=None))[0]
|
|
assert "Propose ONE concrete cost-saving measure for this project." in base
|
|
assert _APPROACH.label not in base
|
|
|
|
|
|
async def test_commissioned_approach_gets_no_validator_discount(project) -> None:
|
|
"""A commissioned approach whose numbers do not hold is REJECTED — the deterministic gate is
|
|
blocking, and being asked for by a domain expert is not a reason to pass it.
|
|
|
|
This is the one rule krav 1 must not be allowed to erode: the expert directs *what is
|
|
evaluated*, never *what is approved*.
|
|
"""
|
|
client = FakeChatClient(scripted=[_INFEASIBLE], default_reply=_INFEASIBLE)
|
|
result = await generate_via_llm(
|
|
client, project, "", _meter(), max_attempts=1, approach=_APPROACH
|
|
)
|
|
assert isinstance(result, Rejection)
|
|
|
|
|
|
async def test_commissioned_approach_still_validates_when_the_numbers_hold(project) -> None:
|
|
"""...and the same commissioned path DOES produce a validated proposal when the numbers are
|
|
feasible — so the rejection above is the validator working, not the approach path being
|
|
broken end to end."""
|
|
client = FakeChatClient(scripted=[_VALID], default_reply=_VALID)
|
|
result = await generate_via_llm(
|
|
client, project, "", _meter(), max_attempts=1, approach=_APPROACH
|
|
)
|
|
assert isinstance(result, ValidatedProposal)
|
|
assert _APPROACH.label in client.received_texts[0][0] # it went through the commissioned prompt
|