fix(validator): stage 0 navngir ALLE baseline-overtredelser, ikke bare den foerste
K2-funn (b), maalt live i oekt 94 (docs/2026-09-06-major2-levende-k2.md § 4,
kjoering 3, max_attempts=3):
1000/500 -> "quantity 1000 ... baseline 1250" -> 1188/350
-> "unit_cost 350 ... baseline 850" -> 1000/850 -> forsoekene brukt opp
Steg 5 mater avvisningsgrunnen ORDRETT inn i neste forsoeks prompt, saa en
melding som navngir ETT felt leses som en instruks om aa rette det feltet.
Modellen fant hver riktig verdi og aldri begge samtidig: den rettet feltet
avvisningen navnga og brakk det andre. Loekka oscillerte i stedet for aa
konvergere.
Endringen gjelder KUN meldingens fullstendighet. D6 er uendret: en hvilken som
helst overtredelse avviser fortsatt, i samme stage, foer loeseren.
max_attempts heves IKKE og eksponeres IKKE.
Hver overtredelse beholder dagens setning ORDRETT, sammenfoeyd med "; ", saa
NOEYAKTIG EN overtredelse rendres byte-identisk med foer - det er dette som
holder de eksisterende delstreng-assertene i S4.0-, reserve- og
levert-bundle-gatene staaende. Skilletegnet er valgt framfor linjeskift fordi
Rejection.reason ogsaa lander i outbox-JSON, de hostede payloadene og
terminal-notisene.
Rekkefoelgen er FORSLAGETS egen (linjer i oppgitt rekkefoelge, quantity foer
unit_cost i en linje), saa to identiske forsoek gir to identiske prompter. En
ukjent kostkode bidrar med sin ENE setning og ingen magnitude-setninger: det
finnes ingen baseline-linje aa avvike fra, og en sammenligning mot ingenting er
nettopp den fabrikasjonen dette steget finnes for.
Load-bearing MAALT (tests/test_stage0_all_violations_loadbearing.py, 6 armer),
seks mutasjoner ALLE ROEDE mot HELE suiten + groenn kontroll 1387/5 (fra
1381/5; +6, 0 fjernet - strengt supersett) og golden demo-transcript.stdout
BYTE-UENDRET (shasum -a 1 av INNHOLDET = ea8c534773acdbe41ae68f2c55724d69aaf8be4f):
returner ved foerste overtredelse (4 roede) - rapporter kun den siste (4) -
ustabil rekkefoelge i linja (1) - over-rapporter et felt som er INNENFOR
toleransen (28) - drift enkelt-overtredelsens form (1) - la en ukjent kode
ogsaa emittere magnitude-setninger (19).
Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
parent
ab747bc7e8
commit
78e8e39147
2 changed files with 187 additions and 16 deletions
154
tests/test_stage0_all_violations_loadbearing.py
Normal file
154
tests/test_stage0_all_violations_loadbearing.py
Normal file
|
|
@ -0,0 +1,154 @@
|
|||
"""Stage 0 reports EVERY baseline violation of an attempt, not just the first one it meets.
|
||||
|
||||
K2 finding (b), measured live in økt 94 (``docs/2026-09-06-major2-levende-k2.md`` § 4, run 3, at
|
||||
``max_attempts=3``)::
|
||||
|
||||
1000/500 -> "quantity 1000 ... baseline 1250" -> 1188/350 -> "unit_cost 350 ... baseline 850"
|
||||
-> 1000/850 -> attempts exhausted
|
||||
|
||||
The model found each correct value and never both at once: Step 5 feeds the rejection reason
|
||||
VERBATIM into the next attempt's prompt, so a message naming ONE field is an instruction to fix
|
||||
that field — and the proposer duly rebroke the other. The loop oscillated instead of converging,
|
||||
under a cap that is deliberately NOT raised (``max_attempts`` stays 3 and stays unexposed).
|
||||
|
||||
The change is to the message's COMPLETENESS only. D6 is untouched: a proposal with any violation is
|
||||
still rejected, by the same stage, before the solver. Each violation keeps TODAY'S sentence
|
||||
verbatim; a run with exactly one violation is byte-identical to before (T2), which is what keeps
|
||||
every existing substring assertion in ``test_s40_cost_baseline_loadbearing`` / the reserve and
|
||||
delivered-bundle gates standing. The joiner is ``"; "`` — a separator no single-line renderer can
|
||||
break, chosen over a newline because ``Rejection.reason`` also lands in outbox JSON, the hosted
|
||||
400/200 payloads and terminal notices.
|
||||
|
||||
Order is STABLE and is the proposal's own: items in stated order, and within an item ``quantity``
|
||||
before ``unit_cost`` (T3). An unknown cost code contributes its ONE sentence and no magnitude
|
||||
sentences — there is no baseline line to compare against, so a magnitude claim there would be a
|
||||
fabricated comparison (T5).
|
||||
|
||||
Measured mutations (all red against the WHOLE suite): return on the first violation · report only
|
||||
the last · unstable within-item order · over-report a field that is INSIDE tolerance · drift the
|
||||
single-violation form · emit magnitude sentences for an unknown code.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from portfolio_optimiser.ir import AffectedItem, CostBaseline, CostBaselineLine, SavingsProposal
|
||||
from portfolio_optimiser.validator import Rejection, ValidatedProposal, validate_proposal
|
||||
|
||||
# The § 4 run-3 magnitudes, verbatim: the baseline line the live model oscillated around.
|
||||
_BASELINE = CostBaseline(
|
||||
project_id="K2",
|
||||
items={
|
||||
"A": CostBaselineLine(code="A", quantity=1250.0, unit_cost=850.0),
|
||||
"B": CostBaselineLine(code="B", quantity=400.0, unit_cost=200.0),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _proposal(*items: AffectedItem, claimed: float = 1000.0) -> SavingsProposal:
|
||||
return SavingsProposal(
|
||||
project_id="K2",
|
||||
measure="Reduce scope",
|
||||
affected_items=list(items),
|
||||
claimed_saving_nok=claimed,
|
||||
assumptions={},
|
||||
)
|
||||
|
||||
|
||||
def _reason(proposal: SavingsProposal) -> str:
|
||||
result = validate_proposal(proposal, baseline=_BASELINE)
|
||||
assert isinstance(result, Rejection), "stage 0 must still REJECT — D6 is unchanged"
|
||||
return result.reason
|
||||
|
||||
|
||||
# --- T1: the finding itself ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_both_field_violations_are_named_in_one_rejection() -> None:
|
||||
"""RED (finding b): the live attempt 1 broke BOTH fields of one line (1000 vs baseline 1250,
|
||||
500 vs baseline 850) and was told only about ``quantity`` — so attempt 2 fixed the quantity and
|
||||
rebroke the unit cost. Both must be named in the same reason, each in today's sentence form."""
|
||||
reason = _reason(_proposal(AffectedItem(code="A", quantity=1000.0, unit_cost=500.0)))
|
||||
assert (
|
||||
"quantity 1000 for cost code 'A' is outside the 5.0% tolerance "
|
||||
"around the baseline quantity 1250" in reason
|
||||
)
|
||||
assert (
|
||||
"unit_cost 500 for cost code 'A' is outside the 5.0% tolerance "
|
||||
"around the baseline unit_cost 850" in reason
|
||||
)
|
||||
|
||||
|
||||
def test_a_single_violation_is_byte_identical_to_before() -> None:
|
||||
"""Control AND the form gate: exactly one violation must render EXACTLY today's one sentence —
|
||||
no joiner, no count prefix, no enumeration. This is what keeps the pre-existing substring
|
||||
assertions in the S4.0, reserve and delivered-bundle gates standing, and it is the arm that
|
||||
goes red if the collect-all rewrite drifts the per-violation wording."""
|
||||
reason = _reason(_proposal(AffectedItem(code="A", quantity=1000.0, unit_cost=850.0)))
|
||||
assert reason == (
|
||||
"quantity 1000 for cost code 'A' is outside the 5.0% tolerance "
|
||||
"around the baseline quantity 1250"
|
||||
)
|
||||
|
||||
|
||||
def test_violation_order_is_the_proposal_s_own() -> None:
|
||||
"""Stable order, and it is the proposal's: items in STATED order, and within an item
|
||||
``quantity`` before ``unit_cost``. A set- or dict-ordered implementation passes T1 and fails
|
||||
here — an unstable reason is a prompt that differs between two identical attempts."""
|
||||
reason = _reason(
|
||||
_proposal(
|
||||
AffectedItem(code="B", quantity=100.0, unit_cost=999.0),
|
||||
AffectedItem(code="A", quantity=1000.0, unit_cost=500.0),
|
||||
)
|
||||
)
|
||||
positions = [
|
||||
reason.index("quantity 100 for cost code 'B'"),
|
||||
reason.index("unit_cost 999 for cost code 'B'"),
|
||||
reason.index("quantity 1000 for cost code 'A'"),
|
||||
reason.index("unit_cost 500 for cost code 'A'"),
|
||||
]
|
||||
assert positions == sorted(positions), f"unstable violation order: {reason}"
|
||||
|
||||
|
||||
def test_a_reconciling_proposal_still_validates() -> None:
|
||||
"""Causality control: the collect-all loop must not fabricate a violation. A proposal whose
|
||||
every field sits ON the baseline still reaches ``ValidatedProposal`` — so T1's rejection is
|
||||
caused by the two deviations, not by the new stage rejecting everything."""
|
||||
result = validate_proposal(
|
||||
_proposal(AffectedItem(code="A", quantity=1250.0, unit_cost=850.0), claimed=1000.0),
|
||||
baseline=_BASELINE,
|
||||
)
|
||||
assert isinstance(result, ValidatedProposal)
|
||||
|
||||
|
||||
def test_an_unknown_code_contributes_one_sentence_and_no_magnitudes() -> None:
|
||||
"""An unknown code has NO baseline line, so there is nothing its quantity/unit_cost could
|
||||
deviate FROM: it contributes its one sentence and no magnitude sentences. The second item's
|
||||
real deviations are still reported in the same reason — an unknown code must not swallow the
|
||||
rest of the attempt's feedback, which is exactly the defect one level up."""
|
||||
reason = _reason(
|
||||
_proposal(
|
||||
AffectedItem(code="ZZ", quantity=7.0, unit_cost=9.0),
|
||||
AffectedItem(code="A", quantity=1000.0, unit_cost=500.0),
|
||||
)
|
||||
)
|
||||
assert "unknown cost code 'ZZ'" in reason
|
||||
assert "for cost code 'ZZ' is outside" not in reason, "no baseline line to compare against"
|
||||
assert "quantity 1000 for cost code 'A' is outside" in reason
|
||||
assert "unit_cost 500 for cost code 'A' is outside" in reason
|
||||
|
||||
|
||||
def test_the_reason_reaches_the_next_attempt_s_prompt_whole() -> None:
|
||||
"""The seam the finding lives on: Step 5 feeds ``Rejection.reason`` verbatim into the next
|
||||
attempt's prompt, so completeness at the validator IS completeness at the proposer. A message
|
||||
truncated on its way into the prompt would leave the oscillation in place with a green T1."""
|
||||
from portfolio_optimiser.generate import _build_messages
|
||||
from portfolio_optimiser.reference_domain import load_reference_projects
|
||||
|
||||
rejection = validate_proposal(
|
||||
_proposal(AffectedItem(code="A", quantity=1000.0, unit_cost=500.0)), baseline=_BASELINE
|
||||
)
|
||||
assert isinstance(rejection, Rejection)
|
||||
messages = _build_messages(load_reference_projects()[0], "ctx", rejection)
|
||||
prompt = "\n".join(m.text for m in messages)
|
||||
assert "quantity 1000 for cost code 'A'" in prompt
|
||||
assert "unit_cost 500 for cost code 'A'" in prompt
|
||||
Loading…
Add table
Add a link
Reference in a new issue