open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

fix(validator): stage 0 navngir ALLE baseline-overtredelser, ikke bare den foerste

Browse source 
K2-funn (b), maalt live i oekt 94 (docs/2026-09-06-major2-levende-k2.md § 4,
kjoering 3, max_attempts=3):

  1000/500 -> "quantity 1000 ... baseline 1250" -> 1188/350
           -> "unit_cost 350 ... baseline 850"  -> 1000/850 -> forsoekene brukt opp

Steg 5 mater avvisningsgrunnen ORDRETT inn i neste forsoeks prompt, saa en
melding som navngir ETT felt leses som en instruks om aa rette det feltet.
Modellen fant hver riktig verdi og aldri begge samtidig: den rettet feltet
avvisningen navnga og brakk det andre. Loekka oscillerte i stedet for aa
konvergere.

Endringen gjelder KUN meldingens fullstendighet. D6 er uendret: en hvilken som
helst overtredelse avviser fortsatt, i samme stage, foer loeseren.
max_attempts heves IKKE og eksponeres IKKE.

Hver overtredelse beholder dagens setning ORDRETT, sammenfoeyd med "; ", saa
NOEYAKTIG EN overtredelse rendres byte-identisk med foer - det er dette som
holder de eksisterende delstreng-assertene i S4.0-, reserve- og
levert-bundle-gatene staaende. Skilletegnet er valgt framfor linjeskift fordi
Rejection.reason ogsaa lander i outbox-JSON, de hostede payloadene og
terminal-notisene.

Rekkefoelgen er FORSLAGETS egen (linjer i oppgitt rekkefoelge, quantity foer
unit_cost i en linje), saa to identiske forsoek gir to identiske prompter. En
ukjent kostkode bidrar med sin ENE setning og ingen magnitude-setninger: det
finnes ingen baseline-linje aa avvike fra, og en sammenligning mot ingenting er
nettopp den fabrikasjonen dette steget finnes for.

Load-bearing MAALT (tests/test_stage0_all_violations_loadbearing.py, 6 armer),
seks mutasjoner ALLE ROEDE mot HELE suiten + groenn kontroll 1387/5 (fra
1381/5; +6, 0 fjernet - strengt supersett) og golden demo-transcript.stdout
BYTE-UENDRET (shasum -a 1 av INNHOLDET = ea8c534773acdbe41ae68f2c55724d69aaf8be4f):
returner ved foerste overtredelse (4 roede) - rapporter kun den siste (4) -
ustabil rekkefoelge i linja (1) - over-rapporter et felt som er INNENFOR
toleransen (28) - drift enkelt-overtredelsens form (1) - la en ukjent kode
ogsaa emittere magnitude-setninger (19).

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 00:38:50 +02:00
commit 78e8e39147
2 changed files with 187 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

39
src/portfolio_optimiser/validator.py
View file

@ -163,32 +163,49 @@ def _reconcile_against_baseline(
than ``tolerance`` (relative to the BASELINE value, which is the ground truth) — a real code
carrying a fabricated magnitude.
Returns the first ``Rejection`` (validator's own type — never a new gate), or ``None`` when the
proposal reconciles. Items are checked in their stated order so the reason is deterministic.
A validation, never a repair: the proposal is rejected, not silently corrected to the baseline."""
Returns ONE ``Rejection`` (validator's own type — never a new gate) naming EVERY violation the
attempt carries, or ``None`` when the proposal reconciles. A validation, never a repair: the
proposal is rejected, not silently corrected to the baseline.
**Completeness is load-bearing (K2 finding (b), measured live in økt 94).** Step 5 feeds this
reason VERBATIM into the next attempt's prompt, so a message naming only the FIRST violation
reads as an instruction to fix that one field. Measured on a line broken in both fields
(``docs/2026-09-06-major2-levende-k2.md`` § 4, run 3): the proposer fixed the quantity and
rebroke the unit cost, then fixed the unit cost and rebroke the quantity, and ran out of
attempts — it found each correct value and never both at once. Reporting them together is what
lets the loop converge under the EXISTING cap (``max_attempts`` is not raised, and stays
unexposed). The verdict itself (D6) is unchanged: any violation still rejects, in this stage,
before the solver.
Each violation keeps its sentence VERBATIM, joined with ``"; "``, so exactly one violation
renders byte-identically to before. The joiner is a separator no single-line renderer can
break — chosen over a newline because ``Rejection.reason`` also lands in outbox JSON, the
hosted payloads and terminal notices. Order is the PROPOSAL's own — items in stated order,
``quantity`` before ``unit_cost`` within an item — so two identical attempts get two identical
prompts. An unknown code contributes its one sentence and NO magnitude sentences: there is no
baseline line for its figures to deviate from, and a comparison against nothing is exactly the
fabrication this stage exists to catch."""
violations: list[str] = []
for item in proposal.affected_items:
line = baseline.items.get(item.code)
if line is None:
return Rejection(
proposal=proposal,
reason=(
violations.append(
f"unknown cost code {item.code!r}: not in project {baseline.project_id}'s "
f"cost baseline ({len(baseline.items)} known codes)"
),
)
continue
for field, claimed, actual in (
("quantity", item.quantity, line.quantity),
("unit_cost", item.unit_cost, line.unit_cost),
):
if abs(claimed - actual) > tolerance * actual:
return Rejection(
proposal=proposal,
reason=(
violations.append(
f"{field} {claimed:g} for cost code {item.code!r} is outside the "
f"{tolerance:.1%} tolerance around the baseline {field} {actual:g}"
),
)
if not violations:
return None
return Rejection(proposal=proposal, reason="; ".join(violations))
def validate_proposal(

154
tests/test_stage0_all_violations_loadbearing.py Normal file
View file

@ -0,0 +1,154 @@
"""Stage 0 reports EVERY baseline violation of an attempt, not just the first one it meets.
K2 finding (b), measured live in økt 94 (``docs/2026-09-06-major2-levende-k2.md`` § 4, run 3, at
``max_attempts=3``)::
1000/500 -> "quantity 1000 ... baseline 1250" -> 1188/350 -> "unit_cost 350 ... baseline 850"
-> 1000/850 -> attempts exhausted
The model found each correct value and never both at once: Step 5 feeds the rejection reason
VERBATIM into the next attempt's prompt, so a message naming ONE field is an instruction to fix
that field — and the proposer duly rebroke the other. The loop oscillated instead of converging,
under a cap that is deliberately NOT raised (``max_attempts`` stays 3 and stays unexposed).
The change is to the message's COMPLETENESS only. D6 is untouched: a proposal with any violation is
still rejected, by the same stage, before the solver. Each violation keeps TODAY'S sentence
verbatim; a run with exactly one violation is byte-identical to before (T2), which is what keeps
every existing substring assertion in ``test_s40_cost_baseline_loadbearing`` / the reserve and
delivered-bundle gates standing. The joiner is ``"; "`` — a separator no single-line renderer can
break, chosen over a newline because ``Rejection.reason`` also lands in outbox JSON, the hosted
400/200 payloads and terminal notices.
Order is STABLE and is the proposal's own: items in stated order, and within an item ``quantity``
before ``unit_cost`` (T3). An unknown cost code contributes its ONE sentence and no magnitude
sentences — there is no baseline line to compare against, so a magnitude claim there would be a
fabricated comparison (T5).
Measured mutations (all red against the WHOLE suite): return on the first violation · report only
the last · unstable within-item order · over-report a field that is INSIDE tolerance · drift the
single-violation form · emit magnitude sentences for an unknown code.
"""
from __future__ import annotations
from portfolio_optimiser.ir import AffectedItem, CostBaseline, CostBaselineLine, SavingsProposal
from portfolio_optimiser.validator import Rejection, ValidatedProposal, validate_proposal
# The § 4 run-3 magnitudes, verbatim: the baseline line the live model oscillated around.
_BASELINE = CostBaseline(
project_id="K2",
items={
"A": CostBaselineLine(code="A", quantity=1250.0, unit_cost=850.0),
"B": CostBaselineLine(code="B", quantity=400.0, unit_cost=200.0),
},
)
def _proposal(*items: AffectedItem, claimed: float = 1000.0) -> SavingsProposal:
return SavingsProposal(
project_id="K2",
measure="Reduce scope",
affected_items=list(items),
claimed_saving_nok=claimed,
assumptions={},
)
def _reason(proposal: SavingsProposal) -> str:
result = validate_proposal(proposal, baseline=_BASELINE)
assert isinstance(result, Rejection), "stage 0 must still REJECT — D6 is unchanged"
return result.reason
# --- T1: the finding itself ----------------------------------------------------------------------
def test_both_field_violations_are_named_in_one_rejection() -> None:
"""RED (finding b): the live attempt 1 broke BOTH fields of one line (1000 vs baseline 1250,
500 vs baseline 850) and was told only about ``quantity`` — so attempt 2 fixed the quantity and
rebroke the unit cost. Both must be named in the same reason, each in today's sentence form."""
reason = _reason(_proposal(AffectedItem(code="A", quantity=1000.0, unit_cost=500.0)))
assert (
"quantity 1000 for cost code 'A' is outside the 5.0% tolerance "
"around the baseline quantity 1250" in reason
)
assert (
"unit_cost 500 for cost code 'A' is outside the 5.0% tolerance "
"around the baseline unit_cost 850" in reason
)
def test_a_single_violation_is_byte_identical_to_before() -> None:
"""Control AND the form gate: exactly one violation must render EXACTLY today's one sentence —
no joiner, no count prefix, no enumeration. This is what keeps the pre-existing substring
assertions in the S4.0, reserve and delivered-bundle gates standing, and it is the arm that
goes red if the collect-all rewrite drifts the per-violation wording."""
reason = _reason(_proposal(AffectedItem(code="A", quantity=1000.0, unit_cost=850.0)))
assert reason == (
"quantity 1000 for cost code 'A' is outside the 5.0% tolerance "
"around the baseline quantity 1250"
)
def test_violation_order_is_the_proposal_s_own() -> None:
"""Stable order, and it is the proposal's: items in STATED order, and within an item
``quantity`` before ``unit_cost``. A set- or dict-ordered implementation passes T1 and fails
here — an unstable reason is a prompt that differs between two identical attempts."""
reason = _reason(
_proposal(
AffectedItem(code="B", quantity=100.0, unit_cost=999.0),
AffectedItem(code="A", quantity=1000.0, unit_cost=500.0),
)
)
positions = [
reason.index("quantity 100 for cost code 'B'"),
reason.index("unit_cost 999 for cost code 'B'"),
reason.index("quantity 1000 for cost code 'A'"),
reason.index("unit_cost 500 for cost code 'A'"),
]
assert positions == sorted(positions), f"unstable violation order: {reason}"
def test_a_reconciling_proposal_still_validates() -> None:
"""Causality control: the collect-all loop must not fabricate a violation. A proposal whose
every field sits ON the baseline still reaches ``ValidatedProposal`` — so T1's rejection is
caused by the two deviations, not by the new stage rejecting everything."""
result = validate_proposal(
_proposal(AffectedItem(code="A", quantity=1250.0, unit_cost=850.0), claimed=1000.0),
baseline=_BASELINE,
)
assert isinstance(result, ValidatedProposal)
def test_an_unknown_code_contributes_one_sentence_and_no_magnitudes() -> None:
"""An unknown code has NO baseline line, so there is nothing its quantity/unit_cost could
deviate FROM: it contributes its one sentence and no magnitude sentences. The second item's
real deviations are still reported in the same reason — an unknown code must not swallow the
rest of the attempt's feedback, which is exactly the defect one level up."""
reason = _reason(
_proposal(
AffectedItem(code="ZZ", quantity=7.0, unit_cost=9.0),
AffectedItem(code="A", quantity=1000.0, unit_cost=500.0),
)
)
assert "unknown cost code 'ZZ'" in reason
assert "for cost code 'ZZ' is outside" not in reason, "no baseline line to compare against"
assert "quantity 1000 for cost code 'A' is outside" in reason
assert "unit_cost 500 for cost code 'A' is outside" in reason
def test_the_reason_reaches_the_next_attempt_s_prompt_whole() -> None:
"""The seam the finding lives on: Step 5 feeds ``Rejection.reason`` verbatim into the next
attempt's prompt, so completeness at the validator IS completeness at the proposer. A message
truncated on its way into the prompt would leave the oscillation in place with a green T1."""
from portfolio_optimiser.generate import _build_messages
from portfolio_optimiser.reference_domain import load_reference_projects
rejection = validate_proposal(
_proposal(AffectedItem(code="A", quantity=1000.0, unit_cost=500.0)), baseline=_BASELINE
)
assert isinstance(rejection, Rejection)
messages = _build_messages(load_reference_projects()[0], "ctx", rejection)
prompt = "\n".join(m.text for m in messages)
assert "quantity 1000 for cost code 'A'" in prompt
assert "unit_cost 500 for cost code 'A'" in prompt