fix(validator): stage 0 navngir ALLE baseline-overtredelser, ikke bare den foerste
K2-funn (b), maalt live i oekt 94 (docs/2026-09-06-major2-levende-k2.md § 4,
kjoering 3, max_attempts=3):
1000/500 -> "quantity 1000 ... baseline 1250" -> 1188/350
-> "unit_cost 350 ... baseline 850" -> 1000/850 -> forsoekene brukt opp
Steg 5 mater avvisningsgrunnen ORDRETT inn i neste forsoeks prompt, saa en
melding som navngir ETT felt leses som en instruks om aa rette det feltet.
Modellen fant hver riktig verdi og aldri begge samtidig: den rettet feltet
avvisningen navnga og brakk det andre. Loekka oscillerte i stedet for aa
konvergere.
Endringen gjelder KUN meldingens fullstendighet. D6 er uendret: en hvilken som
helst overtredelse avviser fortsatt, i samme stage, foer loeseren.
max_attempts heves IKKE og eksponeres IKKE.
Hver overtredelse beholder dagens setning ORDRETT, sammenfoeyd med "; ", saa
NOEYAKTIG EN overtredelse rendres byte-identisk med foer - det er dette som
holder de eksisterende delstreng-assertene i S4.0-, reserve- og
levert-bundle-gatene staaende. Skilletegnet er valgt framfor linjeskift fordi
Rejection.reason ogsaa lander i outbox-JSON, de hostede payloadene og
terminal-notisene.
Rekkefoelgen er FORSLAGETS egen (linjer i oppgitt rekkefoelge, quantity foer
unit_cost i en linje), saa to identiske forsoek gir to identiske prompter. En
ukjent kostkode bidrar med sin ENE setning og ingen magnitude-setninger: det
finnes ingen baseline-linje aa avvike fra, og en sammenligning mot ingenting er
nettopp den fabrikasjonen dette steget finnes for.
Load-bearing MAALT (tests/test_stage0_all_violations_loadbearing.py, 6 armer),
seks mutasjoner ALLE ROEDE mot HELE suiten + groenn kontroll 1387/5 (fra
1381/5; +6, 0 fjernet - strengt supersett) og golden demo-transcript.stdout
BYTE-UENDRET (shasum -a 1 av INNHOLDET = ea8c534773acdbe41ae68f2c55724d69aaf8be4f):
returner ved foerste overtredelse (4 roede) - rapporter kun den siste (4) -
ustabil rekkefoelge i linja (1) - over-rapporter et felt som er INNENFOR
toleransen (28) - drift enkelt-overtredelsens form (1) - la en ukjent kode
ogsaa emittere magnitude-setninger (19).
Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
parent
ab747bc7e8
commit
78e8e39147
2 changed files with 187 additions and 16 deletions
|
|
@ -163,32 +163,49 @@ def _reconcile_against_baseline(
|
||||||
than ``tolerance`` (relative to the BASELINE value, which is the ground truth) — a real code
|
than ``tolerance`` (relative to the BASELINE value, which is the ground truth) — a real code
|
||||||
carrying a fabricated magnitude.
|
carrying a fabricated magnitude.
|
||||||
|
|
||||||
Returns the first ``Rejection`` (validator's own type — never a new gate), or ``None`` when the
|
Returns ONE ``Rejection`` (validator's own type — never a new gate) naming EVERY violation the
|
||||||
proposal reconciles. Items are checked in their stated order so the reason is deterministic.
|
attempt carries, or ``None`` when the proposal reconciles. A validation, never a repair: the
|
||||||
A validation, never a repair: the proposal is rejected, not silently corrected to the baseline."""
|
proposal is rejected, not silently corrected to the baseline.
|
||||||
|
|
||||||
|
**Completeness is load-bearing (K2 finding (b), measured live in økt 94).** Step 5 feeds this
|
||||||
|
reason VERBATIM into the next attempt's prompt, so a message naming only the FIRST violation
|
||||||
|
reads as an instruction to fix that one field. Measured on a line broken in both fields
|
||||||
|
(``docs/2026-09-06-major2-levende-k2.md`` § 4, run 3): the proposer fixed the quantity and
|
||||||
|
rebroke the unit cost, then fixed the unit cost and rebroke the quantity, and ran out of
|
||||||
|
attempts — it found each correct value and never both at once. Reporting them together is what
|
||||||
|
lets the loop converge under the EXISTING cap (``max_attempts`` is not raised, and stays
|
||||||
|
unexposed). The verdict itself (D6) is unchanged: any violation still rejects, in this stage,
|
||||||
|
before the solver.
|
||||||
|
|
||||||
|
Each violation keeps its sentence VERBATIM, joined with ``"; "``, so exactly one violation
|
||||||
|
renders byte-identically to before. The joiner is a separator no single-line renderer can
|
||||||
|
break — chosen over a newline because ``Rejection.reason`` also lands in outbox JSON, the
|
||||||
|
hosted payloads and terminal notices. Order is the PROPOSAL's own — items in stated order,
|
||||||
|
``quantity`` before ``unit_cost`` within an item — so two identical attempts get two identical
|
||||||
|
prompts. An unknown code contributes its one sentence and NO magnitude sentences: there is no
|
||||||
|
baseline line for its figures to deviate from, and a comparison against nothing is exactly the
|
||||||
|
fabrication this stage exists to catch."""
|
||||||
|
violations: list[str] = []
|
||||||
for item in proposal.affected_items:
|
for item in proposal.affected_items:
|
||||||
line = baseline.items.get(item.code)
|
line = baseline.items.get(item.code)
|
||||||
if line is None:
|
if line is None:
|
||||||
return Rejection(
|
violations.append(
|
||||||
proposal=proposal,
|
|
||||||
reason=(
|
|
||||||
f"unknown cost code {item.code!r}: not in project {baseline.project_id}'s "
|
f"unknown cost code {item.code!r}: not in project {baseline.project_id}'s "
|
||||||
f"cost baseline ({len(baseline.items)} known codes)"
|
f"cost baseline ({len(baseline.items)} known codes)"
|
||||||
),
|
|
||||||
)
|
)
|
||||||
|
continue
|
||||||
for field, claimed, actual in (
|
for field, claimed, actual in (
|
||||||
("quantity", item.quantity, line.quantity),
|
("quantity", item.quantity, line.quantity),
|
||||||
("unit_cost", item.unit_cost, line.unit_cost),
|
("unit_cost", item.unit_cost, line.unit_cost),
|
||||||
):
|
):
|
||||||
if abs(claimed - actual) > tolerance * actual:
|
if abs(claimed - actual) > tolerance * actual:
|
||||||
return Rejection(
|
violations.append(
|
||||||
proposal=proposal,
|
|
||||||
reason=(
|
|
||||||
f"{field} {claimed:g} for cost code {item.code!r} is outside the "
|
f"{field} {claimed:g} for cost code {item.code!r} is outside the "
|
||||||
f"{tolerance:.1%} tolerance around the baseline {field} {actual:g}"
|
f"{tolerance:.1%} tolerance around the baseline {field} {actual:g}"
|
||||||
),
|
|
||||||
)
|
)
|
||||||
|
if not violations:
|
||||||
return None
|
return None
|
||||||
|
return Rejection(proposal=proposal, reason="; ".join(violations))
|
||||||
|
|
||||||
|
|
||||||
def validate_proposal(
|
def validate_proposal(
|
||||||
|
|
|
||||||
154
tests/test_stage0_all_violations_loadbearing.py
Normal file
154
tests/test_stage0_all_violations_loadbearing.py
Normal file
|
|
@ -0,0 +1,154 @@
|
||||||
|
"""Stage 0 reports EVERY baseline violation of an attempt, not just the first one it meets.
|
||||||
|
|
||||||
|
K2 finding (b), measured live in økt 94 (``docs/2026-09-06-major2-levende-k2.md`` § 4, run 3, at
|
||||||
|
``max_attempts=3``)::
|
||||||
|
|
||||||
|
1000/500 -> "quantity 1000 ... baseline 1250" -> 1188/350 -> "unit_cost 350 ... baseline 850"
|
||||||
|
-> 1000/850 -> attempts exhausted
|
||||||
|
|
||||||
|
The model found each correct value and never both at once: Step 5 feeds the rejection reason
|
||||||
|
VERBATIM into the next attempt's prompt, so a message naming ONE field is an instruction to fix
|
||||||
|
that field — and the proposer duly rebroke the other. The loop oscillated instead of converging,
|
||||||
|
under a cap that is deliberately NOT raised (``max_attempts`` stays 3 and stays unexposed).
|
||||||
|
|
||||||
|
The change is to the message's COMPLETENESS only. D6 is untouched: a proposal with any violation is
|
||||||
|
still rejected, by the same stage, before the solver. Each violation keeps TODAY'S sentence
|
||||||
|
verbatim; a run with exactly one violation is byte-identical to before (T2), which is what keeps
|
||||||
|
every existing substring assertion in ``test_s40_cost_baseline_loadbearing`` / the reserve and
|
||||||
|
delivered-bundle gates standing. The joiner is ``"; "`` — a separator no single-line renderer can
|
||||||
|
break, chosen over a newline because ``Rejection.reason`` also lands in outbox JSON, the hosted
|
||||||
|
400/200 payloads and terminal notices.
|
||||||
|
|
||||||
|
Order is STABLE and is the proposal's own: items in stated order, and within an item ``quantity``
|
||||||
|
before ``unit_cost`` (T3). An unknown cost code contributes its ONE sentence and no magnitude
|
||||||
|
sentences — there is no baseline line to compare against, so a magnitude claim there would be a
|
||||||
|
fabricated comparison (T5).
|
||||||
|
|
||||||
|
Measured mutations (all red against the WHOLE suite): return on the first violation · report only
|
||||||
|
the last · unstable within-item order · over-report a field that is INSIDE tolerance · drift the
|
||||||
|
single-violation form · emit magnitude sentences for an unknown code.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from portfolio_optimiser.ir import AffectedItem, CostBaseline, CostBaselineLine, SavingsProposal
|
||||||
|
from portfolio_optimiser.validator import Rejection, ValidatedProposal, validate_proposal
|
||||||
|
|
||||||
|
# The § 4 run-3 magnitudes, verbatim: the baseline line the live model oscillated around.
|
||||||
|
_BASELINE = CostBaseline(
|
||||||
|
project_id="K2",
|
||||||
|
items={
|
||||||
|
"A": CostBaselineLine(code="A", quantity=1250.0, unit_cost=850.0),
|
||||||
|
"B": CostBaselineLine(code="B", quantity=400.0, unit_cost=200.0),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _proposal(*items: AffectedItem, claimed: float = 1000.0) -> SavingsProposal:
|
||||||
|
return SavingsProposal(
|
||||||
|
project_id="K2",
|
||||||
|
measure="Reduce scope",
|
||||||
|
affected_items=list(items),
|
||||||
|
claimed_saving_nok=claimed,
|
||||||
|
assumptions={},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _reason(proposal: SavingsProposal) -> str:
|
||||||
|
result = validate_proposal(proposal, baseline=_BASELINE)
|
||||||
|
assert isinstance(result, Rejection), "stage 0 must still REJECT — D6 is unchanged"
|
||||||
|
return result.reason
|
||||||
|
|
||||||
|
|
||||||
|
# --- T1: the finding itself ----------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_both_field_violations_are_named_in_one_rejection() -> None:
|
||||||
|
"""RED (finding b): the live attempt 1 broke BOTH fields of one line (1000 vs baseline 1250,
|
||||||
|
500 vs baseline 850) and was told only about ``quantity`` — so attempt 2 fixed the quantity and
|
||||||
|
rebroke the unit cost. Both must be named in the same reason, each in today's sentence form."""
|
||||||
|
reason = _reason(_proposal(AffectedItem(code="A", quantity=1000.0, unit_cost=500.0)))
|
||||||
|
assert (
|
||||||
|
"quantity 1000 for cost code 'A' is outside the 5.0% tolerance "
|
||||||
|
"around the baseline quantity 1250" in reason
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
"unit_cost 500 for cost code 'A' is outside the 5.0% tolerance "
|
||||||
|
"around the baseline unit_cost 850" in reason
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_single_violation_is_byte_identical_to_before() -> None:
|
||||||
|
"""Control AND the form gate: exactly one violation must render EXACTLY today's one sentence —
|
||||||
|
no joiner, no count prefix, no enumeration. This is what keeps the pre-existing substring
|
||||||
|
assertions in the S4.0, reserve and delivered-bundle gates standing, and it is the arm that
|
||||||
|
goes red if the collect-all rewrite drifts the per-violation wording."""
|
||||||
|
reason = _reason(_proposal(AffectedItem(code="A", quantity=1000.0, unit_cost=850.0)))
|
||||||
|
assert reason == (
|
||||||
|
"quantity 1000 for cost code 'A' is outside the 5.0% tolerance "
|
||||||
|
"around the baseline quantity 1250"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_violation_order_is_the_proposal_s_own() -> None:
|
||||||
|
"""Stable order, and it is the proposal's: items in STATED order, and within an item
|
||||||
|
``quantity`` before ``unit_cost``. A set- or dict-ordered implementation passes T1 and fails
|
||||||
|
here — an unstable reason is a prompt that differs between two identical attempts."""
|
||||||
|
reason = _reason(
|
||||||
|
_proposal(
|
||||||
|
AffectedItem(code="B", quantity=100.0, unit_cost=999.0),
|
||||||
|
AffectedItem(code="A", quantity=1000.0, unit_cost=500.0),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
positions = [
|
||||||
|
reason.index("quantity 100 for cost code 'B'"),
|
||||||
|
reason.index("unit_cost 999 for cost code 'B'"),
|
||||||
|
reason.index("quantity 1000 for cost code 'A'"),
|
||||||
|
reason.index("unit_cost 500 for cost code 'A'"),
|
||||||
|
]
|
||||||
|
assert positions == sorted(positions), f"unstable violation order: {reason}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_reconciling_proposal_still_validates() -> None:
|
||||||
|
"""Causality control: the collect-all loop must not fabricate a violation. A proposal whose
|
||||||
|
every field sits ON the baseline still reaches ``ValidatedProposal`` — so T1's rejection is
|
||||||
|
caused by the two deviations, not by the new stage rejecting everything."""
|
||||||
|
result = validate_proposal(
|
||||||
|
_proposal(AffectedItem(code="A", quantity=1250.0, unit_cost=850.0), claimed=1000.0),
|
||||||
|
baseline=_BASELINE,
|
||||||
|
)
|
||||||
|
assert isinstance(result, ValidatedProposal)
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unknown_code_contributes_one_sentence_and_no_magnitudes() -> None:
|
||||||
|
"""An unknown code has NO baseline line, so there is nothing its quantity/unit_cost could
|
||||||
|
deviate FROM: it contributes its one sentence and no magnitude sentences. The second item's
|
||||||
|
real deviations are still reported in the same reason — an unknown code must not swallow the
|
||||||
|
rest of the attempt's feedback, which is exactly the defect one level up."""
|
||||||
|
reason = _reason(
|
||||||
|
_proposal(
|
||||||
|
AffectedItem(code="ZZ", quantity=7.0, unit_cost=9.0),
|
||||||
|
AffectedItem(code="A", quantity=1000.0, unit_cost=500.0),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert "unknown cost code 'ZZ'" in reason
|
||||||
|
assert "for cost code 'ZZ' is outside" not in reason, "no baseline line to compare against"
|
||||||
|
assert "quantity 1000 for cost code 'A' is outside" in reason
|
||||||
|
assert "unit_cost 500 for cost code 'A' is outside" in reason
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_reason_reaches_the_next_attempt_s_prompt_whole() -> None:
|
||||||
|
"""The seam the finding lives on: Step 5 feeds ``Rejection.reason`` verbatim into the next
|
||||||
|
attempt's prompt, so completeness at the validator IS completeness at the proposer. A message
|
||||||
|
truncated on its way into the prompt would leave the oscillation in place with a green T1."""
|
||||||
|
from portfolio_optimiser.generate import _build_messages
|
||||||
|
from portfolio_optimiser.reference_domain import load_reference_projects
|
||||||
|
|
||||||
|
rejection = validate_proposal(
|
||||||
|
_proposal(AffectedItem(code="A", quantity=1000.0, unit_cost=500.0)), baseline=_BASELINE
|
||||||
|
)
|
||||||
|
assert isinstance(rejection, Rejection)
|
||||||
|
messages = _build_messages(load_reference_projects()[0], "ctx", rejection)
|
||||||
|
prompt = "\n".join(m.text for m in messages)
|
||||||
|
assert "quantity 1000 for cost code 'A'" in prompt
|
||||||
|
assert "unit_cost 500 for cost code 'A'" in prompt
|
||||||
Loading…
Add table
Add a link
Reference in a new issue