Kjell Tore Guttormsen
d6f3359fae
generate_via_llm consumed each validator Rejection internally (`last`), fed it into the next attempt's prompt, and dropped it. So Step 5 was real but unobservable: a caller could see THAT a proposal validated, never that it validated on attempt 2 after the deterministic validator falsified attempt 1. It was the one step of the eight with no output to show. The seam is a typed return value -- GenerationResult(outcome, refinements) -- rather than an out-parameter or a callback: a returned value cannot be silently lost by a caller that forgets to pass a collector, and mypy forces every call site to acknowledge it. refinements carries ONLY rejections that were actually fed back. When the attempt budget runs out the final rejection IS outcome; counting it here would be double-counting, and the bounded control test goes red on the collect-everything implementation that gets this wrong. The loop's bound is untouched: max_attempts and meter.tick_round stand, and `last` still drives the prompt alone, so prompt growth is unchanged. run.py accumulates across _evaluate calls, so _evaluate_mandate is untouched; RunResult.refinements defaults (the coverage precedent) and is concatenated across approaches rather than keyed per approach -- stated as an honesty limit. The simulation now shows it: the scripted proposer overclaims 250000, which the validator falsifies against P90 = 90000, and the corrected 30000 validates. Only the overclaim is scripted -- the rejection is computed. scripted_factory takes a per-role reply selector so this needs no second scripted client body. README records the two accuracy changes only (Step 5 is now inspectable; the simulation trace shows the correction). The level-2 publishing claim stays deferred until after the demo (O4). Load-bearing MEASURED against the full suite with a control, four mutations all red: detach the returned history (4 tests) - collect-everything (control only) - detach the run wiring (2 tests) - revert the simulation's proposer to a constant (the demo-protection test). Control: 759 passed / 4 skipped; ruff, format and mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CcWFcREUi6YPjEpN3ACDP
63 lines
2.8 KiB
Python
63 lines
2.8 KiB
Python
"""Step 10 tests — LLM->IR generation + validator-as-retry (FakeChatClient, no LLM).
|
|
|
|
A well-formed reply validates; a malformed/text-leaked reply is retried (not silently
|
|
accepted); exhausting attempts OR crossing the meter cap is a typed failure (never a
|
|
malformed proposal). Pattern: tests/spikes/test_harness.py + tests/spikes/test_c_validator.py.
|
|
"""
|
|
|
|
import pytest
|
|
from spikes._harness import FakeChatClient
|
|
|
|
from portfolio_optimiser.budget import Budget, BudgetExceeded, TokenMeter
|
|
from portfolio_optimiser.generate import generate_via_llm, generate_with_validation
|
|
from portfolio_optimiser.ir import SavingsProposal
|
|
from portfolio_optimiser.reference_domain import load_reference_projects
|
|
from portfolio_optimiser.validator import Rejection, ValidatedProposal, proposal_for
|
|
|
|
# A feasible proposal for FV42-GSV-E1: 200k <= ~445k (30% of the affected total).
|
|
_VALID = (
|
|
'{"project_id":"FV42-GSV-E1","measure":"Reduce scope",'
|
|
'"affected_items":[{"code":"05.2","quantity":4300,"unit_cost":215},'
|
|
'{"code":"03.1","quantity":1800,"unit_cost":310}],"claimed_saving_nok":200000}'
|
|
)
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def project():
|
|
return load_reference_projects()[0] # FV42-GSV-E1
|
|
|
|
|
|
def _meter() -> TokenMeter:
|
|
return TokenMeter(Budget(max_tokens=10**9, max_rounds=20))
|
|
|
|
|
|
async def test_wellformed_reply_yields_validated_proposal(project) -> None:
|
|
client = FakeChatClient(scripted=[_VALID], default_reply=_VALID)
|
|
result = await generate_via_llm(client, project, "", _meter(), max_attempts=3)
|
|
assert isinstance(result.outcome, ValidatedProposal)
|
|
assert isinstance(result.outcome.proposal, SavingsProposal)
|
|
|
|
|
|
async def test_malformed_reply_is_retried_not_silently_accepted(project) -> None:
|
|
client = FakeChatClient(scripted=["not json at all {{{", _VALID], default_reply=_VALID)
|
|
result = await generate_via_llm(client, project, "", _meter(), max_attempts=3)
|
|
assert isinstance(result.outcome, ValidatedProposal) # the malformed reply was NOT accepted
|
|
assert client.call_count >= 2 # it retried past the malformed reply
|
|
|
|
|
|
def test_generate_with_validation_is_bounded_and_typed(project) -> None:
|
|
calls = {"n": 0}
|
|
|
|
def always_out_of_range(_attempt: int) -> SavingsProposal:
|
|
calls["n"] += 1
|
|
return proposal_for(project, ["05.2", "03.1"], claimed_saving_nok=800_000)
|
|
|
|
# Exhausting attempts -> typed Rejection (never a malformed/validated proposal); self_repair bounded.
|
|
result = generate_with_validation(always_out_of_range, _meter(), max_attempts=3)
|
|
assert isinstance(result, Rejection)
|
|
assert calls["n"] == 3
|
|
|
|
# Crossing the meter cap -> typed failure (BudgetExceeded).
|
|
tiny = TokenMeter(Budget(max_tokens=10**9, max_rounds=1))
|
|
with pytest.raises(BudgetExceeded):
|
|
generate_with_validation(always_out_of_range, tiny, max_attempts=3)
|