feat(step5): the falsification that informed the next hypothesis now leaves the loop

generate_via_llm consumed each validator Rejection internally (`last`), fed it into the
next attempt's prompt, and dropped it. So Step 5 was real but unobservable: a caller could
see THAT a proposal validated, never that it validated on attempt 2 after the deterministic
validator falsified attempt 1. It was the one step of the eight with no output to show.

The seam is a typed return value -- GenerationResult(outcome, refinements) -- rather than an
out-parameter or a callback: a returned value cannot be silently lost by a caller that forgets
to pass a collector, and mypy forces every call site to acknowledge it.

refinements carries ONLY rejections that were actually fed back. When the attempt budget runs
out the final rejection IS outcome; counting it here would be double-counting, and the bounded
control test goes red on the collect-everything implementation that gets this wrong.

The loop's bound is untouched: max_attempts and meter.tick_round stand, and `last` still drives
the prompt alone, so prompt growth is unchanged. run.py accumulates across _evaluate calls, so
_evaluate_mandate is untouched; RunResult.refinements defaults (the coverage precedent) and is
concatenated across approaches rather than keyed per approach -- stated as an honesty limit.

The simulation now shows it: the scripted proposer overclaims 250000, which the validator
falsifies against P90 = 90000, and the corrected 30000 validates. Only the overclaim is
scripted -- the rejection is computed. scripted_factory takes a per-role reply selector so this
needs no second scripted client body.

README records the two accuracy changes only (Step 5 is now inspectable; the simulation trace
shows the correction). The level-2 publishing claim stays deferred until after the demo (O4).

Load-bearing MEASURED against the full suite with a control, four mutations all red:
detach the returned history (4 tests) - collect-everything (control only) - detach the run
wiring (2 tests) - revert the simulation's proposer to a constant (the demo-protection test).
Control: 759 passed / 4 skipped; ruff, format and mypy clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CcWFcREUi6YPjEpN3ACDP

src/portfolio_optimiser/generate.py

@@ -15,14 +15,15 @@ Two entry points, because the LLM call is async while ``validator.self_repair``
   attempts. Used for deterministic candidate sources.
 * ``generate_via_llm`` — the ASYNC LLM path: an async mirror of the same bounded retry that
   awaits the chat call (parse-retry inside the meter budget, then ``validate_proposal``).
-  Returns ``ValidatedProposal | Rejection``; never a malformed proposal; raises
-  ``BudgetExceeded`` when the meter cap is crossed.
+  Returns a ``GenerationResult`` (the outcome PLUS the falsifications that informed it); never a
+  malformed proposal; raises ``BudgetExceeded`` when the meter cap is crossed.
 """
 
 from __future__ import annotations
 
 import json
 from collections.abc import Callable
+from dataclasses import dataclass, field
 
 from agent_framework import BaseChatClient, Message
 from pydantic import ValidationError
@@ -43,6 +44,28 @@ class GenerationError(RuntimeError):
     """No parseable proposal could be produced within the attempt budget."""
 
 
+@dataclass(frozen=True)
+class GenerationResult:
+    """What one ``generate_via_llm`` call produced: the outcome, and the falsification history that
+    informed it (Step 5, målbilde §5/§7).
+
+    A TYPED RETURN VALUE rather than an out-parameter or a callback, deliberately: the informed
+    refinement loop already computed this history internally and then dropped it, so Step 5 was the
+    one step of the eight with no observable output. A returned value cannot be silently lost by a
+    caller that forgets to pass a collector, and it forces every call site to acknowledge the seam.
+
+    ``refinements`` holds ONLY the rejections that were actually fed back into a later attempt's
+    prompt — the honest reading of "informed refinement". When the attempt budget runs out, the
+    final rejection IS ``outcome``: it informed nothing and is not repeated here. So the total
+    number of validator falsifications this call produced is ``len(refinements)`` plus one when
+    ``outcome`` is itself a ``Rejection``. It is empty on the common single-attempt path, which is
+    honest rather than merely convenient: nothing was falsified, so there is nothing to show.
+    """
+
+    outcome: ValidatedProposal | Rejection
+    refinements: tuple[Rejection, ...] = field(default=())
+
+
 def _build_messages(
     project: Project,
     context: str,
@@ -140,7 +163,7 @@ async def generate_via_llm(
     max_attempts: int = 3,
     baseline: CostBaseline | None = None,
     approach: Approach | None = None,
-) -> ValidatedProposal | Rejection:
+) -> GenerationResult:
     """Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds,
     the meter checked in this loop:
 
@@ -165,8 +188,12 @@ async def generate_via_llm(
     ``baseline`` (S4.0) is handed straight to ``validate_proposal``, so a fabricated cost line is
     falsified per ATTEMPT like any other rejection — and its reason feeds the next attempt's prompt
     through the SAME informed-refinement path (Step 5), which is why no new loop appears here.
-    Returns
-    ``ValidatedProposal | Rejection``; never a malformed proposal; raises ``BudgetExceeded``
+
+    Returns a ``GenerationResult``: the ``ValidatedProposal | Rejection`` outcome plus every
+    rejection that was fed back into a later attempt's prompt. Surfacing that history changes
+    nothing about the loop's BOUND — ``max_attempts`` and ``meter.tick_round`` are exactly as
+    before ("refine until good enough" without a cap stays forbidden, §6); it only stops the loop
+    from discarding what it already knew. Never a malformed proposal; raises ``BudgetExceeded``
     when the meter cap is crossed."""
 
     async def _fetch_parsed(messages: list[Message]) -> SavingsProposal:
@@ -181,16 +208,25 @@ async def generate_via_llm(
                 continue
 
     last: Rejection | None = None
+    # The falsifications that were FED BACK, in attempt order. ``last`` still drives the PROMPT and
+    # is still overwritten each round -- only the most-recent falsification reaches the model, so
+    # prompt growth is unchanged. This list is a record for the CALLER, appended to only once a
+    # rejection is about to inform a further attempt; it is never read back into a prompt.
+    fed_back: list[Rejection] = []
     for _ in range(max_attempts):
         # Informed refinement: feed the PREVIOUS attempt's validator rejection into this
         # attempt's prompt. ``last`` is None on attempt 1 -> the unchanged base prompt; it is
         # overwritten each round -> only the most-recent falsification ("forrige"), never an
         # accumulated history (bounded prompt growth).
+        if last is not None:
+            fed_back.append(last)
         messages = _build_messages(project, context, prior_rejection=last, approach=approach)
         candidate = await _fetch_parsed(messages)
         result = validate_proposal(candidate, baseline=baseline)
         if isinstance(result, ValidatedProposal):
-            return result
+            return GenerationResult(outcome=result, refinements=tuple(fed_back))
         last = result
     assert last is not None  # max_attempts >= 1, so at least one validation ran
-    return last  # validation never passed within the attempt budget -> typed Rejection
+    # Validation never passed within the attempt budget -> typed Rejection. ``last`` is the outcome
+    # and was never fed back, so it is deliberately absent from ``refinements``.
+    return GenerationResult(outcome=last, refinements=tuple(fed_back))

src/portfolio_optimiser/run.py

@@ -131,6 +131,15 @@ class RunResult:
     #: report is honest there, because nothing was ordered. It defaults so every existing
     #: constructor call and every frozen aggregate over ``RunResult`` is unaffected.
     coverage: tuple[ApproachOutcome, ...] = ()
+    #: Step 5 (målbilde §5/§7): the validator falsifications that informed a LATER generation
+    #: attempt, in attempt order — what ``generate_via_llm`` corrected in response to, rather than
+    #: only what it ended up with. EMPTY on the common path where the first candidate validates:
+    #: nothing was falsified, so there is nothing to show. Honesty limit: with a mandate this is
+    #: the run's refinements CONCATENATED across every commissioned approach, not keyed per
+    #: approach — ``coverage`` is the per-approach report, and hanging proposals off its rows is
+    #: what ``_evaluate_mandate`` deliberately avoids. It defaults, so every existing constructor
+    #: call is unaffected (mirrors ``coverage``).
+    refinements: tuple[Rejection, ...] = ()
 
 
 @dataclass(frozen=True)
@@ -635,10 +644,18 @@ async def run_project(
     # each under the SAME meter — no new unbounded loop; the caps already in force are the bound.
     proposer_client = factory("proposer")
 
+    # Step 5 (målbilde §5/§7): generation now returns its falsification history alongside the
+    # outcome. ``_evaluate`` keeps its ``ValidatedProposal | Rejection`` shape so ``_evaluate_mandate``
+    # is untouched, and the history is accumulated here in call order — one entry per approach that
+    # needed correcting, concatenated (see ``RunResult.refinements`` for that honesty limit).
+    refinements: list[Rejection] = []
+
     async def _evaluate(approach: Approach | None) -> ValidatedProposal | Rejection:
-        return await generate_via_llm(
+        generated = await generate_via_llm(
             proposer_client, project, gen_context, meter, baseline=baseline, approach=approach
         )
+        refinements.extend(generated.refinements)
+        return generated.outcome
 
     coverage: tuple[ApproachOutcome, ...] = ()
     evaluated: tuple[tuple[str, ValidatedProposal | Rejection], ...] = ()
@@ -774,6 +791,7 @@ async def run_project(
         debate_output=debate_output,
         checker_verdict=checker_decision,
         coverage=coverage,
+        refinements=tuple(refinements),
     )
 
 

src/portfolio_optimiser/simulation.py

@@ -51,12 +51,34 @@ def _default_bundle_dir() -> Path:
     return shared_root() / "examples" / "bygg-energi-mikro"
 
 
-# A VALID SavingsProposal for BYGG-KONTOR-NORD: total = 300000 x 1.0, P90 = 0.30 x 300000 = 90000,
-# claimed 30000 <= 90000 -> validates on the first attempt (no `assumptions` -> degenerate MC).
+# Two SavingsProposals for BYGG-KONTOR-NORD: total = 300000 x 1.0, so the degenerate Monte Carlo
+# P90 = 0.30 x 300000 = 90000 (no `assumptions`). The OVERCLAIMED one asks for 250000 — parseable,
+# and internally consistent, but above P90, so the DETERMINISTIC validator falsifies it. The
+# corrected one claims 30000 <= 90000 and validates. Together they drive Step 5 (informed
+# refinement): the proposer is scripted, but the rejection that turns proposal 1 into proposal 2 is
+# genuinely computed by the validator, not scripted.
+_OVERCLAIMED_PROPOSAL = (
+    '{"measure":"LED-retrofit av kontorbelysning","affected_items":'
+    '[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":250000}'
+)
 _VALID_PROPOSAL = (
     '{"measure":"LED-retrofit av kontorbelysning","affected_items":'
     '[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
 )
+# The flip key: the overclaimed figure, which the validator's rejection reason carries and
+# ``generate._build_messages`` appends to the NEXT attempt's prompt. Verified ABSENT from the demo
+# bundle, so it cannot pre-exist in attempt 1's prompt — the correction is caused by the
+# falsification travelling back, never by the proposer simply being asked twice.
+_REJECTED_CLAIM_KEY = "250000"
+
+
+def _proposer_reply(prompt: str, _role: str) -> str:
+    """The scripted proposer, keyed on PROMPT CONTENT (the canonical client's ``reply_selector``
+    seam): it overclaims until the validator's rejection comes back in the prompt, then corrects.
+    Stateless — no per-turn counter — so the debate turns and the generation attempts share it."""
+    return _VALID_PROPOSAL if _REJECTED_CLAIM_KEY in prompt else _OVERCLAIMED_PROPOSAL
+
+
 # The checker's debate turn ends with the gate marker the run parses (run._checker_verdict).
 _CHECKER_APPROVE = "Tallene er innenfor feasibelt område og resonnementet holder. VERDICT: APPROVE"
 
@@ -145,14 +167,24 @@ class ScriptedChatClient(OpenAIChatCompletionClient):
         return _coro()
 
 
-def scripted_factory(replies: dict[str, str], sink: list[str]) -> Callable[[str], BaseChatClient]:
+def scripted_factory(
+    replies: Mapping[str, str | Callable[[str, str], str]], sink: list[str]
+) -> Callable[[str], BaseChatClient]:
     """A role-keyed client factory: ``factory("proposer")`` and ``factory("checker")`` each return a
     fresh ``ScriptedChatClient`` with that role's reply, all sharing ONE ``sink``. MAF stamps the
     proposer/checker identity from the agent name, so role-keyed stateless replies suffice (no
-    per-turn counter); the shared ``sink`` spans the debate turns and the generation call."""
+    per-turn counter); the shared ``sink`` spans the debate turns and the generation call.
+
+    A role's value may be a constant reply OR a ``reply_selector`` over ``(prompt_blob, role)`` —
+    the canonical client's existing seam, passed straight through. That is what lets a role answer
+    DIFFERENTLY on a later attempt (Step 5: the proposer corrects once the validator's rejection
+    comes back in the prompt) without a per-turn counter and without a second scripted body."""
 
     def factory(role: str) -> BaseChatClient:
-        return ScriptedChatClient(replies[role], sink, role=role)
+        reply = replies[role]
+        if callable(reply):
+            return ScriptedChatClient(sink=sink, role=role, reply_selector=reply)
+        return ScriptedChatClient(reply, sink, role=role)
 
     return factory
 
@@ -212,7 +244,10 @@ async def simulate_learning_loop(
     copy = Path(work_dir) / "bundle"
     shutil.copytree(bundle_dir, copy)
     copy_s = str(copy)
-    replies = {"proposer": _VALID_PROPOSAL, "checker": _CHECKER_APPROVE}
+    replies: dict[str, str | Callable[[str, str], str]] = {
+        "proposer": _proposer_reply,
+        "checker": _CHECKER_APPROVE,
+    }
     verdict_input = {"decision": example.decision, "rationale": persona_rationale}
 
     # Run A — empty wiki isolates the persona's NEW knowledge.
@@ -283,6 +318,21 @@ def _outcome_line(result: RunResult) -> str:
     return f"REJECTED   ({o.reason})"
 
 
+def _refinement_lines(result: RunResult) -> list[str]:
+    """Step 5 made visible: every falsification that was fed back into a further hypothesis. Empty
+    when the first candidate validated — printing nothing is the honest output there."""
+    lines = []
+    for n, rejected in enumerate(result.refinements, start=1):
+        lines.append(
+            f"  steg 5 #{n} : REJECTED   (claimed "
+            f"{rejected.proposal.claimed_saving_nok:.0f} NOK) — {rejected.reason}"
+        )
+        lines.append(
+            "             -> grunnen mates tilbake i neste hypotese (bundet av max_attempts)"
+        )
+    return lines
+
+
 def main(argv: list[str] | None = None) -> int:  # pragma: no cover - console trace
     """Run the simulation against the energi bundle in a throwaway temp dir and print an honest,
     readable trace. Invoke: ``uv run python -m portfolio_optimiser.simulation``."""
@@ -299,6 +349,8 @@ def main(argv: list[str] | None = None) -> int:  # pragma: no cover - console tr
     print("=" * 78)
 
     print("\nRUN A  (fresh wiki — no prior verdicts)")
+    for line in _refinement_lines(result.run_a):
+        print(line)
     print(f"  validator : {_outcome_line(result.run_a)}")
     print(f"  checker   : VERDICT={result.run_a.checker_verdict.upper()}")
     print(f"  persona   : {result.run_a.verdict.decision}  ->  {result.run_a.verdict.rationale}")
@@ -310,6 +362,8 @@ def main(argv: list[str] | None = None) -> int:  # pragma: no cover - console tr
     print(f"  wrote     : {result.promoted_path.name}  (linked into index.md, neutral label)")
 
     print("\nRUN B  (re-seeded wiki — reads the promoted verdict)")
+    for line in _refinement_lines(result.run_b):
+        print(line)
     print(f"  validator : {_outcome_line(result.run_b)}")
     print(
         f"  prompt has marker '{result.marker}': {result.marker_in_run_b_prompt}  (expected True)"