feat(step5): the falsification that informed the next hypothesis now leaves the loop
generate_via_llm consumed each validator Rejection internally (`last`), fed it into the next attempt's prompt, and dropped it. So Step 5 was real but unobservable: a caller could see THAT a proposal validated, never that it validated on attempt 2 after the deterministic validator falsified attempt 1. It was the one step of the eight with no output to show. The seam is a typed return value -- GenerationResult(outcome, refinements) -- rather than an out-parameter or a callback: a returned value cannot be silently lost by a caller that forgets to pass a collector, and mypy forces every call site to acknowledge it. refinements carries ONLY rejections that were actually fed back. When the attempt budget runs out the final rejection IS outcome; counting it here would be double-counting, and the bounded control test goes red on the collect-everything implementation that gets this wrong. The loop's bound is untouched: max_attempts and meter.tick_round stand, and `last` still drives the prompt alone, so prompt growth is unchanged. run.py accumulates across _evaluate calls, so _evaluate_mandate is untouched; RunResult.refinements defaults (the coverage precedent) and is concatenated across approaches rather than keyed per approach -- stated as an honesty limit. The simulation now shows it: the scripted proposer overclaims 250000, which the validator falsifies against P90 = 90000, and the corrected 30000 validates. Only the overclaim is scripted -- the rejection is computed. scripted_factory takes a per-role reply selector so this needs no second scripted client body. README records the two accuracy changes only (Step 5 is now inspectable; the simulation trace shows the correction). The level-2 publishing claim stays deferred until after the demo (O4). Load-bearing MEASURED against the full suite with a control, four mutations all red: detach the returned history (4 tests) - collect-everything (control only) - detach the run wiring (2 tests) - revert the simulation's proposer to a constant (the demo-protection test). Control: 759 passed / 4 skipped; ruff, format and mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CcWFcREUi6YPjEpN3ACDP
This commit is contained in:
parent
cd011c4ac7
commit
d6f3359fae
11 changed files with 381 additions and 26 deletions
|
|
@ -15,14 +15,15 @@ Two entry points, because the LLM call is async while ``validator.self_repair``
|
|||
attempts. Used for deterministic candidate sources.
|
||||
* ``generate_via_llm`` — the ASYNC LLM path: an async mirror of the same bounded retry that
|
||||
awaits the chat call (parse-retry inside the meter budget, then ``validate_proposal``).
|
||||
Returns ``ValidatedProposal | Rejection``; never a malformed proposal; raises
|
||||
``BudgetExceeded`` when the meter cap is crossed.
|
||||
Returns a ``GenerationResult`` (the outcome PLUS the falsifications that informed it); never a
|
||||
malformed proposal; raises ``BudgetExceeded`` when the meter cap is crossed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from agent_framework import BaseChatClient, Message
|
||||
from pydantic import ValidationError
|
||||
|
|
@ -43,6 +44,28 @@ class GenerationError(RuntimeError):
|
|||
"""No parseable proposal could be produced within the attempt budget."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GenerationResult:
|
||||
"""What one ``generate_via_llm`` call produced: the outcome, and the falsification history that
|
||||
informed it (Step 5, målbilde §5/§7).
|
||||
|
||||
A TYPED RETURN VALUE rather than an out-parameter or a callback, deliberately: the informed
|
||||
refinement loop already computed this history internally and then dropped it, so Step 5 was the
|
||||
one step of the eight with no observable output. A returned value cannot be silently lost by a
|
||||
caller that forgets to pass a collector, and it forces every call site to acknowledge the seam.
|
||||
|
||||
``refinements`` holds ONLY the rejections that were actually fed back into a later attempt's
|
||||
prompt — the honest reading of "informed refinement". When the attempt budget runs out, the
|
||||
final rejection IS ``outcome``: it informed nothing and is not repeated here. So the total
|
||||
number of validator falsifications this call produced is ``len(refinements)`` plus one when
|
||||
``outcome`` is itself a ``Rejection``. It is empty on the common single-attempt path, which is
|
||||
honest rather than merely convenient: nothing was falsified, so there is nothing to show.
|
||||
"""
|
||||
|
||||
outcome: ValidatedProposal | Rejection
|
||||
refinements: tuple[Rejection, ...] = field(default=())
|
||||
|
||||
|
||||
def _build_messages(
|
||||
project: Project,
|
||||
context: str,
|
||||
|
|
@ -140,7 +163,7 @@ async def generate_via_llm(
|
|||
max_attempts: int = 3,
|
||||
baseline: CostBaseline | None = None,
|
||||
approach: Approach | None = None,
|
||||
) -> ValidatedProposal | Rejection:
|
||||
) -> GenerationResult:
|
||||
"""Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds,
|
||||
the meter checked in this loop:
|
||||
|
||||
|
|
@ -165,8 +188,12 @@ async def generate_via_llm(
|
|||
``baseline`` (S4.0) is handed straight to ``validate_proposal``, so a fabricated cost line is
|
||||
falsified per ATTEMPT like any other rejection — and its reason feeds the next attempt's prompt
|
||||
through the SAME informed-refinement path (Step 5), which is why no new loop appears here.
|
||||
Returns
|
||||
``ValidatedProposal | Rejection``; never a malformed proposal; raises ``BudgetExceeded``
|
||||
|
||||
Returns a ``GenerationResult``: the ``ValidatedProposal | Rejection`` outcome plus every
|
||||
rejection that was fed back into a later attempt's prompt. Surfacing that history changes
|
||||
nothing about the loop's BOUND — ``max_attempts`` and ``meter.tick_round`` are exactly as
|
||||
before ("refine until good enough" without a cap stays forbidden, §6); it only stops the loop
|
||||
from discarding what it already knew. Never a malformed proposal; raises ``BudgetExceeded``
|
||||
when the meter cap is crossed."""
|
||||
|
||||
async def _fetch_parsed(messages: list[Message]) -> SavingsProposal:
|
||||
|
|
@ -181,16 +208,25 @@ async def generate_via_llm(
|
|||
continue
|
||||
|
||||
last: Rejection | None = None
|
||||
# The falsifications that were FED BACK, in attempt order. ``last`` still drives the PROMPT and
|
||||
# is still overwritten each round -- only the most-recent falsification reaches the model, so
|
||||
# prompt growth is unchanged. This list is a record for the CALLER, appended to only once a
|
||||
# rejection is about to inform a further attempt; it is never read back into a prompt.
|
||||
fed_back: list[Rejection] = []
|
||||
for _ in range(max_attempts):
|
||||
# Informed refinement: feed the PREVIOUS attempt's validator rejection into this
|
||||
# attempt's prompt. ``last`` is None on attempt 1 -> the unchanged base prompt; it is
|
||||
# overwritten each round -> only the most-recent falsification ("forrige"), never an
|
||||
# accumulated history (bounded prompt growth).
|
||||
if last is not None:
|
||||
fed_back.append(last)
|
||||
messages = _build_messages(project, context, prior_rejection=last, approach=approach)
|
||||
candidate = await _fetch_parsed(messages)
|
||||
result = validate_proposal(candidate, baseline=baseline)
|
||||
if isinstance(result, ValidatedProposal):
|
||||
return result
|
||||
return GenerationResult(outcome=result, refinements=tuple(fed_back))
|
||||
last = result
|
||||
assert last is not None # max_attempts >= 1, so at least one validation ran
|
||||
return last # validation never passed within the attempt budget -> typed Rejection
|
||||
# Validation never passed within the attempt budget -> typed Rejection. ``last`` is the outcome
|
||||
# and was never fed back, so it is deliberately absent from ``refinements``.
|
||||
return GenerationResult(outcome=last, refinements=tuple(fed_back))
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue