feat(fase4): route the validator falsification into the next hypothesis (informed refinement)
Closes gap #5 (maalbilde §5/§7): generate_via_llm's outer max_attempts loop built the prompt ONCE and re-sent it identically — a BLIND retry. The validator's per-attempt Rejection.reason was captured in `last` but never reached the next prompt, so the proposer re-answered the same question with no knowledge of WHY the prior candidate failed. Step 5 routes that reason into the next attempt. - generate.py: _build_messages() gains prior_rejection; when set it appends a revision block carrying ONLY the falsification reason verbatim (never the rejected proposal JSON). None -> the byte-identical base prompt, so attempt 1 is unchanged. generate_via_llm() rebuilds messages inside the outer loop with prior_rejection=`last` (None on attempt 1); _fetch_parsed() takes messages as an explicit parameter. `last` is overwritten each round -> only the most-recent falsification ("forrige"), never an accumulated history. Bound unchanged: max_attempts + meter.tick_round (no new loop; §6 — "improve until good enough" without a ceiling stays impossible). - Scope honesty: the only per-attempt falsifier here is the validator. The checker is a run-level, one-shot signal (run.py, before generation); seeding generation with the checker critique is separately scoped and NOT done here. The boundary is written into the generate_via_llm docstring + README + CLAUDE. Load-bearing (maalbilde §7): tests/test_step5_refine_loadbearing.py is a PAIR — the positive test keys the proposer's flip on the validator REASON PAYLOAD (the rejected claim value, derived from validate_proposal(bad).reason so test and SUT share one source of truth), and asserts the reason reached attempt 2's prompt VERBATIM (the green-but-dead guard). It goes RED on detach (build messages once): the flip token never arrives, so the outcome never flips AND the verbatim assertion fails — proven double-red. The bounded control proves a never-fixed proposer exhausts exactly max_attempts and returns a Rejection. Adversarial Plan agent hardened the design pre-implementation (flip on payload not wrapper/call-count; derive flip-key from the validator reason; drive through generate_via_llm directly; docstring honesty). Suite 136->138 passed, 4 skipped; mypy + ruff check clean. New test ruff-formatted; pre-existing ruff-format drift (budget/verdicts/test_contracts) left untouched for a surgical diff. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MHR8iKxJRxDiDfNw8HZmWE
This commit is contained in:
parent
4ec778c855
commit
f7c81b45ec
4 changed files with 188 additions and 10 deletions
|
|
@ -42,7 +42,15 @@ class GenerationError(RuntimeError):
|
|||
"""No parseable proposal could be produced within the attempt budget."""
|
||||
|
||||
|
||||
def _build_messages(project: Project, context: str) -> list[Message]:
|
||||
def _build_messages(
|
||||
project: Project, context: str, prior_rejection: Rejection | None = None
|
||||
) -> list[Message]:
|
||||
"""Build the hypothesis prompt. When ``prior_rejection`` is set (Step 5, målbilde §5/§7),
|
||||
append a revision block carrying ONLY the falsification *reason* verbatim — never the prior
|
||||
proposal JSON (minimal honest payload: the model must address the falsification, not parrot
|
||||
the rejected candidate back). ``None`` -> the byte-identical base prompt, so attempt 1 is
|
||||
unchanged. The reason carries only the rejected claim/feasible figures, which deliberately
|
||||
do not collide with other load-bearing prompt markers."""
|
||||
prompt = (
|
||||
"Propose ONE concrete cost-saving measure for this project.\n"
|
||||
f"Project: {project.id} - {project.name}\n"
|
||||
|
|
@ -51,6 +59,12 @@ def _build_messages(project: Project, context: str) -> list[Message]:
|
|||
"measure, affected_items (list of {code, quantity, unit_cost}), claimed_saving_nok, "
|
||||
"and optional assumptions."
|
||||
)
|
||||
if prior_rejection is not None:
|
||||
prompt += (
|
||||
"\n\nYour previous proposal was REJECTED by the deterministic validator.\n"
|
||||
f"Reason: {prior_rejection.reason}\n"
|
||||
"Produce a REVISED SavingsProposal that resolves this."
|
||||
)
|
||||
return [Message(role="user", contents=[prompt])]
|
||||
|
||||
|
||||
|
|
@ -97,13 +111,23 @@ async def generate_via_llm(
|
|||
*,
|
||||
max_attempts: int = 3,
|
||||
) -> ValidatedProposal | Rejection:
|
||||
"""Async LLM path: non-streaming chat -> parse -> validate, with bounded retries and the
|
||||
meter checked in this loop. A malformed/text-leaked reply is retried (never silently
|
||||
accepted); a validation rejection is retried; returns ``ValidatedProposal | Rejection``;
|
||||
raises ``BudgetExceeded`` when the meter cap is crossed."""
|
||||
messages = _build_messages(project, context)
|
||||
"""Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds,
|
||||
the meter checked in this loop:
|
||||
|
||||
async def _fetch_parsed() -> SavingsProposal:
|
||||
* malformed/text-leaked reply -> BLIND parse-retry (inner loop): re-fetch until the reply
|
||||
parses; never silently accepted.
|
||||
* validator rejection -> INFORMED refinement (outer ``max_attempts`` loop; Step 5,
|
||||
målbilde §5/§7): the previous attempt's ``Rejection.reason`` is fed into the next
|
||||
attempt's prompt (``_build_messages(prior_rejection=...)``) so the proposer can correct
|
||||
rather than re-answer blindly. Bounded by ``max_attempts`` + the meter (no new loop; §6).
|
||||
|
||||
The only per-attempt falsifier here is the deterministic validator (the numbers). The
|
||||
checker is a run-level, one-shot signal (run.py, before generation); seeding generation
|
||||
with the checker critique is separately scoped and NOT done here. Returns
|
||||
``ValidatedProposal | Rejection``; never a malformed proposal; raises ``BudgetExceeded``
|
||||
when the meter cap is crossed."""
|
||||
|
||||
async def _fetch_parsed(messages: list[Message]) -> SavingsProposal:
|
||||
# Parse-robust: a malformed/text-leaked reply is retried; the meter caps total work.
|
||||
while True:
|
||||
meter.tick_round() # between-attempt bound (BudgetExceeded over cap)
|
||||
|
|
@ -116,7 +140,12 @@ async def generate_via_llm(
|
|||
|
||||
last: Rejection | None = None
|
||||
for _ in range(max_attempts):
|
||||
candidate = await _fetch_parsed()
|
||||
# Informed refinement: feed the PREVIOUS attempt's validator rejection into this
|
||||
# attempt's prompt. ``last`` is None on attempt 1 -> the unchanged base prompt; it is
|
||||
# overwritten each round -> only the most-recent falsification ("forrige"), never an
|
||||
# accumulated history (bounded prompt growth).
|
||||
messages = _build_messages(project, context, prior_rejection=last)
|
||||
candidate = await _fetch_parsed(messages)
|
||||
result = validate_proposal(candidate)
|
||||
if isinstance(result, ValidatedProposal):
|
||||
return result
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue