feat(step5): the falsification that informed the next hypothesis now leaves the loop

generate_via_llm consumed each validator Rejection internally (`last`), fed it into the
next attempt's prompt, and dropped it. So Step 5 was real but unobservable: a caller could
see THAT a proposal validated, never that it validated on attempt 2 after the deterministic
validator falsified attempt 1. It was the one step of the eight with no output to show.

The seam is a typed return value -- GenerationResult(outcome, refinements) -- rather than an
out-parameter or a callback: a returned value cannot be silently lost by a caller that forgets
to pass a collector, and mypy forces every call site to acknowledge it.

refinements carries ONLY rejections that were actually fed back. When the attempt budget runs
out the final rejection IS outcome; counting it here would be double-counting, and the bounded
control test goes red on the collect-everything implementation that gets this wrong.

The loop's bound is untouched: max_attempts and meter.tick_round stand, and `last` still drives
the prompt alone, so prompt growth is unchanged. run.py accumulates across _evaluate calls, so
_evaluate_mandate is untouched; RunResult.refinements defaults (the coverage precedent) and is
concatenated across approaches rather than keyed per approach -- stated as an honesty limit.

The simulation now shows it: the scripted proposer overclaims 250000, which the validator
falsifies against P90 = 90000, and the corrected 30000 validates. Only the overclaim is
scripted -- the rejection is computed. scripted_factory takes a per-role reply selector so this
needs no second scripted client body.

README records the two accuracy changes only (Step 5 is now inspectable; the simulation trace
shows the correction). The level-2 publishing claim stays deferred until after the demo (O4).

Load-bearing MEASURED against the full suite with a control, four mutations all red:
detach the returned history (4 tests) - collect-everything (control only) - detach the run
wiring (2 tests) - revert the simulation's proposer to a constant (the demo-protection test).
Control: 759 passed / 4 skipped; ruff, format and mypy clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CcWFcREUi6YPjEpN3ACDP
This commit is contained in:
Kjell Tore Guttormsen 2026-08-06 15:12:06 +02:00
commit d6f3359fae
11 changed files with 381 additions and 26 deletions

View file

@ -57,8 +57,10 @@ demonstrably informed by the first:
uv run python -m portfolio_optimiser.simulation
```
The trace ends with the approved verdict's marker present in Run B's prompt and absent from Run A's
— knowledge crossing runs purely through the file-backed wiki (promote → re-seed → fold).
Each run shows the refinement step: the proposer's first claim is falsified by the deterministic
validator, and the corrected claim validates. The trace then ends with the approved verdict's marker
present in Run B's prompt and absent from Run A's — knowledge crossing runs purely through the
file-backed wiki (promote → re-seed → fold).
**3 — Run the loop over a knowledge base, with answers you supply.** Write the stand-in replies,
then point the CLI at the bundle:
@ -262,7 +264,9 @@ One run, one project, eight steps — with the learning loop closing across runs
is anchored to the project's declared cost baseline, so a proposal cannot invent the cost
lines it claims to save against.
5. **Refine** — a rejected attempt retries *informed* by the rejection reason, under hard
attempt and token caps. Unbounded loops are forbidden everywhere.
attempt and token caps. Unbounded loops are forbidden everywhere. The falsifications that
informed a later attempt are surfaced on the result (`RunResult.refinements`), so what the
run corrected in response to is inspectable, not just what it ended up with.
6. **Propose or discard** — a validated proposal with risk percentiles, or a typed rejection.
7. **Expert feedback** — days later, an expert drops a verdict file in an inbox folder; a
later run picks it up. Fully resumable; no live session assumed.