open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(step5): the falsification that informed the next hypothesis now leaves the loop

Browse source 
generate_via_llm consumed each validator Rejection internally (`last`), fed it into the
next attempt's prompt, and dropped it. So Step 5 was real but unobservable: a caller could
see THAT a proposal validated, never that it validated on attempt 2 after the deterministic
validator falsified attempt 1. It was the one step of the eight with no output to show.

The seam is a typed return value -- GenerationResult(outcome, refinements) -- rather than an
out-parameter or a callback: a returned value cannot be silently lost by a caller that forgets
to pass a collector, and mypy forces every call site to acknowledge it.

refinements carries ONLY rejections that were actually fed back. When the attempt budget runs
out the final rejection IS outcome; counting it here would be double-counting, and the bounded
control test goes red on the collect-everything implementation that gets this wrong.

The loop's bound is untouched: max_attempts and meter.tick_round stand, and `last` still drives
the prompt alone, so prompt growth is unchanged. run.py accumulates across _evaluate calls, so
_evaluate_mandate is untouched; RunResult.refinements defaults (the coverage precedent) and is
concatenated across approaches rather than keyed per approach -- stated as an honesty limit.

The simulation now shows it: the scripted proposer overclaims 250000, which the validator
falsifies against P90 = 90000, and the corrected 30000 validates. Only the overclaim is
scripted -- the rejection is computed. scripted_factory takes a per-role reply selector so this
needs no second scripted client body.

README records the two accuracy changes only (Step 5 is now inspectable; the simulation trace
shows the correction). The level-2 publishing claim stays deferred until after the demo (O4).

Load-bearing MEASURED against the full suite with a control, four mutations all red:
detach the returned history (4 tests) - collect-everything (control only) - detach the run
wiring (2 tests) - revert the simulation's proposer to a constant (the demo-protection test).
Control: 759 passed / 4 skipped; ruff, format and mypy clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CcWFcREUi6YPjEpN3ACDP
This commit is contained in:
Kjell Tore Guttormsen 2026-08-06 15:12:06 +02:00
commit d6f3359fae
11 changed files with 381 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

15
CHANGELOG.md
View file

@ -5,6 +5,21 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Added
- Step 5 is now observable: `generate_via_llm` returns a `GenerationResult` carrying the validator
falsifications that informed a later attempt, surfaced on `RunResult.refinements`. The offline
simulation exercises it — the scripted proposer overclaims, the deterministic validator falsifies
the number, and the refined proposal validates.
### Changed
- **Breaking (library API):** `generate_via_llm` returns `GenerationResult` instead of
`ValidatedProposal | Rejection`; read `.outcome` for the previous value. The refinement loop's
bound is unchanged (`max_attempts` + token meter).
- `simulation.scripted_factory` accepts a per-role reply *selector* over `(prompt, role)` as well as
a constant reply, so a scripted role can answer differently on a later attempt.
## [0.1.0] - 2026-08-06
First tagged release. There is no prior release, so the entries below describe what this version