The context sets, the packaged knowledge bases and the example bundles are
replaced by one fictitious example set about IT operations in an invented
organisation: three context sets (serverrom-2027, driftsavtale-2027 and the
two-base drift-og-avtale-2027), two synthetic knowledge bases under
src/portfolio_optimiser/data/kunnskapsbaser and two example bundles under
src/portfolio_optimiser/data/bundles. Numbers, codes and structural values in
tests and fixtures are kept; names, ids and wording change. Dated measurement
documents that only recorded runs on the replaced material are deleted.
Gate figures measured on the new set are not comparable with earlier ones.
The exclusion gate from the previous commit is green: 0 tracked files hit
outside the shared/ subtree.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
generate_via_llm consumed each validator Rejection internally (`last`), fed it into the
next attempt's prompt, and dropped it. So Step 5 was real but unobservable: a caller could
see THAT a proposal validated, never that it validated on attempt 2 after the deterministic
validator falsified attempt 1. It was the one step of the eight with no output to show.
The seam is a typed return value -- GenerationResult(outcome, refinements) -- rather than an
out-parameter or a callback: a returned value cannot be silently lost by a caller that forgets
to pass a collector, and mypy forces every call site to acknowledge it.
refinements carries ONLY rejections that were actually fed back. When the attempt budget runs
out the final rejection IS outcome; counting it here would be double-counting, and the bounded
control test goes red on the collect-everything implementation that gets this wrong.
The loop's bound is untouched: max_attempts and meter.tick_round stand, and `last` still drives
the prompt alone, so prompt growth is unchanged. run.py accumulates across _evaluate calls, so
_evaluate_mandate is untouched; RunResult.refinements defaults (the coverage precedent) and is
concatenated across approaches rather than keyed per approach -- stated as an honesty limit.
The simulation now shows it: the scripted proposer overclaims 250000, which the validator
falsifies against P90 = 90000, and the corrected 30000 validates. Only the overclaim is
scripted -- the rejection is computed. scripted_factory takes a per-role reply selector so this
needs no second scripted client body.
README records the two accuracy changes only (Step 5 is now inspectable; the simulation trace
shows the correction). The level-2 publishing claim stays deferred until after the demo (O4).
Load-bearing MEASURED against the full suite with a control, four mutations all red:
detach the returned history (4 tests) - collect-everything (control only) - detach the run
wiring (2 tests) - revert the simulation's proposer to a constant (the demo-protection test).
Control: 759 passed / 4 skipped; ruff, format and mypy clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CcWFcREUi6YPjEpN3ACDP
Closes gap #5 (maalbilde §5/§7): generate_via_llm's outer max_attempts loop
built the prompt ONCE and re-sent it identically — a BLIND retry. The validator's
per-attempt Rejection.reason was captured in `last` but never reached the next
prompt, so the proposer re-answered the same question with no knowledge of WHY
the prior candidate failed. Step 5 routes that reason into the next attempt.
- generate.py: _build_messages() gains prior_rejection; when set it appends a
revision block carrying ONLY the falsification reason verbatim (never the
rejected proposal JSON). None -> the byte-identical base prompt, so attempt 1
is unchanged. generate_via_llm() rebuilds messages inside the outer loop with
prior_rejection=`last` (None on attempt 1); _fetch_parsed() takes messages as
an explicit parameter. `last` is overwritten each round -> only the most-recent
falsification ("forrige"), never an accumulated history. Bound unchanged:
max_attempts + meter.tick_round (no new loop; §6 — "improve until good enough"
without a ceiling stays impossible).
- Scope honesty: the only per-attempt falsifier here is the validator. The
checker is a run-level, one-shot signal (run.py, before generation); seeding
generation with the checker critique is separately scoped and NOT done here.
The boundary is written into the generate_via_llm docstring + README + CLAUDE.
Load-bearing (maalbilde §7): tests/test_step5_refine_loadbearing.py is a PAIR —
the positive test keys the proposer's flip on the validator REASON PAYLOAD (the
rejected claim value, derived from validate_proposal(bad).reason so test and SUT
share one source of truth), and asserts the reason reached attempt 2's prompt
VERBATIM (the green-but-dead guard). It goes RED on detach (build messages once):
the flip token never arrives, so the outcome never flips AND the verbatim
assertion fails — proven double-red. The bounded control proves a never-fixed
proposer exhausts exactly max_attempts and returns a Rejection.
Adversarial Plan agent hardened the design pre-implementation (flip on payload
not wrapper/call-count; derive flip-key from the validator reason; drive through
generate_via_llm directly; docstring honesty). Suite 136->138 passed, 4 skipped;
mypy + ruff check clean. New test ruff-formatted; pre-existing ruff-format drift
(budget/verdicts/test_contracts) left untouched for a surgical diff.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MHR8iKxJRxDiDfNw8HZmWE
