Kjell Tore Guttormsen f7c81b45ec feat(fase4): route the validator falsification into the next hypothesis (informed refinement)

Closes gap #5 (maalbilde §5/§7): generate_via_llm's outer max_attempts loop
built the prompt ONCE and re-sent it identically — a BLIND retry. The validator's
per-attempt Rejection.reason was captured in `last` but never reached the next
prompt, so the proposer re-answered the same question with no knowledge of WHY
the prior candidate failed. Step 5 routes that reason into the next attempt.

- generate.py: _build_messages() gains prior_rejection; when set it appends a
  revision block carrying ONLY the falsification reason verbatim (never the
  rejected proposal JSON). None -> the byte-identical base prompt, so attempt 1
  is unchanged. generate_via_llm() rebuilds messages inside the outer loop with
  prior_rejection=`last` (None on attempt 1); _fetch_parsed() takes messages as
  an explicit parameter. `last` is overwritten each round -> only the most-recent
  falsification ("forrige"), never an accumulated history. Bound unchanged:
  max_attempts + meter.tick_round (no new loop; §6 — "improve until good enough"
  without a ceiling stays impossible).
- Scope honesty: the only per-attempt falsifier here is the validator. The
  checker is a run-level, one-shot signal (run.py, before generation); seeding
  generation with the checker critique is separately scoped and NOT done here.
  The boundary is written into the generate_via_llm docstring + README + CLAUDE.

Load-bearing (maalbilde §7): tests/test_step5_refine_loadbearing.py is a PAIR —
the positive test keys the proposer's flip on the validator REASON PAYLOAD (the
rejected claim value, derived from validate_proposal(bad).reason so test and SUT
share one source of truth), and asserts the reason reached attempt 2's prompt
VERBATIM (the green-but-dead guard). It goes RED on detach (build messages once):
the flip token never arrives, so the outcome never flips AND the verbatim
assertion fails — proven double-red. The bounded control proves a never-fixed
proposer exhausts exactly max_attempts and returns a Rejection.

Adversarial Plan agent hardened the design pre-implementation (flip on payload
not wrapper/call-count; derive flip-key from the validator reason; drive through
generate_via_llm directly; docstring honesty). Suite 136->138 passed, 4 skipped;
mypy + ruff check clean. New test ruff-formatted; pre-existing ruff-format drift
(budget/verdicts/test_contracts) left untouched for a surgical diff.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MHR8iKxJRxDiDfNw8HZmWE

2026-06-30 09:08:02 +02:00

5.4 KiB

Raw Blame History

portfolio-optimiser

Generic, open framework on Microsoft Agent Framework (MAF) for finding cost-savings / efficiency proposals within each project of a portfolio of independent projects. Multiple agents collaborate to generate candidate proposals; a mandatory deterministic validator (solver + Monte Carlo) decides the numbers; domain experts review via human-in-the-loop, and the system learns from their verdicts.

Status: Early development. The deterministic backbone is solid; the agentic learning loop is being wired one load-bearing seam at a time — Step 1 (OKF-navigated context → hypothesis) is wired, with the verdict layer kept out of the read-context (see below). Not yet end-to-end usable.

Disclaimer — technical framework only. This project is a technical framework. Organizations that deploy it are themselves responsible for ensuring a valid processing purpose and for any required assessments (DPIA, risk/ROS, security reviews, etc.). The framework ships technical affordances (local-only mode, provenance/audit logging, no silent data egress) to enable compliant use, but makes no compliance guarantees.

Design philosophy

The result will never fit any single customer 100%. The goal is a ~90% genuinely generic core plus clear extension points, so competent people can configure the last mile per customer. We deliberately do not chase the final 10%.

Agentic loop — wiring status

The mandatory deterministic backbone (validator + budget meter + provenance) is solid and load-bearing. The agentic learning loop (see the target picture §11) is wired one seam at a time:

Step 1 — OKF context → hypothesis (wired). run_project(..., bundle_dir=...) wires the first agentic seam in two halves:
- Context by navigation, not stuffing. The agent read-context is built by navigating the project's OKF bundle (index.md + frontmatter + cross-links, progressive disclosure) — never keyword chunk-stuffing (target picture §2/§4).
- Verdict layer gated out of context. The type: verdict layer is excluded from that context; the candidate's prior expert verdicts reach the hypothesis prompt only through the gated ExpeL fold (folded in before generation), so a prior verdict provably — and exclusively — influences the next hypothesis.
- Two load-bearing tests fail when the seam is detached: the realization signal must reach the prompt via the fold (tests/test_step1_expel_loadbearing.py), and must never leak via context (tests/test_okf.py::test_bundle_context_excludes_verdict_layer).
Steps 3/4 — checker gates the reasoning (wired). Two falsifiers now act on the same candidate: the deterministic validator gates the numbers (blocking, as before), and the maker-checker's checker gates the reasoning (target picture §2/§6). The checker ends its turn with a VERDICT: APPROVE / VERDICT: REJECT — <reason> line; an explicit reject blocks an otherwise-validated proposal (run_project surfaces both debate participants via output_from=agents and overrides the outcome to a checker-sourced Rejection). The gate is opt-in-reject (fail-open on a missing marker), and provenance.validator_decision stays honest — it reflects the validator only, never the checker. Load-bearing: tests/test_checker_gate_loadbearing.py goes red on either detach (revert output_from, or drop the override).
Step 5 — informed refinement (wired). The proposer's bounded retry is no longer blind: the validator's previous Rejection.reason is fed into the next attempt's prompt (generate.py generate_via_llm → _build_messages(prior_rejection=...)), so the model corrects against the falsification instead of re-answering identically (target picture §5/§7). It carries only the most-recent reason (never an accumulated history, never the rejected proposal JSON) and runs under the existing max_attempts + token-meter cap — no new loop, so "improve until good enough" without a ceiling stays impossible. The only per-attempt falsifier here is the validator; seeding generation with the checker's critique is a run-level, separately-scoped concern and is deliberately not done here. Load-bearing: tests/test_step5_refine_loadbearing.py goes red when the reason is detached from the prompt (the outcome never flips and the verbatim-reason assertion fails), with a bounded control proving the loop still stops at max_attempts.
Steps 7–8 (async file feedback, gated wiki promotion) — not yet wired.

Docs

docs/plan/2026-06-26-maalbilde-agentic-loop.md — target picture: the agentic cost-saving loop + OKF knowledge architecture (north star).
docs/research/2026-06-23-prior-art-platform.md — prior-art & platform research (incl. implementation register §15).
docs/plan/2026-06-23-incremental-plan.md — incremental delivery plan (deterministic backbone).
shared/ — framework-neutral shared core (concept + example OKF knowledge bundles), reused unchanged by both reference implementations.

Stack

Python ≥3.10 · MAF (agent-framework) · uv. Backend profiles: Azure/Foundry (full) + local (fallback).

Develop

uv sync
uv run pytest
uv run ruff check .

5.4 KiB Raw Blame History Unescape Escape