open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(outbox): every evaluated approach becomes something an expert can judge

Browse source 
A run commissioned to evaluate three approaches wrote ONE proposal artefact, so
only the approach it selected could ever receive a verdict. The other two were
evaluated, reported in the settlement, and then taught the learning loop nothing.

The defect class is a key collapse, and it had two halves — fixing either alone
leaves it intact:

* the WRITER wrote one pair per run, so the non-selected approaches never existed
  on disk;
* the READER (hitl._read_outbox_proposals) joins proposal to outcome on the
  run_id FIELD read from file CONTENT, never the filename. Three files sharing
  one run_id collapse onto one dict key, last write wins — so widening only the
  filename would have produced three artefacts and still one pending row. This is
  the S3.2 collision class: two rows under one key silently become one.

Artefacts are now keyed {run_id}-{approach_id}-*.json AND carry approach_id in the
payload; the join key is (run_id, approach_id). Two properties make them genuinely
judgeable rather than merely present:

* verdict_id is minted per approach (verdicts.verdict_key, the S3.2 content hash)
  — reusing the run's single id would let one delivered verdict clear all three
  from the queue;
* provenance.validator_decision follows ITS OWN approach — the run's stamp would
  report a rejected candidate as validated, and nothing downstream could correct it.

verdicts.verdict_key is public so a run can stamp the key a verdict WILL arrive
under without capturing a decision nobody has made; it delegates to _mint_id
rather than restating the hash (the (p) rule: one keying rule, one copy).

The per-approach set REPLACES the run-level pair rather than joining it — the
selected approach is already among them, and writing both would count it twice in
hitl pending. The selected one carries the run's final outcome, so the outbox can
never disagree with the RunResult; the others carry the validator's verdict, the
only falsifier that ran on them.

mandate.py is deliberately untouched: hanging a ValidatedProposal off a coverage
row would drag validator — and pulp — into a module kept to pydantic+stdlib for
D7 portability, so _evaluate_mandate returns the evaluated outcomes alongside.

Ran it, not just tested it: a real CLI run wrote six artefacts and hitl pending
listed three rows. It also showed the honest edge — three approaches that produce
an identical candidate share one content-hash key, so one verdict settles all
three. That is correct (they were one candidate), and it is now documented.

Load-bearing MEASURED (tests/test_a5_per_approach_artifacts_loadbearing.py) against
the whole 750-test suite, five mutations all red: detach the per-approach writer ·
drop approach_id from the join key · reuse the run's verdict id · reuse the run's
provenance stamp · widen the filename but not the payload. Control: on a full
detach exactly the 5 new tests fail and 745 pre-existing ones stay green — the
no-mandate path is inert, and writes neither the filename segment nor the field.

Docs: bestille-en-kjoring.md (what the commissioner gets) + ekspert-svar.md (what
the expert's queue looks like, and that "rejected" is the validator's verdict on
the numbers, never a professional judgement of the idea).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtRd8y1PDPGwkrRXFhubqr
This commit is contained in:
Kjell Tore Guttormsen 2026-08-05 21:12:09 +02:00
commit 455d93d33e
7 changed files with 443 additions and 33 deletions
Download patch file Download diff file Expand all files Collapse all files

54
src/portfolio_optimiser/hitl.py
View file

@ -50,43 +50,59 @@ _REQUIRED_FEATURE_KEYS = {"affected_codes", "measure_type", "claimed_saving_nok"
@dataclass(frozen=True)
class PendingProposal:
"""One outbox proposal still awaiting an expert verdict. ``codes``/``measure`` carry the routing
keys (Step 3); ``verdict_id`` is the id-join key against the inbox."""
keys (Step 3); ``verdict_id`` is the id-join key against the inbox. ``approach_id`` names the
commissioned approach the artefact belongs to (A5), and is ``""`` for an artefact written
without a mandate — a run nobody commissioned has no approach to name."""
run_id: str
verdict_id: str
outcome_type: str
measure: str
codes: frozenset[str]
approach_id: str = ""
def _join_key(data: dict[str, Any]) -> tuple[str, str]:
"""The key one artefact is filed under: ``(run_id, approach_id)``, read from file CONTENT.
``run_id`` alone was the key until A5 gave a run several judgeable approaches. Once it does, a
``run_id``-only key collapses every approach of one run onto a single dict entry (last write
wins) and the expert's queue silently reports one candidate where three were evaluated — the
S3.2 key-collision class. An artefact with no ``approach_id`` keys on ``""``, which is exactly
the pre-A5 behaviour for pre-A5 files."""
approach_id = data.get("approach_id")
return str(data["run_id"]), str(approach_id) if isinstance(approach_id, str) else ""
def _read_outbox_proposals(outbox_dir: str) -> list[PendingProposal]:
"""Read the outbox, joining ``{run_id}-proposal.json`` and ``{run_id}-outcome.json`` on the
``run_id`` FIELD read from file content (never the filename). TOLERANT (RAW layer, contrast
``okf.load_ir_projection``'s fail-fast): a missing dir yields ``[]``; unparseable files, and
orphans (a proposal without its outcome or vice-versa), are SKIPPED, never raised — a live run
writes the pair non-atomically, so half-written state is realistic."""
"""Read the outbox, joining ``{run_id}[-{approach_id}]-proposal.json`` and its ``-outcome.json``
on the ``run_id``/``approach_id`` FIELDS read from file content (never the filename). TOLERANT
(RAW layer, contrast ``okf.load_ir_projection``'s fail-fast): a missing dir yields ``[]``;
unparseable files, and orphans (a proposal without its outcome or vice-versa), are SKIPPED,
never raised — a live run writes the pair non-atomically, so half-written state is realistic."""
directory = Path(outbox_dir)
if not directory.is_dir():
return []
proposals: dict[str, dict[str, Any]] = {}
proposals: dict[tuple[str, str], dict[str, Any]] = {}
for file in sorted(directory.glob("*-proposal.json")):
data = _load_json_dict(file)
if data is None or "run_id" not in data or not isinstance(data.get("proposal"), dict):
continue
proposals[str(data["run_id"])] = data
proposals[_join_key(data)] = data
outcomes: dict[str, dict[str, Any]] = {}
outcomes: dict[tuple[str, str], dict[str, Any]] = {}
for file in sorted(directory.glob("*-outcome.json")):
data = _load_json_dict(file)
if data is None or "run_id" not in data:
continue
outcomes[str(data["run_id"])] = data
outcomes[_join_key(data)] = data
result: list[PendingProposal] = []
for run_id in proposals.keys() & outcomes.keys(): # inner join — orphans on either side dropped
proposal = proposals[run_id]["proposal"]
outcome = outcomes[run_id]
for key in proposals.keys() & outcomes.keys(): # inner join — orphans on either side dropped
run_id, approach_id = key
proposal = proposals[key]["proposal"]
outcome = outcomes[key]
verdict_id = outcome.get("verdict_id")
outcome_type = outcome.get("outcome_type")
if not isinstance(verdict_id, str) or not isinstance(outcome_type, str):
@ -102,6 +118,7 @@ def _read_outbox_proposals(outbox_dir: str) -> list[PendingProposal]:
outcome_type=outcome_type,
measure=str(proposal.get("measure", "")),
codes=codes,
approach_id=approach_id,
)
)
return result
@ -141,10 +158,11 @@ def _inbox_verdict_ids(verdict_dir: str) -> set[str]:
def pending(outbox_dir: str, verdict_dir: str) -> list[PendingProposal]:
"""The pending registry: outbox proposals whose ``verdict_id`` is NOT yet in the inbox id-set,
sorted deterministically by ``(run_id, verdict_id)``."""
sorted deterministically by ``(run_id, approach_id, verdict_id)`` — one row per evaluated
approach, since each is judged on its own key."""
judged = _inbox_verdict_ids(verdict_dir)
unjudged = [p for p in _read_outbox_proposals(outbox_dir) if p.verdict_id not in judged]
return sorted(unjudged, key=lambda p: (p.run_id, p.verdict_id))
return sorted(unjudged, key=lambda p: (p.run_id, p.approach_id, p.verdict_id))
# --- Routing config: self-contained dimension→expert table (fail-fast) ----------------------------
@ -286,7 +304,11 @@ def main(argv: list[str] | None = None) -> int:
if args.command == "pending":
for proposal in pending(args.outbox_dir, args.verdict_dir):
print(f"{proposal.run_id} {proposal.verdict_id} {proposal.outcome_type}")
# The approach is appended only when there is one: an artefact written without a
# mandate has no approach, and printing an empty column would suggest a missing value
# rather than a run nobody commissioned by approach.
approach = f" [{proposal.approach_id}]" if proposal.approach_id else ""
print(f"{proposal.run_id} {proposal.verdict_id} {proposal.outcome_type}{approach}")
return 0
try: