feat(run): every commissioned approach is evaluated, and every one is reported

Carrying an approach into the prompt only half-answers krav 1. The expert asked
for their approaches to be CONCRETELY EVALUATED, which means each must reach a
verdict and each verdict must be visible. `run_project(mandate=...)` evaluates
every commissioned approach in turn — the run's own proposal last, when allowed —
each under the SAME meter. No new loop: the caps already in force are the bound.

`RunResult.coverage` is the settlement, one row per approach: validated (with the
figure), rejected (with the validator's reason verbatim), or not_evaluated (with
why). `not_evaluated` is the row that earns the type its keep — an approach the
run never reached must be reported as unreached, because an omitted row is
indistinguishable from an approach nobody ordered. That silence is the defect
class krav 1 is asking us to remove.

Budget exhaustion mid-list is reported, not swallowed. But if the FIRST approach
exhausts it there is nothing honest to return, so BudgetExceeded propagates
exactly as before — a run that produced nothing must still fail loudly.

RunResult stays single-outcome (portfolio aggregation, outbox artefacts and HITL
keying all rest on that). The choice is deterministic: highest validated saving,
ties by mandate order — never whichever ran last.

Load-bearing MEASURED against the whole 702-test suite. FIVE mutations red:
evaluate only the first approach (5 red) · drop the rejected rows (3) · ignore
allow_own_proposals (1) · select produced[-1] (1) · select produced[0] (1).

The sixth measurement is why this commit exists in this shape: the ordering
mutation FIRST STAYED GREEN. The test had placed the bigger approach last, where
"highest saving" and "whichever ran last" give the same answer, so an
order-dependent implementation passed it. A scenario that cannot separate two
implementations proves nothing about either — the test now pins BOTH orderings,
and each mutation direction fails one of them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULCqjLF61rehj5cZmdUoR3
This commit is contained in:
Kjell Tore Guttormsen 2026-08-05 16:05:48 +02:00
commit 61bf5b78ea
3 changed files with 326 additions and 5 deletions

View file

@ -29,7 +29,9 @@ Two refusals are load-bearing, both at construction time:
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Literal
from pydantic import BaseModel, Field, model_validator
@ -86,6 +88,26 @@ class Mandate(BaseModel):
return self
@dataclass(frozen=True)
class ApproachOutcome:
"""What became of ONE commissioned approach — one row of the run's coverage report.
``not_evaluated`` is the row that earns this type its keep: an approach the run never got to
(budget exhausted, pass stopped) must be reported as *not evaluated*, never omitted. An omitted
row is indistinguishable from an approach nobody ordered, which is exactly the silence krav 1
is asking us to remove.
``detail`` carries the validator's reason on a rejection, or why an approach went unevaluated;
``saving_nok`` is set only for a validated row (the claimed figure the validator admitted).
"""
id: str
label: str
status: Literal["validated", "rejected", "not_evaluated"]
detail: str = ""
saving_nok: float | None = None
def load_mandate(path: str | Path) -> Mandate:
"""Fail-fast standalone loader for a run mandate (mirrors ``dimension.load_dimension``).