feat(run): every commissioned approach is evaluated, and every one is reported
Carrying an approach into the prompt only half-answers krav 1. The expert asked for their approaches to be CONCRETELY EVALUATED, which means each must reach a verdict and each verdict must be visible. `run_project(mandate=...)` evaluates every commissioned approach in turn — the run's own proposal last, when allowed — each under the SAME meter. No new loop: the caps already in force are the bound. `RunResult.coverage` is the settlement, one row per approach: validated (with the figure), rejected (with the validator's reason verbatim), or not_evaluated (with why). `not_evaluated` is the row that earns the type its keep — an approach the run never reached must be reported as unreached, because an omitted row is indistinguishable from an approach nobody ordered. That silence is the defect class krav 1 is asking us to remove. Budget exhaustion mid-list is reported, not swallowed. But if the FIRST approach exhausts it there is nothing honest to return, so BudgetExceeded propagates exactly as before — a run that produced nothing must still fail loudly. RunResult stays single-outcome (portfolio aggregation, outbox artefacts and HITL keying all rest on that). The choice is deterministic: highest validated saving, ties by mandate order — never whichever ran last. Load-bearing MEASURED against the whole 702-test suite. FIVE mutations red: evaluate only the first approach (5 red) · drop the rejected rows (3) · ignore allow_own_proposals (1) · select produced[-1] (1) · select produced[0] (1). The sixth measurement is why this commit exists in this shape: the ordering mutation FIRST STAYED GREEN. The test had placed the bigger approach last, where "highest saving" and "whichever ran last" give the same answer, so an order-dependent implementation passed it. A scenario that cannot separate two implementations proves nothing about either — the test now pins BOTH orderings, and each mutation direction fails one of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ULCqjLF61rehj5cZmdUoR3
This commit is contained in:
parent
fa1fa5aafd
commit
61bf5b78ea
3 changed files with 326 additions and 5 deletions
|
|
@ -29,7 +29,9 @@ Two refusals are load-bearing, both at construction time:
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
|
|
@ -86,6 +88,26 @@ class Mandate(BaseModel):
|
|||
return self
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ApproachOutcome:
|
||||
"""What became of ONE commissioned approach — one row of the run's coverage report.
|
||||
|
||||
``not_evaluated`` is the row that earns this type its keep: an approach the run never got to
|
||||
(budget exhausted, pass stopped) must be reported as *not evaluated*, never omitted. An omitted
|
||||
row is indistinguishable from an approach nobody ordered, which is exactly the silence krav 1
|
||||
is asking us to remove.
|
||||
|
||||
``detail`` carries the validator's reason on a rejection, or why an approach went unevaluated;
|
||||
``saving_nok`` is set only for a validated row (the claimed figure the validator admitted).
|
||||
"""
|
||||
|
||||
id: str
|
||||
label: str
|
||||
status: Literal["validated", "rejected", "not_evaluated"]
|
||||
detail: str = ""
|
||||
saving_nok: float | None = None
|
||||
|
||||
|
||||
def load_mandate(path: str | Path) -> Mandate:
|
||||
"""Fail-fast standalone loader for a run mandate (mirrors ``dimension.load_dimension``).
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue