open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(hitl): ekspertdommen kan ikke oppstaa av stillhet (F2, ORDRE 20260825T214801Z)

Browse source 
run_project KREVDE verdict_input og kjorte capture_verdict ubetinget; CLI-en
defaultet det til {"approved", "reviewed by expert"} og hosting listet det som
PAAKREVD. Netto: hver flaggloes kjoering myntet en ekspertgodkjenning ingen ga,
den gikk inn i den delte storen, og run_portfolio bar den inn i neste prosjekts
hypotese-prompt som en prior expert verdict -- paa flaten som ble overlevert
14.08. Non-goal 3, brutt i en soem.

RunResult.verdict er naa Verdict | None, og None er hva stillhet produserer:
ingenting myntes, ingenting lagres, ingenting varsles. Prinsippet sto allerede i
repoet -- RunFailure sin docstring: aa fylle et felt med en dummy legger
FABRIKKERT proveniens inn i aggregatet.

Traceability koster ingenting: RunResult.verdict_key (property, derivert fra
kandidaten) er verdicts.verdict_key sitt alt dokumenterte formaal -- identisk
med verdict.id naar en dom BLE gitt, og fortsatt meningsfull naar ingen ble det.
Det er den outboxen og den hostede responsen stempler.

Halv dom NEKTES paa begge doerer (FeedbackContract er eneste sted formen
valideres; CLI-en nekter ved navn FOER enhver mode-dispatch). Validering, aldri
reparasjon. De to mode-partisjonene fikk --decision/--rationale inn: kommentarene
sa ordrett at en aerlig nekt var uimplementerbar fordi de non-None
argparse-defaultene gjorde en eksplisitt verdi uskillbar fra defaulten -- med
defaultene borte er den implementerbar.

Hosting er WIDENING, ikke bryting: verdict_input flyttet fra _REQUIRED_FIELDS
til _OPTIONAL_FIELDS. Ingen ekstern kaller brekker.

AERLIGHETS-GRENSE: referanse-fixturens SYNTETISKE verdict_input-rader staar
uroert -- de er merket SYNTETISK paa fire steder og er reviewens F5 (maaling av
misjonspaastanden), ikke F2. Project.verdict_input er naa valgfri.

Load-bearing MAALT (tests/test_ungiven_verdict_loadbearing.py, 15 armer), aatte
mutasjoner alle roede mot HELE suiten + gronn kontroll 1080/5 og golden
demo-transcript.stdout BYTE-UENDRET (ea8c534773acdbe41ae68f2c55724d69aaf8be4f).
En mutasjon falsifiserte testen foerst (vakuoes-gate-klassen, ellevte gang):
--report-armen brukte et bart --report, som nekter rc 1 uansett fordi --ledger
mangler.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-08-27 01:22:07 +02:00
commit 56f4f6d084
12 changed files with 685 additions and 49 deletions
Download patch file Download diff file Expand all files Collapse all files

151
src/portfolio_optimiser/run.py
View file

@ -145,7 +145,13 @@ class RunResult:
outcome: ValidatedProposal | Rejection
provenance: ProvenanceStamp
verdict: Verdict
#: The Layer-2 expert verdict — ``None`` when NOBODY gave one (F2, non-goal 3). Absence is a
#: first-class state, not a hole to fill: the previous unconditional capture minted an
#: ``approved`` verdict for every run whose caller stayed silent, and ``run_portfolio`` then
#: carried it into the next project's hypothesis prompt as a prior expert judgement. The
#: sibling ``RunFailure`` docstring already states the principle this now honours — filling a
#: field with a dummy puts FABRICATED provenance into the aggregate.
verdict: Verdict | None
retrieved: list[Verdict]
store: VerdictStore
debate_output: str
@ -170,6 +176,17 @@ class RunResult:
#: and on any bundle that was read whole; it defaults for the same reason ``coverage`` does.
skipped_links: tuple[okf.SkippedLink, ...] = ()
@property
def verdict_key(self) -> str:
"""The id an expert verdict on THIS run's candidate will arrive under — always available,
including on a run nobody has reviewed. DERIVED from the candidate (never from a decision),
which is exactly what ``verdicts.verdict_key`` exists for, so 'no verdict' costs no
traceability: the outbox artefact and the hosted response can still name the key the honest
Step-7 inbox channel will join back on. A PROPERTY rather than a stored field because a
second copy of a keying rule is the ``(p)`` defect — and because a defaulted field would
have to state a value for a fact that is always derivable."""
return verdict_key(_features_of(self.outcome.proposal))
@dataclass(frozen=True)
class RunFailure:
@ -422,8 +439,8 @@ def _project_from_bundle(
) -> Project:
"""Derive a minimal ``Project`` from an OKF bundle (so a bundle the loop runs need NOT be a
road reference-domain project). Only ``id`` + ``name`` reach the generation prompt
(``generate._build_messages``), so ``cost_items`` is empty and ``verdict_input`` is unused here
(the Layer-2 decision flows via the ``verdict_input`` argument). Fail-fast: the bundle's IR
(``generate._build_messages``), so ``cost_items`` is empty and ``verdict_input`` is left unset
here (the Layer-2 decision flows via ``run_project``'s own ``verdict_input`` argument). Fail-fast: the bundle's IR
``project_id`` must match the requested id. ``bundle`` reuses an already-navigated bundle to
avoid a second navigation."""
ir = okf.load_ir_projection(bundle_dir)
@ -443,10 +460,32 @@ def _project_from_bundle(
currency="NOK",
cost_items=(),
docs_dir=bundle_dir,
verdict_input={},
)
def _verdict_input_from_args(args: Any) -> dict[str, str] | None:
"""The CLI's verdict, or ``None`` when the operator recorded none (F2). ``main`` has already
refused the half-given case by name, so both flags are set together or neither is. ``args`` is
typed ``Any`` because ``argparse`` is imported inside ``_build_parser``, not at module scope."""
if args.decision is None:
return None
return {"decision": args.decision, "rationale": args.rationale}
def verdict_notice(result: RunResult) -> str:
"""The ONE renderer for a run's verdict identity on stdout (F2). Present: the unchanged
``verdict id=…, decision=…`` — read off the run's OWN captured verdict rather than off argv, so
stdout and the store cannot disagree about what was recorded (the ``cost_baseline_notice``
precedent). Absent: it SAYS so, and names the key an expert verdict on this candidate would
arrive under — the operator's join back into the honest Step-7 inbox channel. Not an omission
like the ``*_notice`` renderers above: those describe an event that may not have happened,
whereas every run has a verdict identity to report, and a blank there would read as a missing
line rather than as 'nobody reviewed this'."""
if result.verdict is None:
return f"no expert verdict given; verdict key={result.verdict_key}"
return f"verdict id={result.verdict.id}, decision={result.verdict.decision}"
def _features_of(proposal: SavingsProposal) -> ProposalFeatures:
return ProposalFeatures(
affected_codes=frozenset(item.code for item in proposal.affected_items),
@ -533,7 +572,7 @@ async def run_project(
profile: Profile | str = Profile.LOCAL,
*,
docs_dir: str,
verdict_input: dict[str, str],
verdict_input: dict[str, str] | None = None,
bundle_dir: str | None = None,
dimension: Dimension | None = None,
store: VerdictStore | None = None,
@ -555,7 +594,10 @@ async def run_project(
) -> RunResult | DryRunReport:
"""Run the vertical slice for ONE project. ``client_factory`` is the test-injection seam
(defaults to the real backend). ``verdict_input`` carries the expert decision/rationale
(Layer-2). ``bundle_dir`` (Fase 2a) makes the run OKF-bundle-driven: the project is derived
(Layer-2) WHEN an expert gave one; omitted (the default) it means nobody reviewed this run, so
no verdict is minted, none enters ``store``, and ``RunResult.verdict`` is ``None`` (F2,
non-goal 3). Supplying it with only one of the two keys raises ``ValueError``: the missing half
is the expert's to write, never ours to default. ``bundle_dir`` (Fase 2a) makes the run OKF-bundle-driven: the project is derived
from the bundle and, before generation, the candidate's prior verdicts in ``store`` are folded
into the hypothesis prompt (Step-1 ExpeL wiring, målbilde §5/§7). ``verdict_dir`` (Fase 5,
Steg 7, målbilde §3/§7) is the async file inbox: a folder of expert/persona-authored verdict
@ -869,10 +911,20 @@ async def run_project(
retrieved = store.retrieve(features, k=top_k, retriever=ranker) if store.verdicts else []
# 8. Layer-2 (out-of-band): capture the durable verdict + persist; B11 notify is a stub.
verdict = capture_verdict(features, verdict_input["decision"], verdict_input["rationale"])
store.add(verdict)
if notify is not None:
notify(verdict)
# ONLY when an expert actually gave one (F2, non-goal 3). Absent ``verdict_input`` means
# nobody reviewed this run: nothing is minted, nothing enters the store, and nothing is
# notified — so silence cannot become an ``approved`` that propagates into the next
# project's hypothesis prompt as a prior expert judgement. A half-given verdict is a
# CALLER error, refused by name rather than completed on the expert's behalf (validation,
# never repair — the ``write_concept_file`` precedent).
# The SHAPE of a supplied verdict is not re-checked here: step 1's ``load_contracts``
# already ran ``FeedbackContract`` over it and refused a half-given one by field name.
verdict: Verdict | None = None
if verdict_input is not None:
verdict = capture_verdict(features, verdict_input["decision"], verdict_input["rationale"])
store.add(verdict)
if notify is not None:
notify(verdict)
# S2.1 outbox (RAW output layer, målbilde §3): persist the run's proposal + outcome artefacts
# when configured. Wired ONLY here — no new consumer (S5.1/S5.2 are Non-Goals this bolk). run_id
@ -886,7 +938,12 @@ async def run_project(
outcome=outcome,
provenance=stamp,
checker_verdict=checker_decision,
verdict_id=verdict.id,
# The artefact carries the candidate's KEY, not evidence that anybody decided:
# identical to ``verdict.id`` whenever a verdict WAS given (both mint from the
# same features), and still meaningful on a run nobody reviewed. This is the
# documented purpose of ``verdict_key`` and it is what keeps the per-approach
# branch below and this one speaking the same language.
verdict_id=verdict_key(features),
)
else:
# A5: one judgeable artefact PER evaluated approach. Without this the expert can only
@ -1401,7 +1458,7 @@ async def run_mandate_across_bundles(
bundle_dirs: Sequence[str],
profile: Profile | str = Profile.LOCAL,
*,
verdict_input: dict[str, str],
verdict_input: dict[str, str] | None = None,
store: VerdictStore | None = None,
verdict_dir: str | None = None,
dimension: Dimension | None = None,
@ -1749,8 +1806,25 @@ def main(argv: list[str] | None = None) -> int:
"--verdict-dir (without them it cannot take effect, and is refused rather than ignored). "
"OFF by default, and off means the structural ranking is unchanged",
)
parser.add_argument("--decision", default="approved", choices=["approved", "rejected"])
parser.add_argument("--rationale", default="reviewed by expert")
# F2 (non-goal 3): NO defaults. Silence means nobody reviewed the run, and the previous
# ``approved``/``reviewed by expert`` pair minted an expert judgement out of that silence —
# which then propagated into the next project's hypothesis prompt as a prior verdict. The two
# belong together: half a verdict is refused by name below, never completed on the expert's
# behalf.
parser.add_argument(
"--decision",
default=None,
choices=["approved", "rejected"],
help="the expert's recorded decision for this run. Omit it when nobody reviewed the run — "
"no verdict is then minted, nothing enters the learning store, and the summary line says "
"so. Requires --rationale",
)
parser.add_argument(
"--rationale",
default=None,
help="the expert's reasoning behind --decision (required with it; an expert verdict is a "
"decision AND its reasoning)",
)
parser.add_argument(
"--live-dry-run",
action="store_true",
@ -1796,6 +1870,23 @@ def main(argv: list[str] | None = None) -> int:
if tracing_line is not None:
print(tracing_line, file=sys.stderr)
# F2: half a verdict is refused BY NAME, before every mode dispatch below — an expert verdict
# is a decision AND its reasoning, and defaulting the missing half is exactly the seam that let
# an approval nobody spoke enter the learning store. Placed here (ahead of report mode, which
# RETURNS) so the pairing holds on every path, not only the ones that run a model.
if (args.decision is None) != (args.rationale is None):
given, absent = (
("--decision", "--rationale")
if args.decision is not None
else ("--rationale", "--decision")
)
print(
f"run refused: {absent} is required together with {given} (an expert verdict is a "
"decision AND its reasoning; omit BOTH when nobody reviewed the run)",
file=sys.stderr,
)
return 1
# S5.4: read-only value-report dispatch — placed FIRST (right after parse_args, BEFORE the
# mode-exclusivity block below) so it returns before any model/portfolio path can start and no
# later branch can shadow it (the bare `--ledger`-outside-portfolio refusal at the elif below is
@ -1808,9 +1899,11 @@ def main(argv: list[str] | None = None) -> int:
# Mode-exclusivity as an ALLOWLIST (not a short blocklist): report mode permits ONLY --ledger
# and --json; ANY other distinguishable mode/config flag is refused — else --report --goals
# would silently drop --goals, whereas bare --goals is refused below (adding --report must not
# suppress an existing refusal). --decision/--rationale are excluded: their non-None argparse
# defaults are indistinguishable from an explicit value (exactly as the block below excludes
# them); they are inert in report mode.
# suppress an existing refusal). --decision/--rationale ARE listed now: before F2 their
# non-None argparse defaults made an explicit value indistinguishable from the default, so
# an honest refusal was unimplementable and they had to be excluded. With the defaults gone
# they are distinguishable, and an operator who typed a real expert verdict must not have
# it silently dropped — 'refused, never ignored' is this partition's own rule.
report_forbidden = {
"--portfolio": args.portfolio,
"--live-dry-run": args.live_dry_run,
@ -1836,6 +1929,10 @@ def main(argv: list[str] | None = None) -> int:
"--checkpoint-dir": args.checkpoint_dir is not None,
"--review-inbox": args.review_inbox is not None,
"--resume": args.resume is not None,
# Distinguishable only since F2 removed their defaults. They are refused TOGETHER above
# when only one is given, so at most one situation reaches this list: both set.
"--decision": args.decision is not None,
"--rationale": args.rationale is not None,
}
if any(report_forbidden.values()):
print(
@ -1864,9 +1961,12 @@ def main(argv: list[str] | None = None) -> int:
# Step 4: mode-exclusivity validation (structured refusal, NOT argparse.error — keeps the rc 1
# refusal contract). The two CLI modes are a documented partition: single-project-only flags are
# refused in portfolio mode, and --goals/--ledger are refused outside it — never silently ignored.
# --decision/--rationale are excluded: their non-None argparse defaults make an explicit value
# indistinguishable from the default, so an honest refusal is unimplementable (they are inert in
# portfolio mode; the README documents that). --dimension-config is valid in BOTH modes.
# --decision/--rationale are single-project-only and REFUSED in portfolio mode since F2: a pass
# takes each project's verdict from its OWN row, so a run-level verdict flag has nowhere to go
# and silently dropping a real expert judgement is the thing this partition exists to prevent.
# Before F2 their non-None argparse defaults made an explicit value indistinguishable from the
# default and an honest refusal was unimplementable; that is no longer true.
# --dimension-config is valid in BOTH modes.
if args.portfolio:
single_only = {
"--docs-dir": args.docs_dir,
@ -1891,6 +1991,9 @@ def main(argv: list[str] | None = None) -> int:
"--checkpoint-dir": args.checkpoint_dir,
"--review-inbox": args.review_inbox,
"--resume": args.resume,
# See the block comment above: distinguishable only since F2.
"--decision": args.decision,
"--rationale": args.rationale,
}
offending = [name for name, value in single_only.items() if value]
if offending:
@ -2455,7 +2558,7 @@ def main(argv: list[str] | None = None) -> int:
print(f"portfolio run refused: {exc}", file=sys.stderr)
return 1
for r in portfolio_result.runs:
print(f"{type(r.outcome).__name__}: verdict id={r.verdict.id}")
print(f"{type(r.outcome).__name__}: {verdict_notice(r)}")
# Per project, because anchoring is a per-project fact. DEFENSIVE and currently
# unreachable from this branch — measured, and said out loud for the same reason the
# ``budget_stop`` arm below is: no reference project sets ``bundle_dir``, so every
@ -2531,7 +2634,7 @@ def main(argv: list[str] | None = None) -> int:
),
outbox_dir=args.outbox_dir,
run_id=args.run_id,
verdict_input={"decision": args.decision, "rationale": args.rationale},
verdict_input=_verdict_input_from_args(args),
mcp_servers=mcp_servers,
live_dry_run=True,
)
@ -2589,7 +2692,7 @@ def main(argv: list[str] | None = None) -> int:
),
outbox_dir=args.outbox_dir,
run_id=args.run_id,
verdict_input={"decision": args.decision, "rationale": args.rationale},
verdict_input=_verdict_input_from_args(args),
semantic_retrieval=args.semantic_retrieval,
client_factory=scripted_client_factory,
mandate=mandate,
@ -2603,7 +2706,7 @@ def main(argv: list[str] | None = None) -> int:
print(f"run refused: {exc}", file=sys.stderr)
return 1
kind = type(result.outcome).__name__
print(f"{args.project_id}: {kind} (verdict id={result.verdict.id}, decision={args.decision})")
print(f"{args.project_id}: {kind} ({verdict_notice(result)})")
# Same notice, same renderer, read off the run's OWN stamp — so stdout and the outbox artefact
# cannot disagree about whether the gate was anchored.
notice = cost_baseline_notice(result.provenance.cost_baseline_anchored)