open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(hitl): ekspertdommen kan ikke oppstaa av stillhet (F2, ORDRE 20260825T214801Z)

Browse source 
run_project KREVDE verdict_input og kjorte capture_verdict ubetinget; CLI-en
defaultet det til {"approved", "reviewed by expert"} og hosting listet det som
PAAKREVD. Netto: hver flaggloes kjoering myntet en ekspertgodkjenning ingen ga,
den gikk inn i den delte storen, og run_portfolio bar den inn i neste prosjekts
hypotese-prompt som en prior expert verdict -- paa flaten som ble overlevert
14.08. Non-goal 3, brutt i en soem.

RunResult.verdict er naa Verdict | None, og None er hva stillhet produserer:
ingenting myntes, ingenting lagres, ingenting varsles. Prinsippet sto allerede i
repoet -- RunFailure sin docstring: aa fylle et felt med en dummy legger
FABRIKKERT proveniens inn i aggregatet.

Traceability koster ingenting: RunResult.verdict_key (property, derivert fra
kandidaten) er verdicts.verdict_key sitt alt dokumenterte formaal -- identisk
med verdict.id naar en dom BLE gitt, og fortsatt meningsfull naar ingen ble det.
Det er den outboxen og den hostede responsen stempler.

Halv dom NEKTES paa begge doerer (FeedbackContract er eneste sted formen
valideres; CLI-en nekter ved navn FOER enhver mode-dispatch). Validering, aldri
reparasjon. De to mode-partisjonene fikk --decision/--rationale inn: kommentarene
sa ordrett at en aerlig nekt var uimplementerbar fordi de non-None
argparse-defaultene gjorde en eksplisitt verdi uskillbar fra defaulten -- med
defaultene borte er den implementerbar.

Hosting er WIDENING, ikke bryting: verdict_input flyttet fra _REQUIRED_FIELDS
til _OPTIONAL_FIELDS. Ingen ekstern kaller brekker.

AERLIGHETS-GRENSE: referanse-fixturens SYNTETISKE verdict_input-rader staar
uroert -- de er merket SYNTETISK paa fire steder og er reviewens F5 (maaling av
misjonspaastanden), ikke F2. Project.verdict_input er naa valgfri.

Load-bearing MAALT (tests/test_ungiven_verdict_loadbearing.py, 15 armer), aatte
mutasjoner alle roede mot HELE suiten + gronn kontroll 1080/5 og golden
demo-transcript.stdout BYTE-UENDRET (ea8c534773acdbe41ae68f2c55724d69aaf8be4f).
En mutasjon falsifiserte testen foerst (vakuoes-gate-klassen, ellevte gang):
--report-armen brukte et bart --report, som nekter rc 1 uansett fordi --ledger
mangler.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-08-27 01:22:07 +02:00
commit 56f4f6d084
12 changed files with 685 additions and 49 deletions
Download patch file Download diff file Expand all files Collapse all files

40
CLAUDE.md
View file

@ -1073,6 +1073,46 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
nye verktøyet, så at en manager velger BEDRE med et utdrag enn med hele indeksen er ikke bevist
(structured-output-grensens klasse); og ordrens nevner for N100:2023 var 34 mens disken viser 40 —
tallene bruker den målte nevneren. Måling: `docs/2026-08-26-katalogkostnaden.md`.
- **Ekspertdommen kan ikke oppstå av STILLHET, og fraværet er en FØRSTEKLASSES tilstand (F2,
non-goal 3, økt 66):** `run_project` KREVDE `verdict_input` og kjørte `capture_verdict`
ubetinget, CLI-en defaultet det til `{"approved", "reviewed by expert"}`, og hosting listet det
som PÅKREVD. Netto: hver flaggløs kjøring myntet en ekspertgodkjenning ingen ga, den gikk inn i
den delte storen, og `run_portfolio` bar den inn i neste prosjekts hypotese-prompt som en
*prior expert verdict* — på flaten som ble overlevert 14.08. **`RunResult.verdict` er nå
`Verdict | None`**, og `None` er hva stillhet produserer: ingenting myntes, ingenting lagres,
ingenting varsles. Prinsippet sto allerede skrevet i repoet — `RunFailure`s docstring: å fylle et
felt med en dummy legger FABRIKKERT proveniens inn i aggregatet. **Traceability koster ingenting,
fordi nøkkelen DERIVERES fra kandidaten:** `RunResult.verdict_key` (property, ikke lagret felt —
en andre kopi av en nøklingsregel er kø-(p)) er `verdicts.verdict_key`s alt dokumenterte formål,
identisk med `verdict.id` når en dom BLE gitt, og fortsatt meningsfull når ingen ble det; det er
den outboxen og den hostede responsen stempler, så et artefakt fra en ukommentert kjøring er
fortsatt dømbart og joiner tilbake via Steg-7-innboksen. **Halv dom NEKTES på begge dører**
(`FeedbackContract` er ENESTE sted formen valideres, og CLI-en nekter ved navn FØR enhver
mode-dispatch): den manglende halvdelen er ekspertens å skrive, aldri vår å defaulte — validering,
ALDRI reparasjon (`write_concept_file`-presedensen). **Hosting er WIDENING, ikke bryting:**
`verdict_input` flyttet `_REQUIRED_FIELDS` → `_OPTIONAL_FIELDS`, så hvert kall som finnes ute
virker uendret; en kaller som utelot det fikk før 400 på et felt som ikke KUNNE fylles ærlig.
**De to mode-partisjonene fikk `--decision`/`--rationale` inn — og det er en KONSEKVENS, ikke
scope-krype:** kommentarene på begge stedene sa ordrett at en ærlig nekt var *uimplementerbar*
fordi de non-None argparse-defaultene gjorde en eksplisitt verdi uskillbar fra defaulten. Med
defaultene borte er den implementerbar, og «refused, never ignored» er partisjonens egen regel.
`run.verdict_notice` er ENESTE renderer og leser dommen av kjøringens EGET stempel, ikke av argv.
**Ærlighets-grense, uttalt:** referanse-fixturens SYNTETISKE `verdict_input`-rader står URØRT —
de er merket SYNTETISK på fire steder og er reviewens F5 (måling av misjonspåstanden), ikke F2;
`Project.verdict_input` er nå valgfri, så en rad UTEN dom er lovlig. Load-bearing MÅLT
(`tests/test_ungiven_verdict_loadbearing.py`, 15 armer), **åtte mutasjoner alle røde mot HELE
suiten** + grønn kontroll 1080/5 og golden `demo-transcript.stdout` BYTE-UENDRET
(`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`): detach fangst-gaten (5 røde) · gjeninnfør
argparse-defaultene (22) · hosting krever fortsatt feltet (1) · CLI-en REPARERER en halv dom (2) ·
kontrakten reparerer en halv dom (1) · `verdict_key` lest av dommen i stedet for derivert (2) ·
begge partisjons-radene fjernet (2) · rendereren påstår en dom som aldri ble gitt (2). **ÉN
MUTASJON FALSIFISERTE TESTEN FØRST (repoets vakuøs-gate-klasse, ELLEVTE gang):** `--report`-armen
brukte et bart `--report`, som nekter med rc 1 uansett fordi `--ledger` mangler — testen sto
GRØNN med partisjons-raden fjernet. Den kjører nå mot en argv report-modus ellers ville AKSEPTERT
(gyldig `--ledger` + en kontroll som beviser rc 0 uten flaggene), så rc 1 er mutantens motsatte
utfall. **Migreringsnote:** ingen ekstern kaller brekker — hosting utvider, CLI-ens gamle
flaggform er uendret, og det som ENDRER seg er at en flaggløs kjøring nå SIER at ingen dømte i
stedet for å påstå `decision=approved`.
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.

2
DEPLOY.md
View file

@ -195,7 +195,7 @@ name with HTTP 400, never silently ignored:
|---|---|---|
| `project_id` | yes | The project to analyse. |
| `docs_dir` | yes | Source documents. |
| `verdict_input` | yes | The expert verdict for this run. |
| `verdict_input` | no | An expert verdict for this run, `{"decision", "rationale"}`, **when a human actually gave one**. Omit it otherwise — no verdict is then recorded and the response's `verdict_id` is simply the key a later verdict on this candidate will arrive under. Supplying only one of the two keys is refused (400). |
| `bundle_dir` | no | Knowledge base to navigate. |
| `profile` | no | Defaults to `azure` on this HTTP surface. |
| `max_rounds` | no | Debate round cap. |

40
README.md
View file

@ -154,11 +154,28 @@ project's own numbers cannot support, no matter how confidently the proposer ass
the part of the method that carries the weight — the agents propose, and something that cannot be
argued with decides.
Read that summary line carefully: `Rejection (verdict id=…, decision=approved)` is not a
contradiction. `Rejection` is the **validator's** outcome, while `decision=` echoes the
**human's** recorded verdict — here the `--decision` default, since nobody reviewed this run.
The two are deliberately separate: a machine gate that blocks, and a human judgement that
approves, are different questions and are never collapsed into one field.
Read that summary line carefully. Nobody reviewed this run, so it says exactly that:
`Rejection (no expert verdict given; verdict key=…)`. `Rejection` is the **validator's** outcome;
the second half is about the **human**, and there was no human here. The key it quotes is the id
under which an expert verdict on this candidate will arrive later — your join back into the
[expert-answer channel](docs/ekspert-svar.md).
Record one and the line changes:
```bash
uv run python -m portfolio_optimiser.run BYGG-KONTOR-NORD \
--docs-dir shared/examples/bygg-energi-mikro \
--bundle-dir shared/examples/bygg-energi-mikro \
--scripted-replies replies.json \
--decision approved --rationale "the retrofit is within scope"
```
Now it reads `Rejection (verdict id=…, decision=approved)`, which is not a contradiction: a machine
gate that blocks and a human judgement that approves are different questions and are never
collapsed into one field. The two flags go together or not at all — half a verdict is refused by
name, because the missing half is the expert's to write and never ours to default. Until 1.1.0
`--decision` defaulted to `approved`, so every flagless run recorded an approval nobody gave and
carried it into the next project's hypothesis; that default is gone.
**5 — See what it would cost with a real model**, before spending anything:
@ -570,7 +587,10 @@ when the seam is detached, so the loop cannot silently degrade into theater.
The **prior-verdict fold — the learning step — happens only on the `--bundle-dir` path**; a
plain `--docs-dir`-only run is single-shot (no fold). `--decision`/`--rationale` apply to the
single-project path only and are inert in portfolio mode. **`--outbox-dir` must differ from
single-project path only and are **refused** in portfolio and report mode rather than ignored —
a portfolio pass takes each project's verdict from its own row, so a run-level verdict flag has
nowhere to go, and silently dropping a judgement an expert actually typed is the failure this
partition exists to prevent. **`--outbox-dir` must differ from
`--verdict-dir`**: writing the raw outbox into a folder later read as an inbox would re-ingest
raw agent output past the promotion gate (self-contamination) — documented here, deliberately
not CLI-enforced. Stop criteria and budget caps are required at startup. Try the offline
@ -632,9 +652,11 @@ command) was **deleted with them** rather than weakened into a check that could
start command now has exactly one copy, in [`DEPLOY.md`](DEPLOY.md), and
`tests/test_handover_package_loadbearing.py` is what keeps it there.
An invocation is a JSON object whitelisted onto `run_project`'s signature — `project_id`,
`docs_dir` and `verdict_input` required; `bundle_dir`, `profile`, `max_rounds`, `max_tokens`
and `top_k` optional. Unknown fields are refused by name (400), never silently dropped.
An invocation is a JSON object whitelisted onto `run_project`'s signature — `project_id` and
`docs_dir` required; `verdict_input`, `bundle_dir`, `profile`, `max_rounds`, `max_tokens`
and `top_k` optional. `verdict_input` was required until 1.1.0, which forced an external caller to
invent an expert verdict just to get a run at all; omitting it now means nobody reviewed the run,
and the response's `verdict_id` is the key one would arrive under. Unknown fields are refused by name (400), never silently dropped.
`profile` defaults to `azure` on this surface: the AZURE profile reads its endpoint and
credential from the environment at call time, so the same process runs hosted (managed identity)
and locally (`az login`) without rewiring.

9
docs/ekspert-svar.md
View file

@ -23,6 +23,15 @@ Systemet tar imot en ekspertdom på to steder, og de gjør forskjellige ting:
Form B er den som lærer systemet noe. Form A er en synkron snarvei.
**Ingen av dem er default, og det er poenget.** Utelater du begge flaggene, blir det ikke registrert
noen dom i det hele tatt — kjøringen sier det høyt (`no expert verdict given; verdict key=…`), og
ingenting går inn i læringsstoren. Fram til 1.1.0 defaultet `--decision` til `approved`, slik at
hver flaggløs kjøring produserte en ekspertgodkjenning ingen ekspert hadde gitt, og bar den videre
inn i neste prosjekts hypotese. Nøkkelen linja skriver ut er den du fører inn i `id`-feltet i Form
B, og det er slik en dom avgitt dager senere finner tilbake til kandidaten den gjelder.
**Halv dom nektes:** `--decision` og `--rationale` må gis sammen. Den manglende halvdelen er
ekspertens å skrive, aldri systemets å fylle inn.
**`decision` er binær på kjørestien: `approved` eller `rejected`.** Det finnes ikke et tredje
«godkjent med justering» her — en realiseringskorreksjon er en **godkjenning** der forbeholdet står
i begrunnelsen. Det er med vilje: forbeholdet er fagkunnskap som skal leses, ikke en enum-verdi som

14
src/portfolio_optimiser/contracts.py
View file

@ -107,7 +107,11 @@ class Contracts(BaseModel):
data_source: DataSourceContract
model_map: ModelMapContract
termination: TerminationContract
feedback: FeedbackContract
#: ``None`` when the run carries no expert verdict at all (F2, non-goal 3). A run nobody
#: reviewed has no feedback to validate; a run that DOES claim one is validated exactly as
#: before, so a half-given verdict still fails fast here rather than being completed for the
#: expert further down.
feedback: FeedbackContract | None
def _bundled_model_map() -> dict[str, Any]:
@ -119,17 +123,19 @@ def _bundled_model_map() -> dict[str, Any]:
def load_contracts(
data_source: dict[str, Any],
termination: dict[str, Any],
feedback: dict[str, Any],
feedback: dict[str, Any] | None,
*,
model_map: dict[str, Any] | None = None,
) -> Contracts:
"""Validate ALL contracts at startup (fail-fast, before any chat-client is built). Raises
``pydantic.ValidationError`` on the first malformed contract. ``model_map`` defaults to the
bundled ``data/model_map.json`` (the same file Step 8 ships)."""
bundled ``data/model_map.json`` (the same file Step 8 ships). ``feedback`` is ``None`` for a
run nobody reviewed — the ONE place the expert-verdict shape is validated, so a caller that
supplies half a verdict is refused here by field name (F2)."""
raw_map = _bundled_model_map() if model_map is None else model_map
return Contracts(
data_source=DataSourceContract(**data_source),
model_map=ModelMapContract(**raw_map),
termination=TerminationContract(**termination),
feedback=FeedbackContract(**feedback),
feedback=None if feedback is None else FeedbackContract(**feedback),
)

21
src/portfolio_optimiser/hosting.py
View file

@ -77,8 +77,20 @@ from portfolio_optimiser.tracing import configure_tracing, tracing_notice
DEFAULT_PORT = 8088
_HOSTED_DEFAULT_PROFILE = "azure"
_REQUIRED_FIELDS = ("project_id", "docs_dir", "verdict_input")
_OPTIONAL_FIELDS = ("bundle_dir", "profile", "max_rounds", "max_tokens", "top_k")
_REQUIRED_FIELDS = ("project_id", "docs_dir")
#: ``verdict_input`` is OPTIONAL since F2 (non-goal 3). It used to be required, which forced an
#: external caller to invent an expert verdict just to get a run at all — a field that could not be
#: filled honestly, on the surface handed over 2026-08-14. The move is a pure WIDENING: a caller
#: that still sends it is unaffected, and one that omits it now gets a run whose verdict is
#: honestly absent.
_OPTIONAL_FIELDS = (
"bundle_dir",
"profile",
"max_rounds",
"max_tokens",
"top_k",
"verdict_input",
)
#: Fields this surface CONSUMES rather than forwards (U4). They are not ``run_project``
#: parameters — the exploration runs first and hands ``run_project`` a ``Mandate`` — so passing one
#: through would be a ``TypeError`` answered as a 500. The whitelist is therefore a THREE-way
@ -185,7 +197,10 @@ def _response_payload(result: RunResult) -> dict[str, Any]:
**outcome_payload(
result.outcome,
checker_verdict=result.checker_verdict,
verdict_id=result.verdict.id,
# The candidate's KEY, not evidence that anybody decided (F2): identical to the
# captured verdict's id whenever one was given, and still the id under which a later
# expert verdict on this candidate will arrive when none was.
verdict_id=result.verdict_key,
),
"proposal": result.outcome.proposal.model_dump(),
"provenance": result.provenance.model_dump(),

16
src/portfolio_optimiser/reference_domain.py
View file

@ -45,7 +45,12 @@ class Project:
currency: str
cost_items: tuple[CostItem, ...]
docs_dir: str # absolute path to this project's bundled cost-docs folder (config-driven)
verdict_input: dict[str, str] # SYNTHETIC Layer-2 expert decision/rationale (config-driven)
#: The Layer-2 expert decision/rationale, when an expert actually gave one. OPTIONAL and
#: ``None`` by default (F2, non-goal 3): a project nobody has reviewed has no honest value
#: here, and the previous required field made silence indistinguishable from an approval.
#: The shipped reference fixture DOES carry values, and labels them SYNTHETIC at every point —
#: that dataset's honesty is F5's subject (measuring the mission claim), not F2's.
verdict_input: dict[str, str] | None = None
bundle_dir: str | None = None # abs path to an OKF bundle backing this project (Fase 2a S2.0)
verdict_dir: str | None = None # abs path to this project's async verdict inbox (Fase 2a S2.0)
@ -59,8 +64,9 @@ def load_reference_projects() -> tuple[Project, ...]:
Each project's ``docs_dir`` is stored in the JSON relative to the package ``data/`` root
and resolved here to an absolute filesystem path; ``verdict_input`` carries the SYNTHETIC
Layer-2 expert decision/rationale. Missing keys raise ``KeyError`` (fail-fast, matching the
existing loader contract)."""
Layer-2 expert decision/rationale when the row declares one, and is ``None`` when it does not
(F2). Missing keys raise ``KeyError`` (fail-fast, matching the existing loader contract) —
``verdict_input`` is deliberately NOT among them."""
resource = files("portfolio_optimiser").joinpath(_DATA_RESOURCE)
raw = json.loads(resource.read_text(encoding="utf-8"))
@ -87,7 +93,9 @@ def load_reference_projects() -> tuple[Project, ...]:
for c in p["cost_items"]
),
docs_dir=str(files("portfolio_optimiser").joinpath(f"data/{p['docs_dir']}")),
verdict_input=p["verdict_input"],
# ``.get`` — never fail-fast: a reference row without a reviewed verdict is now legal
# (F2), and a missing key must mean "nobody reviewed it", not a load error.
verdict_input=p.get("verdict_input"),
bundle_dir=_resolve(p.get("bundle_dir")),
verdict_dir=_resolve(p.get("verdict_dir")),
)

151
src/portfolio_optimiser/run.py
View file

@ -145,7 +145,13 @@ class RunResult:
outcome: ValidatedProposal | Rejection
provenance: ProvenanceStamp
verdict: Verdict
#: The Layer-2 expert verdict — ``None`` when NOBODY gave one (F2, non-goal 3). Absence is a
#: first-class state, not a hole to fill: the previous unconditional capture minted an
#: ``approved`` verdict for every run whose caller stayed silent, and ``run_portfolio`` then
#: carried it into the next project's hypothesis prompt as a prior expert judgement. The
#: sibling ``RunFailure`` docstring already states the principle this now honours — filling a
#: field with a dummy puts FABRICATED provenance into the aggregate.
verdict: Verdict | None
retrieved: list[Verdict]
store: VerdictStore
debate_output: str
@ -170,6 +176,17 @@ class RunResult:
#: and on any bundle that was read whole; it defaults for the same reason ``coverage`` does.
skipped_links: tuple[okf.SkippedLink, ...] = ()
@property
def verdict_key(self) -> str:
"""The id an expert verdict on THIS run's candidate will arrive under — always available,
including on a run nobody has reviewed. DERIVED from the candidate (never from a decision),
which is exactly what ``verdicts.verdict_key`` exists for, so 'no verdict' costs no
traceability: the outbox artefact and the hosted response can still name the key the honest
Step-7 inbox channel will join back on. A PROPERTY rather than a stored field because a
second copy of a keying rule is the ``(p)`` defect — and because a defaulted field would
have to state a value for a fact that is always derivable."""
return verdict_key(_features_of(self.outcome.proposal))
@dataclass(frozen=True)
class RunFailure:
@ -422,8 +439,8 @@ def _project_from_bundle(
) -> Project:
"""Derive a minimal ``Project`` from an OKF bundle (so a bundle the loop runs need NOT be a
road reference-domain project). Only ``id`` + ``name`` reach the generation prompt
(``generate._build_messages``), so ``cost_items`` is empty and ``verdict_input`` is unused here
(the Layer-2 decision flows via the ``verdict_input`` argument). Fail-fast: the bundle's IR
(``generate._build_messages``), so ``cost_items`` is empty and ``verdict_input`` is left unset
here (the Layer-2 decision flows via ``run_project``'s own ``verdict_input`` argument). Fail-fast: the bundle's IR
``project_id`` must match the requested id. ``bundle`` reuses an already-navigated bundle to
avoid a second navigation."""
ir = okf.load_ir_projection(bundle_dir)
@ -443,10 +460,32 @@ def _project_from_bundle(
currency="NOK",
cost_items=(),
docs_dir=bundle_dir,
verdict_input={},
)
def _verdict_input_from_args(args: Any) -> dict[str, str] | None:
"""The CLI's verdict, or ``None`` when the operator recorded none (F2). ``main`` has already
refused the half-given case by name, so both flags are set together or neither is. ``args`` is
typed ``Any`` because ``argparse`` is imported inside ``_build_parser``, not at module scope."""
if args.decision is None:
return None
return {"decision": args.decision, "rationale": args.rationale}
def verdict_notice(result: RunResult) -> str:
"""The ONE renderer for a run's verdict identity on stdout (F2). Present: the unchanged
``verdict id=…, decision=…`` — read off the run's OWN captured verdict rather than off argv, so
stdout and the store cannot disagree about what was recorded (the ``cost_baseline_notice``
precedent). Absent: it SAYS so, and names the key an expert verdict on this candidate would
arrive under — the operator's join back into the honest Step-7 inbox channel. Not an omission
like the ``*_notice`` renderers above: those describe an event that may not have happened,
whereas every run has a verdict identity to report, and a blank there would read as a missing
line rather than as 'nobody reviewed this'."""
if result.verdict is None:
return f"no expert verdict given; verdict key={result.verdict_key}"
return f"verdict id={result.verdict.id}, decision={result.verdict.decision}"
def _features_of(proposal: SavingsProposal) -> ProposalFeatures:
return ProposalFeatures(
affected_codes=frozenset(item.code for item in proposal.affected_items),
@ -533,7 +572,7 @@ async def run_project(
profile: Profile | str = Profile.LOCAL,
*,
docs_dir: str,
verdict_input: dict[str, str],
verdict_input: dict[str, str] | None = None,
bundle_dir: str | None = None,
dimension: Dimension | None = None,
store: VerdictStore | None = None,
@ -555,7 +594,10 @@ async def run_project(
) -> RunResult | DryRunReport:
"""Run the vertical slice for ONE project. ``client_factory`` is the test-injection seam
(defaults to the real backend). ``verdict_input`` carries the expert decision/rationale
(Layer-2). ``bundle_dir`` (Fase 2a) makes the run OKF-bundle-driven: the project is derived
(Layer-2) WHEN an expert gave one; omitted (the default) it means nobody reviewed this run, so
no verdict is minted, none enters ``store``, and ``RunResult.verdict`` is ``None`` (F2,
non-goal 3). Supplying it with only one of the two keys raises ``ValueError``: the missing half
is the expert's to write, never ours to default. ``bundle_dir`` (Fase 2a) makes the run OKF-bundle-driven: the project is derived
from the bundle and, before generation, the candidate's prior verdicts in ``store`` are folded
into the hypothesis prompt (Step-1 ExpeL wiring, målbilde §5/§7). ``verdict_dir`` (Fase 5,
Steg 7, målbilde §3/§7) is the async file inbox: a folder of expert/persona-authored verdict
@ -869,10 +911,20 @@ async def run_project(
retrieved = store.retrieve(features, k=top_k, retriever=ranker) if store.verdicts else []
# 8. Layer-2 (out-of-band): capture the durable verdict + persist; B11 notify is a stub.
verdict = capture_verdict(features, verdict_input["decision"], verdict_input["rationale"])
store.add(verdict)
if notify is not None:
notify(verdict)
# ONLY when an expert actually gave one (F2, non-goal 3). Absent ``verdict_input`` means
# nobody reviewed this run: nothing is minted, nothing enters the store, and nothing is
# notified — so silence cannot become an ``approved`` that propagates into the next
# project's hypothesis prompt as a prior expert judgement. A half-given verdict is a
# CALLER error, refused by name rather than completed on the expert's behalf (validation,
# never repair — the ``write_concept_file`` precedent).
# The SHAPE of a supplied verdict is not re-checked here: step 1's ``load_contracts``
# already ran ``FeedbackContract`` over it and refused a half-given one by field name.
verdict: Verdict | None = None
if verdict_input is not None:
verdict = capture_verdict(features, verdict_input["decision"], verdict_input["rationale"])
store.add(verdict)
if notify is not None:
notify(verdict)
# S2.1 outbox (RAW output layer, målbilde §3): persist the run's proposal + outcome artefacts
# when configured. Wired ONLY here — no new consumer (S5.1/S5.2 are Non-Goals this bolk). run_id
@ -886,7 +938,12 @@ async def run_project(
outcome=outcome,
provenance=stamp,
checker_verdict=checker_decision,
verdict_id=verdict.id,
# The artefact carries the candidate's KEY, not evidence that anybody decided:
# identical to ``verdict.id`` whenever a verdict WAS given (both mint from the
# same features), and still meaningful on a run nobody reviewed. This is the
# documented purpose of ``verdict_key`` and it is what keeps the per-approach
# branch below and this one speaking the same language.
verdict_id=verdict_key(features),
)
else:
# A5: one judgeable artefact PER evaluated approach. Without this the expert can only
@ -1401,7 +1458,7 @@ async def run_mandate_across_bundles(
bundle_dirs: Sequence[str],
profile: Profile | str = Profile.LOCAL,
*,
verdict_input: dict[str, str],
verdict_input: dict[str, str] | None = None,
store: VerdictStore | None = None,
verdict_dir: str | None = None,
dimension: Dimension | None = None,
@ -1749,8 +1806,25 @@ def main(argv: list[str] | None = None) -> int:
"--verdict-dir (without them it cannot take effect, and is refused rather than ignored). "
"OFF by default, and off means the structural ranking is unchanged",
)
parser.add_argument("--decision", default="approved", choices=["approved", "rejected"])
parser.add_argument("--rationale", default="reviewed by expert")
# F2 (non-goal 3): NO defaults. Silence means nobody reviewed the run, and the previous
# ``approved``/``reviewed by expert`` pair minted an expert judgement out of that silence —
# which then propagated into the next project's hypothesis prompt as a prior verdict. The two
# belong together: half a verdict is refused by name below, never completed on the expert's
# behalf.
parser.add_argument(
"--decision",
default=None,
choices=["approved", "rejected"],
help="the expert's recorded decision for this run. Omit it when nobody reviewed the run — "
"no verdict is then minted, nothing enters the learning store, and the summary line says "
"so. Requires --rationale",
)
parser.add_argument(
"--rationale",
default=None,
help="the expert's reasoning behind --decision (required with it; an expert verdict is a "
"decision AND its reasoning)",
)
parser.add_argument(
"--live-dry-run",
action="store_true",
@ -1796,6 +1870,23 @@ def main(argv: list[str] | None = None) -> int:
if tracing_line is not None:
print(tracing_line, file=sys.stderr)
# F2: half a verdict is refused BY NAME, before every mode dispatch below — an expert verdict
# is a decision AND its reasoning, and defaulting the missing half is exactly the seam that let
# an approval nobody spoke enter the learning store. Placed here (ahead of report mode, which
# RETURNS) so the pairing holds on every path, not only the ones that run a model.
if (args.decision is None) != (args.rationale is None):
given, absent = (
("--decision", "--rationale")
if args.decision is not None
else ("--rationale", "--decision")
)
print(
f"run refused: {absent} is required together with {given} (an expert verdict is a "
"decision AND its reasoning; omit BOTH when nobody reviewed the run)",
file=sys.stderr,
)
return 1
# S5.4: read-only value-report dispatch — placed FIRST (right after parse_args, BEFORE the
# mode-exclusivity block below) so it returns before any model/portfolio path can start and no
# later branch can shadow it (the bare `--ledger`-outside-portfolio refusal at the elif below is
@ -1808,9 +1899,11 @@ def main(argv: list[str] | None = None) -> int:
# Mode-exclusivity as an ALLOWLIST (not a short blocklist): report mode permits ONLY --ledger
# and --json; ANY other distinguishable mode/config flag is refused — else --report --goals
# would silently drop --goals, whereas bare --goals is refused below (adding --report must not
# suppress an existing refusal). --decision/--rationale are excluded: their non-None argparse
# defaults are indistinguishable from an explicit value (exactly as the block below excludes
# them); they are inert in report mode.
# suppress an existing refusal). --decision/--rationale ARE listed now: before F2 their
# non-None argparse defaults made an explicit value indistinguishable from the default, so
# an honest refusal was unimplementable and they had to be excluded. With the defaults gone
# they are distinguishable, and an operator who typed a real expert verdict must not have
# it silently dropped — 'refused, never ignored' is this partition's own rule.
report_forbidden = {
"--portfolio": args.portfolio,
"--live-dry-run": args.live_dry_run,
@ -1836,6 +1929,10 @@ def main(argv: list[str] | None = None) -> int:
"--checkpoint-dir": args.checkpoint_dir is not None,
"--review-inbox": args.review_inbox is not None,
"--resume": args.resume is not None,
# Distinguishable only since F2 removed their defaults. They are refused TOGETHER above
# when only one is given, so at most one situation reaches this list: both set.
"--decision": args.decision is not None,
"--rationale": args.rationale is not None,
}
if any(report_forbidden.values()):
print(
@ -1864,9 +1961,12 @@ def main(argv: list[str] | None = None) -> int:
# Step 4: mode-exclusivity validation (structured refusal, NOT argparse.error — keeps the rc 1
# refusal contract). The two CLI modes are a documented partition: single-project-only flags are
# refused in portfolio mode, and --goals/--ledger are refused outside it — never silently ignored.
# --decision/--rationale are excluded: their non-None argparse defaults make an explicit value
# indistinguishable from the default, so an honest refusal is unimplementable (they are inert in
# portfolio mode; the README documents that). --dimension-config is valid in BOTH modes.
# --decision/--rationale are single-project-only and REFUSED in portfolio mode since F2: a pass
# takes each project's verdict from its OWN row, so a run-level verdict flag has nowhere to go
# and silently dropping a real expert judgement is the thing this partition exists to prevent.
# Before F2 their non-None argparse defaults made an explicit value indistinguishable from the
# default and an honest refusal was unimplementable; that is no longer true.
# --dimension-config is valid in BOTH modes.
if args.portfolio:
single_only = {
"--docs-dir": args.docs_dir,
@ -1891,6 +1991,9 @@ def main(argv: list[str] | None = None) -> int:
"--checkpoint-dir": args.checkpoint_dir,
"--review-inbox": args.review_inbox,
"--resume": args.resume,
# See the block comment above: distinguishable only since F2.
"--decision": args.decision,
"--rationale": args.rationale,
}
offending = [name for name, value in single_only.items() if value]
if offending:
@ -2455,7 +2558,7 @@ def main(argv: list[str] | None = None) -> int:
print(f"portfolio run refused: {exc}", file=sys.stderr)
return 1
for r in portfolio_result.runs:
print(f"{type(r.outcome).__name__}: verdict id={r.verdict.id}")
print(f"{type(r.outcome).__name__}: {verdict_notice(r)}")
# Per project, because anchoring is a per-project fact. DEFENSIVE and currently
# unreachable from this branch — measured, and said out loud for the same reason the
# ``budget_stop`` arm below is: no reference project sets ``bundle_dir``, so every
@ -2531,7 +2634,7 @@ def main(argv: list[str] | None = None) -> int:
),
outbox_dir=args.outbox_dir,
run_id=args.run_id,
verdict_input={"decision": args.decision, "rationale": args.rationale},
verdict_input=_verdict_input_from_args(args),
mcp_servers=mcp_servers,
live_dry_run=True,
)
@ -2589,7 +2692,7 @@ def main(argv: list[str] | None = None) -> int:
),
outbox_dir=args.outbox_dir,
run_id=args.run_id,
verdict_input={"decision": args.decision, "rationale": args.rationale},
verdict_input=_verdict_input_from_args(args),
semantic_retrieval=args.semantic_retrieval,
client_factory=scripted_client_factory,
mandate=mandate,
@ -2603,7 +2706,7 @@ def main(argv: list[str] | None = None) -> int:
print(f"run refused: {exc}", file=sys.stderr)
return 1
kind = type(result.outcome).__name__
print(f"{args.project_id}: {kind} (verdict id={result.verdict.id}, decision={args.decision})")
print(f"{args.project_id}: {kind} ({verdict_notice(result)})")
# Same notice, same renderer, read off the run's OWN stamp — so stdout and the outbox artefact
# cannot disagree about whether the gate was anchored.
notice = cost_baseline_notice(result.provenance.cost_baseline_anchored)

6
src/portfolio_optimiser/simulation.py
View file

@ -579,6 +579,9 @@ async def simulate_learning_loop(
)
# Gate-promote the persona verdict from the raw output layer into the OKF wiki (Steg 8).
# Narrowed, not defaulted: this demo ALWAYS hands Run A a persona verdict (``verdict_input``
# above), so ``None`` here would mean the wiring was cut — an assertion, not a fallback (F2).
assert run_a.verdict is not None, "Run A recorded no persona verdict — the wiring is detached"
promoted_path = promote_verdict(
copy_s,
run_a.verdict,
@ -972,6 +975,9 @@ def _run_trace_lines(result: RunResult, *, marker: str, marker_in_prompt: bool)
# captured while the run is live. The long one — a verdict file dropped into a folder after the
# run — is printed by ``main`` between the two runs, because that is when it happens.
lines.append(" Steg 7 — SVAR PÅ TILBAKEMELDING (ekspert-persona, kort løkke i kjøringen)")
# Same narrowing as above: every simulated run records a persona verdict, so ``None`` is a
# detached wiring rather than a state this renderer has to have a word for.
assert result.verdict is not None, "the simulated run recorded no persona verdict"
lines.append(f" dom: {result.verdict.decision}")
lines.append(f" begrunnelse: {_clip(result.verdict.rationale, 300)}")
return lines

8
tests/test_hosting_loadbearing.py
View file

@ -234,7 +234,9 @@ async def test_invocations_wires_payload_into_run_project(
assert body["outcome_type"] == "validated"
assert body["p90"] == 200.0
assert body["checker_verdict"] == "approve"
assert body["verdict_id"] == "vid-hosted"
# F2: the response quotes the candidate's verdict KEY, not the captured verdict's id — the two
# coincide on a real run that recorded one, and only the key survives a run nobody reviewed.
assert body["verdict_id"] == _result(_VALIDATED).verdict_key
assert body["proposal"]["measure"] == "LED-retrofit av kontorbelysning"
assert body["provenance"]["model"] == "synthetic"
@ -271,7 +273,9 @@ async def test_unknown_field_is_refused_never_repaired(
assert len(recorder.calls) == 1
@pytest.mark.parametrize("missing", ["project_id", "docs_dir", "verdict_input"])
# ``verdict_input`` is deliberately ABSENT from this list since F2: it is optional, and the arms
# proving the relaxation live in ``tests/test_ungiven_verdict_loadbearing.py``.
@pytest.mark.parametrize("missing", ["project_id", "docs_dir"])
async def test_missing_required_field_is_400(
served: str, monkeypatch: pytest.MonkeyPatch, missing: str
) -> None:

7
tests/test_scripted_cli_door_loadbearing.py
View file

@ -86,9 +86,12 @@ def test_scripted_door_runs_the_whole_loop_offline(bundle, replies_file, capsys)
rc = run.main(_argv(bundle, replies_file))
out = capsys.readouterr().out
assert rc == 0, out
# A full run reports its outcome type + the minted verdict id; a dry-run never gets this far.
# A full run reports its outcome type + the candidate's verdict key; a dry-run never gets this
# far. Since F2 this argv records NO expert verdict (no --decision/--rationale), so the line
# says so and quotes the key one would arrive under — asserting "verdict id=" here would be
# asserting that a run nobody reviewed minted an approval.
assert "BYGG-KONTOR-NORD:" in out
assert "verdict id=" in out
assert "no expert verdict given; verdict key=" in out
assert "LIVE-DRY-RUN" not in out

420
tests/test_ungiven_verdict_loadbearing.py Normal file
View file

@ -0,0 +1,420 @@
"""F2 (misjonsreview ``docs/2026-08-25-fable-misjonsreview.md``, non-goal 3): an expert verdict
must not be able to ARISE without an expert having given one, and a verdict nobody gave must not
PROPAGATE into the next project's hypothesis prompt as if it were one.
Before this seam, ``run_project`` REQUIRED ``verdict_input`` and unconditionally ran
``capture_verdict(features, verdict_input["decision"], ...)``. The CLI defaulted that to
``{"decision": "approved", "rationale": "reviewed by expert"}``, so every flagless run minted an
approval nobody spoke; the hosted surface listed the field as REQUIRED, so an external caller had
to invent one to get a run at all; and in ``run_portfolio`` the minted verdict entered the shared
store and reached the next project's ExpeL few-shot.
The tests here fasten the GOAL, not the mechanism: absence of a verdict must be REPRESENTABLE and
must be what silence produces. Every negative assert is paired with a control that proves the
event it denies can actually happen — a test that can only be green proves nothing (this repo's
recurring vacuous-gate class).
"""
from __future__ import annotations
import json
import shutil
from collections.abc import Callable
from importlib.resources import files
from pathlib import Path
from typing import Any
import pytest
from agent_framework import BaseChatClient
from pydantic import ValidationError
from conftest import SyntheticUsageChatClient
from portfolio_optimiser import hosting, run
from portfolio_optimiser.ledger import LedgerEntry, SavingsLedger
from portfolio_optimiser.reference_domain import Project
from portfolio_optimiser.run import run_portfolio, run_project
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
_PID = "BYGG-KONTOR-NORD"
_MINI_BUNDLE = str(files("portfolio_optimiser").joinpath("data/bundles/bygg-energi-mikro-a"))
_VALID_PROPOSER_REPLY = (
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
)
_CHECKER_APPROVE = "The retrofit is supported by the cited documents. VERDICT: APPROVE"
_GIVEN = {"decision": "approved", "rationale": "expert reviewed (sim)"}
@pytest.fixture(autouse=True)
def _isolate_model_env(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False)
def _role_factory(proposer_reply: str, checker_reply: str) -> Callable[[str], BaseChatClient]:
def factory(role: str) -> BaseChatClient:
return SyntheticUsageChatClient(
default_reply=checker_reply if role == "checker" else proposer_reply
)
return factory
@pytest.fixture()
def bundle(tmp_path: Path) -> Path:
"""A throwaway COPY — the shared fixture is commons-owned and is never mutated by a test."""
dst = tmp_path / "bundle"
shutil.copytree(BUNDLE_DIR, dst)
return dst
@pytest.fixture()
def replies_file(tmp_path: Path) -> Path:
path = tmp_path / "replies.json"
path.write_text(
json.dumps({"proposer": _VALID_PROPOSER_REPLY, "checker": _CHECKER_APPROVE}),
encoding="utf-8",
)
return path
def _argv(bundle: Path, replies_file: Path) -> list[str]:
return [
_PID,
"--docs-dir",
str(bundle),
"--bundle-dir",
str(bundle),
"--scripted-replies",
str(replies_file),
]
# --- T1: a verdict cannot ARISE from silence (library seam) ---------------------------------
async def test_run_without_verdict_input_captures_no_verdict(tmp_path) -> None:
"""T1 GOAL: a run nobody reviewed produces NO verdict — the field is ``None`` and the store
stays empty. RED before the seam: ``verdict_input`` was a REQUIRED keyword and step 8 minted
an approval unconditionally."""
result = await run_project(
_PID,
"local",
docs_dir=str(BUNDLE_DIR),
bundle_dir=str(BUNDLE_DIR),
client_factory=_role_factory(_VALID_PROPOSER_REPLY, _CHECKER_APPROVE),
max_rounds=2,
)
assert result.verdict is None, "a verdict nobody gave was minted anyway"
assert result.store.verdicts == [], "an ungiven verdict entered the learning store"
async def test_run_with_verdict_input_still_captures_it(tmp_path) -> None:
"""T1 CONTROL: when an expert DOES speak, the verdict is captured and stored exactly as
before. Without this arm T1 would pass against an implementation that never captures at all."""
result = await run_project(
_PID,
"local",
docs_dir=str(BUNDLE_DIR),
bundle_dir=str(BUNDLE_DIR),
verdict_input=_GIVEN,
client_factory=_role_factory(_VALID_PROPOSER_REPLY, _CHECKER_APPROVE),
max_rounds=2,
)
assert result.verdict is not None
assert result.verdict.decision == "approved"
assert result.verdict.rationale == "expert reviewed (sim)"
assert [v.id for v in result.store.verdicts] == [result.verdict.id]
async def test_verdict_key_is_available_even_with_no_verdict(tmp_path) -> None:
"""T1b: the KEY an expert verdict on this candidate will arrive under is always available —
it is derived from the candidate, not from a decision. This is what keeps the outbox artefact
and the hosted response judgeable on a run nobody has reviewed yet (``verdicts.verdict_key``'s
own documented purpose), so 'no verdict' costs no traceability."""
result = await run_project(
_PID,
"local",
docs_dir=str(BUNDLE_DIR),
bundle_dir=str(BUNDLE_DIR),
client_factory=_role_factory(_VALID_PROPOSER_REPLY, _CHECKER_APPROVE),
max_rounds=2,
)
assert result.verdict is None
assert result.verdict_key, "the run must still name the key a verdict on it would arrive under"
given = await run_project(
_PID,
"local",
docs_dir=str(BUNDLE_DIR),
bundle_dir=str(BUNDLE_DIR),
verdict_input=_GIVEN,
client_factory=_role_factory(_VALID_PROPOSER_REPLY, _CHECKER_APPROVE),
max_rounds=2,
)
# Same candidate -> same key, and a captured verdict keys under exactly it.
assert given.verdict is not None
assert given.verdict.id == given.verdict_key == result.verdict_key
# --- T2: the CLI's silence is silence, not an approval --------------------------------------
def test_cli_without_decision_flags_reports_no_verdict(bundle, replies_file, capsys) -> None:
"""T2 GOAL: a flagless CLI run says a verdict was NOT given. RED before the seam: argparse
defaulted to ``approved``/``reviewed by expert`` and the line read ``verdict id=…,
decision=approved`` for a run nobody reviewed."""
rc = run.main(_argv(bundle, replies_file))
out = capsys.readouterr().out
assert rc == 0, out
assert "verdict id=" not in out, out
assert "no expert verdict" in out, out
def test_cli_with_decision_flags_reports_the_verdict(bundle, replies_file, capsys) -> None:
"""T2 CONTROL: an operator who DOES record a verdict gets the unchanged line. Proves T2's
negative is caused by the absent flags, not by the reporting having been removed."""
rc = run.main(
[*_argv(bundle, replies_file), "--decision", "approved", "--rationale", "I reviewed it"]
)
out = capsys.readouterr().out
assert rc == 0, out
assert "verdict id=" in out, out
assert "decision=approved" in out, out
def test_cli_outbox_artefact_still_carries_a_verdict_key(
bundle, replies_file, tmp_path, capsys
) -> None:
"""T2b: 'no verdict' must not cost the artefact its identity — the outcome file still carries
the key a later expert verdict on this candidate will arrive under, which is how the honest
inbox channel (Step 7) joins back to this run."""
outbox = tmp_path / "outbox"
rc = run.main([*_argv(bundle, replies_file), "--outbox-dir", str(outbox), "--run-id", "r1"])
assert rc == 0, capsys.readouterr().out
payload = json.loads((outbox / "r1-outcome.json").read_text(encoding="utf-8"))
assert payload["verdict_id"], "the outbox lost the candidate's verdict key"
@pytest.mark.parametrize(
("flags", "missing"),
[
(["--decision", "approved"], "--rationale"),
(["--rationale", "I reviewed it"], "--decision"),
],
)
def test_cli_half_a_verdict_is_refused_by_name(
bundle, replies_file, capsys, flags: list[str], missing: str
) -> None:
"""T3: a verdict is a decision AND its reasoning. Half of one is refused BY NAME before any
model call — validation, never repair (the alternative is filling the other half in on the
expert's behalf, which is the very defect F2 closes). RED before the seam: the missing half
silently took its argparse default."""
rc = run.main([*_argv(bundle, replies_file), *flags])
captured = capsys.readouterr()
assert rc == 1, captured.out
assert missing in captured.err, captured.err
assert "verdict id=" not in captured.out, captured.out
# --- T4/T5: an ungiven verdict cannot PROPAGATE to the next project --------------------------
_SENTINEL = "SENTINEL-F2-3d71ac realiseringskorreksjon fra prosjekt k"
_ALIGNED_REPLY = (
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
'[{"code":"ENERGI-TOTAL-EL","quantity":180000,"unit_cost":1.0}],"claimed_saving_nok":18000}'
)
def _generation_prompts(sink: list[str]) -> list[str]:
return [p for p in sink if "SavingsProposal" in p]
def _make_docs(tmp_path: Path, name: str) -> str:
d = tmp_path / name
d.mkdir()
(d / "cost.txt").write_text(
"Asphalt Ab11 unit rate renegotiation reduced the paving cost on the school stretch.",
encoding="utf-8",
)
return str(d)
def _road_k(tmp_path: Path, *, verdict_input: dict[str, str] | None) -> Project:
return Project(
id="ROAD-K",
name="Road k",
description="road-backed project k",
currency="NOK",
cost_items=(),
docs_dir=_make_docs(tmp_path, "k-docs"),
verdict_input=verdict_input,
bundle_dir=None,
verdict_dir=None,
)
def _bundle_kplus1(tmp_path: Path) -> Project:
return Project(
id="BYGG-ENERGI-MIKRO-A",
name="Bundle k+1",
description="bundle-backed project k+1",
currency="NOK",
cost_items=(),
docs_dir=_make_docs(tmp_path, "kplus1-docs"),
verdict_input={"decision": "approved", "rationale": "k+1 reviewed (sim)"},
bundle_dir=_MINI_BUNDLE,
verdict_dir=None,
)
async def test_a_given_verdict_on_k_does_reach_kplus1(
tmp_path, monkeypatch, make_recording_client_factory
) -> None:
"""T4 CONTROL (mirrors ``test_portfolio_learning_loadbearing``): when an expert DID review
project *k*, that verdict reaches *k+1*'s hypothesis prompt. This is the event T5 denies —
proving first that it can happen is what makes T5's negative assert mean anything."""
k = _road_k(tmp_path, verdict_input={"decision": "approved", "rationale": _SENTINEL})
kplus1 = _bundle_kplus1(tmp_path)
monkeypatch.setattr("portfolio_optimiser.run.load_reference_projects", lambda: (k, kplus1))
factory, recorded = make_recording_client_factory(_ALIGNED_REPLY)
result = await run_portfolio(profile="local", client_factory=factory)
assert result.runs[0].verdict is not None
prompts = _generation_prompts(recorded)
assert any(_SENTINEL in p for p in prompts)
assert any(result.runs[0].verdict.id in p for p in prompts)
async def test_an_ungiven_verdict_on_k_never_reaches_kplus1(
tmp_path, monkeypatch, make_recording_client_factory
) -> None:
"""T5 GOAL: project *k* that nobody reviewed contributes NOTHING to *k+1*'s hypothesis
prompt — no verdict is minted, none enters the shared store, and *k*'s candidate key never
shows up as a prior judgement. RED before the seam: *k* minted an ``approved`` verdict from a
``verdict_input`` no expert supplied, and the threaded store carried it forward."""
k = _road_k(tmp_path, verdict_input=None)
kplus1 = _bundle_kplus1(tmp_path)
monkeypatch.setattr("portfolio_optimiser.run.load_reference_projects", lambda: (k, kplus1))
factory, recorded = make_recording_client_factory(_ALIGNED_REPLY)
result = await run_portfolio(profile="local", client_factory=factory)
assert result.runs[0].verdict is None, "k minted a verdict nobody gave"
k_key = result.runs[0].verdict_key
prompts = _generation_prompts(recorded)
assert prompts, "the generation calls must have happened"
assert not any(k_key in p for p in prompts), (
"k's ungiven verdict propagated into k+1's hypothesis prompt as a prior judgement"
)
assert [v.id for v in result.store.verdicts] == [result.runs[1].verdict.id], (
"the shared store holds a verdict for a project nobody reviewed"
)
# --- T6: the hosted surface no longer FORCES a caller to invent a verdict ---------------------
async def test_hosted_invocation_without_verdict_input_is_accepted(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""T6 GOAL: ``verdict_input`` is no longer a REQUIRED field, so an external caller who has no
expert verdict can run at all. A RELAXATION — a caller that still sends the field is
unaffected (the control below). RED before the seam: 400, naming ``verdict_input``."""
calls: list[dict[str, Any]] = []
async def _runner(*args: Any, **kwargs: Any):
calls.append(kwargs)
raise AssertionError("stop after the whitelist") # pragma: no cover
monkeypatch.setattr(hosting, "run_project", _runner)
payload = {"project_id": "P1", "docs_dir": "docs"}
with pytest.raises(AssertionError, match="stop after the whitelist"):
await hosting.invoke(payload)
assert calls, "the whitelist refused a payload that omitted verdict_input"
assert calls[0].get("verdict_input") is None
async def test_hosted_invocation_with_verdict_input_still_forwards_it(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""T6 CONTROL: the outward-facing contract is only WIDENED — a caller that sends the field
still has it forwarded verbatim, so no call that exists out there breaks."""
calls: list[dict[str, Any]] = []
async def _runner(*args: Any, **kwargs: Any):
calls.append(kwargs)
raise AssertionError("stop after the whitelist") # pragma: no cover
monkeypatch.setattr(hosting, "run_project", _runner)
payload = {"project_id": "P1", "docs_dir": "docs", "verdict_input": _GIVEN}
with pytest.raises(AssertionError, match="stop after the whitelist"):
await hosting.invoke(payload)
assert calls[0]["verdict_input"] == _GIVEN
# --- T7: the library refuses half a verdict too, and the partitions no longer drop a real one ---
async def test_library_half_a_verdict_is_refused_by_field_name(tmp_path) -> None:
"""T7: the CLI is not the only door. A caller that hands ``run_project`` half a verdict is
refused at step 1 by ``FeedbackContract`` — the ONE place the shape is validated — rather than
having the missing half filled in for the expert. RED if that contract is made tolerant."""
with pytest.raises(ValidationError):
await run_project(
_PID,
"local",
docs_dir=str(BUNDLE_DIR),
bundle_dir=str(BUNDLE_DIR),
verdict_input={"decision": "approved"},
client_factory=_role_factory(_VALID_PROPOSER_REPLY, _CHECKER_APPROVE),
max_rounds=2,
)
def test_portfolio_mode_refuses_a_recorded_verdict_by_name(bundle, replies_file, capsys) -> None:
"""T8a: before F2 the argparse defaults made ``--decision`` indistinguishable from its default,
so the partition could not refuse it and an operator's real expert verdict was silently
dropped in portfolio mode (a pass takes each project's verdict from its own row). It is
distinguishable now, so it is refused BY NAME — 'refused, never ignored' is the partition's own
rule. RED when the rows are dropped from ``single_only``."""
rc = run.main(["--portfolio", "--decision", "approved", "--rationale", "I reviewed it"])
captured = capsys.readouterr()
assert rc == 1, captured.out
assert "--decision" in captured.err, captured.err
assert "--rationale" in captured.err, captured.err
def test_report_mode_refuses_a_recorded_verdict(tmp_path, capsys) -> None:
"""T8b: same rule on the OTHER partition. Report mode's refusal is generic by construction (it
names the allowlist, not the offender), so the discriminator has to be the OUTCOME — and the
argv must be one that report mode would otherwise ACCEPT. Measured while mutating: with a bare
``--report`` this test was VACUOUS (green with the rows dropped), because ``--report`` without
``--ledger`` refuses with rc 1 for a completely different reason. A valid ``--ledger`` makes
rc 0 the mutant's outcome, so rc 1 here means the partition refused."""
led = SavingsLedger()
led.add_realized(
LedgerEntry(
project_id="P1",
dimension="energi",
candidate_identity="c-a",
amount_ore=100000,
verdict_id="v1",
provenance="p1",
)
)
ledger_path = tmp_path / "ledger.json"
led.save(str(ledger_path))
# CONTROL: the same argv WITHOUT the verdict flags is accepted and prints a report.
assert run.main(["--report", "--ledger", str(ledger_path)]) == 0
capsys.readouterr()
rc = run.main(
["--report", "--ledger", str(ledger_path), "--decision", "approved", "--rationale", "x"]
)
captured = capsys.readouterr()
assert rc == 1, captured.out
assert "--report" in captured.err, captured.err