open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 4 Activity
portfolio-optimiser/src/portfolio_optimiser/run.py
Kjell Tore Guttormsen 5e4c497a84 feat(p17b): ONE commission, SEVERAL bases -- reachable from the command line 
``run_mandate_across_bundles`` has existed since session 58, reachable from FIVE
test files and from NO command line (measured: ``grep -n across-bundle run.py``
= 0 hits). ``--across-bundle <dir>``, repeated once per base, is that door.

The engine takes a CALLBACK rather than an outbox directory. Its own docstring
has always said N runs need N ``run_id``s and that minting them there would
default a key this repo requires a caller to supply -- so ``outbox_for`` is that
contract KEPT, not relaxed, and the operator-chosen ``<run-id>-<bundle_id>``
rule lives in ``main()`` where the decision was made. The order's alternative (a
caller running ``run_project`` itself over ``route_by_bundle``'s sub-mandates)
would be a second copy of the loop's id reconciliation, shared store, per-base
project resolution, collision accounting and both budget teeth.

``resolve_bundle_routing`` is ONE resolution shared by the engine and the
dry-run arm: a free trip answering with a different project id, or tolerating a
duplicate id the paid dispatch refuses, would rehearse a different run.

``{run-id}-multibase.json`` is written from a ``finally`` and every row is built
from the resolution plus disk, so the pass a cap cut short still leaves the
record. ``completed`` is a required field for ``ExplorationTrace.completed``'s
reason. ``stop_reason`` is read BACK from each base's own coverage artefact.

Load-bearing MEASURED (17 arms), four mutations all red against the WHOLE suite,
green control 1761/5 (from 1744/5, superset, 0 removed), golden byte-unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-15 04:24:48 +02:00

4510 lines
258 KiB
Python
Raw Blame History

"""Vertical-slice orchestrator + single-command entry (two-layer HITL wiring).
``run_project`` composes the whole method for ONE synthetic project on real (or injected)
chat clients:
1. ``load_contracts`` — fail-fast: every config (incl. the verdict-feedback shape) is
validated BEFORE any chat client is built.
2. load the project + retrieve cited chunks via the Step-7 data source -> first-class
``provenance.Citation`` list.
3. a FRESH maker-checker ``GroupChat`` debate (``fresh_workflow``), round-capped
(``with_max_rounds``); **Layer-1 HITL** = the optional in-run ``with_request_info`` gate.
4. ``generate_via_llm`` -> blocking ``validate_proposal`` -> ``ValidatedProposal | Rejection``
(the token bound is the ``meter`` checked in the generate loop).
5. attach a first-class ``ProvenanceStamp``.
6. **Layer-2 (out-of-band)**: ``capture_verdict`` mints a stable id from the proposal's
features; the decision/rationale come from ``verdict_input`` (function arg / CLI / fixture).
The B11 expert notification is a STUB (``notify``) in Fase 2.
7. persist to the ``VerdictStore`` -> the next run's ``ExpeLContextProvider`` retrieval (the
learning loop; this run also exercises the two-arg ``extend_instructions`` injection).
The two HITL layers are deliberately distinct: **Layer-1** is the optional synchronous in-run
review gate (no checkpoint — research 01: durable resume is fragile); **Layer-2** is the
durable learned verdict captured out-of-band in the VerdictStore (D7-portable).
"""
from __future__ import annotations
import asyncio
import json
from collections.abc import Awaitable, Callable, Iterable, Sequence
from contextlib import AsyncExitStack
from dataclasses import asdict, dataclass, replace
from pathlib import Path
from typing import Any, Literal, cast
from agent_framework import BaseChatClient, SessionContext
from agent_framework.exceptions import ChatClientException
from pydantic import ValidationError
from portfolio_optimiser.backends import Profile, get_backend, resolve_model
from portfolio_optimiser.budget import (
Budget,
BudgetExceeded,
BudgetMiddleware,
BudgetRefused,
PortfolioMeter,
TokenMeter,
)
from portfolio_optimiser.contracts import GoalConfig, GoalContract, load_contracts, load_goal_config
from portfolio_optimiser.ledger import SavingsLedger, to_ore
from portfolio_optimiser.datasource import (
bundle_citations,
bundle_excerpt_citations,
chunk_dict_to_citation,
make_retrieval_tool,
retrieve_chunks,
)
from portfolio_optimiser.dimension import Dimension, admits, load_dimension
from portfolio_optimiser.explore import (
HYPOTHESISER_ROLE,
MANAGER_ROLE,
NAVIGATOR_ROLE,
ExplorationContract,
ExplorationResult,
ExplorationToolRecorder,
ExplorationTrace,
DeclaredRequirement,
ParkedStateError,
PlanReviewDecision,
PlanReviewParked,
ToolCall,
explore,
exploration_notice,
load_exploration_contract,
load_parked,
parked_notice,
parked_payload,
navigator_tools,
requirement_payload,
resume_exploration,
terminal_plan_reviewer,
tool_call_payload,
trace_payload,
)
from portfolio_optimiser.generate import (
GroundingOffer,
ParseFailure,
generate_via_llm,
grounding_offer,
)
from portfolio_optimiser.ir import SavingsProposal
from portfolio_optimiser.mandate import (
OWN_PROPOSAL_ID,
Approach,
ApproachOutcome,
Mandate,
MandateCandidateError,
MandateRoutingError,
announce,
candidate_from_approach,
load_mandate,
route_by_bundle,
settle,
)
from portfolio_optimiser.mcp_tools import (
McpServerConfig,
ToolCallRecorder,
build_mcp_tools,
load_mcp_config,
service_labels,
tool_server_index,
)
from portfolio_optimiser.proposal_review import (
ProposalReview,
ProposalReviewer,
ProposalReviewInputError,
proposal_review_notice,
proposal_reviews_payload,
terminal_proposal_reviewer,
)
from portfolio_optimiser.provenance import ProvenanceStamp
from portfolio_optimiser.reference_domain import Project, load_reference_projects
from portfolio_optimiser.tracing import TracingConfigError, configure_tracing, tracing_notice
from portfolio_optimiser.validator import (
Grounding,
Rejection,
ValidatedProposal,
baseline_from_project,
classify_codes,
validate_proposal,
)
from portfolio_optimiser import hitl, okf, outbox, prepass
from portfolio_optimiser.prepass import load_prepass_payload
from portfolio_optimiser.semretrieval import (
SEMANTIC_WEIGHT_DEFAULT,
Embedder,
FakeEmbedder,
HybridRanker,
build_embedder,
load_embedder_config,
)
from portfolio_optimiser.verdicts import (
VerdictCollision,
ExpeLContextProvider,
ProposalFeatures,
Verdict,
VerdictStore,
optional_bundle_candidate_features,
capture_verdict,
load_verdicts_from_dir,
similarity,
verdict_key,
)
from portfolio_optimiser.value_report import (
build_value_report,
dump_report_json,
format_report_text,
)
from portfolio_optimiser.workflow import _MAKER_CHECKER_ROLES, fresh_workflow
#: The caps a CLI run actually uses. Named constants rather than repeated literals because the
#: run announcement (Trekk A2) PRINTS them: a second copy could drift and make the announcement
#: describe a run that never happened.
_DEFAULT_MAX_ROUNDS = 3
_DEFAULT_MAX_TOKENS = 100_000
@dataclass(frozen=True)
class RunResult:
"""The outcome of one project run: the validated/rejected proposal, its first-class
provenance, the captured (Layer-2) verdict, the ExpeL hits surfaced for it, the store, the
debate's converged output that the candidate was generated from (F1 traceability), and the
checker's gate decision (Step 3/4: ``"approve" | "reject" | "absent"``). ``checker_verdict``
records the checker's decision distinctly from ``provenance.validator_decision`` so the two
falsifiers (reasoning vs numbers) are never conflated."""
outcome: ValidatedProposal | Rejection
provenance: ProvenanceStamp
#: The Layer-2 expert verdict — ``None`` when NOBODY gave one (F2, non-goal 3). Absence is a
#: first-class state, not a hole to fill: the previous unconditional capture minted an
#: ``approved`` verdict for every run whose caller stayed silent, and ``run_portfolio`` then
#: carried it into the next project's hypothesis prompt as a prior expert judgement. The
#: sibling ``RunFailure`` docstring already states the principle this now honours — filling a
#: field with a dummy puts FABRICATED provenance into the aggregate.
verdict: Verdict | None
retrieved: list[Verdict]
store: VerdictStore
debate_output: str
checker_verdict: str = "absent"
#: One row per commissioned approach (Trekk A4). EMPTY when the run had no mandate — an empty
#: report is honest there, because nothing was ordered. It defaults so every existing
#: constructor call and every frozen aggregate over ``RunResult`` is unaffected.
coverage: tuple[ApproachOutcome, ...] = ()
#: Step 5 (målbilde §5/§7): the validator falsifications that informed a LATER generation
#: attempt, in attempt order — what ``generate_via_llm`` corrected in response to, rather than
#: only what it ended up with. EMPTY on the common path where the first candidate validates:
#: nothing was falsified, so there is nothing to show. Honesty limit: with a mandate this is
#: the run's refinements CONCATENATED across every commissioned approach, not keyed per
#: approach — ``coverage`` is the per-approach report, and hanging proposals off its rows is
#: what ``_evaluate_mandate`` deliberately avoids. It defaults, so every existing constructor
#: call is unaffected (mirrors ``coverage``).
refinements: tuple[Rejection, ...] = ()
#: Every cross-link the bundle navigation could not follow. A RUN-level fact, carried here and
#: NOT on ``provenance``: navigation happens ONCE per run, before any proposal exists, and the
#: same walk backs every refinement attempt — whereas ``ProvenanceStamp.cost_baseline_anchored``
#: describes the gate that judged ONE candidate. EMPTY on the road path (no bundle is navigated)
#: and on any bundle that was read whole; it defaults for the same reason ``coverage`` does.
skipped_links: tuple[okf.SkippedLink, ...] = ()
#: How many prior expert verdicts were in the store but could NOT be folded into the hypothesis
#: prompt, because the knowledge base declares no IR projection to key retrieval against (S7b
#: søm 1). 0 is the honest POSITIVE statement — every verdict the store held was keyable, which
#: covers both "the fold ran" and "there was nothing to fold" — so it defaults, exactly as
#: ``skipped_links``' empty tuple does.
#:
#: A COUNT, not a flag: "the fold did not happen" and "two judgements never reached the model"
#: are different operative facts, which is ``BudgetExceeded``'s kø-(y) rule one level down.
#:
#: Carried HERE and on neither of the other two carriers, and that placement is MEASURED.
#: ``ProvenanceStamp`` describes the gate that judged ONE candidate, whereas this is settled
#: once per run before any candidate exists (``skipped_links``' own reason). ``DryRunReport``
#: cannot carry it at all: the dry-run cut returns ABOVE the fold, so a field there could only
#: ever report zero — unlike ``cost_baseline_anchored`` and ``skipped_links``, both resolved
#: above that cut.
unkeyed_verdicts: int = 0
#: Which documents the DEBATE opened, in call order (S2c). Since the debate navigates the base
#: instead of being handed ``bundle_context``, "what did this run read" is no longer answerable
#: from the prompts — that is the whole saving — so the run carries the trace itself.
#:
#: EMPTY is an honest POSITIVE statement ("the debate opened nothing"), which is why it
#: defaults, exactly as ``skipped_links`` does; and it is also the S2c regression signal, which
#: is why the outbox artefact is written even when it is empty rather than only on activity.
#: The RESULT of each call is deliberately absent — that is the base's content, i.e. the very
#: thing too big to ride along (``ToolCall``'s own rule, MAJOR-1).
debate_tool_calls: tuple[ToolCall, ...] = ()
#: What a human answered about each candidate the validator accepted, in order (MAJOR-2).
#: Built FROM the caller-owned sink ``run_project`` hands to ``generate_via_llm``, never
#: accumulated beside it: two containers holding one fact drift (kø-(p)), and a drifted pair
#: would let this result and the written artefact describe different runs.
#:
#: KEYED per approach, unlike ``refinements`` above — a human wrote these words about ONE
#: specific candidate, and an artefact that cannot say which is one nobody can act on. EMPTY
#: is an honest POSITIVE statement ("nobody was asked, or nothing validated"), so it defaults,
#: exactly as ``skipped_links`` does.
#:
#: Carried HERE and on neither other carrier: ``ProvenanceStamp`` describes the gate that
#: judged ONE candidate, and ``DryRunReport`` returns above generation entirely.
expert_revisions: tuple[ProposalReview, ...] = ()
#: The cut this run was GIVEN, when it was given one (order 20260907T080223Z): the base's ref,
#: the question it was computed for, the three denominators and the withheld rules by count.
#:
#: DEFAULTED, unlike ``cost_baseline_anchored``, and the difference is the one that row states:
#: both of that field's possible defaults would assert something about an event, whereas
#: ``None`` here is the true statement "no payload was supplied" — and there is exactly one way
#: to supply one. This is ``skipped_links``' half of the rule.
#:
#: Carried HERE and not on ``ProvenanceStamp``: the stamp describes the gate that judged ONE
#: candidate, while this is a RUN-level fact about what the run was allowed to read at all.
prepass: prepass.PrepassDeclaration | None = None
#: What this run's DELIVERED input could ground an ``affected_item`` code in (P8): the size of
#: the text P7's gate searched, how many distinct identifiers of a measured form it carries,
#: and how many cost lines the run can anchor one of them AS.
#:
#: A RUN-level fact settled ONCE, before any candidate exists, which is why it is here and not
#: on ``ProvenanceStamp`` (that describes the gate that judged ONE candidate) — ``skipped_links``'
#: own placement rule. DEFAULTED, ``prepass``' half of the rule: ``None`` is the true statement
#: "no measurement was made", and there is exactly one place that makes it.
grounding_offer: GroundingOffer | None = None
@property
def verdict_key(self) -> str:
"""The id an expert verdict on THIS run's candidate will arrive under — always available,
including on a run nobody has reviewed. DERIVED from the candidate (never from a decision),
which is exactly what ``verdicts.verdict_key`` exists for, so 'no verdict' costs no
traceability: the outbox artefact and the hosted response can still name the key the honest
Step-7 inbox channel will join back on. A PROPERTY rather than a stored field because a
second copy of a keying rule is the ``(p)`` defect — and because a defaulted field would
have to state a value for a fact that is always derivable."""
return verdict_key(_features_of(self.outcome.proposal))
@dataclass(frozen=True)
class RunFailure:
"""One project that RAISED during a portfolio pass (S3.3, SC3 collect-and-continue).
A DISTINCT type from ``RunResult`` rather than an error field on it, and deliberately so: the
session spec's wording was "a ``RunResult`` slot with an error field", but ``RunResult`` is
frozen with six required non-defaulted fields (``:89-94``) — a run that raised before producing
an outcome has no honest value for ``outcome``, ``provenance`` or ``verdict``. Filling them with
dummies would put FABRICATED provenance into the aggregate, which is the failure mode this
repo's provenance rules exist to prevent. The exception is recorded as text (``error``) plus its
class name (``error_type``) rather than the live exception object, so a ``PortfolioResult``
stays a plain frozen value with no traceback frames held alive."""
project_id: str
error: str
error_type: str
@dataclass(frozen=True)
class DryRunReport:
"""S4.2 offline ``--live-dry-run`` outcome (comparison protocol §4 pkt 3): everything a real run
would use — profile, the resolved model-id per BUILT role, and the round/token parameters —
captured WITHOUT a model call. A DISTINCT type from ``RunResult``, whose post-generation fields
(outcome/provenance/verdict) do not exist yet on a run that stopped before the first model call."""
profile: str
resolved_models: dict[str, str]
max_rounds: int
max_tokens: int
top_k: int
#: Whether a REAL run of this configuration would have its deterministic gate anchored to the
#: project's own cost lines (see ``ProvenanceStamp.cost_baseline_anchored``). Carried here too
#: because a dry run stops before any proposal exists, so there is no stamp to read it off —
#: and this surface is precisely where the un-anchored case was measured to be silent.
cost_baseline_anchored: bool
#: Which knowledge base a real run would judge, and how its identity was established
#: (``ProvenanceStamp.bundle_id_source``). Carried here for the same reason
#: ``cost_baseline_anchored`` is: a dry run stops before any stamp exists, and this surface is
#: precisely where a mount that disagrees with the declaration would otherwise be silent.
#: ``None`` on the road path.
bundle_id_source: okf.ResolvedBundleId | None
#: Every cross-link the bundle navigation could not follow (``okf.Bundle.skipped``). EMPTY is a
#: positive statement — "every cross-link was followed" — which is why it DEFAULTS, unlike
#: ``cost_baseline_anchored`` above: a missing bool would have to claim something about an event
#: (and both claims would sometimes be false), while a missing trace asserts only that the event
#: list is empty. The road path navigates no bundle, so empty is literally true there too.
skipped_links: tuple[okf.SkippedLink, ...] = ()
#: The cut a dry run was given, when it was given one. Carried here for ``skipped_links``'
#: reason and NOT for ``unkeyed_verdicts``' one: the dry-run cut returns BELOW the fork that
#: resolves a payload, so a dry run can honestly report what it would have read — whereas a
#: field resolved above that cut could only ever report zero.
prepass: prepass.PrepassDeclaration | None = None
#: What the delivered input of a REAL run of this configuration could ground an
#: ``affected_item`` code in (P8). Carried here for ``cost_baseline_anchored``'s reason and,
#: more sharply, because this is the surface on which "before it spends its three attempts"
#: is provable at all: the dry-run cut returns before the first model call, so a dry run that
#: reports a null offer has said the run cannot succeed WITHOUT paying to find out.
grounding_offer: GroundingOffer | None = None
@dataclass(frozen=True)
class GoalReached:
"""A savings-goal signal VALUE (Step 8, SC6) — NOT an exception. Structured like
``BudgetExceeded`` (``budget.py:22-34``) but semantically SUCCESS (the goal was reached), not
resource exhaustion (H1). Used as a ``stop_reason`` value + a loop ``break``, never ``raise``d.
``scope`` is ``"portfolio"`` (the whole pass) or ``"project"`` (one pid); ``limit_ore`` is the
threshold that was met, ``observed_ore`` the accumulated realized sum that met it (``>=``)."""
scope: Literal["project", "portfolio"]
project_id: str | None
limit_ore: int
observed_ore: int
@dataclass(frozen=True)
class BudgetStop:
"""A GLOBAL token-cap stop signal VALUE (S3.4/F10) — NOT an exception, and NOT a goal.
Structured like ``GoalReached`` and carried the same way (a ``stop_reason``-shaped value plus a
loop ``break``), but it is kept as its OWN field rather than widening ``stop_reason``: the two
stops mean opposite things. A goal-stop is success (the savings target was met); this is
resource exhaustion (the pass ran out of tokens). Folding them into one field would let a
caller read "we stopped" without being able to tell which happened.
``required_tokens`` is what one more run would have needed; ``remaining_tokens`` is what was
actually left. Both are recorded because their DIFFERENCE is the operator's next decision."""
limit_tokens: int
spent_tokens: int
remaining_tokens: int
required_tokens: int
@dataclass(frozen=True)
class PortfolioResult:
"""The outcome of a sequential fan-out over N projects (SC2).
``runs`` is one ``RunResult`` per project in input order; ``store`` is the ONE shared
``VerdictStore`` threaded across every run (the cross-project ExpeL learning loop).
The remaining fields are a thin aggregate over ``runs``: ``validated_count`` /
``rejected_count`` partition the outcomes; ``sum_claimed_saving_nok`` totals the claimed
saving of the validated proposals only; ``sum_token_usage`` totals every run's
provenance token usage. ``stopped_early`` / ``stop_reason`` record a Step-8 goal-stop,
``failures`` records the projects that RAISED (S3.3 collect-and-continue), and ``budget_stop``
records a S3.4 global-token-cap stop (which also sets ``stopped_early``, but is kept apart from
``stop_reason`` because exhaustion is not success): all four default so the frozen aggregate and
every existing constructor call are unaffected.
``runs`` and ``failures`` PARTITION the projects that were actually submitted — a project
appears in exactly one of them, never both, and the counts do not overlap. ``validated_count`` /
``rejected_count`` therefore total ``len(runs)``, not the portfolio size: a failure is neither a
validation nor a rejection, and folding it into either would misreport the pass."""
runs: tuple[RunResult, ...]
store: VerdictStore
validated_count: int
rejected_count: int
sum_claimed_saving_nok: float
sum_token_usage: int
stopped_early: bool = False
stop_reason: GoalReached | None = None
failures: tuple[RunFailure, ...] = ()
budget_stop: BudgetStop | None = None
def _coverage_row(
row_id: str, label: str, outcome: ValidatedProposal | Rejection
) -> ApproachOutcome:
"""One coverage row from one evaluated approach. A rejection carries the validator's reason
verbatim — a bare status would tell the expert their approach failed without telling them why,
which is the part they can actually act on."""
if isinstance(outcome, ValidatedProposal):
return ApproachOutcome(
id=row_id,
label=label,
status="validated",
saving_nok=outcome.proposal.claimed_saving_nok,
)
return ApproachOutcome(id=row_id, label=label, status="rejected", detail=outcome.reason)
def evaluate_mandate_candidates(
mandate: Mandate, *, bundle_dir: str, project_id: str
) -> tuple[ApproachOutcome, ...]:
"""Judge every commissioned approach as a DETERMINISTIC candidate, built from the commission and
the knowledge base's own priced schedule — no model anywhere (S7b).
The use case this serves is "documents + a concrete task -> a judged proposal". Until now the
only candidate source was ``generate_via_llm``, and the bundle arm additionally required a
hand-written ``validator-input.json`` for the project's identity — so an ingested tender corpus
could be navigated and never run (measured: ``docs/2026-09-03-forslag-fra-mandat.md``). This is
the candidate source; making that projection optional is the OTHER seam and is not built here.
**SYNC, and that is the design rather than an omission.** A sync function cannot await a chat
call, so "this path makes no model calls" is a property of its type instead of a promise its body
has to keep. No mutation of the body can quietly reintroduce one.
**Nothing here is re-implemented.** Routing is ``route_by_bundle`` against the base's DECLARED id
(S7a-3, so a base delivered under a directory name of its own routes as itself); the baseline is
``okf.derive_cost_baseline``; the judgement is ``validate_proposal`` with that same baseline, so a
commissioned candidate gets **no discount at the deterministic gate** — exactly the rule the LLM
path states for ``approach``. The coverage rows are ``_coverage_row``'s.
**Every candidate is built BEFORE any is judged.** A commission that cannot be executed as
written is refused whole rather than settled in part (``load_mandate``'s rule): a partial
settlement would describe work nobody ordered. It also keeps the refusal ahead of the work, which
is the økt-57 hoist applied to CBC solves rather than to model calls.
**``allow_own_proposals`` gets a ``not_evaluated`` row, not a refusal.** A run's own proposal
needs a model and this path has none, so the row cannot be filled — but omitting it would make it
indistinguishable from an approach nobody commissioned, which is the silence ``ApproachOutcome``
exists to remove. Refusing the whole run would be wrong the other way: the field defaults to
``True``, so every mandate written before today carries it.
:raises MandateRoutingError: the commission names a base this run was not given.
:raises MandateCandidateError: an approach carries no estimate, no codes, or an unknown code.
:raises okf.CostBaselineDerivationError: the bundle's schedule cannot be derived from (an
unpriced schedule refuses in full — MAJOR-4's rule, propagated rather than routed around).
"""
bundle = okf.navigate_bundle(bundle_dir)
okf.assert_declared_ids_agree(bundle)
declared = okf.reconcile_bundle_id(bundle_dir).id
routed = route_by_bundle(mandate, [declared])
_, scoped = routed[0]
baseline = okf.derive_cost_baseline(bundle, project_id=project_id)
candidates = [
(approach, candidate_from_approach(approach, baseline=baseline, project_id=project_id))
for approach in scoped.approaches
]
rows = [
_coverage_row(approach.id, approach.label, validate_proposal(candidate, baseline=baseline))
for approach, candidate in candidates
]
if scoped.allow_own_proposals:
rows.append(
ApproachOutcome(
id=OWN_PROPOSAL_ID,
label="the system's own proposal",
status="not_evaluated",
detail=(
"this path builds candidates from the commission alone, so there is no model "
"to originate one"
),
)
)
return tuple(rows)
def _select_outcome(
produced: list[tuple[int, ValidatedProposal | Rejection]],
) -> ValidatedProposal | Rejection:
"""Pick the ONE outcome a ``RunResult`` carries out of everything the mandate produced.
``RunResult`` stays single-outcome on purpose: portfolio aggregation, the outbox artefacts and
the HITL verdict keying all rest on there being exactly one. The choice is deterministic —
highest validated saving, ties broken by mandate order — so it can never depend on which
approach happened to run last. When nothing validated, the FIRST rejection stands, which keeps
a fully-rejected mandate reporting a typed ``Rejection`` rather than a fabricated success.
"""
validated = [(i, o) for i, o in produced if isinstance(o, ValidatedProposal)]
if validated:
return min(validated, key=lambda t: (-t[1].proposal.claimed_saving_nok, t[0]))[1]
return produced[0][1]
async def _evaluate_mandate(
mandate: Mandate,
evaluate: Callable[[Approach | None], Awaitable[ValidatedProposal | Rejection]],
budget_stops: list[str] | None = None,
) -> tuple[
ValidatedProposal | Rejection,
tuple[ApproachOutcome, ...],
tuple[tuple[str, ValidatedProposal | Rejection], ...],
]:
"""Evaluate every commissioned approach, then the run's own proposal when allowed, and report
what became of each (Trekk A3/A4).
Returns the selected outcome, the coverage report, and — third — every EVALUATED approach's own
outcome paired with its id (A5), which is what lets each of them be written as a judgeable
outbox artefact. It is returned alongside rather than folded into ``ApproachOutcome`` on
purpose: ``mandate.py`` imports only ``pydantic`` + stdlib to stay D7-portable (guarded by
``test_okf_is_maf_free``), and hanging a ``ValidatedProposal`` off a coverage row would drag
``validator`` — and with it ``pulp`` — into that deliberately thin module. Rows the run never
reached are absent here by construction: a ``not_evaluated`` approach has no proposal, so there
is nothing to write and nothing to judge.
Budget exhaustion mid-list is REPORTED, not swallowed: the approaches that were never reached
become ``not_evaluated`` rows. But if the very first approach exhausts the budget there is
nothing honest to return, so ``BudgetExceeded`` propagates exactly as it did before — a run
that produced nothing must still fail loudly rather than hand back an empty report.
"""
plan: list[tuple[str, str, Approach | None]] = [(a.id, a.label, a) for a in mandate.approaches]
if mandate.allow_own_proposals:
plan.append((OWN_PROPOSAL_ID, "the system's own proposal", None))
rows: list[ApproachOutcome] = []
produced: list[tuple[int, ValidatedProposal | Rejection]] = []
evaluated: list[tuple[str, ValidatedProposal | Rejection]] = []
for index, (row_id, label, approach) in enumerate(plan):
try:
outcome = await evaluate(approach)
except BudgetExceeded as stop:
if not produced:
raise
# P19 D2: WHICH cap bound, recorded on a caller-owned sink before the rows are built.
# The exception is SWALLOWED here (the approaches that were reached are a real result),
# so ``run_project``'s own ``in_flight`` never sees it — and a coverage artefact that
# said "nothing stopped this run" about a commission cut in half would be the silence
# the artefact exists to remove.
if budget_stops is not None:
budget_stops.append(stop.kind)
rows.extend(
ApproachOutcome(
id=rid,
label=lbl,
status="not_evaluated",
detail="budget exhausted before this approach was evaluated",
)
for rid, lbl, _ in plan[index:]
)
break
produced.append((index, outcome))
evaluated.append((row_id, outcome))
rows.append(_coverage_row(row_id, label, outcome))
return _select_outcome(produced), tuple(rows), tuple(evaluated)
def _authored_texts(result: Any, name: str) -> list[str]:
"""The texts of ``get_outputs()`` entries authored by participant ``name`` (proposer/checker),
in surfaced order. MAF surfaces ``author_name`` on each output's ``messages`` — NOT on the
``AgentResponse`` itself (verified against 1.9.0) — so we match through ``messages``. This
separates the proposer's converged output (fed to generation, F1) from the checker's gate
verdict (Step 3/4); the orchestrator's termination notice is authored by neither, so it is
excluded automatically."""
texts: list[str] = []
for out in result.get_outputs():
if not any(getattr(m, "author_name", None) == name for m in getattr(out, "messages", [])):
continue
text = out if isinstance(out, str) else getattr(out, "text", None)
if text:
texts.append(text)
return texts
#: The pointer's shape is fixed text plus the base id, its document count and (when scoped) the
#: dimension — O(1) in the corpus by construction. The ceiling that guards it lives in the TEST
#: (``test_debate_navigation_cost_loadbearing``), for ``_CATALOGUE_EXCERPT_CHARS``' reason: a bound
#: imported from the implementation moves with it, and widening this is the regression the gate
#: exists to catch.
def _bundle_pointer(bundle: okf.Bundle, bundle_id: str, *, dimension: str | None = None) -> str:
"""What the debate is told about the knowledge base INSTEAD of being given it (S2c).
It must do exactly two things: NAME the base by the id the tools take — a bounded prompt that
omits it is a debate that cannot make a single call, which is this seam's vacuous form — and
say the ladder exists. It carries no content: the whole point is that the corpus is read on
demand and each result rides only from the call that asked for it.
The document COUNT is the price signal a proposer chooses against (``directory_listing``'s own
``chars`` rule, one rung up), and it is the count IN SCOPE: under a dimension, advertising
documents the tools will then refuse would be a number that describes a different run.
"""
documents = sum(1 for f in bundle.context_files if okf.in_dimension(f, dimension))
scope = (
f" Scope: dimension {dimension!r} — only knowledge in that scope is readable."
if dimension
else ""
)
return (
f"Knowledge base: {bundle_id} ({documents} concept documents)."
f"{scope}\n"
"It is NOT included here — read it with your tools: list_bundles() for the bases, "
f"read_bundle({bundle_id!r}) for its top level, read_dir({bundle_id!r}, path) for one "
f"directory, read_file({bundle_id!r}, path) for one document. Open what you need.\n"
"A listing is a WINDOW: it reports 'total' for the level and gives you 'limit' entries "
"from 'offset'. When 'total' is large, do not page through it — narrow it: "
f"read_dir({bundle_id!r}, path, filter='<word>') answers with the entries whose title, "
"requirement number or path contains that word, and reports 'total_matches'.\n"
"Before you settle on a measure, name the ONE requirement of this base that BINDS it: "
"find it with a filter, read it with read_file, then call "
f"declare_requirement({bundle_id!r}, path, ref) with the requirement's own number. A "
"declaration naming a document this run never opened is refused; reading it is the fix."
)
def _debate_text(result: Any) -> str:
"""The PROPOSER's converged output (fed into generation, F1). With ``output_from=agents`` both
participants surface, so we select proposer-authored outputs specifically — taking the last of
ALL surfaced outputs would feed the checker's verdict to generation at even round counts.
Returns ``""`` when the proposer produced no surfaced text."""
proposer_texts = _authored_texts(result, "proposer")
return proposer_texts[-1] if proposer_texts else ""
def _checker_verdict(result: Any) -> tuple[str, str]:
"""Parse the checker's gate verdict from its surfaced debate output (Step 3/4, målbilde §2/§6).
Returns ``(decision, reason)``: ``"reject"`` ONLY on an explicit ``VERDICT: REJECT`` (with the
trailing reason), ``"approve"`` on an explicit ``VERDICT: APPROVE``, else ``"absent"``. The gate
is opt-in-reject (fail-open): a missing/unparseable marker never blocks, so the deterministic
validator stays the sole gate on those runs."""
checker_texts = _authored_texts(result, "checker")
text = checker_texts[-1] if checker_texts else ""
upper = text.upper()
if "VERDICT: REJECT" in upper:
reason = text[upper.index("VERDICT: REJECT") + len("VERDICT: REJECT") :]
return "reject", reason.lstrip(" -:—").strip()
if "VERDICT: APPROVE" in upper:
return "approve", ""
return "absent", ""
def _project_by_id(project_id: str) -> Project:
for project in load_reference_projects():
if project.id == project_id:
return project
raise ValueError(f"unknown project_id: {project_id!r}")
def _project_from_bundle(
bundle_dir: str, project_id: str, *, bundle: okf.Bundle | None = None
) -> Project:
"""Derive a minimal ``Project`` from an OKF bundle (so a bundle the loop runs need NOT be a
road reference-domain project). Only ``id`` + ``name`` reach the generation prompt
(``generate._build_messages``), so ``cost_items`` is empty and ``verdict_input`` is left unset
here (the Layer-2 decision flows via ``run_project``'s own ``verdict_input`` argument).
**The IR projection is OPTIONAL, and the tolerance stops at absence** (S7b søm 1). A base that
HAS one must still agree with the requested id — that divergence guard is the existing contract
multi-base dispatch rests on ("den eksisterende fail-fasten blir rutingsnøkkelen"), so loosening
absence must not loosen disagreement. A base WITHOUT one has nothing to check against, and an
ingested corpus is exactly that case.
**The name is unaffected, which is measured rather than assumed.** ``SavingsProposal`` has no
name field, so the projection has never been a name source: ``Project.name`` comes from the
``type: project`` concept's ``title``, with the requested id as the last resort — before and
after this change alike.
``bundle`` reuses an already-navigated bundle to avoid a second navigation."""
ir = okf.load_optional_ir_projection(bundle_dir)
if ir is not None and ir["project_id"] != project_id:
raise ValueError(f"bundle project_id {ir['project_id']!r} != requested {project_id!r}")
nav = bundle if bundle is not None else okf.navigate_bundle(bundle_dir)
project_file = next((f for f in nav.files if f.type == "project"), None)
name = (
project_file.frontmatter.get("title", project_id).strip('"')
if project_file is not None
else project_id
)
return Project(
id=project_id,
name=name,
description="",
currency="NOK",
cost_items=(),
docs_dir=bundle_dir,
)
def _verdict_input_from_args(args: Any) -> dict[str, str] | None:
"""The CLI's verdict, or ``None`` when the operator recorded none (F2). ``main`` has already
refused the half-given case by name, so both flags are set together or neither is. ``args`` is
typed ``Any`` because ``argparse`` is imported inside ``_build_parser``, not at module scope."""
if args.decision is None:
return None
return {"decision": args.decision, "rationale": args.rationale}
def verdict_notice(result: RunResult) -> str:
"""The ONE renderer for a run's verdict identity on stdout (F2). Present: the unchanged
``verdict id=…, decision=…`` — read off the run's OWN captured verdict rather than off argv, so
stdout and the store cannot disagree about what was recorded (the ``cost_baseline_notice``
precedent). Absent: it SAYS so, and names the key an expert verdict on this candidate would
arrive under — the operator's join back into the honest Step-7 inbox channel. Not an omission
like the ``*_notice`` renderers above: those describe an event that may not have happened,
whereas every run has a verdict identity to report, and a blank there would read as a missing
line rather than as 'nobody reviewed this'."""
if result.verdict is None:
return f"no expert verdict given; verdict key={result.verdict_key}"
return f"verdict id={result.verdict.id}, decision={result.verdict.decision}"
def _features_of(proposal: SavingsProposal) -> ProposalFeatures:
return ProposalFeatures(
affected_codes=frozenset(item.code for item in proposal.affected_items),
measure_type=proposal.measure,
claimed_saving_nok=proposal.claimed_saving_nok,
description=proposal.measure,
)
def _write_or_report(
write: Callable[[], object], *, what: str, in_flight: BaseException | None
) -> None:
"""Run ONE ``finally`` writer so a disk error can never DISPLACE the run's stop reason.
Both artefacts below are written from a ``finally``, which is what makes them survive the run
that most needs them — and is also what put them in the propagation path of the very exception
they are evidence for. An ``OSError`` raised there REPLACES the in-flight ``BudgetExceeded`` or
``ProposalReviewInputError``: neither the CLI's ``except ProposalReviewInputError`` nor its
refusal tuple catches ``OSError``, so the operator got a traceback and the reason the run
stopped was gone.
The guard is CONDITIONAL, never a blanket ``except``: with nothing in flight there is no stop
reason to protect, and a run that could not write its outbox must say so by failing —
downgrading that to a clean return reports a success the run cannot evidence. Either way the
failure is SAID, because a missing artefact would otherwise read as a run that had nothing to
record, which is exactly the distinction T10/T11 exist to keep.
ONE helper for two call sites (kø-(p)): a rule about what a writer may do to an in-flight
exception, copied, is a rule that ends up applied to only one of them."""
import sys # deferred exactly as ``main()`` does — this module keeps ``sys`` off its top level
try:
write()
except OSError as exc:
print(f"run warning: could not write {what}: {exc}", file=sys.stderr)
if in_flight is None:
raise
def _default_factory(profile: Profile | str) -> Callable[[str], BaseChatClient]:
def factory(role: str) -> BaseChatClient:
return get_backend(profile).create_chat_client(model=resolve_model(profile, role))
return factory
class UnanchoredRunRefused(ValueError):
"""A run was required to be anchored (F4) and the bundle offered no cost baseline.
Measured live (``docs/2026-09-07-syretest-s7-prepass-k2.md`` § 8): all three paid arms stamped
``cost_baseline_anchored: False``, so the validator's stage 0 — the one stage that tells a
fabricated cost line from a real one — was skipped, and each arm invented its codes
(``ENGRAVE_MARK``, ``RITB-HOURS``, ``Material_Cost_Concrete``). The run SAID so on stdout
(``cost_baseline_notice``), so this was never a silence; what visibility cannot do is stop a
machine-readable artefact reading ``validator_decision: validated`` over lines nothing anchored.
OPT-IN, never a default: a bundle written before the S4.0 amendment — every commons-owned
golden — is legitimately un-anchored, and making the requirement the default would refuse them
all. A caller who needs the guarantee asks for it by name and composes it with
``--derive-cost-baseline`` when the base carries a priced schedule instead of the file.
A ``ValueError``, the ``BundleIdMismatch``/``CostBaselineDerivationError`` precedent: an argv
that is wrong about what this base can offer belongs on the CLI's refusal tuple and hosting's
400 arm, never on the crash channel.
"""
#: The one line a run prints about its own anchoring. Rendered ONLY when the run is un-anchored:
#: an anchored run has nothing to warn about, and ``mandate.announce``'s rule is that a line for
#: something the run does not have is OMITTED rather than rendered empty.
_UNANCHORED_NOTICE = (
" Cost baseline: NONE in the bundle — this run is un-anchored: the validator's stage 0 "
"(reconciling each proposed cost line against the project's own) is SKIPPED"
)
def cost_baseline_notice(anchored: bool) -> str | None:
"""Render the un-anchored notice, or ``None`` when the run IS anchored.
ONE renderer with N callsites, never N copies of the wording (kø-(p)) — and it takes the
already-resolved BOOLEAN rather than a bundle path, so the printed line and the machine-readable
``ProvenanceStamp.cost_baseline_anchored`` can never disagree: both descend from the single
``okf.load_optional_cost_baseline`` call inside ``run_project``. A renderer that re-read the
bundle would be a second resolution of the same rule, free to drift from the run it describes.
**Not folded into ``mandate.announce``, and that is a measurement rather than a preference:**
``announce`` is printed only when ``--mandate`` is given, so the runs this notice exists for —
the bare bundle dry-runs that exit 0 in silence — would still say nothing. It also renders
BEFORE ``run_project`` is called, i.e. before anyone has resolved the baseline; putting the line
there would have required ``main`` to open the bundle itself.
English, like every other line this CLI prints; the Norwegian explanation of what an un-anchored
run means belongs in ``docs/kunnskapsbase-for-en-kjoring.md``, next to the domain expert.
Tense-neutral on purpose ("is SKIPPED"): the same string serves ``--live-dry-run`` (where the
run has not happened) and a completed run (where it has)."""
return None if anchored else _UNANCHORED_NOTICE
def grounding_offer_notice(offer: GroundingOffer | None) -> str | None:
"""Render the one line that says what this run's delivered input can ground, or ``None`` when
there is nothing to warn about (P8).
ONE renderer with N callsites, never N copies of the wording (kø-(p)), taking the
ALREADY-MEASURED value rather than a text: a renderer that re-composed the grounding would be
a second resolution of the same rule, free to drift from the run it describes
(``cost_baseline_notice``'s rule).
``None`` when the run CAN anchor a cost line — omission, never an empty row
(``mandate.announce``'s rule) — and ``None`` on ``None``, which is the honest reading of "no
measurement was made". This is deliberately NOT ``proposal_review_notice``'s deviation: there,
silence on zero was ambiguous; here, a run that can anchor its lines has nothing to report that
the outcome does not already say.
BOTH numbers, because the pair is the diagnosis. "0 cost lines" alone reads as a restatement of
``cost_baseline_notice``; "50 identifiers" alone reads as good news. Together they say the
thing P8 measured: the input offers plenty to cite and nothing to cost, while the proposer
prompt asks for a cost line.
English, like every other line this CLI prints."""
if offer is None or offer.cost_lines > 0:
return None
return (
f" Grounding offer: {offer.identifiers} distinct identifier(s) and "
f"{offer.cost_lines} cost line(s) in the {offer.chars} characters this run was given — "
"the proposer is asked to restate a cost line this input does not carry, so every "
"candidate it invents will be refused as ungrounded"
)
def bundle_id_notice(resolved: okf.ResolvedBundleId | None) -> str | None:
"""Render the one line that says a base was mounted under a name it does not answer to, or
``None`` when there is nothing to say.
The warning half of the S7a-3 slacken. A declared id that disagrees with its directory is no
longer refused — it is a filesystem accident, and refusing it made the first delivered corpus
that declares its own id unopenable — but it must not become SILENT either: every artefact the
run stamps names the declared id, while the operator typed the mount.
ONE renderer with N callsites, never N copies of the wording (kø-(p)), and it takes the
ALREADY-RESOLVED value rather than a bundle path: a renderer that re-read the base would be a
second resolution of the same rule, free to drift from the run it describes
(``cost_baseline_notice``'s rule, and the reason ``ResolvedBundleId`` carries the mount).
``None`` on agreement AND on ``None`` — omission, never an empty row (``mandate.announce``'s
rule). A run with no knowledge base has no identity to disagree about.
Both names are printed. A warning that says only "mismatch" leaves the operator to go and look
for the two values it is warning about."""
if resolved is None or resolved.id == resolved.mount:
return None
return (
f" Knowledge base: declares bundle_id {resolved.id!r} (source: {resolved.origin}) but is "
f"mounted as {resolved.mount!r} — the DECLARED id is the identity, so every artefact this "
f"run stamps names {resolved.id!r}"
)
def collision_notice(collisions: tuple[VerdictCollision, ...]) -> str | None:
"""Render which candidates two bases both described, or ``None`` when none did.
``None`` on an empty tuple — omission, never an empty row (``mandate.announce``'s rule, as
``skipped_links_notice`` and ``cost_baseline_notice`` already follow). A dispatch where every
candidate belonged to exactly one base has nothing to report.
ONE renderer, taking the ALREADY-RESOLVED trace rather than a store to re-scan: a renderer that
recomputed the collisions would be a second resolution of the same fact, free to disagree with
the dispatch it describes (kø-(p)).
**Surface, stated plainly:** ``MultiBaseResult`` has no production caller today, so this notice
is library-facing. It is written now because the field would otherwise be a value nothing can
display — the same principle Step 4 applies one level down: a signal that was not stored must
not become an asserted absent one.
"""
if not collisions:
return None
lines = ["Cross-base candidates (one verdict each; the later base's was dropped):"]
lines.extend(
f" - {c.verdict_id}: first from {c.first_bundle_id!r}, again from {c.second_bundle_id!r}"
for c in collisions
)
return "\n".join(lines)
def unkeyed_verdicts_notice(unkeyed: int) -> str | None:
"""Render the prior verdicts that could NOT reach the hypothesis prompt, or ``None`` when every
one of them could (S7b søm 1).
The measured silence this closes: making the IR projection optional lets an ingested corpus run
the whole loop, but it also removes the pre-hypothesis candidate the Step-1 ExpeL fold is keyed
on. Without a line here, a base holding a dozen prior expert judgements would run and simply not
use them — indistinguishable, on stdout and in the artefact alike, from a base that had never
been judged at all. That is the same class of silence ``skipped_links_notice`` exists for.
ONE renderer, taking the already-resolved COUNT rather than a store or a bundle path: a renderer
that re-read either would be a second resolution of the run's own fold, free to disagree with it.
``None`` at zero — omission, never an empty row (``mandate.announce``'s rule, the one
``cost_baseline_notice`` and ``skipped_links_notice`` both follow). Zero is the honest positive
statement, and a run that folded everything has nothing to report."""
if unkeyed <= 0:
return None
plural = "" if unkeyed == 1 else "s"
return (
f" Knowledge base: {unkeyed} prior expert verdict{plural} NOT folded into the hypothesis "
"prompt — the base declares no IR projection (validator-input.json), so there is no "
"candidate to key retrieval against"
)
def prepass_notice(declaration: prepass.PrepassDeclaration | None) -> str | None:
"""Render the CUT this run was given, or ``None`` when it was given none.
ONE renderer, N callsites (kø-(p)), taking the ALREADY-RESOLVED declaration rather than a
payload path: a renderer that re-read the file would be a second resolution free to disagree
with the run it describes.
``None`` without a payload — omission, never an empty row (``mandate.announce``'s rule, which
``cost_baseline_notice`` and ``skipped_links_notice`` both follow). Here the omission is
unambiguous in a way ``proposal_review_notice``'s deliberately is not: a run says nothing about
a cut because the operator supplied no payload, and there is exactly one way to supply one. The
golden transcript is an independent, pre-existing witness for that half — a renderer that
always returned a line would print in the demo and go red there.
The withheld concepts appear as rule -> COUNT, never as ids: the same rule the rendering
follows, for the same measured reason (34 451 o200k tokens of ids on a real corpus), and
because an operator reading a run summary wants to know WHAT was dropped and HOW MUCH, not
which. The full list is in the payload the operator already holds."""
if declaration is None:
return None
rules = ", ".join(f"{rule} ({count})" for rule, count in declaration.withheld_rules)
# ONE renderer, two arms — because the number of concepts delivered says nothing at all about
# whether the run could still open the rest, and an operator reading a summary that reported
# "8 of 630" without that would draw the wrong conclusion in exactly one of the two cases.
role = (
"a DECLARED CUT SEEDED the exploration (the rest of the base stayed reachable with the "
"navigation tools)"
if declaration.rest_reachable
else "a DECLARED CUT was used"
)
return (
f" Knowledge base: {role} — {declaration.delivered} of "
f"{declaration.considered} concept(s) delivered, {declaration.withheld} withheld"
f"{' by rule: ' + rules if rules else ''}. "
f"Base {declaration.bundle_id} at ref {declaration.ref}; "
f"cut computed for: {declaration.question}"
)
def skipped_links_notice(skipped: tuple[okf.SkippedLink, ...]) -> str | None:
"""Render what the run could NOT read, or ``None`` when every cross-link was followed.
The measured silence this closes: ``okf._walk`` tolerates an unfollowable link exactly as OKF
SPEC §4 requires (skip, never raise) — correct, and unchanged here — but it left no trace, so a
knowledge base whose other half was never reached looked identical to one where those documents
were never written, and ``--live-dry-run`` exited 0 over both.
ONE renderer with N callsites, never N copies of the wording (kø-(p)), and it takes the
already-resolved trace rather than a bundle path: a renderer that re-navigated the bundle would
be a second resolution of the same walk, free to disagree with the run it describes. Both
callsites read it off the value ``run_project`` returned from its ONE
``okf.navigate_bundle`` call.
``None`` when the trace is empty — omission, never an empty row (``mandate.announce``'s rule,
the same one ``cost_baseline_notice`` follows). A run that reached everything has nothing to
report.
The per-link line prints the reason TOKEN itself rather than a prose translation of it: a second
display vocabulary keyed off ``SkipReason`` would be the duplicate free to drift, and the token
is already the operative word ("missing" vs "outside-bundle"). English, like every other line
this CLI prints; the Norwegian explanation belongs in
``docs/kunnskapsbase-for-en-kjoring.md``, next to the domain expert."""
if not skipped:
return None
lines = [
f" Knowledge base: {len(skipped)} cross-link(s) NOT followed — "
"the agents never read the document(s) behind them:"
]
lines += [f" - {s.from_file} -> {s.target} ({s.reason})" for s in skipped]
return "\n".join(lines)
async def run_project(
project_id: str,
profile: Profile | str = Profile.LOCAL,
*,
docs_dir: str,
verdict_input: dict[str, str] | None = None,
bundle_dir: str | None = None,
#: Derive the validator's cost baseline from a priced schedule IN the bundle
#: (``okf.derive_cost_baseline``) instead of loading a hand-written ``cost-baseline.json``.
#: Bundle path only, and OPT-IN by construction: the default leaves every existing run on the
#: file loader, byte-identically.
derive_cost_baseline: bool = False,
require_cost_baseline: bool = False,
dimension: Dimension | None = None,
store: VerdictStore | None = None,
verdict_dir: str | None = None,
outbox_dir: str | None = None,
run_id: str | None = None,
client_factory: Callable[[str], BaseChatClient] | None = None,
max_rounds: int = _DEFAULT_MAX_ROUNDS,
max_tokens: int = _DEFAULT_MAX_TOKENS,
top_k: int = 3,
enable_layer1_hitl: bool = False,
notify: Callable[[Verdict], None] | None = None,
meter: TokenMeter | None = None,
live_dry_run: bool = False,
semantic_retrieval: bool = False,
embedder: Embedder | None = None,
mandate: Mandate | None = None,
mcp_servers: tuple[McpServerConfig, ...] = (),
#: MAJOR-2: the synchronous HITL door onto a candidate the deterministic validator has just
#: ACCEPTED. ``None`` (the default) is byte-identical to a pre-MAJOR-2 run — same prompts, same
#: outbox files, same golden transcript. Given one, it is called once per validated attempt of
#: every commissioned approach, and a ``revise`` buys ONE more attempt out of the budget the
#: loop already has. It mints no verdict and gates nothing (F2).
proposal_reviewer: ProposalReviewer | None = None,
#: A verified OKF consumption pre-pass payload (order 20260907T080223Z). Bundle path only, and
#: OPT-IN by construction: ``None`` (the default) leaves the debate on ``_bundle_pointer``'s
#: pointer and the four navigator tools, byte-identically. Given one, the debate is handed a
#: DECLARED CUT and the tools are withdrawn — a debate holding both would be free to walk
#: around the cut it just declared, which is the undeclared cut in a costume (contract SS 2.2).
#: A LOADED object, never a path: the library seam takes the validated artefact and the CLI
#: owns the file, exactly as ``mandate=`` and ``dimension=`` already do.
prepass_payload: prepass.PrepassPayload | None = None,
) -> RunResult | DryRunReport:
"""Run the vertical slice for ONE project. ``client_factory`` is the test-injection seam
(defaults to the real backend). ``verdict_input`` carries the expert decision/rationale
(Layer-2) WHEN an expert gave one; omitted (the default) it means nobody reviewed this run, so
no verdict is minted, none enters ``store``, and ``RunResult.verdict`` is ``None`` (F2,
non-goal 3). Supplying it with only one of the two keys raises ``ValueError``: the missing half
is the expert's to write, never ours to default. ``bundle_dir`` (Fase 2a) makes the run OKF-bundle-driven: the project is derived
from the bundle and, before generation, the candidate's prior verdicts in ``store`` are folded
into the hypothesis prompt (Step-1 ExpeL wiring, målbilde §5/§7). ``verdict_dir`` (Fase 5,
Steg 7, målbilde §3/§7) is the async file inbox: a folder of expert/persona-authored verdict
files (plain JSON, R2 raw layer) MERGED into the store BEFORE the Step-1 fold, so a verdict
dropped after an earlier run is consumed by this separate, later run — the long feedback loop,
fully resumable across runs separated in time. The system READS this folder; it does not write
to it (the expert/persona writes, målbilde §3). ``outbox_dir`` (Fase 2a, Steg 7 output layer,
målbilde §3, R2) is the RAW OUTBOX: when set, the run's proposal + outcome artefacts are written
there via ``outbox.write_outbox`` (``run_id`` is then REQUIRED — no wall-clock/uuid default, for
byte-determinism). The outbox dir should be DISTINCT from any ``verdict_dir`` inbox: writing the
outbox into a folder later read as an inbox would re-ingest raw agent output and bypass the
Step-8 promotion gate (self-contamination) — documented here, not enforced. Raises
``pydantic.ValidationError`` on a bad contract and ``BudgetExceeded`` when the token/round cap is
crossed, and ``ValueError`` when ``outbox_dir`` is set without a ``run_id``. ``live_dry_run``
(S4.2, comparison protocol §4 pkt 2/3) is the offline drill: it walks the whole path up to the
EAGER client build, writes the run-config artefact (when ``outbox_dir`` is set), and returns a
``DryRunReport`` BEFORE the first model call (``debate.run``) — zero chat calls.
``semantic_retrieval`` (S3.1) is the opt-in scaling SEAM — the deliverable is the extension
point, not better retrieval. When true, a ``HybridRanker`` blends a cosine term over the
embedded feature triple (sorted cost codes, measure type, magnitude bucket) with the structural
score, which lets a prior verdict on a DIFFERENT cost-code set outrank one that ties
structurally. The shipped ``FakeEmbedder`` is a deterministic sha256 projection carrying NO
semantics, so over a structural tie the resulting order is deterministic but arbitrary;
retrieval *quality* arrives only with an embedder injected via ``embedder=`` or
``--embedder-config``. Default false keeps the structural ranking exactly as before."""
# 0. Fail-fast: an outbox write is byte-deterministic and keyed on run_id — no wall-clock default.
if outbox_dir is not None and run_id is None:
raise ValueError(
"run_id is required when outbox_dir is set (no wall-clock/uuid default — the outbox "
"artefacts are byte-deterministic and keyed on run_id)"
)
# 1. Fail-fast: validate ALL contracts (incl. the verdict-feedback shape) before any client.
load_contracts(
{"docs_dir": docs_dir, "top_k": top_k},
{"max_rounds": max_rounds, "max_tokens": max_tokens},
verdict_input,
)
# 1b. Long loop (Steg 7): ingest the async verdict inbox INTO the store before the Step-1 fold.
# Merge (not replace) into the passed store so run_portfolio's cross-project threading stays
# intact; store.add is idempotent on the content-hash id. A verdict that landed after an earlier
# run thus reaches THIS run's hypothesis via the existing fold below — no change to the fold.
if verdict_dir is not None:
store = store if store is not None else VerdictStore(verdicts=[])
for dropped in load_verdicts_from_dir(verdict_dir):
store.add(dropped)
# 2-3. Project + agent read-context + first-class citations. A bundle run derives ALL THREE from
# the navigated OKF bundle via progressive disclosure (verdict layer EXCLUDED — målbilde §2/§4),
# NOT keyword chunk-stuffing; the road path keeps the chunk-retrieval data source. ``debate_tools``
# is the query-time retrieval surface: since S2c the bundle path carries the four NAVIGATOR
# tools there (progressive disclosure taken to its conclusion — the agents open what they need
# instead of being handed the base), and a ``docs_dir==bundle_dir`` chunk tool is still refused,
# because that one would re-leak the verdict layer the navigation excludes by construction.
# S4.0 (F3): the run path SETS the validator's cost baseline, so the deterministic gate is
# anchored to the project's real cost lines instead of the ones the proposal asserts.
# * road path: the reference project's own ``cost_items`` ARE the baseline -> always anchored.
# * bundle path: anchored only when the bundle SHIPS a ``cost-baseline.json``. A bundle written
# before the amendment (every commons-owned golden) is legitimately un-anchored -> None =
# pre-S4.0 behaviour. A baseline that exists but is malformed still raises (fail-closed).
if prepass_payload is not None and bundle_dir is None:
# Hoisted above the arm below, and refused rather than ignored: a payload is a cut OF a
# knowledge base, and the road path has none for it to agree with. A silently dropped
# payload would leave the caller with a navigating run that reported a declared one.
raise prepass.PrepassRefused(
"a pre-pass payload declares a cut of a knowledge base, so it needs the bundle it "
"was cut from; this run was given no bundle_dir"
)
# P7: the delivered base is the run's own evidence for what identifiers EXIST. Built from
# ``context_files`` (MAJOR-3/S7a-3's rule), so the ``type: verdict`` layer stays out — a
# proposal grounded in a prior verdict would reach the ExpeL fold's material around its gate.
# P18/B1: ONE document per concept file, not one blob. The boundaries ARE the denominator the
# gate's share rule needs, and composing them here — where the base is already walked — is what
# keeps them from being a second, drifting reconstruction (kø-(p)).
bundle_grounding: tuple[str, ...] = ()
# S2c: a CALLER-OWNED sink for what the debate opens (the ``parse_failures``/``ExplorationTrace``
# shape). A returned value would be lost on exactly the run that most needs the evidence — a
# budget stop mid-debate raises out of ``debate.run`` and constructs no ``RunResult`` at all.
# ``ExplorationToolRecorder`` is REUSED rather than re-implemented: it is already the recorder
# for in-process navigator calls, ordered and un-deduplicated, which is exactly the question
# here too ("did this run open anything, and in what sequence"). Its sibling
# ``mcp_tools.ToolCallRecorder`` stays what it is — a sorted, de-duplicated EGRESS claim.
#
# BOUND HERE, above the fork, because the bundle arm hands both lists to ``navigator_tools``:
# the declaration rung refuses against the very trace the recorder writes, and a second list
# would be free to disagree with it about what this run opened (kø-(p)).
debate_tool_calls: list[ToolCall] = []
#: P19 DEL A: which requirement the debate declared as binding, in declaration order.
debate_requirements: list[DeclaredRequirement] = []
if bundle_dir is not None:
bundle = okf.navigate_bundle(bundle_dir)
bundle_grounding = tuple(
"\n".join([f.name, *f.frontmatter.values(), f.body]) for f in bundle.context_files
)
# ONE bundle-id rule (Step 10, slackened S7a-3 pkt. 1): the DECLARED id is the identity and
# the mount is carried alongside, so a base delivered under a directory name of its own is
# opened rather than refused. What is still refused, before a single model call: a base
# whose concepts declare two different corpora.
resolved = okf.reconcile_bundle_id(bundle_dir)
resolved_bundle_id: okf.ResolvedBundleId | None = resolved
okf.assert_declared_ids_agree(bundle)
project = _project_from_bundle(bundle_dir, project_id, bundle=bundle)
# The THIRD projection into ``CostBaseline`` (MAJOR-4), behind an EXPLICIT commission and
# never silent. The refusal PROPAGATES rather than degrading to the file loader: a caller
# who asked for derivation and got an un-anchored run instead would have been answered by a
# silently downgraded order, which is what ``load_mandate`` fail-fasts against. This one
# resolution serves BOTH the full run and the ``live_dry_run`` report below, so the dry-run
# arm cannot drift away from what a real run would anchor on.
baseline = (
okf.derive_cost_baseline(bundle, project_id=project_id)
if derive_cost_baseline
else okf.load_optional_cost_baseline(bundle_dir)
)
# S2c: the debate NAVIGATES the base; it is never handed the whole of it. Measured on K2
# (630 concepts, docs/2026-09-04-syretest-s7b-k2.md § 3.4) the rendered context was 648 962
# o200k tokens riding in THREE prompts — 99,1 % of a run's prompt cost, none of it asked
# for twice. The task message now carries a POINTER, and the agents get the SAME four
# tools the exploration uses; ``gen_context = debate_output or context`` below means the
# generation fallback is bounded by the same change rather than by a second policy.
dimension_id = dimension.id if dimension else None
# The pre-pass fork (order 20260907T080223Z). WITHOUT a payload every line below is what
# it was: the pointer, the four tools, and citations over the whole navigated base.
#
# WITH one, the debate is handed a DECLARED CUT and the tools are withdrawn. The refusal
# PROPAGATES — never a silent degrade back to the pointer, which is ``load_mandate``'s
# rule: a caller who asked for a declared cut and got a navigating run instead was
# answered by a silently downgraded order.
prepass_declaration: prepass.PrepassDeclaration | None = None
debate_tools: list[Any]
if prepass_payload is not None:
# ONE admission gate, shared with the exploration's seeding door: shape, then the
# mounted base, then the empty-delivery refusal. Two copies of what makes a payload
# admissible would let one door accept what the other refuses (kø-(p)).
prepass.admit_payload(
prepass_payload,
bundle_dir=bundle_dir,
resolved_id=resolved,
dimension=dimension_id,
)
# ``rest_reachable=False`` is the FACT this arm establishes four lines below by
# emptying ``debate_tools`` — stated, never defaulted, because the other arm keeps
# them and a declaration that could not tell the two apart would describe neither.
prepass_declaration = prepass.declaration_of(prepass_payload, rest_reachable=False)
context = prepass.render_context(prepass_payload)
# Citations over the DELIVERED concepts alone: a stamp citing the whole corpus for a
# proposal that saw eight documents re-creates the undeclared claim this seam removes.
# Built from the MOUNTED bodies in ``bundle_citations``' own shape, so ``snippet ==
# body[start:end]`` stays exact by construction rather than indexing normalised text.
citations = bundle_excerpt_citations(
bundle, [excerpt.concept_id for excerpt in prepass_payload.excerpts]
)
# Contract SS 2.2: "Context the pre-pass withheld was withheld deliberately." A debate
# holding both the payload and the ladder could walk around the cut it just declared.
# Measured: an empty list reaches the wire as ``tools: None``, so no untested empty-
# array form is introduced. The MCP append BELOW this fork is deliberately untouched —
# this withdraws the navigator tools, not the tool list.
debate_tools = []
else:
context = _bundle_pointer(bundle, resolved.id, dimension=dimension_id)
citations = bundle_citations(bundle)
# §4.1a context-scope, carried over: the agents read ONLY dimension-matched knowledge.
# The filter used to live in the rendering; with navigation it lives in the TOOLS, on
# both rungs (``navigator_tools``' own gate), because that is now where the bytes
# leave. Under a payload the SAME two gates are re-raised by
# ``prepass.verify_against_bundle`` on the mounted documents instead.
# P19 DEL A: the debate gets the declaration rung too, and the sinks are what create
# it. ``debate_tool_calls`` is bound below — this list is the SAME one
# ``ExplorationToolRecorder`` fills, so the refusal reads the run's own read trace.
debate_tools = list(
navigator_tools(
[bundle_dir],
dimension=dimension_id,
opened=debate_tool_calls,
requirements=debate_requirements,
)
)
# What the navigation could NOT reach, taken from the run's ONE walk. The road path below
# navigates no bundle at all, so its empty tuple is literally true rather than a stand-in.
skipped_links: tuple[okf.SkippedLink, ...] = bundle.skipped
else:
project = _project_by_id(project_id)
baseline = baseline_from_project(project)
chunks = retrieve_chunks("cost saving measure", docs_dir, top_k)
citations = [chunk_dict_to_citation(c) for c in chunks]
context = "\n".join(c["snippet"] for c in chunks)
skipped_links = ()
# No knowledge base, so no bundle identity — said by ABSENCE rather than by minting one.
resolved_bundle_id = None
# Bound in BOTH branches for ``resolved_bundle_id``'s reason: the ``DryRunReport`` and the
# ``RunResult`` below read it unconditionally, and a name bound in one arm only is a
# ``NameError`` waiting for the other caller.
prepass_declaration = None
debate_tools = [make_retrieval_tool(docs_dir, top_k=top_k)]
# F4: the anchoring REQUIREMENT, opt-in and checked here — the one point at which both
# branches have bound ``baseline``, and above the dry-run cut below, so the FREE trip refuses
# too. Before the first model call by construction: at the exit code a refusal after the spend
# is indistinguishable from one before it (session 57's rule). The road path is anchored by
# construction, so on it this can only pass.
if require_cost_baseline and baseline is None:
raise UnanchoredRunRefused(
"this run was required to be anchored, but the knowledge base offers no cost "
"baseline: without one the validator's stage 0 is skipped and nothing ties a proposed "
"cost line to this project. Ship a cost-baseline.json, or pass "
"--derive-cost-baseline when the base carries a priced schedule"
)
# P8: what this run was GIVEN, composed ONCE. Bound HERE and not inside ``_evaluate`` below,
# and that placement is the measurement this seam rests on: this is the first point at which
# both halves exist AND it is above the ``--live-dry-run`` cut, so the offer can be reported
# on the FREE trip — before the first model call at ``debate.run``, let alone the three
# generation attempts. ``generate.py``'s own composition happens per attempt, AFTER
# ``_fetch_parsed`` has returned, so a report from there could only ever speak once an attempt
# had been paid for. ONE binding feeding both the report and the gate: two compositions of one
# text are free to disagree, which is exactly what a report must not be able to do (kø-(p)).
delivered = Grounding(documents=(context, *bundle_grounding))
offer = grounding_offer(project, baseline, delivered)
# Trekk B2 (krav 3): configured MCP servers become tools the AGENTS can call during the debate.
# Appended to BOTH paths — on the bundle path they are the first tools that path has ever had.
# Constructed here but NOT connected: an ``MCPTool`` is an async context manager, so the run
# enters them around ``debate.run`` below and exits afterwards. Empty tuple -> the tool list is
# byte-identical to the pre-Trekk-B one, and no network call is possible.
live_mcp_tools = build_mcp_tools(mcp_servers) if mcp_servers else []
debate_tools = debate_tools + live_mcp_tools
if not citations:
raise ValueError(f"no citable content in docs_dir: {docs_dir!r}")
# 4. Budget + maker-checker debate (round-capped; Layer-1 HITL optional). The shared meter is
# driven on the debate's chat calls by the BudgetMiddleware (the brief's named short-circuit
# mechanism); ``debate_tools`` exposes the citation-bearing data source on the road path.
meter = (
meter
if meter is not None
else TokenMeter(Budget(max_tokens=max_tokens, max_rounds=max(max_rounds * 4, 4)))
)
factory = client_factory if client_factory is not None else _default_factory(profile)
budget_mw = BudgetMiddleware(meter)
# Trekk B4: the egress DECLARATION says what a run may contact; this records what it actually
# called. Attached only when servers are configured — with none there is nothing to attribute a
# call to, and the middleware list stays exactly what it was before Trekk B.
call_recorder = ToolCallRecorder(tool_server_index(mcp_servers)) if mcp_servers else None
# S2c: a CALLER-OWNED sink for what the debate opens (the ``parse_failures``/``ExplorationTrace``
# shape). A returned value would be lost on exactly the run that most needs the evidence — a
# budget stop mid-debate raises out of ``debate.run`` and constructs no ``RunResult`` at all.
# ``ExplorationToolRecorder`` is REUSED rather than re-implemented: it is already the recorder
# for in-process navigator calls, ordered and un-deduplicated, which is exactly the question
# here too ("did this run open anything, and in what sequence"). Its sibling
# ``mcp_tools.ToolCallRecorder`` stays what it is — a sorted, de-duplicated EGRESS claim.
debate_middleware: list[Any] = [budget_mw, ExplorationToolRecorder(debate_tool_calls)]
if call_recorder is not None:
debate_middleware.append(call_recorder)
debate = fresh_workflow(
factory,
max_rounds=max_rounds,
enable_layer1_hitl=enable_layer1_hitl,
tools=debate_tools,
middleware=debate_middleware,
)
# S4.2 cut (comparison protocol §4 pkt 2/3): everything above is offline — contracts, budget, and
# the EAGER client build (fresh_workflow constructs the proposer+checker clients, workflow.py:64).
# Capture the run-config (resolved model per BUILT role, profile, params, token cap) and, for a
# ``--live-dry-run``, STOP HERE — before the first (paid) model call at ``debate.run`` below.
if outbox_dir is not None or live_dry_run:
# ``resolved_models`` reflects the configured MAP (the default factory's model-ids for the M2
# run). Under an injected ``client_factory`` the built clients may differ (e.g. "synthetic");
# ``provenance.model`` (below) stays the authority on the client actually built.
resolved_models = {role: resolve_model(profile, role) for role in _MAKER_CHECKER_ROLES}
if outbox_dir is not None:
assert run_id is not None # narrowed by the step-0 guard (no wall-clock default)
outbox.write_run_config(
outbox_dir,
run_id,
profile=Profile(profile).value,
resolved_models=resolved_models,
max_rounds=max_rounds,
max_tokens=max_tokens,
top_k=top_k,
)
if live_dry_run:
return DryRunReport(
profile=Profile(profile).value,
resolved_models=resolved_models,
max_rounds=max_rounds,
max_tokens=max_tokens,
top_k=top_k,
cost_baseline_anchored=baseline is not None,
bundle_id_source=resolved_bundle_id,
skipped_links=skipped_links,
prepass=prepass_declaration,
# P8, and this surface is the point: the offer is measured ABOVE this cut, so a dry
# run reports it having made no model call at all.
grounding_offer=offer,
)
# The MCP lifecycle (Trekk B2): entered HERE, after the dry-run cut above, so a dry run never
# opens a connection — its promise to stop before the first call covers egress too. Constructed
# tools that are never entered expose nothing, and ones never exited leave the process hanging,
# so the stack owns both halves.
try:
async with AsyncExitStack() as mcp_stack:
for live_tool in live_mcp_tools:
await mcp_stack.enter_async_context(live_tool)
result = await debate.run(
f"Find a cost-saving measure for {project.id}.\nContext:\n{context}"
)
finally:
# ``finally``, the ``write_parse_failures`` precedent: any exception leaving the debate —
# a budget stop is today's known one — destroys the same evidence, and a list of exception
# types is a list that goes stale. Written even when EMPTY: "the debate opened nothing" is
# the S2c regression itself, so it must be readable rather than inferred from an absence.
if outbox_dir is not None:
assert run_id is not None # narrowed by the step-0 guard (no wall-clock default)
if prepass_declaration is not None:
# BEFORE the debate trace, and by a DIRECT call rather than ``_write_or_report``:
# that helper's required ``in_flight`` is bound only inside the generation block
# below, and passing ``None`` here would let an OSError displace an in-flight
# ``BudgetExceeded`` — the very defect it exists to prevent. Written from the
# ``finally`` so a budget stop mid-debate still leaves the declaration, and built
# from the SAME object ``RunResult.prepass`` carries, never a second load.
outbox.write_prepass(
outbox_dir,
run_id,
declaration=prepass.declaration_payload(prepass_declaration),
)
outbox.write_debate_tools(
outbox_dir,
run_id,
tool_calls=tool_call_payload(debate_tool_calls),
requirements=requirement_payload(debate_requirements),
)
# F1: the candidate must derive from the DEBATE. Feed the proposer's converged output into
# generation (retrieval context is the last-resort fallback only). The checker's verdict
# (Step 3/4) is parsed from the SAME debate result and gates the outcome below.
debate_output = _debate_text(result)
checker_decision, checker_reason = _checker_verdict(result)
gen_context = debate_output or context
# Step-1 ExpeL wiring (Fase 2a, målbilde §5/§7): fold the candidate's prior verdicts INTO the
# hypothesis context BEFORE generation, keyed on the OKF bundle's candidate features (available
# pre-hypothesis). THIS is the one missing dataflow — previously ExpeL was computed
# post-generation into a discarded SessionContext (step 7 below), so a prior verdict could not
# reach the next hypothesis. Bundle-driven path with a populated store only.
#
# Scope of the --semantic-retrieval opt-in, stated precisely (an earlier version of this
# comment claimed "the road path is untouched", which the flag made false): the ranker built
# below is passed to ALL THREE retrievals this run performs — this fold, and the post-hoc
# ExpeLContextProvider + store.retrieve in step 7 — so the flag reaches the road path's
# proposal-keyed retrieval too. What IS untouched on the road path is the fold itself: it stays
# bundle-gated, so a --docs-dir-only run remains single-shot either way.
# S3.1 opt-in: build the hybrid ranker as a LOCAL, then pass it explicitly at each retrieval
# this run performs. It is deliberately not assigned to ``store.retriever``: the store is
# caller-owned (``run_portfolio`` threads one store across every project, and a library caller
# may reuse theirs), so a store-global assignment leaked this run's opt-in into every later use
# of that object — including a subsequent run with the flag OFF. Flag off => ranker stays None
# => ``retrieve`` falls through to the StructuralRetriever default.
ranker = (
HybridRanker(
embedder if embedder is not None else FakeEmbedder(),
similarity,
SEMANTIC_WEIGHT_DEFAULT,
)
if semantic_retrieval
else None
)
# S7b søm 1: the fold needs a pre-hypothesis CANDIDATE to rank prior verdicts against, and that
# candidate comes from the base's IR projection. A base without one (every ingested corpus) can
# now run — but it cannot key retrieval, and the verdicts it holds would otherwise be dropped in
# silence. Counted here and reported; the fold itself is unchanged when the key exists.
unkeyed_verdicts = 0
if bundle_dir is not None and store is not None and store.verdicts:
expel_query = optional_bundle_candidate_features(bundle_dir)
if expel_query is None:
unkeyed_verdicts = len(store.verdicts)
else:
fewshot = ExpeLContextProvider(
store, expel_query, k=top_k, retriever=ranker
).format_fewshot()
gen_context = f"{fewshot}\n\n{gen_context}"
# 5. Structured candidate(s) -> blocking validation on the NUMBERS; token bound = the meter.
# Without a mandate this is the single pre-Trekk-A call, unchanged. With one, every
# commissioned approach is evaluated in turn (and the run's own proposal last, when allowed),
# each under the SAME meter — no new unbounded loop; the caps already in force are the bound.
proposer_client = factory("proposer")
# Step 5 (målbilde §5/§7): generation now returns its falsification history alongside the
# outcome. ``_evaluate`` keeps its ``ValidatedProposal | Rejection`` shape so ``_evaluate_mandate``
# is untouched, and the history is accumulated here in call order — one entry per approach that
# needed correcting, concatenated (see ``RunResult.refinements`` for that honesty limit).
refinements: list[Rejection] = []
# Fase 1b, funn 1: the raw replies that did not parse. Owned HERE, beside ``meter``, and handed
# down — not read back off a return value. ``generate_via_llm`` raises ``BudgetExceeded`` from
# inside its own fetch loop when the round ledger runs out on unparseable replies (the measured
# live failure), and on that path it returns nothing at all; a caller-owned accumulator is the
# only shape that still holds the evidence afterwards. Concatenated across commissioned
# approaches rather than keyed per approach, mirroring ``RunResult.refinements``' honesty limit.
parse_failures: list[ParseFailure] = []
# MAJOR-2: what a human answered about each validated candidate. A caller-owned sink for the
# same measured reason ``parse_failures`` is one, one notch sharper: the round ledger can fire
# on the very attempt a revise bought, and on that path ``generate_via_llm`` returns nothing —
# so the run whose record matters most is exactly the one a return value cannot reach.
expert_reviews: list[ProposalReview] = []
# P19 D2: which cap, if any, cut the commission short. Caller-owned for the reason every other
# sink here is: ``_evaluate_mandate`` SWALLOWS the stop once something has been produced, so a
# return value would not reach the ``finally`` that writes the artefact.
budget_stops: list[str] = []
async def _evaluate(approach: Approach | None) -> ValidatedProposal | Rejection:
# Which candidate the expert is being asked about. With a mandate every entry is keyed —
# the run's own proposal by ``OWN_PROPOSAL_ID``, the first-class row ``_evaluate_mandate``
# already uses — and ``None`` means only one thing: there was no mandate at all. Recording
# ``None`` for the own proposal would make it indistinguishable from a non-mandate run's
# entry, which is the property keying exists for.
if approach is not None:
review_key: tuple[str | None, str | None] = (approach.id, approach.label)
elif mandate is not None:
review_key = (OWN_PROPOSAL_ID, "the system's own proposal")
else:
review_key = (None, project.id)
generated = await generate_via_llm(
proposer_client,
project,
gen_context,
meter,
baseline=baseline,
approach=approach,
parse_failures=parse_failures,
reviewer=proposal_reviewer,
reviews=expert_reviews,
review_key=review_key,
# D3: the reasoning gate's own answer, READ-ONLY. An expert deciding whether to spend
# an attempt is helped by knowing it; it never enters the record, because the two
# falsifiers are never blended.
checker_verdict=checker_decision,
# P7: what this run was GIVEN, as opposed to what the debate said about it.
# ``context`` is the DELIVERED rendering (pre-pass cut / bundle pointer / retrieved
# chunks) — never ``gen_context``, which on the debate path is the model's own
# summary and would let a code the debate invented ground the proposal repeating it.
grounding=delivered,
)
refinements.extend(generated.refinements)
return generated.outcome
coverage: tuple[ApproachOutcome, ...] = ()
evaluated: tuple[tuple[str, ValidatedProposal | Rejection], ...] = ()
in_flight: BaseException | None = None
try:
if mandate is None:
validator_outcome = await _evaluate(None)
else:
validator_outcome, coverage, evaluated = await _evaluate_mandate(
mandate, _evaluate, budget_stops
)
except BaseException as stop:
# Recorded and re-raised UNTOUCHED. This arm decides nothing about the exception itself —
# only what the two writers in the ``finally`` are allowed to do to it (``_write_or_report``).
in_flight = stop
raise
finally:
# ``finally``, not ``except BudgetExceeded``: the round ledger is today's known way out, but
# any exception leaving generation destroys the same evidence, and a per-exception-type list
# is a list that goes stale. Written only when something actually failed to parse, so the
# file's presence is the signal (a run whose replies all parse leaves the outbox unchanged).
if outbox_dir is not None and parse_failures:
assert run_id is not None # narrowed by the step-0 guard (no wall-clock default)
_write_or_report(
lambda: outbox.write_parse_failures(
outbox_dir,
run_id,
failures=[{"text": f.text, "error": f.error} for f in parse_failures],
),
what=f"{run_id}-parse-failures.json",
in_flight=in_flight,
)
# Same ``finally``, a THIRD write rule: IFF a mandate was given, including when the run
# stopped before a single approach was evaluated (P19 D2). Coverage is the MANDATE's
# report by construction — without one there are no approaches and the file would describe
# nothing — so a mandate-less run leaves the outbox byte-identical, which two existing
# tests pin as an exact listing. The stop reason comes from the in-flight exception rather
# than being inferred: a ``BudgetExceeded`` carries ``kind`` as a field precisely so that
# "which cap bound" is readable by machine (kø-(y)), and a run that finished says so with
# an empty string rather than with a missing key.
if outbox_dir is not None and mandate is not None:
assert run_id is not None # narrowed by the step-0 guard (no wall-clock default)
_write_or_report(
lambda: outbox.write_coverage(
outbox_dir,
run_id,
rows=[
{
"id": row.id,
"label": row.label,
"status": row.status,
"detail": row.detail,
"saving_nok": row.saving_nok,
}
for row in coverage
],
stop_reason=(
budget_stops[0]
if budget_stops
else (in_flight.kind if isinstance(in_flight, BudgetExceeded) else "")
),
),
what=f"{run_id}-coverage.json",
in_flight=in_flight,
)
# Same ``finally``, different write rule: IFF a reviewer was given, including when the
# list is empty (D4). A reviewer-less run must leave the outbox byte-identical, while a
# reviewer that was offered and never consulted is a fact the artefact must be able to
# state rather than one an operator infers from an absent file.
if outbox_dir is not None and proposal_reviewer is not None:
assert run_id is not None # narrowed by the step-0 guard (no wall-clock default)
_write_or_report(
lambda: outbox.write_proposal_reviews(
outbox_dir,
run_id,
payload=proposal_reviews_payload(
expert_reviews, key_of=lambda p: verdict_key(_features_of(p))
),
),
what=f"{run_id}-proposal-reviews.json",
in_flight=in_flight,
)
proposal = validator_outcome.proposal
# 6. First-class provenance stamp (authoritative; independent of MAF Annotation).
# F1: an injected client_factory stamps the injected client's REAL model ("unknown" is the
# neutral fallback for a client that doesn't surface one -- never a fabricated name); the
# default path keeps the deterministic resolve_model. validator_decision reflects the VALIDATOR
# (the numbers) ONLY -- stamped from validator_outcome BEFORE the checker override below, so a
# checker-gated proposal whose numbers passed is never mislabelled as validator-rejected.
model = (
(getattr(proposer_client, "model", None) or "unknown")
if client_factory is not None
else resolve_model(profile, "proposer")
)
stamp = ProvenanceStamp(
citations=citations,
model=model,
role="proposer",
validator_decision=(
"validated" if isinstance(validator_outcome, ValidatedProposal) else "rejected"
),
token_usage=meter.tokens,
# Whether stage 0 of the deterministic gate had a baseline to reconcile against. Read off
# the SAME ``baseline`` the validator was handed, so the record cannot describe a different
# anchoring than the one that ran. The road path is anchored by construction (the reference
# project's own cost_items ARE the baseline); a bundle is anchored iff it ships the file.
cost_baseline_anchored=baseline is not None,
# P19/B2: what the run made of each code it was handed. Derived from the SAME classifier
# the gate uses (kø-(p)), off the proposal being stamped — never re-read from anywhere.
code_forms=classify_codes([item.code for item in proposal.affected_items]),
# WHICH corpus was judged, and whether the base named itself or the mount named it for it.
# Read off the SAME resolution the run opened the base with (kø-(p)); ``None`` on the road
# path, where no knowledge base exists to name.
bundle_id_source=resolved_bundle_id,
# B4: which external service the debate actually called. Read AFTER the debate, so it is a
# record rather than an intention. The honesty limit lives on ``ExternalCall`` itself: this
# is the call and its source, not a verified rendering of the service's answer.
external_calls=call_recorder.calls() if call_recorder is not None else [],
)
# 6b. Step 3/4 checker gate (målbilde §2/§6): the validator falsifies the numbers, the checker
# falsifies the reasoning. An explicit checker REJECT blocks an otherwise-validated proposal; a
# validator rejection (the numbers) already stands. Fail-open: APPROVE/absent never blocks.
outcome: ValidatedProposal | Rejection
if isinstance(validator_outcome, ValidatedProposal) and checker_decision == "reject":
outcome = Rejection(proposal=proposal, reason=f"checker rejected: {checker_reason}")
else:
outcome = validator_outcome
# 6c. Step 2 dimension scope gate (§4.1b): a candidate whose measure_type/codes fall OUTSIDE the
# run's dimension is rejected. A scope/type gate placed AFTER the checker override (preserves
# test_checker_gate_loadbearing) — NOT a new numeric gate: validate_proposal stays the only
# blocking numeric gate and provenance.validator_decision (the numbers) is untouched. Mirrors the
# override form: only an otherwise-standing ValidatedProposal can be flipped to a Rejection.
if dimension is not None and isinstance(outcome, ValidatedProposal):
feats = _features_of(proposal)
if not admits(
measure_type=feats.measure_type, codes=feats.affected_codes, dimension=dimension
):
outcome = Rejection(
proposal=proposal,
reason=f"outside dimension {dimension.id!r}: measure_type={feats.measure_type!r}",
)
# 7. ExpeL (regression guard + traceability): exercises the two-arg extend_instructions
# injection on a REAL SessionContext (the Critical Fase-1 GA-signature guard), and surfaces
# the proposal-keyed retrieval for RunResult.retrieved. On the bundle path the load-bearing
# ExpeL->prompt dataflow already happened pre-generation (above); this block's SessionContext
# is NOT what reaches the prompt.
store = store if store is not None else VerdictStore(verdicts=[])
features = _features_of(proposal)
provider = ExpeLContextProvider(store, features, k=top_k, retriever=ranker)
sctx = SessionContext(input_messages=[], instructions=[])
await provider.before_run(agent=None, session=None, context=sctx, state={})
retrieved = store.retrieve(features, k=top_k, retriever=ranker) if store.verdicts else []
# 8. Layer-2 (out-of-band): capture the durable verdict + persist; B11 notify is a stub.
# ONLY when an expert actually gave one (F2, non-goal 3). Absent ``verdict_input`` means
# nobody reviewed this run: nothing is minted, nothing enters the store, and nothing is
# notified — so silence cannot become an ``approved`` that propagates into the next
# project's hypothesis prompt as a prior expert judgement. A half-given verdict is a
# CALLER error, refused by name rather than completed on the expert's behalf (validation,
# never repair — the ``write_concept_file`` precedent).
# The SHAPE of a supplied verdict is not re-checked here: step 1's ``load_contracts``
# already ran ``FeedbackContract`` over it and refused a half-given one by field name.
verdict: Verdict | None = None
if verdict_input is not None:
verdict = capture_verdict(features, verdict_input["decision"], verdict_input["rationale"])
store.add(verdict)
if notify is not None:
notify(verdict)
# S2.1 outbox (RAW output layer, målbilde §3): persist the run's proposal + outcome artefacts
# when configured. Wired ONLY here — no new consumer (S5.1/S5.2 are Non-Goals this bolk). run_id
# is guaranteed non-None by the fail-fast guard at the top.
if outbox_dir is not None:
assert run_id is not None # narrowed by the step-0 guard; keeps the type checker honest
if not evaluated:
outbox.write_outbox(
outbox_dir,
run_id,
outcome=outcome,
provenance=stamp,
checker_verdict=checker_decision,
# The artefact carries the candidate's KEY, not evidence that anybody decided:
# identical to ``verdict.id`` whenever a verdict WAS given (both mint from the
# same features), and still meaningful on a run nobody reviewed. This is the
# documented purpose of ``verdict_key`` and it is what keeps the per-approach
# branch below and this one speaking the same language.
verdict_id=verdict_key(features),
)
else:
# A5: one judgeable artefact PER evaluated approach. Without this the expert can only
# judge the approach the run happened to select, so every other approach they
# commissioned teaches the learning loop nothing. The per-approach set REPLACES the
# single run-level pair rather than joining it — the selected approach is already among
# these, and writing both would make ``hitl pending`` count it twice.
for approach_id, approach_outcome in evaluated:
# The SELECTED approach carries the run's final outcome, so the outbox can never
# disagree with the ``RunResult``: the checker/dimension overrides above apply to
# that one. The others carry the validator's verdict, which is the only falsifier
# that ran on them.
final = outcome if approach_outcome is validator_outcome else approach_outcome
outbox.write_outbox(
outbox_dir,
run_id,
outcome=final,
# ``validator_decision`` must follow ITS OWN approach — stamping every artefact
# with the selected approach's decision would report a rejected candidate as
# validated. Everything else (model, citations, token usage) is the run's.
provenance=stamp.model_copy(
update={
"validator_decision": (
"validated"
if isinstance(approach_outcome, ValidatedProposal)
else "rejected"
)
}
),
checker_verdict=checker_decision,
# The key an expert verdict on THIS candidate will arrive under (S3.2 content
# hash). Reusing the run's single verdict id would let one delivered verdict
# clear every approach from the pending queue.
verdict_id=verdict_key(_features_of(approach_outcome.proposal)),
approach_id=approach_id,
)
return RunResult(
outcome=outcome,
provenance=stamp,
verdict=verdict,
retrieved=retrieved,
store=store,
debate_output=debate_output,
checker_verdict=checker_decision,
coverage=coverage,
refinements=tuple(refinements),
skipped_links=skipped_links,
unkeyed_verdicts=unkeyed_verdicts,
debate_tool_calls=tuple(debate_tool_calls),
expert_revisions=tuple(expert_reviews),
prepass=prepass_declaration,
# P8: read off the SAME single measurement the gate's own grounding descends from, so the
# record and the refusals cannot describe different inputs.
grounding_offer=offer,
)
def _aggregate(runs: tuple[RunResult, ...], store: VerdictStore) -> PortfolioResult:
"""Thin aggregate over the per-project runs (SC2): partition validated/rejected, total the
validated claimed saving, and total every run's provenance token usage."""
validated = [r for r in runs if isinstance(r.outcome, ValidatedProposal)]
rejected = [r for r in runs if isinstance(r.outcome, Rejection)]
return PortfolioResult(
runs=runs,
store=store,
validated_count=len(validated),
rejected_count=len(rejected),
sum_claimed_saving_nok=sum(r.outcome.proposal.claimed_saving_nok for r in validated),
sum_token_usage=sum(r.provenance.token_usage for r in runs),
)
def _baseline_ore(projects: Iterable[Project]) -> int:
"""Addressable baseline in øre, quantized PER COST LINE and summed as integers (Kø-(p)).
Both sides of the goal comparison must be computed in the same order. ``observed_ore`` is
``SavingsLedger``'s sum of per-candidate integer øre; a baseline that summed
``Project.total_cost`` floats and quantized the total ONCE put the threshold on a different
scale — three ``60000.005`` NOK lines are ``18000003`` øre per line but ``18000001`` summed
first, enough to flip a percent goal. Each cost line is a real amount, so the per-line value
is the one that exists; integer addition also keeps the total order-independent, which
``Project.total_cost``'s float fold is not under the D-D wave model."""
return sum(to_ore(item.total_cost) for p in projects for item in p.cost_items)
def _goal_limit_if_reached(goal: GoalContract, observed_ore: int, baseline_ore: int) -> int | None:
"""The threshold ``observed_ore`` MET (``>=``), or ``None`` if the goal is not yet reached. An
absolute-øre target compares directly; a percent target is taken against ``baseline_ore`` (the
addressable cost in øre). When both are set, reaching EITHER counts as met."""
if goal.absolute_ore is not None and observed_ore >= goal.absolute_ore:
return goal.absolute_ore
if goal.percent is not None:
if baseline_ore <= 0:
raise ValueError(
f"percent goal ({goal.percent}%) is meaningless against a non-positive baseline "
f"({baseline_ore} øre): int(percent/100 * 0) == 0 would falsely read as 'goal "
"reached'. Supply an absolute_ore target, or ensure the addressable baseline is > 0."
)
threshold = int(goal.percent / 100 * baseline_ore)
if observed_ore >= threshold:
return threshold
return None
def _wave_snapshot(store: VerdictStore) -> VerdictStore:
"""A per-project copy of the shared store's CURRENT verdicts (D-D wave model).
Two coroutines appending to one list would interleave by completion order, which no barrier
could then undo. Giving every project in a wave its own copy removes the race at its source
rather than serializing it away with a lock — a lock would order the appends by whoever won,
which is exactly the nondeterminism being eliminated.
``retriever`` is carried across deliberately: it is the S3.1 opt-in seam, and a snapshot that
dropped it would silently downgrade a caller-owned store's semantic retrieval to the
structural default mid-pass.
**Field-complete by construction.** ``dataclasses.replace`` carries over every field
``VerdictStore`` declares and overrides only ``verdicts``, so a field added later is copied
without this function being touched. The hand-enumerated version this replaced could silently
drop one — the exact defect it was first written with (the ``retriever`` omission), which its
own docstring then recorded as a standing hazard. Deriving the copy from the dataclass removes
the hazard instead of documenting it, and does so without the field-count assertion that idea
was rejected for: there is nothing left to keep in sync.
That docstring credited a Step-2 contract test with catching the omission. MEASURED while this
change was made: no such coverage remained — reinstating the hand-enumerated form left the
whole suite green, so the S3.1 retriever seam could be downgraded mid-pass in silence. The gate
is now ``test_wave_snapshot_carries_every_field_except_the_copied_verdicts``.
``verdicts`` is still listed explicitly, and must be: ``replace`` copies field REFERENCES, so
omitting it would hand back a store sharing the caller's list — the very race this snapshot
exists to remove, reintroduced by the call that looks tidiest."""
return replace(store, verdicts=list(store.verdicts))
def _merge_wave(store: VerdictStore, wave: Sequence[tuple[str, VerdictStore]]) -> None:
"""The deterministic merge barrier — the seam S3.3 rests on.
Each project ran against its own snapshot, so its NEW verdicts are the ones absent from the
wave-start store. They are merged back in **wave-submission order** — the order the caller
listed the projects in — so the shared store's contents depend on the portfolio's membership
and never on which project's model round-trips happened to finish first.
**Submission order, NOT lexicographic project_id.** The plan specified a sort on ``project_id``;
the Step-2 contract test measured that it produces a deterministic order which is nevertheless
the WRONG one. On the shipped fixture, lexicographic order is BRU/FV42/RV13 while the sequential
pass yields FV42/RV13/BRU, so a ``project_id`` sort satisfies "deterministic" while breaking
"identical to ``concurrency=1``" — and the second is the actual contract. ``wave`` arrives in
submission order, so preserving it is the fix.
**Detach point: this function's ordering discipline is only half the seam — see
``_wave_snapshot``, which is the half that carries the load.** Because each project writes to
its own copy, ``wave`` is never reordered by completion, so iterating it in order is already
deterministic. Removing the SNAPSHOT is what turns the shared list back into a race and takes
``test_concurrent_pass_is_byte_identical_to_sequential`` RED. This is recorded plainly rather
than dressing the loop in a ``sorted(...)`` that would re-sort an already-ordered list and read
as a guard while guarding nothing.
Safe to apply as-is because ``VerdictStore.add`` is first-write-wins per content-hash id
(``verdicts.py:303-304``) — the wave-start verdicts every snapshot carries are re-offered and
dropped, so only the new ones land."""
seen = {v.id for v in store.verdicts}
for _pid, snapshot in wave:
for verdict in snapshot.verdicts:
if verdict.id not in seen:
store.add(verdict)
seen.add(verdict.id)
def _waves(ids: list[str], k: int) -> list[list[str]]:
"""Partition ``ids`` into consecutive waves of at most ``k``, preserving caller order (D-D).
Order preservation is the whole point: the wave model may change WHEN a project runs, never
WHICH projects run nor in what sequence they are submitted. At ``k=1`` every wave holds a
single project, so the wave loop degenerates to the pre-S3.3 sequential order by
construction — that is what makes the parameter behaviour-preserving at its default."""
return [ids[i : i + k] for i in range(0, len(ids), k)]
def _run_meter(
meter_factory: Callable[[], TokenMeter] | None,
portfolio_meter: PortfolioMeter | None,
max_rounds: int,
) -> TokenMeter | None:
"""The per-project meter ``run_portfolio`` hands to one run: the injected test seam, a meter
BOUND to the portfolio ledger, or ``None`` (letting ``run_project`` build its own, unchanged).
The bound meter mirrors ``run_project``'s own construction (``max(max_rounds * 4, 4)``) so the
only difference the portfolio cap introduces is the token ceiling and the shared ledger — the
round budget is not quietly redefined along the way."""
if meter_factory is not None:
return meter_factory()
if portfolio_meter is None:
return None
return TokenMeter(
Budget(
max_tokens=portfolio_meter.budget.max_tokens_per_run,
max_rounds=max(max_rounds * 4, 4),
),
portfolio=portfolio_meter,
)
async def run_portfolio(
project_ids: Sequence[str] | None = None,
profile: Profile | str = Profile.LOCAL,
*,
dimension: Dimension | None = None,
ledger: SavingsLedger | None = None,
goals: GoalConfig | None = None,
store: VerdictStore | None = None,
client_factory: Callable[[str], BaseChatClient] | None = None,
max_rounds: int = _DEFAULT_MAX_ROUNDS,
max_tokens: int = _DEFAULT_MAX_TOKENS,
top_k: int = 3,
concurrency: int = 1,
meter_factory: Callable[[], TokenMeter] | None = None,
portfolio_meter: PortfolioMeter | None = None,
semantic_retrieval: bool = False,
embedder: Embedder | None = None,
mandate: Mandate | None = None,
mcp_servers: tuple[McpServerConfig, ...] = (),
) -> PortfolioResult:
"""Fan out over a portfolio of independent projects SEQUENTIALLY, composing ``run_project``
as-is (every project's execution state — meter, debate, retrieval context — is built fresh
per call, so sequential reuse is inherently isolated). ONE ``VerdictStore`` is threaded
across every run, AND each project's ``bundle_dir``/``verdict_dir`` are threaded into
``run_project`` (Fase 2a S2.0): a bundle-backed project's ``bundle_dir``-gated Step-1 ExpeL fold
then folds the store's prior verdicts into its hypothesis prompt, so a verdict captured on
project k materially reaches project k+1's hypothesis — the cross-project learning loop is
delivered, not merely asserted. ``verdict_dir`` also lets each project merge its async file inbox
(Steg 7) on the portfolio path. ``project_ids`` defaults to every loaded project; an unknown id
raises ``ValueError``. ``meter_factory`` (test seam) supplies a per-project meter — inject a
shared meter to make the isolation guard go red (SC3).
Step 8 goal-stop (SC6): ``ledger`` carries EARLIER, out-of-band HITL realizations (the long file
loop / Steg-7 role split — ``run_project`` never realizes mid-pass, C3), so the check reads an
ACCUMULATED sum, it does not build one during the pass. BEFORE running each pid, the accumulated
realized sum is compared to ``goals`` with a ``>=`` boundary (reached, not exceeded — H1): a HARD
portfolio goal ``break``s the pass (``stopped_early``); a HARD per-project goal SKIPS that pid
(its further runs are future passes, not more runs here); a SOFT goal flags ``stop_reason`` but
continues. Because the ledger is static during the pass, a reached goal is observed on the first
iteration. ``stop_reason`` surfaces the first goal event; a per-project skip is also observable
as the pid's absence from ``runs``.
``concurrency`` (S3.3, D-D wave model) caps how many projects run at once. It is validated
BEFORE anything loads — an unusable cap must surface as an error, never as a silently-empty
pass. At the default ``1`` the pass is the sequential one described above.
Error policy (S3.3, SC3) is COLLECT-AND-CONTINUE: a project that raises does not cancel its
siblings and does not abort the pass. Its exception is recorded as a ``RunFailure`` in
``failures`` while every project that completed keeps its full ``RunResult``, so a portfolio of
independent projects degrades one project at a time rather than all at once. This holds at
every ``k``, including ``k=1``. The policy is carried by ``return_exceptions=True`` on the
wave's ``gather``; ``asyncio.TaskGroup`` would cancel the siblings and is rejected for that
reason. Note the pass still raises for errors that are NOT one project's failure — an unknown
``project_id`` and a non-positive ``concurrency`` are caller mistakes and fail fast.
Goal-stop under waves (S3.3 reading of the Step-8 semantics above — a reading, NOT a redesign):
the checks run at WAVE ASSEMBLY, per member, before the wave starts. A HARD per-project goal
removes that pid from its wave, so an excluded project is never STARTED — it costs no model
round-trip and leaves no verdict behind, which a post-hoc filter over completed runs could not
achieve. A HARD portfolio goal stops the pass, and the wave already assembled still crosses its
merge barrier before the loop exits, so a stop never strands verdicts outside the store. A SOFT
goal flags ``stop_reason`` and continues. Membership is identical at every ``concurrency`` — but
the reason is that the ledger is STATIC during a pass (C3: no realization happens on the run
path, which ``test_concurrent_pass_does_not_write_on_the_run_path`` pins), so every check reads
the same accumulated sum regardless of when it runs. Wave-assembly placement buys the
never-started property, not determinism; determinism is the one-writer rule's.
**The one semantic difference ``k > 1`` introduces, stated plainly (OQ1).** Every project in a
wave reads the WAVE-START store, so a verdict captured by project A does NOT reach project B's
hypothesis prompt when A and B share a wave — sequentially it would. This is a deliberate trade,
not an oversight: the snapshot is what removes the append race, and restoring intra-wave
visibility would restore exactly the completion-order dependence the barrier exists to
eliminate (what B saw would depend on whether A happened to finish first). Learning therefore
flows ACROSS wave boundaries, not within them, and ``concurrency`` is the knob that trades
learning granularity for wall-clock — at ``k=1`` nothing changes, and at ``k=len(portfolio)``
the pass learns nothing from itself.
On the SHIPPED reference fixture this difference is invisible in outcomes, because the Step-1
ExpeL fold is ``bundle_dir``-gated and no reference project sets ``bundle_dir`` — the chain is
live for store CONTENT but inert for OUTCOMES. That is a property of the fixture, never a
property of the design, so it is pinned on a bundle-backed pair where the fold does fire
(``test_intra_wave_visibility_is_the_documented_semantic_difference``): same fixture, same
sentinel, only the wave boundary moves. Store content stays identical across ``k`` — the
difference is confined to what each project READ, never to what the pass produced or persisted.
``portfolio_meter`` (S3.4, F10) installs the GLOBAL token cap. Without it nothing changes: each
run is bounded only by its own ``max_tokens``, so N projects can cost N times that with no
ceiling over the pass. With it, one ``PortfolioMeter`` is shared by every run (each run's meter
is BOUND to it, which is why ``meter_factory`` — whose meters are unbound — is refused
alongside it: accepting both would run a pass that looks capped and is not), and the cap is
enforced in three places that are deliberately different:
- **At startup**, a remainder that cannot fund one run raises ``BudgetRefused`` — a pass that
can afford zero projects is a caller mistake, not a result.
- **At wave assembly**, a project that cannot be funded is NEVER STARTED, and the pass stops
with ``budget_stop`` + ``stopped_early``, every completed run preserved. Never-started is the
property that matters: an unfunded project that is merely interrupted mid-run has already
cost model round-trips. Because every member of a wave is checked against the SAME pre-wave
remainder, admission RESERVES each member's requirement as it goes — otherwise a wave of k
would over-commit the cap by up to k runs. This makes membership identical at every
``concurrency``, matching the goal-stop's guarantee above.
- **Before each chat call** (``BudgetMiddleware``'s pre-call guard), a call the remainder cannot
pay for is refused rather than made. This is what bounds a run that was funded at admission
but whose siblings drained the pass while it was in flight; such a run surfaces as a
``RunFailure``, not as overspend.
Spend is carried ACROSS passes by seeding the meter from ``budget.read_spend`` and writing
``budget.write_spend`` afterwards. Those are the caller's calls, not this function's — the pass
reads its ledger, it does not own the file (mirroring the Steg-7 role split)."""
if concurrency < 1:
raise ValueError(
f"concurrency must be >= 1, got {concurrency}: a non-positive wave size would run no "
"projects at all and read as an empty portfolio. Pass 1 for the sequential pass."
)
if portfolio_meter is not None and meter_factory is not None:
raise ValueError(
"portfolio_meter and meter_factory are mutually exclusive: a meter_factory meter is "
"not bound to the portfolio ledger, so the global cap would be silently unenforced"
)
# Startup refusal (fail-fast, before anything loads): a pass that cannot fund its first run
# must not begin. Raised, not returned — an empty PortfolioResult would read as "nothing to do".
if portfolio_meter is not None and not portfolio_meter.can_fund_run():
raise BudgetRefused(portfolio_meter.remaining(), portfolio_meter.required_per_run)
projects = {p.id: p for p in load_reference_projects()}
ids = list(project_ids) if project_ids is not None else list(projects)
store = store if store is not None else VerdictStore(verdicts=[])
# S3.1: the flag is FORWARDED per project rather than installed on the shared store here. The
# store is threaded across every project in the pass (and may be caller-owned), so a pre-loop
# assignment outlived the pass; forwarding keeps the opt-in scoped to each run's own retrievals.
ledger = ledger if ledger is not None else SavingsLedger(entries=[])
goals = goals if goals is not None else GoalConfig()
portfolio_baseline_ore = _baseline_ore(projects[p] for p in ids if p in projects)
runs: list[RunResult] = []
failures: list[RunFailure] = []
stopped_early = False
stop_reason: GoalReached | None = None
budget_stop: BudgetStop | None = None
for wave_ids in _waves(ids, concurrency):
members: list[str] = []
# Tokens committed to members already admitted to THIS wave but that have not spent yet.
# Every member reads the same pre-wave remainder, so without this a wave of k would admit
# k projects off one project's worth of budget.
reserved = 0
for pid in wave_ids:
if pid not in projects:
raise ValueError(f"unknown project_id: {pid!r}")
project = projects[pid]
if goals.portfolio is not None:
observed = ledger.portfolio_total()
limit = _goal_limit_if_reached(goals.portfolio, observed, portfolio_baseline_ore)
if limit is not None:
if goals.portfolio.mode == "hard":
stopped_early = True
stop_reason = GoalReached("portfolio", None, limit, observed)
break
if stop_reason is None:
stop_reason = GoalReached("portfolio", None, limit, observed) # soft flag
per_project_goal = goals.per_project.get(pid)
if per_project_goal is not None:
observed = ledger.per_project_total(pid)
limit = _goal_limit_if_reached(
per_project_goal, observed, _baseline_ore((project,))
)
if limit is not None:
if stop_reason is None:
stop_reason = GoalReached("project", pid, limit, observed)
if per_project_goal.mode == "hard":
continue # skip THIS pid; the rest of the pass proceeds
# S3.4 funding check, placed AFTER the goal checks: a reached goal is success and owns
# the stop when both apply, and a pid a hard per-project goal already skipped costs no
# budget, so it must not consume a reservation.
if portfolio_meter is not None and not portfolio_meter.can_fund_run(reserved=reserved):
stopped_early = True
budget_stop = BudgetStop(
limit_tokens=portfolio_meter.budget.max_total_tokens,
spent_tokens=portfolio_meter.spent,
remaining_tokens=portfolio_meter.remaining(),
required_tokens=portfolio_meter.required_per_run,
)
break
if portfolio_meter is not None:
reserved += portfolio_meter.required_per_run
members.append(pid)
# Every project in the wave reads the SAME wave-start state and writes only its own copy,
# so no two coroutines touch one list. The snapshot is what makes the barrier sufficient.
snapshots = [(pid, _wave_snapshot(store)) for pid in members]
# run_portfolio only drives full runs (never dry-run), so the return narrows to RunResult;
# the cast keeps the widened run_project signature honest without an @overload duplication.
# ``return_exceptions=True`` is the collect-and-continue seam (SC3): one project's
# exception must not cancel its siblings. ``asyncio.TaskGroup`` is deliberately NOT used —
# it cancels the remaining tasks on first exception, which is this policy's exact negation.
wave_results = await asyncio.gather(
*(
run_project(
pid,
profile,
docs_dir=projects[pid].docs_dir,
verdict_input=projects[pid].verdict_input,
bundle_dir=projects[pid].bundle_dir,
verdict_dir=projects[pid].verdict_dir,
dimension=dimension,
store=snapshot,
client_factory=client_factory,
max_rounds=max_rounds,
max_tokens=max_tokens,
top_k=top_k,
semantic_retrieval=semantic_retrieval,
embedder=embedder,
mandate=mandate,
mcp_servers=mcp_servers,
meter=_run_meter(meter_factory, portfolio_meter, max_rounds),
)
for pid, snapshot in snapshots
),
return_exceptions=True,
)
# ``gather`` resolves in ARGUMENT order, not completion order, and waves follow
# ``project_ids`` — so ``runs`` stays in caller order however the schedule interleaved, and
# position is a sound key for pairing each result back to the pid that produced it.
#
# ``strict=True`` is FUTURE-PROOFING and is deliberately untested — measured, not assumed:
# dropping it leaves the whole suite green, because ``gather`` is constructed from exactly
# ``snapshots``, so the two lengths cannot diverge today. A test could only go red by
# manufacturing a mismatch, which would exercise ``zip`` rather than this pass. It earns its
# place against a later edit that filters or extends one sequence without the other — then a
# silent truncation would mis-attribute every result after the gap, and this fails instead.
for (pid, _snapshot), outcome in zip(snapshots, wave_results, strict=True):
# ``BaseException``, NOT ``Exception``, and the width is load-bearing: a member raising
# ``asyncio.CancelledError`` (a BaseException since 3.8) is COLLECTED by
# ``return_exceptions=True`` and must land in ``failures``. Narrowed to ``Exception`` it
# would fall to the ``cast`` below and put a live exception object into ``runs``, which
# dies later in ``_aggregate`` — a crash pointing away from its cause. Gated by
# ``tests/test_portfolio_failure_accounting_loadbearing.py``.
if isinstance(outcome, BaseException):
failures.append(
RunFailure(
project_id=pid, error=str(outcome), error_type=type(outcome).__name__
)
)
else:
runs.append(cast(RunResult, outcome))
# ``snapshots`` is handed over UNFILTERED. ``_merge_wave`` consumes the ORDER this sequence
# arrives in and nothing else, so that order is the contract. Measured honestly: filtering
# the failed members out here would in fact be harmless, because filtering preserves
# relative order — the variant is green. What is NOT harmless is anything that REORDERS
# (or re-pairs) the sequence, and keeping the list whole is what leaves no place for that
# mistake to be written. A failed project's snapshot holds no new verdicts, so it costs
# nothing to pass through.
_merge_wave(store, snapshots)
if stopped_early:
break
base = _aggregate(tuple(runs), store)
if stopped_early or stop_reason is not None or failures or budget_stop is not None:
return replace(
base,
stopped_early=stopped_early,
stop_reason=stop_reason,
failures=tuple(failures),
budget_stop=budget_stop,
)
return base
@dataclass(frozen=True)
class BundleRun:
"""One knowledge base's run inside a multi-base dispatch (§ C.7).
``bundle_id`` and ``project_id`` are carried BESIDE the result rather than read back off it,
because they are what the dispatch ROUTED on: the id is how the mandate named the base, and the
project is what that base's own IR projection says it is. A reader pairing a coverage row back
to a base should not have to re-derive either.
"""
bundle_id: str
bundle_dir: str
project_id: str
result: RunResult
#: The ``run_id`` this base's artefacts were written under, or ``""`` when the pass wrote no
#: outbox at all. Carried here rather than re-derived by the caller for the reason the two
#: fields above are: the CALLER minted it (P17b — N runs need N ids, and this engine refuses
#: to default a key the repo requires a caller to supply), so a reader pairing an artefact
#: back to a base must not have to reconstruct the naming convention to do it.
run_id: str = ""
@dataclass(frozen=True)
class MultiBaseResult:
"""What one multi-base dispatch produced: one run per base the commission named (§ C.7).
A DISTINCT type from ``PortfolioResult``, and deliberately so. ``PortfolioResult`` keys on
"one ``RunResult`` per PROJECT in input order" and its ``runs``/``failures`` partition *the
projects that were actually submitted*; this keys on the KNOWLEDGE BASE the mandate routed each
approach to. Two bases may legitimately describe one project, which the project-keyed shape
cannot express at all — ``run_portfolio`` reads each base off ``projects[pid].bundle_dir``, so
a pid admits exactly one base there. Reusing the type would fuse two axes.
``unreached`` is the ``not_evaluated`` rule at the dispatch layer: when the global cap stops the
pass, every approach in a base that was never started is reported as unreached rather than
omitted. An omitted row is indistinguishable from an approach nobody commissioned, which is the
silence the coverage report exists to remove.
"""
runs: tuple[BundleRun, ...]
store: VerdictStore
stopped_early: bool = False
budget_stop: BudgetStop | None = None
unreached: tuple[ApproachOutcome, ...] = ()
#: Candidates two bases both described (D2). DEFAULTS to an empty tuple, which is the
#: ``skipped_links`` half of the required-vs-default rule and not the
#: ``cost_baseline_anchored`` half: an empty trace is an honest POSITIVE statement ("no
#: candidate was described by two bases"), whereas a missing bool would have to assert
#: something about an event and both assertions would sometimes be untrue.
collisions: tuple[VerdictCollision, ...] = ()
def _coverage_stop_reason(outbox_dir: str, run_id: str) -> str:
"""``BudgetExceeded.kind`` this base recorded, read back off its OWN coverage artefact.
Never recomputed from the ``RunResult``: P19 D2 put "why did this run stop" in
``{run_id}-coverage.json`` precisely because ``_evaluate_mandate`` swallows the exception once
something has been produced, so the dispatcher never sees it. A second derivation here would
be free to disagree with the file the judge reads.
``"absent"`` is a THIRD value, and not the same as ``""``: a base that wrote no coverage file
at all is a different finding from one that finished with nothing stopping it — the
``stress`` judge's own vocabulary, reused rather than re-invented."""
path = Path(outbox_dir) / f"{run_id}-coverage.json"
if not path.is_file():
return "absent"
try:
return str(json.loads(path.read_text(encoding="utf-8")).get("stop_reason", ""))
except (OSError, json.JSONDecodeError):
return "absent"
def _write_multibase_summary(
outbox_dir: str,
run_id: str,
*,
resolved: Sequence[tuple[str, str, str]],
mint: Callable[[str], tuple[str, str]],
multi: MultiBaseResult | None,
) -> None:
"""Write ``{run_id}-multibase.json`` for a multi-base pass, COMPLETED or not (P17b).
Called from a ``finally``, which is ``write_parse_failures``' rule applied one layer up: the
pass that most needs a record of what it spent is the one a cap or a provider cut short, and
the engine's documented limit is that a base which RAISES propagates. Every per-base row is
therefore built from the RESOLUTION and from DISK — the configured bases, the caller's own
minting rule, and each base's own ``{run_id}-coverage.json`` — none of which needs the
dispatch to have returned.
``completed`` is a REQUIRED field of the artefact and not an inference from an empty
``unreached``: ``ExplorationTrace.completed``'s reason verbatim, because "nothing was left
unreached" and "we never found out" must not be the same value. When the pass did not
complete, ``unreached``/``collisions``/``budget_stop`` are what the dispatch never got to say,
and they are written as empty/``None`` UNDER that flag rather than as findings.
"""
rows = []
for bundle_id, bundle_dir, project_id in resolved:
_, base_run_id = mint(bundle_id)
rows.append(
{
"bundle_id": bundle_id,
"bundle_dir": bundle_dir,
"project_id": project_id,
"run_id": base_run_id,
"stop_reason": _coverage_stop_reason(outbox_dir, base_run_id),
}
)
outbox.write_multibase(
outbox_dir,
run_id,
runs=rows,
completed=multi is not None,
unreached=[asdict(row) for row in multi.unreached] if multi is not None else [],
collisions=[asdict(row) for row in multi.collisions] if multi is not None else [],
stopped_early=multi.stopped_early if multi is not None else False,
budget_stop=(
asdict(multi.budget_stop)
if multi is not None and multi.budget_stop is not None
else None
),
)
def resolve_bundle_routing(
bundle_dirs: Sequence[str],
) -> tuple[tuple[str, str, str], ...]:
"""``(bundle_id, bundle_dir, project_id)`` per configured base, in CONFIGURED order.
The ONE resolution shared by ``run_mandate_across_bundles`` and the CLI's multi-base dry run
(P17b). Extracted rather than copied for the reason the id derivation itself was unified in
Step 10: a drill that answered with a different project id, or tolerated a duplicate id the
paid dispatch refuses, would be a free trip that fails to measure the very run it precedes.
``bundle_id`` is ``okf.reconcile_bundle_id``'s — the declared id wins over the mount (S7a-3).
Two bases answering to ONE id refuse here, the same refusal ``explore._bundle_index`` makes
and for the same reason: the id is how the mandate NAMES a base, so a collision would let an
approach be evaluated against A while the report says B (the S3.2 key-collision class).
``project_id`` is S7b søm 1's precedence, and it is load-bearing in BOTH directions: the
hand-written IR projection FIRST (that file is what every existing base has always been routed
by), the base's own DECLARED id as the fallback (an ingested corpus carries no projection, so
file-only could not route it at all). No caller-supplied constant is admitted: it could only
ever be right for one base out of N.
:raises MandateRoutingError: two configured bases share one id.
"""
out: list[tuple[str, str, str]] = []
seen: dict[str, str] = {}
for raw in bundle_dirs:
bundle_id = okf.reconcile_bundle_id(raw).id
if bundle_id in seen:
raise MandateRoutingError(
f"two knowledge bases share the id {bundle_id!r} ({seen[bundle_id]!r} and "
f"{raw!r}); an approach names a base by that id, so it must be unique"
)
seen[bundle_id] = raw
declared_ir = okf.load_optional_ir_projection(raw)
project_id = str(declared_ir["project_id"]) if declared_ir is not None else bundle_id
out.append((bundle_id, raw, project_id))
return tuple(out)
async def run_mandate_across_bundles(
mandate: Mandate,
bundle_dirs: Sequence[str],
profile: Profile | str = Profile.LOCAL,
*,
verdict_input: dict[str, str] | None = None,
store: VerdictStore | None = None,
verdict_dir: str | None = None,
dimension: Dimension | None = None,
client_factory: Callable[[str], BaseChatClient] | None = None,
max_rounds: int = _DEFAULT_MAX_ROUNDS,
max_tokens: int = _DEFAULT_MAX_TOKENS,
top_k: int = 3,
portfolio_meter: PortfolioMeter | None = None,
#: MAJOR-2, threaded unchanged into every per-base ``run_project``. ONE object, never a copy
#: per base: the dispatch is SEQUENTIAL, so a single terminal reviewer composes. Which base is
#: being asked about is visible to the expert because the request carries the approach label
#: AND ``project_id`` — each base's own, read by ``_project_from_bundle`` — so bases can be
#: told apart even when a multi-base commission reuses approach ids.
proposal_reviewer: ProposalReviewer | None = None,
#: P17b. Where THIS base's artefacts go, and under which ``run_id`` — supplied by the caller
#: per base, never minted here. That is the engine's own long-standing contract kept rather
#: than relaxed: N runs need N ``run_id``s, and minting one here would default a key this repo
#: requires a caller to supply, for byte-determinism. A CALLBACK rather than an
#: ``outbox_dir``/``run_id`` pair because the naming rule is an OPERATOR decision
#: (``<run-id>-<bundle_id>``, chosen 14.09) and belongs at the call site that made it; the
#: alternative the order offered — a caller running ``run_project`` itself over
#: ``route_by_bundle``'s sub-mandates — would be a SECOND copy of this loop's id
#: reconciliation, shared store, per-base project resolution, collision accounting and both
#: budget teeth (kø-(p), over five rules that each have exactly one home).
outbox_for: Callable[[str], tuple[str, str]] | None = None,
) -> MultiBaseResult:
"""Evaluate ONE commission across SEVERAL knowledge bases — the multi-base dispatch (§ C.7).
``mandate.route_by_bundle`` partitions the commission by ``Approach.bundle_id``, and this runs
the EXISTING ``run_project`` once per base with that base's own sub-mandate. Composition, never
a widening: ``run_project`` derives the project, the validator's stage-0 baseline, the agents'
read context and the ExpeL query key from THE bundle it is handed, and returns one stamped
``RunResult`` — a second ``bundle_dir`` on that signature would force a silent pick-one for all
four. § C.7 words it the same way: *pipelinen kjøres per bundle som i dag*.
**There is no ``project_id`` argument, and that is the design rather than an omission.** Each
base's project is read from THAT base's own IR projection (``okf.load_ir_projection``), which is
the same value ``_project_from_bundle`` already fail-fasts against — so a caller-supplied
constant could only ever be right for one base out of N. Turning the existing fail-fast into the
routing key removes the guess entirely.
ONE ``VerdictStore`` is threaded across every base, exactly as ``run_portfolio`` threads one
across every project: a verdict captured while evaluating base k must be able to reach base
k+1's hypothesis, and a fresh store per base would leave the loop looking wired while carrying
nothing between the runs.
**The budget has the two S3.4 teeth that apply here, and no more.** With a ``portfolio_meter``:
a remainder that cannot fund one run refuses at STARTUP (``BudgetRefused`` — a pass with room
for zero runs is a caller error, not a result), and a base that cannot be funded is NEVER
STARTED, the pass stopping with ``budget_stop`` and every unreached approach reported. Never
started is the whole point: a base merely aborted mid-run has already cost real model calls.
The wave-reservation tooth has no counterpart here — this dispatch is SEQUENTIAL, so there is
no wave of runs funded off one pre-wave remainder to over-commit.
Honesty limits, stated rather than implied. (1) A base that RAISES propagates; the
collect-and-continue policy belongs to ``run_portfolio``, where the caller submitted a batch of
independent projects, whereas here the caller asked for ONE commission to be evaluated.
(2) Without a ``portfolio_meter`` the pass's ceiling is the number of routed bases times
``max_tokens``, each run bounded on its own — the global ledger is opt-in, and this does not
re-implement it (``_run_meter`` is the one copy of the binding rule). (3) The outbox is
CALLER-KEYED: N runs need N ``run_id``s, and minting them here would default a key this repo
requires a caller to supply, for byte-determinism — so ``outbox_for`` hands each base its
directory and its id, and a caller that offers no callback still writes nothing (every
pre-P17b call site, unchanged).
:raises MandateRoutingError: the commission cannot be routed against ``bundle_dirs``.
:raises BudgetRefused: a global remainder that cannot fund a single run.
"""
resolved = resolve_bundle_routing(bundle_dirs)
by_id = {bundle_id: bundle_dir for bundle_id, bundle_dir, _ in resolved}
project_by_id = {bundle_id: project_id for bundle_id, _, project_id in resolved}
routed = route_by_bundle(mandate, tuple(by_id))
if portfolio_meter is not None and not portfolio_meter.can_fund_run():
raise BudgetRefused(portfolio_meter.remaining(), portfolio_meter.required_per_run)
store = store if store is not None else VerdictStore(verdicts=[])
runs: list[BundleRun] = []
unreached: list[ApproachOutcome] = []
collisions: list[VerdictCollision] = []
#: Which base FIRST produced each verdict id — the map the store deliberately does not keep,
#: and the reason this accounting lives in the dispatcher rather than in ``VerdictStore.add``.
first_base_by_verdict: dict[str, str] = {}
budget_stop: BudgetStop | None = None
for index, (bundle_id, sub_mandate) in enumerate(routed):
if portfolio_meter is not None and not portfolio_meter.can_fund_run():
budget_stop = BudgetStop(
limit_tokens=portfolio_meter.budget.max_total_tokens,
spent_tokens=portfolio_meter.spent,
remaining_tokens=portfolio_meter.remaining(),
required_tokens=portfolio_meter.required_per_run,
)
unreached.extend(
ApproachOutcome(
id=approach.id,
label=approach.label,
status="not_evaluated",
detail=(f"budget exhausted before knowledge base {pending_id!r} was run"),
)
for pending_id, pending in routed[index:]
for approach in pending.approaches
)
break
bundle_dir = by_id[bundle_id]
project_id = project_by_id[bundle_id]
# D2: the id of the verdict THIS base minted, taken from ``run_project``'s existing
# ``notify`` seam rather than off the returned ``RunResult``. ``notify`` fires inside the
# capture block, so it is called exactly when a verdict exists (F2: never when nobody
# reviewed the run) and it carries the minted object, id included. The slot is built FRESH
# each iteration — a shared accumulator would let a later base read the previous base's
# verdict and manufacture a collision that never happened.
minted_here: list[str] = []
base_outbox, base_run_id = outbox_for(bundle_id) if outbox_for is not None else (None, "")
result = cast(
RunResult,
await run_project(
project_id,
profile,
docs_dir=bundle_dir,
bundle_dir=bundle_dir,
outbox_dir=base_outbox,
run_id=base_run_id or None,
notify=lambda verdict: minted_here.append(verdict.id),
verdict_input=verdict_input,
verdict_dir=verdict_dir,
dimension=dimension,
store=store,
client_factory=client_factory,
max_rounds=max_rounds,
max_tokens=max_tokens,
top_k=top_k,
mandate=sub_mandate,
meter=_run_meter(None, portfolio_meter, max_rounds),
proposal_reviewer=proposal_reviewer,
),
)
# D2, dispatcher-side accounting. The condition is the dispatcher's OWN map and nothing
# else: a base that already claimed this id is the only thing that makes "a SECOND base"
# a statement worth making. A Step-7 inbox verdict or a bundle seed sharing the id is NOT
# a cross-base collision, and is excluded exactly by never appearing in this map.
#
# The plan asked for a before/after snapshot of ``{v.id for v in store.verdicts}`` beside
# this. MEASURED redundant and therefore left out: ``store.add`` is first-write-wins and
# never removes, so any id in this map is necessarily in the store when a later base runs.
# The snapshot could not change the outcome of a single dispatch — a conjunct no mutation
# can redden is dead code wearing a guard's clothes, which is the class this repo writes
# rows against rather than ships.
for minted_id in minted_here:
first = first_base_by_verdict.get(minted_id)
if first is None:
first_base_by_verdict[minted_id] = bundle_id
else:
collisions.append(
VerdictCollision(
verdict_id=minted_id,
first_bundle_id=first,
second_bundle_id=bundle_id,
)
)
runs.append(
BundleRun(
bundle_id=bundle_id,
bundle_dir=bundle_dir,
project_id=project_id,
result=result,
run_id=base_run_id,
)
)
return MultiBaseResult(
runs=tuple(runs),
store=store,
collisions=tuple(collisions),
stopped_early=budget_stop is not None,
budget_stop=budget_stop,
unreached=tuple(unreached),
)
# The roles ``debate``/``generate`` ask the factory for. Fixed here so a malformed replies file is
# caught at the door instead of mid-run.
_SCRIPTED_ROLES = ("proposer", "checker")
# The THREE more roles ``explore()`` asks the same factory for (``explore.py:576``) — added to
# ``_SCRIPTED_ROLES`` at the door only when ``--explore`` is in play (MAJOR-2,
# docs/2026-08-25-syretest-vei-ab.md): a plain debate-only run must not be made to answer for
# roles it never uses.
_EXPLORATION_SCRIPTED_ROLES = (MANAGER_ROLE, NAVIGATOR_ROLE, HYPOTHESISER_ROLE)
# The honesty banner for the scripted door. It is a REQUIREMENT, not decoration (målbilde §1):
# a scripted run that reads like a model run is worse than having no offline mode at all, so this
# prints on every scripted invocation and mirrors ``simulation.main``'s banner.
_SCRIPTED_BANNER = (
"=" * 78
+ "\nSCRIPTED OFFLINE RUN — every agent reply is read from your --scripted-replies file."
+ "\nNO MODEL WAS CALLED (ingen modellkall gjort). The context navigation, the debate"
+ "\nplumbing, the deterministic validator and the verdict are real; the agents' answers"
+ "\nare yours, not a model's. This proves the loop closes — not that an LLM would say this."
+ "\n"
+ "=" * 78
)
def _validate_script(role: str, steps: Any, path: str) -> None:
"""Refuse a malformed step script BY NAME (MAJOR-1 b). Validation, NEVER repair.
A step is either a plain reply (``str``) or one tool call
(``{"call": "<tool>", "args": {...}}``). Skipping a step nobody could parse would run a
rehearsal that proves less than the operator wrote while looking exactly like one that proved
it — the failure mode this whole seam exists to make visible. An EMPTY list is refused for the
same reason: it is the constant form written obscurely, and it would fall through to the
default reply.
"""
if not isinstance(steps, list) or not steps:
raise ValueError(f"--scripted-replies[{role!r}] must be a non-empty list of steps ({path})")
for index, step in enumerate(steps):
if isinstance(step, str):
continue
if not isinstance(step, dict) or not isinstance(step.get("call"), str):
raise ValueError(
f"--scripted-replies[{role!r}] step {index} is neither a text reply nor a tool "
f'call {{"call": "<tool>", "args": {{...}}}} ({path})'
)
unknown = sorted(set(step) - {"call", "args"})
if unknown:
raise ValueError(
f"--scripted-replies[{role!r}] step {index} names unknown key(s) "
f"{', '.join(unknown)}; a step carries only 'call' and 'args' ({path})"
)
if step.get("args") is not None and not isinstance(step["args"], dict):
raise ValueError(
f"--scripted-replies[{role!r}] step {index}: 'args' must be an object ({path})"
)
def _load_scripted_replies(
path: str, required_roles: Sequence[str] = _SCRIPTED_ROLES
) -> dict[str, Any]:
"""Load the caller's scripted answers, fail-fast. Every role ``required_roles`` names must be
present AND readable: a missing role would otherwise surface as a ``KeyError`` deep inside
``scripted_factory``'s lookup, mid-run, long after the run appeared to start cleanly (MAJOR-2:
measured for the three ``explore()`` adds on top of the debate's own two).
**ANY role may be given a step LIST** (MAJOR-1 b), because a single constant string can never
emit a ``function_call`` — measured: 0 tool calls / 0 approaches / 1 round on 4/4 bases, an
offline rehearsal that was vacuous by construction.
Until S2c the list form was REFUSED for the debate's two roles, on the stated ground that "the
proposer answers ``generate``'s own call, not an agent loop that would invoke a tool between
turns, so a script of calls there describes a rehearsal that cannot happen". That ground is now
measurably false: the debate's proposer and checker are ``Agent``s in a GroupChat and they hold
the four navigator tools, so a constant-string rehearsal proves the debate RUNS while proving
nothing about whether it OPENS the base — which is the identical vacuity MAJOR-1 closed one
surface over. Keeping the refusal would have made the free half of the measurement ladder
unable to reach the very seam S2c builds.
**Honesty limit, stated rather than encoded.** Scripts are per-CLIENT and each client gets its
own copy, so a ``proposer`` script is consumed once by the DEBATE client and again, from the
start, by the fresh client ``generate_via_llm`` builds. A script whose first step is a tool
call therefore answers the generation call with a ``function_call`` too, which will not parse.
That is the operator's to write correctly: guessing which steps were "meant for" which call
site would be repair, and this loader validates.
The same limit binds ``--proposal-review`` (MAJOR-2), where it is easiest to trip: each
``revise`` the piped answers request buys ONE more generation attempt, so a proposer script
one entry short of them degrades to the selector's default reply, which never parses — and the
round ledger fires on the parse-retry. Write the script long enough."""
try:
raw = json.loads(Path(path).read_text(encoding="utf-8"))
except FileNotFoundError as exc:
raise ValueError(f"--scripted-replies file not found: {path}") from exc
except json.JSONDecodeError as exc:
raise ValueError(f"--scripted-replies is not valid JSON ({path}): {exc}") from exc
if not isinstance(raw, dict):
raise ValueError(f"--scripted-replies must be a JSON object of role -> reply ({path})")
missing = [r for r in required_roles if not isinstance(raw.get(r), (str, list))]
if missing:
raise ValueError(
f"--scripted-replies needs a reply for each of {', '.join(required_roles)}; "
f"missing or of an unusable type: {', '.join(missing)} ({path})"
)
for role in required_roles:
if isinstance(raw[role], str):
continue
_validate_script(role, raw[role], path)
return {role: raw[role] for role in required_roles}
def main(argv: list[str] | None = None) -> int:
"""Single-command console entry: run the slice for one project against a docs folder."""
import argparse
import asyncio
import sys
parser = argparse.ArgumentParser(description="portfolio-optimiser vertical slice")
# project_id + --docs-dir are relaxed from required to a mode-conditional refusal (below): the
# single-project path still requires both, but portfolio mode takes neither. The compensating
# guard keeps the legacy contract failing loudly (rc 1 refusal) instead of via argparse exit 2.
parser.add_argument("project_id", nargs="?", default=None)
parser.add_argument("--profile", default="local")
parser.add_argument("--docs-dir", default=None)
parser.add_argument(
"--bundle-dir", default=None, help="OKF bundle dir (enables the Step-1 fold)"
)
parser.add_argument(
"--across-bundle",
action="append",
default=None,
metavar="DIR",
help="P17b: run ONE commission across SEVERAL knowledge bases — repeat the flag once per "
"base. The mandate is partitioned by each approach's bundle_id and the existing pipeline "
"runs once per base, sequentially, threading ONE verdict store so a verdict minted against "
"base k reaches base k+1. Each base writes its own artefact set under <run-id>-<bundle_id>, "
"plus one <run-id>-multibase.json summary. Requires --mandate, --run-id and --outbox-dir; "
"--bundle-dir stays ONE directory and is refused here",
)
parser.add_argument(
"--verdict-dir",
default=None,
help="async verdict inbox: a folder of dropped expert verdicts, ingested before generation "
"(the long loop — a verdict that landed after an earlier run is consumed by this run)",
)
parser.add_argument(
"--dimension-config",
default=None,
help="fail-fast dimension scope config (JSON): scopes the run to one cost axis; a "
"missing or malformed file refuses the run (authoritative startup config, not a RAW inbox)",
)
parser.add_argument(
"--mandate",
default=None,
metavar="FILE",
help="run mandate (JSON, fail-fast): what a domain expert commissions this run to "
"evaluate — named approaches and/or the system's own — plus the objective and success "
"criteria. The run ANNOUNCES it before the first model call and SETTLES against it "
"afterwards, one row per approach. Valid in both modes; in portfolio mode it applies to "
"every project in the pass",
)
parser.add_argument(
"--explore",
default=None,
metavar="PROMPT",
help="U4 opt-in: run a Magentic EXPLORATION over the knowledge base first and let it shape "
"the mandate this run then evaluates. The exploration chooses which base to open and which "
"directions are worth testing; every number it produces is still gated by the same "
"deterministic validator, and the exploration itself writes nothing. REQUIRES "
"--explore-config and --bundle-dir; refused together with --mandate (two sources of one "
"mandate)",
)
parser.add_argument(
"--explore-config",
default=None,
metavar="FILE",
help="the exploration's bounds (JSON, fail-fast, REQUIRES --explore): max_rounds, "
"max_tokens, max_stall_count, max_reset_count, max_plan_revisions, enable_plan_review. "
"Every field is required and none has a default — an omitted bound would fall back to "
"MAF's unbounded loop, not to something conservative. enable_plan_review requires "
"--plan-review, which is what answers it",
)
parser.add_argument(
"--plan-review",
action="store_true",
help="U13 synchronous HITL door (REQUIRES --explore, and --explore-config must set "
"enable_plan_review): answer the exploration's plan review AT THIS TERMINAL. Before the "
'loop is allowed to run you are shown the plan and answer "approve" or "revise <what to '
'change>"; a revision goes back to the manager, which replans and asks you again. Every '
"round trip is recorded in {run_id}-exploration.json, feedback verbatim. Input that ends "
"without an answer is an error, NEVER a sign-off",
)
parser.add_argument(
"--prepass-payload",
default=None,
metavar="FILE",
help="Hand the debate a DECLARED CUT of the knowledge base instead of the pointer and the "
"four navigator tools. FILE is one contract-conformant OKF consumption pre-pass payload "
"(okf-consumption/1): the delivered excerpts plus the three denominators — how many "
"concepts were considered, how many withheld, and by which rule. Every excerpt is verified "
"against the mounted document (identity, sha256, and the delivered text re-derived from "
"the base) before a single model call, so a payload cannot deliver bytes the base does not "
"hold; the verdict layer and the dimension scope are refused here exactly as the navigator "
"tools refuse them. The cut is recorded in {run_id}-prepass.json and printed. This makes "
"the run's cut DECLARED rather than emergent; it does not make it cheaper. Requires "
"--bundle-dir; refused with --portfolio, --report, --proposals-from-mandate, "
"--dimension-config and --explore.",
)
parser.add_argument(
"--prepass-seed",
default=None,
metavar="FILE",
help="The OTHER arm of the same door (Q5=B): hand the EXPLORATION a declared cut as its "
"STARTING POINT and KEEP the four navigator tools. FILE is one contract-conformant OKF "
"consumption pre-pass payload (okf-consumption/1), admitted by exactly the checks "
"--prepass-payload applies — identity, sha256, the delivered text re-derived from the "
"mounted document, the verdict layer — before a single model call. The difference is what "
"happens next: the cut and its three denominators join the exploration's task message, "
"and the loop may still open anything else in the base (contract SS 2.2 permits what the "
"payload 'explicitly names as reachable'). The cut is printed and recorded in "
"{run_id}-exploration.json, which says rest_reachable so a reader can tell a seeded run "
"from a bounded one. REQUIRES --explore; refused with --prepass-payload (two opposite "
"arms of one decision), --portfolio, --report and --checkpoint-dir.",
)
parser.add_argument(
"--proposal-review",
action="store_true",
help="MAJOR-2 synchronous HITL door: after the deterministic validator ACCEPTS a "
'candidate, show it AT THIS TERMINAL and read "approve" or "revise <what to change>". A '
"revision buys ONE more attempt under the existing attempt/round budget — no new loop and "
"no new cap — and the words go into the next hypothesis prompt verbatim. Every answer is "
"recorded in {run_id}-proposal-reviews.json. Input that ends without an answer STOPS the "
"run, NEVER a sign-off. An approve is not an expert verdict (that still arrives via "
"--decision/--rationale or the verdict inbox). Refused with --portfolio, --report, "
"--live-dry-run, --proposals-from-mandate and --checkpoint-dir (pass it at --resume "
"instead); it composes with --explore --plan-review, which reads two doors from one stdin "
"in sequence",
)
parser.add_argument(
"--checkpoint-dir",
default=None,
metavar="DIR",
help="U12 ASYNCHRONOUS HITL door (REQUIRES --explore, --explore-config with "
"enable_plan_review, --run-id and --outbox-dir; refused together with --plan-review): "
"instead of blocking on a human at this terminal, park the exploration's plan review to "
"disk. The workflow's checkpoints go here and the question goes to "
"{run_id}-plan-review.json in the outbox; an expert answers days later by dropping "
"{run_id}-plan-review-answer.json into a review inbox, and --resume picks it up",
)
parser.add_argument(
"--review-inbox",
default=None,
metavar="DIR",
help="where the expert drops their answer to a parked plan review (READ-only, and never "
"the same folder as --outbox-dir: a run that read its own output as input would be "
"answering itself). Required by --resume",
)
parser.add_argument(
"--resume",
default=None,
metavar="RUN_ID",
help="resume the exploration parked under RUN_ID (REQUIRES --checkpoint-dir and "
"--review-inbox): read the open question from the outbox, the answer from the review "
"inbox, and drive the exploration onward in THIS process. The prompt, the bounds and the "
"knowledge bases are read from the parked state, not from argv — the workflow has to be "
"rebuilt exactly as it was for the checkpoint to be accepted at all. A revision makes the "
"manager replan and park a NEW question; an approval lets the run continue into the "
"pipeline as usual",
)
parser.add_argument(
"--mcp-config",
default=None,
metavar="FILE",
help="external MCP servers this run may contact (JSON, fail-fast): name, transport "
"(stdio|http), coordinates, the ALLOWED tool names, a timeout, and optionally the NAME of "
"an env var holding the credential (never the credential itself). Every server and tool is "
"named in the run announcement BEFORE the first call; without this flag no network call is "
"made at all",
)
parser.add_argument(
"--embedder-config",
default=None,
help='fail-fast embedder config (JSON, e.g. {"type": "fake"}): selects the embedder '
"used by --semantic-retrieval from a CLOSED registry. Not an import path — a config file "
"can never name arbitrary code to load (a new embedder is added as a registry branch)",
)
parser.add_argument(
"--outbox-dir",
default=None,
help="RAW outbox dir for the run's proposal/outcome artefacts (REQUIRES --run-id). Should "
"differ from --verdict-dir: the two folders have OPPOSITE ownership (the system writes the "
"outbox, the expert writes the inbox). Measured: sharing one folder is inert TODAY, because "
"the outbox artefacts are named {run_id}-*.json and carry none of the verdict keys, so the "
"tolerant inbox loader skips them — the hazard is a future verdict-shaped artefact in the "
"outbox being re-ingested past the Step-8 promotion gate (self-contamination). Not "
"CLI-enforced: there is no reachable contamination to refuse",
)
parser.add_argument(
"--run-id",
default=None,
help="stable run id for --outbox-dir artefacts (required when --outbox-dir is set; no "
"wall-clock/uuid default — the outbox artefacts are byte-deterministic)",
)
parser.add_argument(
"--portfolio",
action="store_true",
help="portfolio mode: dispatch to run_portfolio over all reference projects (or the single "
"given PROJECT_ID). Takes --goals/--ledger/--dimension-config; the single-project-only flags "
"are refused in this mode (the two CLI modes are a documented partition)",
)
parser.add_argument(
"--goals",
default=None,
help="portfolio mode: goal config JSON (fail-fast) — the GoalReached stop is checked against "
"the ledger before each project",
)
parser.add_argument(
"--ledger",
default=None,
help="portfolio mode: accumulated savings ledger JSON (fail-fast) read for the goal-stop "
"(earlier out-of-band HITL realizations — never built during the pass)",
)
parser.add_argument(
"--semantic-retrieval",
action="store_true",
help="S3.1 opt-in scaling SEAM: blend a cosine term over the embedded feature triple with "
"the structural score, so a prior verdict on a DIFFERENT cost-code set can outrank one that "
"ties structurally. The shipped embedder is a semantics-free sha256 projection — this buys "
"the extension point, not better retrieval; inject a real one with --embedder-config. "
"Accepted in both modes, but in single-project mode it REQUIRES --bundle-dir and "
"--verdict-dir (without them it cannot take effect, and is refused rather than ignored). "
"OFF by default, and off means the structural ranking is unchanged",
)
# F2 (non-goal 3): NO defaults. Silence means nobody reviewed the run, and the previous
# ``approved``/``reviewed by expert`` pair minted an expert judgement out of that silence —
# which then propagated into the next project's hypothesis prompt as a prior verdict. The two
# belong together: half a verdict is refused by name below, never completed on the expert's
# behalf.
parser.add_argument(
"--decision",
default=None,
choices=["approved", "rejected"],
help="the expert's recorded decision for this run. Omit it when nobody reviewed the run — "
"no verdict is then minted, nothing enters the learning store, and the summary line says "
"so. Requires --rationale",
)
parser.add_argument(
"--rationale",
default=None,
help="the expert's reasoning behind --decision (required with it; an expert verdict is a "
"decision AND its reasoning)",
)
parser.add_argument(
"--max-rounds",
type=int,
default=_DEFAULT_MAX_ROUNDS,
help=(
"per-run round cap (P16 B2). DEFAULTS to what every CLI run was implicitly bound to "
"before this flag existed, so nothing changes unless it is stated"
),
)
parser.add_argument(
"--max-tokens",
type=int,
default=_DEFAULT_MAX_TOKENS,
help="per-run token cap (P16 B2). Same default, same reason as --max-rounds",
)
parser.add_argument(
"--live-dry-run",
action="store_true",
help="offline drill: build contracts/clients/budget, STOP before the first model call",
)
parser.add_argument(
"--derive-cost-baseline",
action="store_true",
help=(
"derive the validator's cost baseline from a priced schedule inside --bundle-dir "
"instead of loading a hand-written cost-baseline.json (MAJOR-4). Refuses rather than "
"guesses: an unpriced or ambiguous schedule stops the run"
),
)
parser.add_argument(
"--require-cost-baseline",
action="store_true",
help=(
"REFUSE the run unless the validator's stage 0 has a cost baseline to reconcile "
"against (F4). Opt-in: a bundle that ships none is legitimately un-anchored and runs "
"unchanged without this flag. Combine with --derive-cost-baseline to satisfy it from "
"a priced schedule inside the base"
),
)
parser.add_argument(
"--proposals-from-mandate",
action="store_true",
help=(
"build each candidate DETERMINISTICALLY from --mandate and the schedule "
"--derive-cost-baseline reads, and judge it with the ordinary validator — ZERO model "
"calls (S7b). The expert supplies the measure, the cost codes and the estimate; the "
"document supplies the quantities and prices. Requires --mandate and "
"--derive-cost-baseline. Refuses rather than invents: an approach with no estimate or "
"no affected_codes stops the run by name"
),
)
parser.add_argument(
"--scripted-replies",
default=None,
metavar="FILE",
help="offline WHOLE-LOOP run over your own bundle with ZERO model calls: FILE is JSON "
'{"proposer": "<reply>", "checker": "<reply>"} and those fixed strings stand in for every '
"model answer. Unlike --live-dry-run (which stops before the first call) the complete loop "
"runs — hypothesis, debate, deterministic validator, verdict. The answers are yours, not a "
"model's, and the run says so on every invocation",
)
parser.add_argument(
"--report",
action="store_true",
help="S5.4 read-only value report: roll up the --ledger's realized savings (per-project + "
"portfolio totals, flagged cross-dimension overlaps, per-entry provenance) to stdout. "
"Mode-exclusive: only --ledger/--json are permitted alongside it; makes NO model calls",
)
parser.add_argument(
"--json",
action="store_true",
help="value report output form (requires --report): emit the roll-up as deterministic JSON "
"instead of the human table",
)
args = parser.parse_args(argv)
# U14: the tracing seam, resolved FIRST — ahead of every branch that can return, because MAF's
# contract is "call once at startup, before any telemetry is captured". Without PORTFOLIO_OTEL
# this configures nothing at all and prints nothing (omission, never an empty row), so every
# existing stderr expectation in the suite is untouched. A malformed request exits through this
# CLI's own refusal surface (printed line + rc 1) rather than as a traceback: it is something
# the operator exported and can fix, which is exactly what that surface is for.
try:
tracing_setup = configure_tracing()
except TracingConfigError as exc:
print(f"run refused: {exc}", file=sys.stderr)
return 1
tracing_line = tracing_notice(tracing_setup)
if tracing_line is not None:
print(tracing_line, file=sys.stderr)
# F2: half a verdict is refused BY NAME, before every mode dispatch below — an expert verdict
# is a decision AND its reasoning, and defaulting the missing half is exactly the seam that let
# an approval nobody spoke enter the learning store. Placed here (ahead of report mode, which
# RETURNS) so the pairing holds on every path, not only the ones that run a model.
if (args.decision is None) != (args.rationale is None):
given, absent = (
("--decision", "--rationale")
if args.decision is not None
else ("--rationale", "--decision")
)
print(
f"run refused: {absent} is required together with {given} (an expert verdict is a "
"decision AND its reasoning; omit BOTH when nobody reviewed the run)",
file=sys.stderr,
)
return 1
# S5.4: read-only value-report dispatch — placed FIRST (right after parse_args, BEFORE the
# mode-exclusivity block below) so it returns before any model/portfolio path can start and no
# later branch can shadow it (the bare `--ledger`-outside-portfolio refusal at the elif below is
# left UNCHANGED — a bare --ledger with no --report still flows there and refuses as before).
if args.json and not args.report:
# A stray --json is never silently ignored (honors S5.3's "refused, never ignored" partition).
print("run refused: --json requires --report", file=sys.stderr)
return 1
if args.report:
# Mode-exclusivity as an ALLOWLIST (not a short blocklist): report mode permits ONLY --ledger
# and --json; ANY other distinguishable mode/config flag is refused — else --report --goals
# would silently drop --goals, whereas bare --goals is refused below (adding --report must not
# suppress an existing refusal). --decision/--rationale ARE listed now: before F2 their
# non-None argparse defaults made an explicit value indistinguishable from the default, so
# an honest refusal was unimplementable and they had to be excluded. With the defaults gone
# they are distinguishable, and an operator who typed a real expert verdict must not have
# it silently dropped — 'refused, never ignored' is this partition's own rule.
report_forbidden = {
"--portfolio": args.portfolio,
"--live-dry-run": args.live_dry_run,
# Report mode returns before the run dispatch, so an omission here is a SILENT DROP,
# not a refusal — the gap F4 measured on --plan-review.
"--derive-cost-baseline": args.derive_cost_baseline,
# Same reason, same rung: report mode returns above every run dispatch, so leaving it
# out is a SILENT DROP of a guarantee the operator asked for by name.
"--require-cost-baseline": args.require_cost_baseline,
# Same reason, one flag later: report mode returns above the S7b dispatch too.
"--proposals-from-mandate": args.proposals_from_mandate,
"PROJECT_ID": args.project_id is not None,
"--goals": args.goals is not None,
"--docs-dir": args.docs_dir is not None,
"--bundle-dir": args.bundle_dir is not None,
# P17b, and for its neighbours' reason: report mode returns ABOVE every dispatch,
# including the multi-base one, so an omission here is a SILENT DROP of a whole pass
# rather than a refusal (the F4 class).
"--across-bundle": bool(args.across_bundle),
"--verdict-dir": args.verdict_dir is not None,
"--outbox-dir": args.outbox_dir is not None,
"--run-id": args.run_id is not None,
"--dimension-config": args.dimension_config is not None,
"--semantic-retrieval": args.semantic_retrieval,
"--embedder-config": args.embedder_config is not None,
"--scripted-replies": args.scripted_replies is not None,
"--explore": args.explore is not None,
"--explore-config": args.explore_config is not None,
# Report mode returns BELOW, before every exploration refusal, so a flag missing from
# this list is silently dropped rather than refused — which is the whole reason the
# list enumerates every distinguishable flag instead of the ones that would misbehave.
"--plan-review": args.plan_review,
# MAJOR-2, and for the identical reason: report mode returns above the run dispatch, so
# an omission here is a SILENT DROP — the door would be accepted and never asked.
"--proposal-review": args.proposal_review,
# The pre-pass door, listed for the same reason as its neighbours: report mode returns
# ABOVE every dispatch, so an omission here is a silent DROP — the operator would be
# told nothing and the declared cut would simply never happen.
"--prepass-payload": args.prepass_payload is not None,
# And the seeding arm, listed for the identical reason: report mode returns above the
# exploration dispatch too, so an omission here is a silent DROP and not a refusal.
"--prepass-seed": args.prepass_seed is not None,
# The three U12 flags, listed for exactly that reason: report mode returns before the
# resume dispatch, so an omission here is a silent drop, not a refusal.
"--checkpoint-dir": args.checkpoint_dir is not None,
"--review-inbox": args.review_inbox is not None,
"--resume": args.resume is not None,
# Distinguishable only since F2 removed their defaults. They are refused TOGETHER above
# when only one is given, so at most one situation reaches this list: both set.
"--decision": args.decision is not None,
"--rationale": args.rationale is not None,
# DEL C, the side-finding økt 82 measured and reported rather than fixed: --mandate is
# OLDER than this partition and never got a row, so ``--report --ledger X --mandate Y``
# dropped the commission in SILENCE — announced nothing, settled nothing, exit 0. The
# F4 class exactly: report mode returns above every dispatch, so an omission here is a
# silent drop and not a refusal.
"--mandate": args.mandate is not None,
# P16 B2, same rung: a cap stated in report mode binds nothing, and report mode returns
# above the dispatch that would have honoured it. Distinguishable only because the
# argparse default IS the historic value, so "stated" and "omitted" differ.
"--max-rounds": args.max_rounds != _DEFAULT_MAX_ROUNDS,
"--max-tokens": args.max_tokens != _DEFAULT_MAX_TOKENS,
}
if any(report_forbidden.values()):
print(
"run report refused: mode-exclusive (only --ledger/--json permitted with --report)",
file=sys.stderr,
)
return 1
if not args.ledger:
# Guards SavingsLedger.load(None) -> Path(None) TypeError (NOT in the load except tuple).
print("run report refused: --report requires --ledger <file>", file=sys.stderr)
return 1
try:
# `report_ledger`, not `ledger`: the portfolio branch below binds `ledger` as
# `SavingsLedger | None`, so reusing that name here (type `SavingsLedger`) collides on
# mypy's function-scoped declared type.
report_ledger = SavingsLedger.load(args.ledger)
except (FileNotFoundError, ValidationError, ValueError) as exc:
# A load failure must never masquerade as a real zero-savings result (SC5): stderr + rc 1,
# no table. Only a successfully-loaded (possibly empty) ledger prints.
print(f"run report refused: {exc}", file=sys.stderr)
return 1
rep = build_value_report(report_ledger) # NB: `rep`, not `report` (`report` is bound below)
print(dump_report_json(rep) if args.json else format_report_text(rep))
return 0
# Step 4: mode-exclusivity validation (structured refusal, NOT argparse.error — keeps the rc 1
# refusal contract). The two CLI modes are a documented partition: single-project-only flags are
# refused in portfolio mode, and --goals/--ledger are refused outside it — never silently ignored.
# --decision/--rationale are single-project-only and REFUSED in portfolio mode since F2: a pass
# takes each project's verdict from its OWN row, so a run-level verdict flag has nowhere to go
# and silently dropping a real expert judgement is the thing this partition exists to prevent.
# Before F2 their non-None argparse defaults made an explicit value indistinguishable from the
# default and an honest refusal was unimplementable; that is no longer true.
# --dimension-config is valid in BOTH modes.
if args.portfolio:
single_only = {
"--docs-dir": args.docs_dir,
"--bundle-dir": args.bundle_dir,
# P17b. A portfolio pass keys on PROJECTS and reads each project's base off its own
# row, so a run-level list of bases has nowhere to go; the two are different axes
# (``MultiBaseResult`` is a distinct type from ``PortfolioResult`` for exactly that
# reason). BY NAME, like its neighbours: falling through to "--across-bundle requires
# --mandate" would tell an operator who wrote --portfolio --across-bundle to add a
# flag that is legal in both modes, which answers the wrong question.
"--across-bundle": bool(args.across_bundle),
"--verdict-dir": args.verdict_dir,
"--outbox-dir": args.outbox_dir,
"--run-id": args.run_id,
"--live-dry-run": args.live_dry_run,
# BY NAME, not by falling through to "--derive-cost-baseline requires --bundle-dir":
# --bundle-dir is already single-project-only, so that message would tell an operator
# who wrote --portfolio --derive-cost-baseline to add the one flag this mode also
# refuses. Same reason --explore is listed here rather than left to fall through.
"--derive-cost-baseline": args.derive_cost_baseline,
# It guards ONE base's anchoring, and the portfolio pass takes the road path, which is
# anchored by construction — so here the flag could only ever pass. BY NAME rather
# than falling through to the --bundle-dir requirement, its neighbours' reason.
"--require-cost-baseline": args.require_cost_baseline,
# It reads ONE base's schedule and settles ONE commission against it, so it sits on the
# same side of the partition as the flag it requires. BY NAME rather than falling
# through to "requires --derive-cost-baseline": an operator who wrote --portfolio
# --proposals-from-mandate must not be told to add a flag this mode also refuses.
"--proposals-from-mandate": args.proposals_from_mandate,
# One exploration shapes ONE mandate against ONE knowledge base, and --bundle-dir (its
# only source of bases here) is already single-project-only. Refusing it by NAME beats
# letting it fall through to the --bundle-dir requirement below: an operator who wrote
# --portfolio --explore has to hear which of the two is wrong.
"--explore": args.explore,
"--explore-config": args.explore_config,
# It answers --explore's review, so it lives on the same side of the partition. By
# NAME for the same reason --explore is: falling through to "--plan-review requires
# --explore" would tell an operator who wrote --portfolio --plan-review to add the one
# flag this mode also refuses.
"--plan-review": args.plan_review,
# MAJOR-2: one terminal, and a portfolio pass interleaves its waves as coroutines —
# several projects' candidates would arrive at the same prompt with nothing to tell
# them apart. BY NAME, like its neighbours: falling through to a --bundle-dir
# requirement would tell the operator to add a flag this mode also refuses.
"--proposal-review": args.proposal_review,
# ONE payload is a cut of ONE base at ONE ref, and --bundle-dir (its only source of a
# base here) is already single-project-only. BY NAME rather than falling through to
# the --bundle-dir requirement below, whose message ALSO names --prepass-payload: an
# operator who wrote --portfolio --prepass-payload has to hear which of the two is
# wrong, and an arm asserting on the shared token could not tell the two apart.
"--prepass-payload": args.prepass_payload,
# The seeding arm sits on the same side and by NAME for the same reason: it requires
# --explore, which this mode also refuses, so falling through would tell an operator
# who wrote --portfolio --prepass-seed to add a second flag --portfolio refuses too.
"--prepass-seed": args.prepass_seed,
# And the asynchronous half of the same door, on the same side of the partition and by
# NAME for the same reason.
"--checkpoint-dir": args.checkpoint_dir,
"--review-inbox": args.review_inbox,
"--resume": args.resume,
# See the block comment above: distinguishable only since F2.
"--decision": args.decision,
"--rationale": args.rationale,
}
offending = [name for name, value in single_only.items() if value]
if offending:
print(
f"portfolio run refused: {', '.join(offending)} belong to single-project mode, "
"not --portfolio (the two CLI modes are a documented partition)",
file=sys.stderr,
)
return 1
elif args.goals is not None or args.ledger is not None:
print(
"run refused: --goals/--ledger require --portfolio mode",
file=sys.stderr,
)
return 1
# P17b — the multi-base door's own refusals, at FUNCTION level and never nested under another
# flag's branch, for the F4 reason its neighbours are: under one, a bare combination falls
# straight through to a dispatch that drops the flag in silence. Placed ABOVE the required-args
# guard because this mode takes NO ``PROJECT_ID`` at all — each base's project is read from
# THAT base's own IR projection, which is the whole reason the dispatch has no such parameter.
if args.across_bundle:
# The three things a multi-base pass cannot invent. ``--mandate`` because the commission IS
# the partition key (without ``Approach.bundle_id`` there is nothing to route on), and the
# outbox pair because N runs need N ``run_id``s: the engine refuses to default that key,
# so the caller must supply the stem it mints them from.
required = {
"--mandate": args.mandate,
"--run-id": args.run_id,
"--outbox-dir": args.outbox_dir,
}
missing = [name for name, value in required.items() if not value]
if missing:
print(
f"run refused: --across-bundle requires {', '.join(missing)} (the commission is "
"what partitions the pass by base, and each base writes its own artefact set "
"under <run-id>-<bundle_id> — a key this repo requires a caller to supply)",
file=sys.stderr,
)
return 1
# Four single-base modes, each refused BY NAME rather than by falling through. Every one
# of them resolves something from THE base — one directory, one cut, one exploration, one
# derived schedule — and silently picking which of N that means is the guessed-shape class
# this repo refuses outright.
conflicting = {
"--bundle-dir": args.bundle_dir,
"--explore": args.explore,
"--prepass-payload": args.prepass_payload,
"--proposals-from-mandate": args.proposals_from_mandate,
}
clash = [name for name, value in conflicting.items() if value]
if clash:
print(
f"run refused: --across-bundle cannot be combined with {', '.join(clash)} (each "
"of those resolves ONE knowledge base — a directory, a declared cut, an "
"exploration or a derived schedule — and this mode configures several; which one "
"was meant is not something this layer may decide)",
file=sys.stderr,
)
return 1
# Single-project mode requires PROJECT_ID + --docs-dir (compensating for the relaxed argparse
# required/positional so the legacy contract keeps failing loudly via the refusal surface).
# HOISTED above the scripted door (below) so an incomplete argv is refused BEFORE the honesty
# banner could claim a scripted run happened; the refusal ORDER within single-project mode
# (required args -> semantic-retrieval -> scripted) is unchanged.
#
# ``not args.across_bundle`` is a MODE test, not a relaxation: the multi-base pass takes no
# PROJECT_ID and no single ``--bundle-dir``, and both of those are refused above by name.
if (
not args.portfolio
and not args.across_bundle
and (args.project_id is None or (args.docs_dir is None and args.bundle_dir is None))
):
print(
"run refused: single-project mode requires PROJECT_ID and either --docs-dir or "
"--bundle-dir (use --portfolio for portfolio mode)",
file=sys.stderr,
)
return 1
# P18/C1 (P16 FUNN 2): on the BUNDLE path ``docs_dir`` is never read — retrieval, the chunk
# tool and the citation check all live in the road branch — yet the guard above demanded it,
# so the documented command had to name the same directory twice. It is now bound ONCE, from
# ``--bundle-dir`` when ``--docs-dir`` is absent, which is byte-identically what the README
# tells an operator to type by hand; every existing invocation, including the two-flag form,
# is unchanged. This is NOT the "--docs-dir omvei" (feeding project documents through
# retrieval INSTEAD of ingesting them into a knowledge base): no such path is opened, and the
# road branch still refuses without a real ``--docs-dir``.
docs_dir = args.docs_dir if args.docs_dir is not None else args.bundle_dir
# The third projection reads a table INSIDE a bundle, so without one there is nothing to derive
# from: the road path's baseline comes from ``Project.cost_items`` and is anchored by
# construction. Refused by NAME here rather than left to surface later as a project-lookup
# failure, which names neither the flag nor what it needs.
if not args.portfolio and args.derive_cost_baseline and args.bundle_dir is None:
print(
"run refused: --derive-cost-baseline requires --bundle-dir (the schedule it derives "
"from is a concept file in the knowledge base; the road path is already anchored by "
"its own cost_items)",
file=sys.stderr,
)
return 1
# The road path's baseline comes from ``Project.cost_items`` and is anchored by construction,
# so without a bundle this flag could never fire — and a flag that cannot fire is a claim the
# surface makes about itself (the Fase-3 class). Refused BY NAME, its neighbour's reason.
if not args.portfolio and args.require_cost_baseline and args.bundle_dir is None:
print(
"run refused: --require-cost-baseline requires --bundle-dir (the road path is already "
"anchored by its own cost_items, so on it the requirement could never fire)",
file=sys.stderr,
)
return 1
# The SEEDING arm's three refusals, placed ABOVE the replacing arm's block on purpose: given
# both flags, the block below would answer with "--prepass-payload and --explore cannot be
# combined", which names neither of the two flags the operator actually put in conflict. At
# FUNCTION level, never nested under another flag's branch, for the F4 reason its neighbours
# are: under one, a bare combination falls through to a dispatch that drops the flag silently.
if not args.portfolio and args.prepass_seed is not None:
if args.prepass_payload is not None:
# TWO OPPOSITE ARMS OF ONE DECISION, and merging them is not defined: one WITHDRAWS
# the navigator tools because the cut replaces the base, the other KEEPS them because
# the cut is where to start. A run holding both would have to silently pick, and the
# picked one would be a policy nobody wrote down.
print(
"run refused: --prepass-payload and --prepass-seed are two opposite arms of one "
"decision (the first REPLACES the knowledge base with the cut and withdraws the "
"navigation tools; the second uses the cut as a STARTING POINT and keeps them). "
"Choose which one this run is",
file=sys.stderr,
)
return 1
if args.explore is None:
# The ``--explore-config`` case verbatim: a seed for a loop that never runs would be
# loaded, verified, paid for in I/O and then dropped.
print(
"run refused: --prepass-seed requires --explore (the cut seeds an EXPLORATION's "
"starting point; without one there is no loop to seed, and the debate's own arm "
"is --prepass-payload)",
file=sys.stderr,
)
return 1
if args.checkpoint_dir is not None:
# A parked leg writes {run_id}-exploration.json from its ``finally`` WITH the
# declaration; the resumed leg runs in a process that never saw the payload and
# OVERWRITES the same file with ``prepass: null``. A declaration that evaporates
# halfway is worse than one refused, and it would do so silently — so this is refused
# rather than left to erase itself.
print(
"run refused: --prepass-seed and --checkpoint-dir cannot be combined (the resumed "
"leg runs in a process that never saw the payload, and its own artefact would "
"overwrite the parked leg's declaration of the cut with nothing)",
file=sys.stderr,
)
return 1
# The pre-pass door's four remaining refusals, at FUNCTION level and never nested under
# another flag's branch: under one, a bare combination would fall straight through to a
# dispatch that drops the payload in silence (the F4 class).
if not args.portfolio and args.prepass_payload is not None:
if args.bundle_dir is None:
print(
"run refused: --prepass-payload requires --bundle-dir (a payload is a cut OF a "
"knowledge base, and every excerpt in it is verified against the mounted "
"document before the run starts; the road path has no base to verify against)",
file=sys.stderr,
)
return 1
if args.proposals_from_mandate:
# That mode settles a commission against the derived schedule and RETURNS above the
# debate entirely, so a payload there would be accepted and silently inert.
print(
"run refused: --prepass-payload is not used by --proposals-from-mandate (that "
"mode builds candidates deterministically from the commission and the derived "
"schedule, and returns before any debate — so the cut would be declared and then "
"never read)",
file=sys.stderr,
)
return 1
if args.dimension_config is not None:
# The pre-pass has no dimension concept, so its CUT is unscoped. Discarding delivered
# excerpts here would make the payload's own denominators wrong for the run that
# published them, which is the denominator failure this whole seam exists to remove.
print(
"run refused: --prepass-payload and --dimension-config cannot be combined (the "
"pre-pass cut the base without knowing about dimensions, so honouring the scope "
"would drop excerpts its declaration counts as delivered — the denominators would "
"then be wrong for the run that published them; cut for the dimension instead, or "
"run unscoped)",
file=sys.stderr,
)
return 1
if args.explore:
# The exploration reads the WHOLE base with all four navigator tools and then shapes
# the mandate the debate is told it must not go outside. That is precisely the ground
# on which the debate's own tools are withdrawn, one caller over.
print(
"run refused: --prepass-payload and --explore cannot be combined (the exploration "
"navigates the whole knowledge base with the same four tools the payload "
"withdraws, so the run as a whole would read far outside the cut it declares)",
file=sys.stderr,
)
return 1
# S7b: the deterministic candidate source needs BOTH halves of its input, and each missing half
# is refused by its own name. Neither is inferable — a commission is what a person wrote, and
# the derived schedule is the only thing that can supply a quantity and a price — so a run that
# proceeded without one would have either nothing to quantify or nothing to quantify WITH.
# Placed AFTER the --bundle-dir requirement above, so a base-less argv is still answered by the
# message naming --bundle-dir rather than by one of these.
if not args.portfolio and args.proposals_from_mandate:
if args.mandate is None:
print(
"run refused: --proposals-from-mandate requires --mandate (the commission IS the "
"candidate source here — the measure, the cost codes and the estimate all come "
"from the approaches, and none of them is ours to invent)",
file=sys.stderr,
)
return 1
if not args.derive_cost_baseline:
print(
"run refused: --proposals-from-mandate requires --derive-cost-baseline (the "
"derived schedule is where each candidate's quantities and unit costs come from; "
"without it there is nothing to build affected_items out of)",
file=sys.stderr,
)
return 1
# --semantic-retrieval is refused, never silently ignored (the repo's flag contract). In
# single-project mode it can only do observable work with BOTH of these: the Step-1 fold is
# gated on ``bundle_dir``, and ``--verdict-dir`` is the only route by which ``main()`` can hand
# ``run_project`` a non-empty store (``main()`` never passes ``store=``, and ``run_project``
# never seeds one). Without them the flag would rank nothing that reaches a prompt, and
# ``RunResult.retrieved`` never leaves the process — ``main()`` prints one outcome line only.
#
# DELIBERATELY STATIC. There is no runtime "refuse if the store ends up empty" check: a
# missing, empty or partially-skipped inbox is the Steg-7 tolerant-load contract, so refusing
# there would fire on a legitimate first run. The refusal is therefore necessary, not
# sufficient — it catches the configuration that CANNOT work, not every run that finds nothing.
#
# main() only. As a library API, ``run_project(semantic_retrieval=True, store=…)`` with a
# caller-supplied store stays legitimate — that is the path the tests drive. Portfolio mode is
# unaffected: ``run_portfolio`` always resolves a store and populates it by cross-project capture.
if not args.portfolio and args.semantic_retrieval:
required = {"--bundle-dir": args.bundle_dir, "--verdict-dir": args.verdict_dir}
missing = [name for name, value in required.items() if not value]
if missing:
print(
f"run refused: --semantic-retrieval requires {' and '.join(missing)} in "
"single-project mode (the Step-1 fold is bundle-path-only, and --verdict-dir is "
"the only route to a non-empty store)",
file=sys.stderr,
)
return 1
# --embedder-config selects the embedder for the HybridRanker, and that ranker is built ONLY
# when semantic_retrieval is on; the default StructuralRetriever takes no embedder at all. So
# without --semantic-retrieval the config is loaded fail-fast and then dropped on the floor —
# MEASURED, not inferred: an injected embedder is consulted ZERO times with the flag off and
# once with it on (tests/test_run_cli.py::test_injected_embedder_is_never_consulted_with_the
# _flag_off + its control). That is the silent-ignore this CLI's flag contract exists to
# prevent, and the same ground on which --semantic-retrieval itself is refused above when it
# cannot take effect.
#
# REFUSED, not wired — the opposite call from --scripted-replies in portfolio mode, and for a
# stated reason: there the seam already existed (run_portfolio takes the same client_factory),
# so refusing would have left a whole mode without an offline door. Here there is nothing to
# wire to; an embedder has no job outside the hybrid ranker.
#
# MODE-INDEPENDENT (hence above the portfolio dispatch, not inside either branch): both modes
# gate the embedder on the same flag, since run_portfolio forwards it to run_project unchanged.
# Placed ABOVE the scripted door for the reason the required-args guard was hoisted there — a
# refused run must not first print a banner claiming a scripted loop closed.
if args.embedder_config is not None and not args.semantic_retrieval:
print(
"run refused: --embedder-config requires --semantic-retrieval (the embedder is only "
"consulted by the hybrid ranker that flag builds — without it the config would be "
"loaded and then ignored)",
file=sys.stderr,
)
return 1
# The exploration door (U4). Every refusal here is BY NAME and happens before anything runs,
# for the reason the whole block above exists: a flag that cannot take effect is refused, never
# silently ignored. Four of them, each closing a different way this could go quietly wrong.
#
# 1. --explore-config alone would be loaded and dropped (the --embedder-config case verbatim).
# 2. --explore alone has no bounds, and the CLI may not invent them: EVERY ExplorationContract
# field is required without a default precisely because MagenticBuilder's own fallback is
# "unbounded", which is the one shape shared/method-spec.md §8 forbids outright.
# 3. --explore + --mandate are TWO SOURCES OF ONE MANDATE. Refused rather than merged, and the
# decision is deliberate: ``explore()`` takes the objective from the prompt and hardcodes
# ``allow_own_proposals=True``, so composing them would silently overwrite three fields an
# operator wrote by hand — the silent merge this repo's flag contract forbids. § C.6 door 1
# (the expert's own hypotheses seeding the exploration) is a real need, and it is served by
# the library API; the refusal names it rather than only forbidding.
# 4. --explore + --live-dry-run contradict: the drill stops before the first model call and an
# exploration IS model calls (the --scripted-replies precedent, same words).
# 5. --plan-review alone answers a review that is never requested (the (1) case, for the U13
# door). Its config-dependent half — the flag against a config that asks for no review, and
# a config that asks for one with no flag — is refused below, once the bounds are loaded.
if args.plan_review and args.explore is None:
print(
"run refused: --plan-review requires --explore (there is no plan to review without an "
"exploration, so the flag would be accepted and then never used)",
file=sys.stderr,
)
return 1
# --- U12, the asynchronous half. Every refusal names its flags, and every one of them fires
# BEFORE the first model call: a resume that is going to be refused must be refused while it
# is still free (the økt-57 hoist), and a park that cannot write its question must not run at
# all — the whole point of the door is that somebody can answer it afterwards.
if args.checkpoint_dir is not None and args.plan_review:
print(
"run refused: --plan-review and --checkpoint-dir are two doors onto one review — the "
"first answers it at this terminal, the second parks it for another process. Refused "
"rather than ranked: silently preferring either would block an operator who asked for "
"the other",
file=sys.stderr,
)
return 1
# --- MAJOR-2. THREE modes return ABOVE generation, so each would accept the door and then
# never ask it — the silent-drop class ``report_forbidden`` exists for. The block is at
# FUNCTION scope on purpose, measured rather than placed by eye: the neighbouring
# ``--scripted-replies``/``--live-dry-run`` refusal is nested under ``if args.scripted_replies
# is not None`` and the ``--explore``/``--live-dry-run`` one under ``if args.explore is not
# None``, so under either guard a bare ``--live-dry-run --proposal-review`` would fall straight
# through to the dry-run dispatch and drop the flag. Every message carries prose the others do
# not share, and each fires before the first model call.
if args.proposal_review:
if args.live_dry_run:
print(
"run refused: --proposal-review and --live-dry-run contradict (the dry run stops "
"before the first model call, so no candidate ever reaches a reviewer) — drop one",
file=sys.stderr,
)
return 1
if args.proposals_from_mandate:
print(
"run refused: --proposal-review has nothing to answer under "
"--proposals-from-mandate (that mode settles the commission deterministically and "
"never generates a candidate) — drop one",
file=sys.stderr,
)
return 1
# ``and args.resume is None`` is load-bearing, not defensive: --resume REQUIRES
# --checkpoint-dir (:2776 below), so a bare --checkpoint-dir test refused the very
# composition this message recommends, and the help text, README and the invariant row all
# described a path no argv could take. What is being refused is a PARK (a leg that returns
# before any candidate exists), never a LIFT (a leg that runs the pipeline to a candidate).
if args.checkpoint_dir is not None and args.resume is None:
print(
"run refused: --proposal-review and --checkpoint-dir: a parked exploration "
"returns before any candidate exists, so the review would never be asked — pass "
"--proposal-review at --resume instead",
file=sys.stderr,
)
return 1
if args.resume is not None:
if args.explore is not None:
print(
"run refused: --resume and --explore are two sources of one exploration. --resume "
"continues the one recorded in the parked state (its own prompt, bounds and "
"bases); --explore starts a new one. Merging would silently drop a prompt",
file=sys.stderr,
)
return 1
if args.mandate is not None:
print(
"run refused: --resume and --mandate are two sources of one mandate — the resumed "
"exploration SHAPES one (the --explore + --mandate refusal, one time-scale later)",
file=sys.stderr,
)
return 1
if args.run_id is not None:
print(
"run refused: --resume and --run-id are two sources of one run id. --resume names "
"the parked run, and the resumed leg keeps writing under that same id",
file=sys.stderr,
)
return 1
if args.live_dry_run:
print(
"run refused: --resume and --live-dry-run contradict each other (the drill stops "
"before the first model call; resuming an exploration is model calls) — pick one",
file=sys.stderr,
)
return 1
if args.checkpoint_dir is None:
print(
"run refused: --resume requires --checkpoint-dir (the workflow state a resume "
"restores from lives there; without it there is nothing to resume)",
file=sys.stderr,
)
return 1
if args.review_inbox is None:
print(
"run refused: --resume requires --review-inbox (the expert's answer lives there, "
"and a resume with no answer would have to invent one)",
file=sys.stderr,
)
return 1
if not args.outbox_dir:
print(
"run refused: --resume requires --outbox-dir (the open question was written "
"there as {run_id}-plan-review.json, and it is what names the review to answer)",
file=sys.stderr,
)
return 1
if not args.bundle_dir:
print(
"run refused: --resume requires --bundle-dir (the resumed exploration navigates "
"knowledge bases, exactly as the parked one did)",
file=sys.stderr,
)
return 1
# ONE run id across the suspension. --run-id was refused above precisely so this
# assignment is the only source, and the resumed leg keeps writing under the id the parked
# leg used — an artefact set split across two ids would describe two runs that never were.
args.run_id = args.resume
elif args.checkpoint_dir is not None and args.explore is None:
print(
"run refused: --checkpoint-dir requires --explore (to park a plan review) or --resume "
"(to lift one); on its own it names a folder nothing would ever be written to",
file=sys.stderr,
)
return 1
if args.review_inbox is not None and args.resume is None:
print(
"run refused: --review-inbox requires --resume (the answers there are read by a "
"resume and by nothing else, so the folder would be named and never opened)",
file=sys.stderr,
)
return 1
if args.checkpoint_dir is not None and args.explore is not None:
# The HOIST again, and it is the one that matters most here: the question artefact IS the
# asynchronous door. Without somewhere to write it the exploration would spend its whole
# budget and then have no way to say what it stopped to ask — a park indistinguishable
# from a crash, days before anybody noticed.
if not args.outbox_dir or not args.run_id:
print(
"run refused: --checkpoint-dir requires --outbox-dir and --run-id, settled BEFORE "
"the exploration runs: the parked question is written as "
"{run_id}-plan-review.json, and without it the review could never be answered",
file=sys.stderr,
)
return 1
if args.explore_config is not None and args.explore is None:
print(
"run refused: --explore-config requires --explore (the bounds describe an exploration "
"that would never run, so the config would be loaded and then ignored)",
file=sys.stderr,
)
return 1
if args.explore is not None:
if args.explore_config is None:
print(
"run refused: --explore requires --explore-config (an exploration's bounds are "
"never defaulted — an omitted cap falls back to an unbounded loop, not to a "
"conservative one)",
file=sys.stderr,
)
return 1
if args.mandate is not None:
print(
"run refused: --explore and --mandate are two sources of one mandate. The "
"exploration SHAPES a mandate (objective from the prompt, own proposals allowed), "
"so merging would silently overwrite what you wrote. To seed an exploration with "
"an expert's own hypotheses, use the library door: "
"explore(..., seed_approaches=[Approach(...)])",
file=sys.stderr,
)
return 1
if args.live_dry_run:
print(
"run refused: --explore and --live-dry-run contradict each other (the drill stops "
"before the first model call; an exploration is model calls) — pick one",
file=sys.stderr,
)
return 1
if not args.bundle_dir:
print(
"run refused: --explore requires --bundle-dir (the exploration navigates knowledge "
"bases, and with none configured it would spend its budget reading nothing)",
file=sys.stderr,
)
return 1
if args.outbox_dir and not args.run_id:
# A HOIST, not a second copy of the rule: ``run_project`` owns the outbox contract and
# refuses on its first statement, which is early enough for every path that existed
# before U4. The exploration runs AHEAD of that call, so without this the whole
# exploration budget is spent on model calls and only THEN refused — and the artefact
# write is skipped as well, so not even the record of what was spent survives. The same
# hoist ``main()`` performs for the required-args guard, for the same reason.
print(
"run refused: --outbox-dir requires --run-id, and with --explore that has to be "
"settled BEFORE the exploration runs (otherwise the loop spends its whole budget "
"on an argv that cannot finish)",
file=sys.stderr,
)
return 1
exploration_contract: ExplorationContract | None = None
if args.explore_config is not None:
try:
exploration_contract = load_exploration_contract(args.explore_config)
except (FileNotFoundError, ValidationError, ValueError) as exc:
print(f"run refused: {exc}", file=sys.stderr)
return 1
# Both halves are refused HERE rather than left to ``explore()``, which refuses them too:
# ExplorationError is a RuntimeError and therefore outside this CLI's (ValueError,
# FileNotFoundError, ValidationError) refusal tuple, so either would leave as a traceback
# instead of the rc 1 line every other misconfiguration produces.
#
# The two messages share the token ``enable_plan_review`` and must NOT share their
# distinguishing wording: a test asserting on the shared substring passes against a
# surface missing one of the branches entirely (measured in økt 57 on --explore).
if args.checkpoint_dir is not None and not exploration_contract.enable_plan_review:
print(
"run refused: --checkpoint-dir was given but --explore-config sets "
"enable_plan_review false, so nothing would ever park and the checkpoints would "
"be written and never read (refused, never silently ignored)",
file=sys.stderr,
)
return 1
if (
exploration_contract.enable_plan_review
and not args.plan_review
and args.checkpoint_dir is None
):
# The refusal SURVIVES F4 — a run must never stop at a review nobody can answer — but
# its old wording ("the synchronous door is the library API") stopped being true the
# moment this CLI grew one, so it names the flag instead. A claim a surface makes about
# itself is exactly what Fase 3 measured drifting.
print(
"run refused: --explore-config sets enable_plan_review but no reviewer was "
"offered, so the run would stop at a review nobody can answer. Add --plan-review "
"to answer it at this terminal, or --checkpoint-dir to park it for an expert to "
"answer later, or set enable_plan_review to false",
file=sys.stderr,
)
return 1
if args.plan_review and not exploration_contract.enable_plan_review:
print(
"run refused: --plan-review was given but --explore-config sets enable_plan_review "
"false, so no review is ever requested and the reviewer would never be asked "
"anything (refused, never silently ignored)",
file=sys.stderr,
)
return 1
# The commission, loaded fail-fast BEFORE anything runs: a missing or malformed mandate is
# REFUSED rather than degraded to "no mandate", because the settlement would then describe work
# nobody ordered. Placed with the other refusals and ABOVE the scripted banner, for the same
# reason the required-args guard was hoisted there — a refused run must not first print a
# banner claiming a scripted loop closed.
mandate: Mandate | None = None
if args.mandate is not None:
try:
mandate = load_mandate(args.mandate)
except (FileNotFoundError, ValidationError, ValueError) as exc:
print(f"run refused: {exc}", file=sys.stderr)
return 1
# The declared cut, loaded fail-fast alongside the commission and for the same reason: a
# payload that cannot be read is not a run to start with a navigating debate instead. Missing,
# not JSON, or not the shape the models require — all three land on the refusal surface with
# rc 1 and no traceback, which is what ``PrepassRefused`` subclasses ``ValueError`` for.
prepass_payload: prepass.PrepassPayload | None = None
if args.prepass_payload is not None:
try:
prepass_payload = load_prepass_payload(args.prepass_payload)
except (FileNotFoundError, ValidationError, ValueError) as exc:
print(f"run refused: {exc}", file=sys.stderr)
return 1
# The seeding arm's payload, loaded here and by the SAME loader: one reader of one file
# format, never a second (kø-(p)). What differs between the two arms starts after admission.
prepass_seed: prepass.PrepassPayload | None = None
if args.prepass_seed is not None:
try:
prepass_seed = load_prepass_payload(args.prepass_seed)
except (FileNotFoundError, ValidationError, ValueError) as exc:
print(f"run refused: {exc}", file=sys.stderr)
return 1
# The egress config, loaded fail-fast alongside the commission. Degrading a broken one to "no
# external services" would make the announcement describe a run nobody configured, and a
# partially-parsed one could contact a subset nobody chose.
mcp_servers: tuple[McpServerConfig, ...] = ()
if args.mcp_config is not None:
try:
mcp_servers = load_mcp_config(args.mcp_config)
except (FileNotFoundError, ValidationError, ValueError) as exc:
print(f"run refused: {exc}", file=sys.stderr)
return 1
# The scripted door (offline WHOLE-loop run over the caller's own data). Resolved BEFORE the
# dry-run branch so the two offline modes cannot both be honoured — and BEFORE the portfolio
# dispatch, because the door serves BOTH modes. It originally sat below that dispatch, which
# made ``--portfolio --scripted-replies`` silently drop the flag: no banner, and four real
# model calls attempted (measured). That is the failure mode the "refused, never ignored"
# partition exists to prevent, and here the honest resolution is to WIRE it — ``run_portfolio``
# already exposes the same ``client_factory`` seam ``run_project`` does, so refusing would have
# left portfolio mode with no offline door at all for an adopter without a model budget.
scripted_client_factory: Callable[[str], BaseChatClient] | None = None
if args.scripted_replies is not None:
if args.live_dry_run:
# Both are offline, and they contradict: --live-dry-run stops before the first model
# call while --scripted-replies answers every one of them. Refuse rather than let one
# silently win (S5.3's "refused, never ignored" partition). Unreachable in portfolio
# mode, where --live-dry-run is already refused by the single_only partition above.
print(
"run refused: --scripted-replies and --live-dry-run are both offline modes and "
"contradict each other (dry-run stops before the first model call; scripted "
"answers all of them) — pick one",
file=sys.stderr,
)
return 1
# ``--explore`` asks the same factory for three roles the debate never uses (MAJOR-2) — the
# door must know about them BEFORE loading the file, or a missing one crashes deep inside
# ``explore()`` instead of being refused here, at the door, by name.
required_scripted_roles: Sequence[str] = _SCRIPTED_ROLES
if args.explore is not None or args.resume is not None:
# ``--resume`` rebuilds the SAME workflow with the SAME three participants, so it needs
# the same three replies. Measured, not reasoned: without ``--resume`` here the child
# process died on ``KeyError: 'navigator'`` deep inside ``fresh_exploration_workflow``
# — the identical defect MAJOR-2 closed for ``--explore`` in økt 62, reappearing on the
# second surface that builds an exploration. A gate that names one door and not the
# other is the drift this comment exists to stop happening a third time.
required_scripted_roles = _SCRIPTED_ROLES + _EXPLORATION_SCRIPTED_ROLES
try:
replies = _load_scripted_replies(args.scripted_replies, required_scripted_roles)
except (OSError, ValueError) as exc:
print(f"run refused: {exc}", file=sys.stderr)
return 1
# Imported HERE rather than at module scope: ``simulation`` imports ``run``, so a top-level
# import would be circular. The scripted client already exists as MAF-side scaffolding —
# this flag is a DOOR onto that one seam, never a second implementation of it.
from portfolio_optimiser.simulation import scripted_factory
scripted_client_factory = scripted_factory(replies, [])
print(_SCRIPTED_BANNER)
# U4: the exploration runs BEFORE the announcement, because what it produces IS the mandate the
# announcement describes. Its own model calls are therefore un-announced — stated plainly
# rather than papered over: the announcement's contract is that a COMMISSION is declared before
# the work it commissions, and until the exploration returns there is no commission to declare.
# ``exploration_notice`` is what covers the gap, printed the moment the loop is done.
#
# Every ExplorationError ``explore()`` can raise for a CONFIG reason is unreachable from here by
# construction: both plan-review preconditions are refused above, and the duplicate-base-id
# refusal needs two bases where this surface passes one. What can still escape — an unreadable
# marked hypothesis, an exhausted budget, and (since F4) a plan review the operator left
# unanswered — is the RUN failing, not the caller erring, and leaves as it does for the debate
# today.
# The resume's two loads happen HERE, before the trace block below: they are refusals, and a
# refusal must not first overwrite {run_id}-exploration.json with an empty trace — the record
# of what the PARKED leg did is the only evidence of the run so far. This is also the økt-57
# hoist in its purest form: not answered yet is the NORMAL state of this door, so it has to be
# free. Both errors are ``ValueError``s (``PlanReviewAnswerError``) or ``ExplorationError``
# (``ParkedStateError``), and both are caught by NAME rather than left to escape as tracebacks.
resumed: tuple[Any, PlanReviewDecision] | None = None
if args.resume is not None:
question = hitl.read_plan_review_question(args.outbox_dir, args.resume)
if question is None:
print(
f"run refused: no parked plan review for run {args.resume!r} in "
f"{args.outbox_dir!r} (expected {args.resume}-plan-review.json) — there is "
f"nothing to resume",
file=sys.stderr,
)
return 1
try:
parked_state = load_parked(question)
answer = hitl.load_plan_review_answer(
args.review_inbox, args.resume, request_id=parked_state.request_id
)
except (hitl.PlanReviewAnswerError, ParkedStateError) as exc:
print(f"run refused: {exc}", file=sys.stderr)
return 1
resumed = (
parked_state,
PlanReviewDecision.approve()
if answer.decision == "approve"
else PlanReviewDecision.revise(answer.feedback),
)
# The seeding arm's admission, HOISTED above the try/finally below for the økt-57 reason the
# resume loads are hoisted: this is a refusal, and a refusal must not first overwrite
# {run_id}-exploration.json with an empty trace. It also has to happen before the first model
# call — at the exit code, a refusal after the spend looks exactly like one before it.
seed_declaration: prepass.PrepassDeclaration | None = None
seed_context = ""
if prepass_seed is not None:
assert args.bundle_dir is not None # guarded: --prepass-seed -> --explore -> --bundle-dir
try:
# This door OPENS a base, so it carries that gate itself rather than trusting the
# tool-side one to fire later (S7a-3: every opening door gets its own, and its own
# mutation). A base whose concepts name two corpora cannot be the one a cut is OF.
okf.assert_declared_ids_agree(okf.navigate_bundle(args.bundle_dir))
prepass.admit_payload(
prepass_seed,
bundle_dir=args.bundle_dir,
resolved_id=okf.reconcile_bundle_id(args.bundle_dir),
# NOT scoped, and that is measured rather than overlooked: the exploration builds
# ``navigator_tools(bundle_dirs)`` with no dimension at all, so scoping the seed
# would refuse text the very same loop can open with ``read_file`` a moment later.
# The DEBATE downstream stays scoped exactly as it is today.
dimension=None,
)
except (FileNotFoundError, ValidationError, ValueError) as exc:
print(f"run refused: {exc}", file=sys.stderr)
return 1
seed_declaration = prepass.declaration_of(prepass_seed, rest_reachable=True)
seed_context = prepass.render_seed(prepass_seed)
# Printed HERE, before the loop it seeds, rather than after: the announcement contract is
# that a commission is declared before the work it commissions, and a run whose budget is
# exhausted mid-exploration must still have said what it started from.
cut_notice = prepass_notice(seed_declaration)
assert cut_notice is not None # a declaration always renders; ``None`` means no cut
print(cut_notice)
if args.explore is not None or resumed is not None:
exploration_trace = ExplorationTrace()
exploration: ExplorationResult | None = None
parked_now: PlanReviewParked | None = None
budget_now: BudgetExceeded | None = None
provider_now: ChatClientException | None = None
try:
if resumed is not None:
# The parked state, not argv, is what rebuilds the workflow: the graph has to match
# the checkpoint's signature for the restore to be accepted at all, so an operator
# who had to re-supply the prompt and the bounds could get one wrong and find out
# as a restore failure days later.
parked_state, decision = resumed
exploration = asyncio.run(
resume_exploration(
parked_state,
decision,
checkpoint_dir=args.checkpoint_dir,
profile=args.profile,
client_factory=scripted_client_factory,
trace=exploration_trace,
)
)
else:
assert (
exploration_contract is not None
) # guarded above: --explore requires --explore-config
exploration = asyncio.run(
explore(
args.explore,
contract=exploration_contract,
bundle_dirs=(args.bundle_dir,),
profile=args.profile,
client_factory=scripted_client_factory,
trace=exploration_trace,
# The F4 door. Built here and never inside ``explore()``: the loop owns the
# seam, the CLI owns which reviewer fills it, and a library that reached
# for stdin on its own would answer for a caller that never offered to.
plan_reviewer=terminal_plan_reviewer() if args.plan_review else None,
# The U12 door. Mutually exclusive with the one above, refused at the top.
checkpoint_dir=args.checkpoint_dir,
# Q5=B. Empty unless --prepass-seed was given, and an empty string leaves
# the task message byte-identical to what every run built before today.
seed_context=seed_context,
)
)
except PlanReviewParked as parked_exc:
# NOT an error, and not a completed run either — the third channel, for the reason
# ``BudgetExceeded`` has its own: the exploration produced no mandate, so returning one
# would let a caller book "explored" for a loop suspended mid-plan. Caught here rather
# than left to escape, because parking is what the operator ASKED for by giving
# --checkpoint-dir; the artefact is where a machine reads that it happened.
parked_now = parked_exc
except ChatClientException as provider_exc:
# Funn 99. The model endpoint rejected the request, so the run ENDED — it was never
# refused. ``run stopped:``, not ``run refused:``, and the class is what routes it here
# (the MAJOR-2 precedent, verbatim: "the argv was fine and the run had already spent
# tokens, so 'refused' would mislabel it"). MEASURED on the paid Q5=B run: an Azure 400
# ("No tool call found for function call output with call_id ...") left ``main()`` as a
# traceback, because ``ChatClientException`` is in none of this function's refusal
# tuples — the same gap ``BudgetExceeded`` was added to this very block for. Caught
# INSIDE the try/finally so the ``finally`` still writes ``{run_id}-exploration.json``:
# the run that most needs the evidence is the one a provider cut short.
provider_now = provider_exc
except BudgetExceeded as budget_exc:
# A cap that fired is a refusal at this door, not a programming error — the CLI's
# existing contract for every other loader mistake below (stderr + rc 1, never a
# traceback), applied to the one raise this block did not yet catch (measured by
# accident on K2, S7b's own uttalte grense). Caught here, one frame above every other
# refusal, because ``explore()``/``resume_exploration()`` are the only two calls in
# this block that can raise it — a handler placed lower would never see it.
budget_now = budget_exc
finally:
# From a ``finally``, exactly as ``write_parse_failures`` is (Fase 1b, funn 1): the run
# that most needs this evidence is the one a cap cut short, and that run returns
# nothing. ``completed`` says which of the two happened, so a reader never has to infer
# it from an absent ``stop``.
if args.outbox_dir and args.run_id:
outbox.write_exploration(
args.outbox_dir,
args.run_id,
payload=trace_payload(
exploration_trace,
stop=exploration.stop if exploration is not None else None,
completed=exploration is not None,
mandate=exploration.mandate if exploration is not None else None,
# From the SAME object the notice was rendered from, never a second load:
# stdout and the artefact must describe one cut, not two (kø-(p)).
prepass=(
prepass.declaration_payload(seed_declaration)
if seed_declaration is not None
else None
),
),
)
if provider_now is not None:
# One line, rc 1, no traceback — and reported BEFORE the budget arm only because the
# two are mutually exclusive by construction (a single exception left the block).
print(f"run stopped: {provider_now}", file=sys.stderr)
return 1
if budget_now is not None:
# Same shape as every other refusal in this function: one line on stderr, rc 1, no
# traceback. The artefact was already written by the ``finally`` above (``completed``
# is ``False`` there, exactly as it is for a park) — this only decides what the
# terminal says.
print(f"run refused: {budget_now}", file=sys.stderr)
return 1
if parked_now is not None:
outbox.write_plan_review(
args.outbox_dir, args.run_id, payload=parked_payload(parked_now.parked)
)
print(parked_notice(parked_now.parked, run_id=args.run_id))
return 0
assert exploration is not None # the only other way out of the block above is an exception
print(exploration_notice(exploration))
mandate = exploration.mandate
if mandate is not None:
# The scope line reads the dimension config only to NAME it. A config that fails to load is
# left unnamed here and refused a moment later by the dispatch below, which stays the single
# owner of that refusal (and of its mode-specific wording) — announcing must never change
# which error an operator sees.
dimension_label: str | None = None
if args.dimension_config is not None:
try:
_dim = load_dimension(args.dimension_config)
except (FileNotFoundError, ValidationError, ValueError):
dimension_label = None
else:
dimension_label = f"{_dim.id} ({_dim.label})"
print(
announce(
mandate,
project_id=args.project_id or "the portfolio",
# From ARGV, never the constants (P16 B2). The announcement is the one thing
# printed BEFORE the first paid call, and its whole job is to say what this run
# will do; reading the defaults was correct only while main() could not do
# otherwise. MEASURED on the first free drill after --max-rounds landed: the same
# stdout said "Stops at: 3 rounds / 100000 tokens" two lines above
# "max_rounds=8, max_tokens=120000" -- the Fase-3 class, introduced by the very
# flag being announced.
max_rounds=args.max_rounds,
max_tokens=args.max_tokens,
dimension_label=dimension_label,
external_services=service_labels(mcp_servers),
)
)
elif mcp_servers:
# The egress declaration must NOT depend on a commission being present. Without this
# branch, configuring servers and omitting --mandate would contact third parties with
# nothing printed at all — silent egress, which this repo forbids outright.
print("Contacts: " + ", ".join(service_labels(mcp_servers)))
# S7b: the deterministic door. Placed AFTER the announcement — the commission is declared before
# the work it commissions, exactly as the announcement's contract requires, and here that
# contract is trivially kept because there IS no un-announced spend: this path makes no model
# calls at all. Placed BEFORE the portfolio dispatch and every run dispatch below, because it is
# a terminal mode rather than a modifier: it settles the commission and returns.
#
# ``mandate`` is narrowed by the refusal above, which is why this reads it without a guard. The
# refusals of the underlying seam surface through the SAME structured contract as every other
# single-project loader failure — stderr + rc 1, never a traceback — which is what
# ``MandateCandidateError``/``MandateRoutingError``/``CostBaselineDerivationError`` all
# subclassing ``ValueError`` buys.
if args.proposals_from_mandate:
assert mandate is not None # narrowed by the --mandate refusal above
assert args.bundle_dir is not None # narrowed by --derive-cost-baseline's requirement
# NARROWED, never defaulted. ``args.project_id or ""`` would reach
# ``derive_cost_baseline`` and mint a ``CostBaseline(project_id="")`` — a fabricated
# identity, which is the shape ``cost_baseline_anchored`` is required-without-default to
# forbid. Unreachable today (the required-args guard fires far above), so this is the
# assert that says so rather than a default that quietly disagrees with it.
assert args.project_id is not None # narrowed by the required-args guard
try:
coverage = evaluate_mandate_candidates(
mandate,
bundle_dir=args.bundle_dir,
project_id=args.project_id,
)
except (
MandateCandidateError,
MandateRoutingError,
okf.BundleIdMismatch,
okf.CostBaselineDerivationError,
FileNotFoundError,
) as exc:
# NARROWED to the classes this seam owns, never a blanket ``except ValueError``: the
# five named here are all caller-configuration mistakes, and swallowing anything else
# would turn a programming error into a polite refusal (the ``_unwrap_ingest_error``
# ownership rule). ``FileNotFoundError`` is ``navigate_bundle``'s, for a --bundle-dir
# with no readable index.
print(f"run refused: {exc}", file=sys.stderr)
return 1
print(settle(coverage))
return 0
# P17b — the multi-base dispatch. Placed BELOW the announcement (the commission is declared
# before the work it commissions) and ABOVE the portfolio dispatch, because it is a MODE
# rather than a modifier: it runs the pass and returns. The dry-run arm lives INSIDE this
# block rather than in the generic ``--live-dry-run`` branch further down, which addresses
# ``args.project_id``/``args.bundle_dir`` — neither of which this argv has — and would
# therefore drop the whole pass in silence (the F4 class).
if args.across_bundle:
assert mandate is not None # narrowed by the required-flags refusal above
assert args.run_id is not None and args.outbox_dir is not None # same refusal
bases = list(args.across_bundle)
try:
# Resolved and ROUTED on the free trip too: a commission that cannot be executed as
# written must be refused while it is still free (the økt-57 hoist), and a drill that
# tolerated a routing error the paid pass refuses would be a rehearsal of a different
# run.
resolved = resolve_bundle_routing(bases)
route_by_bundle(mandate, tuple(bundle_id for bundle_id, _, _ in resolved))
except (MandateRoutingError, okf.BundleIdMismatch, FileNotFoundError, ValueError) as exc:
print(f"run refused: {exc}", file=sys.stderr)
return 1
if args.live_dry_run:
for bundle_id, bundle_dir, project_id in resolved:
try:
report = asyncio.run(
run_project(
project_id,
args.profile,
docs_dir=bundle_dir,
bundle_dir=bundle_dir,
verdict_dir=args.verdict_dir,
dimension=(
load_dimension(args.dimension_config)
if args.dimension_config
else None
),
max_rounds=args.max_rounds,
max_tokens=args.max_tokens,
require_cost_baseline=args.require_cost_baseline,
mcp_servers=mcp_servers,
live_dry_run=True,
)
)
except (ValueError, FileNotFoundError, ValidationError) as exc:
print(f"live-dry-run refused: {bundle_id}: {exc}", file=sys.stderr)
return 1
assert isinstance(report, DryRunReport)
print(
f"{bundle_id} ({project_id}): LIVE-DRY-RUN OK (profile={report.profile}, "
f"models={report.resolved_models}, max_rounds={report.max_rounds}, "
f"max_tokens={report.max_tokens}, top_k={report.top_k}) — "
"ingen modellkall gjort (stoppet før første debate.run)"
)
# Every notice the single-base drill prints, per base: a pass whose SECOND base
# cannot be anchored is exactly as unanchored as one whose first cannot, and a
# drill that said it once would leave the operator guessing which base it meant.
for notice in (
cost_baseline_notice(report.cost_baseline_anchored),
grounding_offer_notice(report.grounding_offer),
skipped_links_notice(report.skipped_links),
bundle_id_notice(report.bundle_id_source),
):
if notice is not None:
print(notice)
return 0
outbox_dir = args.outbox_dir
stem = args.run_id
def outbox_for(bundle_id: str) -> tuple[str, str]:
"""The OPERATOR-CHOSEN minting rule (14.09): ``<run-id>-<bundle_id>``, one stem per
base. Written here, at the call site that made the decision, rather than inside the
dispatch — which is precisely why the engine takes a callback and not a directory."""
return outbox_dir, f"{stem}-{bundle_id}"
#: Bound by the dispatch when it RETURNS; still ``None`` when a base raised. The summary
#: is written from a ``finally`` either way (``write_parse_failures``' precedent), because
#: the pass that most needs a record of what it spent is the one a cap or a provider cut
#: short — and the engine's documented limit is that a base which RAISES propagates.
multi: MultiBaseResult | None = None
try:
multi = asyncio.run(
run_mandate_across_bundles(
mandate,
tuple(bases),
args.profile,
verdict_dir=args.verdict_dir,
dimension=(
load_dimension(args.dimension_config) if args.dimension_config else None
),
client_factory=scripted_client_factory,
max_rounds=args.max_rounds,
max_tokens=args.max_tokens,
verdict_input=_verdict_input_from_args(args),
proposal_reviewer=(
terminal_proposal_reviewer() if args.proposal_review else None
),
outbox_for=outbox_for,
)
)
except ChatClientException as exc:
print(f"run stopped: {exc}", file=sys.stderr)
return 1
except ProposalReviewInputError as exc:
print(f"run stopped: {exc}", file=sys.stderr)
return 1
except (
MandateRoutingError,
okf.BundleIdMismatch,
FileNotFoundError,
ValidationError,
ValueError,
BudgetExceeded,
) as exc:
# ``BudgetExceeded`` is in the tuple for the single-project path's measured reason: it
# is a ``RuntimeError``, and the FIRST approach to hit the cap re-raises by design
# (``_evaluate_mandate`` only swallows it mid-list). Over several bases that is not an
# edge case — round 3 measured ``stop_reason: rounds`` in 5 of 5 runs — so without this
# arm the ordinary outcome of a multi-base pass is a traceback.
print(f"run refused: {exc}", file=sys.stderr)
return 1
finally:
_write_multibase_summary(
outbox_dir,
args.run_id,
resolved=resolved,
mint=outbox_for,
multi=multi,
)
for bundle_run in multi.runs:
print(f"--- {bundle_run.bundle_id} ({bundle_run.project_id}) ---")
print(
f"{bundle_run.project_id}: {type(bundle_run.result.outcome).__name__} "
f"({verdict_notice(bundle_run.result)})"
)
# Every per-run notice the single-base path prints, read off THIS base's own stamp —
# a pass that said them once could only be talking about one of N bases, and the
# reader could not tell which.
for notice in (
cost_baseline_notice(bundle_run.result.provenance.cost_baseline_anchored),
bundle_id_notice(bundle_run.result.provenance.bundle_id_source),
skipped_links_notice(bundle_run.result.skipped_links),
unkeyed_verdicts_notice(bundle_run.result.unkeyed_verdicts),
):
if notice is not None:
print(notice)
notice = collision_notice(multi.collisions)
if notice is not None:
print(notice)
return 0
if args.portfolio:
# Portfolio mode (Step 3): dispatch to the EXISTING run_portfolio via the fail-fast loaders
# (run_portfolio itself is unchanged). Loader/ValueError failures surface through the same
# structured-refusal contract as the single-project path (stderr + rc 1, no traceback).
try:
goals = load_goal_config(args.goals) if args.goals else None
ledger = SavingsLedger.load(args.ledger) if args.ledger else None
dimension = load_dimension(args.dimension_config) if args.dimension_config else None
embedder = (
build_embedder(load_embedder_config(args.embedder_config))
if args.embedder_config
else None
)
project_ids = (args.project_id,) if args.project_id is not None else None
portfolio_result = asyncio.run(
run_portfolio(
project_ids,
args.profile,
dimension=dimension,
embedder=embedder,
ledger=ledger,
goals=goals,
semantic_retrieval=args.semantic_retrieval,
client_factory=scripted_client_factory,
mandate=mandate,
mcp_servers=mcp_servers,
max_rounds=args.max_rounds,
max_tokens=args.max_tokens,
)
)
except (ValueError, FileNotFoundError, ValidationError) as exc:
print(f"portfolio run refused: {exc}", file=sys.stderr)
return 1
for r in portfolio_result.runs:
print(f"{type(r.outcome).__name__}: {verdict_notice(r)}")
# Per project, because anchoring is a per-project fact. DEFENSIVE and currently
# unreachable from this branch — measured, and said out loud for the same reason the
# ``budget_stop`` arm below is: no reference project sets ``bundle_dir``, so every
# portfolio run today takes the road path and is anchored by construction. The test
# that covers it drives a crafted ``PortfolioResult``, and says so.
run_notice = cost_baseline_notice(r.provenance.cost_baseline_anchored)
if run_notice is not None:
print(run_notice)
# One settlement per project: the mandate applies to each project in the pass, so
# each project answers for it separately. Empty without a mandate.
project_settlement = settle(r.coverage)
if project_settlement:
print(project_settlement)
if portfolio_result.stop_reason is not None:
sr = portfolio_result.stop_reason
print(
f"goal reached: scope={sr.scope} project={sr.project_id or '-'} "
f"observed_ore={sr.observed_ore} limit_ore={sr.limit_ore} "
f"stopped_early={portfolio_result.stopped_early}"
)
# A ``PortfolioResult`` has FOUR outcome channels and this branch reported one of them:
# ``failures`` and ``budget_stop`` never reached the operator, and rc was unconditionally 0
# — so a pass in which every project died printed nothing and exited 0 (measured: four
# projects, four APIConnectionError, silent success). ``BudgetStop`` is kept apart from
# ``stop_reason`` precisely so a caller can tell exhaustion from success; showing neither
# collapsed the distinction the dataclass was split to preserve.
# The budget-stop arm is DEFENSIVE and currently UNREACHABLE from here — measured, not
# assumed, and said out loud for the same reason ``strict=True`` below is: ``main()`` never
# constructs a ``PortfolioMeter``, and every write to ``budget_stop`` is gated on one, so
# only a LIBRARY caller passing ``portfolio_meter=`` can produce this field today. It is
# printed anyway because the field exists and a CLI door onto the global cap is a natural
# next step; the test that covers it drives a crafted ``PortfolioResult``, and says so.
# TRAP for whoever wires that door: ``BudgetRefused`` is a ``RuntimeError``, so the
# ``except`` above would NOT catch the startup refusal — it needs adding explicitly.
if portfolio_result.budget_stop is not None:
bs = portfolio_result.budget_stop
print(
f"budget stop: limit_tokens={bs.limit_tokens} spent_tokens={bs.spent_tokens} "
f"remaining_tokens={bs.remaining_tokens} required_tokens={bs.required_tokens} "
f"stopped_early={portfolio_result.stopped_early}"
)
for failure in portfolio_result.failures:
print(
f"project failed: {failure.project_id} [{failure.error_type}] {failure.error}",
file=sys.stderr,
)
# rc 1 iff something RAISED. Collect-and-continue (S3.3) exists so a partial pass does not
# LOSE the work that completed — every finished run still printed above — not so a pass with
# dead projects can report success to a scripted caller. A ``budget_stop`` alone stays rc 0:
# exhaustion is a structured stop the operator asked for by setting a cap, not a crash.
return 1 if portfolio_result.failures else 0
if args.live_dry_run:
# S4.2 drill (comparison protocol §4 pkt 2/3): walk the offline path, STOP before the first
# model call, print the run-config. A misconfigured profile (e.g. AZURE with a
# REPLACE-WITH-* placeholder) makes resolve_model raise inside the eager factory build —
# refuse cleanly (mirror preflight.main) instead of tracebacking; S4.1 is the config gate.
try:
report = asyncio.run(
run_project(
args.project_id,
args.profile,
docs_dir=docs_dir,
bundle_dir=args.bundle_dir,
verdict_dir=args.verdict_dir,
dimension=(
load_dimension(args.dimension_config) if args.dimension_config else None
),
embedder=(
build_embedder(load_embedder_config(args.embedder_config))
if args.embedder_config
else None
),
max_rounds=args.max_rounds,
max_tokens=args.max_tokens,
outbox_dir=args.outbox_dir,
run_id=args.run_id,
prepass_payload=prepass_payload,
verdict_input=_verdict_input_from_args(args),
derive_cost_baseline=args.derive_cost_baseline,
require_cost_baseline=args.require_cost_baseline,
mcp_servers=mcp_servers,
live_dry_run=True,
)
)
except (ValueError, FileNotFoundError, ValidationError) as exc:
# Structured refusal (rc 1, no traceback) for ANY offline-path ValueError. The
# azure-preflight remediation is only meaningful for the AZURE config gate (S4.1), so
# scope it to that profile — a LOCAL-profile ValueError (unknown project_id, empty
# docs_dir, bundle mismatch) must not carry an irrelevant azure hint.
msg = f"live-dry-run refused: {exc}"
if args.profile == "azure":
msg += (
"\nkjør 'python -m portfolio_optimiser.preflight --profile azure' først "
"(S4.1 offline config-gate)"
)
print(msg, file=sys.stderr)
return 1
assert isinstance(report, DryRunReport) # live_dry_run=True always returns a DryRunReport
print(
f"{args.project_id}: LIVE-DRY-RUN OK (profile={report.profile}, "
f"models={report.resolved_models}, max_rounds={report.max_rounds}, "
f"max_tokens={report.max_tokens}, top_k={report.top_k}) — "
"ingen modellkall gjort (stoppet før første debate.run)"
)
# The measured silence this closes: a bundle without ``cost-baseline.json`` used to dry-run
# to rc 0 with nothing said about the gate's stage 0 being skipped. Printed AFTER the line
# it qualifies, and only when there is something to say.
notice = cost_baseline_notice(report.cost_baseline_anchored)
if notice is not None:
print(notice)
# P8, printed next to the line it qualifies: "stage 0 is skipped" says the gate lost a
# falsifier; this says what the input could have offered it instead. On the FREE trip, so
# an operator learns a run cannot be grounded without paying three attempts to find out.
offer_notice = grounding_offer_notice(report.grounding_offer)
if offer_notice is not None:
print(offer_notice)
# The second measured silence on this surface: a bundle with an unfollowable cross-link
# dry-ran to rc 0 with nothing said, so a half-read base looked exactly like a small one.
nav_notice = skipped_links_notice(report.skipped_links)
if nav_notice is not None:
print(nav_notice)
# The third silence on this surface (S7a-3 pkt. 1): a base delivered under a directory name
# of its own now OPENS, so the disagreement has to be said out loud or nothing ever reports
# that the run's artefacts name something other than the path the operator typed.
id_notice = bundle_id_notice(report.bundle_id_source)
cut_notice = prepass_notice(report.prepass)
if cut_notice is not None:
print(cut_notice)
if id_notice is not None:
print(id_notice)
return 0
try:
result = cast(
RunResult,
asyncio.run(
run_project(
args.project_id,
args.profile,
docs_dir=docs_dir,
bundle_dir=args.bundle_dir,
verdict_dir=args.verdict_dir,
dimension=(
load_dimension(args.dimension_config) if args.dimension_config else None
),
embedder=(
build_embedder(load_embedder_config(args.embedder_config))
if args.embedder_config
else None
),
max_rounds=args.max_rounds,
max_tokens=args.max_tokens,
outbox_dir=args.outbox_dir,
run_id=args.run_id,
prepass_payload=prepass_payload,
verdict_input=_verdict_input_from_args(args),
semantic_retrieval=args.semantic_retrieval,
derive_cost_baseline=args.derive_cost_baseline,
require_cost_baseline=args.require_cost_baseline,
client_factory=scripted_client_factory,
mandate=mandate,
mcp_servers=mcp_servers,
# Built at the CALL SITE, never inside the library (the ``--plan-review``
# precedent): the terminal is the operator's, and ``run_project`` must stay
# answerable by a persona, a test double or nobody at all.
proposal_reviewer=(
terminal_proposal_reviewer() if args.proposal_review else None
),
)
),
)
except ChatClientException as exc:
# Funn 99, the SECOND seam: the debate's own model calls go through the same provider, so
# an arm on the exploration block alone leaves an ordinary ``run.main([...])`` tracebacking.
# Same channel and same reason as ``ProposalReviewInputError`` below — the argv was fine and
# tokens were already spent — and a distinct class from every ``ValueError``-shaped refusal,
# so a reader can tell "the request was wrong" from "the endpoint rejected it".
print(f"run stopped: {exc}", file=sys.stderr)
return 1
except ProposalReviewInputError as exc:
# A DISTINCT channel from ``run refused:`` below, and the class is what routes it here.
# The argv was fine and the run had already spent tokens, so "refused" would mislabel it;
# and a ``ValueError``-shaped error would sit one frame from ``_fetch_parsed``'s
# ``except (ValidationError, ValueError, TypeError)`` and be captured as a parse failure
# instead. ``run_project``'s ``finally`` has already written the review record.
print(f"run stopped: {exc}", file=sys.stderr)
return 1
except (ValueError, FileNotFoundError, ValidationError, BudgetExceeded) as exc:
# Structured refusal (rc 1, no traceback) for the full-run path: run_project's fail-fast
# loaders (contracts, load_dimension, outbox run_id guard) surface here as one clean line.
# ``BudgetExceeded`` joined this tuple for fiks-ordre 20260904T070930Z: a mandate this run
# is EVALUATING (whether commissioned via ``--explore`` or ``--mandate``) can still exhaust
# the round/token cap inside ``generate_via_llm``'s retry loop, and ``_evaluate_mandate``
# only swallows that mid-list — the FIRST approach hitting the cap re-raises by design
# (``produced`` empty, "nothing honest to return"). Measured by accident on K2's syretest:
# an unparseable proposer reply burned the round cap and the process tracebacked instead of
# refusing, because this tuple did not yet know ``BudgetExceeded`` is a ``RuntimeError``,
# not a ``ValueError``.
print(f"run refused: {exc}", file=sys.stderr)
return 1
kind = type(result.outcome).__name__
print(f"{args.project_id}: {kind} ({verdict_notice(result)})")
# Same notice, same renderer, read off the run's OWN stamp — so stdout and the outbox artefact
# cannot disagree about whether the gate was anchored.
notice = cost_baseline_notice(result.provenance.cost_baseline_anchored)
if notice is not None:
print(notice)
# Same renderer on the full run, read off the run's OWN measurement: a run that spent every
# attempt being refused as ungrounded is exactly where the input-side fact costs the most.
offer_notice = grounding_offer_notice(result.grounding_offer)
if offer_notice is not None:
print(offer_notice)
# Same renderer on the full run, and deliberately so: a run that PRODUCED a proposal from a
# half-read base is where the silence cost the most — the dry run at least produced nothing.
nav_notice = skipped_links_notice(result.skipped_links)
if nav_notice is not None:
print(nav_notice)
# Same renderer on the full run, read off the run's OWN stamp — so stdout and the outbox
# artefact cannot disagree about which corpus was judged.
id_notice = bundle_id_notice(result.provenance.bundle_id_source)
if id_notice is not None:
print(id_notice)
# The CUT this run was given, read off the run's OWN declaration rather than off argv, for
# the reason above: stdout and ``{run_id}-prepass.json`` are built from the same object.
cut_notice = prepass_notice(result.prepass)
if cut_notice is not None:
print(cut_notice)
# Full run only, and structurally so: the fold happens BELOW the ``--live-dry-run`` cut, so a
# dry run has nothing to report here (contrast the three notices above, all resolved above it).
fold_notice = unkeyed_verdicts_notice(result.unkeyed_verdicts)
if fold_notice is not None:
print(fold_notice)
# MAJOR-2: what the human answered, read off the run's OWN record rather than off argv — so
# stdout and ``{run_id}-proposal-reviews.json`` cannot disagree. ``offered`` IS argv, because
# "a reviewer was offered and never consulted" is a fact only the flag can supply.
review_notice = proposal_review_notice(result.expert_revisions, offered=args.proposal_review)
if review_notice is not None:
print(review_notice)
# The settlement against the commission (Trekk A4). Empty without a mandate, so an
# un-commissioned run prints exactly what it printed before.
settlement = settle(result.coverage)
if settlement:
print(settlement)
return 0
if __name__ == "__main__": # pragma: no cover - console entry
raise SystemExit(main())
Reference in a new issue View git blame Copy permalink