feat(explore): U4+U13 del 3 - multi-base er en PARTISJON, ikke en videre signatur (ORDRE 20260825T080753Z) [skip-docs]
Ordrens premiss («endrer run_project sin signatur») er FALSIFISERT foer bygging, og ordren ba selv om nettopp den sjekken. run_project kan ikke ta mer enn en bundle_dir: paa bundle-stien avleder den FIRE enkeltverdier fra DEN basen - prosjektet (_project_from_bundle fail-faster naar basens egen project_id ikke er den forespurte), validatorens stage-0-baseline, agentenes lesekontekst og ExpeL-noekkelen - og returnerer ETT stemplet RunResult. En andre katalog paa den signaturen ville tvunget et stille velg-en for alle fire. Planens egen setning sier det samme lest naert: «pipelinen kjoeres per bundle som i dag (run_portfolio-formen)» = N kall, ikke ett kall med N. Levert form, tre soemmer: - mandate.Approach.bundle_id (default "" - hvert mandat skrevet foer i dag er fortsatt gyldig og dispatchbart uendret) - mandate.route_by_bundle - ren partisjon, fail-fast paa et mandat som ikke kan utfoeres som skrevet (load_mandate-regelen). En base som ingen approach navngir kjoeres ikke; med NOEYAKTIG en base absorberer den alt uten navn, som ikke er en gjetning men det eneste mulige svaret. - run.run_mandate_across_bundles - dispatchen. INGEN project_id-parameter: hver base sitt prosjekt leses fra DEN basens egen IR-projeksjon, altsaa den verdien _project_from_bundle allerede fail-faster mot. En delt VerdictStore traades paa tvers (kryss-base-laering, run_portfolio-formen), og med portfolio_meter gjelder de to S3.4-tennene som HAR mening her: oppstartsnekt (BudgetRefused) og aldri-startet + budget_stop + not_evaluated-rader. INGEN eksisterende kaller endrer signatur - CLI, hosting og simulation sender fortsatt en base hver, og kan fortsatt gjoere det. explore(): hver myntet approach baerer bundle_id; markoeren kan navngi basen; en umerket markoer med flere baser NEKTES (HypothesisParseError), med en base resolveres den. Froe-approaches VALIDERES men skrives ALDRI om (§ C.6 doer 1 er en bevaringsregel) - og valideringen skjer FOER foerste modellkall, samme oekt-57-hoist-grunn: ved unntaket alene ser en nekt etter forbruket identisk ut. explore()-docstringens gamle aerlighets-grense («venter paa at run_project tar mer enn en bundle_dir») er RETTET - den ville vaert usann fra i dag. Load-bearing MAALT, 12 mutasjoner alle roede mot HELE suiten, groenn kontroll 1020/5 (fra 999/5 + 21 nye tester); golden demo-transcript.stdout BYTE-UENDRET (ea8c534773acdbe41ae68f2c55724d69aaf8be4f): detach myntet bundle_id (2) · stille gjennomfall ved >1 base (1) · ukjent id resolvert etter rekkefoelge (1) · froe-sjekk etter forbruket (2) · ruteren gjetter foerste base (1) · uroutbar approach droppet (1) · dispatchen kollapser til en base (5) · project_id fra foerste base (2) · detach aldri-startet-tannen (1) · unreached urapportert (1) · fersk store per base (1) · detach oppstartsnekten (2). EN MUTASJON FALSIFISERTE TESTEN FOERST (repoets vakuoes-gate-klasse, aattende gang): store-testen sammenlignet med ==, og VerdictStore er en pydantic-modell med VERDI-likhet - tre ulike tomme stores er alle like, saa «fersk store per base» lot hele suiten staa groenn. Delt INSTANS er paastanden, saa testen asserterer naa paa `is`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L3YHobQC3WzYVoxSgZus4d
This commit is contained in:
parent
3cbea91a72
commit
18af86e422
6 changed files with 1051 additions and 17 deletions
|
|
@ -703,9 +703,46 @@ class HypothesisParseError(ExplorationError):
|
|||
"""
|
||||
|
||||
|
||||
def _parse_hypotheses(texts: Sequence[str]) -> list[tuple[str, str]]:
|
||||
"""Every marked ``(label, rationale)`` pair the hypothesiser committed to, in turn order."""
|
||||
found: list[tuple[str, str]] = []
|
||||
def _resolve_hypothesis_bundle(raw: str, bundle_ids: Sequence[str]) -> str:
|
||||
"""Which knowledge base a marked hypothesis belongs to (§ C.7), fail-closed.
|
||||
|
||||
Three rules, and the middle one is the whole reason the field can default at all:
|
||||
|
||||
* a stated id must be one that was CONFIGURED — refused by name, never resolved by position,
|
||||
exactly as ``_resolve_bundle`` refuses an unknown base for the read tools. A hypothesis that
|
||||
names a base must not be treated more leniently than a read of one.
|
||||
* a silent marker with at most one base configured resolves to that base (or to ``""`` with
|
||||
none). With one configured base there is no other value the field could take, so this is the
|
||||
only answer rather than a guess.
|
||||
* a silent marker with SEVERAL configured refuses. A marked line is a claim the loop committed
|
||||
to; one that cannot be routed is a claim it could not finish making, and the marker is
|
||||
precisely what makes that affordable to refuse — an unmarked turn is not a hypothesis, so
|
||||
there is no silence being turned into an error.
|
||||
"""
|
||||
if raw:
|
||||
if raw not in bundle_ids:
|
||||
known = ", ".join(bundle_ids) or "(none configured)"
|
||||
raise ExplorationError(
|
||||
f"hypothesis names knowledge base {raw!r}, which is not configured for this "
|
||||
f"exploration; configured: {known}"
|
||||
)
|
||||
return raw
|
||||
if len(bundle_ids) <= 1:
|
||||
return bundle_ids[0] if bundle_ids else ""
|
||||
raise HypothesisParseError(
|
||||
f"a marked hypothesis must name its knowledge base when several are configured "
|
||||
f'({", ".join(bundle_ids)}): add "bundle_id" to the marker payload. Routing it here '
|
||||
"would evaluate a direction against a project nobody asked about"
|
||||
)
|
||||
|
||||
|
||||
def _parse_hypotheses(
|
||||
texts: Sequence[str], bundle_ids: Sequence[str]
|
||||
) -> list[tuple[str, str, str]]:
|
||||
"""Every marked ``(label, rationale, bundle_id)`` triple the hypothesiser committed to, in turn
|
||||
order. The base is resolved HERE rather than at minting time so an unroutable claim is refused
|
||||
while the line that made it is still in hand for the error message."""
|
||||
found: list[tuple[str, str, str]] = []
|
||||
for text in texts:
|
||||
for line in text.splitlines():
|
||||
stripped = line.strip()
|
||||
|
|
@ -724,12 +761,48 @@ def _parse_hypotheses(texts: Sequence[str]) -> list[tuple[str, str]]:
|
|||
"a marked hypothesis needs a non-empty 'label' and 'rationale'; got: "
|
||||
f"{stripped}"
|
||||
)
|
||||
found.append((str(data["label"]), str(data["rationale"])))
|
||||
found.append(
|
||||
(
|
||||
str(data["label"]),
|
||||
str(data["rationale"]),
|
||||
_resolve_hypothesis_bundle(str(data.get("bundle_id") or ""), bundle_ids),
|
||||
)
|
||||
)
|
||||
return found
|
||||
|
||||
|
||||
def refuse_unroutable_seeds(seeds: Sequence[Approach], bundle_ids: Sequence[str]) -> None:
|
||||
"""Refuse an expert's seed that could not be dispatched — BEFORE the first model call.
|
||||
|
||||
Validation, ALDRI reparasjon, and here the second half is the load-bearing one: a seed is the
|
||||
expert's own words and § C.6 door 1 is a PRESERVATION rule, so filling in a missing
|
||||
``bundle_id`` on their behalf would put their name on a routing decision they did not make.
|
||||
The single-base default belongs to ``route_by_bundle``, at consumption, where it is unambiguous
|
||||
by construction — never to a rewrite of the input.
|
||||
|
||||
Called before the loop starts rather than after it returns, and that placement is the økt-57
|
||||
outbox-hoist precedent: at the exception alone, a refusal after the exploration has spent its
|
||||
whole budget looks identical to one before it spent anything. The mandate is undispatchable
|
||||
either way; what is at stake is whether the expert pays to find that out.
|
||||
"""
|
||||
for seed in seeds:
|
||||
if seed.bundle_id:
|
||||
if seed.bundle_id not in bundle_ids:
|
||||
known = ", ".join(bundle_ids) or "(none configured)"
|
||||
raise ExplorationError(
|
||||
f"seed approach {seed.id!r} names knowledge base {seed.bundle_id!r}, which is "
|
||||
f"not configured for this exploration; configured: {known}"
|
||||
)
|
||||
elif len(bundle_ids) > 1:
|
||||
raise ExplorationError(
|
||||
f"seed approach {seed.id!r} names no knowledge base and {len(bundle_ids)} are "
|
||||
f"configured ({', '.join(bundle_ids)}); set Approach.bundle_id so the mandate can "
|
||||
"be dispatched to the base the hypothesis is actually about"
|
||||
)
|
||||
|
||||
|
||||
def _mint_approaches(
|
||||
seeds: Sequence[Approach], discovered: Sequence[tuple[str, str]]
|
||||
seeds: Sequence[Approach], discovered: Sequence[tuple[str, str, str]]
|
||||
) -> tuple[Approach, ...]:
|
||||
"""Seeds FIRST, untouched, then one approach per discovered direction.
|
||||
|
||||
|
|
@ -742,7 +815,7 @@ def _mint_approaches(
|
|||
taken = {approach.id for approach in seeds}
|
||||
minted: list[Approach] = list(seeds)
|
||||
counter = 0
|
||||
for label, rationale in discovered:
|
||||
for label, rationale, bundle_id in discovered:
|
||||
counter += 1
|
||||
candidate = f"hypothesis-{counter}"
|
||||
while candidate in taken or candidate == OWN_PROPOSAL_ID:
|
||||
|
|
@ -752,7 +825,12 @@ def _mint_approaches(
|
|||
# description is the hypothesiser's own words, VERBATIM: ``generate._build_messages`` feeds
|
||||
# an Approach.description to the proposer unchanged, and the reason a direction is worth
|
||||
# trying is exactly the half a model cannot re-derive from the cost table.
|
||||
minted.append(Approach(id=candidate, label=label, description=rationale))
|
||||
#
|
||||
# ``bundle_id`` is stamped on a MINTED approach because there is nothing here to preserve —
|
||||
# the opposite call from the seeds above, which pass through untouched (§ C.6 door 1).
|
||||
minted.append(
|
||||
Approach(id=candidate, label=label, description=rationale, bundle_id=bundle_id)
|
||||
)
|
||||
return tuple(minted)
|
||||
|
||||
|
||||
|
|
@ -792,13 +870,19 @@ async def explore(
|
|||
returned mandate whatever the loop found, including when the loop found nothing and including
|
||||
when it stopped early.
|
||||
|
||||
Honesty limits, stated rather than implied. (1) A run is bound to ONE mandate; multi-base
|
||||
dispatch (``Approach.bundle_id``, § C.7) waits for ``run_project`` to accept more than one
|
||||
``bundle_dir``, and shipping the field before its consumer would be a shape guessed instead of
|
||||
measured. (2) The ``quick_validate`` verdicts the hypothesiser saw are not in
|
||||
**Multi-base (§ C.7).** Every approach in the returned mandate carries ``bundle_id``, and
|
||||
``mandate.route_by_bundle`` partitions the commission by it so ``run.run_mandate_across_bundles``
|
||||
can run the pipeline once per base. ``run_project`` itself still takes ONE ``bundle_dir``, and
|
||||
that is the measured shape rather than a step not yet taken: it derives the project, the
|
||||
validator's cost baseline, the read context and the ExpeL query key from THE bundle and returns
|
||||
one stamped ``RunResult``, so a second directory on that signature would force a silent
|
||||
pick-one for all four.
|
||||
|
||||
Honesty limits, stated rather than implied. (1) The ``quick_validate`` verdicts the
|
||||
hypothesiser saw are not in
|
||||
``ExplorationResult``: they are level-1 advisory, and their home is the
|
||||
``{run_id}-exploration.json`` artefact the CLI wiring writes — pass an ``ExplorationTrace`` to
|
||||
collect them. (3) The exploration roles resolve through ``resolve_model``'s ``default``
|
||||
collect them. (2) The exploration roles resolve through ``resolve_model``'s ``default``
|
||||
fallback unless an operator maps them explicitly.
|
||||
|
||||
``trace`` is the caller's accumulator and is the ONLY way to see what a run that RAISED
|
||||
|
|
@ -816,6 +900,12 @@ async def explore(
|
|||
"requested and the reviewer would never be called (refused, never silently ignored)"
|
||||
)
|
||||
|
||||
# The base ids, resolved ONCE and before anything is built: they are what a seed and a marked
|
||||
# hypothesis are both routed against, and ``_bundle_index`` is the one place that decides them
|
||||
# (a duplicate basename refuses here rather than letting the manager read A believing it read B).
|
||||
bundle_ids = tuple(_bundle_index(bundle_dirs))
|
||||
refuse_unroutable_seeds(seed_approaches, bundle_ids)
|
||||
|
||||
if meter is None:
|
||||
meter = TokenMeter(Budget(max_tokens=contract.max_tokens, max_rounds=contract.max_rounds))
|
||||
if client_factory is None:
|
||||
|
|
@ -912,7 +1002,9 @@ async def explore(
|
|||
if stop is None:
|
||||
stop = _classify_stop(ledger_log, replans=replans, contract=contract)
|
||||
|
||||
discovered = _parse_hypotheses(hypothesis_texts) if stop != "unknown_speaker" else []
|
||||
discovered = (
|
||||
_parse_hypotheses(hypothesis_texts, bundle_ids) if stop != "unknown_speaker" else []
|
||||
)
|
||||
return ExplorationResult(
|
||||
mandate=Mandate(
|
||||
objective=prompt,
|
||||
|
|
|
|||
|
|
@ -29,6 +29,7 @@ Two refusals are load-bearing, both at construction time:
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
|
@ -50,6 +51,14 @@ class Approach(BaseModel):
|
|||
id: str = Field(min_length=1)
|
||||
label: str = Field(min_length=1)
|
||||
description: str = ""
|
||||
#: Which knowledge base this approach belongs to (§ C.7), by the base's id — the directory's
|
||||
#: BASENAME, exactly as ``explore._bundle_index`` names it. DEFAULTS to empty, meaning "no base
|
||||
#: named": a legitimate statement when the run has only one base to name, and what keeps every
|
||||
#: mandate written before multi-base existed valid and dispatchable unchanged.
|
||||
#:
|
||||
#: The field is a ROUTING key, never a claim about content. It says which pipeline the approach
|
||||
#: must be evaluated in, and ``route_by_bundle`` is the one place that reads it.
|
||||
bundle_id: str = ""
|
||||
|
||||
|
||||
class Mandate(BaseModel):
|
||||
|
|
@ -88,6 +97,85 @@ class Mandate(BaseModel):
|
|||
return self
|
||||
|
||||
|
||||
class MandateRoutingError(ValueError):
|
||||
"""A commission that cannot be executed against the bases it was given (§ C.7).
|
||||
|
||||
**A ``ValueError`` by construction, and that is a measurement rather than a taxonomy note.**
|
||||
økt 57 paid for the opposite: ``ExplorationError`` is a ``RuntimeError`` and therefore fell
|
||||
outside ``run.main``'s ``(ValueError, FileNotFoundError, ValidationError)`` refusal tuple and
|
||||
outside ``hosting``'s 400 arm, so a caller's configuration mistake would have left as a
|
||||
traceback on one surface and a 500 — the crash channel — on the other. A routing refusal is
|
||||
exactly that class of caller mistake, so it is born inside both nets instead of being
|
||||
retrofitted into them later.
|
||||
"""
|
||||
|
||||
|
||||
def route_by_bundle(mandate: Mandate, bundle_ids: Sequence[str]) -> tuple[tuple[str, Mandate], ...]:
|
||||
"""Partition one commission into one sub-mandate PER knowledge base (§ C.7).
|
||||
|
||||
This is the whole of "multi-base", and it is deliberately a partition rather than a widening.
|
||||
``run_project`` derives four single-valued things from THE bundle it is given — the project
|
||||
(``_project_from_bundle`` fail-fasts when the bundle's own ``project_id`` is not the requested
|
||||
one), the validator's stage-0 cost baseline, the agents' read context and the ExpeL query key —
|
||||
and returns ONE ``RunResult`` with ONE ``ProvenanceStamp``. A second ``bundle_dir`` on that
|
||||
signature would force a silent pick-one for all four. § C.7 says the same thing in its own
|
||||
words: *pipelinen kjøres per bundle som i dag* — N calls, not one call taking N.
|
||||
|
||||
Order is taken from ``bundle_ids``, never from first appearance among the approaches, so the
|
||||
dispatch's spend order is a property of how the run was configured rather than of how a model
|
||||
happened to sequence its hypotheses.
|
||||
|
||||
**Fail-fast on a commission that cannot be executed as written**, mirroring ``load_mandate``'s
|
||||
contract for exactly its reason: a run must never proceed on a *silently degraded* commission,
|
||||
because the coverage report would then describe work nobody ordered. Two ways that happens, and
|
||||
both refuse by name rather than resolving by position (the S3.2 key-collision class):
|
||||
|
||||
* an approach naming a base that was not configured;
|
||||
* an approach naming NO base while more than one is configured — with a single base there is no
|
||||
other value the field could take, so resolving it there is the only answer rather than a
|
||||
guess, and it is what keeps every pre-multi-base mandate dispatchable unchanged.
|
||||
|
||||
A base that no approach names is NOT run: a run costs money and the commission ordered nothing
|
||||
for it. The single-base case keeps its own rule above, so an own-proposals-only mandate still
|
||||
reaches the one base it could possibly mean.
|
||||
|
||||
:raises MandateRoutingError: no bases configured, or an approach that cannot be routed.
|
||||
"""
|
||||
if not bundle_ids:
|
||||
raise MandateRoutingError(
|
||||
"a mandate cannot be routed against zero knowledge bases: an empty plan reads as "
|
||||
"'there was nothing to do', which is indistinguishable from a commission that was "
|
||||
"fully evaluated against nothing"
|
||||
)
|
||||
|
||||
known = tuple(bundle_ids)
|
||||
sole = known[0] if len(known) == 1 else None
|
||||
grouped: dict[str, list[Approach]] = {bundle_id: [] for bundle_id in known}
|
||||
for approach in mandate.approaches:
|
||||
target = approach.bundle_id or sole
|
||||
if target is None:
|
||||
raise MandateRoutingError(
|
||||
f"approach {approach.id!r} names no knowledge base and {len(known)} are "
|
||||
f"configured ({', '.join(known)}); which one it belongs to is not something this "
|
||||
"layer may decide on the expert's behalf"
|
||||
)
|
||||
if target not in grouped:
|
||||
raise MandateRoutingError(
|
||||
f"approach {approach.id!r} names knowledge base {target!r}, which is not "
|
||||
f"configured for this run; configured: {', '.join(known)}"
|
||||
)
|
||||
grouped[target].append(approach)
|
||||
|
||||
return tuple(
|
||||
(
|
||||
bundle_id,
|
||||
mandate.model_copy(update={"approaches": tuple(grouped[bundle_id])}),
|
||||
)
|
||||
for bundle_id in known
|
||||
if grouped[bundle_id] or sole is not None
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ApproachOutcome:
|
||||
"""What became of ONE commissioned approach — one row of the run's coverage report.
|
||||
|
|
|
|||
|
|
@ -70,8 +70,10 @@ from portfolio_optimiser.mandate import (
|
|||
Approach,
|
||||
ApproachOutcome,
|
||||
Mandate,
|
||||
MandateRoutingError,
|
||||
announce,
|
||||
load_mandate,
|
||||
route_by_bundle,
|
||||
settle,
|
||||
)
|
||||
from portfolio_optimiser.mcp_tools import (
|
||||
|
|
@ -1343,6 +1345,187 @@ async def run_portfolio(
|
|||
return base
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BundleRun:
|
||||
"""One knowledge base's run inside a multi-base dispatch (§ C.7).
|
||||
|
||||
``bundle_id`` and ``project_id`` are carried BESIDE the result rather than read back off it,
|
||||
because they are what the dispatch ROUTED on: the id is how the mandate named the base, and the
|
||||
project is what that base's own IR projection says it is. A reader pairing a coverage row back
|
||||
to a base should not have to re-derive either.
|
||||
"""
|
||||
|
||||
bundle_id: str
|
||||
bundle_dir: str
|
||||
project_id: str
|
||||
result: RunResult
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MultiBaseResult:
|
||||
"""What one multi-base dispatch produced: one run per base the commission named (§ C.7).
|
||||
|
||||
A DISTINCT type from ``PortfolioResult``, and deliberately so. ``PortfolioResult`` keys on
|
||||
"one ``RunResult`` per PROJECT in input order" and its ``runs``/``failures`` partition *the
|
||||
projects that were actually submitted*; this keys on the KNOWLEDGE BASE the mandate routed each
|
||||
approach to. Two bases may legitimately describe one project, which the project-keyed shape
|
||||
cannot express at all — ``run_portfolio`` reads each base off ``projects[pid].bundle_dir``, so
|
||||
a pid admits exactly one base there. Reusing the type would fuse two axes.
|
||||
|
||||
``unreached`` is the ``not_evaluated`` rule at the dispatch layer: when the global cap stops the
|
||||
pass, every approach in a base that was never started is reported as unreached rather than
|
||||
omitted. An omitted row is indistinguishable from an approach nobody commissioned, which is the
|
||||
silence the coverage report exists to remove.
|
||||
"""
|
||||
|
||||
runs: tuple[BundleRun, ...]
|
||||
store: VerdictStore
|
||||
stopped_early: bool = False
|
||||
budget_stop: BudgetStop | None = None
|
||||
unreached: tuple[ApproachOutcome, ...] = ()
|
||||
|
||||
|
||||
async def run_mandate_across_bundles(
|
||||
mandate: Mandate,
|
||||
bundle_dirs: Sequence[str],
|
||||
profile: Profile | str = Profile.LOCAL,
|
||||
*,
|
||||
verdict_input: dict[str, str],
|
||||
store: VerdictStore | None = None,
|
||||
verdict_dir: str | None = None,
|
||||
dimension: Dimension | None = None,
|
||||
client_factory: Callable[[str], BaseChatClient] | None = None,
|
||||
max_rounds: int = _DEFAULT_MAX_ROUNDS,
|
||||
max_tokens: int = _DEFAULT_MAX_TOKENS,
|
||||
top_k: int = 3,
|
||||
portfolio_meter: PortfolioMeter | None = None,
|
||||
) -> MultiBaseResult:
|
||||
"""Evaluate ONE commission across SEVERAL knowledge bases — the multi-base dispatch (§ C.7).
|
||||
|
||||
``mandate.route_by_bundle`` partitions the commission by ``Approach.bundle_id``, and this runs
|
||||
the EXISTING ``run_project`` once per base with that base's own sub-mandate. Composition, never
|
||||
a widening: ``run_project`` derives the project, the validator's stage-0 baseline, the agents'
|
||||
read context and the ExpeL query key from THE bundle it is handed, and returns one stamped
|
||||
``RunResult`` — a second ``bundle_dir`` on that signature would force a silent pick-one for all
|
||||
four. § C.7 words it the same way: *pipelinen kjøres per bundle som i dag*.
|
||||
|
||||
**There is no ``project_id`` argument, and that is the design rather than an omission.** Each
|
||||
base's project is read from THAT base's own IR projection (``okf.load_ir_projection``), which is
|
||||
the same value ``_project_from_bundle`` already fail-fasts against — so a caller-supplied
|
||||
constant could only ever be right for one base out of N. Turning the existing fail-fast into the
|
||||
routing key removes the guess entirely.
|
||||
|
||||
ONE ``VerdictStore`` is threaded across every base, exactly as ``run_portfolio`` threads one
|
||||
across every project: a verdict captured while evaluating base k must be able to reach base
|
||||
k+1's hypothesis, and a fresh store per base would leave the loop looking wired while carrying
|
||||
nothing between the runs.
|
||||
|
||||
**The budget has the two S3.4 teeth that apply here, and no more.** With a ``portfolio_meter``:
|
||||
a remainder that cannot fund one run refuses at STARTUP (``BudgetRefused`` — a pass with room
|
||||
for zero runs is a caller error, not a result), and a base that cannot be funded is NEVER
|
||||
STARTED, the pass stopping with ``budget_stop`` and every unreached approach reported. Never
|
||||
started is the whole point: a base merely aborted mid-run has already cost real model calls.
|
||||
The wave-reservation tooth has no counterpart here — this dispatch is SEQUENTIAL, so there is
|
||||
no wave of runs funded off one pre-wave remainder to over-commit.
|
||||
|
||||
Honesty limits, stated rather than implied. (1) A base that RAISES propagates; the
|
||||
collect-and-continue policy belongs to ``run_portfolio``, where the caller submitted a batch of
|
||||
independent projects, whereas here the caller asked for ONE commission to be evaluated.
|
||||
(2) Without a ``portfolio_meter`` the pass's ceiling is the number of routed bases times
|
||||
``max_tokens``, each run bounded on its own — the global ledger is opt-in, and this does not
|
||||
re-implement it (``_run_meter`` is the one copy of the binding rule). (3) The outbox is NOT
|
||||
wired: N runs need N ``run_id``s, and minting them here would default a key this repo requires
|
||||
a caller to supply, for byte-determinism. A caller who needs artefacts per base calls
|
||||
``run_project`` itself with the sub-mandates ``route_by_bundle`` hands back.
|
||||
|
||||
:raises MandateRoutingError: the commission cannot be routed against ``bundle_dirs``.
|
||||
:raises BudgetRefused: a global remainder that cannot fund a single run.
|
||||
"""
|
||||
by_id: dict[str, str] = {}
|
||||
for raw in bundle_dirs:
|
||||
bundle_id = Path(raw).name
|
||||
if bundle_id in by_id:
|
||||
# The same refusal ``explore._bundle_index`` makes, for the same reason: the id is how
|
||||
# the mandate names a base, so two bases answering to one name would let an approach be
|
||||
# evaluated against A while the report says B (the S3.2 key-collision class).
|
||||
raise MandateRoutingError(
|
||||
f"two knowledge bases share the id {bundle_id!r} ({by_id[bundle_id]!r} and "
|
||||
f"{raw!r}); an approach names a base by that id, so it must be unique"
|
||||
)
|
||||
by_id[bundle_id] = raw
|
||||
|
||||
routed = route_by_bundle(mandate, tuple(by_id))
|
||||
|
||||
if portfolio_meter is not None and not portfolio_meter.can_fund_run():
|
||||
raise BudgetRefused(portfolio_meter.remaining(), portfolio_meter.required_per_run)
|
||||
|
||||
store = store if store is not None else VerdictStore(verdicts=[])
|
||||
runs: list[BundleRun] = []
|
||||
unreached: list[ApproachOutcome] = []
|
||||
budget_stop: BudgetStop | None = None
|
||||
|
||||
for index, (bundle_id, sub_mandate) in enumerate(routed):
|
||||
if portfolio_meter is not None and not portfolio_meter.can_fund_run():
|
||||
budget_stop = BudgetStop(
|
||||
limit_tokens=portfolio_meter.budget.max_total_tokens,
|
||||
spent_tokens=portfolio_meter.spent,
|
||||
remaining_tokens=portfolio_meter.remaining(),
|
||||
required_tokens=portfolio_meter.required_per_run,
|
||||
)
|
||||
unreached.extend(
|
||||
ApproachOutcome(
|
||||
id=approach.id,
|
||||
label=approach.label,
|
||||
status="not_evaluated",
|
||||
detail=(f"budget exhausted before knowledge base {pending_id!r} was run"),
|
||||
)
|
||||
for pending_id, pending in routed[index:]
|
||||
for approach in pending.approaches
|
||||
)
|
||||
break
|
||||
|
||||
bundle_dir = by_id[bundle_id]
|
||||
# ONE reading of the base's own project id, used both to ADDRESS the run and to LABEL it.
|
||||
# A second lookup for the label would be the kø-(p) duplicate free to drift from the value
|
||||
# the run was actually dispatched with.
|
||||
project_id = str(okf.load_ir_projection(bundle_dir)["project_id"])
|
||||
result = cast(
|
||||
RunResult,
|
||||
await run_project(
|
||||
project_id,
|
||||
profile,
|
||||
docs_dir=bundle_dir,
|
||||
bundle_dir=bundle_dir,
|
||||
verdict_input=verdict_input,
|
||||
verdict_dir=verdict_dir,
|
||||
dimension=dimension,
|
||||
store=store,
|
||||
client_factory=client_factory,
|
||||
max_rounds=max_rounds,
|
||||
max_tokens=max_tokens,
|
||||
top_k=top_k,
|
||||
mandate=sub_mandate,
|
||||
meter=_run_meter(None, portfolio_meter, max_rounds),
|
||||
),
|
||||
)
|
||||
runs.append(
|
||||
BundleRun(
|
||||
bundle_id=bundle_id,
|
||||
bundle_dir=bundle_dir,
|
||||
project_id=project_id,
|
||||
result=result,
|
||||
)
|
||||
)
|
||||
|
||||
return MultiBaseResult(
|
||||
runs=tuple(runs),
|
||||
store=store,
|
||||
stopped_early=budget_stop is not None,
|
||||
budget_stop=budget_stop,
|
||||
unreached=tuple(unreached),
|
||||
)
|
||||
|
||||
|
||||
# The roles ``debate``/``generate`` ask the factory for. Fixed here so a malformed replies file is
|
||||
# caught at the door instead of mid-run.
|
||||
_SCRIPTED_ROLES = ("proposer", "checker")
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue