open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 1 Activity

feat(budget): enforce a global portfolio token cap before the call, not after it (S3.4/F10)

Browse source 
PortfolioBudget + PortfolioMeter carry ONE token ledger over a whole portfolio
pass -- and, seeded from a persisted spend file, across passes -- while the
per-run Budget/TokenMeter pair is untouched. Three enforcement points, each
doing a different job:

- startup: a remainder that cannot fund one run raises BudgetRefused before
  anything loads (a pass that can afford zero projects is a caller mistake,
  not a result);
- wave assembly: an unfundable project is NEVER STARTED and the pass stops
  structurally (budget_stop + stopped_early, completed runs preserved).
  Because every member of a wave is funded against the SAME pre-wave
  remainder, admission RESERVES each member's requirement -- otherwise a wave
  of k over-commits the cap by up to k runs;
- pre-call: BudgetMiddleware refuses a call the remainder cannot pay for
  instead of making it. The post-charge check stays: real usage is only
  knowable after the response, so the guard stops the NEXT call, never the
  one in flight.

budget_stop is its own field rather than a widened stop_reason -- a goal-stop
is success, this is resource exhaustion, and fusing them would make "we
stopped" unreadable. PortfolioMeter splits record/check so tokens the provider
already billed reach the ledger even when the same charge breaks the run's own
cap. read_spend raises on corrupt content (our own accounting state, unlike
the tolerant RAW inbox layer); write_spend takes a REQUIRED stamp with no
wall-clock default, mirroring promote_verdict.

Load-bearing MEASURED, not asserted -- 6 mutations, all red: detach the wave
check; detach the pre-call guard; detach the wave reservation; check the run
cap before crediting the global ledger; detach the startup refusal; make
read_spend tolerant. Files restored from shasum-verified copies after each.

Two findings worth keeping: the pre-call guard MASKS a detached wave check if
the test asserts on overspend (spend stays under the cap either way), so the
load-bearing assertion had to become failures == () plus never-started; and
the token arithmetic is probed (32 tokens/run at tokens=8), not guessed.

537 -> 553 tests, ruff + mypy green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015EaxFnaDAbMQkmTeX4u7sd
This commit is contained in:
Kjell Tore Guttormsen 2026-07-31 21:34:48 +02:00
commit a831aa1e3b
6 changed files with 733 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

122
src/portfolio_optimiser/run.py
View file

@ -35,7 +35,13 @@ from agent_framework import BaseChatClient, SessionContext
from pydantic import ValidationError
from portfolio_optimiser.backends import Profile, get_backend, resolve_model
from portfolio_optimiser.budget import Budget, BudgetMiddleware, TokenMeter
from portfolio_optimiser.budget import (
Budget,
BudgetMiddleware,
BudgetRefused,
PortfolioMeter,
TokenMeter,
)
from portfolio_optimiser.contracts import GoalConfig, GoalContract, load_contracts, load_goal_config
from portfolio_optimiser.ledger import SavingsLedger
from portfolio_optimiser.datasource import (
@ -141,6 +147,25 @@ class GoalReached:
observed_ore: int
@dataclass(frozen=True)
class BudgetStop:
"""A GLOBAL token-cap stop signal VALUE (S3.4/F10) — NOT an exception, and NOT a goal.
Structured like ``GoalReached`` and carried the same way (a ``stop_reason``-shaped value plus a
loop ``break``), but it is kept as its OWN field rather than widening ``stop_reason``: the two
stops mean opposite things. A goal-stop is success (the savings target was met); this is
resource exhaustion (the pass ran out of tokens). Folding them into one field would let a
caller read "we stopped" without being able to tell which happened.
``required_tokens`` is what one more run would have needed; ``remaining_tokens`` is what was
actually left. Both are recorded because their DIFFERENCE is the operator's next decision."""
limit_tokens: int
spent_tokens: int
remaining_tokens: int
required_tokens: int
@dataclass(frozen=True)
class PortfolioResult:
"""The outcome of a sequential fan-out over N projects (SC2).
@ -150,9 +175,11 @@ class PortfolioResult:
The remaining fields are a thin aggregate over ``runs``: ``validated_count`` /
``rejected_count`` partition the outcomes; ``sum_claimed_saving_nok`` totals the claimed
saving of the validated proposals only; ``sum_token_usage`` totals every run's
provenance token usage. ``stopped_early`` / ``stop_reason`` record a Step-8 goal-stop, and
``failures`` records the projects that RAISED (S3.3 collect-and-continue): all three default so
the frozen aggregate and every existing constructor call are unaffected.
provenance token usage. ``stopped_early`` / ``stop_reason`` record a Step-8 goal-stop,
``failures`` records the projects that RAISED (S3.3 collect-and-continue), and ``budget_stop``
records a S3.4 global-token-cap stop (which also sets ``stopped_early``, but is kept apart from
``stop_reason`` because exhaustion is not success): all four default so the frozen aggregate and
every existing constructor call are unaffected.
``runs`` and ``failures`` PARTITION the projects that were actually submitted — a project
appears in exactly one of them, never both, and the counts do not overlap. ``validated_count`` /
@ -168,6 +195,7 @@ class PortfolioResult:
stopped_early: bool = False
stop_reason: GoalReached | None = None
failures: tuple[RunFailure, ...] = ()
budget_stop: BudgetStop | None = None
def _authored_texts(result: Any, name: str) -> list[str]:
@ -647,6 +675,30 @@ def _waves(ids: list[str], k: int) -> list[list[str]]:
return [ids[i : i + k] for i in range(0, len(ids), k)]
def _run_meter(
meter_factory: Callable[[], TokenMeter] | None,
portfolio_meter: PortfolioMeter | None,
max_rounds: int,
) -> TokenMeter | None:
"""The per-project meter ``run_portfolio`` hands to one run: the injected test seam, a meter
BOUND to the portfolio ledger, or ``None`` (letting ``run_project`` build its own, unchanged).
The bound meter mirrors ``run_project``'s own construction (``max(max_rounds * 4, 4)``) so the
only difference the portfolio cap introduces is the token ceiling and the shared ledger — the
round budget is not quietly redefined along the way."""
if meter_factory is not None:
return meter_factory()
if portfolio_meter is None:
return None
return TokenMeter(
Budget(
max_tokens=portfolio_meter.budget.max_tokens_per_run,
max_rounds=max(max_rounds * 4, 4),
),
portfolio=portfolio_meter,
)
async def run_portfolio(
project_ids: Sequence[str] | None = None,
profile: Profile | str = Profile.LOCAL,
@ -661,6 +713,7 @@ async def run_portfolio(
top_k: int = 3,
concurrency: int = 1,
meter_factory: Callable[[], TokenMeter] | None = None,
portfolio_meter: PortfolioMeter | None = None,
semantic_retrieval: bool = False,
embedder: Embedder | None = None,
) -> PortfolioResult:
@ -727,12 +780,46 @@ async def run_portfolio(
property of the design, so it is pinned on a bundle-backed pair where the fold does fire
(``test_intra_wave_visibility_is_the_documented_semantic_difference``): same fixture, same
sentinel, only the wave boundary moves. Store content stays identical across ``k`` — the
difference is confined to what each project READ, never to what the pass produced or persisted."""
difference is confined to what each project READ, never to what the pass produced or persisted.
``portfolio_meter`` (S3.4, F10) installs the GLOBAL token cap. Without it nothing changes: each
run is bounded only by its own ``max_tokens``, so N projects can cost N times that with no
ceiling over the pass. With it, one ``PortfolioMeter`` is shared by every run (each run's meter
is BOUND to it, which is why ``meter_factory`` — whose meters are unbound — is refused
alongside it: accepting both would run a pass that looks capped and is not), and the cap is
enforced in three places that are deliberately different:
- **At startup**, a remainder that cannot fund one run raises ``BudgetRefused`` — a pass that
can afford zero projects is a caller mistake, not a result.
- **At wave assembly**, a project that cannot be funded is NEVER STARTED, and the pass stops
with ``budget_stop`` + ``stopped_early``, every completed run preserved. Never-started is the
property that matters: an unfunded project that is merely interrupted mid-run has already
cost model round-trips. Because every member of a wave is checked against the SAME pre-wave
remainder, admission RESERVES each member's requirement as it goes — otherwise a wave of k
would over-commit the cap by up to k runs. This makes membership identical at every
``concurrency``, matching the goal-stop's guarantee above.
- **Before each chat call** (``BudgetMiddleware``'s pre-call guard), a call the remainder cannot
pay for is refused rather than made. This is what bounds a run that was funded at admission
but whose siblings drained the pass while it was in flight; such a run surfaces as a
``RunFailure``, not as overspend.
Spend is carried ACROSS passes by seeding the meter from ``budget.read_spend`` and writing
``budget.write_spend`` afterwards. Those are the caller's calls, not this function's — the pass
reads its ledger, it does not own the file (mirroring the Steg-7 role split)."""
if concurrency < 1:
raise ValueError(
f"concurrency must be >= 1, got {concurrency}: a non-positive wave size would run no "
"projects at all and read as an empty portfolio. Pass 1 for the sequential pass."
)
if portfolio_meter is not None and meter_factory is not None:
raise ValueError(
"portfolio_meter and meter_factory are mutually exclusive: a meter_factory meter is "
"not bound to the portfolio ledger, so the global cap would be silently unenforced"
)
# Startup refusal (fail-fast, before anything loads): a pass that cannot fund its first run
# must not begin. Raised, not returned — an empty PortfolioResult would read as "nothing to do".
if portfolio_meter is not None and not portfolio_meter.can_fund_run():
raise BudgetRefused(portfolio_meter.remaining(), portfolio_meter.required_per_run)
projects = {p.id: p for p in load_reference_projects()}
ids = list(project_ids) if project_ids is not None else list(projects)
store = store if store is not None else VerdictStore(verdicts=[])
@ -747,8 +834,13 @@ async def run_portfolio(
failures: list[RunFailure] = []
stopped_early = False
stop_reason: GoalReached | None = None
budget_stop: BudgetStop | None = None
for wave_ids in _waves(ids, concurrency):
members: list[str] = []
# Tokens committed to members already admitted to THIS wave but that have not spent yet.
# Every member reads the same pre-wave remainder, so without this a wave of k would admit
# k projects off one project's worth of budget.
reserved = 0
for pid in wave_ids:
if pid not in projects:
raise ValueError(f"unknown project_id: {pid!r}")
@ -777,6 +869,21 @@ async def run_portfolio(
if per_project_goal.mode == "hard":
continue # skip THIS pid; the rest of the pass proceeds
# S3.4 funding check, placed AFTER the goal checks: a reached goal is success and owns
# the stop when both apply, and a pid a hard per-project goal already skipped costs no
# budget, so it must not consume a reservation.
if portfolio_meter is not None and not portfolio_meter.can_fund_run(reserved=reserved):
stopped_early = True
budget_stop = BudgetStop(
limit_tokens=portfolio_meter.budget.max_total_tokens,
spent_tokens=portfolio_meter.spent,
remaining_tokens=portfolio_meter.remaining(),
required_tokens=portfolio_meter.required_per_run,
)
break
if portfolio_meter is not None:
reserved += portfolio_meter.required_per_run
members.append(pid)
# Every project in the wave reads the SAME wave-start state and writes only its own copy,
@ -805,7 +912,7 @@ async def run_portfolio(
top_k=top_k,
semantic_retrieval=semantic_retrieval,
embedder=embedder,
meter=meter_factory() if meter_factory is not None else None,
meter=_run_meter(meter_factory, portfolio_meter, max_rounds),
)
for pid, snapshot in snapshots
),
@ -836,12 +943,13 @@ async def run_portfolio(
break
base = _aggregate(tuple(runs), store)
if stopped_early or stop_reason is not None or failures:
if stopped_early or stop_reason is not None or failures or budget_stop is not None:
return replace(
base,
stopped_early=stopped_early,
stop_reason=stop_reason,
failures=tuple(failures),
budget_stop=budget_stop,
)
return base