feat(budget): enforce a global portfolio token cap before the call, not after it (S3.4/F10)

PortfolioBudget + PortfolioMeter carry ONE token ledger over a whole portfolio
pass -- and, seeded from a persisted spend file, across passes -- while the
per-run Budget/TokenMeter pair is untouched. Three enforcement points, each
doing a different job:

- startup: a remainder that cannot fund one run raises BudgetRefused before
  anything loads (a pass that can afford zero projects is a caller mistake,
  not a result);
- wave assembly: an unfundable project is NEVER STARTED and the pass stops
  structurally (budget_stop + stopped_early, completed runs preserved).
  Because every member of a wave is funded against the SAME pre-wave
  remainder, admission RESERVES each member's requirement -- otherwise a wave
  of k over-commits the cap by up to k runs;
- pre-call: BudgetMiddleware refuses a call the remainder cannot pay for
  instead of making it. The post-charge check stays: real usage is only
  knowable after the response, so the guard stops the NEXT call, never the
  one in flight.

budget_stop is its own field rather than a widened stop_reason -- a goal-stop
is success, this is resource exhaustion, and fusing them would make "we
stopped" unreadable. PortfolioMeter splits record/check so tokens the provider
already billed reach the ledger even when the same charge breaks the run's own
cap. read_spend raises on corrupt content (our own accounting state, unlike
the tolerant RAW inbox layer); write_spend takes a REQUIRED stamp with no
wall-clock default, mirroring promote_verdict.

Load-bearing MEASURED, not asserted -- 6 mutations, all red: detach the wave
check; detach the pre-call guard; detach the wave reservation; check the run
cap before crediting the global ledger; detach the startup refusal; make
read_spend tolerant. Files restored from shasum-verified copies after each.

Two findings worth keeping: the pre-call guard MASKS a detached wave check if
the test asserts on overspend (spend stays under the cap either way), so the
load-bearing assertion had to become failures == () plus never-started; and
the token arithmetic is probed (32 tokens/run at tokens=8), not guessed.

537 -> 553 tests, ruff + mypy green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015EaxFnaDAbMQkmTeX4u7sd
This commit is contained in:
Kjell Tore Guttormsen 2026-07-31 21:34:48 +02:00
commit a831aa1e3b
6 changed files with 733 additions and 11 deletions

View file

@ -105,6 +105,29 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
(`tests/test_step8_promotion_loadbearing.py`): gaten avviser ikke-godkjent dom (RØD uten gate); godkjent
dom er navigerbar (RØD når `link_in_index` detaches); promotert signal holdes ute av `bundle_context`
(RØD når en beskrivende index-label lekker det inn). Index-RMW er ikke-atomisk (enprosess-MVP).
- **Globalt token-tak håndheves FØR kall, aldri bare etterpå (S3.4, F10):** `PortfolioBudget` +
`PortfolioMeter` er ÉN ledger over hele porteføljepasset (og — seedet av `read_spend` — på tvers
av pass), mens per-run `Budget`/`TokenMeter` er uendret. Taket har tre tenner, med hver sin jobb:
(1) **oppstartsnekt** — en rest som ikke kan finansiere én kjøring raiser `BudgetRefused` FØR noe
lastes (et pass som har råd til null prosjekter er en caller-feil, ikke et resultat);
(2) **wave-assembly** — et prosjekt som ikke kan finansieres blir ALDRI STARTET, og passet stopper
strukturert (`budget_stop` + `stopped_early`, fullførte runs bevart). Aldri-startet er poenget:
et ufinansiert prosjekt som bare avbrytes har allerede kostet kall. Fordi hele bølgen sjekkes mot
SAMME før-bølge-rest, **reserverer** admission hver members krav — ellers overforplikter en bølge
av k taket med inntil k kjøringer; (3) **pre-call-guard** i `BudgetMiddleware` — et kall resten
ikke kan betale for NEKTES i stedet for å gjøres (post-charge-sjekken består: ekte usage kjennes
først etterpå, så guarden stopper NESTE kall, aldri det som er i lufta). `budget_stop` er et EGET
felt, aldri `stop_reason`: et mål-stopp er suksess, dette er ressurs-utmattelse — å slå dem sammen
ville gjort «vi stoppet» uleselig. `record`/`check` er SPLITTET i `PortfolioMeter` fordi tokens
leverandøren allerede har fakturert må nå ledgeren selv når samme charge bryter run-taket.
Spend-fila er vår EGEN regnskapstilstand: `read_spend` raiser på korrupt innhold (kontrast det
tolerante RAW-inbox-laget — å lese korrupt som null ville gitt tilbake et allerede brukt budsjett),
og `write_spend` tar et PÅKREVD `stamp` uten wall-clock-default (byte-determinisme, speiler
`promote_verdict`). `portfolio_meter` og `meter_factory` er gjensidig utelukkende — en
factory-meter er ubundet, så begge sammen ville gitt et pass som SER capped ut uten å være det.
Load-bearing MÅLT (`tests/test_portfolio_budget_loadbearing.py` + `tests/test_budget.py`), seks
mutasjoner alle røde: detach wave-sjekken · detach pre-call-guarden · detach bølge-reservasjonen ·
sjekk run-taket før global kreditering · detach oppstartsnekten · gjør `read_spend` tolerant.
- **Kostnadsdisiplin:** utvikle primært på lokal profil (gratis); Foundry/Azure (privat tenant finnes) kun til målrettet, minimal verifisering; billigste modeller + små syntetiske data + harde token-tak. Ingen tunge test-kjøringer.
- **Offline simulering = primært metode-bevis (kostnadsdrevet, erstatter §11.8):** operatøren kjører
IKKE MAF mot ekte modell (verken Azure/Foundry eller Ollama — API for begge repoene er for kostbart

View file

@ -149,6 +149,18 @@ when the seam is detached, so the loop cannot silently degrade into theater.
not CLI-enforced. Stop criteria and budget caps are required at startup. Try the offline
end-to-end proof (no model, no network): `uv run python -m portfolio_optimiser.simulation`.
- **A global token cap across the whole portfolio, enforced before the call.** Per-run caps alone
let N projects cost N times that with no ceiling over the pass. Pass a
`PortfolioMeter(PortfolioBudget(max_total_tokens=…, max_tokens_per_run=…))` to `run_portfolio`
and one ledger bounds the entire pass — and, seeded from `budget.read_spend`, a *series* of
passes. It bites in three places: a remainder that cannot fund one run refuses the pass at
startup (`BudgetRefused`); a project that cannot be funded is **never started**, stopping the
pass structurally (`budget_stop`, completed runs preserved); and a chat call the remainder
cannot pay for is **refused rather than made** (the post-charge check remains, since real usage
is only knowable after the response). Spend persists via `budget.write_spend`, which takes an
explicit stamp and no wall-clock default, so the file is byte-deterministic. Python API only —
not yet exposed on the CLI.
## What this enables
The reference case is portfolio cost review (the example bundle is a building-energy measure),

View file

@ -8,12 +8,21 @@ call and short-circuits with ``BudgetExceeded`` the moment the cap is crossed.
``strict_usage`` (default ``True``) makes a missing usage a HARD FAIL (``UsageUnavailable``):
a usage regression must never silently disable the cap. Test doubles that legitimately supply
a synthetic ``UsageDetails`` do not trip it.
S3.4 (F10) adds the cross-project half. ``PortfolioBudget`` + ``PortfolioMeter`` carry ONE token
ledger over a whole portfolio pass and, seeded from ``read_spend``, across passes while the
per-run ``Budget``/``TokenMeter`` pair stays exactly what it was. Two teeth follow from that:
the middleware now refuses a call BEFORE making it once the binding cap is exhausted (money not
spent, not money spent and regretted), and ``run_portfolio`` refuses to START a project it cannot
fund. Both are enforcement, never repair nothing is trimmed, retried, or scaled down.
"""
from __future__ import annotations
import json
from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from pathlib import Path
from agent_framework import ChatContext, ChatMiddleware
@ -37,6 +46,24 @@ class UsageUnavailable(RuntimeError):
must fail closed rather than silently stop counting (research 03 Rec 3)."""
class BudgetRefused(RuntimeError):
"""Raised at STARTUP when the portfolio's remaining tokens cannot fund one run (S3.4).
Distinct from ``BudgetExceeded`` on purpose: nothing was crossed and nothing was spent the
pass is refused before it begins, because a pass that can afford zero projects is a caller
mistake, not a resource event. Named for the repo's fail-closed gate family
(``PromotionRefused``, ``IngestStampError``): refusal, never repair.
"""
def __init__(self, remaining: int, required: int) -> None:
self.remaining = remaining
self.required = required
super().__init__(
f"portfolio budget refused: remaining={remaining} cannot fund one run "
f"(requires {required}); raise the global cap or lower min_run_reserve"
)
@dataclass(frozen=True)
class Budget:
"""Hard token + round/iteration caps, required at startup (A4 / D6).
@ -54,22 +81,142 @@ class Budget:
raise ValueError(f"max_rounds must be positive, got {self.max_rounds}")
@dataclass(frozen=True)
class PortfolioBudget:
"""The GLOBAL token cap over a whole portfolio pass, plus the per-run cap it funds (S3.4/F10).
``max_total_tokens`` bounds everything a pass may spend and, when a ``PortfolioMeter`` is
seeded from persisted spend, everything a SERIES of passes may spend. ``max_tokens_per_run``
is the cap each individual run is given. ``min_run_reserve`` (defaulting to the per-run cap)
is the remainder a run must be able to claim before it is allowed to START: without it a pass
would happily launch a project it can only half pay for, spending real tokens on a run that
cannot finish.
Fail-fast on configurations that cannot mean what they say: a per-run cap above the global one
would let a single run cross the pass's own ceiling, and a reserve above the per-run cap would
demand more than any run can ever spend, refusing every pass forever.
"""
max_total_tokens: int
max_tokens_per_run: int
min_run_reserve: int | None = None
def __post_init__(self) -> None:
if self.max_total_tokens <= 0:
raise ValueError(f"max_total_tokens must be positive, got {self.max_total_tokens}")
if self.max_tokens_per_run <= 0:
raise ValueError(f"max_tokens_per_run must be positive, got {self.max_tokens_per_run}")
if self.max_tokens_per_run > self.max_total_tokens:
raise ValueError(
f"max_tokens_per_run ({self.max_tokens_per_run}) exceeds max_total_tokens "
f"({self.max_total_tokens}): one run could cross the portfolio cap on its own"
)
if self.min_run_reserve is not None:
if self.min_run_reserve <= 0:
raise ValueError(f"min_run_reserve must be positive, got {self.min_run_reserve}")
if self.min_run_reserve > self.max_tokens_per_run:
raise ValueError(
f"min_run_reserve ({self.min_run_reserve}) exceeds max_tokens_per_run "
f"({self.max_tokens_per_run}): no run could ever meet it"
)
@property
def required_per_run(self) -> int:
"""The remainder one run must be able to claim to be allowed to start."""
return self.min_run_reserve if self.min_run_reserve is not None else self.max_tokens_per_run
class PortfolioMeter:
"""The ONE token ledger for a portfolio pass, shared by every run's ``TokenMeter`` (S3.4).
``spent`` seeds from an earlier pass (``read_spend``), which is what makes the cap hold ACROSS
passes and not merely within one.
``record`` and ``check`` are deliberately SPLIT rather than fused into one ``charge``. Tokens
the provider already billed must reach this ledger even when the charge simultaneously breaks
the RUN's own cap and raises there — a fused charge that returned early on the run cap would
silently forget that spend and hand the next project a budget that was never really there.
"""
def __init__(self, budget: PortfolioBudget, *, spent: int = 0) -> None:
if spent < 0:
raise ValueError(f"spent must be non-negative, got {spent}")
self.budget = budget
self.spent = spent
@property
def required_per_run(self) -> int:
return self.budget.required_per_run
def remaining(self) -> int:
"""Tokens left in the global cap (never negative — a crossed cap reads as zero left)."""
return max(0, self.budget.max_total_tokens - self.spent)
def can_fund_run(self, *, reserved: int = 0) -> bool:
"""Whether one more run can be started. ``reserved`` is what the caller has already
committed to runs it admitted but that have not spent yet (the wave case): every member of
a concurrent wave is funded off the same pre-wave remainder, so admitting them without
reserving would over-commit the cap by exactly the wave size."""
return self.remaining() - reserved >= self.required_per_run
def record(self, tokens: int) -> int:
"""Accumulate ``tokens`` into the global ledger. Never raises — see the class docstring."""
self.spent += tokens
return self.spent
def check(self) -> None:
"""Raise ``BudgetExceeded`` if the global cap has been crossed."""
if self.spent > self.budget.max_total_tokens:
raise BudgetExceeded("portfolio_tokens", self.budget.max_total_tokens, self.spent)
class TokenMeter:
"""Accumulates token and round usage against a ``Budget``; raises the moment a cap is
crossed."""
crossed. When ``portfolio`` is supplied (S3.4) every charge also lands in the shared portfolio
ledger, so the run is bounded by BOTH its own cap and the pass's global one."""
def __init__(self, budget: Budget) -> None:
def __init__(self, budget: Budget, *, portfolio: PortfolioMeter | None = None) -> None:
self.budget = budget
self.tokens = 0
self.rounds = 0
self.portfolio = portfolio
def charge(self, tokens: int) -> int:
"""Add ``tokens`` to the running total; raise ``BudgetExceeded`` if over cap."""
"""Add ``tokens`` to the running total; raise ``BudgetExceeded`` if over cap.
Both ledgers are credited BEFORE either cap is tested: the spend happened regardless of
which cap it broke, so recording must not depend on the outcome of a check."""
self.tokens += tokens
if self.portfolio is not None:
self.portfolio.record(tokens)
if self.tokens > self.budget.max_tokens:
raise BudgetExceeded("tokens", self.budget.max_tokens, self.tokens)
if self.portfolio is not None:
self.portfolio.check()
return self.tokens
def remaining(self) -> int:
"""Tokens left under whichever cap BINDS — the run's own, or the portfolio's."""
own = self.budget.max_tokens - self.tokens
if self.portfolio is None:
return own
return min(own, self.portfolio.remaining())
def exhausted(self) -> BudgetExceeded | None:
"""The structured error to refuse a call with, or ``None`` while budget remains.
Exhaustion is ``remaining() <= 0``, not ``< 0``: a chat call that costs zero tokens does
not exist, so at exactly-zero the next call can only overspend. Naming WHICH cap binds is
the point a run refused because a sibling drained the pass reads as ``portfolio_tokens``,
not as its own overrun."""
if self.remaining() > 0:
return None
if self.portfolio is not None and self.portfolio.remaining() <= 0:
return BudgetExceeded(
"portfolio_tokens", self.portfolio.budget.max_total_tokens, self.portfolio.spent
)
return BudgetExceeded("tokens", self.budget.max_tokens, self.tokens)
def tick_round(self) -> int:
"""Increment the round counter; raise ``BudgetExceeded`` if over cap."""
self.rounds += 1
@ -80,13 +227,21 @@ class TokenMeter:
class BudgetMiddleware(ChatMiddleware):
"""Chat middleware that charges a ``TokenMeter`` from each response's real
``UsageDetails`` and short-circuits when the cap is crossed."""
``UsageDetails`` and short-circuits when the cap is crossed.
Two teeth, not one. The PRE-call guard (S3.4) refuses to make a call the budget cannot pay
for that is money not spent. The post-charge check is what it always was: the real usage is
only knowable after the response, so a call that crosses the cap can only be caught behind it.
The guard does not replace the check; it stops the NEXT call, never the one in flight."""
def __init__(self, meter: TokenMeter, *, strict_usage: bool = True) -> None:
self._meter = meter
self._strict = strict_usage
async def process(self, context: ChatContext, call_next: Callable[[], Awaitable[None]]) -> None:
exhausted = self._meter.exhausted()
if exhausted is not None:
raise exhausted # BEFORE call_next: the call must never be made
await call_next()
usage = getattr(context.result, "usage_details", None)
total = usage.get("total_token_count") if usage is not None else None
@ -98,3 +253,39 @@ class BudgetMiddleware(ChatMiddleware):
)
return
self._meter.charge(int(total)) # raises BudgetExceeded if over cap
def write_spend(path: str | Path, spent: int, *, stamp: str) -> Path:
"""Persist a portfolio pass's accumulated ``spent`` tokens as deterministic JSON.
``stamp`` is a REQUIRED keyword with no wall-clock default (mirroring ``promote_verdict`` and
the outbox's ``run_id``): the file must be byte-identical for the same spend, so a diff means
the spend changed and nothing else. This is what carries the global cap ACROSS passes the
next pass seeds its ``PortfolioMeter`` from ``read_spend``."""
target = Path(path)
target.parent.mkdir(parents=True, exist_ok=True)
payload = {"spent_tokens": int(spent), "stamp": stamp}
target.write_text(json.dumps(payload, sort_keys=True, indent=2), encoding="utf-8")
return target
def read_spend(path: str | Path) -> int:
"""Read persisted spend; a MISSING file reads as ``0`` (there was no earlier pass).
Malformed content RAISES deliberately unlike the tolerant raw verdict inbox
(``load_verdicts_from_dir``). That folder is written out of band by other parties, so skipping
junk is correct there; this file is our OWN accounting state, and reading a corrupt one as zero
would silently hand back a budget that had already been spent."""
target = Path(path)
if not target.exists():
return 0
try:
payload = json.loads(target.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise ValueError(f"malformed spend file {str(target)!r}: {exc}") from exc
if not isinstance(payload, dict) or not isinstance(payload.get("spent_tokens"), int):
raise ValueError(f"spend file {str(target)!r} lacks an integer 'spent_tokens'")
spent = int(payload["spent_tokens"])
if spent < 0:
raise ValueError(f"spend file {str(target)!r} carries negative spent_tokens: {spent}")
return spent

View file

@ -35,7 +35,13 @@ from agent_framework import BaseChatClient, SessionContext
from pydantic import ValidationError
from portfolio_optimiser.backends import Profile, get_backend, resolve_model
from portfolio_optimiser.budget import Budget, BudgetMiddleware, TokenMeter
from portfolio_optimiser.budget import (
Budget,
BudgetMiddleware,
BudgetRefused,
PortfolioMeter,
TokenMeter,
)
from portfolio_optimiser.contracts import GoalConfig, GoalContract, load_contracts, load_goal_config
from portfolio_optimiser.ledger import SavingsLedger
from portfolio_optimiser.datasource import (
@ -141,6 +147,25 @@ class GoalReached:
observed_ore: int
@dataclass(frozen=True)
class BudgetStop:
"""A GLOBAL token-cap stop signal VALUE (S3.4/F10) — NOT an exception, and NOT a goal.
Structured like ``GoalReached`` and carried the same way (a ``stop_reason``-shaped value plus a
loop ``break``), but it is kept as its OWN field rather than widening ``stop_reason``: the two
stops mean opposite things. A goal-stop is success (the savings target was met); this is
resource exhaustion (the pass ran out of tokens). Folding them into one field would let a
caller read "we stopped" without being able to tell which happened.
``required_tokens`` is what one more run would have needed; ``remaining_tokens`` is what was
actually left. Both are recorded because their DIFFERENCE is the operator's next decision."""
limit_tokens: int
spent_tokens: int
remaining_tokens: int
required_tokens: int
@dataclass(frozen=True)
class PortfolioResult:
"""The outcome of a sequential fan-out over N projects (SC2).
@ -150,9 +175,11 @@ class PortfolioResult:
The remaining fields are a thin aggregate over ``runs``: ``validated_count`` /
``rejected_count`` partition the outcomes; ``sum_claimed_saving_nok`` totals the claimed
saving of the validated proposals only; ``sum_token_usage`` totals every run's
provenance token usage. ``stopped_early`` / ``stop_reason`` record a Step-8 goal-stop, and
``failures`` records the projects that RAISED (S3.3 collect-and-continue): all three default so
the frozen aggregate and every existing constructor call are unaffected.
provenance token usage. ``stopped_early`` / ``stop_reason`` record a Step-8 goal-stop,
``failures`` records the projects that RAISED (S3.3 collect-and-continue), and ``budget_stop``
records a S3.4 global-token-cap stop (which also sets ``stopped_early``, but is kept apart from
``stop_reason`` because exhaustion is not success): all four default so the frozen aggregate and
every existing constructor call are unaffected.
``runs`` and ``failures`` PARTITION the projects that were actually submitted a project
appears in exactly one of them, never both, and the counts do not overlap. ``validated_count`` /
@ -168,6 +195,7 @@ class PortfolioResult:
stopped_early: bool = False
stop_reason: GoalReached | None = None
failures: tuple[RunFailure, ...] = ()
budget_stop: BudgetStop | None = None
def _authored_texts(result: Any, name: str) -> list[str]:
@ -647,6 +675,30 @@ def _waves(ids: list[str], k: int) -> list[list[str]]:
return [ids[i : i + k] for i in range(0, len(ids), k)]
def _run_meter(
meter_factory: Callable[[], TokenMeter] | None,
portfolio_meter: PortfolioMeter | None,
max_rounds: int,
) -> TokenMeter | None:
"""The per-project meter ``run_portfolio`` hands to one run: the injected test seam, a meter
BOUND to the portfolio ledger, or ``None`` (letting ``run_project`` build its own, unchanged).
The bound meter mirrors ``run_project``'s own construction (``max(max_rounds * 4, 4)``) so the
only difference the portfolio cap introduces is the token ceiling and the shared ledger the
round budget is not quietly redefined along the way."""
if meter_factory is not None:
return meter_factory()
if portfolio_meter is None:
return None
return TokenMeter(
Budget(
max_tokens=portfolio_meter.budget.max_tokens_per_run,
max_rounds=max(max_rounds * 4, 4),
),
portfolio=portfolio_meter,
)
async def run_portfolio(
project_ids: Sequence[str] | None = None,
profile: Profile | str = Profile.LOCAL,
@ -661,6 +713,7 @@ async def run_portfolio(
top_k: int = 3,
concurrency: int = 1,
meter_factory: Callable[[], TokenMeter] | None = None,
portfolio_meter: PortfolioMeter | None = None,
semantic_retrieval: bool = False,
embedder: Embedder | None = None,
) -> PortfolioResult:
@ -727,12 +780,46 @@ async def run_portfolio(
property of the design, so it is pinned on a bundle-backed pair where the fold does fire
(``test_intra_wave_visibility_is_the_documented_semantic_difference``): same fixture, same
sentinel, only the wave boundary moves. Store content stays identical across ``k`` the
difference is confined to what each project READ, never to what the pass produced or persisted."""
difference is confined to what each project READ, never to what the pass produced or persisted.
``portfolio_meter`` (S3.4, F10) installs the GLOBAL token cap. Without it nothing changes: each
run is bounded only by its own ``max_tokens``, so N projects can cost N times that with no
ceiling over the pass. With it, one ``PortfolioMeter`` is shared by every run (each run's meter
is BOUND to it, which is why ``meter_factory`` whose meters are unbound is refused
alongside it: accepting both would run a pass that looks capped and is not), and the cap is
enforced in three places that are deliberately different:
- **At startup**, a remainder that cannot fund one run raises ``BudgetRefused`` a pass that
can afford zero projects is a caller mistake, not a result.
- **At wave assembly**, a project that cannot be funded is NEVER STARTED, and the pass stops
with ``budget_stop`` + ``stopped_early``, every completed run preserved. Never-started is the
property that matters: an unfunded project that is merely interrupted mid-run has already
cost model round-trips. Because every member of a wave is checked against the SAME pre-wave
remainder, admission RESERVES each member's requirement as it goes — otherwise a wave of k
would over-commit the cap by up to k runs. This makes membership identical at every
``concurrency``, matching the goal-stop's guarantee above.
- **Before each chat call** (``BudgetMiddleware``'s pre-call guard), a call the remainder cannot
pay for is refused rather than made. This is what bounds a run that was funded at admission
but whose siblings drained the pass while it was in flight; such a run surfaces as a
``RunFailure``, not as overspend.
Spend is carried ACROSS passes by seeding the meter from ``budget.read_spend`` and writing
``budget.write_spend`` afterwards. Those are the caller's calls, not this function's the pass
reads its ledger, it does not own the file (mirroring the Steg-7 role split)."""
if concurrency < 1:
raise ValueError(
f"concurrency must be >= 1, got {concurrency}: a non-positive wave size would run no "
"projects at all and read as an empty portfolio. Pass 1 for the sequential pass."
)
if portfolio_meter is not None and meter_factory is not None:
raise ValueError(
"portfolio_meter and meter_factory are mutually exclusive: a meter_factory meter is "
"not bound to the portfolio ledger, so the global cap would be silently unenforced"
)
# Startup refusal (fail-fast, before anything loads): a pass that cannot fund its first run
# must not begin. Raised, not returned — an empty PortfolioResult would read as "nothing to do".
if portfolio_meter is not None and not portfolio_meter.can_fund_run():
raise BudgetRefused(portfolio_meter.remaining(), portfolio_meter.required_per_run)
projects = {p.id: p for p in load_reference_projects()}
ids = list(project_ids) if project_ids is not None else list(projects)
store = store if store is not None else VerdictStore(verdicts=[])
@ -747,8 +834,13 @@ async def run_portfolio(
failures: list[RunFailure] = []
stopped_early = False
stop_reason: GoalReached | None = None
budget_stop: BudgetStop | None = None
for wave_ids in _waves(ids, concurrency):
members: list[str] = []
# Tokens committed to members already admitted to THIS wave but that have not spent yet.
# Every member reads the same pre-wave remainder, so without this a wave of k would admit
# k projects off one project's worth of budget.
reserved = 0
for pid in wave_ids:
if pid not in projects:
raise ValueError(f"unknown project_id: {pid!r}")
@ -777,6 +869,21 @@ async def run_portfolio(
if per_project_goal.mode == "hard":
continue # skip THIS pid; the rest of the pass proceeds
# S3.4 funding check, placed AFTER the goal checks: a reached goal is success and owns
# the stop when both apply, and a pid a hard per-project goal already skipped costs no
# budget, so it must not consume a reservation.
if portfolio_meter is not None and not portfolio_meter.can_fund_run(reserved=reserved):
stopped_early = True
budget_stop = BudgetStop(
limit_tokens=portfolio_meter.budget.max_total_tokens,
spent_tokens=portfolio_meter.spent,
remaining_tokens=portfolio_meter.remaining(),
required_tokens=portfolio_meter.required_per_run,
)
break
if portfolio_meter is not None:
reserved += portfolio_meter.required_per_run
members.append(pid)
# Every project in the wave reads the SAME wave-start state and writes only its own copy,
@ -805,7 +912,7 @@ async def run_portfolio(
top_k=top_k,
semantic_retrieval=semantic_retrieval,
embedder=embedder,
meter=meter_factory() if meter_factory is not None else None,
meter=_run_meter(meter_factory, portfolio_meter, max_rounds),
)
for pid, snapshot in snapshots
),
@ -836,12 +943,13 @@ async def run_portfolio(
break
base = _aggregate(tuple(runs), store)
if stopped_early or stop_reason is not None or failures:
if stopped_early or stop_reason is not None or failures or budget_stop is not None:
return replace(
base,
stopped_early=stopped_early,
stop_reason=stop_reason,
failures=tuple(failures),
budget_stop=budget_stop,
)
return base

View file

@ -15,6 +15,8 @@ from portfolio_optimiser.budget import (
Budget,
BudgetExceeded,
BudgetMiddleware,
PortfolioBudget,
PortfolioMeter,
TokenMeter,
UsageUnavailable,
)
@ -85,6 +87,90 @@ async def test_budget_middleware_fires_on_real_agent_chat(make_client_factory) -
assert meter.tokens == 8 # charged from the synthetic UsageDetails via the middleware
async def test_pre_call_guard_does_not_await_call_next_when_exhausted() -> None:
"""S3.4 unit arm of the pre-call guard: the middleware refuses BEFORE ``call_next``. Counting
the awaits is the whole measurement the post-charge arm raises either way."""
calls = {"n": 0}
async def _counted() -> None:
calls["n"] += 1
meter = TokenMeter(Budget(max_tokens=10, max_rounds=10))
meter.charge(10) # exactly at the cap -> nothing left to fund a call with
mw = BudgetMiddleware(meter)
with pytest.raises(BudgetExceeded):
await mw.process(_Ctx(_resp(5)), _counted) # type: ignore[arg-type]
assert calls["n"] == 0
async def test_below_the_cap_still_calls_through() -> None:
"""Control for the guard: with budget left, the call goes through exactly as before — the
guard must gate exhaustion, not traffic."""
calls = {"n": 0}
async def _counted() -> None:
calls["n"] += 1
meter = TokenMeter(Budget(max_tokens=10, max_rounds=10))
mw = BudgetMiddleware(meter)
await mw.process(_Ctx(_resp(4)), _counted) # type: ignore[arg-type]
assert calls["n"] == 1 and meter.tokens == 4 and meter.remaining() == 6
def test_portfolio_budget_rejects_unusable_configs() -> None:
"""Fail-fast at construction (mirroring ``Budget``): non-positive caps, a per-run cap larger
than the global one (one run could then cross the pass's own ceiling), and a reserve larger
than a run can ever spend (which would refuse every pass forever)."""
with pytest.raises(ValueError):
PortfolioBudget(max_total_tokens=0, max_tokens_per_run=10)
with pytest.raises(ValueError):
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=0)
with pytest.raises(ValueError):
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=200)
with pytest.raises(ValueError):
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=50, min_run_reserve=60)
with pytest.raises(ValueError):
PortfolioMeter(PortfolioBudget(max_total_tokens=100, max_tokens_per_run=50), spent=-1)
def test_portfolio_meter_accumulates_and_crosses() -> None:
budget = PortfolioBudget(max_total_tokens=100, max_tokens_per_run=50)
meter = PortfolioMeter(budget, spent=40)
assert meter.remaining() == 60 and meter.required_per_run == 50
meter.record(60)
meter.check() # exactly at the cap is still within it (mirrors TokenMeter's `>` boundary)
meter.record(1)
with pytest.raises(BudgetExceeded) as exc:
meter.check()
assert exc.value.kind == "portfolio_tokens"
assert exc.value.limit == 100 and exc.value.observed == 101
async def test_run_meter_bound_to_portfolio_charges_both_ledgers() -> None:
"""A bound run meter charges the global ledger on every call, and ``remaining`` reads whichever
cap binds that is what lets one project's spend refuse another's next call."""
portfolio = PortfolioMeter(PortfolioBudget(max_total_tokens=100, max_tokens_per_run=80))
meter = TokenMeter(Budget(max_tokens=80, max_rounds=10), portfolio=portfolio)
mw = BudgetMiddleware(meter)
await mw.process(_Ctx(_resp(30)), _noop) # type: ignore[arg-type]
assert meter.tokens == 30 and portfolio.spent == 30
assert meter.remaining() == 50 # run has 50 left, portfolio 70 -> the RUN binds
await mw.process(_Ctx(_resp(45)), _noop) # type: ignore[arg-type]
assert meter.remaining() == 5 # run 5, portfolio 25 -> still the run
async def test_spend_is_ledgered_even_when_the_per_run_cap_raises() -> None:
"""Order matters: tokens the provider already billed must reach the global ledger even though
the RUN's own cap raises on the same charge. Checking the run cap first and returning early
would lose that spend and hand the next project a budget that was never really there."""
portfolio = PortfolioMeter(PortfolioBudget(max_total_tokens=1000, max_tokens_per_run=50))
meter = TokenMeter(Budget(max_tokens=50, max_rounds=10), portfolio=portfolio)
with pytest.raises(BudgetExceeded) as exc:
meter.charge(60)
assert exc.value.kind == "tokens" # the RUN's cap is the one that broke
assert portfolio.spent == 60 # ...and the global ledger still saw the spend
def test_no_word_count_token_proxy_in_src() -> None:
# The meter is fed from real UsageDetails, never a len(text.split()) word-count proxy
# (research 03 Rec 3 — the Fase 1 _word_tokens anti-pattern is retired). NOTE: a bare

View file

@ -0,0 +1,302 @@
"""S3.4 load-bearing — the GLOBAL token cap across a portfolio pass (F10).
Two properties the session plan names as RED conditions, plus the wave-size and control arms
that keep them honest:
1. **The global cap stops the pass mid-way, with completed runs preserved.** Detach the
wave-assembly funding check and the third project is STARTED anyway it then dies on the
meter mid-run, so the pass reports a failure instead of a structured stop. The property is
never-started, not merely never-overspent: a project that cannot be funded must cost zero
model round-trips.
2. **The pre-call guard refuses the call BEFORE it is made.** The client double counts its own
invocations, so "the call never happened" is measured, not inferred. Today's post-charge
middleware would let the call through and only then raise which is exactly the money the
guard exists to not spend.
Token arithmetic is MEASURED, not guessed: every project in the reference portfolio spends
4 chat calls x ``tokens`` per reply. At ``tokens=8`` that is a flat 32 tokens per run (probed
against ``provenance.token_usage`` before these tests were written), which is what makes the
80/64/32 budget below land the stop between run 2 and run 3.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from agent_framework import Agent
from portfolio_optimiser.budget import (
Budget,
BudgetExceeded,
BudgetMiddleware,
BudgetRefused,
PortfolioBudget,
PortfolioMeter,
TokenMeter,
read_spend,
write_spend,
)
from portfolio_optimiser.run import run_portfolio
_PORTFOLIO_IDS = ["FV42-GSV-E1", "RV13-RAS-TP", "BRU-LAKS-REHAB"]
# Per-project replies (the tested constants of tests/test_portfolio.py, unchanged): all three
# validate, and their claimed savings are DISTINCT — which is how a run is keyed back to its
# project here, since ``RunResult`` carries no project id. 200000 + 130000 = the first two.
REPLIES = {
"FV42-GSV-E1": (
'{"measure":"Reduce scope","affected_items":['
'{"code":"05.2","quantity":4300,"unit_cost":215},'
'{"code":"03.1","quantity":1800,"unit_cost":310}],"claimed_saving_nok":200000}'
),
"RV13-RAS-TP": (
'{"measure":"Material substitution","affected_items":['
'{"code":"88.2","quantity":180,"unit_cost":4200}],"claimed_saving_nok":130000}'
),
"BRU-LAKS-REHAB": (
'{"measure":"Reduce scope","affected_items":['
'{"code":"05.2","quantity":4300,"unit_cost":215},'
'{"code":"07.4","quantity":2400,"unit_cost":690}],"claimed_saving_nok":210000}'
),
}
_FIRST_TWO_SAVING = 330000
# Measured: 32 tokens per run at tokens=8. 80 total funds two runs (32 + 32 = 64) and leaves 16,
# which is below the 32-token reserve one more run requires -> the third is never started.
_TOTAL = 80
_PER_RUN = 64
_RESERVE = 32
_PER_RUN_SPEND = 32
def _budget() -> PortfolioBudget:
return PortfolioBudget(
max_total_tokens=_TOTAL, max_tokens_per_run=_PER_RUN, min_run_reserve=_RESERVE
)
async def test_global_cap_stops_the_pass_mid_way_with_completed_runs_preserved(
make_portfolio_client_factory, fresh_store
) -> None:
"""RED 1: the pass stops the moment the REMAINING global budget cannot fund another run, and
every completed run survives the stop.
Detach the wave-assembly funding check and this goes red three ways at once: project 3 is
started (so it lands in ``failures`` when its charge crosses the cap), ``stopped_early`` stays
False, and ``budget_stop`` stays None. The strongest of the three is ``failures == ()``: an
unfunded project that is merely *interrupted* has already cost model round-trips, which is the
precise thing a global cap is for."""
meter = PortfolioMeter(_budget())
result = await run_portfolio(
_PORTFOLIO_IDS,
"local",
store=fresh_store,
client_factory=make_portfolio_client_factory(REPLIES, tokens=8),
portfolio_meter=meter,
)
assert result.stopped_early is True
assert result.budget_stop is not None
assert result.budget_stop.limit_tokens == _TOTAL
assert result.budget_stop.spent_tokens == 2 * _PER_RUN_SPEND
assert result.budget_stop.remaining_tokens == _TOTAL - 2 * _PER_RUN_SPEND
assert result.budget_stop.required_tokens == _RESERVE
# Completed runs preserved, and the unfunded project NEVER STARTED: absent from BOTH sides of
# the runs/failures partition, not merely absent from runs. A project that had been started and
# then killed by the meter would land in ``failures`` — which is what makes ``== ()`` the
# sharpest assertion here. The claimed-saving sum names WHICH two ran (the replies are distinct).
assert len(result.runs) == 2
assert result.failures == ()
assert result.validated_count == 2
assert result.sum_claimed_saving_nok == _FIRST_TWO_SAVING
# The cap itself held: the pass spent what two runs cost and no more.
assert meter.spent == 2 * _PER_RUN_SPEND
assert meter.spent <= _TOTAL
assert meter.spent == result.sum_token_usage
async def test_generous_budget_runs_the_whole_pass(
make_portfolio_client_factory, fresh_store
) -> None:
"""Control: the stop above is CAUSED by the cap, not by wiring a meter in at all. With a global
budget that funds every project the pass completes untouched a gate that can only fire proves
nothing about what fires it."""
meter = PortfolioMeter(PortfolioBudget(max_total_tokens=10_000, max_tokens_per_run=_PER_RUN))
result = await run_portfolio(
_PORTFOLIO_IDS,
"local",
store=fresh_store,
client_factory=make_portfolio_client_factory({}, tokens=8),
portfolio_meter=meter,
)
assert result.stopped_early is False
assert result.budget_stop is None
assert len(result.runs) == 3
assert meter.spent == 3 * _PER_RUN_SPEND
async def test_global_cap_holds_at_wave_size_three(
make_portfolio_client_factory, fresh_store
) -> None:
"""The cap is a property of the PASS, not of the sequential schedule: at ``concurrency=3`` the
whole portfolio is assembled into ONE wave, and every member's funding check reads the same
pre-wave ``remaining``. Admitting all three off that shared reading would overspend by design.
The wave therefore RESERVES each admitted member's requirement as it admits it, so the same 80
tokens fund the same two projects at k=3 as at k=1."""
meter = PortfolioMeter(_budget())
result = await run_portfolio(
_PORTFOLIO_IDS,
"local",
store=fresh_store,
client_factory=make_portfolio_client_factory({}, tokens=8),
concurrency=3,
portfolio_meter=meter,
)
assert len(result.runs) == 2
assert result.failures == ()
assert result.stopped_early is True
assert meter.spent <= _TOTAL
async def test_pre_call_guard_refuses_the_call_before_it_is_made(make_client_factory) -> None:
"""RED 2: with the budget exhausted, the chat call is REFUSED — the client is never invoked.
``call_count`` on the scripted client is the measurement: post-charge middleware alone would
make the call, read its usage, and only then raise, so ``pytest.raises`` would pass either way.
Only the counter separates 'refused' from 'made and then regretted'."""
client = make_client_factory("ok", tokens=8)("proposer")
meter = TokenMeter(Budget(max_tokens=10, max_rounds=10))
meter.charge(10) # exactly at the cap: within budget, but nothing left to fund a call with
assert meter.remaining() == 0
agent = Agent(
client, "propose a measure", name="proposer", middleware=[BudgetMiddleware(meter)]
)
with pytest.raises(BudgetExceeded) as exc:
await agent.run("hi")
assert exc.value.kind == "tokens"
assert client.call_count == 0, "the call must NEVER be made once the budget is exhausted"
async def test_pre_call_guard_fires_on_the_GLOBAL_remainder_too(make_client_factory) -> None:
"""The guard reads whichever cap binds. Here the run's OWN budget is untouched and only the
portfolio remainder is exhausted a run that starts funded can still be refused mid-flight
because a sibling spent the rest of the global budget."""
client = make_client_factory("ok", tokens=8)("proposer")
portfolio = PortfolioMeter(
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=100), spent=100
)
meter = TokenMeter(Budget(max_tokens=100, max_rounds=10), portfolio=portfolio)
assert meter.tokens == 0 # the RUN has spent nothing
assert meter.remaining() == 0 # ...but the portfolio has nothing left
agent = Agent(
client, "propose a measure", name="proposer", middleware=[BudgetMiddleware(meter)]
)
with pytest.raises(BudgetExceeded) as exc:
await agent.run("hi")
assert exc.value.kind == "portfolio_tokens"
assert client.call_count == 0
async def test_startup_refusal_when_the_remainder_cannot_fund_one_run(
make_portfolio_client_factory, fresh_store
) -> None:
"""Fail-fast at startup (never a silently truncated pass): a meter whose remainder is already
below one run's requirement refuses BEFORE anything runs. This is the across-passes arm — the
remainder here is what an earlier pass left behind."""
meter = PortfolioMeter(_budget(), spent=_TOTAL - (_RESERVE - 1))
with pytest.raises(BudgetRefused) as exc:
await run_portfolio(
_PORTFOLIO_IDS,
"local",
store=fresh_store,
client_factory=make_portfolio_client_factory({}, tokens=8),
portfolio_meter=meter,
)
assert exc.value.remaining == _RESERVE - 1
assert exc.value.required == _RESERVE
assert meter.spent == _TOTAL - (_RESERVE - 1) # nothing ran, nothing charged
async def test_spend_persists_across_passes(
make_portfolio_client_factory, fresh_store, tmp_path: Path
) -> None:
"""The cap holds ACROSS passes, which is what the persisted spend buys: pass A writes its
spend, pass B is seeded from that file and stops one run earlier than it otherwise would."""
spend_file = tmp_path / "spend.json"
assert read_spend(spend_file) == 0 # no prior pass -> no prior spend
budget = PortfolioBudget(
max_total_tokens=3 * _PER_RUN_SPEND, max_tokens_per_run=_PER_RUN, min_run_reserve=_RESERVE
)
meter_a = PortfolioMeter(budget)
result_a = await run_portfolio(
_PORTFOLIO_IDS[:1],
"local",
store=fresh_store,
client_factory=make_portfolio_client_factory({}, tokens=8),
portfolio_meter=meter_a,
)
assert len(result_a.runs) == 1
write_spend(spend_file, meter_a.spent, stamp="pass-a")
meter_b = PortfolioMeter(budget, spent=read_spend(spend_file))
assert meter_b.spent == _PER_RUN_SPEND # pass B starts where pass A stopped
result_b = await run_portfolio(
_PORTFOLIO_IDS,
"local",
store=fresh_store,
client_factory=make_portfolio_client_factory({}, tokens=8),
portfolio_meter=meter_b,
)
# 96 total funds three runs; one is already spent, so pass B affords two and stops before the
# third — the earlier pass's spend is what moved the stop.
assert len(result_b.runs) == 2
assert result_b.stopped_early is True
assert meter_b.spent == 3 * _PER_RUN_SPEND
def test_spend_file_is_deterministic_and_stamped(tmp_path: Path) -> None:
"""Byte-determinism with an EXPLICIT stamp (mirroring ``promote_verdict``): no wall-clock, so
two writes of the same spend are byte-identical and a diff means the spend changed."""
a, b = tmp_path / "a.json", tmp_path / "b.json"
write_spend(a, 1234, stamp="run-7")
write_spend(b, 1234, stamp="run-7")
assert a.read_bytes() == b.read_bytes()
assert json.loads(a.read_text(encoding="utf-8")) == {"spent_tokens": 1234, "stamp": "run-7"}
assert read_spend(a) == 1234
def test_spend_file_malformed_fails_fast(tmp_path: Path) -> None:
"""The spend file is OUR OWN accounting state, not the tolerant raw inbox layer: a corrupt file
raises rather than silently reading as zero spend, which would hand back the whole budget."""
bad = tmp_path / "bad.json"
bad.write_text("{not json", encoding="utf-8")
with pytest.raises(ValueError):
read_spend(bad)
negative = tmp_path / "neg.json"
negative.write_text('{"spent_tokens": -5, "stamp": "x"}', encoding="utf-8")
with pytest.raises(ValueError):
read_spend(negative)
async def test_meter_factory_and_portfolio_meter_are_mutually_exclusive(
make_portfolio_client_factory, fresh_store
) -> None:
"""Fail-fast on the one wiring that would silently disable the global cap: a ``meter_factory``
meter is not bound to the portfolio ledger, so accepting both would run a pass that LOOKS
capped and is not."""
with pytest.raises(ValueError):
await run_portfolio(
_PORTFOLIO_IDS,
"local",
store=fresh_store,
client_factory=make_portfolio_client_factory({}, tokens=8),
meter_factory=lambda: TokenMeter(Budget(max_tokens=1000, max_rounds=10)),
portfolio_meter=PortfolioMeter(_budget()),
)