feat(budget): enforce a global portfolio token cap before the call, not after it (S3.4/F10)
PortfolioBudget + PortfolioMeter carry ONE token ledger over a whole portfolio pass -- and, seeded from a persisted spend file, across passes -- while the per-run Budget/TokenMeter pair is untouched. Three enforcement points, each doing a different job: - startup: a remainder that cannot fund one run raises BudgetRefused before anything loads (a pass that can afford zero projects is a caller mistake, not a result); - wave assembly: an unfundable project is NEVER STARTED and the pass stops structurally (budget_stop + stopped_early, completed runs preserved). Because every member of a wave is funded against the SAME pre-wave remainder, admission RESERVES each member's requirement -- otherwise a wave of k over-commits the cap by up to k runs; - pre-call: BudgetMiddleware refuses a call the remainder cannot pay for instead of making it. The post-charge check stays: real usage is only knowable after the response, so the guard stops the NEXT call, never the one in flight. budget_stop is its own field rather than a widened stop_reason -- a goal-stop is success, this is resource exhaustion, and fusing them would make "we stopped" unreadable. PortfolioMeter splits record/check so tokens the provider already billed reach the ledger even when the same charge breaks the run's own cap. read_spend raises on corrupt content (our own accounting state, unlike the tolerant RAW inbox layer); write_spend takes a REQUIRED stamp with no wall-clock default, mirroring promote_verdict. Load-bearing MEASURED, not asserted -- 6 mutations, all red: detach the wave check; detach the pre-call guard; detach the wave reservation; check the run cap before crediting the global ledger; detach the startup refusal; make read_spend tolerant. Files restored from shasum-verified copies after each. Two findings worth keeping: the pre-call guard MASKS a detached wave check if the test asserts on overspend (spend stays under the cap either way), so the load-bearing assertion had to become failures == () plus never-started; and the token arithmetic is probed (32 tokens/run at tokens=8), not guessed. 537 -> 553 tests, ruff + mypy green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EaxFnaDAbMQkmTeX4u7sd
This commit is contained in:
parent
0d50ab89d3
commit
a831aa1e3b
6 changed files with 733 additions and 11 deletions
23
CLAUDE.md
23
CLAUDE.md
|
|
@ -105,6 +105,29 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
(`tests/test_step8_promotion_loadbearing.py`): gaten avviser ikke-godkjent dom (RØD uten gate); godkjent
|
||||
dom er navigerbar (RØD når `link_in_index` detaches); promotert signal holdes ute av `bundle_context`
|
||||
(RØD når en beskrivende index-label lekker det inn). Index-RMW er ikke-atomisk (enprosess-MVP).
|
||||
- **Globalt token-tak håndheves FØR kall, aldri bare etterpå (S3.4, F10):** `PortfolioBudget` +
|
||||
`PortfolioMeter` er ÉN ledger over hele porteføljepasset (og — seedet av `read_spend` — på tvers
|
||||
av pass), mens per-run `Budget`/`TokenMeter` er uendret. Taket har tre tenner, med hver sin jobb:
|
||||
(1) **oppstartsnekt** — en rest som ikke kan finansiere én kjøring raiser `BudgetRefused` FØR noe
|
||||
lastes (et pass som har råd til null prosjekter er en caller-feil, ikke et resultat);
|
||||
(2) **wave-assembly** — et prosjekt som ikke kan finansieres blir ALDRI STARTET, og passet stopper
|
||||
strukturert (`budget_stop` + `stopped_early`, fullførte runs bevart). Aldri-startet er poenget:
|
||||
et ufinansiert prosjekt som bare avbrytes har allerede kostet kall. Fordi hele bølgen sjekkes mot
|
||||
SAMME før-bølge-rest, **reserverer** admission hver members krav — ellers overforplikter en bølge
|
||||
av k taket med inntil k kjøringer; (3) **pre-call-guard** i `BudgetMiddleware` — et kall resten
|
||||
ikke kan betale for NEKTES i stedet for å gjøres (post-charge-sjekken består: ekte usage kjennes
|
||||
først etterpå, så guarden stopper NESTE kall, aldri det som er i lufta). `budget_stop` er et EGET
|
||||
felt, aldri `stop_reason`: et mål-stopp er suksess, dette er ressurs-utmattelse — å slå dem sammen
|
||||
ville gjort «vi stoppet» uleselig. `record`/`check` er SPLITTET i `PortfolioMeter` fordi tokens
|
||||
leverandøren allerede har fakturert må nå ledgeren selv når samme charge bryter run-taket.
|
||||
Spend-fila er vår EGEN regnskapstilstand: `read_spend` raiser på korrupt innhold (kontrast det
|
||||
tolerante RAW-inbox-laget — å lese korrupt som null ville gitt tilbake et allerede brukt budsjett),
|
||||
og `write_spend` tar et PÅKREVD `stamp` uten wall-clock-default (byte-determinisme, speiler
|
||||
`promote_verdict`). `portfolio_meter` og `meter_factory` er gjensidig utelukkende — en
|
||||
factory-meter er ubundet, så begge sammen ville gitt et pass som SER capped ut uten å være det.
|
||||
Load-bearing MÅLT (`tests/test_portfolio_budget_loadbearing.py` + `tests/test_budget.py`), seks
|
||||
mutasjoner alle røde: detach wave-sjekken · detach pre-call-guarden · detach bølge-reservasjonen ·
|
||||
sjekk run-taket før global kreditering · detach oppstartsnekten · gjør `read_spend` tolerant.
|
||||
- **Kostnadsdisiplin:** utvikle primært på lokal profil (gratis); Foundry/Azure (privat tenant finnes) kun til målrettet, minimal verifisering; billigste modeller + små syntetiske data + harde token-tak. Ingen tunge test-kjøringer.
|
||||
- **Offline simulering = primært metode-bevis (kostnadsdrevet, erstatter §11.8):** operatøren kjører
|
||||
IKKE MAF mot ekte modell (verken Azure/Foundry eller Ollama — API for begge repoene er for kostbart
|
||||
|
|
|
|||
12
README.md
12
README.md
|
|
@ -149,6 +149,18 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
|||
not CLI-enforced. Stop criteria and budget caps are required at startup. Try the offline
|
||||
end-to-end proof (no model, no network): `uv run python -m portfolio_optimiser.simulation`.
|
||||
|
||||
- **A global token cap across the whole portfolio, enforced before the call.** Per-run caps alone
|
||||
let N projects cost N times that with no ceiling over the pass. Pass a
|
||||
`PortfolioMeter(PortfolioBudget(max_total_tokens=…, max_tokens_per_run=…))` to `run_portfolio`
|
||||
and one ledger bounds the entire pass — and, seeded from `budget.read_spend`, a *series* of
|
||||
passes. It bites in three places: a remainder that cannot fund one run refuses the pass at
|
||||
startup (`BudgetRefused`); a project that cannot be funded is **never started**, stopping the
|
||||
pass structurally (`budget_stop`, completed runs preserved); and a chat call the remainder
|
||||
cannot pay for is **refused rather than made** (the post-charge check remains, since real usage
|
||||
is only knowable after the response). Spend persists via `budget.write_spend`, which takes an
|
||||
explicit stamp and no wall-clock default, so the file is byte-deterministic. Python API only —
|
||||
not yet exposed on the CLI.
|
||||
|
||||
## What this enables
|
||||
|
||||
The reference case is portfolio cost review (the example bundle is a building-energy measure),
|
||||
|
|
|
|||
|
|
@ -8,12 +8,21 @@ call and short-circuits with ``BudgetExceeded`` the moment the cap is crossed.
|
|||
``strict_usage`` (default ``True``) makes a missing usage a HARD FAIL (``UsageUnavailable``):
|
||||
a usage regression must never silently disable the cap. Test doubles that legitimately supply
|
||||
a synthetic ``UsageDetails`` do not trip it.
|
||||
|
||||
S3.4 (F10) adds the cross-project half. ``PortfolioBudget`` + ``PortfolioMeter`` carry ONE token
|
||||
ledger over a whole portfolio pass — and, seeded from ``read_spend``, across passes — while the
|
||||
per-run ``Budget``/``TokenMeter`` pair stays exactly what it was. Two teeth follow from that:
|
||||
the middleware now refuses a call BEFORE making it once the binding cap is exhausted (money not
|
||||
spent, not money spent and regretted), and ``run_portfolio`` refuses to START a project it cannot
|
||||
fund. Both are enforcement, never repair — nothing is trimmed, retried, or scaled down.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from agent_framework import ChatContext, ChatMiddleware
|
||||
|
||||
|
|
@ -37,6 +46,24 @@ class UsageUnavailable(RuntimeError):
|
|||
must fail closed rather than silently stop counting (research 03 Rec 3)."""
|
||||
|
||||
|
||||
class BudgetRefused(RuntimeError):
|
||||
"""Raised at STARTUP when the portfolio's remaining tokens cannot fund one run (S3.4).
|
||||
|
||||
Distinct from ``BudgetExceeded`` on purpose: nothing was crossed and nothing was spent — the
|
||||
pass is refused before it begins, because a pass that can afford zero projects is a caller
|
||||
mistake, not a resource event. Named for the repo's fail-closed gate family
|
||||
(``PromotionRefused``, ``IngestStampError``): refusal, never repair.
|
||||
"""
|
||||
|
||||
def __init__(self, remaining: int, required: int) -> None:
|
||||
self.remaining = remaining
|
||||
self.required = required
|
||||
super().__init__(
|
||||
f"portfolio budget refused: remaining={remaining} cannot fund one run "
|
||||
f"(requires {required}); raise the global cap or lower min_run_reserve"
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Budget:
|
||||
"""Hard token + round/iteration caps, required at startup (A4 / D6).
|
||||
|
|
@ -54,22 +81,142 @@ class Budget:
|
|||
raise ValueError(f"max_rounds must be positive, got {self.max_rounds}")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PortfolioBudget:
|
||||
"""The GLOBAL token cap over a whole portfolio pass, plus the per-run cap it funds (S3.4/F10).
|
||||
|
||||
``max_total_tokens`` bounds everything a pass may spend — and, when a ``PortfolioMeter`` is
|
||||
seeded from persisted spend, everything a SERIES of passes may spend. ``max_tokens_per_run``
|
||||
is the cap each individual run is given. ``min_run_reserve`` (defaulting to the per-run cap)
|
||||
is the remainder a run must be able to claim before it is allowed to START: without it a pass
|
||||
would happily launch a project it can only half pay for, spending real tokens on a run that
|
||||
cannot finish.
|
||||
|
||||
Fail-fast on configurations that cannot mean what they say: a per-run cap above the global one
|
||||
would let a single run cross the pass's own ceiling, and a reserve above the per-run cap would
|
||||
demand more than any run can ever spend, refusing every pass forever.
|
||||
"""
|
||||
|
||||
max_total_tokens: int
|
||||
max_tokens_per_run: int
|
||||
min_run_reserve: int | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.max_total_tokens <= 0:
|
||||
raise ValueError(f"max_total_tokens must be positive, got {self.max_total_tokens}")
|
||||
if self.max_tokens_per_run <= 0:
|
||||
raise ValueError(f"max_tokens_per_run must be positive, got {self.max_tokens_per_run}")
|
||||
if self.max_tokens_per_run > self.max_total_tokens:
|
||||
raise ValueError(
|
||||
f"max_tokens_per_run ({self.max_tokens_per_run}) exceeds max_total_tokens "
|
||||
f"({self.max_total_tokens}): one run could cross the portfolio cap on its own"
|
||||
)
|
||||
if self.min_run_reserve is not None:
|
||||
if self.min_run_reserve <= 0:
|
||||
raise ValueError(f"min_run_reserve must be positive, got {self.min_run_reserve}")
|
||||
if self.min_run_reserve > self.max_tokens_per_run:
|
||||
raise ValueError(
|
||||
f"min_run_reserve ({self.min_run_reserve}) exceeds max_tokens_per_run "
|
||||
f"({self.max_tokens_per_run}): no run could ever meet it"
|
||||
)
|
||||
|
||||
@property
|
||||
def required_per_run(self) -> int:
|
||||
"""The remainder one run must be able to claim to be allowed to start."""
|
||||
return self.min_run_reserve if self.min_run_reserve is not None else self.max_tokens_per_run
|
||||
|
||||
|
||||
class PortfolioMeter:
|
||||
"""The ONE token ledger for a portfolio pass, shared by every run's ``TokenMeter`` (S3.4).
|
||||
|
||||
``spent`` seeds from an earlier pass (``read_spend``), which is what makes the cap hold ACROSS
|
||||
passes and not merely within one.
|
||||
|
||||
``record`` and ``check`` are deliberately SPLIT rather than fused into one ``charge``. Tokens
|
||||
the provider already billed must reach this ledger even when the charge simultaneously breaks
|
||||
the RUN's own cap and raises there — a fused charge that returned early on the run cap would
|
||||
silently forget that spend and hand the next project a budget that was never really there.
|
||||
"""
|
||||
|
||||
def __init__(self, budget: PortfolioBudget, *, spent: int = 0) -> None:
|
||||
if spent < 0:
|
||||
raise ValueError(f"spent must be non-negative, got {spent}")
|
||||
self.budget = budget
|
||||
self.spent = spent
|
||||
|
||||
@property
|
||||
def required_per_run(self) -> int:
|
||||
return self.budget.required_per_run
|
||||
|
||||
def remaining(self) -> int:
|
||||
"""Tokens left in the global cap (never negative — a crossed cap reads as zero left)."""
|
||||
return max(0, self.budget.max_total_tokens - self.spent)
|
||||
|
||||
def can_fund_run(self, *, reserved: int = 0) -> bool:
|
||||
"""Whether one more run can be started. ``reserved`` is what the caller has already
|
||||
committed to runs it admitted but that have not spent yet (the wave case): every member of
|
||||
a concurrent wave is funded off the same pre-wave remainder, so admitting them without
|
||||
reserving would over-commit the cap by exactly the wave size."""
|
||||
return self.remaining() - reserved >= self.required_per_run
|
||||
|
||||
def record(self, tokens: int) -> int:
|
||||
"""Accumulate ``tokens`` into the global ledger. Never raises — see the class docstring."""
|
||||
self.spent += tokens
|
||||
return self.spent
|
||||
|
||||
def check(self) -> None:
|
||||
"""Raise ``BudgetExceeded`` if the global cap has been crossed."""
|
||||
if self.spent > self.budget.max_total_tokens:
|
||||
raise BudgetExceeded("portfolio_tokens", self.budget.max_total_tokens, self.spent)
|
||||
|
||||
|
||||
class TokenMeter:
|
||||
"""Accumulates token and round usage against a ``Budget``; raises the moment a cap is
|
||||
crossed."""
|
||||
crossed. When ``portfolio`` is supplied (S3.4) every charge also lands in the shared portfolio
|
||||
ledger, so the run is bounded by BOTH its own cap and the pass's global one."""
|
||||
|
||||
def __init__(self, budget: Budget) -> None:
|
||||
def __init__(self, budget: Budget, *, portfolio: PortfolioMeter | None = None) -> None:
|
||||
self.budget = budget
|
||||
self.tokens = 0
|
||||
self.rounds = 0
|
||||
self.portfolio = portfolio
|
||||
|
||||
def charge(self, tokens: int) -> int:
|
||||
"""Add ``tokens`` to the running total; raise ``BudgetExceeded`` if over cap."""
|
||||
"""Add ``tokens`` to the running total; raise ``BudgetExceeded`` if over cap.
|
||||
|
||||
Both ledgers are credited BEFORE either cap is tested: the spend happened regardless of
|
||||
which cap it broke, so recording must not depend on the outcome of a check."""
|
||||
self.tokens += tokens
|
||||
if self.portfolio is not None:
|
||||
self.portfolio.record(tokens)
|
||||
if self.tokens > self.budget.max_tokens:
|
||||
raise BudgetExceeded("tokens", self.budget.max_tokens, self.tokens)
|
||||
if self.portfolio is not None:
|
||||
self.portfolio.check()
|
||||
return self.tokens
|
||||
|
||||
def remaining(self) -> int:
|
||||
"""Tokens left under whichever cap BINDS — the run's own, or the portfolio's."""
|
||||
own = self.budget.max_tokens - self.tokens
|
||||
if self.portfolio is None:
|
||||
return own
|
||||
return min(own, self.portfolio.remaining())
|
||||
|
||||
def exhausted(self) -> BudgetExceeded | None:
|
||||
"""The structured error to refuse a call with, or ``None`` while budget remains.
|
||||
|
||||
Exhaustion is ``remaining() <= 0``, not ``< 0``: a chat call that costs zero tokens does
|
||||
not exist, so at exactly-zero the next call can only overspend. Naming WHICH cap binds is
|
||||
the point — a run refused because a sibling drained the pass reads as ``portfolio_tokens``,
|
||||
not as its own overrun."""
|
||||
if self.remaining() > 0:
|
||||
return None
|
||||
if self.portfolio is not None and self.portfolio.remaining() <= 0:
|
||||
return BudgetExceeded(
|
||||
"portfolio_tokens", self.portfolio.budget.max_total_tokens, self.portfolio.spent
|
||||
)
|
||||
return BudgetExceeded("tokens", self.budget.max_tokens, self.tokens)
|
||||
|
||||
def tick_round(self) -> int:
|
||||
"""Increment the round counter; raise ``BudgetExceeded`` if over cap."""
|
||||
self.rounds += 1
|
||||
|
|
@ -80,13 +227,21 @@ class TokenMeter:
|
|||
|
||||
class BudgetMiddleware(ChatMiddleware):
|
||||
"""Chat middleware that charges a ``TokenMeter`` from each response's real
|
||||
``UsageDetails`` and short-circuits when the cap is crossed."""
|
||||
``UsageDetails`` and short-circuits when the cap is crossed.
|
||||
|
||||
Two teeth, not one. The PRE-call guard (S3.4) refuses to make a call the budget cannot pay
|
||||
for — that is money not spent. The post-charge check is what it always was: the real usage is
|
||||
only knowable after the response, so a call that crosses the cap can only be caught behind it.
|
||||
The guard does not replace the check; it stops the NEXT call, never the one in flight."""
|
||||
|
||||
def __init__(self, meter: TokenMeter, *, strict_usage: bool = True) -> None:
|
||||
self._meter = meter
|
||||
self._strict = strict_usage
|
||||
|
||||
async def process(self, context: ChatContext, call_next: Callable[[], Awaitable[None]]) -> None:
|
||||
exhausted = self._meter.exhausted()
|
||||
if exhausted is not None:
|
||||
raise exhausted # BEFORE call_next: the call must never be made
|
||||
await call_next()
|
||||
usage = getattr(context.result, "usage_details", None)
|
||||
total = usage.get("total_token_count") if usage is not None else None
|
||||
|
|
@ -98,3 +253,39 @@ class BudgetMiddleware(ChatMiddleware):
|
|||
)
|
||||
return
|
||||
self._meter.charge(int(total)) # raises BudgetExceeded if over cap
|
||||
|
||||
|
||||
def write_spend(path: str | Path, spent: int, *, stamp: str) -> Path:
|
||||
"""Persist a portfolio pass's accumulated ``spent`` tokens as deterministic JSON.
|
||||
|
||||
``stamp`` is a REQUIRED keyword with no wall-clock default (mirroring ``promote_verdict`` and
|
||||
the outbox's ``run_id``): the file must be byte-identical for the same spend, so a diff means
|
||||
the spend changed and nothing else. This is what carries the global cap ACROSS passes — the
|
||||
next pass seeds its ``PortfolioMeter`` from ``read_spend``."""
|
||||
target = Path(path)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
payload = {"spent_tokens": int(spent), "stamp": stamp}
|
||||
target.write_text(json.dumps(payload, sort_keys=True, indent=2), encoding="utf-8")
|
||||
return target
|
||||
|
||||
|
||||
def read_spend(path: str | Path) -> int:
|
||||
"""Read persisted spend; a MISSING file reads as ``0`` (there was no earlier pass).
|
||||
|
||||
Malformed content RAISES — deliberately unlike the tolerant raw verdict inbox
|
||||
(``load_verdicts_from_dir``). That folder is written out of band by other parties, so skipping
|
||||
junk is correct there; this file is our OWN accounting state, and reading a corrupt one as zero
|
||||
would silently hand back a budget that had already been spent."""
|
||||
target = Path(path)
|
||||
if not target.exists():
|
||||
return 0
|
||||
try:
|
||||
payload = json.loads(target.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"malformed spend file {str(target)!r}: {exc}") from exc
|
||||
if not isinstance(payload, dict) or not isinstance(payload.get("spent_tokens"), int):
|
||||
raise ValueError(f"spend file {str(target)!r} lacks an integer 'spent_tokens'")
|
||||
spent = int(payload["spent_tokens"])
|
||||
if spent < 0:
|
||||
raise ValueError(f"spend file {str(target)!r} carries negative spent_tokens: {spent}")
|
||||
return spent
|
||||
|
|
|
|||
|
|
@ -35,7 +35,13 @@ from agent_framework import BaseChatClient, SessionContext
|
|||
from pydantic import ValidationError
|
||||
|
||||
from portfolio_optimiser.backends import Profile, get_backend, resolve_model
|
||||
from portfolio_optimiser.budget import Budget, BudgetMiddleware, TokenMeter
|
||||
from portfolio_optimiser.budget import (
|
||||
Budget,
|
||||
BudgetMiddleware,
|
||||
BudgetRefused,
|
||||
PortfolioMeter,
|
||||
TokenMeter,
|
||||
)
|
||||
from portfolio_optimiser.contracts import GoalConfig, GoalContract, load_contracts, load_goal_config
|
||||
from portfolio_optimiser.ledger import SavingsLedger
|
||||
from portfolio_optimiser.datasource import (
|
||||
|
|
@ -141,6 +147,25 @@ class GoalReached:
|
|||
observed_ore: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BudgetStop:
|
||||
"""A GLOBAL token-cap stop signal VALUE (S3.4/F10) — NOT an exception, and NOT a goal.
|
||||
|
||||
Structured like ``GoalReached`` and carried the same way (a ``stop_reason``-shaped value plus a
|
||||
loop ``break``), but it is kept as its OWN field rather than widening ``stop_reason``: the two
|
||||
stops mean opposite things. A goal-stop is success (the savings target was met); this is
|
||||
resource exhaustion (the pass ran out of tokens). Folding them into one field would let a
|
||||
caller read "we stopped" without being able to tell which happened.
|
||||
|
||||
``required_tokens`` is what one more run would have needed; ``remaining_tokens`` is what was
|
||||
actually left. Both are recorded because their DIFFERENCE is the operator's next decision."""
|
||||
|
||||
limit_tokens: int
|
||||
spent_tokens: int
|
||||
remaining_tokens: int
|
||||
required_tokens: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PortfolioResult:
|
||||
"""The outcome of a sequential fan-out over N projects (SC2).
|
||||
|
|
@ -150,9 +175,11 @@ class PortfolioResult:
|
|||
The remaining fields are a thin aggregate over ``runs``: ``validated_count`` /
|
||||
``rejected_count`` partition the outcomes; ``sum_claimed_saving_nok`` totals the claimed
|
||||
saving of the validated proposals only; ``sum_token_usage`` totals every run's
|
||||
provenance token usage. ``stopped_early`` / ``stop_reason`` record a Step-8 goal-stop, and
|
||||
``failures`` records the projects that RAISED (S3.3 collect-and-continue): all three default so
|
||||
the frozen aggregate and every existing constructor call are unaffected.
|
||||
provenance token usage. ``stopped_early`` / ``stop_reason`` record a Step-8 goal-stop,
|
||||
``failures`` records the projects that RAISED (S3.3 collect-and-continue), and ``budget_stop``
|
||||
records a S3.4 global-token-cap stop (which also sets ``stopped_early``, but is kept apart from
|
||||
``stop_reason`` because exhaustion is not success): all four default so the frozen aggregate and
|
||||
every existing constructor call are unaffected.
|
||||
|
||||
``runs`` and ``failures`` PARTITION the projects that were actually submitted — a project
|
||||
appears in exactly one of them, never both, and the counts do not overlap. ``validated_count`` /
|
||||
|
|
@ -168,6 +195,7 @@ class PortfolioResult:
|
|||
stopped_early: bool = False
|
||||
stop_reason: GoalReached | None = None
|
||||
failures: tuple[RunFailure, ...] = ()
|
||||
budget_stop: BudgetStop | None = None
|
||||
|
||||
|
||||
def _authored_texts(result: Any, name: str) -> list[str]:
|
||||
|
|
@ -647,6 +675,30 @@ def _waves(ids: list[str], k: int) -> list[list[str]]:
|
|||
return [ids[i : i + k] for i in range(0, len(ids), k)]
|
||||
|
||||
|
||||
def _run_meter(
|
||||
meter_factory: Callable[[], TokenMeter] | None,
|
||||
portfolio_meter: PortfolioMeter | None,
|
||||
max_rounds: int,
|
||||
) -> TokenMeter | None:
|
||||
"""The per-project meter ``run_portfolio`` hands to one run: the injected test seam, a meter
|
||||
BOUND to the portfolio ledger, or ``None`` (letting ``run_project`` build its own, unchanged).
|
||||
|
||||
The bound meter mirrors ``run_project``'s own construction (``max(max_rounds * 4, 4)``) so the
|
||||
only difference the portfolio cap introduces is the token ceiling and the shared ledger — the
|
||||
round budget is not quietly redefined along the way."""
|
||||
if meter_factory is not None:
|
||||
return meter_factory()
|
||||
if portfolio_meter is None:
|
||||
return None
|
||||
return TokenMeter(
|
||||
Budget(
|
||||
max_tokens=portfolio_meter.budget.max_tokens_per_run,
|
||||
max_rounds=max(max_rounds * 4, 4),
|
||||
),
|
||||
portfolio=portfolio_meter,
|
||||
)
|
||||
|
||||
|
||||
async def run_portfolio(
|
||||
project_ids: Sequence[str] | None = None,
|
||||
profile: Profile | str = Profile.LOCAL,
|
||||
|
|
@ -661,6 +713,7 @@ async def run_portfolio(
|
|||
top_k: int = 3,
|
||||
concurrency: int = 1,
|
||||
meter_factory: Callable[[], TokenMeter] | None = None,
|
||||
portfolio_meter: PortfolioMeter | None = None,
|
||||
semantic_retrieval: bool = False,
|
||||
embedder: Embedder | None = None,
|
||||
) -> PortfolioResult:
|
||||
|
|
@ -727,12 +780,46 @@ async def run_portfolio(
|
|||
property of the design, so it is pinned on a bundle-backed pair where the fold does fire
|
||||
(``test_intra_wave_visibility_is_the_documented_semantic_difference``): same fixture, same
|
||||
sentinel, only the wave boundary moves. Store content stays identical across ``k`` — the
|
||||
difference is confined to what each project READ, never to what the pass produced or persisted."""
|
||||
difference is confined to what each project READ, never to what the pass produced or persisted.
|
||||
|
||||
``portfolio_meter`` (S3.4, F10) installs the GLOBAL token cap. Without it nothing changes: each
|
||||
run is bounded only by its own ``max_tokens``, so N projects can cost N times that with no
|
||||
ceiling over the pass. With it, one ``PortfolioMeter`` is shared by every run (each run's meter
|
||||
is BOUND to it, which is why ``meter_factory`` — whose meters are unbound — is refused
|
||||
alongside it: accepting both would run a pass that looks capped and is not), and the cap is
|
||||
enforced in three places that are deliberately different:
|
||||
|
||||
- **At startup**, a remainder that cannot fund one run raises ``BudgetRefused`` — a pass that
|
||||
can afford zero projects is a caller mistake, not a result.
|
||||
- **At wave assembly**, a project that cannot be funded is NEVER STARTED, and the pass stops
|
||||
with ``budget_stop`` + ``stopped_early``, every completed run preserved. Never-started is the
|
||||
property that matters: an unfunded project that is merely interrupted mid-run has already
|
||||
cost model round-trips. Because every member of a wave is checked against the SAME pre-wave
|
||||
remainder, admission RESERVES each member's requirement as it goes — otherwise a wave of k
|
||||
would over-commit the cap by up to k runs. This makes membership identical at every
|
||||
``concurrency``, matching the goal-stop's guarantee above.
|
||||
- **Before each chat call** (``BudgetMiddleware``'s pre-call guard), a call the remainder cannot
|
||||
pay for is refused rather than made. This is what bounds a run that was funded at admission
|
||||
but whose siblings drained the pass while it was in flight; such a run surfaces as a
|
||||
``RunFailure``, not as overspend.
|
||||
|
||||
Spend is carried ACROSS passes by seeding the meter from ``budget.read_spend`` and writing
|
||||
``budget.write_spend`` afterwards. Those are the caller's calls, not this function's — the pass
|
||||
reads its ledger, it does not own the file (mirroring the Steg-7 role split)."""
|
||||
if concurrency < 1:
|
||||
raise ValueError(
|
||||
f"concurrency must be >= 1, got {concurrency}: a non-positive wave size would run no "
|
||||
"projects at all and read as an empty portfolio. Pass 1 for the sequential pass."
|
||||
)
|
||||
if portfolio_meter is not None and meter_factory is not None:
|
||||
raise ValueError(
|
||||
"portfolio_meter and meter_factory are mutually exclusive: a meter_factory meter is "
|
||||
"not bound to the portfolio ledger, so the global cap would be silently unenforced"
|
||||
)
|
||||
# Startup refusal (fail-fast, before anything loads): a pass that cannot fund its first run
|
||||
# must not begin. Raised, not returned — an empty PortfolioResult would read as "nothing to do".
|
||||
if portfolio_meter is not None and not portfolio_meter.can_fund_run():
|
||||
raise BudgetRefused(portfolio_meter.remaining(), portfolio_meter.required_per_run)
|
||||
projects = {p.id: p for p in load_reference_projects()}
|
||||
ids = list(project_ids) if project_ids is not None else list(projects)
|
||||
store = store if store is not None else VerdictStore(verdicts=[])
|
||||
|
|
@ -747,8 +834,13 @@ async def run_portfolio(
|
|||
failures: list[RunFailure] = []
|
||||
stopped_early = False
|
||||
stop_reason: GoalReached | None = None
|
||||
budget_stop: BudgetStop | None = None
|
||||
for wave_ids in _waves(ids, concurrency):
|
||||
members: list[str] = []
|
||||
# Tokens committed to members already admitted to THIS wave but that have not spent yet.
|
||||
# Every member reads the same pre-wave remainder, so without this a wave of k would admit
|
||||
# k projects off one project's worth of budget.
|
||||
reserved = 0
|
||||
for pid in wave_ids:
|
||||
if pid not in projects:
|
||||
raise ValueError(f"unknown project_id: {pid!r}")
|
||||
|
|
@ -777,6 +869,21 @@ async def run_portfolio(
|
|||
if per_project_goal.mode == "hard":
|
||||
continue # skip THIS pid; the rest of the pass proceeds
|
||||
|
||||
# S3.4 funding check, placed AFTER the goal checks: a reached goal is success and owns
|
||||
# the stop when both apply, and a pid a hard per-project goal already skipped costs no
|
||||
# budget, so it must not consume a reservation.
|
||||
if portfolio_meter is not None and not portfolio_meter.can_fund_run(reserved=reserved):
|
||||
stopped_early = True
|
||||
budget_stop = BudgetStop(
|
||||
limit_tokens=portfolio_meter.budget.max_total_tokens,
|
||||
spent_tokens=portfolio_meter.spent,
|
||||
remaining_tokens=portfolio_meter.remaining(),
|
||||
required_tokens=portfolio_meter.required_per_run,
|
||||
)
|
||||
break
|
||||
|
||||
if portfolio_meter is not None:
|
||||
reserved += portfolio_meter.required_per_run
|
||||
members.append(pid)
|
||||
|
||||
# Every project in the wave reads the SAME wave-start state and writes only its own copy,
|
||||
|
|
@ -805,7 +912,7 @@ async def run_portfolio(
|
|||
top_k=top_k,
|
||||
semantic_retrieval=semantic_retrieval,
|
||||
embedder=embedder,
|
||||
meter=meter_factory() if meter_factory is not None else None,
|
||||
meter=_run_meter(meter_factory, portfolio_meter, max_rounds),
|
||||
)
|
||||
for pid, snapshot in snapshots
|
||||
),
|
||||
|
|
@ -836,12 +943,13 @@ async def run_portfolio(
|
|||
break
|
||||
|
||||
base = _aggregate(tuple(runs), store)
|
||||
if stopped_early or stop_reason is not None or failures:
|
||||
if stopped_early or stop_reason is not None or failures or budget_stop is not None:
|
||||
return replace(
|
||||
base,
|
||||
stopped_early=stopped_early,
|
||||
stop_reason=stop_reason,
|
||||
failures=tuple(failures),
|
||||
budget_stop=budget_stop,
|
||||
)
|
||||
return base
|
||||
|
||||
|
|
|
|||
|
|
@ -15,6 +15,8 @@ from portfolio_optimiser.budget import (
|
|||
Budget,
|
||||
BudgetExceeded,
|
||||
BudgetMiddleware,
|
||||
PortfolioBudget,
|
||||
PortfolioMeter,
|
||||
TokenMeter,
|
||||
UsageUnavailable,
|
||||
)
|
||||
|
|
@ -85,6 +87,90 @@ async def test_budget_middleware_fires_on_real_agent_chat(make_client_factory) -
|
|||
assert meter.tokens == 8 # charged from the synthetic UsageDetails via the middleware
|
||||
|
||||
|
||||
async def test_pre_call_guard_does_not_await_call_next_when_exhausted() -> None:
|
||||
"""S3.4 unit arm of the pre-call guard: the middleware refuses BEFORE ``call_next``. Counting
|
||||
the awaits is the whole measurement — the post-charge arm raises either way."""
|
||||
calls = {"n": 0}
|
||||
|
||||
async def _counted() -> None:
|
||||
calls["n"] += 1
|
||||
|
||||
meter = TokenMeter(Budget(max_tokens=10, max_rounds=10))
|
||||
meter.charge(10) # exactly at the cap -> nothing left to fund a call with
|
||||
mw = BudgetMiddleware(meter)
|
||||
with pytest.raises(BudgetExceeded):
|
||||
await mw.process(_Ctx(_resp(5)), _counted) # type: ignore[arg-type]
|
||||
assert calls["n"] == 0
|
||||
|
||||
|
||||
async def test_below_the_cap_still_calls_through() -> None:
|
||||
"""Control for the guard: with budget left, the call goes through exactly as before — the
|
||||
guard must gate exhaustion, not traffic."""
|
||||
calls = {"n": 0}
|
||||
|
||||
async def _counted() -> None:
|
||||
calls["n"] += 1
|
||||
|
||||
meter = TokenMeter(Budget(max_tokens=10, max_rounds=10))
|
||||
mw = BudgetMiddleware(meter)
|
||||
await mw.process(_Ctx(_resp(4)), _counted) # type: ignore[arg-type]
|
||||
assert calls["n"] == 1 and meter.tokens == 4 and meter.remaining() == 6
|
||||
|
||||
|
||||
def test_portfolio_budget_rejects_unusable_configs() -> None:
|
||||
"""Fail-fast at construction (mirroring ``Budget``): non-positive caps, a per-run cap larger
|
||||
than the global one (one run could then cross the pass's own ceiling), and a reserve larger
|
||||
than a run can ever spend (which would refuse every pass forever)."""
|
||||
with pytest.raises(ValueError):
|
||||
PortfolioBudget(max_total_tokens=0, max_tokens_per_run=10)
|
||||
with pytest.raises(ValueError):
|
||||
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=0)
|
||||
with pytest.raises(ValueError):
|
||||
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=200)
|
||||
with pytest.raises(ValueError):
|
||||
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=50, min_run_reserve=60)
|
||||
with pytest.raises(ValueError):
|
||||
PortfolioMeter(PortfolioBudget(max_total_tokens=100, max_tokens_per_run=50), spent=-1)
|
||||
|
||||
|
||||
def test_portfolio_meter_accumulates_and_crosses() -> None:
|
||||
budget = PortfolioBudget(max_total_tokens=100, max_tokens_per_run=50)
|
||||
meter = PortfolioMeter(budget, spent=40)
|
||||
assert meter.remaining() == 60 and meter.required_per_run == 50
|
||||
meter.record(60)
|
||||
meter.check() # exactly at the cap is still within it (mirrors TokenMeter's `>` boundary)
|
||||
meter.record(1)
|
||||
with pytest.raises(BudgetExceeded) as exc:
|
||||
meter.check()
|
||||
assert exc.value.kind == "portfolio_tokens"
|
||||
assert exc.value.limit == 100 and exc.value.observed == 101
|
||||
|
||||
|
||||
async def test_run_meter_bound_to_portfolio_charges_both_ledgers() -> None:
|
||||
"""A bound run meter charges the global ledger on every call, and ``remaining`` reads whichever
|
||||
cap binds — that is what lets one project's spend refuse another's next call."""
|
||||
portfolio = PortfolioMeter(PortfolioBudget(max_total_tokens=100, max_tokens_per_run=80))
|
||||
meter = TokenMeter(Budget(max_tokens=80, max_rounds=10), portfolio=portfolio)
|
||||
mw = BudgetMiddleware(meter)
|
||||
await mw.process(_Ctx(_resp(30)), _noop) # type: ignore[arg-type]
|
||||
assert meter.tokens == 30 and portfolio.spent == 30
|
||||
assert meter.remaining() == 50 # run has 50 left, portfolio 70 -> the RUN binds
|
||||
await mw.process(_Ctx(_resp(45)), _noop) # type: ignore[arg-type]
|
||||
assert meter.remaining() == 5 # run 5, portfolio 25 -> still the run
|
||||
|
||||
|
||||
async def test_spend_is_ledgered_even_when_the_per_run_cap_raises() -> None:
|
||||
"""Order matters: tokens the provider already billed must reach the global ledger even though
|
||||
the RUN's own cap raises on the same charge. Checking the run cap first and returning early
|
||||
would lose that spend and hand the next project a budget that was never really there."""
|
||||
portfolio = PortfolioMeter(PortfolioBudget(max_total_tokens=1000, max_tokens_per_run=50))
|
||||
meter = TokenMeter(Budget(max_tokens=50, max_rounds=10), portfolio=portfolio)
|
||||
with pytest.raises(BudgetExceeded) as exc:
|
||||
meter.charge(60)
|
||||
assert exc.value.kind == "tokens" # the RUN's cap is the one that broke
|
||||
assert portfolio.spent == 60 # ...and the global ledger still saw the spend
|
||||
|
||||
|
||||
def test_no_word_count_token_proxy_in_src() -> None:
|
||||
# The meter is fed from real UsageDetails, never a len(text.split()) word-count proxy
|
||||
# (research 03 Rec 3 — the Fase 1 _word_tokens anti-pattern is retired). NOTE: a bare
|
||||
|
|
|
|||
302
tests/test_portfolio_budget_loadbearing.py
Normal file
302
tests/test_portfolio_budget_loadbearing.py
Normal file
|
|
@ -0,0 +1,302 @@
|
|||
"""S3.4 load-bearing — the GLOBAL token cap across a portfolio pass (F10).
|
||||
|
||||
Two properties the session plan names as RED conditions, plus the wave-size and control arms
|
||||
that keep them honest:
|
||||
|
||||
1. **The global cap stops the pass mid-way, with completed runs preserved.** Detach the
|
||||
wave-assembly funding check and the third project is STARTED anyway — it then dies on the
|
||||
meter mid-run, so the pass reports a failure instead of a structured stop. The property is
|
||||
never-started, not merely never-overspent: a project that cannot be funded must cost zero
|
||||
model round-trips.
|
||||
2. **The pre-call guard refuses the call BEFORE it is made.** The client double counts its own
|
||||
invocations, so "the call never happened" is measured, not inferred. Today's post-charge
|
||||
middleware would let the call through and only then raise — which is exactly the money the
|
||||
guard exists to not spend.
|
||||
|
||||
Token arithmetic is MEASURED, not guessed: every project in the reference portfolio spends
|
||||
4 chat calls x ``tokens`` per reply. At ``tokens=8`` that is a flat 32 tokens per run (probed
|
||||
against ``provenance.token_usage`` before these tests were written), which is what makes the
|
||||
80/64/32 budget below land the stop between run 2 and run 3.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from agent_framework import Agent
|
||||
|
||||
from portfolio_optimiser.budget import (
|
||||
Budget,
|
||||
BudgetExceeded,
|
||||
BudgetMiddleware,
|
||||
BudgetRefused,
|
||||
PortfolioBudget,
|
||||
PortfolioMeter,
|
||||
TokenMeter,
|
||||
read_spend,
|
||||
write_spend,
|
||||
)
|
||||
from portfolio_optimiser.run import run_portfolio
|
||||
|
||||
_PORTFOLIO_IDS = ["FV42-GSV-E1", "RV13-RAS-TP", "BRU-LAKS-REHAB"]
|
||||
|
||||
# Per-project replies (the tested constants of tests/test_portfolio.py, unchanged): all three
|
||||
# validate, and their claimed savings are DISTINCT — which is how a run is keyed back to its
|
||||
# project here, since ``RunResult`` carries no project id. 200000 + 130000 = the first two.
|
||||
REPLIES = {
|
||||
"FV42-GSV-E1": (
|
||||
'{"measure":"Reduce scope","affected_items":['
|
||||
'{"code":"05.2","quantity":4300,"unit_cost":215},'
|
||||
'{"code":"03.1","quantity":1800,"unit_cost":310}],"claimed_saving_nok":200000}'
|
||||
),
|
||||
"RV13-RAS-TP": (
|
||||
'{"measure":"Material substitution","affected_items":['
|
||||
'{"code":"88.2","quantity":180,"unit_cost":4200}],"claimed_saving_nok":130000}'
|
||||
),
|
||||
"BRU-LAKS-REHAB": (
|
||||
'{"measure":"Reduce scope","affected_items":['
|
||||
'{"code":"05.2","quantity":4300,"unit_cost":215},'
|
||||
'{"code":"07.4","quantity":2400,"unit_cost":690}],"claimed_saving_nok":210000}'
|
||||
),
|
||||
}
|
||||
_FIRST_TWO_SAVING = 330000
|
||||
|
||||
# Measured: 32 tokens per run at tokens=8. 80 total funds two runs (32 + 32 = 64) and leaves 16,
|
||||
# which is below the 32-token reserve one more run requires -> the third is never started.
|
||||
_TOTAL = 80
|
||||
_PER_RUN = 64
|
||||
_RESERVE = 32
|
||||
_PER_RUN_SPEND = 32
|
||||
|
||||
|
||||
def _budget() -> PortfolioBudget:
|
||||
return PortfolioBudget(
|
||||
max_total_tokens=_TOTAL, max_tokens_per_run=_PER_RUN, min_run_reserve=_RESERVE
|
||||
)
|
||||
|
||||
|
||||
async def test_global_cap_stops_the_pass_mid_way_with_completed_runs_preserved(
|
||||
make_portfolio_client_factory, fresh_store
|
||||
) -> None:
|
||||
"""RED 1: the pass stops the moment the REMAINING global budget cannot fund another run, and
|
||||
every completed run survives the stop.
|
||||
|
||||
Detach the wave-assembly funding check and this goes red three ways at once: project 3 is
|
||||
started (so it lands in ``failures`` when its charge crosses the cap), ``stopped_early`` stays
|
||||
False, and ``budget_stop`` stays None. The strongest of the three is ``failures == ()``: an
|
||||
unfunded project that is merely *interrupted* has already cost model round-trips, which is the
|
||||
precise thing a global cap is for."""
|
||||
meter = PortfolioMeter(_budget())
|
||||
result = await run_portfolio(
|
||||
_PORTFOLIO_IDS,
|
||||
"local",
|
||||
store=fresh_store,
|
||||
client_factory=make_portfolio_client_factory(REPLIES, tokens=8),
|
||||
portfolio_meter=meter,
|
||||
)
|
||||
|
||||
assert result.stopped_early is True
|
||||
assert result.budget_stop is not None
|
||||
assert result.budget_stop.limit_tokens == _TOTAL
|
||||
assert result.budget_stop.spent_tokens == 2 * _PER_RUN_SPEND
|
||||
assert result.budget_stop.remaining_tokens == _TOTAL - 2 * _PER_RUN_SPEND
|
||||
assert result.budget_stop.required_tokens == _RESERVE
|
||||
|
||||
# Completed runs preserved, and the unfunded project NEVER STARTED: absent from BOTH sides of
|
||||
# the runs/failures partition, not merely absent from runs. A project that had been started and
|
||||
# then killed by the meter would land in ``failures`` — which is what makes ``== ()`` the
|
||||
# sharpest assertion here. The claimed-saving sum names WHICH two ran (the replies are distinct).
|
||||
assert len(result.runs) == 2
|
||||
assert result.failures == ()
|
||||
assert result.validated_count == 2
|
||||
assert result.sum_claimed_saving_nok == _FIRST_TWO_SAVING
|
||||
|
||||
# The cap itself held: the pass spent what two runs cost and no more.
|
||||
assert meter.spent == 2 * _PER_RUN_SPEND
|
||||
assert meter.spent <= _TOTAL
|
||||
assert meter.spent == result.sum_token_usage
|
||||
|
||||
|
||||
async def test_generous_budget_runs_the_whole_pass(
|
||||
make_portfolio_client_factory, fresh_store
|
||||
) -> None:
|
||||
"""Control: the stop above is CAUSED by the cap, not by wiring a meter in at all. With a global
|
||||
budget that funds every project the pass completes untouched — a gate that can only fire proves
|
||||
nothing about what fires it."""
|
||||
meter = PortfolioMeter(PortfolioBudget(max_total_tokens=10_000, max_tokens_per_run=_PER_RUN))
|
||||
result = await run_portfolio(
|
||||
_PORTFOLIO_IDS,
|
||||
"local",
|
||||
store=fresh_store,
|
||||
client_factory=make_portfolio_client_factory({}, tokens=8),
|
||||
portfolio_meter=meter,
|
||||
)
|
||||
assert result.stopped_early is False
|
||||
assert result.budget_stop is None
|
||||
assert len(result.runs) == 3
|
||||
assert meter.spent == 3 * _PER_RUN_SPEND
|
||||
|
||||
|
||||
async def test_global_cap_holds_at_wave_size_three(
|
||||
make_portfolio_client_factory, fresh_store
|
||||
) -> None:
|
||||
"""The cap is a property of the PASS, not of the sequential schedule: at ``concurrency=3`` the
|
||||
whole portfolio is assembled into ONE wave, and every member's funding check reads the same
|
||||
pre-wave ``remaining``. Admitting all three off that shared reading would overspend by design.
|
||||
The wave therefore RESERVES each admitted member's requirement as it admits it, so the same 80
|
||||
tokens fund the same two projects at k=3 as at k=1."""
|
||||
meter = PortfolioMeter(_budget())
|
||||
result = await run_portfolio(
|
||||
_PORTFOLIO_IDS,
|
||||
"local",
|
||||
store=fresh_store,
|
||||
client_factory=make_portfolio_client_factory({}, tokens=8),
|
||||
concurrency=3,
|
||||
portfolio_meter=meter,
|
||||
)
|
||||
assert len(result.runs) == 2
|
||||
assert result.failures == ()
|
||||
assert result.stopped_early is True
|
||||
assert meter.spent <= _TOTAL
|
||||
|
||||
|
||||
async def test_pre_call_guard_refuses_the_call_before_it_is_made(make_client_factory) -> None:
|
||||
"""RED 2: with the budget exhausted, the chat call is REFUSED — the client is never invoked.
|
||||
|
||||
``call_count`` on the scripted client is the measurement: post-charge middleware alone would
|
||||
make the call, read its usage, and only then raise, so ``pytest.raises`` would pass either way.
|
||||
Only the counter separates 'refused' from 'made and then regretted'."""
|
||||
client = make_client_factory("ok", tokens=8)("proposer")
|
||||
meter = TokenMeter(Budget(max_tokens=10, max_rounds=10))
|
||||
meter.charge(10) # exactly at the cap: within budget, but nothing left to fund a call with
|
||||
assert meter.remaining() == 0
|
||||
|
||||
agent = Agent(
|
||||
client, "propose a measure", name="proposer", middleware=[BudgetMiddleware(meter)]
|
||||
)
|
||||
with pytest.raises(BudgetExceeded) as exc:
|
||||
await agent.run("hi")
|
||||
assert exc.value.kind == "tokens"
|
||||
assert client.call_count == 0, "the call must NEVER be made once the budget is exhausted"
|
||||
|
||||
|
||||
async def test_pre_call_guard_fires_on_the_GLOBAL_remainder_too(make_client_factory) -> None:
|
||||
"""The guard reads whichever cap binds. Here the run's OWN budget is untouched and only the
|
||||
portfolio remainder is exhausted — a run that starts funded can still be refused mid-flight
|
||||
because a sibling spent the rest of the global budget."""
|
||||
client = make_client_factory("ok", tokens=8)("proposer")
|
||||
portfolio = PortfolioMeter(
|
||||
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=100), spent=100
|
||||
)
|
||||
meter = TokenMeter(Budget(max_tokens=100, max_rounds=10), portfolio=portfolio)
|
||||
assert meter.tokens == 0 # the RUN has spent nothing
|
||||
assert meter.remaining() == 0 # ...but the portfolio has nothing left
|
||||
|
||||
agent = Agent(
|
||||
client, "propose a measure", name="proposer", middleware=[BudgetMiddleware(meter)]
|
||||
)
|
||||
with pytest.raises(BudgetExceeded) as exc:
|
||||
await agent.run("hi")
|
||||
assert exc.value.kind == "portfolio_tokens"
|
||||
assert client.call_count == 0
|
||||
|
||||
|
||||
async def test_startup_refusal_when_the_remainder_cannot_fund_one_run(
|
||||
make_portfolio_client_factory, fresh_store
|
||||
) -> None:
|
||||
"""Fail-fast at startup (never a silently truncated pass): a meter whose remainder is already
|
||||
below one run's requirement refuses BEFORE anything runs. This is the across-passes arm — the
|
||||
remainder here is what an earlier pass left behind."""
|
||||
meter = PortfolioMeter(_budget(), spent=_TOTAL - (_RESERVE - 1))
|
||||
with pytest.raises(BudgetRefused) as exc:
|
||||
await run_portfolio(
|
||||
_PORTFOLIO_IDS,
|
||||
"local",
|
||||
store=fresh_store,
|
||||
client_factory=make_portfolio_client_factory({}, tokens=8),
|
||||
portfolio_meter=meter,
|
||||
)
|
||||
assert exc.value.remaining == _RESERVE - 1
|
||||
assert exc.value.required == _RESERVE
|
||||
assert meter.spent == _TOTAL - (_RESERVE - 1) # nothing ran, nothing charged
|
||||
|
||||
|
||||
async def test_spend_persists_across_passes(
|
||||
make_portfolio_client_factory, fresh_store, tmp_path: Path
|
||||
) -> None:
|
||||
"""The cap holds ACROSS passes, which is what the persisted spend buys: pass A writes its
|
||||
spend, pass B is seeded from that file and stops one run earlier than it otherwise would."""
|
||||
spend_file = tmp_path / "spend.json"
|
||||
assert read_spend(spend_file) == 0 # no prior pass -> no prior spend
|
||||
|
||||
budget = PortfolioBudget(
|
||||
max_total_tokens=3 * _PER_RUN_SPEND, max_tokens_per_run=_PER_RUN, min_run_reserve=_RESERVE
|
||||
)
|
||||
meter_a = PortfolioMeter(budget)
|
||||
result_a = await run_portfolio(
|
||||
_PORTFOLIO_IDS[:1],
|
||||
"local",
|
||||
store=fresh_store,
|
||||
client_factory=make_portfolio_client_factory({}, tokens=8),
|
||||
portfolio_meter=meter_a,
|
||||
)
|
||||
assert len(result_a.runs) == 1
|
||||
write_spend(spend_file, meter_a.spent, stamp="pass-a")
|
||||
|
||||
meter_b = PortfolioMeter(budget, spent=read_spend(spend_file))
|
||||
assert meter_b.spent == _PER_RUN_SPEND # pass B starts where pass A stopped
|
||||
result_b = await run_portfolio(
|
||||
_PORTFOLIO_IDS,
|
||||
"local",
|
||||
store=fresh_store,
|
||||
client_factory=make_portfolio_client_factory({}, tokens=8),
|
||||
portfolio_meter=meter_b,
|
||||
)
|
||||
# 96 total funds three runs; one is already spent, so pass B affords two and stops before the
|
||||
# third — the earlier pass's spend is what moved the stop.
|
||||
assert len(result_b.runs) == 2
|
||||
assert result_b.stopped_early is True
|
||||
assert meter_b.spent == 3 * _PER_RUN_SPEND
|
||||
|
||||
|
||||
def test_spend_file_is_deterministic_and_stamped(tmp_path: Path) -> None:
|
||||
"""Byte-determinism with an EXPLICIT stamp (mirroring ``promote_verdict``): no wall-clock, so
|
||||
two writes of the same spend are byte-identical and a diff means the spend changed."""
|
||||
a, b = tmp_path / "a.json", tmp_path / "b.json"
|
||||
write_spend(a, 1234, stamp="run-7")
|
||||
write_spend(b, 1234, stamp="run-7")
|
||||
assert a.read_bytes() == b.read_bytes()
|
||||
assert json.loads(a.read_text(encoding="utf-8")) == {"spent_tokens": 1234, "stamp": "run-7"}
|
||||
assert read_spend(a) == 1234
|
||||
|
||||
|
||||
def test_spend_file_malformed_fails_fast(tmp_path: Path) -> None:
|
||||
"""The spend file is OUR OWN accounting state, not the tolerant raw inbox layer: a corrupt file
|
||||
raises rather than silently reading as zero spend, which would hand back the whole budget."""
|
||||
bad = tmp_path / "bad.json"
|
||||
bad.write_text("{not json", encoding="utf-8")
|
||||
with pytest.raises(ValueError):
|
||||
read_spend(bad)
|
||||
negative = tmp_path / "neg.json"
|
||||
negative.write_text('{"spent_tokens": -5, "stamp": "x"}', encoding="utf-8")
|
||||
with pytest.raises(ValueError):
|
||||
read_spend(negative)
|
||||
|
||||
|
||||
async def test_meter_factory_and_portfolio_meter_are_mutually_exclusive(
|
||||
make_portfolio_client_factory, fresh_store
|
||||
) -> None:
|
||||
"""Fail-fast on the one wiring that would silently disable the global cap: a ``meter_factory``
|
||||
meter is not bound to the portfolio ledger, so accepting both would run a pass that LOOKS
|
||||
capped and is not."""
|
||||
with pytest.raises(ValueError):
|
||||
await run_portfolio(
|
||||
_PORTFOLIO_IDS,
|
||||
"local",
|
||||
store=fresh_store,
|
||||
client_factory=make_portfolio_client_factory({}, tokens=8),
|
||||
meter_factory=lambda: TokenMeter(Budget(max_tokens=1000, max_rounds=10)),
|
||||
portfolio_meter=PortfolioMeter(_budget()),
|
||||
)
|
||||
Loading…
Add table
Add a link
Reference in a new issue