feat(budget): enforce a global portfolio token cap before the call, not after it (S3.4/F10)
PortfolioBudget + PortfolioMeter carry ONE token ledger over a whole portfolio pass -- and, seeded from a persisted spend file, across passes -- while the per-run Budget/TokenMeter pair is untouched. Three enforcement points, each doing a different job: - startup: a remainder that cannot fund one run raises BudgetRefused before anything loads (a pass that can afford zero projects is a caller mistake, not a result); - wave assembly: an unfundable project is NEVER STARTED and the pass stops structurally (budget_stop + stopped_early, completed runs preserved). Because every member of a wave is funded against the SAME pre-wave remainder, admission RESERVES each member's requirement -- otherwise a wave of k over-commits the cap by up to k runs; - pre-call: BudgetMiddleware refuses a call the remainder cannot pay for instead of making it. The post-charge check stays: real usage is only knowable after the response, so the guard stops the NEXT call, never the one in flight. budget_stop is its own field rather than a widened stop_reason -- a goal-stop is success, this is resource exhaustion, and fusing them would make "we stopped" unreadable. PortfolioMeter splits record/check so tokens the provider already billed reach the ledger even when the same charge breaks the run's own cap. read_spend raises on corrupt content (our own accounting state, unlike the tolerant RAW inbox layer); write_spend takes a REQUIRED stamp with no wall-clock default, mirroring promote_verdict. Load-bearing MEASURED, not asserted -- 6 mutations, all red: detach the wave check; detach the pre-call guard; detach the wave reservation; check the run cap before crediting the global ledger; detach the startup refusal; make read_spend tolerant. Files restored from shasum-verified copies after each. Two findings worth keeping: the pre-call guard MASKS a detached wave check if the test asserts on overspend (spend stays under the cap either way), so the load-bearing assertion had to become failures == () plus never-started; and the token arithmetic is probed (32 tokens/run at tokens=8), not guessed. 537 -> 553 tests, ruff + mypy green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EaxFnaDAbMQkmTeX4u7sd
This commit is contained in:
parent
0d50ab89d3
commit
a831aa1e3b
6 changed files with 733 additions and 11 deletions
23
CLAUDE.md
23
CLAUDE.md
|
|
@ -105,6 +105,29 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
||||||
(`tests/test_step8_promotion_loadbearing.py`): gaten avviser ikke-godkjent dom (RØD uten gate); godkjent
|
(`tests/test_step8_promotion_loadbearing.py`): gaten avviser ikke-godkjent dom (RØD uten gate); godkjent
|
||||||
dom er navigerbar (RØD når `link_in_index` detaches); promotert signal holdes ute av `bundle_context`
|
dom er navigerbar (RØD når `link_in_index` detaches); promotert signal holdes ute av `bundle_context`
|
||||||
(RØD når en beskrivende index-label lekker det inn). Index-RMW er ikke-atomisk (enprosess-MVP).
|
(RØD når en beskrivende index-label lekker det inn). Index-RMW er ikke-atomisk (enprosess-MVP).
|
||||||
|
- **Globalt token-tak håndheves FØR kall, aldri bare etterpå (S3.4, F10):** `PortfolioBudget` +
|
||||||
|
`PortfolioMeter` er ÉN ledger over hele porteføljepasset (og — seedet av `read_spend` — på tvers
|
||||||
|
av pass), mens per-run `Budget`/`TokenMeter` er uendret. Taket har tre tenner, med hver sin jobb:
|
||||||
|
(1) **oppstartsnekt** — en rest som ikke kan finansiere én kjøring raiser `BudgetRefused` FØR noe
|
||||||
|
lastes (et pass som har råd til null prosjekter er en caller-feil, ikke et resultat);
|
||||||
|
(2) **wave-assembly** — et prosjekt som ikke kan finansieres blir ALDRI STARTET, og passet stopper
|
||||||
|
strukturert (`budget_stop` + `stopped_early`, fullførte runs bevart). Aldri-startet er poenget:
|
||||||
|
et ufinansiert prosjekt som bare avbrytes har allerede kostet kall. Fordi hele bølgen sjekkes mot
|
||||||
|
SAMME før-bølge-rest, **reserverer** admission hver members krav — ellers overforplikter en bølge
|
||||||
|
av k taket med inntil k kjøringer; (3) **pre-call-guard** i `BudgetMiddleware` — et kall resten
|
||||||
|
ikke kan betale for NEKTES i stedet for å gjøres (post-charge-sjekken består: ekte usage kjennes
|
||||||
|
først etterpå, så guarden stopper NESTE kall, aldri det som er i lufta). `budget_stop` er et EGET
|
||||||
|
felt, aldri `stop_reason`: et mål-stopp er suksess, dette er ressurs-utmattelse — å slå dem sammen
|
||||||
|
ville gjort «vi stoppet» uleselig. `record`/`check` er SPLITTET i `PortfolioMeter` fordi tokens
|
||||||
|
leverandøren allerede har fakturert må nå ledgeren selv når samme charge bryter run-taket.
|
||||||
|
Spend-fila er vår EGEN regnskapstilstand: `read_spend` raiser på korrupt innhold (kontrast det
|
||||||
|
tolerante RAW-inbox-laget — å lese korrupt som null ville gitt tilbake et allerede brukt budsjett),
|
||||||
|
og `write_spend` tar et PÅKREVD `stamp` uten wall-clock-default (byte-determinisme, speiler
|
||||||
|
`promote_verdict`). `portfolio_meter` og `meter_factory` er gjensidig utelukkende — en
|
||||||
|
factory-meter er ubundet, så begge sammen ville gitt et pass som SER capped ut uten å være det.
|
||||||
|
Load-bearing MÅLT (`tests/test_portfolio_budget_loadbearing.py` + `tests/test_budget.py`), seks
|
||||||
|
mutasjoner alle røde: detach wave-sjekken · detach pre-call-guarden · detach bølge-reservasjonen ·
|
||||||
|
sjekk run-taket før global kreditering · detach oppstartsnekten · gjør `read_spend` tolerant.
|
||||||
- **Kostnadsdisiplin:** utvikle primært på lokal profil (gratis); Foundry/Azure (privat tenant finnes) kun til målrettet, minimal verifisering; billigste modeller + små syntetiske data + harde token-tak. Ingen tunge test-kjøringer.
|
- **Kostnadsdisiplin:** utvikle primært på lokal profil (gratis); Foundry/Azure (privat tenant finnes) kun til målrettet, minimal verifisering; billigste modeller + små syntetiske data + harde token-tak. Ingen tunge test-kjøringer.
|
||||||
- **Offline simulering = primært metode-bevis (kostnadsdrevet, erstatter §11.8):** operatøren kjører
|
- **Offline simulering = primært metode-bevis (kostnadsdrevet, erstatter §11.8):** operatøren kjører
|
||||||
IKKE MAF mot ekte modell (verken Azure/Foundry eller Ollama — API for begge repoene er for kostbart
|
IKKE MAF mot ekte modell (verken Azure/Foundry eller Ollama — API for begge repoene er for kostbart
|
||||||
|
|
|
||||||
12
README.md
12
README.md
|
|
@ -149,6 +149,18 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
||||||
not CLI-enforced. Stop criteria and budget caps are required at startup. Try the offline
|
not CLI-enforced. Stop criteria and budget caps are required at startup. Try the offline
|
||||||
end-to-end proof (no model, no network): `uv run python -m portfolio_optimiser.simulation`.
|
end-to-end proof (no model, no network): `uv run python -m portfolio_optimiser.simulation`.
|
||||||
|
|
||||||
|
- **A global token cap across the whole portfolio, enforced before the call.** Per-run caps alone
|
||||||
|
let N projects cost N times that with no ceiling over the pass. Pass a
|
||||||
|
`PortfolioMeter(PortfolioBudget(max_total_tokens=…, max_tokens_per_run=…))` to `run_portfolio`
|
||||||
|
and one ledger bounds the entire pass — and, seeded from `budget.read_spend`, a *series* of
|
||||||
|
passes. It bites in three places: a remainder that cannot fund one run refuses the pass at
|
||||||
|
startup (`BudgetRefused`); a project that cannot be funded is **never started**, stopping the
|
||||||
|
pass structurally (`budget_stop`, completed runs preserved); and a chat call the remainder
|
||||||
|
cannot pay for is **refused rather than made** (the post-charge check remains, since real usage
|
||||||
|
is only knowable after the response). Spend persists via `budget.write_spend`, which takes an
|
||||||
|
explicit stamp and no wall-clock default, so the file is byte-deterministic. Python API only —
|
||||||
|
not yet exposed on the CLI.
|
||||||
|
|
||||||
## What this enables
|
## What this enables
|
||||||
|
|
||||||
The reference case is portfolio cost review (the example bundle is a building-energy measure),
|
The reference case is portfolio cost review (the example bundle is a building-energy measure),
|
||||||
|
|
|
||||||
|
|
@ -8,12 +8,21 @@ call and short-circuits with ``BudgetExceeded`` the moment the cap is crossed.
|
||||||
``strict_usage`` (default ``True``) makes a missing usage a HARD FAIL (``UsageUnavailable``):
|
``strict_usage`` (default ``True``) makes a missing usage a HARD FAIL (``UsageUnavailable``):
|
||||||
a usage regression must never silently disable the cap. Test doubles that legitimately supply
|
a usage regression must never silently disable the cap. Test doubles that legitimately supply
|
||||||
a synthetic ``UsageDetails`` do not trip it.
|
a synthetic ``UsageDetails`` do not trip it.
|
||||||
|
|
||||||
|
S3.4 (F10) adds the cross-project half. ``PortfolioBudget`` + ``PortfolioMeter`` carry ONE token
|
||||||
|
ledger over a whole portfolio pass — and, seeded from ``read_spend``, across passes — while the
|
||||||
|
per-run ``Budget``/``TokenMeter`` pair stays exactly what it was. Two teeth follow from that:
|
||||||
|
the middleware now refuses a call BEFORE making it once the binding cap is exhausted (money not
|
||||||
|
spent, not money spent and regretted), and ``run_portfolio`` refuses to START a project it cannot
|
||||||
|
fund. Both are enforcement, never repair — nothing is trimmed, retried, or scaled down.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
from collections.abc import Awaitable, Callable
|
from collections.abc import Awaitable, Callable
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
from agent_framework import ChatContext, ChatMiddleware
|
from agent_framework import ChatContext, ChatMiddleware
|
||||||
|
|
||||||
|
|
@ -37,6 +46,24 @@ class UsageUnavailable(RuntimeError):
|
||||||
must fail closed rather than silently stop counting (research 03 Rec 3)."""
|
must fail closed rather than silently stop counting (research 03 Rec 3)."""
|
||||||
|
|
||||||
|
|
||||||
|
class BudgetRefused(RuntimeError):
|
||||||
|
"""Raised at STARTUP when the portfolio's remaining tokens cannot fund one run (S3.4).
|
||||||
|
|
||||||
|
Distinct from ``BudgetExceeded`` on purpose: nothing was crossed and nothing was spent — the
|
||||||
|
pass is refused before it begins, because a pass that can afford zero projects is a caller
|
||||||
|
mistake, not a resource event. Named for the repo's fail-closed gate family
|
||||||
|
(``PromotionRefused``, ``IngestStampError``): refusal, never repair.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, remaining: int, required: int) -> None:
|
||||||
|
self.remaining = remaining
|
||||||
|
self.required = required
|
||||||
|
super().__init__(
|
||||||
|
f"portfolio budget refused: remaining={remaining} cannot fund one run "
|
||||||
|
f"(requires {required}); raise the global cap or lower min_run_reserve"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class Budget:
|
class Budget:
|
||||||
"""Hard token + round/iteration caps, required at startup (A4 / D6).
|
"""Hard token + round/iteration caps, required at startup (A4 / D6).
|
||||||
|
|
@ -54,22 +81,142 @@ class Budget:
|
||||||
raise ValueError(f"max_rounds must be positive, got {self.max_rounds}")
|
raise ValueError(f"max_rounds must be positive, got {self.max_rounds}")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class PortfolioBudget:
|
||||||
|
"""The GLOBAL token cap over a whole portfolio pass, plus the per-run cap it funds (S3.4/F10).
|
||||||
|
|
||||||
|
``max_total_tokens`` bounds everything a pass may spend — and, when a ``PortfolioMeter`` is
|
||||||
|
seeded from persisted spend, everything a SERIES of passes may spend. ``max_tokens_per_run``
|
||||||
|
is the cap each individual run is given. ``min_run_reserve`` (defaulting to the per-run cap)
|
||||||
|
is the remainder a run must be able to claim before it is allowed to START: without it a pass
|
||||||
|
would happily launch a project it can only half pay for, spending real tokens on a run that
|
||||||
|
cannot finish.
|
||||||
|
|
||||||
|
Fail-fast on configurations that cannot mean what they say: a per-run cap above the global one
|
||||||
|
would let a single run cross the pass's own ceiling, and a reserve above the per-run cap would
|
||||||
|
demand more than any run can ever spend, refusing every pass forever.
|
||||||
|
"""
|
||||||
|
|
||||||
|
max_total_tokens: int
|
||||||
|
max_tokens_per_run: int
|
||||||
|
min_run_reserve: int | None = None
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
if self.max_total_tokens <= 0:
|
||||||
|
raise ValueError(f"max_total_tokens must be positive, got {self.max_total_tokens}")
|
||||||
|
if self.max_tokens_per_run <= 0:
|
||||||
|
raise ValueError(f"max_tokens_per_run must be positive, got {self.max_tokens_per_run}")
|
||||||
|
if self.max_tokens_per_run > self.max_total_tokens:
|
||||||
|
raise ValueError(
|
||||||
|
f"max_tokens_per_run ({self.max_tokens_per_run}) exceeds max_total_tokens "
|
||||||
|
f"({self.max_total_tokens}): one run could cross the portfolio cap on its own"
|
||||||
|
)
|
||||||
|
if self.min_run_reserve is not None:
|
||||||
|
if self.min_run_reserve <= 0:
|
||||||
|
raise ValueError(f"min_run_reserve must be positive, got {self.min_run_reserve}")
|
||||||
|
if self.min_run_reserve > self.max_tokens_per_run:
|
||||||
|
raise ValueError(
|
||||||
|
f"min_run_reserve ({self.min_run_reserve}) exceeds max_tokens_per_run "
|
||||||
|
f"({self.max_tokens_per_run}): no run could ever meet it"
|
||||||
|
)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def required_per_run(self) -> int:
|
||||||
|
"""The remainder one run must be able to claim to be allowed to start."""
|
||||||
|
return self.min_run_reserve if self.min_run_reserve is not None else self.max_tokens_per_run
|
||||||
|
|
||||||
|
|
||||||
|
class PortfolioMeter:
|
||||||
|
"""The ONE token ledger for a portfolio pass, shared by every run's ``TokenMeter`` (S3.4).
|
||||||
|
|
||||||
|
``spent`` seeds from an earlier pass (``read_spend``), which is what makes the cap hold ACROSS
|
||||||
|
passes and not merely within one.
|
||||||
|
|
||||||
|
``record`` and ``check`` are deliberately SPLIT rather than fused into one ``charge``. Tokens
|
||||||
|
the provider already billed must reach this ledger even when the charge simultaneously breaks
|
||||||
|
the RUN's own cap and raises there — a fused charge that returned early on the run cap would
|
||||||
|
silently forget that spend and hand the next project a budget that was never really there.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, budget: PortfolioBudget, *, spent: int = 0) -> None:
|
||||||
|
if spent < 0:
|
||||||
|
raise ValueError(f"spent must be non-negative, got {spent}")
|
||||||
|
self.budget = budget
|
||||||
|
self.spent = spent
|
||||||
|
|
||||||
|
@property
|
||||||
|
def required_per_run(self) -> int:
|
||||||
|
return self.budget.required_per_run
|
||||||
|
|
||||||
|
def remaining(self) -> int:
|
||||||
|
"""Tokens left in the global cap (never negative — a crossed cap reads as zero left)."""
|
||||||
|
return max(0, self.budget.max_total_tokens - self.spent)
|
||||||
|
|
||||||
|
def can_fund_run(self, *, reserved: int = 0) -> bool:
|
||||||
|
"""Whether one more run can be started. ``reserved`` is what the caller has already
|
||||||
|
committed to runs it admitted but that have not spent yet (the wave case): every member of
|
||||||
|
a concurrent wave is funded off the same pre-wave remainder, so admitting them without
|
||||||
|
reserving would over-commit the cap by exactly the wave size."""
|
||||||
|
return self.remaining() - reserved >= self.required_per_run
|
||||||
|
|
||||||
|
def record(self, tokens: int) -> int:
|
||||||
|
"""Accumulate ``tokens`` into the global ledger. Never raises — see the class docstring."""
|
||||||
|
self.spent += tokens
|
||||||
|
return self.spent
|
||||||
|
|
||||||
|
def check(self) -> None:
|
||||||
|
"""Raise ``BudgetExceeded`` if the global cap has been crossed."""
|
||||||
|
if self.spent > self.budget.max_total_tokens:
|
||||||
|
raise BudgetExceeded("portfolio_tokens", self.budget.max_total_tokens, self.spent)
|
||||||
|
|
||||||
|
|
||||||
class TokenMeter:
|
class TokenMeter:
|
||||||
"""Accumulates token and round usage against a ``Budget``; raises the moment a cap is
|
"""Accumulates token and round usage against a ``Budget``; raises the moment a cap is
|
||||||
crossed."""
|
crossed. When ``portfolio`` is supplied (S3.4) every charge also lands in the shared portfolio
|
||||||
|
ledger, so the run is bounded by BOTH its own cap and the pass's global one."""
|
||||||
|
|
||||||
def __init__(self, budget: Budget) -> None:
|
def __init__(self, budget: Budget, *, portfolio: PortfolioMeter | None = None) -> None:
|
||||||
self.budget = budget
|
self.budget = budget
|
||||||
self.tokens = 0
|
self.tokens = 0
|
||||||
self.rounds = 0
|
self.rounds = 0
|
||||||
|
self.portfolio = portfolio
|
||||||
|
|
||||||
def charge(self, tokens: int) -> int:
|
def charge(self, tokens: int) -> int:
|
||||||
"""Add ``tokens`` to the running total; raise ``BudgetExceeded`` if over cap."""
|
"""Add ``tokens`` to the running total; raise ``BudgetExceeded`` if over cap.
|
||||||
|
|
||||||
|
Both ledgers are credited BEFORE either cap is tested: the spend happened regardless of
|
||||||
|
which cap it broke, so recording must not depend on the outcome of a check."""
|
||||||
self.tokens += tokens
|
self.tokens += tokens
|
||||||
|
if self.portfolio is not None:
|
||||||
|
self.portfolio.record(tokens)
|
||||||
if self.tokens > self.budget.max_tokens:
|
if self.tokens > self.budget.max_tokens:
|
||||||
raise BudgetExceeded("tokens", self.budget.max_tokens, self.tokens)
|
raise BudgetExceeded("tokens", self.budget.max_tokens, self.tokens)
|
||||||
|
if self.portfolio is not None:
|
||||||
|
self.portfolio.check()
|
||||||
return self.tokens
|
return self.tokens
|
||||||
|
|
||||||
|
def remaining(self) -> int:
|
||||||
|
"""Tokens left under whichever cap BINDS — the run's own, or the portfolio's."""
|
||||||
|
own = self.budget.max_tokens - self.tokens
|
||||||
|
if self.portfolio is None:
|
||||||
|
return own
|
||||||
|
return min(own, self.portfolio.remaining())
|
||||||
|
|
||||||
|
def exhausted(self) -> BudgetExceeded | None:
|
||||||
|
"""The structured error to refuse a call with, or ``None`` while budget remains.
|
||||||
|
|
||||||
|
Exhaustion is ``remaining() <= 0``, not ``< 0``: a chat call that costs zero tokens does
|
||||||
|
not exist, so at exactly-zero the next call can only overspend. Naming WHICH cap binds is
|
||||||
|
the point — a run refused because a sibling drained the pass reads as ``portfolio_tokens``,
|
||||||
|
not as its own overrun."""
|
||||||
|
if self.remaining() > 0:
|
||||||
|
return None
|
||||||
|
if self.portfolio is not None and self.portfolio.remaining() <= 0:
|
||||||
|
return BudgetExceeded(
|
||||||
|
"portfolio_tokens", self.portfolio.budget.max_total_tokens, self.portfolio.spent
|
||||||
|
)
|
||||||
|
return BudgetExceeded("tokens", self.budget.max_tokens, self.tokens)
|
||||||
|
|
||||||
def tick_round(self) -> int:
|
def tick_round(self) -> int:
|
||||||
"""Increment the round counter; raise ``BudgetExceeded`` if over cap."""
|
"""Increment the round counter; raise ``BudgetExceeded`` if over cap."""
|
||||||
self.rounds += 1
|
self.rounds += 1
|
||||||
|
|
@ -80,13 +227,21 @@ class TokenMeter:
|
||||||
|
|
||||||
class BudgetMiddleware(ChatMiddleware):
|
class BudgetMiddleware(ChatMiddleware):
|
||||||
"""Chat middleware that charges a ``TokenMeter`` from each response's real
|
"""Chat middleware that charges a ``TokenMeter`` from each response's real
|
||||||
``UsageDetails`` and short-circuits when the cap is crossed."""
|
``UsageDetails`` and short-circuits when the cap is crossed.
|
||||||
|
|
||||||
|
Two teeth, not one. The PRE-call guard (S3.4) refuses to make a call the budget cannot pay
|
||||||
|
for — that is money not spent. The post-charge check is what it always was: the real usage is
|
||||||
|
only knowable after the response, so a call that crosses the cap can only be caught behind it.
|
||||||
|
The guard does not replace the check; it stops the NEXT call, never the one in flight."""
|
||||||
|
|
||||||
def __init__(self, meter: TokenMeter, *, strict_usage: bool = True) -> None:
|
def __init__(self, meter: TokenMeter, *, strict_usage: bool = True) -> None:
|
||||||
self._meter = meter
|
self._meter = meter
|
||||||
self._strict = strict_usage
|
self._strict = strict_usage
|
||||||
|
|
||||||
async def process(self, context: ChatContext, call_next: Callable[[], Awaitable[None]]) -> None:
|
async def process(self, context: ChatContext, call_next: Callable[[], Awaitable[None]]) -> None:
|
||||||
|
exhausted = self._meter.exhausted()
|
||||||
|
if exhausted is not None:
|
||||||
|
raise exhausted # BEFORE call_next: the call must never be made
|
||||||
await call_next()
|
await call_next()
|
||||||
usage = getattr(context.result, "usage_details", None)
|
usage = getattr(context.result, "usage_details", None)
|
||||||
total = usage.get("total_token_count") if usage is not None else None
|
total = usage.get("total_token_count") if usage is not None else None
|
||||||
|
|
@ -98,3 +253,39 @@ class BudgetMiddleware(ChatMiddleware):
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
self._meter.charge(int(total)) # raises BudgetExceeded if over cap
|
self._meter.charge(int(total)) # raises BudgetExceeded if over cap
|
||||||
|
|
||||||
|
|
||||||
|
def write_spend(path: str | Path, spent: int, *, stamp: str) -> Path:
|
||||||
|
"""Persist a portfolio pass's accumulated ``spent`` tokens as deterministic JSON.
|
||||||
|
|
||||||
|
``stamp`` is a REQUIRED keyword with no wall-clock default (mirroring ``promote_verdict`` and
|
||||||
|
the outbox's ``run_id``): the file must be byte-identical for the same spend, so a diff means
|
||||||
|
the spend changed and nothing else. This is what carries the global cap ACROSS passes — the
|
||||||
|
next pass seeds its ``PortfolioMeter`` from ``read_spend``."""
|
||||||
|
target = Path(path)
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
payload = {"spent_tokens": int(spent), "stamp": stamp}
|
||||||
|
target.write_text(json.dumps(payload, sort_keys=True, indent=2), encoding="utf-8")
|
||||||
|
return target
|
||||||
|
|
||||||
|
|
||||||
|
def read_spend(path: str | Path) -> int:
|
||||||
|
"""Read persisted spend; a MISSING file reads as ``0`` (there was no earlier pass).
|
||||||
|
|
||||||
|
Malformed content RAISES — deliberately unlike the tolerant raw verdict inbox
|
||||||
|
(``load_verdicts_from_dir``). That folder is written out of band by other parties, so skipping
|
||||||
|
junk is correct there; this file is our OWN accounting state, and reading a corrupt one as zero
|
||||||
|
would silently hand back a budget that had already been spent."""
|
||||||
|
target = Path(path)
|
||||||
|
if not target.exists():
|
||||||
|
return 0
|
||||||
|
try:
|
||||||
|
payload = json.loads(target.read_text(encoding="utf-8"))
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise ValueError(f"malformed spend file {str(target)!r}: {exc}") from exc
|
||||||
|
if not isinstance(payload, dict) or not isinstance(payload.get("spent_tokens"), int):
|
||||||
|
raise ValueError(f"spend file {str(target)!r} lacks an integer 'spent_tokens'")
|
||||||
|
spent = int(payload["spent_tokens"])
|
||||||
|
if spent < 0:
|
||||||
|
raise ValueError(f"spend file {str(target)!r} carries negative spent_tokens: {spent}")
|
||||||
|
return spent
|
||||||
|
|
|
||||||
|
|
@ -35,7 +35,13 @@ from agent_framework import BaseChatClient, SessionContext
|
||||||
from pydantic import ValidationError
|
from pydantic import ValidationError
|
||||||
|
|
||||||
from portfolio_optimiser.backends import Profile, get_backend, resolve_model
|
from portfolio_optimiser.backends import Profile, get_backend, resolve_model
|
||||||
from portfolio_optimiser.budget import Budget, BudgetMiddleware, TokenMeter
|
from portfolio_optimiser.budget import (
|
||||||
|
Budget,
|
||||||
|
BudgetMiddleware,
|
||||||
|
BudgetRefused,
|
||||||
|
PortfolioMeter,
|
||||||
|
TokenMeter,
|
||||||
|
)
|
||||||
from portfolio_optimiser.contracts import GoalConfig, GoalContract, load_contracts, load_goal_config
|
from portfolio_optimiser.contracts import GoalConfig, GoalContract, load_contracts, load_goal_config
|
||||||
from portfolio_optimiser.ledger import SavingsLedger
|
from portfolio_optimiser.ledger import SavingsLedger
|
||||||
from portfolio_optimiser.datasource import (
|
from portfolio_optimiser.datasource import (
|
||||||
|
|
@ -141,6 +147,25 @@ class GoalReached:
|
||||||
observed_ore: int
|
observed_ore: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class BudgetStop:
|
||||||
|
"""A GLOBAL token-cap stop signal VALUE (S3.4/F10) — NOT an exception, and NOT a goal.
|
||||||
|
|
||||||
|
Structured like ``GoalReached`` and carried the same way (a ``stop_reason``-shaped value plus a
|
||||||
|
loop ``break``), but it is kept as its OWN field rather than widening ``stop_reason``: the two
|
||||||
|
stops mean opposite things. A goal-stop is success (the savings target was met); this is
|
||||||
|
resource exhaustion (the pass ran out of tokens). Folding them into one field would let a
|
||||||
|
caller read "we stopped" without being able to tell which happened.
|
||||||
|
|
||||||
|
``required_tokens`` is what one more run would have needed; ``remaining_tokens`` is what was
|
||||||
|
actually left. Both are recorded because their DIFFERENCE is the operator's next decision."""
|
||||||
|
|
||||||
|
limit_tokens: int
|
||||||
|
spent_tokens: int
|
||||||
|
remaining_tokens: int
|
||||||
|
required_tokens: int
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class PortfolioResult:
|
class PortfolioResult:
|
||||||
"""The outcome of a sequential fan-out over N projects (SC2).
|
"""The outcome of a sequential fan-out over N projects (SC2).
|
||||||
|
|
@ -150,9 +175,11 @@ class PortfolioResult:
|
||||||
The remaining fields are a thin aggregate over ``runs``: ``validated_count`` /
|
The remaining fields are a thin aggregate over ``runs``: ``validated_count`` /
|
||||||
``rejected_count`` partition the outcomes; ``sum_claimed_saving_nok`` totals the claimed
|
``rejected_count`` partition the outcomes; ``sum_claimed_saving_nok`` totals the claimed
|
||||||
saving of the validated proposals only; ``sum_token_usage`` totals every run's
|
saving of the validated proposals only; ``sum_token_usage`` totals every run's
|
||||||
provenance token usage. ``stopped_early`` / ``stop_reason`` record a Step-8 goal-stop, and
|
provenance token usage. ``stopped_early`` / ``stop_reason`` record a Step-8 goal-stop,
|
||||||
``failures`` records the projects that RAISED (S3.3 collect-and-continue): all three default so
|
``failures`` records the projects that RAISED (S3.3 collect-and-continue), and ``budget_stop``
|
||||||
the frozen aggregate and every existing constructor call are unaffected.
|
records a S3.4 global-token-cap stop (which also sets ``stopped_early``, but is kept apart from
|
||||||
|
``stop_reason`` because exhaustion is not success): all four default so the frozen aggregate and
|
||||||
|
every existing constructor call are unaffected.
|
||||||
|
|
||||||
``runs`` and ``failures`` PARTITION the projects that were actually submitted — a project
|
``runs`` and ``failures`` PARTITION the projects that were actually submitted — a project
|
||||||
appears in exactly one of them, never both, and the counts do not overlap. ``validated_count`` /
|
appears in exactly one of them, never both, and the counts do not overlap. ``validated_count`` /
|
||||||
|
|
@ -168,6 +195,7 @@ class PortfolioResult:
|
||||||
stopped_early: bool = False
|
stopped_early: bool = False
|
||||||
stop_reason: GoalReached | None = None
|
stop_reason: GoalReached | None = None
|
||||||
failures: tuple[RunFailure, ...] = ()
|
failures: tuple[RunFailure, ...] = ()
|
||||||
|
budget_stop: BudgetStop | None = None
|
||||||
|
|
||||||
|
|
||||||
def _authored_texts(result: Any, name: str) -> list[str]:
|
def _authored_texts(result: Any, name: str) -> list[str]:
|
||||||
|
|
@ -647,6 +675,30 @@ def _waves(ids: list[str], k: int) -> list[list[str]]:
|
||||||
return [ids[i : i + k] for i in range(0, len(ids), k)]
|
return [ids[i : i + k] for i in range(0, len(ids), k)]
|
||||||
|
|
||||||
|
|
||||||
|
def _run_meter(
|
||||||
|
meter_factory: Callable[[], TokenMeter] | None,
|
||||||
|
portfolio_meter: PortfolioMeter | None,
|
||||||
|
max_rounds: int,
|
||||||
|
) -> TokenMeter | None:
|
||||||
|
"""The per-project meter ``run_portfolio`` hands to one run: the injected test seam, a meter
|
||||||
|
BOUND to the portfolio ledger, or ``None`` (letting ``run_project`` build its own, unchanged).
|
||||||
|
|
||||||
|
The bound meter mirrors ``run_project``'s own construction (``max(max_rounds * 4, 4)``) so the
|
||||||
|
only difference the portfolio cap introduces is the token ceiling and the shared ledger — the
|
||||||
|
round budget is not quietly redefined along the way."""
|
||||||
|
if meter_factory is not None:
|
||||||
|
return meter_factory()
|
||||||
|
if portfolio_meter is None:
|
||||||
|
return None
|
||||||
|
return TokenMeter(
|
||||||
|
Budget(
|
||||||
|
max_tokens=portfolio_meter.budget.max_tokens_per_run,
|
||||||
|
max_rounds=max(max_rounds * 4, 4),
|
||||||
|
),
|
||||||
|
portfolio=portfolio_meter,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
async def run_portfolio(
|
async def run_portfolio(
|
||||||
project_ids: Sequence[str] | None = None,
|
project_ids: Sequence[str] | None = None,
|
||||||
profile: Profile | str = Profile.LOCAL,
|
profile: Profile | str = Profile.LOCAL,
|
||||||
|
|
@ -661,6 +713,7 @@ async def run_portfolio(
|
||||||
top_k: int = 3,
|
top_k: int = 3,
|
||||||
concurrency: int = 1,
|
concurrency: int = 1,
|
||||||
meter_factory: Callable[[], TokenMeter] | None = None,
|
meter_factory: Callable[[], TokenMeter] | None = None,
|
||||||
|
portfolio_meter: PortfolioMeter | None = None,
|
||||||
semantic_retrieval: bool = False,
|
semantic_retrieval: bool = False,
|
||||||
embedder: Embedder | None = None,
|
embedder: Embedder | None = None,
|
||||||
) -> PortfolioResult:
|
) -> PortfolioResult:
|
||||||
|
|
@ -727,12 +780,46 @@ async def run_portfolio(
|
||||||
property of the design, so it is pinned on a bundle-backed pair where the fold does fire
|
property of the design, so it is pinned on a bundle-backed pair where the fold does fire
|
||||||
(``test_intra_wave_visibility_is_the_documented_semantic_difference``): same fixture, same
|
(``test_intra_wave_visibility_is_the_documented_semantic_difference``): same fixture, same
|
||||||
sentinel, only the wave boundary moves. Store content stays identical across ``k`` — the
|
sentinel, only the wave boundary moves. Store content stays identical across ``k`` — the
|
||||||
difference is confined to what each project READ, never to what the pass produced or persisted."""
|
difference is confined to what each project READ, never to what the pass produced or persisted.
|
||||||
|
|
||||||
|
``portfolio_meter`` (S3.4, F10) installs the GLOBAL token cap. Without it nothing changes: each
|
||||||
|
run is bounded only by its own ``max_tokens``, so N projects can cost N times that with no
|
||||||
|
ceiling over the pass. With it, one ``PortfolioMeter`` is shared by every run (each run's meter
|
||||||
|
is BOUND to it, which is why ``meter_factory`` — whose meters are unbound — is refused
|
||||||
|
alongside it: accepting both would run a pass that looks capped and is not), and the cap is
|
||||||
|
enforced in three places that are deliberately different:
|
||||||
|
|
||||||
|
- **At startup**, a remainder that cannot fund one run raises ``BudgetRefused`` — a pass that
|
||||||
|
can afford zero projects is a caller mistake, not a result.
|
||||||
|
- **At wave assembly**, a project that cannot be funded is NEVER STARTED, and the pass stops
|
||||||
|
with ``budget_stop`` + ``stopped_early``, every completed run preserved. Never-started is the
|
||||||
|
property that matters: an unfunded project that is merely interrupted mid-run has already
|
||||||
|
cost model round-trips. Because every member of a wave is checked against the SAME pre-wave
|
||||||
|
remainder, admission RESERVES each member's requirement as it goes — otherwise a wave of k
|
||||||
|
would over-commit the cap by up to k runs. This makes membership identical at every
|
||||||
|
``concurrency``, matching the goal-stop's guarantee above.
|
||||||
|
- **Before each chat call** (``BudgetMiddleware``'s pre-call guard), a call the remainder cannot
|
||||||
|
pay for is refused rather than made. This is what bounds a run that was funded at admission
|
||||||
|
but whose siblings drained the pass while it was in flight; such a run surfaces as a
|
||||||
|
``RunFailure``, not as overspend.
|
||||||
|
|
||||||
|
Spend is carried ACROSS passes by seeding the meter from ``budget.read_spend`` and writing
|
||||||
|
``budget.write_spend`` afterwards. Those are the caller's calls, not this function's — the pass
|
||||||
|
reads its ledger, it does not own the file (mirroring the Steg-7 role split)."""
|
||||||
if concurrency < 1:
|
if concurrency < 1:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"concurrency must be >= 1, got {concurrency}: a non-positive wave size would run no "
|
f"concurrency must be >= 1, got {concurrency}: a non-positive wave size would run no "
|
||||||
"projects at all and read as an empty portfolio. Pass 1 for the sequential pass."
|
"projects at all and read as an empty portfolio. Pass 1 for the sequential pass."
|
||||||
)
|
)
|
||||||
|
if portfolio_meter is not None and meter_factory is not None:
|
||||||
|
raise ValueError(
|
||||||
|
"portfolio_meter and meter_factory are mutually exclusive: a meter_factory meter is "
|
||||||
|
"not bound to the portfolio ledger, so the global cap would be silently unenforced"
|
||||||
|
)
|
||||||
|
# Startup refusal (fail-fast, before anything loads): a pass that cannot fund its first run
|
||||||
|
# must not begin. Raised, not returned — an empty PortfolioResult would read as "nothing to do".
|
||||||
|
if portfolio_meter is not None and not portfolio_meter.can_fund_run():
|
||||||
|
raise BudgetRefused(portfolio_meter.remaining(), portfolio_meter.required_per_run)
|
||||||
projects = {p.id: p for p in load_reference_projects()}
|
projects = {p.id: p for p in load_reference_projects()}
|
||||||
ids = list(project_ids) if project_ids is not None else list(projects)
|
ids = list(project_ids) if project_ids is not None else list(projects)
|
||||||
store = store if store is not None else VerdictStore(verdicts=[])
|
store = store if store is not None else VerdictStore(verdicts=[])
|
||||||
|
|
@ -747,8 +834,13 @@ async def run_portfolio(
|
||||||
failures: list[RunFailure] = []
|
failures: list[RunFailure] = []
|
||||||
stopped_early = False
|
stopped_early = False
|
||||||
stop_reason: GoalReached | None = None
|
stop_reason: GoalReached | None = None
|
||||||
|
budget_stop: BudgetStop | None = None
|
||||||
for wave_ids in _waves(ids, concurrency):
|
for wave_ids in _waves(ids, concurrency):
|
||||||
members: list[str] = []
|
members: list[str] = []
|
||||||
|
# Tokens committed to members already admitted to THIS wave but that have not spent yet.
|
||||||
|
# Every member reads the same pre-wave remainder, so without this a wave of k would admit
|
||||||
|
# k projects off one project's worth of budget.
|
||||||
|
reserved = 0
|
||||||
for pid in wave_ids:
|
for pid in wave_ids:
|
||||||
if pid not in projects:
|
if pid not in projects:
|
||||||
raise ValueError(f"unknown project_id: {pid!r}")
|
raise ValueError(f"unknown project_id: {pid!r}")
|
||||||
|
|
@ -777,6 +869,21 @@ async def run_portfolio(
|
||||||
if per_project_goal.mode == "hard":
|
if per_project_goal.mode == "hard":
|
||||||
continue # skip THIS pid; the rest of the pass proceeds
|
continue # skip THIS pid; the rest of the pass proceeds
|
||||||
|
|
||||||
|
# S3.4 funding check, placed AFTER the goal checks: a reached goal is success and owns
|
||||||
|
# the stop when both apply, and a pid a hard per-project goal already skipped costs no
|
||||||
|
# budget, so it must not consume a reservation.
|
||||||
|
if portfolio_meter is not None and not portfolio_meter.can_fund_run(reserved=reserved):
|
||||||
|
stopped_early = True
|
||||||
|
budget_stop = BudgetStop(
|
||||||
|
limit_tokens=portfolio_meter.budget.max_total_tokens,
|
||||||
|
spent_tokens=portfolio_meter.spent,
|
||||||
|
remaining_tokens=portfolio_meter.remaining(),
|
||||||
|
required_tokens=portfolio_meter.required_per_run,
|
||||||
|
)
|
||||||
|
break
|
||||||
|
|
||||||
|
if portfolio_meter is not None:
|
||||||
|
reserved += portfolio_meter.required_per_run
|
||||||
members.append(pid)
|
members.append(pid)
|
||||||
|
|
||||||
# Every project in the wave reads the SAME wave-start state and writes only its own copy,
|
# Every project in the wave reads the SAME wave-start state and writes only its own copy,
|
||||||
|
|
@ -805,7 +912,7 @@ async def run_portfolio(
|
||||||
top_k=top_k,
|
top_k=top_k,
|
||||||
semantic_retrieval=semantic_retrieval,
|
semantic_retrieval=semantic_retrieval,
|
||||||
embedder=embedder,
|
embedder=embedder,
|
||||||
meter=meter_factory() if meter_factory is not None else None,
|
meter=_run_meter(meter_factory, portfolio_meter, max_rounds),
|
||||||
)
|
)
|
||||||
for pid, snapshot in snapshots
|
for pid, snapshot in snapshots
|
||||||
),
|
),
|
||||||
|
|
@ -836,12 +943,13 @@ async def run_portfolio(
|
||||||
break
|
break
|
||||||
|
|
||||||
base = _aggregate(tuple(runs), store)
|
base = _aggregate(tuple(runs), store)
|
||||||
if stopped_early or stop_reason is not None or failures:
|
if stopped_early or stop_reason is not None or failures or budget_stop is not None:
|
||||||
return replace(
|
return replace(
|
||||||
base,
|
base,
|
||||||
stopped_early=stopped_early,
|
stopped_early=stopped_early,
|
||||||
stop_reason=stop_reason,
|
stop_reason=stop_reason,
|
||||||
failures=tuple(failures),
|
failures=tuple(failures),
|
||||||
|
budget_stop=budget_stop,
|
||||||
)
|
)
|
||||||
return base
|
return base
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -15,6 +15,8 @@ from portfolio_optimiser.budget import (
|
||||||
Budget,
|
Budget,
|
||||||
BudgetExceeded,
|
BudgetExceeded,
|
||||||
BudgetMiddleware,
|
BudgetMiddleware,
|
||||||
|
PortfolioBudget,
|
||||||
|
PortfolioMeter,
|
||||||
TokenMeter,
|
TokenMeter,
|
||||||
UsageUnavailable,
|
UsageUnavailable,
|
||||||
)
|
)
|
||||||
|
|
@ -85,6 +87,90 @@ async def test_budget_middleware_fires_on_real_agent_chat(make_client_factory) -
|
||||||
assert meter.tokens == 8 # charged from the synthetic UsageDetails via the middleware
|
assert meter.tokens == 8 # charged from the synthetic UsageDetails via the middleware
|
||||||
|
|
||||||
|
|
||||||
|
async def test_pre_call_guard_does_not_await_call_next_when_exhausted() -> None:
|
||||||
|
"""S3.4 unit arm of the pre-call guard: the middleware refuses BEFORE ``call_next``. Counting
|
||||||
|
the awaits is the whole measurement — the post-charge arm raises either way."""
|
||||||
|
calls = {"n": 0}
|
||||||
|
|
||||||
|
async def _counted() -> None:
|
||||||
|
calls["n"] += 1
|
||||||
|
|
||||||
|
meter = TokenMeter(Budget(max_tokens=10, max_rounds=10))
|
||||||
|
meter.charge(10) # exactly at the cap -> nothing left to fund a call with
|
||||||
|
mw = BudgetMiddleware(meter)
|
||||||
|
with pytest.raises(BudgetExceeded):
|
||||||
|
await mw.process(_Ctx(_resp(5)), _counted) # type: ignore[arg-type]
|
||||||
|
assert calls["n"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
async def test_below_the_cap_still_calls_through() -> None:
|
||||||
|
"""Control for the guard: with budget left, the call goes through exactly as before — the
|
||||||
|
guard must gate exhaustion, not traffic."""
|
||||||
|
calls = {"n": 0}
|
||||||
|
|
||||||
|
async def _counted() -> None:
|
||||||
|
calls["n"] += 1
|
||||||
|
|
||||||
|
meter = TokenMeter(Budget(max_tokens=10, max_rounds=10))
|
||||||
|
mw = BudgetMiddleware(meter)
|
||||||
|
await mw.process(_Ctx(_resp(4)), _counted) # type: ignore[arg-type]
|
||||||
|
assert calls["n"] == 1 and meter.tokens == 4 and meter.remaining() == 6
|
||||||
|
|
||||||
|
|
||||||
|
def test_portfolio_budget_rejects_unusable_configs() -> None:
|
||||||
|
"""Fail-fast at construction (mirroring ``Budget``): non-positive caps, a per-run cap larger
|
||||||
|
than the global one (one run could then cross the pass's own ceiling), and a reserve larger
|
||||||
|
than a run can ever spend (which would refuse every pass forever)."""
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
PortfolioBudget(max_total_tokens=0, max_tokens_per_run=10)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=0)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=200)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=50, min_run_reserve=60)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
PortfolioMeter(PortfolioBudget(max_total_tokens=100, max_tokens_per_run=50), spent=-1)
|
||||||
|
|
||||||
|
|
||||||
|
def test_portfolio_meter_accumulates_and_crosses() -> None:
|
||||||
|
budget = PortfolioBudget(max_total_tokens=100, max_tokens_per_run=50)
|
||||||
|
meter = PortfolioMeter(budget, spent=40)
|
||||||
|
assert meter.remaining() == 60 and meter.required_per_run == 50
|
||||||
|
meter.record(60)
|
||||||
|
meter.check() # exactly at the cap is still within it (mirrors TokenMeter's `>` boundary)
|
||||||
|
meter.record(1)
|
||||||
|
with pytest.raises(BudgetExceeded) as exc:
|
||||||
|
meter.check()
|
||||||
|
assert exc.value.kind == "portfolio_tokens"
|
||||||
|
assert exc.value.limit == 100 and exc.value.observed == 101
|
||||||
|
|
||||||
|
|
||||||
|
async def test_run_meter_bound_to_portfolio_charges_both_ledgers() -> None:
|
||||||
|
"""A bound run meter charges the global ledger on every call, and ``remaining`` reads whichever
|
||||||
|
cap binds — that is what lets one project's spend refuse another's next call."""
|
||||||
|
portfolio = PortfolioMeter(PortfolioBudget(max_total_tokens=100, max_tokens_per_run=80))
|
||||||
|
meter = TokenMeter(Budget(max_tokens=80, max_rounds=10), portfolio=portfolio)
|
||||||
|
mw = BudgetMiddleware(meter)
|
||||||
|
await mw.process(_Ctx(_resp(30)), _noop) # type: ignore[arg-type]
|
||||||
|
assert meter.tokens == 30 and portfolio.spent == 30
|
||||||
|
assert meter.remaining() == 50 # run has 50 left, portfolio 70 -> the RUN binds
|
||||||
|
await mw.process(_Ctx(_resp(45)), _noop) # type: ignore[arg-type]
|
||||||
|
assert meter.remaining() == 5 # run 5, portfolio 25 -> still the run
|
||||||
|
|
||||||
|
|
||||||
|
async def test_spend_is_ledgered_even_when_the_per_run_cap_raises() -> None:
|
||||||
|
"""Order matters: tokens the provider already billed must reach the global ledger even though
|
||||||
|
the RUN's own cap raises on the same charge. Checking the run cap first and returning early
|
||||||
|
would lose that spend and hand the next project a budget that was never really there."""
|
||||||
|
portfolio = PortfolioMeter(PortfolioBudget(max_total_tokens=1000, max_tokens_per_run=50))
|
||||||
|
meter = TokenMeter(Budget(max_tokens=50, max_rounds=10), portfolio=portfolio)
|
||||||
|
with pytest.raises(BudgetExceeded) as exc:
|
||||||
|
meter.charge(60)
|
||||||
|
assert exc.value.kind == "tokens" # the RUN's cap is the one that broke
|
||||||
|
assert portfolio.spent == 60 # ...and the global ledger still saw the spend
|
||||||
|
|
||||||
|
|
||||||
def test_no_word_count_token_proxy_in_src() -> None:
|
def test_no_word_count_token_proxy_in_src() -> None:
|
||||||
# The meter is fed from real UsageDetails, never a len(text.split()) word-count proxy
|
# The meter is fed from real UsageDetails, never a len(text.split()) word-count proxy
|
||||||
# (research 03 Rec 3 — the Fase 1 _word_tokens anti-pattern is retired). NOTE: a bare
|
# (research 03 Rec 3 — the Fase 1 _word_tokens anti-pattern is retired). NOTE: a bare
|
||||||
|
|
|
||||||
302
tests/test_portfolio_budget_loadbearing.py
Normal file
302
tests/test_portfolio_budget_loadbearing.py
Normal file
|
|
@ -0,0 +1,302 @@
|
||||||
|
"""S3.4 load-bearing — the GLOBAL token cap across a portfolio pass (F10).
|
||||||
|
|
||||||
|
Two properties the session plan names as RED conditions, plus the wave-size and control arms
|
||||||
|
that keep them honest:
|
||||||
|
|
||||||
|
1. **The global cap stops the pass mid-way, with completed runs preserved.** Detach the
|
||||||
|
wave-assembly funding check and the third project is STARTED anyway — it then dies on the
|
||||||
|
meter mid-run, so the pass reports a failure instead of a structured stop. The property is
|
||||||
|
never-started, not merely never-overspent: a project that cannot be funded must cost zero
|
||||||
|
model round-trips.
|
||||||
|
2. **The pre-call guard refuses the call BEFORE it is made.** The client double counts its own
|
||||||
|
invocations, so "the call never happened" is measured, not inferred. Today's post-charge
|
||||||
|
middleware would let the call through and only then raise — which is exactly the money the
|
||||||
|
guard exists to not spend.
|
||||||
|
|
||||||
|
Token arithmetic is MEASURED, not guessed: every project in the reference portfolio spends
|
||||||
|
4 chat calls x ``tokens`` per reply. At ``tokens=8`` that is a flat 32 tokens per run (probed
|
||||||
|
against ``provenance.token_usage`` before these tests were written), which is what makes the
|
||||||
|
80/64/32 budget below land the stop between run 2 and run 3.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from agent_framework import Agent
|
||||||
|
|
||||||
|
from portfolio_optimiser.budget import (
|
||||||
|
Budget,
|
||||||
|
BudgetExceeded,
|
||||||
|
BudgetMiddleware,
|
||||||
|
BudgetRefused,
|
||||||
|
PortfolioBudget,
|
||||||
|
PortfolioMeter,
|
||||||
|
TokenMeter,
|
||||||
|
read_spend,
|
||||||
|
write_spend,
|
||||||
|
)
|
||||||
|
from portfolio_optimiser.run import run_portfolio
|
||||||
|
|
||||||
|
_PORTFOLIO_IDS = ["FV42-GSV-E1", "RV13-RAS-TP", "BRU-LAKS-REHAB"]
|
||||||
|
|
||||||
|
# Per-project replies (the tested constants of tests/test_portfolio.py, unchanged): all three
|
||||||
|
# validate, and their claimed savings are DISTINCT — which is how a run is keyed back to its
|
||||||
|
# project here, since ``RunResult`` carries no project id. 200000 + 130000 = the first two.
|
||||||
|
REPLIES = {
|
||||||
|
"FV42-GSV-E1": (
|
||||||
|
'{"measure":"Reduce scope","affected_items":['
|
||||||
|
'{"code":"05.2","quantity":4300,"unit_cost":215},'
|
||||||
|
'{"code":"03.1","quantity":1800,"unit_cost":310}],"claimed_saving_nok":200000}'
|
||||||
|
),
|
||||||
|
"RV13-RAS-TP": (
|
||||||
|
'{"measure":"Material substitution","affected_items":['
|
||||||
|
'{"code":"88.2","quantity":180,"unit_cost":4200}],"claimed_saving_nok":130000}'
|
||||||
|
),
|
||||||
|
"BRU-LAKS-REHAB": (
|
||||||
|
'{"measure":"Reduce scope","affected_items":['
|
||||||
|
'{"code":"05.2","quantity":4300,"unit_cost":215},'
|
||||||
|
'{"code":"07.4","quantity":2400,"unit_cost":690}],"claimed_saving_nok":210000}'
|
||||||
|
),
|
||||||
|
}
|
||||||
|
_FIRST_TWO_SAVING = 330000
|
||||||
|
|
||||||
|
# Measured: 32 tokens per run at tokens=8. 80 total funds two runs (32 + 32 = 64) and leaves 16,
|
||||||
|
# which is below the 32-token reserve one more run requires -> the third is never started.
|
||||||
|
_TOTAL = 80
|
||||||
|
_PER_RUN = 64
|
||||||
|
_RESERVE = 32
|
||||||
|
_PER_RUN_SPEND = 32
|
||||||
|
|
||||||
|
|
||||||
|
def _budget() -> PortfolioBudget:
|
||||||
|
return PortfolioBudget(
|
||||||
|
max_total_tokens=_TOTAL, max_tokens_per_run=_PER_RUN, min_run_reserve=_RESERVE
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_global_cap_stops_the_pass_mid_way_with_completed_runs_preserved(
|
||||||
|
make_portfolio_client_factory, fresh_store
|
||||||
|
) -> None:
|
||||||
|
"""RED 1: the pass stops the moment the REMAINING global budget cannot fund another run, and
|
||||||
|
every completed run survives the stop.
|
||||||
|
|
||||||
|
Detach the wave-assembly funding check and this goes red three ways at once: project 3 is
|
||||||
|
started (so it lands in ``failures`` when its charge crosses the cap), ``stopped_early`` stays
|
||||||
|
False, and ``budget_stop`` stays None. The strongest of the three is ``failures == ()``: an
|
||||||
|
unfunded project that is merely *interrupted* has already cost model round-trips, which is the
|
||||||
|
precise thing a global cap is for."""
|
||||||
|
meter = PortfolioMeter(_budget())
|
||||||
|
result = await run_portfolio(
|
||||||
|
_PORTFOLIO_IDS,
|
||||||
|
"local",
|
||||||
|
store=fresh_store,
|
||||||
|
client_factory=make_portfolio_client_factory(REPLIES, tokens=8),
|
||||||
|
portfolio_meter=meter,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.stopped_early is True
|
||||||
|
assert result.budget_stop is not None
|
||||||
|
assert result.budget_stop.limit_tokens == _TOTAL
|
||||||
|
assert result.budget_stop.spent_tokens == 2 * _PER_RUN_SPEND
|
||||||
|
assert result.budget_stop.remaining_tokens == _TOTAL - 2 * _PER_RUN_SPEND
|
||||||
|
assert result.budget_stop.required_tokens == _RESERVE
|
||||||
|
|
||||||
|
# Completed runs preserved, and the unfunded project NEVER STARTED: absent from BOTH sides of
|
||||||
|
# the runs/failures partition, not merely absent from runs. A project that had been started and
|
||||||
|
# then killed by the meter would land in ``failures`` — which is what makes ``== ()`` the
|
||||||
|
# sharpest assertion here. The claimed-saving sum names WHICH two ran (the replies are distinct).
|
||||||
|
assert len(result.runs) == 2
|
||||||
|
assert result.failures == ()
|
||||||
|
assert result.validated_count == 2
|
||||||
|
assert result.sum_claimed_saving_nok == _FIRST_TWO_SAVING
|
||||||
|
|
||||||
|
# The cap itself held: the pass spent what two runs cost and no more.
|
||||||
|
assert meter.spent == 2 * _PER_RUN_SPEND
|
||||||
|
assert meter.spent <= _TOTAL
|
||||||
|
assert meter.spent == result.sum_token_usage
|
||||||
|
|
||||||
|
|
||||||
|
async def test_generous_budget_runs_the_whole_pass(
|
||||||
|
make_portfolio_client_factory, fresh_store
|
||||||
|
) -> None:
|
||||||
|
"""Control: the stop above is CAUSED by the cap, not by wiring a meter in at all. With a global
|
||||||
|
budget that funds every project the pass completes untouched — a gate that can only fire proves
|
||||||
|
nothing about what fires it."""
|
||||||
|
meter = PortfolioMeter(PortfolioBudget(max_total_tokens=10_000, max_tokens_per_run=_PER_RUN))
|
||||||
|
result = await run_portfolio(
|
||||||
|
_PORTFOLIO_IDS,
|
||||||
|
"local",
|
||||||
|
store=fresh_store,
|
||||||
|
client_factory=make_portfolio_client_factory({}, tokens=8),
|
||||||
|
portfolio_meter=meter,
|
||||||
|
)
|
||||||
|
assert result.stopped_early is False
|
||||||
|
assert result.budget_stop is None
|
||||||
|
assert len(result.runs) == 3
|
||||||
|
assert meter.spent == 3 * _PER_RUN_SPEND
|
||||||
|
|
||||||
|
|
||||||
|
async def test_global_cap_holds_at_wave_size_three(
|
||||||
|
make_portfolio_client_factory, fresh_store
|
||||||
|
) -> None:
|
||||||
|
"""The cap is a property of the PASS, not of the sequential schedule: at ``concurrency=3`` the
|
||||||
|
whole portfolio is assembled into ONE wave, and every member's funding check reads the same
|
||||||
|
pre-wave ``remaining``. Admitting all three off that shared reading would overspend by design.
|
||||||
|
The wave therefore RESERVES each admitted member's requirement as it admits it, so the same 80
|
||||||
|
tokens fund the same two projects at k=3 as at k=1."""
|
||||||
|
meter = PortfolioMeter(_budget())
|
||||||
|
result = await run_portfolio(
|
||||||
|
_PORTFOLIO_IDS,
|
||||||
|
"local",
|
||||||
|
store=fresh_store,
|
||||||
|
client_factory=make_portfolio_client_factory({}, tokens=8),
|
||||||
|
concurrency=3,
|
||||||
|
portfolio_meter=meter,
|
||||||
|
)
|
||||||
|
assert len(result.runs) == 2
|
||||||
|
assert result.failures == ()
|
||||||
|
assert result.stopped_early is True
|
||||||
|
assert meter.spent <= _TOTAL
|
||||||
|
|
||||||
|
|
||||||
|
async def test_pre_call_guard_refuses_the_call_before_it_is_made(make_client_factory) -> None:
|
||||||
|
"""RED 2: with the budget exhausted, the chat call is REFUSED — the client is never invoked.
|
||||||
|
|
||||||
|
``call_count`` on the scripted client is the measurement: post-charge middleware alone would
|
||||||
|
make the call, read its usage, and only then raise, so ``pytest.raises`` would pass either way.
|
||||||
|
Only the counter separates 'refused' from 'made and then regretted'."""
|
||||||
|
client = make_client_factory("ok", tokens=8)("proposer")
|
||||||
|
meter = TokenMeter(Budget(max_tokens=10, max_rounds=10))
|
||||||
|
meter.charge(10) # exactly at the cap: within budget, but nothing left to fund a call with
|
||||||
|
assert meter.remaining() == 0
|
||||||
|
|
||||||
|
agent = Agent(
|
||||||
|
client, "propose a measure", name="proposer", middleware=[BudgetMiddleware(meter)]
|
||||||
|
)
|
||||||
|
with pytest.raises(BudgetExceeded) as exc:
|
||||||
|
await agent.run("hi")
|
||||||
|
assert exc.value.kind == "tokens"
|
||||||
|
assert client.call_count == 0, "the call must NEVER be made once the budget is exhausted"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_pre_call_guard_fires_on_the_GLOBAL_remainder_too(make_client_factory) -> None:
|
||||||
|
"""The guard reads whichever cap binds. Here the run's OWN budget is untouched and only the
|
||||||
|
portfolio remainder is exhausted — a run that starts funded can still be refused mid-flight
|
||||||
|
because a sibling spent the rest of the global budget."""
|
||||||
|
client = make_client_factory("ok", tokens=8)("proposer")
|
||||||
|
portfolio = PortfolioMeter(
|
||||||
|
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=100), spent=100
|
||||||
|
)
|
||||||
|
meter = TokenMeter(Budget(max_tokens=100, max_rounds=10), portfolio=portfolio)
|
||||||
|
assert meter.tokens == 0 # the RUN has spent nothing
|
||||||
|
assert meter.remaining() == 0 # ...but the portfolio has nothing left
|
||||||
|
|
||||||
|
agent = Agent(
|
||||||
|
client, "propose a measure", name="proposer", middleware=[BudgetMiddleware(meter)]
|
||||||
|
)
|
||||||
|
with pytest.raises(BudgetExceeded) as exc:
|
||||||
|
await agent.run("hi")
|
||||||
|
assert exc.value.kind == "portfolio_tokens"
|
||||||
|
assert client.call_count == 0
|
||||||
|
|
||||||
|
|
||||||
|
async def test_startup_refusal_when_the_remainder_cannot_fund_one_run(
|
||||||
|
make_portfolio_client_factory, fresh_store
|
||||||
|
) -> None:
|
||||||
|
"""Fail-fast at startup (never a silently truncated pass): a meter whose remainder is already
|
||||||
|
below one run's requirement refuses BEFORE anything runs. This is the across-passes arm — the
|
||||||
|
remainder here is what an earlier pass left behind."""
|
||||||
|
meter = PortfolioMeter(_budget(), spent=_TOTAL - (_RESERVE - 1))
|
||||||
|
with pytest.raises(BudgetRefused) as exc:
|
||||||
|
await run_portfolio(
|
||||||
|
_PORTFOLIO_IDS,
|
||||||
|
"local",
|
||||||
|
store=fresh_store,
|
||||||
|
client_factory=make_portfolio_client_factory({}, tokens=8),
|
||||||
|
portfolio_meter=meter,
|
||||||
|
)
|
||||||
|
assert exc.value.remaining == _RESERVE - 1
|
||||||
|
assert exc.value.required == _RESERVE
|
||||||
|
assert meter.spent == _TOTAL - (_RESERVE - 1) # nothing ran, nothing charged
|
||||||
|
|
||||||
|
|
||||||
|
async def test_spend_persists_across_passes(
|
||||||
|
make_portfolio_client_factory, fresh_store, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""The cap holds ACROSS passes, which is what the persisted spend buys: pass A writes its
|
||||||
|
spend, pass B is seeded from that file and stops one run earlier than it otherwise would."""
|
||||||
|
spend_file = tmp_path / "spend.json"
|
||||||
|
assert read_spend(spend_file) == 0 # no prior pass -> no prior spend
|
||||||
|
|
||||||
|
budget = PortfolioBudget(
|
||||||
|
max_total_tokens=3 * _PER_RUN_SPEND, max_tokens_per_run=_PER_RUN, min_run_reserve=_RESERVE
|
||||||
|
)
|
||||||
|
meter_a = PortfolioMeter(budget)
|
||||||
|
result_a = await run_portfolio(
|
||||||
|
_PORTFOLIO_IDS[:1],
|
||||||
|
"local",
|
||||||
|
store=fresh_store,
|
||||||
|
client_factory=make_portfolio_client_factory({}, tokens=8),
|
||||||
|
portfolio_meter=meter_a,
|
||||||
|
)
|
||||||
|
assert len(result_a.runs) == 1
|
||||||
|
write_spend(spend_file, meter_a.spent, stamp="pass-a")
|
||||||
|
|
||||||
|
meter_b = PortfolioMeter(budget, spent=read_spend(spend_file))
|
||||||
|
assert meter_b.spent == _PER_RUN_SPEND # pass B starts where pass A stopped
|
||||||
|
result_b = await run_portfolio(
|
||||||
|
_PORTFOLIO_IDS,
|
||||||
|
"local",
|
||||||
|
store=fresh_store,
|
||||||
|
client_factory=make_portfolio_client_factory({}, tokens=8),
|
||||||
|
portfolio_meter=meter_b,
|
||||||
|
)
|
||||||
|
# 96 total funds three runs; one is already spent, so pass B affords two and stops before the
|
||||||
|
# third — the earlier pass's spend is what moved the stop.
|
||||||
|
assert len(result_b.runs) == 2
|
||||||
|
assert result_b.stopped_early is True
|
||||||
|
assert meter_b.spent == 3 * _PER_RUN_SPEND
|
||||||
|
|
||||||
|
|
||||||
|
def test_spend_file_is_deterministic_and_stamped(tmp_path: Path) -> None:
|
||||||
|
"""Byte-determinism with an EXPLICIT stamp (mirroring ``promote_verdict``): no wall-clock, so
|
||||||
|
two writes of the same spend are byte-identical and a diff means the spend changed."""
|
||||||
|
a, b = tmp_path / "a.json", tmp_path / "b.json"
|
||||||
|
write_spend(a, 1234, stamp="run-7")
|
||||||
|
write_spend(b, 1234, stamp="run-7")
|
||||||
|
assert a.read_bytes() == b.read_bytes()
|
||||||
|
assert json.loads(a.read_text(encoding="utf-8")) == {"spent_tokens": 1234, "stamp": "run-7"}
|
||||||
|
assert read_spend(a) == 1234
|
||||||
|
|
||||||
|
|
||||||
|
def test_spend_file_malformed_fails_fast(tmp_path: Path) -> None:
|
||||||
|
"""The spend file is OUR OWN accounting state, not the tolerant raw inbox layer: a corrupt file
|
||||||
|
raises rather than silently reading as zero spend, which would hand back the whole budget."""
|
||||||
|
bad = tmp_path / "bad.json"
|
||||||
|
bad.write_text("{not json", encoding="utf-8")
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
read_spend(bad)
|
||||||
|
negative = tmp_path / "neg.json"
|
||||||
|
negative.write_text('{"spent_tokens": -5, "stamp": "x"}', encoding="utf-8")
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
read_spend(negative)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_meter_factory_and_portfolio_meter_are_mutually_exclusive(
|
||||||
|
make_portfolio_client_factory, fresh_store
|
||||||
|
) -> None:
|
||||||
|
"""Fail-fast on the one wiring that would silently disable the global cap: a ``meter_factory``
|
||||||
|
meter is not bound to the portfolio ledger, so accepting both would run a pass that LOOKS
|
||||||
|
capped and is not."""
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
await run_portfolio(
|
||||||
|
_PORTFOLIO_IDS,
|
||||||
|
"local",
|
||||||
|
store=fresh_store,
|
||||||
|
client_factory=make_portfolio_client_factory({}, tokens=8),
|
||||||
|
meter_factory=lambda: TokenMeter(Budget(max_tokens=1000, max_rounds=10)),
|
||||||
|
portfolio_meter=PortfolioMeter(_budget()),
|
||||||
|
)
|
||||||
Loading…
Add table
Add a link
Reference in a new issue