test(budget): defend the third field of the stop event — observed (kø-y)

BudgetExceeded carries kind/limit/observed as ONE structured stop event, but only
`observed` was undefended. Measured against the whole suite before writing anything:
four of five raise sites (TokenMeter.charge, tick_round, and BOTH arms of exhausted())
could report any value at all without a single one of 621 tests noticing. Only
PortfolioMeter.check was covered.

What hid it: spikes/_harness.py carries its OWN copy of BudgetExceeded/TokenMeter, so
the spike suite's `observed` assert never touched the shipped module — the production
tick_round had no direct test whatsoever.

exhausted() is the only site that CHOOSES a ledger (the S3.4 pre-call guard), so a
refusal naming portfolio_tokens while reporting the run's own spend would misdirect
every reader of it. Both arms are pinned with observed != limit on purpose: at
exactly-exhausted the two coincide, and a test written there would pass on an
implementation that echoed the cap back as the spend.

No defect in the values themselves (unlike kø-x and kø-p) — the triple was coherent at
all five sites; the gap was purely coverage.

Load-bearing MEASURED: nine mutations, all red — five observed mutations (including the
control) and four echo mutations. 621 -> 623 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LW749xcXQmVEgdipB6KNm4
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 20:46:36 +02:00
commit 3748011d35
2 changed files with 60 additions and 0 deletions

View file

@ -181,6 +181,20 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
Load-bearing MÅLT (`tests/test_portfolio_budget_loadbearing.py` + `tests/test_budget.py`), seks
mutasjoner alle røde: detach wave-sjekken · detach pre-call-guarden · detach bølge-reservasjonen ·
sjekk run-taket før global kreditering · detach oppstartsnekten · gjør `read_spend` tolerant.
- **`BudgetExceeded`s tre felt beskriver ÉN og samme ledger (kø-(y)):** `kind`/`limit`/`observed`
er ett strukturert stopp-event, og `observed` var det udefenderte tredje feltet — MÅLT: fire av
fem raise-steder (`TokenMeter.charge`, `tick_round`, og BEGGE armene i `exhausted()`) kunne
rapportere hvilken som helst verdi uten at 621 tester merket det; bare `PortfolioMeter.check`
var dekket. **Fellen som skjulte det:** `spikes/_harness.py:41` har sin EGEN kopi av
`BudgetExceeded`/`TokenMeter`, så `spikes/test_harness.py`s `observed`-assert dekker IKKE den
shippede modulen — produksjonens `tick_round` hadde null direkte test. `exhausted()` er det
ENESTE stedet som VELGER ledger (S3.4-guarden), så en refusal som sier `portfolio_tokens` mens
den rapporterer runets eget forbruk ville villedet enhver leser. **Testene bygges med
`observed != limit`**: ved nøyaktig-uttømt sammenfaller de to, og en test skrevet der kan ikke
skille dem — den ville passert på en implementasjon som ekkoet taket tilbake som forbruket.
Ingen defekt funnet i verdiene selv (kontrast (x)/(p)): trippelen var koherent alle fem steder,
gapet var rent dekning. Load-bearing MÅLT (`tests/test_budget.py`), ni mutasjoner alle røde:
fem `observed`-mutasjoner (inkl. kontrollen) + fire ekko-mutasjoner.
- **Pengetall kvantiseres i ÉN orden, fra ÉN kilde (kø-(p)):** `ledger.to_ore` er rammeverkets ENE
NOK→øre-konvertering (Decimal, ROUND_HALF_UP), og den brukes **per pengebeløp — deretter summeres
HELTALL**. `run.py` importerer den; aldri en egen kopi (to kopier av en penge-konvertering drifter,

View file

@ -53,6 +53,21 @@ async def test_cap_crossed_raises_budget_exceeded() -> None:
await mw.process(_Ctx(_resp(60)), _noop) # type: ignore[arg-type]
assert exc.value.kind == "tokens"
assert exc.value.limit == 50
assert exc.value.observed == 60 # the spend that crossed it, not merely that something did
def test_round_cap_crossed_carries_the_round_ledger() -> None:
"""The rounds arm of the same stop event (kø-(y)).
The shipped ``tick_round`` had NO direct test at all only ``spikes/_harness.py``'s separate
copy of ``TokenMeter`` did, and that file is not the one that runs. Its entire error contract
was riding on a module the framework never imports."""
meter = TokenMeter(Budget(max_tokens=1000, max_rounds=2))
assert meter.tick_round() == 1
assert meter.tick_round() == 2 # exactly at the cap is still within it
with pytest.raises(BudgetExceeded) as exc:
meter.tick_round()
assert (exc.value.kind, exc.value.limit, exc.value.observed) == ("rounds", 2, 3)
def test_non_positive_cap_rejected() -> None:
@ -171,6 +186,37 @@ async def test_spend_is_ledgered_even_when_the_per_run_cap_raises() -> None:
assert portfolio.spent == 60 # ...and the global ledger still saw the spend
def test_exhausted_names_one_ledger_in_all_three_fields() -> None:
"""kø-(y): the pre-call guard's refusal must describe ONE ledger consistently.
``exhausted`` is where the S3.4 guard decides what to refuse a call with, and it is the only
raise site that CHOOSES between two ledgers. Naming the binding cap in ``kind`` is already
tested; this pins the other two fields to that same choice a refusal that says
``portfolio_tokens`` while reporting the run's own spend would misdirect every reader of it.
Both arms are built with ``observed != limit`` on purpose. At exactly-exhausted the two
coincide, so a test written at that point cannot tell them apart and would pass on an
implementation that echoed the cap back as the spend. Overrunning first is also the honest
case: a run that crossed its cap and had the error caught is precisely when the next call
must be refused."""
meter = TokenMeter(Budget(max_tokens=10, max_rounds=10))
with pytest.raises(BudgetExceeded):
meter.charge(15) # the overrun is real: the meter now stands at 15 against a cap of 10
own = meter.exhausted()
assert own is not None
assert (own.kind, own.limit, own.observed) == ("tokens", 10, 15)
# The global arm: this run has spent NOTHING — a sibling drained the pass.
portfolio = PortfolioMeter(
PortfolioBudget(max_total_tokens=100, max_tokens_per_run=100), spent=150
)
bound = TokenMeter(Budget(max_tokens=100, max_rounds=10), portfolio=portfolio)
assert bound.tokens == 0
shared = bound.exhausted()
assert shared is not None
assert (shared.kind, shared.limit, shared.observed) == ("portfolio_tokens", 100, 150)
def test_no_word_count_token_proxy_in_src() -> None:
# The meter is fed from real UsageDetails, never a len(text.split()) word-count proxy
# (research 03 Rec 3 — the Fase 1 _word_tokens anti-pattern is retired). NOTE: a bare