feat(portfolio): C3.5 — pre-call run-total USD budget belt (parity row 16/31) [skip-docs]

Add a pre-call USD belt on top of the post-charge token/round meter (§8),
so no future live run can loop past its run budget. Belt-and-braces above
the SDK's per-call max_budget_usd cap.

- budget.py: optional run-total `max_cost_usd` on BudgetMeter (fail-fast on
  non-positive, §10) + `guard_before_call(spent_usd)` raising the same
  structured stop event (BudgetKind widened with "cost_usd"; limit/observed
  → float). Reaching the cap exactly does not stop; crossing it does
  (mirrors the token cap).
- loop.py: `_guarded_complete` helper reads the client's accumulated
  total_cost_usd (0.0 for scripted clients) and guards BEFORE every
  client.complete; all three call sites routed through it — one detach point.
- sdk_client.py: total_cost_usd already exposed/accumulated — untouched.
- tests/test_budget.py: meter-level cap tests + load-bearing loop-wiring
  test (counting client; detach the guard → unguarded loop runs to the round
  cap → kind "rounds" not "cost_usd" → red).

457→462 green, golden byte-exact, full gate clean (ruff+format+mypy strict,
22 src files), run_s10.py/runs/ byte-untouched. README test-count sync ×2 +
budget.py belt note.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RiTwaKLesgcwXx2mDviqpt
This commit is contained in:
Kjell Tore Guttormsen 2026-07-23 22:45:37 +02:00
commit 111b320b75
4 changed files with 145 additions and 14 deletions

View file

@ -13,7 +13,7 @@ human-in-the-loop, and the system learns from the verdicts.
> **Status:** the D7 build (S5S10) is complete, and the deterministic **ingest layer** > **Status:** the D7 build (S5S10) is complete, and the deterministic **ingest layer**
> (CSV and SQL source types) has since been added in front of the loop. The deterministic > (CSV and SQL source types) has since been added in front of the loop. The deterministic
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by > backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
> seam, each proven by load-bearing tests (457 tests, all running offline without an API > seam, each proven by load-bearing tests (462 tests, all running offline without an API
> key). The programme's single budgeted **live model run has been executed and validated** > key). The programme's single budgeted **live model run has been executed and validated**
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below). > its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
@ -57,7 +57,9 @@ offline. Module by module:
never by leaking through context. never by leaking through context.
**Agentic loop** (§3 steps 25, §8) **Agentic loop** (§3 steps 25, §8)
- `budget.py` — the budget meter: no unbounded loop exists anywhere in the framework. - `budget.py` — the budget meter: no unbounded loop exists anywhere in the framework. On
top of the post-charge token/round caps sits an optional pre-call run-total USD belt that
refuses the next model call once the run has crossed its USD budget.
- `loop.py` — generate, makerchecker debate, gate, and informed refinement: the - `loop.py` — generate, makerchecker debate, gate, and informed refinement: the
validator's previous rejection reason is fed into the next bounded attempt, so the validator's previous rejection reason is fed into the next bounded attempt, so the
model corrects against the falsification instead of re-answering identically. model corrects against the falsification instead of re-answering identically.
@ -186,7 +188,7 @@ Python ≥3.10 · [`claude-agent-sdk`](https://pypi.org/project/claude-agent-sdk
```bash ```bash
uv sync # install dependencies uv sync # install dependencies
uv run pytest # 457 tests — run without any API key and without network uv run pytest # 462 tests — run without any API key and without network
uv run ruff check . && uv run ruff format --check . uv run ruff check . && uv run ruff format --check .
uv run mypy src # strict uv run mypy src # strict
``` ```

View file

@ -4,9 +4,16 @@ Token accounting comes from the PROVIDER-REPORTED usage after each model call
never a word-count or character proxy; on counting paths a response missing never a word-count or character proxy; on counting paths a response missing
usage fails CLOSED (``UsageAccountingError``), not silently uncounted. Crossing usage fails CLOSED (``UsageAccountingError``), not silently uncounted. Crossing
a cap raises ``BudgetExceeded``, a STRUCTURED stop event carrying the breached a cap raises ``BudgetExceeded``, a STRUCTURED stop event carrying the breached
kind, the limit, and the observed value never a silent hang. The caps come kind, the limit, and the observed value never a silent hang. The token/round
from the fail-fast startup ``TerminationContract`` (§10), which already refuses caps come from the fail-fast startup ``TerminationContract`` (§10), which
non-positive values. already refuses non-positive values.
On TOP of the post-charge token/round meter sits an optional PRE-call USD belt
(C3.5): a run-total ``max_cost_usd`` cap the loop checks BEFORE every model
call against the SDK client's accumulated ``total_cost_usd``. Once the
run-total cost has crossed the cap the next call is refused with the same
structured stop event (``kind="cost_usd"``) belt-and-braces above the SDK's
per-call USD cap, so no future live run can loop past its run budget.
""" """
from __future__ import annotations from __future__ import annotations
@ -15,7 +22,7 @@ from typing import Literal
from portfolio_optimiser_claude.contracts import TerminationContract from portfolio_optimiser_claude.contracts import TerminationContract
BudgetKind = Literal["tokens", "rounds"] BudgetKind = Literal["tokens", "rounds", "cost_usd"]
class UsageAccountingError(Exception): class UsageAccountingError(Exception):
@ -25,7 +32,7 @@ class UsageAccountingError(Exception):
class BudgetExceeded(Exception): class BudgetExceeded(Exception):
"""The structured stop event: breached kind + limit + observed value (§8).""" """The structured stop event: breached kind + limit + observed value (§8)."""
def __init__(self, kind: BudgetKind, limit: int, observed: int) -> None: def __init__(self, kind: BudgetKind, limit: float, observed: float) -> None:
super().__init__(f"budget exceeded: {kind} observed {observed} > limit {limit}") super().__init__(f"budget exceeded: {kind} observed {observed} > limit {limit}")
self.kind: BudgetKind = kind self.kind: BudgetKind = kind
self.limit = limit self.limit = limit
@ -33,10 +40,20 @@ class BudgetExceeded(Exception):
class BudgetMeter: class BudgetMeter:
"""Run-scoped usage meter over the startup termination contract (§8).""" """Run-scoped usage meter over the startup termination contract (§8).
def __init__(self, termination: TerminationContract) -> None: ``max_cost_usd`` is an OPTIONAL run-total USD cap for the pre-call belt
(C3.5); left unset the belt is a no-op and only the token/round caps apply.
When set it must be positive (§10 fail-fast discipline).
"""
def __init__(
self, termination: TerminationContract, *, max_cost_usd: float | None = None
) -> None:
if max_cost_usd is not None and max_cost_usd <= 0:
raise ValueError(f"max_cost_usd must be positive when set, got {max_cost_usd}")
self._termination = termination self._termination = termination
self._max_cost_usd = max_cost_usd
self.tokens_used = 0 self.tokens_used = 0
self.rounds_used = 0 self.rounds_used = 0
@ -55,3 +72,18 @@ class BudgetMeter:
self.rounds_used += 1 self.rounds_used += 1
if self.rounds_used > self._termination.max_rounds: if self.rounds_used > self._termination.max_rounds:
raise BudgetExceeded("rounds", self._termination.max_rounds, self.rounds_used) raise BudgetExceeded("rounds", self._termination.max_rounds, self.rounds_used)
def guard_before_call(self, spent_usd: float) -> None:
"""Pre-call USD belt (C3.5): refuse the NEXT model call once the
accumulated run-total cost has crossed the cap.
This is the ONLY cap read from OUTSIDE the meter the SDK client
accumulates ``total_cost_usd`` from each ``ResultMessage`` and the loop
passes it here BEFORE every ``client.complete``. No cap configured (or a
scripted client reporting no spend) makes this a no-op, so the offline
suite is untouched. Reaching the cap exactly does not stop (mirrors the
token cap); crossing it raises the structured stop event (§8)."""
if self._max_cost_usd is None:
return
if spent_usd > self._max_cost_usd:
raise BudgetExceeded("cost_usd", self._max_cost_usd, spent_usd)

View file

@ -60,6 +60,22 @@ class ModelClient(Protocol):
def complete(self, prompt: str, *, role: str) -> ModelReply: ... def complete(self, prompt: str, *, role: str) -> ModelReply: ...
def _guarded_complete(
client: ModelClient, prompt: str, *, role: str, meter: BudgetMeter
) -> ModelReply:
"""One model call, pre-guarded by the run-total USD belt (C3.5, §8).
Every model call in the loop goes through here. The pre-call guard reads the
client's accumulated ``total_cost_usd`` (0.0 for scripted clients that carry
no cost) and refuses to make the call once the run-total USD cap is crossed
the belt that stops the loop from spending past its run budget, on TOP of
the per-call SDK cap and the post-charge token/round meter.
"""
spent_usd: float = getattr(client, "total_cost_usd", 0.0)
meter.guard_before_call(spent_usd)
return client.complete(prompt, role=role)
# --- Step 2: hypothesise (structured candidate generation) --------------------------------- # --- Step 2: hypothesise (structured candidate generation) ---------------------------------
@ -87,7 +103,7 @@ def generate_candidate(
bounded by the budget meter: a round tick is charged between attempts (§8). bounded by the budget meter: a round tick is charged between attempts (§8).
""" """
while True: while True:
model_reply = client.complete(prompt, role=_PROPOSER_ROLE) model_reply = _guarded_complete(client, prompt, role=_PROPOSER_ROLE, meter=meter)
meter.charge_tokens(model_reply.usage_tokens) meter.charge_tokens(model_reply.usage_tokens)
try: try:
# JSONDecodeError and pydantic's ValidationError are ValueErrors. # JSONDecodeError and pydantic's ValidationError are ValueErrors.
@ -157,15 +173,17 @@ def run_debate(
for _ in range(max_rounds): for _ in range(max_rounds):
turns += 1 turns += 1
check_turn_safety_net(turns, max_rounds) check_turn_safety_net(turns, max_rounds)
proposer_reply = client.complete( proposer_reply = _guarded_complete(
_proposer_debate_prompt(context, critique), role=_PROPOSER_ROLE client, _proposer_debate_prompt(context, critique), role=_PROPOSER_ROLE, meter=meter
) )
meter.charge_tokens(proposer_reply.usage_tokens) meter.charge_tokens(proposer_reply.usage_tokens)
proposer_output = proposer_reply.text proposer_output = proposer_reply.text
turns += 1 turns += 1
check_turn_safety_net(turns, max_rounds) check_turn_safety_net(turns, max_rounds)
checker_reply = client.complete(_checker_prompt(proposer_output), role=_CHECKER_ROLE) checker_reply = _guarded_complete(
client, _checker_prompt(proposer_output), role=_CHECKER_ROLE, meter=meter
)
meter.charge_tokens(checker_reply.usage_tokens) meter.charge_tokens(checker_reply.usage_tokens)
checker_last = checker_reply.text checker_last = checker_reply.text

View file

@ -12,12 +12,22 @@ import pytest
from portfolio_optimiser_claude.budget import BudgetExceeded, BudgetMeter, UsageAccountingError from portfolio_optimiser_claude.budget import BudgetExceeded, BudgetMeter, UsageAccountingError
from portfolio_optimiser_claude.contracts import TerminationContract from portfolio_optimiser_claude.contracts import TerminationContract
from portfolio_optimiser_claude.loop import ModelReply, generate_candidate
def _meter(max_rounds: int = 5, max_tokens: int = 100) -> BudgetMeter: def _meter(max_rounds: int = 5, max_tokens: int = 100) -> BudgetMeter:
return BudgetMeter(TerminationContract(max_rounds=max_rounds, max_tokens=max_tokens)) return BudgetMeter(TerminationContract(max_rounds=max_rounds, max_tokens=max_tokens))
def _usd_meter(
max_cost_usd: float, *, max_rounds: int = 1000, max_tokens: int = 10_000
) -> BudgetMeter:
return BudgetMeter(
TerminationContract(max_rounds=max_rounds, max_tokens=max_tokens),
max_cost_usd=max_cost_usd,
)
class TestTokenAccounting: class TestTokenAccounting:
def test_accumulates_provider_reported_usage(self) -> None: def test_accumulates_provider_reported_usage(self) -> None:
meter = _meter(max_tokens=100) meter = _meter(max_tokens=100)
@ -73,3 +83,72 @@ def test_caps_come_from_the_startup_contract() -> None:
# construction — the meter builds on that, never on loose ints. # construction — the meter builds on that, never on loose ints.
with pytest.raises(Exception): with pytest.raises(Exception):
TerminationContract(max_rounds=0, max_tokens=100) TerminationContract(max_rounds=0, max_tokens=100)
class _CountingClient:
"""A ``ModelClient`` stand-in that counts calls and accumulates a fixed USD
cost per call, exposing it as ``total_cost_usd`` exactly like the SDK client
the seam the pre-call guard reads (C3.5)."""
def __init__(self, *, cost_per_call: float, text: str) -> None:
self._cost_per_call = cost_per_call
self._text = text
self.calls = 0
self.total_cost_usd = 0.0
def complete(self, prompt: str, *, role: str) -> ModelReply:
self.calls += 1
self.total_cost_usd += self._cost_per_call
return ModelReply(text=self._text, usage_tokens=1, model="counting")
class TestRunTotalUsdCap:
def test_non_positive_usd_cap_is_refused_at_construction(self) -> None:
# §10 fail-fast discipline: a USD cap, when set, must be positive.
with pytest.raises(ValueError):
BudgetMeter(TerminationContract(max_rounds=5, max_tokens=100), max_cost_usd=0.0)
def test_no_cap_makes_the_guard_a_noop(self) -> None:
# No USD cap configured (the default) → the guard never fires, so the
# offline suite and scripted clients that carry no cost are untouched.
meter = _meter()
meter.guard_before_call(9_999.0) # no raise
def test_reaching_the_usd_cap_exactly_does_not_stop(self) -> None:
# Mirrors the token cap (test_reaching_the_cap_exactly_does_not_stop):
# equal-to-cap is allowed; only crossing it stops.
meter = _usd_meter(0.25)
meter.guard_before_call(0.25) # no raise
def test_crossing_the_usd_cap_raises_structured_stop(self) -> None:
# §8: the run-total USD belt raises the SAME structured stop event form
# as the token/round caps — breached kind + limit + observed value.
meter = _usd_meter(0.25)
with pytest.raises(BudgetExceeded) as exc_info:
meter.guard_before_call(0.5)
stop = exc_info.value
assert stop.kind == "cost_usd"
assert stop.limit == 0.25
assert stop.observed == 0.5
class TestPreCallGuardWiredIntoLoop:
"""Load-bearing (§11): the guard runs BEFORE each model call via
``loop._guarded_complete``. Detach point: delete the
``meter.guard_before_call`` line in ``_guarded_complete`` the overspending
call is made anyway the call-count and kind assertions below go RED (the
unguarded loop instead runs to the round cap, raising ``kind == "rounds"``)."""
def test_guard_stops_before_the_call_that_would_overspend(self) -> None:
# +0.125 USD/call, cap 0.25 (all exact binary fractions): calls 1..3
# spend 0.125 / 0.25 / 0.375 (call 3's pre-guard sees exactly 0.25 == cap
# → allowed). BEFORE call 4 the guard sees 0.375 > 0.25 and refuses, so
# call 4 is NEVER made. Unparseable text forces generate_candidate's
# retry loop, so every iteration is a fresh guarded call.
client = _CountingClient(cost_per_call=0.125, text="not json — force the retry loop")
meter = _usd_meter(0.25)
with pytest.raises(BudgetExceeded) as exc_info:
generate_candidate(client, "prompt", meter=meter)
assert exc_info.value.kind == "cost_usd"
assert exc_info.value.observed == 0.375
assert client.calls == 3