feat(portfolio): C3.5 — pre-call run-total USD budget belt (parity row 16/31) [skip-docs]
Add a pre-call USD belt on top of the post-charge token/round meter (§8), so no future live run can loop past its run budget. Belt-and-braces above the SDK's per-call max_budget_usd cap. - budget.py: optional run-total `max_cost_usd` on BudgetMeter (fail-fast on non-positive, §10) + `guard_before_call(spent_usd)` raising the same structured stop event (BudgetKind widened with "cost_usd"; limit/observed → float). Reaching the cap exactly does not stop; crossing it does (mirrors the token cap). - loop.py: `_guarded_complete` helper reads the client's accumulated total_cost_usd (0.0 for scripted clients) and guards BEFORE every client.complete; all three call sites routed through it — one detach point. - sdk_client.py: total_cost_usd already exposed/accumulated — untouched. - tests/test_budget.py: meter-level cap tests + load-bearing loop-wiring test (counting client; detach the guard → unguarded loop runs to the round cap → kind "rounds" not "cost_usd" → red). 457→462 green, golden byte-exact, full gate clean (ruff+format+mypy strict, 22 src files), run_s10.py/runs/ byte-untouched. README test-count sync ×2 + budget.py belt note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RiTwaKLesgcwXx2mDviqpt
This commit is contained in:
parent
a926e4ad46
commit
111b320b75
4 changed files with 145 additions and 14 deletions
|
|
@ -13,7 +13,7 @@ human-in-the-loop, and the system learns from the verdicts.
|
|||
> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
|
||||
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
|
||||
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
|
||||
> seam, each proven by load-bearing tests (457 tests, all running offline without an API
|
||||
> seam, each proven by load-bearing tests (462 tests, all running offline without an API
|
||||
> key). The programme's single budgeted **live model run has been executed and validated** —
|
||||
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
|
||||
|
||||
|
|
@ -57,7 +57,9 @@ offline. Module by module:
|
|||
never by leaking through context.
|
||||
|
||||
**Agentic loop** (§3 steps 2–5, §8)
|
||||
- `budget.py` — the budget meter: no unbounded loop exists anywhere in the framework.
|
||||
- `budget.py` — the budget meter: no unbounded loop exists anywhere in the framework. On
|
||||
top of the post-charge token/round caps sits an optional pre-call run-total USD belt that
|
||||
refuses the next model call once the run has crossed its USD budget.
|
||||
- `loop.py` — generate, maker–checker debate, gate, and informed refinement: the
|
||||
validator's previous rejection reason is fed into the next bounded attempt, so the
|
||||
model corrects against the falsification instead of re-answering identically.
|
||||
|
|
@ -186,7 +188,7 @@ Python ≥3.10 · [`claude-agent-sdk`](https://pypi.org/project/claude-agent-sdk
|
|||
|
||||
```bash
|
||||
uv sync # install dependencies
|
||||
uv run pytest # 457 tests — run without any API key and without network
|
||||
uv run pytest # 462 tests — run without any API key and without network
|
||||
uv run ruff check . && uv run ruff format --check .
|
||||
uv run mypy src # strict
|
||||
```
|
||||
|
|
|
|||
|
|
@ -4,9 +4,16 @@ Token accounting comes from the PROVIDER-REPORTED usage after each model call
|
|||
never a word-count or character proxy; on counting paths a response missing
|
||||
usage fails CLOSED (``UsageAccountingError``), not silently uncounted. Crossing
|
||||
a cap raises ``BudgetExceeded``, a STRUCTURED stop event carrying the breached
|
||||
kind, the limit, and the observed value — never a silent hang. The caps come
|
||||
from the fail-fast startup ``TerminationContract`` (§10), which already refuses
|
||||
non-positive values.
|
||||
kind, the limit, and the observed value — never a silent hang. The token/round
|
||||
caps come from the fail-fast startup ``TerminationContract`` (§10), which
|
||||
already refuses non-positive values.
|
||||
|
||||
On TOP of the post-charge token/round meter sits an optional PRE-call USD belt
|
||||
(C3.5): a run-total ``max_cost_usd`` cap the loop checks BEFORE every model
|
||||
call against the SDK client's accumulated ``total_cost_usd``. Once the
|
||||
run-total cost has crossed the cap the next call is refused with the same
|
||||
structured stop event (``kind="cost_usd"``) — belt-and-braces above the SDK's
|
||||
per-call USD cap, so no future live run can loop past its run budget.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -15,7 +22,7 @@ from typing import Literal
|
|||
|
||||
from portfolio_optimiser_claude.contracts import TerminationContract
|
||||
|
||||
BudgetKind = Literal["tokens", "rounds"]
|
||||
BudgetKind = Literal["tokens", "rounds", "cost_usd"]
|
||||
|
||||
|
||||
class UsageAccountingError(Exception):
|
||||
|
|
@ -25,7 +32,7 @@ class UsageAccountingError(Exception):
|
|||
class BudgetExceeded(Exception):
|
||||
"""The structured stop event: breached kind + limit + observed value (§8)."""
|
||||
|
||||
def __init__(self, kind: BudgetKind, limit: int, observed: int) -> None:
|
||||
def __init__(self, kind: BudgetKind, limit: float, observed: float) -> None:
|
||||
super().__init__(f"budget exceeded: {kind} observed {observed} > limit {limit}")
|
||||
self.kind: BudgetKind = kind
|
||||
self.limit = limit
|
||||
|
|
@ -33,10 +40,20 @@ class BudgetExceeded(Exception):
|
|||
|
||||
|
||||
class BudgetMeter:
|
||||
"""Run-scoped usage meter over the startup termination contract (§8)."""
|
||||
"""Run-scoped usage meter over the startup termination contract (§8).
|
||||
|
||||
def __init__(self, termination: TerminationContract) -> None:
|
||||
``max_cost_usd`` is an OPTIONAL run-total USD cap for the pre-call belt
|
||||
(C3.5); left unset the belt is a no-op and only the token/round caps apply.
|
||||
When set it must be positive (§10 fail-fast discipline).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self, termination: TerminationContract, *, max_cost_usd: float | None = None
|
||||
) -> None:
|
||||
if max_cost_usd is not None and max_cost_usd <= 0:
|
||||
raise ValueError(f"max_cost_usd must be positive when set, got {max_cost_usd}")
|
||||
self._termination = termination
|
||||
self._max_cost_usd = max_cost_usd
|
||||
self.tokens_used = 0
|
||||
self.rounds_used = 0
|
||||
|
||||
|
|
@ -55,3 +72,18 @@ class BudgetMeter:
|
|||
self.rounds_used += 1
|
||||
if self.rounds_used > self._termination.max_rounds:
|
||||
raise BudgetExceeded("rounds", self._termination.max_rounds, self.rounds_used)
|
||||
|
||||
def guard_before_call(self, spent_usd: float) -> None:
|
||||
"""Pre-call USD belt (C3.5): refuse the NEXT model call once the
|
||||
accumulated run-total cost has crossed the cap.
|
||||
|
||||
This is the ONLY cap read from OUTSIDE the meter — the SDK client
|
||||
accumulates ``total_cost_usd`` from each ``ResultMessage`` and the loop
|
||||
passes it here BEFORE every ``client.complete``. No cap configured (or a
|
||||
scripted client reporting no spend) makes this a no-op, so the offline
|
||||
suite is untouched. Reaching the cap exactly does not stop (mirrors the
|
||||
token cap); crossing it raises the structured stop event (§8)."""
|
||||
if self._max_cost_usd is None:
|
||||
return
|
||||
if spent_usd > self._max_cost_usd:
|
||||
raise BudgetExceeded("cost_usd", self._max_cost_usd, spent_usd)
|
||||
|
|
|
|||
|
|
@ -60,6 +60,22 @@ class ModelClient(Protocol):
|
|||
def complete(self, prompt: str, *, role: str) -> ModelReply: ...
|
||||
|
||||
|
||||
def _guarded_complete(
|
||||
client: ModelClient, prompt: str, *, role: str, meter: BudgetMeter
|
||||
) -> ModelReply:
|
||||
"""One model call, pre-guarded by the run-total USD belt (C3.5, §8).
|
||||
|
||||
Every model call in the loop goes through here. The pre-call guard reads the
|
||||
client's accumulated ``total_cost_usd`` (0.0 for scripted clients that carry
|
||||
no cost) and refuses to make the call once the run-total USD cap is crossed
|
||||
— the belt that stops the loop from spending past its run budget, on TOP of
|
||||
the per-call SDK cap and the post-charge token/round meter.
|
||||
"""
|
||||
spent_usd: float = getattr(client, "total_cost_usd", 0.0)
|
||||
meter.guard_before_call(spent_usd)
|
||||
return client.complete(prompt, role=role)
|
||||
|
||||
|
||||
# --- Step 2: hypothesise (structured candidate generation) ---------------------------------
|
||||
|
||||
|
||||
|
|
@ -87,7 +103,7 @@ def generate_candidate(
|
|||
bounded by the budget meter: a round tick is charged between attempts (§8).
|
||||
"""
|
||||
while True:
|
||||
model_reply = client.complete(prompt, role=_PROPOSER_ROLE)
|
||||
model_reply = _guarded_complete(client, prompt, role=_PROPOSER_ROLE, meter=meter)
|
||||
meter.charge_tokens(model_reply.usage_tokens)
|
||||
try:
|
||||
# JSONDecodeError and pydantic's ValidationError are ValueErrors.
|
||||
|
|
@ -157,15 +173,17 @@ def run_debate(
|
|||
for _ in range(max_rounds):
|
||||
turns += 1
|
||||
check_turn_safety_net(turns, max_rounds)
|
||||
proposer_reply = client.complete(
|
||||
_proposer_debate_prompt(context, critique), role=_PROPOSER_ROLE
|
||||
proposer_reply = _guarded_complete(
|
||||
client, _proposer_debate_prompt(context, critique), role=_PROPOSER_ROLE, meter=meter
|
||||
)
|
||||
meter.charge_tokens(proposer_reply.usage_tokens)
|
||||
proposer_output = proposer_reply.text
|
||||
|
||||
turns += 1
|
||||
check_turn_safety_net(turns, max_rounds)
|
||||
checker_reply = client.complete(_checker_prompt(proposer_output), role=_CHECKER_ROLE)
|
||||
checker_reply = _guarded_complete(
|
||||
client, _checker_prompt(proposer_output), role=_CHECKER_ROLE, meter=meter
|
||||
)
|
||||
meter.charge_tokens(checker_reply.usage_tokens)
|
||||
checker_last = checker_reply.text
|
||||
|
||||
|
|
|
|||
|
|
@ -12,12 +12,22 @@ import pytest
|
|||
|
||||
from portfolio_optimiser_claude.budget import BudgetExceeded, BudgetMeter, UsageAccountingError
|
||||
from portfolio_optimiser_claude.contracts import TerminationContract
|
||||
from portfolio_optimiser_claude.loop import ModelReply, generate_candidate
|
||||
|
||||
|
||||
def _meter(max_rounds: int = 5, max_tokens: int = 100) -> BudgetMeter:
|
||||
return BudgetMeter(TerminationContract(max_rounds=max_rounds, max_tokens=max_tokens))
|
||||
|
||||
|
||||
def _usd_meter(
|
||||
max_cost_usd: float, *, max_rounds: int = 1000, max_tokens: int = 10_000
|
||||
) -> BudgetMeter:
|
||||
return BudgetMeter(
|
||||
TerminationContract(max_rounds=max_rounds, max_tokens=max_tokens),
|
||||
max_cost_usd=max_cost_usd,
|
||||
)
|
||||
|
||||
|
||||
class TestTokenAccounting:
|
||||
def test_accumulates_provider_reported_usage(self) -> None:
|
||||
meter = _meter(max_tokens=100)
|
||||
|
|
@ -73,3 +83,72 @@ def test_caps_come_from_the_startup_contract() -> None:
|
|||
# construction — the meter builds on that, never on loose ints.
|
||||
with pytest.raises(Exception):
|
||||
TerminationContract(max_rounds=0, max_tokens=100)
|
||||
|
||||
|
||||
class _CountingClient:
|
||||
"""A ``ModelClient`` stand-in that counts calls and accumulates a fixed USD
|
||||
cost per call, exposing it as ``total_cost_usd`` exactly like the SDK client
|
||||
— the seam the pre-call guard reads (C3.5)."""
|
||||
|
||||
def __init__(self, *, cost_per_call: float, text: str) -> None:
|
||||
self._cost_per_call = cost_per_call
|
||||
self._text = text
|
||||
self.calls = 0
|
||||
self.total_cost_usd = 0.0
|
||||
|
||||
def complete(self, prompt: str, *, role: str) -> ModelReply:
|
||||
self.calls += 1
|
||||
self.total_cost_usd += self._cost_per_call
|
||||
return ModelReply(text=self._text, usage_tokens=1, model="counting")
|
||||
|
||||
|
||||
class TestRunTotalUsdCap:
|
||||
def test_non_positive_usd_cap_is_refused_at_construction(self) -> None:
|
||||
# §10 fail-fast discipline: a USD cap, when set, must be positive.
|
||||
with pytest.raises(ValueError):
|
||||
BudgetMeter(TerminationContract(max_rounds=5, max_tokens=100), max_cost_usd=0.0)
|
||||
|
||||
def test_no_cap_makes_the_guard_a_noop(self) -> None:
|
||||
# No USD cap configured (the default) → the guard never fires, so the
|
||||
# offline suite and scripted clients that carry no cost are untouched.
|
||||
meter = _meter()
|
||||
meter.guard_before_call(9_999.0) # no raise
|
||||
|
||||
def test_reaching_the_usd_cap_exactly_does_not_stop(self) -> None:
|
||||
# Mirrors the token cap (test_reaching_the_cap_exactly_does_not_stop):
|
||||
# equal-to-cap is allowed; only crossing it stops.
|
||||
meter = _usd_meter(0.25)
|
||||
meter.guard_before_call(0.25) # no raise
|
||||
|
||||
def test_crossing_the_usd_cap_raises_structured_stop(self) -> None:
|
||||
# §8: the run-total USD belt raises the SAME structured stop event form
|
||||
# as the token/round caps — breached kind + limit + observed value.
|
||||
meter = _usd_meter(0.25)
|
||||
with pytest.raises(BudgetExceeded) as exc_info:
|
||||
meter.guard_before_call(0.5)
|
||||
stop = exc_info.value
|
||||
assert stop.kind == "cost_usd"
|
||||
assert stop.limit == 0.25
|
||||
assert stop.observed == 0.5
|
||||
|
||||
|
||||
class TestPreCallGuardWiredIntoLoop:
|
||||
"""Load-bearing (§11): the guard runs BEFORE each model call via
|
||||
``loop._guarded_complete``. Detach point: delete the
|
||||
``meter.guard_before_call`` line in ``_guarded_complete`` → the overspending
|
||||
call is made anyway → the call-count and kind assertions below go RED (the
|
||||
unguarded loop instead runs to the round cap, raising ``kind == "rounds"``)."""
|
||||
|
||||
def test_guard_stops_before_the_call_that_would_overspend(self) -> None:
|
||||
# +0.125 USD/call, cap 0.25 (all exact binary fractions): calls 1..3
|
||||
# spend 0.125 / 0.25 / 0.375 (call 3's pre-guard sees exactly 0.25 == cap
|
||||
# → allowed). BEFORE call 4 the guard sees 0.375 > 0.25 and refuses, so
|
||||
# call 4 is NEVER made. Unparseable text forces generate_candidate's
|
||||
# retry loop, so every iteration is a fresh guarded call.
|
||||
client = _CountingClient(cost_per_call=0.125, text="not json — force the retry loop")
|
||||
meter = _usd_meter(0.25)
|
||||
with pytest.raises(BudgetExceeded) as exc_info:
|
||||
generate_candidate(client, "prompt", meter=meter)
|
||||
assert exc_info.value.kind == "cost_usd"
|
||||
assert exc_info.value.observed == 0.375
|
||||
assert client.calls == 3
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue