The egress declaration (Trekk B3) says what a run MAY contact. It cannot say what it DID: after the run, nothing distinguished "the agents queried the price register" from "the agents ignored it", and a proposal resting on an external service should be traceable to it. ToolCallRecorder(FunctionMiddleware) mirrors BudgetMiddleware(ChatMiddleware) one layer down — that one observes the debate's chat calls, this one its tool calls. It observes only: call_next is always awaited, so a trace can never alter the run it traces. The record lands on ProvenanceStamp.external_calls, read AFTER the debate so it is a record rather than an intention. MEASURED, not assumed, before any of it was written: FunctionMiddleware fires for a tool served over a REAL MCP stdio subprocess, and context.function.name carries the BARE tool name with no server prefix. That measurement decided the design — MAF cannot tell us which server a tool came from, so attribution comes from our own config, and a name allowed by two servers is recorded UNATTRIBUTED (server="") rather than credited to the first match. Naming a service that may never have been contacted is the one place a guess must not go. Only CONFIGURED tools are recorded. The middleware fires for every function the agents invoke, including the in-process retrieve_cost_docs on the road path; logging those would turn the record into a false egress claim. An empty list is a positive statement — nothing outside this process was contacted — which is why it is always serialized rather than omitted. Honesty limit, written on ExternalCall itself: this is the call and its source. It is NOT evidence that the service's answer reached the proposal, nor a verified rendering of that answer. One finding, and it is the reason for measuring rather than trusting green: the road-path negative test was VACUOUS. Its scripted tool call named an argument the tool does not declare (code vs query), MAF rejected the call before invocation, and the test asserted an empty record against a run where no tool ran at all — green under the exact mutation it existed to catch. It now spies on the recorder and asserts the invocation genuinely reached it before asserting it was not recorded. This is last session's lesson again: a scenario that cannot distinguish two implementations proves nothing. The tool-call double is registered in test_scripted_client_consolidation.py's _DELEGATING_OVERRIDES — it cannot live in the reply_selector seam, which returns a reply STRING, and a response that is not text is its whole subject. Load-bearing MEASURED (tests/test_b4_mcp_call_trace_loadbearing.py) against the whole 755-test suite, four mutations all red: detach the recorder from the debate middleware · record every function invocation · attribute an ambiguous name to the first server · stop reading the recorder into provenance. Control: a run with no configured servers records nothing, so the empty record is a real answer and not the only one the seam can produce. Ran it, not just tested it: the real recorder against a real MCP server subprocess returns ExternalCall(server='prisregister', tool='lookup_unit_price'), and a scripted CLI run's outbox artefact carries the empty list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtRd8y1PDPGwkrRXFhubqr
128 lines
6.6 KiB
Python
128 lines
6.6 KiB
Python
"""S2.5 (Step 9) consolidation guard (T-2.5e): the four scripted ``_inner_get_response`` bodies
|
|
collapsed to ONE canonical client (``simulation.ScriptedChatClient``); conftest's three test doubles
|
|
now SUBCLASS it. These grep-guards lock that in — they go RED if a divergent ``_inner_get_response``
|
|
is re-added, if a ``src``→``tests`` import creeps in, or if a double stops subclassing the canonical.
|
|
|
|
The guard is a regression lock over an already-verified consolidation: before the collapse there were
|
|
five ``def _inner_get_response`` sites (four scripted + test_step5's own-lineage double); after, two.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
from pathlib import Path
|
|
|
|
_ROOT = Path(__file__).resolve().parents[1]
|
|
|
|
|
|
def _py_files(base: str) -> list[Path]:
|
|
return sorted((_ROOT / base).rglob("*.py"))
|
|
|
|
|
|
# Files permitted to define ``_inner_get_response``. The invariant this guard protects is that the
|
|
# scripted BODY is not duplicated — not that the def-site count is frozen. Adding a file here is a
|
|
# deliberate act: a new entry must either be the canonical, or a thin override that DELEGATES to it
|
|
# (which ``test_delegating_overrides_call_super`` below then enforces mechanically).
|
|
_CANONICAL_SITE = "src/portfolio_optimiser/simulation.py"
|
|
|
|
# Overrides in the SCRIPTED lineage — they subclass ``ScriptedChatClient``, so a body of their own
|
|
# would be a copy of the canonical. They must delegate.
|
|
_DELEGATING_OVERRIDES = [
|
|
# S3.3 ordering probe: yields to the event loop N times, then delegates. It cannot live in the
|
|
# reply-selector seam, which the canonical calls synchronously and so can never await.
|
|
"tests/test_portfolio_concurrent_loadbearing.py",
|
|
# S3.3 failure-accounting probe: RAISES for one project (after N completed calls), otherwise
|
|
# delegates. Like the ordering probe it cannot live in the reply-selector seam — that seam
|
|
# returns a reply string, and this double's whole subject is the absence of one.
|
|
"tests/test_portfolio_failure_accounting_loadbearing.py",
|
|
# B4 tool-call probe: its first response is a function CALL rather than text, then it delegates.
|
|
# It cannot live in the reply-selector seam either — that seam returns a reply STRING, and a
|
|
# response that is not text is precisely this double's subject.
|
|
"tests/test_b4_mcp_call_trace_loadbearing.py",
|
|
]
|
|
|
|
# Doubles in a DIFFERENT lineage (``spikes._harness.FakeChatClient``). There is no canonical
|
|
# scripted body above them to delegate to, so the delegation rule does not apply — but the
|
|
# separation is asserted rather than assumed, so a file cannot be parked here to dodge the rule.
|
|
_FOREIGN_LINEAGE = ["tests/test_step5_refine_loadbearing.py"]
|
|
|
|
|
|
def test_inner_get_response_collapsed_to_two_sites() -> None:
|
|
"""The four scripted clients collapse to ONE canonical ``_inner_get_response``
|
|
(``simulation.py``). Every other def-site must be a registered, DELEGATING override — never a
|
|
fourth copy of the body.
|
|
|
|
The guard originally pinned a literal count of 2. That made it fail on any new legitimate
|
|
subclass while still passing if someone pasted a duplicated body into an already-listed file —
|
|
a count is the wrong shape for the invariant. The list below plus
|
|
``test_delegating_overrides_call_super`` pin the property itself."""
|
|
sites = [
|
|
p.relative_to(_ROOT).as_posix()
|
|
for base in ("src", "tests")
|
|
for p in _py_files(base)
|
|
if p.name != Path(__file__).name # this guard file references the pattern in prose
|
|
and "def _inner_get_response" in p.read_text(encoding="utf-8")
|
|
]
|
|
expected = sorted([_CANONICAL_SITE, *_DELEGATING_OVERRIDES, *_FOREIGN_LINEAGE])
|
|
assert sorted(sites) == expected, (
|
|
f"unregistered ``_inner_get_response`` def-site — the scripted body must not be copied. "
|
|
f"Expected {expected}, got: {sites}"
|
|
)
|
|
|
|
|
|
def test_foreign_lineage_doubles_are_genuinely_foreign() -> None:
|
|
"""A file listed as foreign lineage must NOT subclass the scripted canonical.
|
|
|
|
Without this, ``_FOREIGN_LINEAGE`` would be an escape hatch: any scripted-lineage subclass
|
|
could be moved into that list to skip the delegation rule below."""
|
|
for site in _FOREIGN_LINEAGE:
|
|
text = (_ROOT / site).read_text(encoding="utf-8")
|
|
assert "ScriptedChatClient" not in text, (
|
|
f"{site} is registered as foreign lineage but references ``ScriptedChatClient`` — if it "
|
|
"is in the scripted lineage it belongs in _DELEGATING_OVERRIDES and must delegate"
|
|
)
|
|
|
|
|
|
def test_delegating_overrides_call_super() -> None:
|
|
"""Every scripted-lineage override actually DELEGATES to the canonical rather than
|
|
reimplementing it.
|
|
|
|
This is the strength the literal count never had: without it, a file already on the list could
|
|
grow a full copy of the scripted body and the consolidation would be cosmetic again."""
|
|
for site in _DELEGATING_OVERRIDES:
|
|
text = (_ROOT / site).read_text(encoding="utf-8")
|
|
# Match the delegation ITSELF — ``super()._inner_get_response`` or the explicit
|
|
# ``super(Cls, self)._inner_get_response`` form a nested function needs. Searching for
|
|
# "super(" and "_inner_get_response" independently would pass on any file that merely
|
|
# calls ``super().__init__`` near a def, which is accidental-green, not a guard.
|
|
assert re.search(r"super\([^)]*\)\._inner_get_response", text), (
|
|
f"{site} defines ``_inner_get_response`` but never delegates to the canonical via "
|
|
"``super()._inner_get_response`` — that is a duplicated body, which is exactly what "
|
|
"this guard exists to prevent"
|
|
)
|
|
|
|
|
|
def test_no_src_imports_tests() -> None:
|
|
"""No ``src`` module imports from ``tests`` — the canonical lives in ``src/simulation.py`` so
|
|
``conftest`` imports ``src``, never the reverse (the forbidden src→tests direction)."""
|
|
offenders = [
|
|
p.relative_to(_ROOT).as_posix()
|
|
for p in _py_files("src")
|
|
if ("from tests" in (text := p.read_text(encoding="utf-8")) or "import tests" in text)
|
|
]
|
|
assert offenders == [], f"src must not import tests: {offenders}"
|
|
|
|
|
|
def test_conftest_doubles_subclass_canonical() -> None:
|
|
"""conftest's three test doubles genuinely SUBCLASS the canonical ``ScriptedChatClient``
|
|
(delegating the shared body) — so the consolidation is real, not cosmetic."""
|
|
from conftest import (
|
|
ScriptedChatClient,
|
|
SyntheticUsageChatClient,
|
|
_ProjectAwareUsageChatClient,
|
|
_RecordingChatClient,
|
|
)
|
|
|
|
assert issubclass(SyntheticUsageChatClient, ScriptedChatClient)
|
|
assert issubclass(_ProjectAwareUsageChatClient, ScriptedChatClient)
|
|
assert issubclass(_RecordingChatClient, ScriptedChatClient)
|