feat(provenance): a run records which external service it actually called
The egress declaration (Trekk B3) says what a run MAY contact. It cannot say what it DID: after the run, nothing distinguished "the agents queried the price register" from "the agents ignored it", and a proposal resting on an external service should be traceable to it. ToolCallRecorder(FunctionMiddleware) mirrors BudgetMiddleware(ChatMiddleware) one layer down — that one observes the debate's chat calls, this one its tool calls. It observes only: call_next is always awaited, so a trace can never alter the run it traces. The record lands on ProvenanceStamp.external_calls, read AFTER the debate so it is a record rather than an intention. MEASURED, not assumed, before any of it was written: FunctionMiddleware fires for a tool served over a REAL MCP stdio subprocess, and context.function.name carries the BARE tool name with no server prefix. That measurement decided the design — MAF cannot tell us which server a tool came from, so attribution comes from our own config, and a name allowed by two servers is recorded UNATTRIBUTED (server="") rather than credited to the first match. Naming a service that may never have been contacted is the one place a guess must not go. Only CONFIGURED tools are recorded. The middleware fires for every function the agents invoke, including the in-process retrieve_cost_docs on the road path; logging those would turn the record into a false egress claim. An empty list is a positive statement — nothing outside this process was contacted — which is why it is always serialized rather than omitted. Honesty limit, written on ExternalCall itself: this is the call and its source. It is NOT evidence that the service's answer reached the proposal, nor a verified rendering of that answer. One finding, and it is the reason for measuring rather than trusting green: the road-path negative test was VACUOUS. Its scripted tool call named an argument the tool does not declare (code vs query), MAF rejected the call before invocation, and the test asserted an empty record against a run where no tool ran at all — green under the exact mutation it existed to catch. It now spies on the recorder and asserts the invocation genuinely reached it before asserting it was not recorded. This is last session's lesson again: a scenario that cannot distinguish two implementations proves nothing. The tool-call double is registered in test_scripted_client_consolidation.py's _DELEGATING_OVERRIDES — it cannot live in the reply_selector seam, which returns a reply STRING, and a response that is not text is its whole subject. Load-bearing MEASURED (tests/test_b4_mcp_call_trace_loadbearing.py) against the whole 755-test suite, four mutations all red: detach the recorder from the debate middleware · record every function invocation · attribute an ambiguous name to the first server · stop reading the recorder into provenance. Control: a run with no configured servers records nothing, so the empty record is a real answer and not the only one the seam can produce. Ran it, not just tested it: the real recorder against a real MCP server subprocess returns ExternalCall(server='prisregister', tool='lookup_unit_price'), and a scripted CLI run's outbox artefact carries the empty list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtRd8y1PDPGwkrRXFhubqr
This commit is contained in:
parent
455d93d33e
commit
991131be3f
6 changed files with 402 additions and 2 deletions
|
|
@ -32,12 +32,20 @@ Every refusal below exists because its absence is a live hazard:
|
|||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from collections.abc import Awaitable, Callable
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
|
||||
from agent_framework import MCPStdioTool, MCPStreamableHTTPTool
|
||||
from agent_framework import (
|
||||
FunctionInvocationContext,
|
||||
FunctionMiddleware,
|
||||
MCPStdioTool,
|
||||
MCPStreamableHTTPTool,
|
||||
)
|
||||
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
||||
|
||||
from portfolio_optimiser.provenance import ExternalCall
|
||||
|
||||
|
||||
class McpServerConfig(BaseModel):
|
||||
"""One external MCP server a run is permitted to contact.
|
||||
|
|
@ -170,6 +178,62 @@ def build_mcp_tools(configs: tuple[McpServerConfig, ...]) -> list[Any]:
|
|||
return tools
|
||||
|
||||
|
||||
def tool_server_index(configs: tuple[McpServerConfig, ...]) -> dict[str, str]:
|
||||
"""Map each allowed tool name to the server that serves it — the attribution table B4 needs.
|
||||
|
||||
A name allowed by MORE THAN ONE configured server maps to ``""`` (unattributed) rather than to
|
||||
one of them. MEASURED: MAF hands function middleware the BARE tool name, with no server prefix,
|
||||
so the two really are indistinguishable at that seam. Picking the first match would be a guess
|
||||
written into a provenance record, which is the one place a guess must never go — validation,
|
||||
never repair, applied to attribution.
|
||||
"""
|
||||
index: dict[str, str] = {}
|
||||
for config in configs:
|
||||
for name in config.allowed_tools:
|
||||
index[name] = "" if name in index else config.name
|
||||
return index
|
||||
|
||||
|
||||
class ToolCallRecorder(FunctionMiddleware):
|
||||
"""Function middleware that records WHICH configured external tool a run actually called (B4).
|
||||
|
||||
Mirrors ``budget.BudgetMiddleware(ChatMiddleware)`` one layer down: that one observes the
|
||||
debate's chat calls, this one its tool calls. It observes only — ``call_next`` is always
|
||||
awaited, and nothing here can block, alter or short-circuit an invocation. A trace that changed
|
||||
the run it traces would not be a trace.
|
||||
|
||||
**Only CONFIGURED tools are recorded.** The middleware fires for every function the agents
|
||||
invoke, including the in-process ``retrieve_cost_docs`` on the road path. Logging those would
|
||||
turn the record into a false egress claim — the whole value of the record is that its entries
|
||||
mean "something outside this process was contacted".
|
||||
|
||||
Recording is idempotent per ``(server, tool)`` and returned SORTED: the record answers *what was
|
||||
contacted*, and it is stamped into a byte-deterministic artefact, so it must not vary with how
|
||||
many times an agent happened to ask.
|
||||
"""
|
||||
|
||||
def __init__(self, index: dict[str, str]) -> None:
|
||||
self._index = index
|
||||
self._seen: set[tuple[str, str]] = set()
|
||||
|
||||
def note(self, tool_name: str) -> None:
|
||||
"""Record one invocation by tool name. Unconfigured names are IGNORED, not recorded as
|
||||
unattributed calls — an in-process tool is not an external service with a missing label."""
|
||||
if tool_name in self._index:
|
||||
self._seen.add((self._index[tool_name], tool_name))
|
||||
|
||||
def calls(self) -> list[ExternalCall]:
|
||||
return [ExternalCall(server=s, tool=t) for s, t in sorted(self._seen)]
|
||||
|
||||
async def process(
|
||||
self, context: FunctionInvocationContext, call_next: Callable[[], Awaitable[None]]
|
||||
) -> None:
|
||||
name = getattr(getattr(context, "function", None), "name", None)
|
||||
if isinstance(name, str):
|
||||
self.note(name)
|
||||
await call_next()
|
||||
|
||||
|
||||
def service_labels(configs: tuple[McpServerConfig, ...]) -> tuple[str, ...]:
|
||||
"""The egress declaration, one label per server: what will be contacted and which tools are
|
||||
permitted. Feeds ``mandate.announce(external_services=...)``, so an operator sees the full
|
||||
|
|
|
|||
|
|
@ -29,6 +29,26 @@ class Citation(BaseModel):
|
|||
snippet: str
|
||||
|
||||
|
||||
class ExternalCall(BaseModel):
|
||||
"""One external service call a run actually made (Trekk B4).
|
||||
|
||||
**What this is evidence of, and what it is not.** It records that ``tool`` was invoked and which
|
||||
configured ``server`` it belongs to. It is NOT evidence that the service's answer reached the
|
||||
proposal, and it is not a verified rendering of what the service returned — the framework hands
|
||||
the answer to the agent, and what the agent does with it is the agent's. Reading this as "the
|
||||
figure came from the price register" would claim more than the record supports.
|
||||
|
||||
``server`` is ``""`` when the tool name cannot be attributed to exactly one configured server.
|
||||
MEASURED against a real MCP stdio subprocess: MAF passes the BARE tool name to function
|
||||
middleware, with no server prefix, so two servers exposing one tool name are indistinguishable
|
||||
at this seam. Unattributed is the honest answer there; naming the first match would put a
|
||||
service in the record that may never have been contacted.
|
||||
"""
|
||||
|
||||
server: str
|
||||
tool: str
|
||||
|
||||
|
||||
class ProvenanceStamp(BaseModel):
|
||||
"""Authoritative provenance for one proposal — at least one citation is mandatory."""
|
||||
|
||||
|
|
@ -37,6 +57,9 @@ class ProvenanceStamp(BaseModel):
|
|||
role: str
|
||||
validator_decision: Literal["validated", "rejected"]
|
||||
token_usage: int
|
||||
#: External service calls the run made (B4). EMPTY is a positive statement — "nothing outside
|
||||
#: this process was contacted" — not an absent field, which is why it is always serialized.
|
||||
external_calls: list[ExternalCall] = Field(default_factory=list)
|
||||
|
||||
def to_annotations(self) -> list[Annotation]:
|
||||
"""Map to MAF ``Annotation`` dicts for display only (NOT the source of truth)."""
|
||||
|
|
|
|||
|
|
@ -67,9 +67,11 @@ from portfolio_optimiser.mandate import (
|
|||
)
|
||||
from portfolio_optimiser.mcp_tools import (
|
||||
McpServerConfig,
|
||||
ToolCallRecorder,
|
||||
build_mcp_tools,
|
||||
load_mcp_config,
|
||||
service_labels,
|
||||
tool_server_index,
|
||||
)
|
||||
from portfolio_optimiser.provenance import ProvenanceStamp
|
||||
from portfolio_optimiser.reference_domain import Project, load_reference_projects
|
||||
|
|
@ -536,12 +538,16 @@ async def run_project(
|
|||
)
|
||||
factory = client_factory if client_factory is not None else _default_factory(profile)
|
||||
budget_mw = BudgetMiddleware(meter)
|
||||
# Trekk B4: the egress DECLARATION says what a run may contact; this records what it actually
|
||||
# called. Attached only when servers are configured — with none there is nothing to attribute a
|
||||
# call to, and the middleware list stays exactly what it was before Trekk B.
|
||||
call_recorder = ToolCallRecorder(tool_server_index(mcp_servers)) if mcp_servers else None
|
||||
debate = fresh_workflow(
|
||||
factory,
|
||||
max_rounds=max_rounds,
|
||||
enable_layer1_hitl=enable_layer1_hitl,
|
||||
tools=debate_tools,
|
||||
middleware=[budget_mw],
|
||||
middleware=[budget_mw] if call_recorder is None else [budget_mw, call_recorder],
|
||||
)
|
||||
# S4.2 cut (comparison protocol §4 pkt 2/3): everything above is offline — contracts, budget, and
|
||||
# the EAGER client build (fresh_workflow constructs the proposer+checker clients, workflow.py:64).
|
||||
|
|
@ -661,6 +667,10 @@ async def run_project(
|
|||
"validated" if isinstance(validator_outcome, ValidatedProposal) else "rejected"
|
||||
),
|
||||
token_usage=meter.tokens,
|
||||
# B4: which external service the debate actually called. Read AFTER the debate, so it is a
|
||||
# record rather than an intention. The honesty limit lives on ``ExternalCall`` itself: this
|
||||
# is the call and its source, not a verified rendering of the service's answer.
|
||||
external_calls=call_recorder.calls() if call_recorder is not None else [],
|
||||
)
|
||||
|
||||
# 6b. Step 3/4 checker gate (målbilde §2/§6): the validator falsifies the numbers, the checker
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue