open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(mcp): concrete MCP servers become tools the agents can call during a run

Browse source 
Krav 3, and the operator chose the run path explicitly: the external service must
be reachable WHILE the run works, not only when documents are ingested. Until now
the run path had one in-process tool against a local folder — and on the bundle
path the agents had no tools at all.

MAF already ships the client (MCPStdioTool / MCPStreamableHTTPTool, verified in
the pinned 1.9.0 with allowed_tools and request_timeout), so `mcp_tools.py` owns
only what MAF cannot decide for us: which servers a run may contact, which of
their tools it may call, how long it waits, and where the credential comes from.
This is a DIFFERENT seam from ingest_mcp.py on purpose — that one pulls source
documents before a run and speaks to null-argument tools. Same protocol, different
job.

Every refusal is a live hazard, not tidiness. An empty allowlist would let the far
end decide what the agents may call, so naming the tools is mandatory. A
non-positive timeout is an unbounded wait against a third party. An unknown field
is refused rather than ignored, which is also what keeps a literal secret from
being parked in the config — there is no field for one, only the NAME of an env
var. A named-but-unset credential refuses instead of calling anonymously, because
an anonymous call can succeed with the wrong scope.

Egress is declared, always. Every server and permitted tool is named in the run
announcement before the first call — including when no --mandate is given, which
was a real hole: the announcement only printed with a commission, so configuring
servers without one would have contacted third parties with nothing printed at
all. --live-dry-run still opens nothing, because the tools are entered after the
dry-run cut: the promise to stop before the first call now covers egress too.

Threaded through BOTH modes. A flag accepted in one mode and silently dropped in
the other is the defect class this CLI refuses by name.

Load-bearing MEASURED against the whole 744-test suite, four mutations all red:
build the tools but never hand them to the agents (2) · never enter the
AsyncExitStack, so they are constructed and useless (1) · never declare the egress
(2) · drop the allowlist on the built client (1).

Two live docs claimed MCP was unwired in the run path; both corrected rather than
left to rot.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULCqjLF61rehj5cZmdUoR3
This commit is contained in:
Kjell Tore Guttormsen 2026-08-05 16:53:07 +02:00
commit 9668e17f2f
8 changed files with 706 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

2
CLAUDE.md
View file

@ -13,7 +13,7 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
- `ruff` for lint+format. `pytest` for test.
- Modell-valg som konfig (modell-map rolle→Foundry-deployment), ikke spredt i kode.
- Metode kodifiseres som **Agent Skill** (`agentskills.io`: `SKILL.md` + `scripts/` + `references/`).
- Datatilgang: in-process `FunctionTool` er default-sømmen i kjørestien; **MCP er extension point**, demonstrert via `build_mcp_server` (`datasource.py`) men ikke wiret inn i kjørestien. Data-source-konfig JSON-Schema-validert, fail-fast.
- Datatilgang: in-process `FunctionTool` er default-sømmen i kjørestien. **MCP er wiret som opt-in i kjørestien** (`mcp_tools.py` + `--mcp-config`, Trekk B 2026-08-05): konkrete eksterne servere blir verktøy agentene kan kalle UNDER debatten. Uten konfig gjøres null nettverkskall og verktøylista er uendret. Tre regler er load-bearing: **allowlist er påkrevd** (tom liste ville latt motparten bestemme hva agentene får kalle), **hver server og hvert tillatte verktøy navngis i kunngjøringen før første kall** (også uten `--mandate` — ingen udeklarert egress), og `--live-dry-run` åpner **ingenting**. Egen søm fra `ingest_mcp.py` (kildedokumenter FØR kjøring, null-argument-tools) — samme protokoll, ulik jobb. `build_mcp_server` (`datasource.py`) er fortsatt kun demo. Data-source-konfig JSON-Schema-validert, fail-fast.
- `shared/` er en **git subtree** av [`portfolio-optimiser-commons`](https://git.fromaitochitta.com/open/portfolio-optimiser-commons) (source of truth, R1 realisert 2026-07-03; publisert i `open/` 2026-08-04 — `commons`-remoten peker fortsatt på `ktg/` og virker uendret). Synk er **pull-only**: endringer committes i commons og hentes med `git subtree pull --prefix=shared commons main --squash`. ALDRI `git subtree push` fra konsument — re-split lekker hele konsument-historikken inn i commons (observert + opprydd 2026-07-03). Se `shared/README.md`.
## Kommandoer

26
docs/bestille-en-kjoring.md
View file

@ -130,3 +130,29 @@ tre tilnærminger mot samme linje ga «totalt 90 000», som ingen av dem kunne i
og gjengir tallet hvis det finnes.
- **Den lagrer seg ikke som kunnskap.** Bestillingen er en instruks for én kjøring. Det som lærer
systemet noe, er dommen du avgir etterpå.
## Hvis kjøringen skal spørre en ekstern tjeneste
Skal agentene kunne slå opp i en tjeneste dere allerede har — et prisregister, målerdata, et
avtaleregister — settes den opp med `--mcp-config`:
```json
{"servers": [
{"name": "prisregister", "transport": "http", "url": "https://intern.example/mcp",
"allowed_tools": ["lookup_unit_price"], "timeout_seconds": 15,
"credential_env": "PRISREGISTER_TOKEN"}
]}
```
Tre ting er verdt å vite som bestiller:
- **Du må navngi hvilke verktøy som er tillatt.** Tom liste er ikke lov. Uten det ville tjenesten
selv bestemt hva agentene får lov til å kalle.
- **Alt som vil bli kontaktet, står i kunngjøringen** — før første kall, på `Contacts:`-linja, med
både servernavn og tillatte verktøy. En kjøring når aldri en tjeneste den ikke har navngitt. Det
gjelder også når du ikke bruker `--mandate`.
- **Passord og nøkler skal aldri i fila.** `credential_env` navngir en miljøvariabel; verdien leses
derfra. Er variabelen ikke satt, blir kjøringen nektet — heller det enn å ringe tjenesten anonymt
og få svar med feil tilgang.
Uten `--mcp-config` gjøres ingen nettverkskall i det hele tatt.

17
docs/extending.md
View file

@ -191,8 +191,21 @@ tool is called with an empty argument dict, so `datasource.build_mcp_server` **c
its `retrieve_cost_docs(query)` has a required parameter (verified — it returns an error result).
The two are separate seams by design: `build_mcp_server` serves the agents' retrieval path.
**Still true:** MCP remains **unwired in the optimiser run path** — the in-process `FunctionTool`
seam stays the default there.
**No longer true (Trekk B, 2026-08-05):** MCP used to be unwired in the optimiser run path. It is
now wired, as an **opt-in** — `--mcp-config` (or `run_project(mcp_servers=...)`) hands the agents
live tools from concrete external servers *during* the debate, built by `mcp_tools.py` on MAF's
`MCPStdioTool` / `MCPStreamableHTTPTool`. The in-process `FunctionTool` seam remains the default:
with no config, no network call is possible and the tool list is unchanged.
Three properties hold that config to the repo's data rules, and each is measured by a mutation:
the tool **allowlist is mandatory** (an empty one would let the far end decide what the agents may
call); every configured server and permitted tool is **named in the run announcement before the
first call**, so nothing is contacted undeclared — including when no `--mandate` is given; and
`--live-dry-run` still opens **nothing**, because the tools are entered after the dry-run cut.
This is a **different seam from the ingest path above**, deliberately: that one pulls source
documents into a bundle before a run and speaks to null-argument tools; this one hands live tools
to the agents while they work. They share a protocol, not a job.
**The timeout path moved, and it is covered.** This paragraph used to say the deadline was
`asyncio.wait_for` and that no test exercised it; both halves are now out of date. Measuring

177
src/portfolio_optimiser/mcp_tools.py Normal file
View file

@ -0,0 +1,177 @@
"""Concrete MCP servers as tools on the RUN path (Trekk B1, krav 3).
MAF already ships the MCP client (``MCPStdioTool`` / ``MCPStreamableHTTPTool``, verified present in
the pinned 1.9.0 alongside ``allowed_tools`` / ``request_timeout``), so this module owns only what
MAF cannot decide for us: **which** servers a run may contact, **which** of their tools it may call,
how long it waits, and where the credential comes from.
MAF-SPECIFIC by design, and therefore NOT in ``_MAF_FREE_MODULES``: it builds framework client
objects. That is the deliberate split from ``mandate.py``, which stays framework-neutral and
D7-portable. The D7 sibling will need its own transport module against the same config shape.
**This is a different seam from ``ingest_mcp.py``, on purpose.** That one pulls SOURCE DOCUMENTS
into a bundle before a run and speaks to null-argument tools; this one hands live tools to the
agents DURING the debate. They share a protocol, not a job.
The transport set is CLOSED (``stdio`` | ``http``): a config file can never name arbitrary
machinery to load — the same rule ``--embedder-config`` follows, for the same reason.
Every refusal below exists because its absence is a live hazard:
* an EMPTY ``allowed_tools`` hands the agents whatever the far end chooses to serve — authority
granted by the other party. You must name what you allow;
* a non-positive timeout is an unbounded wait against a third party, which is the fail-fast
invariant this repo applies to every loop and cap;
* an UNKNOWN field is refused rather than ignored, which is also what keeps a literal secret from
being parked in the file and carried along unnoticed — there is no field for one;
* a named credential env var that is NOT set refuses the run rather than calling the service
anonymously: an unauthenticated call can still succeed, with the wrong scope, which is worse
than not calling at all.
"""
from __future__ import annotations
import os
from pathlib import Path
from typing import Any, Literal
from agent_framework import MCPStdioTool, MCPStreamableHTTPTool
from pydantic import BaseModel, ConfigDict, Field, model_validator
class McpServerConfig(BaseModel):
"""One external MCP server a run is permitted to contact.
``extra="forbid"`` is load-bearing, not tidiness: it is what turns a stray ``"credential":
"sk-..."`` into a refusal instead of an ignored key sitting in a config file.
"""
model_config = ConfigDict(extra="forbid", frozen=True)
name: str = Field(min_length=1)
transport: Literal["stdio", "http"]
allowed_tools: tuple[str, ...] = Field(min_length=1)
timeout_seconds: int = Field(gt=0)
command: str | None = None
args: tuple[str, ...] = ()
url: str | None = None
#: The NAME of an environment variable holding the credential — never the credential itself.
credential_env: str | None = None
@model_validator(mode="after")
def _coordinates_match_the_transport(self) -> McpServerConfig:
if self.transport == "stdio":
if not self.command:
raise ValueError(f"{self.name}: stdio transport requires 'command'")
if self.url:
raise ValueError(
f"{self.name}: stdio transport must not carry 'url' — it is ambiguous which "
"endpoint would actually be contacted"
)
else:
if not self.url:
raise ValueError(f"{self.name}: http transport requires 'url'")
if self.command:
raise ValueError(
f"{self.name}: http transport must not carry 'command' — it is ambiguous "
"which endpoint would actually be contacted"
)
return self
class McpConfig(BaseModel):
"""The full set of servers a run may contact."""
model_config = ConfigDict(extra="forbid")
servers: tuple[McpServerConfig, ...] = ()
@model_validator(mode="after")
def _names_are_unique(self) -> McpConfig:
seen: set[str] = set()
for server in self.servers:
if server.name in seen:
raise ValueError(f"duplicate MCP server name: {server.name!r}")
seen.add(server.name)
return self
def load_mcp_config(path: str | Path) -> tuple[McpServerConfig, ...]:
"""Fail-fast loader (mirrors ``load_dimension`` / ``load_mandate``).
Authoritative startup input: missing -> ``FileNotFoundError``, malformed -> ``ValidationError``.
A tolerant read here would be the worst of the three loaders in this repo — degrading a broken
egress config to "no external services" would leave the announcement telling the truth about a
run that was never configured, while a degraded one that *partially* parsed could contact a
subset nobody chose.
"""
p = Path(path)
if not p.is_file():
raise FileNotFoundError(f"MCP config not found: {str(path)!r}")
return McpConfig.model_validate_json(p.read_text(encoding="utf-8")).servers
def _credential(config: McpServerConfig) -> str | None:
"""The credential value, read from the environment at BUILD time. A named-but-unset variable
refuses: calling an external service anonymously can succeed with the wrong scope."""
if config.credential_env is None:
return None
value = os.environ.get(config.credential_env)
if not value:
raise ValueError(
f"{config.name}: credential env var {config.credential_env!r} is not set — refusing "
"to contact the service unauthenticated"
)
return value
def build_mcp_tools(configs: tuple[McpServerConfig, ...]) -> list[Any]:
"""Build one MAF MCP client per configured server, with the allowlist and timeout applied.
NOTE for the caller: an ``MCPTool`` is an async context manager. These objects are constructed
here but NOT connected — the run path must enter them around the debate and exit afterwards.
"""
tools: list[Any] = []
for config in configs:
credential = _credential(config)
if config.transport == "stdio":
# ``credential`` is non-None only when ``credential_env`` named a variable, so the key
# is a str here — narrowed explicitly rather than asserted, so mypy sees it too.
env = (
{config.credential_env: credential}
if credential and config.credential_env
else None
)
tools.append(
MCPStdioTool(
name=config.name,
command=config.command or "",
args=list(config.args),
env=env,
allowed_tools=config.allowed_tools,
request_timeout=config.timeout_seconds,
)
)
else:
header_provider = (
(lambda _ctx, token=credential: {"Authorization": f"Bearer {token}"})
if credential
else None
)
tools.append(
MCPStreamableHTTPTool(
name=config.name,
url=config.url or "",
allowed_tools=config.allowed_tools,
request_timeout=config.timeout_seconds,
header_provider=header_provider,
)
)
return tools
def service_labels(configs: tuple[McpServerConfig, ...]) -> tuple[str, ...]:
"""The egress declaration, one label per server: what will be contacted and which tools are
permitted. Feeds ``mandate.announce(external_services=...)``, so an operator sees the full
reach of a run BEFORE the first call — a run never contacts a service it did not announce."""
return tuple(f"{c.name} ({', '.join(c.allowed_tools)})" for c in configs)

59
src/portfolio_optimiser/run.py
View file

@ -28,6 +28,7 @@ from __future__ import annotations
import asyncio
import json
from collections.abc import Awaitable, Callable, Iterable, Sequence
from contextlib import AsyncExitStack
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Any, Literal, cast
@ -64,6 +65,12 @@ from portfolio_optimiser.mandate import (
load_mandate,
settle,
)
from portfolio_optimiser.mcp_tools import (
McpServerConfig,
build_mcp_tools,
load_mcp_config,
service_labels,
)
from portfolio_optimiser.provenance import ProvenanceStamp
from portfolio_optimiser.reference_domain import Project, load_reference_projects
from portfolio_optimiser.validator import Rejection, ValidatedProposal, baseline_from_project
@ -412,6 +419,7 @@ async def run_project(
semantic_retrieval: bool = False,
embedder: Embedder | None = None,
mandate: Mandate | None = None,
mcp_servers: tuple[McpServerConfig, ...] = (),
) -> RunResult | DryRunReport:
"""Run the vertical slice for ONE project. ``client_factory`` is the test-injection seam
(defaults to the real backend). ``verdict_input`` carries the expert decision/rationale
@ -491,6 +499,14 @@ async def run_project(
citations = [chunk_dict_to_citation(c) for c in chunks]
context = "\n".join(c["snippet"] for c in chunks)
debate_tools = [make_retrieval_tool(docs_dir, top_k=top_k)]
# Trekk B2 (krav 3): configured MCP servers become tools the AGENTS can call during the debate.
# Appended to BOTH paths — on the bundle path they are the first tools that path has ever had.
# Constructed here but NOT connected: an ``MCPTool`` is an async context manager, so the run
# enters them around ``debate.run`` below and exits afterwards. Empty tuple -> the tool list is
# byte-identical to the pre-Trekk-B one, and no network call is possible.
live_mcp_tools = build_mcp_tools(mcp_servers) if mcp_servers else []
debate_tools = debate_tools + live_mcp_tools
if not citations:
raise ValueError(f"no citable content in docs_dir: {docs_dir!r}")
@ -539,7 +555,16 @@ async def run_project(
max_tokens=max_tokens,
top_k=top_k,
)
result = await debate.run(f"Find a cost-saving measure for {project.id}.\nContext:\n{context}")
# The MCP lifecycle (Trekk B2): entered HERE, after the dry-run cut above, so a dry run never
# opens a connection — its promise to stop before the first call covers egress too. Constructed
# tools that are never entered expose nothing, and ones never exited leave the process hanging,
# so the stack owns both halves.
async with AsyncExitStack() as mcp_stack:
for live_tool in live_mcp_tools:
await mcp_stack.enter_async_context(live_tool)
result = await debate.run(
f"Find a cost-saving measure for {project.id}.\nContext:\n{context}"
)
# F1: the candidate must derive from the DEBATE. Feed the proposer's converged output into
# generation (retrieval context is the last-resort fallback only). The checker's verdict
# (Step 3/4) is parsed from the SAME debate result and gates the outcome below.
@ -853,6 +878,7 @@ async def run_portfolio(
semantic_retrieval: bool = False,
embedder: Embedder | None = None,
mandate: Mandate | None = None,
mcp_servers: tuple[McpServerConfig, ...] = (),
) -> PortfolioResult:
"""Fan out over a portfolio of independent projects SEQUENTIALLY, composing ``run_project``
as-is (every project's execution state — meter, debate, retrieval context — is built fresh
@ -1050,6 +1076,7 @@ async def run_portfolio(
semantic_retrieval=semantic_retrieval,
embedder=embedder,
mandate=mandate,
mcp_servers=mcp_servers,
meter=_run_meter(meter_factory, portfolio_meter, max_rounds),
)
for pid, snapshot in snapshots
@ -1182,6 +1209,16 @@ def main(argv: list[str] | None = None) -> int:
"afterwards, one row per approach. Valid in both modes; in portfolio mode it applies to "
"every project in the pass",
)
parser.add_argument(
"--mcp-config",
default=None,
metavar="FILE",
help="external MCP servers this run may contact (JSON, fail-fast): name, transport "
"(stdio|http), coordinates, the ALLOWED tool names, a timeout, and optionally the NAME of "
"an env var holding the credential (never the credential itself). Every server and tool is "
"named in the run announcement BEFORE the first call; without this flag no network call is "
"made at all",
)
parser.add_argument(
"--embedder-config",
default=None,
@ -1432,6 +1469,17 @@ def main(argv: list[str] | None = None) -> int:
print(f"run refused: {exc}", file=sys.stderr)
return 1
# The egress config, loaded fail-fast alongside the commission. Degrading a broken one to "no
# external services" would make the announcement describe a run nobody configured, and a
# partially-parsed one could contact a subset nobody chose.
mcp_servers: tuple[McpServerConfig, ...] = ()
if args.mcp_config is not None:
try:
mcp_servers = load_mcp_config(args.mcp_config)
except (FileNotFoundError, ValidationError, ValueError) as exc:
print(f"run refused: {exc}", file=sys.stderr)
return 1
# The scripted door (offline WHOLE-loop run over the caller's own data). Resolved BEFORE the
# dry-run branch so the two offline modes cannot both be honoured — and BEFORE the portfolio
# dispatch, because the door serves BOTH modes. It originally sat below that dispatch, which
@ -1487,8 +1535,14 @@ def main(argv: list[str] | None = None) -> int:
max_rounds=_DEFAULT_MAX_ROUNDS,
max_tokens=_DEFAULT_MAX_TOKENS,
dimension_label=dimension_label,
external_services=service_labels(mcp_servers),
)
)
elif mcp_servers:
# The egress declaration must NOT depend on a commission being present. Without this
# branch, configuring servers and omitting --mandate would contact third parties with
# nothing printed at all — silent egress, which this repo forbids outright.
print("Contacts: " + ", ".join(service_labels(mcp_servers)))
if args.portfolio:
# Portfolio mode (Step 3): dispatch to the EXISTING run_portfolio via the fail-fast loaders
@ -1515,6 +1569,7 @@ def main(argv: list[str] | None = None) -> int:
semantic_retrieval=args.semantic_retrieval,
client_factory=scripted_client_factory,
mandate=mandate,
mcp_servers=mcp_servers,
)
)
except (ValueError, FileNotFoundError, ValidationError) as exc:
@ -1590,6 +1645,7 @@ def main(argv: list[str] | None = None) -> int:
outbox_dir=args.outbox_dir,
run_id=args.run_id,
verdict_input={"decision": args.decision, "rationale": args.rationale},
mcp_servers=mcp_servers,
live_dry_run=True,
)
)
@ -1639,6 +1695,7 @@ def main(argv: list[str] | None = None) -> int:
semantic_retrieval=args.semantic_retrieval,
client_factory=scripted_client_factory,
mandate=mandate,
mcp_servers=mcp_servers,
)
),
)

87
tests/test_mandate_cli.py
View file

@ -184,3 +184,90 @@ def test_no_mandate_prints_neither_block(bundle, replies_file, capsys) -> None:
assert rc == 0, out
assert "Run mandate for" not in out
assert "Mandate outcome" not in out
# --- the egress declaration (Trekk B3): what a run will contact, before it contacts it ----------
_MCP = {
"servers": [
{
"name": "prisregister",
"transport": "http",
"url": "https://intern.example/mcp",
"allowed_tools": ["lookup_unit_price"],
"timeout_seconds": 15,
}
]
}
@pytest.fixture()
def mcp_file(tmp_path: Path) -> Path:
path = tmp_path / "mcp.json"
path.write_text(json.dumps(_MCP), encoding="utf-8")
return path
def test_configured_servers_are_named_before_the_run(
bundle, mandate_file, mcp_file, capsys
) -> None:
"""Every server and every permitted tool is named in the announcement — a run never reaches a
service it did not declare. ``--live-dry-run`` keeps the assertion offline."""
rc = run.main(
[
"BYGG-KONTOR-NORD",
"--docs-dir",
str(bundle),
"--bundle-dir",
str(bundle),
"--mandate",
str(mandate_file),
"--mcp-config",
str(mcp_file),
"--live-dry-run",
]
)
out = capsys.readouterr().out
assert rc == 0, out
assert "prisregister (lookup_unit_price)" in out
assert "no external services" not in out
def test_egress_is_declared_even_without_a_mandate(bundle, mcp_file, capsys) -> None:
"""The declaration cannot depend on a mandate being present. Without this, configuring servers
and omitting ``--mandate`` would contact third parties with nothing printed at all — silent
egress, which is the one thing this repo's data rules forbid outright."""
rc = run.main(
[
"BYGG-KONTOR-NORD",
"--docs-dir",
str(bundle),
"--bundle-dir",
str(bundle),
"--mcp-config",
str(mcp_file),
"--live-dry-run",
]
)
out = capsys.readouterr().out
assert rc == 0, out
assert "prisregister (lookup_unit_price)" in out
def test_malformed_mcp_config_refuses_the_run(bundle, replies_file, tmp_path, capsys) -> None:
"""An egress config that cannot be read refuses — degrading it to 'no external services' would
make the announcement describe a run nobody configured."""
bad = tmp_path / "mcp.json"
bad.write_text('{"servers": [{"name": "x", "transport": "http"}]}', encoding="utf-8")
rc = run.main(_argv(bundle, replies_file, "--mcp-config", str(bad)))
assert rc == 1
assert "refused" in capsys.readouterr().err.lower()
def test_no_mcp_config_contacts_nothing(bundle, replies_file, mandate_file, capsys) -> None:
"""CONTROL: without ``--mcp-config`` the announcement says so explicitly. An omitted line would
read the same as an unchecked one."""
rc = run.main(_argv(bundle, replies_file, "--mandate", str(mandate_file)))
out = capsys.readouterr().out
assert rc == 0, out
assert "no external services" in out

162
tests/test_mcp_run_loadbearing.py Normal file
View file

@ -0,0 +1,162 @@
"""Load-bearing: configured MCP servers become tools the AGENTS can call during a run, they are
opened and closed around the debate, and nothing is contacted that was not announced (Trekk B2/B3).
Krav 3 is only met if the external service is reachable *while the run works*. Three detach points,
each RED on its own:
* drop the MCP tools from ``debate_tools`` -> the agents never get the tool (and on the bundle path
they get NO tools at all, which is what that path had before);
* skip the ``AsyncExitStack`` entry -> the tools are constructed but never connected, so they are
present and useless — the failure mode that looks like success;
* let a dry run enter them -> ``--live-dry-run`` would contact a third party while claiming to stop
before the first call.
The control asserts an un-configured run's tool list is unchanged, so no assertion above can pass
on something the run does anyway.
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
import pytest
from portfolio_optimiser import run as run_module
from portfolio_optimiser.mcp_tools import McpServerConfig
from portfolio_optimiser.run import run_project
from portfolio_optimiser.simulation import ScriptedChatClient
from portfolio_optimiser.verdicts import VerdictStore
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
_REPLY = (
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
)
_SERVER = McpServerConfig(
name="prisregister",
transport="http",
url="https://intern.example/mcp",
allowed_tools=("lookup_unit_price",),
timeout_seconds=15,
)
class _FakeMcpTool:
"""Stands in for a MAF ``MCPTool``: an async context manager that RECORDS its lifecycle.
A real one would open a connection, which is exactly what a test must not do — and the thing
worth asserting is the lifecycle itself, not the protocol MAF already owns.
"""
def __init__(self, name: str) -> None:
self.name = name
self.entered = 0
self.exited = 0
self.entered_before_debate: bool | None = None
async def __aenter__(self) -> _FakeMcpTool:
self.entered += 1
return self
async def __aexit__(self, *exc: object) -> None:
self.exited += 1
@pytest.fixture()
def fake_tools(monkeypatch: pytest.MonkeyPatch) -> list[_FakeMcpTool]:
"""Replace the MAF client construction with recording doubles — the config still travels the
real path, only the socket-opening object is swapped."""
built: list[_FakeMcpTool] = []
def _build(configs: tuple[McpServerConfig, ...]) -> list[Any]:
# Return ONLY this call's tools. Accumulating across calls handed the second project both
# projects' tools under one name, and MAF refused it ("Duplicate tool name") — a defect in
# the double, but a useful reminder that each run builds its own clients.
fresh = [_FakeMcpTool(c.name) for c in configs]
built.extend(fresh)
return list(fresh)
monkeypatch.setattr(run_module, "build_mcp_tools", _build)
return built
@pytest.fixture()
def captured_tools(monkeypatch: pytest.MonkeyPatch) -> list[list[Any]]:
"""Capture what actually reaches the debate's ``tools=`` — the agents' real surface."""
seen: list[list[Any]] = []
original = run_module.fresh_workflow
def _spy(*args: Any, **kwargs: Any) -> Any:
seen.append(list(kwargs.get("tools") or []))
return original(*args, **kwargs)
monkeypatch.setattr(run_module, "fresh_workflow", _spy)
return seen
def _factory(_role: str) -> ScriptedChatClient:
return ScriptedChatClient(reply=_REPLY)
async def _run(**kwargs: Any):
return await run_project(
"BYGG-KONTOR-NORD",
"local",
docs_dir=str(BUNDLE_DIR),
bundle_dir=str(BUNDLE_DIR),
verdict_input=_VERDICT_INPUT,
store=VerdictStore(verdicts=[]),
client_factory=_factory,
**kwargs,
)
async def test_configured_server_becomes_a_tool_the_agents_have(fake_tools, captured_tools) -> None:
"""On the bundle path the agents had NO tools at all; a configured server is the first one."""
await _run(mcp_servers=(_SERVER,))
assert captured_tools, "the debate was never built"
assert any(isinstance(t, _FakeMcpTool) for t in captured_tools[0])
async def test_tools_are_opened_and_closed_around_the_debate(fake_tools) -> None:
"""Constructed is not connected. An ``MCPTool`` must be entered to expose anything, and exited
or the process hangs — so both halves are asserted, not just the first."""
await _run(mcp_servers=(_SERVER,))
assert fake_tools, "no MCP tool was built"
assert fake_tools[0].entered == 1
assert fake_tools[0].exited == 1
async def test_dry_run_never_contacts_a_configured_server(fake_tools) -> None:
"""``--live-dry-run`` stops before the first model call, and that promise has to cover egress
too: a dry run that opened a connection to a third party would be lying by omission."""
await _run(mcp_servers=(_SERVER,), live_dry_run=True)
assert all(t.entered == 0 for t in fake_tools)
async def test_run_without_mcp_servers_keeps_the_tool_list_unchanged(captured_tools) -> None:
"""CONTROL: with nothing configured the bundle path still hands the agents no tools — the
pre-Trekk-B behaviour, so every assertion above rests on the configuration and not on the run."""
await _run()
assert captured_tools[0] == []
async def test_portfolio_mode_gives_every_project_the_configured_tools(
fake_tools, captured_tools
) -> None:
"""A configured server reaches EVERY project in a portfolio pass. Without the threading the
flag would be accepted and silently dropped in one of the two modes — the defect class this
repo refuses by name ('refused, never ignored')."""
result = await run_module.run_portfolio(
["FV42-GSV-E1", "RV13-RAS-TP"],
"local",
store=VerdictStore(verdicts=[]),
client_factory=_factory,
mcp_servers=(_SERVER,),
)
assert len(result.runs) == 2
assert len(captured_tools) == 2
assert all(any(isinstance(t, _FakeMcpTool) for t in tools) for tools in captured_tools)

180
tests/test_mcp_tools.py Normal file
View file

@ -0,0 +1,180 @@
"""Concrete MCP servers as run-path tools (Trekk B1) — config + tool construction.
Krav 3: a run must be able to reach an external service (price register, meter data, contract
registry) while it is working. MAF already ships the client (``MCPStdioTool`` /
``MCPStreamableHTTPTool``), so what this module owns is the part MAF cannot decide for us: which
servers a run may contact, which of their tools it may call, what it waits, and where the
credential comes from.
Every refusal here exists because its absence is a real hazard, not for tidiness:
* an EMPTY ``allowed_tools`` would let a server expose any tool it likes to the agents — authority
by whatever the far end happens to serve. You must name what you allow;
* a missing/blank timeout would be an unbounded wait against a third party (the repo's fail-fast
invariant: stop criteria are never optional);
* an UNKNOWN field is refused rather than ignored, which is what keeps a literal secret from being
parked in the config file and silently carried along;
* a named credential env var that is not set refuses the run instead of quietly calling the
service unauthenticated.
"""
from __future__ import annotations
import json
import pytest
from agent_framework import MCPStdioTool, MCPStreamableHTTPTool
from pydantic import ValidationError
from portfolio_optimiser.mcp_tools import (
McpServerConfig,
build_mcp_tools,
load_mcp_config,
service_labels,
)
_STDIO = {
"name": "maalerdata",
"transport": "stdio",
"command": "uvx",
"args": ["maalerdata-mcp"],
"allowed_tools": ["read_meter"],
"timeout_seconds": 30,
}
_HTTP = {
"name": "prisregister",
"transport": "http",
"url": "https://intern.example/mcp",
"allowed_tools": ["lookup_unit_price", "list_categories"],
"timeout_seconds": 15,
}
def _write(tmp_path, *servers) -> str:
path = tmp_path / "mcp.json"
path.write_text(json.dumps({"servers": list(servers)}), encoding="utf-8")
return str(path)
# --- config shape ------------------------------------------------------------------------------
def test_stdio_and_http_configs_load(tmp_path) -> None:
configs = load_mcp_config(_write(tmp_path, _STDIO, _HTTP))
assert [c.name for c in configs] == ["maalerdata", "prisregister"]
assert configs[0].transport == "stdio"
assert configs[1].allowed_tools == ("lookup_unit_price", "list_categories")
def test_stdio_without_command_refused(tmp_path) -> None:
"""A stdio server with nothing to launch is not a server."""
with pytest.raises(ValidationError):
load_mcp_config(_write(tmp_path, {**_STDIO, "command": None}))
def test_http_without_url_refused(tmp_path) -> None:
with pytest.raises(ValidationError):
load_mcp_config(_write(tmp_path, {**_HTTP, "url": None}))
def test_transport_and_coordinates_must_agree(tmp_path) -> None:
"""A stdio server carrying a URL (or an http server carrying a command) is ambiguous about
what would actually be contacted — refused rather than resolved by precedence."""
with pytest.raises(ValidationError):
load_mcp_config(_write(tmp_path, {**_STDIO, "url": "https://intern.example/mcp"}))
with pytest.raises(ValidationError):
load_mcp_config(_write(tmp_path, {**_HTTP, "command": "uvx"}))
def test_unknown_transport_refused(tmp_path) -> None:
"""The transport set is CLOSED — a config can never name arbitrary machinery to load
(the same rule ``--embedder-config`` follows)."""
with pytest.raises(ValidationError):
load_mcp_config(_write(tmp_path, {**_STDIO, "transport": "carrier-pigeon"}))
def test_empty_allowed_tools_refused(tmp_path) -> None:
"""Closed by default: you must name the tools a run may call. An empty list would hand the
agents whatever the far end chooses to expose."""
with pytest.raises(ValidationError):
load_mcp_config(_write(tmp_path, {**_STDIO, "allowed_tools": []}))
def test_non_positive_timeout_refused(tmp_path) -> None:
"""Fail-fast: an unbounded wait against a third party is exactly what stop criteria exist
to prevent."""
with pytest.raises(ValidationError):
load_mcp_config(_write(tmp_path, {**_STDIO, "timeout_seconds": 0}))
def test_unknown_field_refused(tmp_path) -> None:
"""Refused, never ignored — and this is the rule that keeps a literal secret out of the file:
there is no field for one, and an unrecognised key does not slip through unnoticed."""
with pytest.raises(ValidationError):
load_mcp_config(_write(tmp_path, {**_STDIO, "credential": "sk-live-not-here-please"}))
def test_duplicate_server_names_refused(tmp_path) -> None:
"""``name`` is how a server is identified in the egress declaration; two servers on one name
would make the announcement ambiguous about what is contacted."""
with pytest.raises(ValidationError):
load_mcp_config(_write(tmp_path, _STDIO, {**_STDIO, "command": "other"}))
def test_missing_config_file_refused(tmp_path) -> None:
with pytest.raises(FileNotFoundError):
load_mcp_config(str(tmp_path / "nope.json"))
def test_malformed_config_refused(tmp_path) -> None:
bad = tmp_path / "mcp.json"
bad.write_text("not json", encoding="utf-8")
with pytest.raises(ValidationError):
load_mcp_config(str(bad))
# --- tool construction -------------------------------------------------------------------------
def test_builds_the_right_client_per_transport() -> None:
tools = build_mcp_tools((McpServerConfig(**_STDIO), McpServerConfig(**_HTTP)))
assert isinstance(tools[0], MCPStdioTool)
assert isinstance(tools[1], MCPStreamableHTTPTool)
def test_allowed_tools_and_timeout_reach_the_client() -> None:
"""The allowlist and the timeout are the two limits that matter, so they are asserted on the
built client — not merely on the config that was meant to produce it."""
tool = build_mcp_tools((McpServerConfig(**_HTTP),))[0]
assert set(tool.allowed_tools or ()) == {"lookup_unit_price", "list_categories"}
assert tool.request_timeout == 15
def test_credential_is_read_from_the_environment_at_build_time(monkeypatch) -> None:
"""The config names an env var; the value never lives in the file."""
monkeypatch.setenv("PRISREGISTER_TOKEN", "s3cret")
cfg = McpServerConfig(**{**_HTTP, "credential_env": "PRISREGISTER_TOKEN"})
tool = build_mcp_tools((cfg,))[0]
assert isinstance(tool, MCPStreamableHTTPTool) # built without raising
def test_missing_credential_env_refuses(monkeypatch) -> None:
"""A named-but-unset credential refuses rather than calling the service unauthenticated —
an anonymous call can succeed with the wrong scope, which is worse than not calling."""
monkeypatch.delenv("PRISREGISTER_TOKEN", raising=False)
cfg = McpServerConfig(**{**_HTTP, "credential_env": "PRISREGISTER_TOKEN"})
with pytest.raises(ValueError, match="PRISREGISTER_TOKEN"):
build_mcp_tools((cfg,))
# --- the egress declaration --------------------------------------------------------------------
def test_service_labels_name_every_server_and_its_allowed_tools() -> None:
"""These strings go straight into the run announcement, so an operator can see WHAT will be
contacted and WHICH tools are permitted before anything is spent."""
labels = service_labels((McpServerConfig(**_STDIO), McpServerConfig(**_HTTP)))
assert labels == (
"maalerdata (read_meter)",
"prisregister (lookup_unit_price, list_categories)",
)