Krav 3, and the operator chose the run path explicitly: the external service must be reachable WHILE the run works, not only when documents are ingested. Until now the run path had one in-process tool against a local folder — and on the bundle path the agents had no tools at all. MAF already ships the client (MCPStdioTool / MCPStreamableHTTPTool, verified in the pinned 1.9.0 with allowed_tools and request_timeout), so `mcp_tools.py` owns only what MAF cannot decide for us: which servers a run may contact, which of their tools it may call, how long it waits, and where the credential comes from. This is a DIFFERENT seam from ingest_mcp.py on purpose — that one pulls source documents before a run and speaks to null-argument tools. Same protocol, different job. Every refusal is a live hazard, not tidiness. An empty allowlist would let the far end decide what the agents may call, so naming the tools is mandatory. A non-positive timeout is an unbounded wait against a third party. An unknown field is refused rather than ignored, which is also what keeps a literal secret from being parked in the config — there is no field for one, only the NAME of an env var. A named-but-unset credential refuses instead of calling anonymously, because an anonymous call can succeed with the wrong scope. Egress is declared, always. Every server and permitted tool is named in the run announcement before the first call — including when no --mandate is given, which was a real hole: the announcement only printed with a commission, so configuring servers without one would have contacted third parties with nothing printed at all. --live-dry-run still opens nothing, because the tools are entered after the dry-run cut: the promise to stop before the first call now covers egress too. Threaded through BOTH modes. A flag accepted in one mode and silently dropped in the other is the defect class this CLI refuses by name. Load-bearing MEASURED against the whole 744-test suite, four mutations all red: build the tools but never hand them to the agents (2) · never enter the AsyncExitStack, so they are constructed and useless (1) · never declare the egress (2) · drop the allowlist on the built client (1). Two live docs claimed MCP was unwired in the run path; both corrected rather than left to rot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ULCqjLF61rehj5cZmdUoR3
273 lines
9.6 KiB
Python
273 lines
9.6 KiB
Python
"""The CLI door onto the mandate (Trekk A6): ``--mandate <file.json>``.
|
|
|
|
Krav 1 and 2 are only real if a domain expert can reach them from a command line. This pins the
|
|
three visible halves:
|
|
|
|
1. the commission is loaded FAIL-FAST — a missing or malformed mandate refuses the run (rc 1),
|
|
because a silently degraded commission would make the settlement describe work nobody ordered;
|
|
2. the run ANNOUNCES what it will do BEFORE it does it, and the announcement precedes the outcome
|
|
line in the output (an after-the-fact statement of intent is not a statement of intent);
|
|
3. the run SETTLES against that commission afterwards, naming every approach.
|
|
|
|
The control asserts both blocks are absent without ``--mandate`` — so no assertion above can pass
|
|
on a constant the CLI prints anyway.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import shutil
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from portfolio_optimiser import run
|
|
|
|
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
|
|
|
_VALID_PROPOSAL = (
|
|
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
|
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
|
|
)
|
|
_CHECKER_APPROVE = "Tallene er innenfor feasibelt område og resonnementet holder. VERDICT: APPROVE"
|
|
|
|
_MANDATE = {
|
|
"objective": "Kutt energikostnad uten ombygging i 2026.",
|
|
"approaches": [
|
|
{
|
|
"id": "led-retrofit",
|
|
"label": "LED-retrofit av kontorbelysning",
|
|
"description": "Drift mener armaturene er originale.",
|
|
}
|
|
],
|
|
"allow_own_proposals": False,
|
|
"success_criteria": "Minst ett tiltak som passerer validatoren.",
|
|
}
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _isolate_model_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
|
|
monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False)
|
|
|
|
|
|
@pytest.fixture()
|
|
def bundle(tmp_path: Path) -> Path:
|
|
"""A throwaway COPY — the commons-owned fixture is never mutated by a test."""
|
|
dst = tmp_path / "bundle"
|
|
shutil.copytree(BUNDLE_DIR, dst)
|
|
return dst
|
|
|
|
|
|
@pytest.fixture()
|
|
def replies_file(tmp_path: Path) -> Path:
|
|
path = tmp_path / "replies.json"
|
|
path.write_text(
|
|
json.dumps({"proposer": _VALID_PROPOSAL, "checker": _CHECKER_APPROVE}), encoding="utf-8"
|
|
)
|
|
return path
|
|
|
|
|
|
@pytest.fixture()
|
|
def mandate_file(tmp_path: Path) -> Path:
|
|
path = tmp_path / "mandate.json"
|
|
path.write_text(json.dumps(_MANDATE), encoding="utf-8")
|
|
return path
|
|
|
|
|
|
def _argv(bundle: Path, replies: Path, *extra: str) -> list[str]:
|
|
return [
|
|
"BYGG-KONTOR-NORD",
|
|
"--docs-dir",
|
|
str(bundle),
|
|
"--bundle-dir",
|
|
str(bundle),
|
|
"--scripted-replies",
|
|
str(replies),
|
|
*extra,
|
|
]
|
|
|
|
|
|
def test_missing_mandate_refuses_the_run(bundle, replies_file, capsys) -> None:
|
|
"""A mandate that is not there refuses — never treated as 'no mandate given'."""
|
|
rc = run.main(_argv(bundle, replies_file, "--mandate", str(bundle / "nope.json")))
|
|
assert rc == 1
|
|
assert "refused" in capsys.readouterr().err.lower()
|
|
|
|
|
|
def test_malformed_mandate_refuses_the_run(bundle, replies_file, tmp_path, capsys) -> None:
|
|
"""A malformed commission refuses too: authoritative startup config, not a tolerant inbox."""
|
|
bad = tmp_path / "bad.json"
|
|
bad.write_text('{"approaches": []}', encoding="utf-8") # no objective
|
|
rc = run.main(_argv(bundle, replies_file, "--mandate", str(bad)))
|
|
assert rc == 1
|
|
assert "refused" in capsys.readouterr().err.lower()
|
|
|
|
|
|
def test_run_announces_the_mandate_before_it_runs(
|
|
bundle, replies_file, mandate_file, capsys
|
|
) -> None:
|
|
"""The announcement names the objective and every commissioned approach, and it appears BEFORE
|
|
the run's outcome line — intent stated up front, not reported after the fact."""
|
|
rc = run.main(_argv(bundle, replies_file, "--mandate", str(mandate_file)))
|
|
out = capsys.readouterr().out
|
|
assert rc == 0, out
|
|
assert "Run mandate for BYGG-KONTOR-NORD" in out
|
|
assert "Kutt energikostnad uten ombygging i 2026." in out
|
|
assert "led-retrofit" in out
|
|
assert out.index("Run mandate for") < out.index("BYGG-KONTOR-NORD: ")
|
|
|
|
|
|
def test_run_settles_against_the_mandate_afterwards(
|
|
bundle, replies_file, mandate_file, capsys
|
|
) -> None:
|
|
"""The settlement names every commissioned approach and its status, AFTER the run."""
|
|
rc = run.main(_argv(bundle, replies_file, "--mandate", str(mandate_file)))
|
|
out = capsys.readouterr().out
|
|
assert rc == 0, out
|
|
assert "Mandate outcome" in out
|
|
assert "led-retrofit" in out.split("Mandate outcome")[1]
|
|
assert out.index("Run mandate for") < out.index("Mandate outcome")
|
|
|
|
|
|
def test_dry_run_announces_without_calling_a_model(bundle, mandate_file, capsys) -> None:
|
|
"""``--live-dry-run`` shows what the run WOULD do — the announcement is offline by
|
|
construction, so an operator can inspect a commission before paying for it."""
|
|
rc = run.main(
|
|
[
|
|
"BYGG-KONTOR-NORD",
|
|
"--docs-dir",
|
|
str(bundle),
|
|
"--bundle-dir",
|
|
str(bundle),
|
|
"--mandate",
|
|
str(mandate_file),
|
|
"--live-dry-run",
|
|
]
|
|
)
|
|
out = capsys.readouterr().out
|
|
assert rc == 0, out
|
|
assert "Run mandate for BYGG-KONTOR-NORD" in out
|
|
assert "LIVE-DRY-RUN OK" in out
|
|
# A dry run stops before the first model call, so there is nothing to settle.
|
|
assert "Mandate outcome" not in out
|
|
|
|
|
|
def test_portfolio_mode_applies_the_mandate_to_every_project(
|
|
replies_file, mandate_file, capsys
|
|
) -> None:
|
|
"""The mandate applies to every project in a portfolio pass, and each project settles for it
|
|
separately — a threading line with no test of its own is the green-but-dead trap this repo
|
|
was bitten by in Fase 2."""
|
|
rc = run.main(
|
|
[
|
|
"--portfolio",
|
|
"--scripted-replies",
|
|
str(replies_file),
|
|
"--mandate",
|
|
str(mandate_file),
|
|
]
|
|
)
|
|
out = capsys.readouterr().out
|
|
assert rc == 0, out
|
|
assert "Run mandate for the portfolio" in out
|
|
settlements = out.count("Mandate outcome")
|
|
assert settlements >= 2, f"expected one settlement per project, saw {settlements}"
|
|
assert "led-retrofit" in out.split("Mandate outcome")[1]
|
|
|
|
|
|
def test_no_mandate_prints_neither_block(bundle, replies_file, capsys) -> None:
|
|
"""CONTROL: without ``--mandate`` the CLI output is the pre-Trekk-A one — no announcement, no
|
|
settlement. Every assertion above would otherwise be able to pass on constant output."""
|
|
rc = run.main(_argv(bundle, replies_file))
|
|
out = capsys.readouterr().out
|
|
assert rc == 0, out
|
|
assert "Run mandate for" not in out
|
|
assert "Mandate outcome" not in out
|
|
|
|
|
|
# --- the egress declaration (Trekk B3): what a run will contact, before it contacts it ----------
|
|
|
|
_MCP = {
|
|
"servers": [
|
|
{
|
|
"name": "prisregister",
|
|
"transport": "http",
|
|
"url": "https://intern.example/mcp",
|
|
"allowed_tools": ["lookup_unit_price"],
|
|
"timeout_seconds": 15,
|
|
}
|
|
]
|
|
}
|
|
|
|
|
|
@pytest.fixture()
|
|
def mcp_file(tmp_path: Path) -> Path:
|
|
path = tmp_path / "mcp.json"
|
|
path.write_text(json.dumps(_MCP), encoding="utf-8")
|
|
return path
|
|
|
|
|
|
def test_configured_servers_are_named_before_the_run(
|
|
bundle, mandate_file, mcp_file, capsys
|
|
) -> None:
|
|
"""Every server and every permitted tool is named in the announcement — a run never reaches a
|
|
service it did not declare. ``--live-dry-run`` keeps the assertion offline."""
|
|
rc = run.main(
|
|
[
|
|
"BYGG-KONTOR-NORD",
|
|
"--docs-dir",
|
|
str(bundle),
|
|
"--bundle-dir",
|
|
str(bundle),
|
|
"--mandate",
|
|
str(mandate_file),
|
|
"--mcp-config",
|
|
str(mcp_file),
|
|
"--live-dry-run",
|
|
]
|
|
)
|
|
out = capsys.readouterr().out
|
|
assert rc == 0, out
|
|
assert "prisregister (lookup_unit_price)" in out
|
|
assert "no external services" not in out
|
|
|
|
|
|
def test_egress_is_declared_even_without_a_mandate(bundle, mcp_file, capsys) -> None:
|
|
"""The declaration cannot depend on a mandate being present. Without this, configuring servers
|
|
and omitting ``--mandate`` would contact third parties with nothing printed at all — silent
|
|
egress, which is the one thing this repo's data rules forbid outright."""
|
|
rc = run.main(
|
|
[
|
|
"BYGG-KONTOR-NORD",
|
|
"--docs-dir",
|
|
str(bundle),
|
|
"--bundle-dir",
|
|
str(bundle),
|
|
"--mcp-config",
|
|
str(mcp_file),
|
|
"--live-dry-run",
|
|
]
|
|
)
|
|
out = capsys.readouterr().out
|
|
assert rc == 0, out
|
|
assert "prisregister (lookup_unit_price)" in out
|
|
|
|
|
|
def test_malformed_mcp_config_refuses_the_run(bundle, replies_file, tmp_path, capsys) -> None:
|
|
"""An egress config that cannot be read refuses — degrading it to 'no external services' would
|
|
make the announcement describe a run nobody configured."""
|
|
bad = tmp_path / "mcp.json"
|
|
bad.write_text('{"servers": [{"name": "x", "transport": "http"}]}', encoding="utf-8")
|
|
rc = run.main(_argv(bundle, replies_file, "--mcp-config", str(bad)))
|
|
assert rc == 1
|
|
assert "refused" in capsys.readouterr().err.lower()
|
|
|
|
|
|
def test_no_mcp_config_contacts_nothing(bundle, replies_file, mandate_file, capsys) -> None:
|
|
"""CONTROL: without ``--mcp-config`` the announcement says so explicitly. An omitted line would
|
|
read the same as an unchecked one."""
|
|
rc = run.main(_argv(bundle, replies_file, "--mandate", str(mandate_file)))
|
|
out = capsys.readouterr().out
|
|
assert rc == 0, out
|
|
assert "no external services" in out
|