feat(mcp): serve OKF bundles over MCP in two shapes, plus the generic skill
The eval was written RED at `5f1772e` with no server in the tree. This is the
capability it was written against.
`okf mcp --bundle <dir>` serves exactly one bundle, whose tools take no bundle
argument. `okf mcp --root <dir>` (repeatable) serves every bundle under the
roots and knows NONE of them by name. Four tools -- `okf_list`,
`okf_describe`, `okf_ask`, `okf_fetch` -- each carrying its reason in the
description a client actually reads.
Gate today: 1 (7/7) - 2 (83/181) - 3 (4/4) - 4 (9/9) - 5 (3/3) - 6 (6/6),
`GATE RED: rows 2`, exit 1.
THE PROTOCOL IS STDLIB, AND THAT IS THE PACKAGING INVARIANT KEPT RATHER THAN
A TASTE. An MCP SDK would be this package's second runtime dependency on the
DEFAULT install path, for four JSON-RPC methods and a newline framing, and
`test_the_only_runtime_dependency_is_the_security_boundary` pins that list
literally. Chosen hand-written because the surface needed is `initialize`,
`notifications/initialized`, `tools/list` and `tools/call`; `uv.lock` is
untouched.
NOTHING IS CACHED ACROSS CALLS, and row 3 is why. Every call re-walks the
roots and recomputes `bundle_ref`, so a bundle added, removed or rebuilt while
the process runs is seen by the next call with no restart, no configuration
edit and no code change -- 9 of 9 discovery checks over three bundles written
while the server was serving. The cost is paid per call and is published
rather than hidden: 0.75 s for the identity of a 2 756-concept bundle, 5.6 s
for one ask, 4 min 13 s for row 2's full run over four bundles.
CONTAINMENT IS TWO INDEPENDENT CHECKS: the bundle's own index must name the
concept, AND `connectors.safe_resolve` must place it inside the bundle. A
mutant removing either one alone still refuses -- with a DIFFERENT code, which
row 6 asserts by name -- and one removing both is killed. Row 6 declares a
code set per case because its first run had the 10 MB concept refused as
`concept_unknown`: the fixture had not named the file in the index, so the
size ceiling never ran and the row was green for a reason unrelated to the
attack.
`okf card <bundle>` and `okf skill --generic` are the one-to-many skill
candidate. The card is DERIVED on every run and never written into the bundle:
storing it would move the bytes of all six `examples/*/expected-bundle` trees
(23 files compared byte-for-byte) and of the pinned reference bundle, to keep
something recomputable in under a second, and a stored card is one more
artefact that can disagree with what is beside it. Measured here rather than
taken from the order: two per-bundle skills are identical on 280 of 312 and
310 lines; the 62 that differ are identity, concept count, the
conditional-field table, the whole-bundle cost and the breaking point. The
generic skill carries none of them, and `render_generic()` takes no argument,
so there is no bundle it could have read.
Row 2 decomposes into three numbers and the middle one is the finding: 99 of
181 (bundle, anchor) pairs are present in the bundles at all, 83 of those 99
were reached, and 0 of 83 were met by `okf_fetch` on the anchor as a concept
id. The set's anchors and this library's concept ids are different
vocabularies, so every pair met was met through the ranker -- 83 is a FLOOR on
the ceiling, never the ceiling.
13 mutants in a scratch copy, never in the working tree: 12 killed, 1 survived
with its mechanism printed, 0 errors, control green first. Suite 2323 passed,
2 skipped. The architecture choice between the two shapes is the OPERATOR's;
these rows are its input. Report: docs/2026-09-20-mcp-to-varianter.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
5f1772e832
commit
df5a1183c9
10 changed files with 1823 additions and 59 deletions
196
tests/test_mcp_server.py
Normal file
196
tests/test_mcp_server.py
Normal file
|
|
@ -0,0 +1,196 @@
|
|||
"""The MCP surface itself (`src/llm_ingestion_okf/mcp_server.py`).
|
||||
|
||||
`tools/okf_mcp_gate.py` measures this module over a real protocol and is the
|
||||
eval it was built against. These tests hold the pieces the gate reaches only
|
||||
through a verdict: what discovery does with a symlink, what the card does NOT
|
||||
do to the bundle, and the property that makes the generic skill's whole claim
|
||||
checkable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from llm_ingestion_okf import mcp_server, skill
|
||||
|
||||
TOOLS = Path(__file__).resolve().parents[1] / "tools"
|
||||
sys.path.insert(0, str(TOOLS))
|
||||
|
||||
import okf_mcp_gate as gate # noqa: E402
|
||||
|
||||
GOLDEN = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "examples"
|
||||
/ "ingest-golden-segmented-okf-v0-2"
|
||||
/ "expected-bundle"
|
||||
)
|
||||
|
||||
|
||||
def _tree(root: Path) -> dict[str, str]:
|
||||
return {
|
||||
str(path.relative_to(root)): hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
for path in sorted(root.rglob("*"))
|
||||
if path.is_file()
|
||||
}
|
||||
|
||||
|
||||
def test_the_card_is_derived_and_writes_nothing_into_the_bundle(tmp_path: Path) -> None:
|
||||
"""The whole reason the card is not a file in the bundle.
|
||||
|
||||
A stored card would move the bytes of all six `examples/*/expected-bundle`
|
||||
trees and of the pinned reference bundle, and would be one more artefact
|
||||
that can disagree with what is beside it.
|
||||
"""
|
||||
bundle = tmp_path / "b"
|
||||
bundle.mkdir()
|
||||
for source in GOLDEN.rglob("*"):
|
||||
target = bundle / source.relative_to(GOLDEN)
|
||||
if source.is_dir():
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
else:
|
||||
target.write_bytes(source.read_bytes())
|
||||
before = _tree(bundle)
|
||||
first = mcp_server.card(bundle, profile=mcp_server.okf_consume.DEFAULT_PROFILE)
|
||||
second = mcp_server.card(bundle, profile=mcp_server.okf_consume.DEFAULT_PROFILE)
|
||||
assert _tree(bundle) == before
|
||||
assert first == second
|
||||
assert first["concept_count"] == 3
|
||||
assert first["ref"].startswith("sha256-tree:")
|
||||
|
||||
|
||||
def test_the_card_command_prints_json_a_reader_can_parse() -> None:
|
||||
"""The generic skill tells its reader to run this. A command whose output
|
||||
could not be read back would make that instruction decorative."""
|
||||
run = subprocess.run(
|
||||
[sys.executable, "-m", "llm_ingestion_okf.cli", "card", str(GOLDEN)],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
assert run.returncode == 0, run.stderr
|
||||
payload = json.loads(run.stdout)
|
||||
assert payload["bundle_id"] == "b-golden-segmented-okf-v0-2"
|
||||
assert set(payload) >= {"ref", "concept_count", "conditional_fields", "whole_bundle_bytes"}
|
||||
|
||||
|
||||
def test_discovery_never_descends_a_symlink(tmp_path: Path) -> None:
|
||||
"""Driven from both sides: the real directory IS found, the link to it is
|
||||
not. Without the control, a rule that found nothing at all would pass."""
|
||||
outside = tmp_path / "outside"
|
||||
gate.write_bundle(outside / "secret", "secret-notes", [("x", "X", "y")])
|
||||
root = tmp_path / "root"
|
||||
gate.write_bundle(root / "real", "real-notes", [("x", "X", "y")])
|
||||
(root / "linked").symlink_to(outside / "secret", target_is_directory=True)
|
||||
|
||||
found = mcp_server.discover([root])
|
||||
assert [served.bundle_id for served in found.bundles] == ["real-notes"]
|
||||
assert mcp_server.discover([outside]).bundles[0].bundle_id == "secret-notes"
|
||||
|
||||
|
||||
def test_a_directory_that_cannot_be_read_as_a_bundle_is_reported_not_skipped(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""An absence with no denominator is not a boundary. A broken manifest that
|
||||
simply vanished from the list would be indistinguishable from a bundle that
|
||||
was never there."""
|
||||
root = tmp_path / "root"
|
||||
gate.write_bundle(root / "good", "good-notes", [("x", "X", "y")])
|
||||
gate.write_bundle(root / "bad", "bad-notes", [("x", "X", "y")])
|
||||
(root / "bad" / "index.md").write_bytes(b"\xff\xfe\x00")
|
||||
found = mcp_server.discover([root])
|
||||
assert [served.bundle_id for served in found.bundles] == ["good-notes"]
|
||||
assert [entry.path for entry in found.unreadable] == ["bad"]
|
||||
|
||||
|
||||
def test_the_one_to_one_server_finds_the_bundle_it_was_started_on(tmp_path: Path) -> None:
|
||||
"""The defect the gate found on this module's first build: `--bundle` points
|
||||
AT a bundle, and discovery that only looked at children found none."""
|
||||
gate.write_bundle(tmp_path / "b", "solo-notes", [("x", "X", "y")])
|
||||
surface = mcp_server.build_surface(bundle=tmp_path / "b", roots=[])
|
||||
assert surface.resolve(None).bundle_id == "solo-notes"
|
||||
|
||||
|
||||
def test_a_one_to_many_call_naming_no_bundle_is_refused_rather_than_guessed(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Picking one would make an answer's provenance depend on directory order."""
|
||||
root = tmp_path / "root"
|
||||
gate.write_bundle(root / "a", "a-notes", [("x", "X", "y")])
|
||||
gate.write_bundle(root / "b", "b-notes", [("x", "X", "y")])
|
||||
surface = mcp_server.build_surface(bundle=None, roots=[root])
|
||||
with pytest.raises(mcp_server.ToolError) as raised:
|
||||
surface.resolve(None)
|
||||
assert raised.value.code == "bundle_id_required"
|
||||
|
||||
|
||||
def test_the_one_to_one_server_offers_no_listing_tool(tmp_path: Path) -> None:
|
||||
"""A tool that always returns the same single row invites a client to treat
|
||||
discovery as available where the deployment does not have it."""
|
||||
gate.write_bundle(tmp_path / "b", "solo-notes", [("x", "X", "y")])
|
||||
one = mcp_server.build_surface(bundle=tmp_path / "b", roots=[])
|
||||
many = mcp_server.build_surface(bundle=None, roots=[tmp_path])
|
||||
assert [tool.name for tool in mcp_server.tools(one)] == list(gate.REQUIRED_TOOLS["one-to-one"])
|
||||
assert [tool.name for tool in mcp_server.tools(many)] == list(
|
||||
gate.REQUIRED_TOOLS["one-to-many"]
|
||||
)
|
||||
|
||||
|
||||
def test_every_tool_carries_a_written_reason() -> None:
|
||||
"""The order's rule: a tool with no reason written down is a tool nobody
|
||||
has to justify keeping."""
|
||||
surface = mcp_server.Surface(
|
||||
(Path("/nonexistent"),), None, mcp_server.okf_consume.DEFAULT_PROFILE
|
||||
)
|
||||
for tool in mcp_server.tools(surface):
|
||||
assert "Exists because" in tool.description, tool.name
|
||||
assert len(tool.description) > 200, tool.name
|
||||
|
||||
|
||||
# --- the one-to-many skill candidate ------------------------------------------
|
||||
|
||||
|
||||
def test_the_generic_skill_carries_no_bundles_identity(tmp_path: Path) -> None:
|
||||
"""The property that makes the candidate's whole claim checkable: it takes
|
||||
no argument, so there is no bundle it could have read.
|
||||
|
||||
Controlled against a per-bundle skill, which must carry exactly what this
|
||||
one does not -- without that control a test asserting an absence would pass
|
||||
on an empty string.
|
||||
"""
|
||||
generic = skill.render_generic()
|
||||
skill.generate(GOLDEN, out=tmp_path / "per-bundle", question="krav", force=True)
|
||||
per_bundle = (tmp_path / "per-bundle" / "SKILL.md").read_text(encoding="utf-8")
|
||||
|
||||
bundle_id = "b-golden-segmented-okf-v0-2"
|
||||
ref = mcp_server.okf_consume.bundle_ref(GOLDEN)
|
||||
assert bundle_id in per_bundle and ref in per_bundle
|
||||
assert bundle_id not in generic and ref not in generic
|
||||
assert skill.CARD_COMMAND in generic
|
||||
|
||||
|
||||
def test_the_generic_skill_is_the_same_bytes_for_anyone(tmp_path: Path) -> None:
|
||||
"""Two calls, and a written file, all identical. A rebuild of any bundle
|
||||
cannot make this artefact wrong, which is stronger than refusing loudly."""
|
||||
first = skill.render_generic()
|
||||
written = skill.generate_generic(out=tmp_path / "g")
|
||||
assert written.read_text(encoding="utf-8") == first == skill.render_generic()
|
||||
|
||||
|
||||
def test_the_generic_skill_leaves_no_per_corpus_hole() -> None:
|
||||
"""A hole left in a generic document is a number the reader is invited to
|
||||
invent -- which is the unfilled template's own defect."""
|
||||
assert skill._PLACEHOLDER.findall(skill.render_generic()) == []
|
||||
|
||||
|
||||
def test_the_generic_skill_keeps_every_section_the_checker_reads() -> None:
|
||||
from llm_ingestion_okf import contract_check
|
||||
|
||||
text = skill.render_generic()
|
||||
for section in contract_check.REQUIRED_SECTIONS:
|
||||
assert f"## {section}" in text, section
|
||||
Loading…
Add table
Add a link
Reference in a new issue