open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(mcp): serve OKF bundles over MCP in two shapes, plus the generic skill

Browse source 
The eval was written RED at `5f1772e` with no server in the tree. This is the
capability it was written against.

`okf mcp --bundle <dir>` serves exactly one bundle, whose tools take no bundle
argument. `okf mcp --root <dir>` (repeatable) serves every bundle under the
roots and knows NONE of them by name. Four tools -- `okf_list`,
`okf_describe`, `okf_ask`, `okf_fetch` -- each carrying its reason in the
description a client actually reads.

Gate today: 1 (7/7) - 2 (83/181) - 3 (4/4) - 4 (9/9) - 5 (3/3) - 6 (6/6),
`GATE RED: rows 2`, exit 1.

THE PROTOCOL IS STDLIB, AND THAT IS THE PACKAGING INVARIANT KEPT RATHER THAN
A TASTE. An MCP SDK would be this package's second runtime dependency on the
DEFAULT install path, for four JSON-RPC methods and a newline framing, and
`test_the_only_runtime_dependency_is_the_security_boundary` pins that list
literally. Chosen hand-written because the surface needed is `initialize`,
`notifications/initialized`, `tools/list` and `tools/call`; `uv.lock` is
untouched.

NOTHING IS CACHED ACROSS CALLS, and row 3 is why. Every call re-walks the
roots and recomputes `bundle_ref`, so a bundle added, removed or rebuilt while
the process runs is seen by the next call with no restart, no configuration
edit and no code change -- 9 of 9 discovery checks over three bundles written
while the server was serving. The cost is paid per call and is published
rather than hidden: 0.75 s for the identity of a 2 756-concept bundle, 5.6 s
for one ask, 4 min 13 s for row 2's full run over four bundles.

CONTAINMENT IS TWO INDEPENDENT CHECKS: the bundle's own index must name the
concept, AND `connectors.safe_resolve` must place it inside the bundle. A
mutant removing either one alone still refuses -- with a DIFFERENT code, which
row 6 asserts by name -- and one removing both is killed. Row 6 declares a
code set per case because its first run had the 10 MB concept refused as
`concept_unknown`: the fixture had not named the file in the index, so the
size ceiling never ran and the row was green for a reason unrelated to the
attack.

`okf card <bundle>` and `okf skill --generic` are the one-to-many skill
candidate. The card is DERIVED on every run and never written into the bundle:
storing it would move the bytes of all six `examples/*/expected-bundle` trees
(23 files compared byte-for-byte) and of the pinned reference bundle, to keep
something recomputable in under a second, and a stored card is one more
artefact that can disagree with what is beside it. Measured here rather than
taken from the order: two per-bundle skills are identical on 280 of 312 and
310 lines; the 62 that differ are identity, concept count, the
conditional-field table, the whole-bundle cost and the breaking point. The
generic skill carries none of them, and `render_generic()` takes no argument,
so there is no bundle it could have read.

Row 2 decomposes into three numbers and the middle one is the finding: 99 of
181 (bundle, anchor) pairs are present in the bundles at all, 83 of those 99
were reached, and 0 of 83 were met by `okf_fetch` on the anchor as a concept
id. The set's anchors and this library's concept ids are different
vocabularies, so every pair met was met through the ranker -- 83 is a FLOOR on
the ceiling, never the ceiling.

13 mutants in a scratch copy, never in the working tree: 12 killed, 1 survived
with its mechanism printed, 0 errors, control green first. Suite 2323 passed,
2 skipped. The architecture choice between the two shapes is the OPERATOR's;
these rows are its input. Report: docs/2026-09-20-mcp-to-varianter.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-20 10:25:55 +02:00
commit df5a1183c9
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
10 changed files with 1823 additions and 59 deletions
Download patch file Download diff file Expand all files Collapse all files

77
tests/test_mcp_gate.py
View file

@ -67,7 +67,7 @@ def test_every_pinned_denominator_is_recounted_here() -> None:
)
assert gate.DISCOVERY_BUNDLES * len(gate.DISCOVERY_CHECKS) == 3 * 3
assert len(gate.CROSS_CHECKS) == 3
assert gate.HOSTILE_CASES == (
assert tuple(gate.HOSTILE_CASES) == (
"traversal-in-bundle-id",
"traversal-in-concept-id",
"symlink-out-of-root",
@ -75,6 +75,9 @@ def test_every_pinned_denominator_is_recounted_here() -> None:
"oversized-concept",
"unknown-bundle-id",
)
# The 10 MB case has exactly one acceptable refusal. Anything else means
# the ceiling did not run -- the false green this row shipped with once.
assert gate.HOSTILE_CASES["oversized-concept"] == frozenset({"concept_too_large"})
assert gate.OVERSIZED_BYTES == 10 * 1024 * 1024
@ -128,7 +131,7 @@ def test_a_question_set_whose_bytes_moved_is_a_usage_error_not_a_red_row(
payload = {
"sporsmal": [
{
"bundle": "bridge-notes",
"bundles": ["bridge-notes"],
"atomer": [{"kilde_anker": "spennvidde", "kilde_sitat": "24 meter"}],
}
]
@ -171,6 +174,15 @@ def test_an_older_freeze_version_is_refused_rather_than_measured(scratch: Path)
gate.read_anchor_set(questions, freeze, want_version=4)
def test_a_refusal_code_is_read_from_the_message_a_client_sees() -> None:
"""Driven from both sides: a coded refusal yields its code, an uncoded one
yields the empty string rather than a plausible guess."""
assert gate.refusal_code(gate.RpcError(-32000, "refused (concept_too_large): 10 MB")) == (
"concept_too_large"
)
assert gate.refusal_code(gate.RpcError(-32000, "something went wrong")) == ""
def test_the_gate_reads_bundle_and_concept_ids_at_any_depth() -> None:
"""An `ask` answer carries one id per excerpt and a `list` answer one per
bundle. A rule reading the top level only would score the shape."""
@ -222,3 +234,64 @@ def test_one_documented_sequence_answers_from_two_bundles(scratch: Path) -> None
def test_hostile_input_is_refused_out_loud(scratch: Path) -> None:
row = _row(6, scratch)
assert row.k == row.m == 6, f"{row.k} of {row.m}: {row.details}"
def test_row_twos_denominator_is_the_sets_and_not_what_the_surface_reached(
scratch: Path,
) -> None:
"""The defect this repository has already met twice: a row that counts
against what the run produced closes by producing less.
Three pairs are declared and one of them names an anchor no bundle carries,
so a denominator taken from the run would read `2 of 2` and call the gate
satisfied. Driven from both sides -- the reachable pairs really are
reachable, so a `0 of 3` would be a different defect.
"""
import hashlib
bundles = gate.corpus(scratch / "base")
questions = scratch / "sporsmal.json"
questions.write_text(
json.dumps(
{
"sporsmal": [
{
"bundles": ["bridge-notes"],
"atomer": [
{"kilde_anker": "spennvidde", "kilde_sitat": "spennvidde 24 meter"},
{"kilde_anker": "rekkverk", "kilde_sitat": "1,2 meter hoeyt"},
{
"kilde_anker": "finnes-ikke",
"kilde_sitat": "dette staar ingen steder",
},
],
}
]
}
),
encoding="utf-8",
)
freeze = scratch / "frys.json"
freeze.write_text(
json.dumps(
{
"versjon": 4,
"sha256": {"sporsmal.json": hashlib.sha256(questions.read_bytes()).hexdigest()},
}
),
encoding="utf-8",
)
anchors = gate.read_anchor_set(questions, freeze, want_version=4)
assert len(anchors.pairs) == 3
rows = {
row.number: row
for row in gate.evaluate(
scratch / "run",
anchors=anchors,
real={"bridge-notes": bundles["bridge-notes"]},
)
}
row = rows[2]
assert row.m == 3, f"the denominator came from the run: {row.k} of {row.m}"
assert row.k == 2, f"{row.k} of {row.m}: {row.details}"
assert row.status == gate.RED

196
tests/test_mcp_server.py Normal file
View file

@ -0,0 +1,196 @@
"""The MCP surface itself (`src/llm_ingestion_okf/mcp_server.py`).
`tools/okf_mcp_gate.py` measures this module over a real protocol and is the
eval it was built against. These tests hold the pieces the gate reaches only
through a verdict: what discovery does with a symlink, what the card does NOT
do to the bundle, and the property that makes the generic skill's whole claim
checkable.
"""
from __future__ import annotations
import hashlib
import json
import subprocess
import sys
from pathlib import Path
import pytest
from llm_ingestion_okf import mcp_server, skill
TOOLS = Path(__file__).resolve().parents[1] / "tools"
sys.path.insert(0, str(TOOLS))
import okf_mcp_gate as gate # noqa: E402
GOLDEN = (
Path(__file__).resolve().parents[1]
/ "examples"
/ "ingest-golden-segmented-okf-v0-2"
/ "expected-bundle"
)
def _tree(root: Path) -> dict[str, str]:
return {
str(path.relative_to(root)): hashlib.sha256(path.read_bytes()).hexdigest()
for path in sorted(root.rglob("*"))
if path.is_file()
}
def test_the_card_is_derived_and_writes_nothing_into_the_bundle(tmp_path: Path) -> None:
"""The whole reason the card is not a file in the bundle.
A stored card would move the bytes of all six `examples/*/expected-bundle`
trees and of the pinned reference bundle, and would be one more artefact
that can disagree with what is beside it.
"""
bundle = tmp_path / "b"
bundle.mkdir()
for source in GOLDEN.rglob("*"):
target = bundle / source.relative_to(GOLDEN)
if source.is_dir():
target.mkdir(parents=True, exist_ok=True)
else:
target.write_bytes(source.read_bytes())
before = _tree(bundle)
first = mcp_server.card(bundle, profile=mcp_server.okf_consume.DEFAULT_PROFILE)
second = mcp_server.card(bundle, profile=mcp_server.okf_consume.DEFAULT_PROFILE)
assert _tree(bundle) == before
assert first == second
assert first["concept_count"] == 3
assert first["ref"].startswith("sha256-tree:")
def test_the_card_command_prints_json_a_reader_can_parse() -> None:
"""The generic skill tells its reader to run this. A command whose output
could not be read back would make that instruction decorative."""
run = subprocess.run(
[sys.executable, "-m", "llm_ingestion_okf.cli", "card", str(GOLDEN)],
capture_output=True,
text=True,
check=False,
)
assert run.returncode == 0, run.stderr
payload = json.loads(run.stdout)
assert payload["bundle_id"] == "b-golden-segmented-okf-v0-2"
assert set(payload) >= {"ref", "concept_count", "conditional_fields", "whole_bundle_bytes"}
def test_discovery_never_descends_a_symlink(tmp_path: Path) -> None:
"""Driven from both sides: the real directory IS found, the link to it is
not. Without the control, a rule that found nothing at all would pass."""
outside = tmp_path / "outside"
gate.write_bundle(outside / "secret", "secret-notes", [("x", "X", "y")])
root = tmp_path / "root"
gate.write_bundle(root / "real", "real-notes", [("x", "X", "y")])
(root / "linked").symlink_to(outside / "secret", target_is_directory=True)
found = mcp_server.discover([root])
assert [served.bundle_id for served in found.bundles] == ["real-notes"]
assert mcp_server.discover([outside]).bundles[0].bundle_id == "secret-notes"
def test_a_directory_that_cannot_be_read_as_a_bundle_is_reported_not_skipped(
tmp_path: Path,
) -> None:
"""An absence with no denominator is not a boundary. A broken manifest that
simply vanished from the list would be indistinguishable from a bundle that
was never there."""
root = tmp_path / "root"
gate.write_bundle(root / "good", "good-notes", [("x", "X", "y")])
gate.write_bundle(root / "bad", "bad-notes", [("x", "X", "y")])
(root / "bad" / "index.md").write_bytes(b"\xff\xfe\x00")
found = mcp_server.discover([root])
assert [served.bundle_id for served in found.bundles] == ["good-notes"]
assert [entry.path for entry in found.unreadable] == ["bad"]
def test_the_one_to_one_server_finds_the_bundle_it_was_started_on(tmp_path: Path) -> None:
"""The defect the gate found on this module's first build: `--bundle` points
AT a bundle, and discovery that only looked at children found none."""
gate.write_bundle(tmp_path / "b", "solo-notes", [("x", "X", "y")])
surface = mcp_server.build_surface(bundle=tmp_path / "b", roots=[])
assert surface.resolve(None).bundle_id == "solo-notes"
def test_a_one_to_many_call_naming_no_bundle_is_refused_rather_than_guessed(
tmp_path: Path,
) -> None:
"""Picking one would make an answer's provenance depend on directory order."""
root = tmp_path / "root"
gate.write_bundle(root / "a", "a-notes", [("x", "X", "y")])
gate.write_bundle(root / "b", "b-notes", [("x", "X", "y")])
surface = mcp_server.build_surface(bundle=None, roots=[root])
with pytest.raises(mcp_server.ToolError) as raised:
surface.resolve(None)
assert raised.value.code == "bundle_id_required"
def test_the_one_to_one_server_offers_no_listing_tool(tmp_path: Path) -> None:
"""A tool that always returns the same single row invites a client to treat
discovery as available where the deployment does not have it."""
gate.write_bundle(tmp_path / "b", "solo-notes", [("x", "X", "y")])
one = mcp_server.build_surface(bundle=tmp_path / "b", roots=[])
many = mcp_server.build_surface(bundle=None, roots=[tmp_path])
assert [tool.name for tool in mcp_server.tools(one)] == list(gate.REQUIRED_TOOLS["one-to-one"])
assert [tool.name for tool in mcp_server.tools(many)] == list(
gate.REQUIRED_TOOLS["one-to-many"]
)
def test_every_tool_carries_a_written_reason() -> None:
"""The order's rule: a tool with no reason written down is a tool nobody
has to justify keeping."""
surface = mcp_server.Surface(
(Path("/nonexistent"),), None, mcp_server.okf_consume.DEFAULT_PROFILE
)
for tool in mcp_server.tools(surface):
assert "Exists because" in tool.description, tool.name
assert len(tool.description) > 200, tool.name
# --- the one-to-many skill candidate ------------------------------------------
def test_the_generic_skill_carries_no_bundles_identity(tmp_path: Path) -> None:
"""The property that makes the candidate's whole claim checkable: it takes
no argument, so there is no bundle it could have read.
Controlled against a per-bundle skill, which must carry exactly what this
one does not -- without that control a test asserting an absence would pass
on an empty string.
"""
generic = skill.render_generic()
skill.generate(GOLDEN, out=tmp_path / "per-bundle", question="krav", force=True)
per_bundle = (tmp_path / "per-bundle" / "SKILL.md").read_text(encoding="utf-8")
bundle_id = "b-golden-segmented-okf-v0-2"
ref = mcp_server.okf_consume.bundle_ref(GOLDEN)
assert bundle_id in per_bundle and ref in per_bundle
assert bundle_id not in generic and ref not in generic
assert skill.CARD_COMMAND in generic
def test_the_generic_skill_is_the_same_bytes_for_anyone(tmp_path: Path) -> None:
"""Two calls, and a written file, all identical. A rebuild of any bundle
cannot make this artefact wrong, which is stronger than refusing loudly."""
first = skill.render_generic()
written = skill.generate_generic(out=tmp_path / "g")
assert written.read_text(encoding="utf-8") == first == skill.render_generic()
def test_the_generic_skill_leaves_no_per_corpus_hole() -> None:
"""A hole left in a generic document is a number the reader is invited to
invent -- which is the unfilled template's own defect."""
assert skill._PLACEHOLDER.findall(skill.render_generic()) == []
def test_the_generic_skill_keeps_every_section_the_checker_reads() -> None:
from llm_ingestion_okf import contract_check
text = skill.render_generic()
for section in contract_check.REQUIRED_SECTIONS:
assert f"## {section}" in text, section