open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(mcp): serve OKF bundles over MCP in two shapes, plus the generic skill

Browse source 
The eval was written RED at `5f1772e` with no server in the tree. This is the
capability it was written against.

`okf mcp --bundle <dir>` serves exactly one bundle, whose tools take no bundle
argument. `okf mcp --root <dir>` (repeatable) serves every bundle under the
roots and knows NONE of them by name. Four tools -- `okf_list`,
`okf_describe`, `okf_ask`, `okf_fetch` -- each carrying its reason in the
description a client actually reads.

Gate today: 1 (7/7) - 2 (83/181) - 3 (4/4) - 4 (9/9) - 5 (3/3) - 6 (6/6),
`GATE RED: rows 2`, exit 1.

THE PROTOCOL IS STDLIB, AND THAT IS THE PACKAGING INVARIANT KEPT RATHER THAN
A TASTE. An MCP SDK would be this package's second runtime dependency on the
DEFAULT install path, for four JSON-RPC methods and a newline framing, and
`test_the_only_runtime_dependency_is_the_security_boundary` pins that list
literally. Chosen hand-written because the surface needed is `initialize`,
`notifications/initialized`, `tools/list` and `tools/call`; `uv.lock` is
untouched.

NOTHING IS CACHED ACROSS CALLS, and row 3 is why. Every call re-walks the
roots and recomputes `bundle_ref`, so a bundle added, removed or rebuilt while
the process runs is seen by the next call with no restart, no configuration
edit and no code change -- 9 of 9 discovery checks over three bundles written
while the server was serving. The cost is paid per call and is published
rather than hidden: 0.75 s for the identity of a 2 756-concept bundle, 5.6 s
for one ask, 4 min 13 s for row 2's full run over four bundles.

CONTAINMENT IS TWO INDEPENDENT CHECKS: the bundle's own index must name the
concept, AND `connectors.safe_resolve` must place it inside the bundle. A
mutant removing either one alone still refuses -- with a DIFFERENT code, which
row 6 asserts by name -- and one removing both is killed. Row 6 declares a
code set per case because its first run had the 10 MB concept refused as
`concept_unknown`: the fixture had not named the file in the index, so the
size ceiling never ran and the row was green for a reason unrelated to the
attack.

`okf card <bundle>` and `okf skill --generic` are the one-to-many skill
candidate. The card is DERIVED on every run and never written into the bundle:
storing it would move the bytes of all six `examples/*/expected-bundle` trees
(23 files compared byte-for-byte) and of the pinned reference bundle, to keep
something recomputable in under a second, and a stored card is one more
artefact that can disagree with what is beside it. Measured here rather than
taken from the order: two per-bundle skills are identical on 280 of 312 and
310 lines; the 62 that differ are identity, concept count, the
conditional-field table, the whole-bundle cost and the breaking point. The
generic skill carries none of them, and `render_generic()` takes no argument,
so there is no bundle it could have read.

Row 2 decomposes into three numbers and the middle one is the finding: 99 of
181 (bundle, anchor) pairs are present in the bundles at all, 83 of those 99
were reached, and 0 of 83 were met by `okf_fetch` on the anchor as a concept
id. The set's anchors and this library's concept ids are different
vocabularies, so every pair met was met through the ranker -- 83 is a FLOOR on
the ceiling, never the ceiling.

13 mutants in a scratch copy, never in the working tree: 12 killed, 1 survived
with its mechanism printed, 0 errors, control green first. Suite 2323 passed,
2 skipped. The architecture choice between the two shapes is the OPERATOR's;
these rows are its input. Report: docs/2026-09-20-mcp-to-varianter.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-20 10:25:55 +02:00
commit df5a1183c9
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
10 changed files with 1823 additions and 59 deletions
Download patch file Download diff file Expand all files Collapse all files

291
tools/okf_mcp_gate.py
View file

@ -42,6 +42,7 @@ import argparse
import hashlib
import json
import os
import re
import subprocess
import sys
import tempfile
@ -104,14 +105,20 @@ CROSS_CHECKS: tuple[str, ...] = (
"documented-sequence",
)
HOSTILE_CASES: tuple[str, ...] = (
"traversal-in-bundle-id",
"traversal-in-concept-id",
"symlink-out-of-root",
"broken-manifest",
"oversized-concept",
"unknown-bundle-id",
)
#: Each hostile case with the refusal CODES that count as the right refusal.
#: A code set rather than a bare "was refused": the first run of this gate had
#: the 10 MB concept refused as `concept_unknown`, because the fixture wrote
#: the file without naming it in the index -- the size ceiling never ran, and
#: the row was green for a reason that had nothing to do with the attack. Two
#: checks giving the same verdict are not the same guarantee.
HOSTILE_CASES: Mapping[str, frozenset[str]] = {
"traversal-in-bundle-id": frozenset({"bundle_unknown", "path_escape", "bundle_id_invalid"}),
"traversal-in-concept-id": frozenset({"concept_unknown", "path_escape"}),
"symlink-out-of-root": frozenset({"bundle_unknown", "path_escape"}),
"broken-manifest": frozenset({"bundle_unreadable"}),
"oversized-concept": frozenset({"concept_too_large"}),
"unknown-bundle-id": frozenset({"bundle_unknown"}),
}
# A concept large enough that reading it whole is a decision rather than an
# accident. The order names 10 MB; the gate writes exactly that.
@ -492,6 +499,20 @@ def _source_ids(answer: Mapping[str, Any]) -> set[str]:
return found
_REFUSAL_CODE = re.compile(r"refused \(([a-z_]+)\)")
def refusal_code(error: RpcError) -> str:
"""The code a refusal carries, or `""`.
Read from the message because that is where a client sees it: a tool-level
refusal travels in the result envelope, and its JSON-RPC number is the same
for every one of them.
"""
match = _REFUSAL_CODE.search(error.message)
return match.group(1) if match else ""
def _probe_arguments(tool: str, bundle_id: str, concept_id: str, *, named: bool) -> dict[str, Any]:
"""One representative call per tool. `named` is False for the one-to-one
variant, whose bundle is fixed at startup and takes no bundle argument."""
@ -615,22 +636,24 @@ def read_anchor_set(questions: Path, freeze: Path, *, want_version: int) -> Anch
document = json.loads(questions.read_text(encoding="utf-8"))
pairs: dict[tuple[str, str], str] = {}
for question in document.get("sporsmal", []):
bundle = str(question.get("bundle") or question.get("kilde") or "")
# The set names its bundles as a LIST per question, and a question may
# name several. A pair is (bundle, anchor), so a question naming two
# bundles and one anchor is two pairs: an anchor reachable in one
# bundle and not the other is two different facts.
named = [str(entry) for entry in (question.get("bundles") or []) if entry]
for atom in question.get("atomer", []) or []:
anchor = atom.get("kilde_anker")
quote = atom.get("kilde_sitat")
quote = str(atom.get("kilde_sitat") or "")
if isinstance(anchor, str) and anchor:
pairs.setdefault((bundle, anchor), str(quote or ""))
for bundle in named:
if pairs.get((bundle, anchor), "") == "":
pairs[(bundle, anchor)] = quote
for cite in question.get("must_cite", []) or []:
if isinstance(cite, str) and cite:
pairs.setdefault((bundle, cite), "")
elif isinstance(cite, Mapping):
anchor = cite.get("anker") or cite.get("kilde_anker")
if isinstance(anchor, str) and anchor:
pairs.setdefault(
(str(cite.get("bundle") or bundle), anchor),
str(cite.get("kilde_sitat") or ""),
)
entries = cite if isinstance(cite, list) else [cite]
for entry in entries:
if isinstance(entry, str) and entry:
for bundle in named:
pairs.setdefault((bundle, entry), "")
return AnchorSet(
label=f"{questions.parent.name}/{questions.name}",
pairs=tuple((bundle, anchor, quote) for (bundle, anchor), quote in sorted(pairs.items())),
@ -651,12 +674,50 @@ SYNTHETIC_ANCHORS: tuple[tuple[str, str, str], ...] = (
)
def _fetch_carries(client: Stdio, bundle_id: str, anchor: str, quote: str, *, named: bool) -> bool:
"""Can the surface hand back the text carrying `quote`, verbatim?
_WHITESPACE = re.compile(r"\s+")
This is a CEILING, never a hit rate: it asks whether the bytes are
reachable at all, not whether a ranker would choose them.
def fold(text: str) -> str:
"""The one normalisation both sides of a quote comparison get.
Two removals and nothing else. U+00AD, because `okf build` strips soft
hyphens from extracted text (`extract.normalise_extracted`) while the
publisher's own JSON keeps them, so a quote carrying one could never match
text that is otherwise identical. And whitespace runs, because a quote cut
out of a paragraph carries the line breaks of wherever it was cut. Case is
NOT folded and no character is transliterated: a quote is a quote.
"""
return _WHITESPACE.sub(" ", text.replace("\u00ad", "")).strip()
def _reach_anchor(client: Stdio, bundle_id: str, anchor: str, quote: str) -> tuple[bool, str]:
"""A documented two-step sequence for one anchor, with the route recorded.
Step one asks whether the anchor IS a concept id -- the cheap case, and a
true ceiling. Step two asks the surface the anchor as a question and reads
the delivered excerpts. Step two goes through the ranker, so a pair met
only there is a FLOOR on the ceiling and never the ceiling itself: the row
prints both counts rather than one.
"""
needle = fold(quote)
try:
answer = client.call("okf_fetch", {BUNDLE_KEY: bundle_id, "concept_id": anchor})
except (RpcError, ServerGone):
pass
else:
if not needle or needle in fold(json.dumps(answer, ensure_ascii=False)):
return True, "fetch"
if not needle:
return False, "no quote to look for"
try:
answer = client.call("okf_ask", {BUNDLE_KEY: bundle_id, "question": anchor})
except (RpcError, ServerGone) as error:
return False, f"ask refused: {error}"
return (needle in fold(json.dumps(answer, ensure_ascii=False))), "ask"
def _fetch_carries(client: Stdio, bundle_id: str, anchor: str, quote: str, *, named: bool) -> bool:
"""The synthetic known-positive's route: fetch by id, nothing else."""
arguments: dict[str, Any] = {"concept_id": anchor}
if named:
arguments[BUNDLE_KEY] = bundle_id
@ -668,8 +729,30 @@ def _fetch_carries(client: Stdio, bundle_id: str, anchor: str, quote: str, *, na
return quote in text if quote else bool(_source_ids(answer))
def present_in_bundle(bundle_root: Path, quotes: Sequence[str]) -> list[bool]:
"""Is the quote anywhere in the bundle's concept bodies?
Read with the LIBRARY rather than the surface, on purpose: this separates
"the bundle does not carry it" from "the surface could not reach it", and
the order asks for the first to be reported as a fact about the bundle
rather than a defect in the server.
"""
from llm_ingestion_okf import consume as okf_consume
haystack = fold(
"\n".join(
(bundle_root / f"{concept_id}.md").read_text(encoding="utf-8", errors="replace")
for concept_id in okf_consume.enumerate_concepts(bundle_root)
)
)
return [bool(quote) and fold(quote) in haystack for quote in quotes]
def row_two(
bundles: Mapping[str, Path], reachable: tuple[bool, str], anchors: AnchorSet | None
bundles: Mapping[str, Path],
reachable: tuple[bool, str],
anchors: AnchorSet | None,
real: Mapping[str, Path] | None = None,
) -> Row:
details: list[str] = []
known_positive = 0
@ -687,10 +770,11 @@ def row_two(
f"known-positive (synthetic, this file's own text): "
f"{known_positive} of {len(SYNTHETIC_ANCHORS)} anchors fetched verbatim"
)
name = "coverage ceiling: every anchor the frozen set points at, fetched verbatim"
if anchors is None:
return _row(
2,
"coverage ceiling: every anchor the frozen set points at, fetched verbatim",
name,
0,
0,
"no frozen question set supplied (--sett/--frys); the denominator is "
@ -699,42 +783,85 @@ def row_two(
)
m = len(anchors.pairs)
details.append(
f"set {anchors.label}, freeze version {anchors.version}, M = {m} (bundle, anchor) pairs"
f"set {anchors.label}, freeze version {anchors.version}, "
f"M = {m} (bundle, anchor) pairs counted from the file"
)
if not reachable[0]:
if not real:
return _row(
2,
"coverage ceiling: every anchor the frozen set points at, fetched verbatim",
name,
0,
m,
f"`python -c 'import {SERVER_MODULE}'` fails: {reachable[1]}",
"no bundle root supplied (--bundle-root); the set's bundles are not "
"committed here and the row cannot be measured without them",
details,
)
if not reachable[0]:
return _row(2, name, 0, m, f"server missing: {reachable[1]}", details)
by_bundle: dict[str, list[tuple[str, str]]] = {}
for bundle_id, anchor, quote in anchors.pairs:
by_bundle.setdefault(bundle_id, []).append((anchor, quote))
root = next(iter(real.values())).parent
k = 0
missing_bundles: set[str] = set()
root = next(iter(bundles.values())).parent
carried = 0
routes = {"fetch": 0, "ask": 0}
with server(variant_argv("one-to-many", root)) as client:
client.handshake()
served = {str(entry) for entry in _bundle_ids(client.call("okf_list", {}))}
for bundle_id, anchor, quote in anchors.pairs:
if bundle_id not in served:
missing_bundles.add(bundle_id)
for bundle_id in sorted(by_bundle):
entries = by_bundle[bundle_id]
target = real.get(bundle_id)
if target is None:
details.append(
f"{bundle_id}: RED for the BUNDLE -- no root supplied holds it "
f"({len(entries)} of {m} pairs)"
)
continue
if _fetch_carries(client, bundle_id, anchor, quote, named=True):
k += 1
if missing_bundles:
details.append(
"red for the BUNDLE, not the server: no bundle served under this root is "
f"named {', '.join(sorted(missing_bundles))}"
)
return _row(
2,
"coverage ceiling: every anchor the frozen set points at, fetched verbatim",
k,
m,
"the ceiling an arm can reach, per (bundle, anchor) pair",
details,
served_id = _served_name(target, served)
if served_id is None:
details.append(
f"{bundle_id}: RED for the BUNDLE -- {target.name} is not served "
f"({len(entries)} of {m} pairs)"
)
continue
here = present_in_bundle(target, [quote for _anchor, quote in entries])
carried += sum(here)
hit = 0
for (anchor, quote), _in_bundle in zip(entries, here, strict=True):
met, route = _reach_anchor(client, served_id, anchor, quote)
if met:
hit += 1
routes[route] = routes.get(route, 0) + 1
k += hit
details.append(
f"{bundle_id} (served as `{served_id}`): {hit} of {len(entries)} reached; "
f"{sum(here)} of {len(entries)} present in the bundle at all"
)
details.append(
f"routes: {routes.get('fetch', 0)} met by `okf_fetch` on the anchor as a concept id "
f"(a true ceiling), {routes.get('ask', 0)} met only through `okf_ask` (a FLOOR on the "
"ceiling: that route runs the ranker)"
)
details.append(
f"present in some bundle at all: {carried} of {m} -- a pair the bundle does not "
"carry is red for the bundle, not for the server"
)
return _row(2, name, k, m, "the ceiling an arm can reach, per (bundle, anchor) pair", details)
def _served_name(target: Path, served: set[str]) -> str | None:
"""The id the server calls this directory, read from the bundle itself.
Never guessed from the directory name: the frozen set names `n100-2023`
and the bundle declares `vegnormal-n100-2023`, so a rule matching names
would have to invent an alias rule of its own.
"""
from llm_ingestion_okf import consume as okf_consume
declared = okf_consume.root_bundle_id_of(target)
return declared if declared in served else None
# --------------------------------------------------------------------------
@ -1031,6 +1158,14 @@ def row_six(scratch: Path) -> Row:
),
encoding="utf-8",
)
# Named in the index, or it is not a concept and the ceiling never runs:
# the bundle's own index is what makes a file reachable at all.
index = root / "bridge-notes" / "index.md"
index.write_text(
index.read_text(encoding="utf-8").rstrip("\n")
+ "\n- [Svulmende](svulmende.md) \u2014 adjudication: proposed\n",
encoding="utf-8",
)
details: list[str] = []
k = 0
@ -1053,8 +1188,16 @@ def row_six(scratch: Path) -> Row:
try:
answer = client.call(tool, arguments)
except RpcError as error:
k += 1
details.append(f"{case}: refused loudly ({error})")
code = refusal_code(error)
if code in HOSTILE_CASES[case]:
k += 1
details.append(f"{case}: refused loudly ({code})")
else:
details.append(
f"{case}: refused as `{code or error.code}`, which is not the "
f"check this case attacks ({'/'.join(sorted(HOSTILE_CASES[case]))}) "
f"-- {error}"
)
continue
except ServerGone as error:
details.append(f"{case}: the server died instead of refusing ({error})")
@ -1099,12 +1242,13 @@ def evaluate(
scratch: Path,
*,
anchors: AnchorSet | None = None,
real: Mapping[str, Path] | None = None,
) -> list[Row]:
reachable = server_exists()
bundles = corpus(scratch / "base")
return [
row_one(bundles, reachable),
row_two(bundles, reachable, anchors),
row_two(bundles, reachable, anchors, real),
row_three(scratch),
row_four(scratch),
row_five(scratch),
@ -1131,6 +1275,35 @@ def render(rows: Sequence[Row]) -> str:
return "\n".join(lines) + "\n"
def _real_bundles(roots: Sequence[Path], anchors: AnchorSet | None) -> dict[str, Path]:
"""Map every bundle the set names to a directory under the given roots.
The match is on the SET's name against the directory name's leading
segment, and the served id is then read from the bundle itself. A set name
matching two directories is a usage error: choosing one would make the
measurement depend on directory order.
"""
if not roots or anchors is None:
return {}
wanted = sorted({bundle for bundle, _anchor, _quote in anchors.pairs})
found: dict[str, Path] = {}
for name in wanted:
hits = [
child
for root in roots
for child in sorted(root.iterdir())
if child.is_dir() and child.name.startswith(name)
]
if len(hits) > 1:
raise GateUsage(
f"`{name}` matches {len(hits)} directories under the given roots: "
f"{', '.join(hit.name for hit in hits)}"
)
if hits:
found[name] = hits[0]
return found
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0])
parser.add_argument("--json", action="store_true", help="emit the rows as JSON")
@ -1144,6 +1317,17 @@ def main(argv: list[str] | None = None) -> int:
type=Path,
help="the freeze file that pins --sett by sha256 and declares its version",
)
parser.add_argument(
"--bundle-root",
type=Path,
action="append",
default=[],
help=(
"a directory holding the set's bundles, for row 2 (repeatable). The "
"bundles are never committed here; the row maps a set name to a served "
"bundle by reading the bundle's own declared id"
),
)
parser.add_argument(
"--sett-versjon",
type=int,
@ -1159,8 +1343,9 @@ def main(argv: list[str] | None = None) -> int:
)
if (args.sett is None) != (args.frys is None):
raise GateUsage("--sett and --frys are given together or not at all")
real = _real_bundles(args.bundle_root, anchors)
with tempfile.TemporaryDirectory(prefix="okf-mcp-gate-") as scratch:
rows = evaluate(Path(scratch), anchors=anchors)
rows = evaluate(Path(scratch), anchors=anchors, real=real)
except GateUsage as error:
print(f"okf-mcp-gate: {error}", file=sys.stderr)
return 2