open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(mcp): serve OKF bundles over MCP in two shapes, plus the generic skill

Browse source 
The eval was written RED at `5f1772e` with no server in the tree. This is the
capability it was written against.

`okf mcp --bundle <dir>` serves exactly one bundle, whose tools take no bundle
argument. `okf mcp --root <dir>` (repeatable) serves every bundle under the
roots and knows NONE of them by name. Four tools -- `okf_list`,
`okf_describe`, `okf_ask`, `okf_fetch` -- each carrying its reason in the
description a client actually reads.

Gate today: 1 (7/7) - 2 (83/181) - 3 (4/4) - 4 (9/9) - 5 (3/3) - 6 (6/6),
`GATE RED: rows 2`, exit 1.

THE PROTOCOL IS STDLIB, AND THAT IS THE PACKAGING INVARIANT KEPT RATHER THAN
A TASTE. An MCP SDK would be this package's second runtime dependency on the
DEFAULT install path, for four JSON-RPC methods and a newline framing, and
`test_the_only_runtime_dependency_is_the_security_boundary` pins that list
literally. Chosen hand-written because the surface needed is `initialize`,
`notifications/initialized`, `tools/list` and `tools/call`; `uv.lock` is
untouched.

NOTHING IS CACHED ACROSS CALLS, and row 3 is why. Every call re-walks the
roots and recomputes `bundle_ref`, so a bundle added, removed or rebuilt while
the process runs is seen by the next call with no restart, no configuration
edit and no code change -- 9 of 9 discovery checks over three bundles written
while the server was serving. The cost is paid per call and is published
rather than hidden: 0.75 s for the identity of a 2 756-concept bundle, 5.6 s
for one ask, 4 min 13 s for row 2's full run over four bundles.

CONTAINMENT IS TWO INDEPENDENT CHECKS: the bundle's own index must name the
concept, AND `connectors.safe_resolve` must place it inside the bundle. A
mutant removing either one alone still refuses -- with a DIFFERENT code, which
row 6 asserts by name -- and one removing both is killed. Row 6 declares a
code set per case because its first run had the 10 MB concept refused as
`concept_unknown`: the fixture had not named the file in the index, so the
size ceiling never ran and the row was green for a reason unrelated to the
attack.

`okf card <bundle>` and `okf skill --generic` are the one-to-many skill
candidate. The card is DERIVED on every run and never written into the bundle:
storing it would move the bytes of all six `examples/*/expected-bundle` trees
(23 files compared byte-for-byte) and of the pinned reference bundle, to keep
something recomputable in under a second, and a stored card is one more
artefact that can disagree with what is beside it. Measured here rather than
taken from the order: two per-bundle skills are identical on 280 of 312 and
310 lines; the 62 that differ are identity, concept count, the
conditional-field table, the whole-bundle cost and the breaking point. The
generic skill carries none of them, and `render_generic()` takes no argument,
so there is no bundle it could have read.

Row 2 decomposes into three numbers and the middle one is the finding: 99 of
181 (bundle, anchor) pairs are present in the bundles at all, 83 of those 99
were reached, and 0 of 83 were met by `okf_fetch` on the anchor as a concept
id. The set's anchors and this library's concept ids are different
vocabularies, so every pair met was met through the ranker -- 83 is a FLOOR on
the ceiling, never the ceiling.

13 mutants in a scratch copy, never in the working tree: 12 killed, 1 survived
with its mechanism printed, 0 errors, control green first. Suite 2323 passed,
2 skipped. The architecture choice between the two shapes is the OPERATOR's;
these rows are its input. Report: docs/2026-09-20-mcp-to-varianter.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-20 10:25:55 +02:00
commit df5a1183c9
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
10 changed files with 1823 additions and 59 deletions
Download patch file Download diff file Expand all files Collapse all files

35
CHANGELOG.md
View file

@ -9,6 +9,41 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added ### Added
- **An MCP surface over OKF bundles, in two shapes, plus a generic
consumption skill.** `okf mcp --bundle <dir>` serves exactly one bundle;
`okf mcp --root <dir>` (repeatable) serves every bundle under the roots and
knows none of them by name. Four tools — `okf_list`, `okf_describe`,
`okf_ask`, `okf_fetch` — each with its reason written into the description a
client reads. The eval was written RED first (`tools/okf_mcp_gate.py`,
`5f1772e`); the capability follows.
- **The protocol is written narrowly with stdlib only, and that is the
packaging invariant kept rather than an aesthetic.** An MCP SDK would be
this package's second runtime dependency on the DEFAULT install path, for
four JSON-RPC methods and a newline framing, and
`test_the_only_runtime_dependency_is_the_security_boundary` pins that list
literally. `uv.lock` is untouched.
- **Nothing is cached across calls.** Every call re-walks the roots and
recomputes the bundle's content identity, so a bundle added, removed or
rebuilt while the server runs is seen by the next call without a restart, a
configuration edit or a code change — measured, 9 of 9 discovery checks over
three bundles written while the process was serving. The cost is paid per
call: 0.75 s for the identity of a 2 756-concept bundle, 5.6 s for one ask.
- **Containment is two independent checks**: the bundle's own index must name
the concept, and the resolved path must be inside the bundle. Removing
either one alone still refuses — with a different code, which the gate
asserts by name — and removing both is caught by the gate's row 6.
- **`okf card <bundle>`** prints one bundle's identity, concept count,
conditional-field counts and whole-bundle cost as JSON, DERIVED on every run
and never written into the bundle. **`okf skill --generic`** writes one
installable consumption skill for ANY bundle, carrying no bundle's identity
or numbers and pointing its reader at the card. Measured: two per-bundle
skills are identical on 280 of 312 and 310 lines, and what differs is
exactly what goes stale on a rebuild.
- Report: `docs/2026-09-20-mcp-to-varianter.md`. The gate stands RED on row 2
(83 of 181 anchors of the frozen graded set reached, of which 99 are present
in the bundles at all and 0 were met by a concept-id lookup), and the
architecture choice between the two shapes is the operator's.
- **Every carried image is now one a model can be SHOWN, and the ones that - **Every carried image is now one a model can be SHOWN, and the ones that
cannot be are refused out loud.** Until this round the asset path carried cannot be are refused out loud.** Until this round the asset path carried
whatever format a publisher shipped. Measured 2026-09-19 over the frozen whatever format a publisher shipped. Measured 2026-09-19 over the frozen

51
CLAUDE.md
View file

@ -1406,6 +1406,57 @@ and fixtures, never code.
held by a test. README publishes this bar behind held by a test. README publishes this bar behind
`<!-- quality-boundary-threshold: ... -->`; SS 7 of the threshold document `<!-- quality-boundary-threshold: ... -->`; SS 7 of the threshold document
carries the seven bundles and the honesty limits. carries the seven bundles and the honesty limits.
- **Serve a bundle over MCP: `okf mcp --bundle <dir>` (one bundle) or
`okf mcp --root <dir>` (every bundle under the roots, none known by name).**
Four tools -- `okf_list`, `okf_describe`, `okf_ask`, `okf_fetch` -- each
carrying its REASON in the description a client reads, and every answer
carrying the bundle id and concept id a claim must be attributed to. The
JSON-RPC is written narrowly with stdlib only: an MCP SDK would be this
package's SECOND runtime dependency on the default install path, and
`test_the_only_runtime_dependency_is_the_security_boundary` pins that list
literally; `uv.lock` is untouched. **NOTHING IS CACHED ACROSS CALLS** -- every
call re-walks the roots and recomputes `bundle_ref`, so a bundle added,
removed or rebuilt while the process runs is seen by the next call with no
restart, no config edit and no code change (measured, 9 of 9 discovery checks
over three bundles written while serving). The cost is paid per call and is
published: **0.75 s** for the identity of a 2 756-concept bundle, **5.6 s**
for one ask. Containment is TWO independent checks -- the bundle's own index
must name the concept AND `connectors.safe_resolve` must place it inside the
bundle -- and a mutant removing either one alone still refuses, with a
different code. A concept above `MAX_CONCEPT_BYTES` is refused whole rather
than truncated, and a directory that cannot be read as a bundle is REPORTED
in `okf_list`'s `unreadable` rather than skipped: an absence with no
denominator is not a boundary. The eval is
`tools/okf_mcp_gate.py`, written RED at `5f1772e` before any server existed;
it speaks real stdio to a SUBPROCESS and never imports the server. Today:
**1 (7/7) - 2 (83/181) - 3 (4/4) - 4 (9/9) - 5 (3/3) - 6 (6/6)**,
`GATE RED: rows 2`. **Row 2 decomposes into three numbers and the middle one
is the finding**: 99 of 181 (bundle, anchor) pairs are present in the bundles
at all, 83 of those 99 were reached, and **0 of 83 were met by `okf_fetch` on
the anchor as a concept id** -- the set's anchors and this library's concept
ids are different vocabularies, so every pair met was met through the ranker
and 83 is a FLOOR on the ceiling, never the ceiling. **The architecture choice
between the two shapes is the OPERATOR's**; the rows are its input. Report:
`docs/2026-09-20-mcp-to-varianter.md`.
- **`okf card <bundle>` and `okf skill --generic` are the one-to-many skill
candidate.** The card is one bundle's identity, concept count,
conditional-field counts and whole-bundle cost as JSON, **DERIVED on every run
and never written into the bundle** -- storing it would move the bytes of all
six `examples/*/expected-bundle` trees (23 files compared byte-for-byte) and
of the pinned reference bundle, to keep something recomputable in under a
second, and a stored card is one more artefact that can disagree with the
bytes beside it. `okf skill --generic` writes ONE installable consumption
skill for ANY bundle: it carries no bundle's identity and no bundle's numbers,
and the property that makes that checkable rather than asserted is that
`skill.render_generic()` **takes no argument** -- there is no bundle it could
have read. Measured 2026-09-20: two per-bundle skills are identical on **280
of 312** and **310** lines, and the 62 that differ are exactly identity,
concept count, the conditional-field table, the whole-bundle cost and the
breaking point -- the five things that go stale on a rebuild. The update
drill, four artefact classes: MCP one-to-one **0 artefacts / 0 steps**, MCP
one-to-many **0 / 0**, today's per-bundle skill **1 / 1 per consuming
project** (it refuses out loud through `bundle_mismatch`, so its cost is not
silence), generic skill **0 / 0**.
- Consume a bundle: `okf consume <bundle> --question "<q>" - Consume a bundle: `okf consume <bundle> --question "<q>"
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass** [--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
`docs/consumption-contract.md` § 1 defines, and the only reading direction `docs/consumption-contract.md` § 1 defines, and the only reading direction

75
README.md
View file

@ -1155,6 +1155,81 @@ the summary back. Install it for your user account after cloning:
mkdir -p ~/.claude/skills && cp -R skills/okf-prosjekt ~/.claude/skills/ mkdir -p ~/.claude/skills && cp -R skills/okf-prosjekt ~/.claude/skills/
``` ```
## Serve a bundle over MCP: `okf mcp`
Two shapes, one implementation, and the difference is what an agent has to be
told in advance.
```sh
okf mcp --bundle .okf/my-bundle # one server, one bundle
okf mcp --root ~/bundles --root ./.okf # one server, every bundle under the roots
```
`--bundle` serves exactly one bundle, fixed at startup; its tools take no
bundle argument, because there is nothing to choose. `--root` (repeatable)
serves every bundle found under the given directories and **knows none of them
by name**: it discovers them per call, so a bundle you add, remove or rebuild
while the server is running is picked up by the next call. No restart, no
configuration edit, no code change.
Four tools, and each one's description says why it exists:
| tool | what it answers |
|---|---|
| `okf_list` | which bundles are reachable right now, with each one's content identity and concept count (multi-bundle servers only) |
| `okf_describe` | what one bundle is: id, ref, concept count, source documents, and how many concepts carry each conditionally-written field |
| `okf_ask` | one question, one bounded payload of excerpts, each with its bundle id, concept id, title and provenance locators. Omitting `bundle_id` on a multi-bundle server asks them all and splits the budget |
| `okf_fetch` | one named concept, verbatim, with its frontmatter and locators |
**Nothing is cached between calls, and that is the design.** Every call
re-reads the directories and recomputes the bundle's content identity, so the
identity in an answer is a fact about the bytes at the moment of the call
rather than at startup — a server that answered from yesterday's bundle is the
one failure you cannot see from the outside. The cost is real and is paid per
call: on a 2 756-concept bundle the identity is a 0.75 s hash of the whole
concept tree, and one `okf_ask` is 5.6 s.
**Refusals are loud.** A path climbing out of the bundle, a symlink leaving the
served root, a bundle id nobody answers to, a directory whose manifest cannot be
read, and a concept above the server's size ceiling each come back as an error
with a code — never as a plausible-looking empty answer. A concept over the
ceiling is refused whole rather than truncated: a truncated concept read as
whole is a wrong answer that looks right. A directory that cannot be read as a
bundle is **reported** in `okf_list`'s `unreadable`, not skipped.
The protocol is written with the standard library only. An MCP SDK would be
this package's second runtime dependency on the default install path, for four
JSON-RPC methods and a newline framing — see
[Requirements](#requirements).
`tools/okf_mcp_gate.py` is the eval: it starts the server as a subprocess,
speaks real stdio to it, and measures six rows. It was written red before the
server existed, and it is red today on row 2. The measurements, the update
drill and the limits are in
[`docs/2026-09-20-mcp-to-varianter.md`](docs/2026-09-20-mcp-to-varianter.md).
### One skill for every bundle: `okf card` and `okf skill --generic`
`okf skill <bundle>` writes a consumption skill for **that** bundle, with its
identity and its numbers measured into the text — which is what makes the file
stale the moment the bundle is rebuilt. `okf skill --generic` writes one
installable skill for **any** bundle instead:
```sh
okf skill --generic --out ~/.claude/skills/okf-consume-any
okf card .okf/my-bundle # the per-bundle numbers, as JSON, on demand
```
The generic skill carries no bundle's id, no ref and no count; it tells its
reader to run `okf card <bundle>` first. The card is **derived on every run and
never written into the bundle**, so there is no second artefact that can
disagree with the bytes beside it.
Measured on two unrelated bundles: two per-bundle skills are identical on 280
of 312 and 310 lines. The 62 lines that differ are exactly identity, concept
count, the conditional-field table, the whole-bundle cost and the breaking
point — the five things a rebuild invalidates.
## Implemented scope (v1) ## Implemented scope (v1)
The library provides three entry points for getting content into an OKF The library provides three entry points for getting content into an OKF

170
docs/2026-09-20-mcp-to-varianter.md Normal file
View file

@ -0,0 +1,170 @@
# An MCP surface over OKF bundles, in two shapes
2026-09-20. Order `20260918T163400Z-6303812376-from-.claude`. Capability loop:
the eval was written RED at `5f1772e`, before any server existed; the capability
follows in its own commit.
The operator's question was not "does MCP work". It was: one server per bundle
or one server for many, and **must these artefacts be made again every time a
bundle is rebuilt or a new one appears?** This round builds the three artefacts
that question compares, and measures the answer.
## What was measured, and against what
`tools/okf_mcp_gate.py`, six rows, one exit code. The server is started as a
subprocess and spoken to over newline-delimited JSON-RPC beginning at
`initialize` -- never imported. A client built from the server's own framing
helpers would agree with the server by construction, so the client is written
separately in the gate.
Denominators are pinned in the gate and recounted a second time in the tests:
7 required tools across the two shapes, 4 artefact classes, 3 bundles times 3
discovery checks, 3 cross-bundle checks, 6 hostile cases. A row that counted
what the server happened to offer would go green by offering less.
| row | what it asks | today |
|---|---|---|
| 1 | every required tool answers over real stdio, carrying bundle id and concept id | **7 of 7** |
| 2 | every anchor the frozen graded set points at, fetched verbatim | **83 of 181** |
| 3 | one concept changes: does the stale artefact refuse, or answer quietly | **4 of 4** |
| 4 | three unknown bundles appear while the server runs | **9 of 9** |
| 5 | one documented sequence, two bundles, both sources | **3 of 3** |
| 6 | traversal, symlink, broken manifest, 10 MB concept, unknown id | **6 of 6** |
`GATE RED: rows 2`, exit 1.
Reproduce:
```sh
uv run python tools/okf_mcp_gate.py \
--sett <the frozen set>/sporsmal.json \
--frys <the frozen set>/frys.json \
--bundle-root <a directory holding its bundles>
```
Without the last three flags row 2 is `0 of 0` with the reason stated: the set
names a consumer's documents, this repository is public, and a gold set is an
input here and never a constant.
## Row 3 is the operator's question, and the answer has four rows
The drill: copy a bundle, start the artefact, change one concept, ask again.
| artefact | stale answer | artefacts to remake | manual steps |
|---|---|---|---|
| one server in front of one bundle | refuses / cannot go stale | 0 | 0 |
| one server in front of many | refuses / cannot go stale | 0 | 0 |
| today's generated skill (per bundle) | refuses out loud (`bundle_mismatch`) | 1 | 1, **per consuming project** |
| the generic skill (one for all) | cannot go stale | 0 | 0 |
**Neither MCP shape needs an update when a bundle is rebuilt, and neither needs
one when a bundle is added.** That is not luck: nothing is cached across calls.
Every call re-walks the roots and recomputes the bundle's content identity, so
the identity in an answer is a fact about the bytes at the moment of the call.
The cost is real and is paid per call -- see the limits below.
Row 3 was **1 of 4 before any capability existed**, which the order did not
predict and is worth stating: today's per-bundle skill already refuses out loud
when its bundle moves, because `okf check`'s `bundle_mismatch` rule compares the
declared ref against the payload's. The skill's cost is not silence. It is that
one artefact has to be regenerated and reinstalled wherever it was installed,
and that number is not measurable from inside this machine.
## The generic skill, measured rather than assumed
The order cited 227 of 285 lines identical between two generated skills,
measured 2026-09-18. Measured again here, on two different bundles
(`examples/ingest-golden-segmented-okf-v0-2` and `tests/fixtures/consume-bundle`):
**280 of 312 and 310 lines identical, 62 lines differing**. Neither number
contradicts the other -- they are different pairs of bundles -- and the shape of
the finding is the same: what differs is identity, concept count, the
conditional-field table, the whole-bundle cost and the breaking point.
`skill.render_generic()` carries none of them. The property that makes that
claim checkable rather than asserted is that **the function takes no argument**:
there is no bundle it could have read, and two calls return the same bytes. A
test controls it against a per-bundle skill, which must carry exactly what the
generic one does not -- without that control, an assertion about an absence
passes on an empty string.
The per-bundle half is `okf card <bundle>`, **derived on every run and never
written into the bundle**. The order proposed storing it there. Writing a card
file into every bundle would move the bytes of all six `examples/*/expected-bundle`
trees (23 files compared byte-for-byte) and of the pinned reference bundle, to
store something recomputable in under a second -- and a stored card is one more
artefact that can disagree with the bytes beside it, which is the defect the
generic skill exists to remove. Chosen as derived because it answers the
maintenance question more completely, not less.
## Row 2 decomposed: the bundle, the ranker, and the vocabulary
**83 of 181** (bundle, anchor) pairs, `M = 181` counted from the set at run time.
The order's own figure of 197 is the set's atom count under a different
definition; 181 is what the pair rule below yields on the file as frozen at
version 4.
Three numbers, and the middle one is the finding:
* **99 of 181 pairs are present in the bundles at all.** 82 are not: the text
the set quotes is not in the bundle, which is red for the BUNDLE and not for
the server. `r761-2025` is the sharpest case at 17 of 33 present.
* **83 of the 99 present were reached**, so the surface reaches 83.8 % of what
is there. `r761-2025` is again the outlier: 2 reached of 17 present.
* **0 of 83 were met by `okf_fetch` on the anchor as a concept id.** The set's
anchors (`Krav 2.3.1—3`) and this library's concept ids are different
vocabularies, so the cheap route -- a true ceiling -- never fires, and every
pair met was met through `okf_ask`, which runs the ranker. **That makes 83 a
FLOOR on the ceiling, never the ceiling.** A surface offering a lookup by the
publisher's own anchor would separate the two, and does not exist today.
Quote comparison folds exactly two things and nothing else: U+00AD, because
`okf build` strips soft hyphens from extracted text while the publisher's JSON
keeps them, and whitespace runs, because a quote cut out of a paragraph carries
the line breaks of wherever it was cut. Case is not folded.
## Hostile input, and why a code set rather than "was refused"
Row 6 declares, per case, the refusal CODES that count as the right refusal.
The first run of this gate had the 10 MB concept refused as `concept_unknown` --
the fixture had written the file without naming it in the index, so the size
ceiling never ran and the row was green for a reason unrelated to the attack.
Two checks giving the same verdict are not the same guarantee.
Containment is two independent checks: the bundle's own index must name the
concept, AND the resolved path must be inside the bundle. A mutant removing the
first one **survives**, and the mechanism is printed: the traversal is then
refused by the second, as `path_escape` instead of `concept_unknown`. A mutant
removing both is killed. That survival is the redundancy working and is reported
as such rather than as a kill.
## Mutants
13 mutants, applied in a scratch copy of the tree and never in the working tree,
with an unmutated control first: **12 killed, 1 survived with a mechanism, 0
errors.** The control's gate rows and pytest targets are green before the first
mutation, so a kill cannot be the call having failed.
Killed: a cached bundle identity (row 3), two bundles known by name in the
many-shape (row 4), a fetched concept without its concept id (row 1), both
containment checks removed (row 6), discovery run once at startup (row 4), row
2's denominator taken from the run (test), a symlink descended (test), the size
ceiling removed (row 6), the generic skill naming a bundle (test), a broken
manifest skipped silently (row 6), a listing tool on the one-shape (test), and
an unknown bundle answered instead of refused (row 6).
## Limits, stated rather than implied
* **Nothing is cached, and it costs.** On the 2 756-concept bundle the content
identity is a 0.75 s hash of the whole concept tree and one `okf_ask` is
5.6 s. Row 2's full run over four bundles and 181 pairs took **4 min 13 s**.
A cache would have to be keyed on something cheaper than the hash and still
correct; no such key is shipped, and the cost is the price of the row-3 result
above.
* **The gate measures a ceiling and a maintenance cost.** Whether an arm answers
WELL is a different question, asked by `tools/okf_retrieval_gate.py`. No arm
was run here and no model was called.
* **The architecture choice is the operator's.** These rows are its input.
* Row 3 counts artefacts and steps inside this machine. A project that has
installed a generated skill pays one more step per project, and that number is
not measurable from here.
* No MCP server was registered in any `settings.json` or `.mcp.json`.

8
src/llm_ingestion_okf/cli.py
View file

@ -105,7 +105,7 @@ __all__ = ["DEFAULT_STAMP", "build", "main", "measure"]
#: #:
#: Imported lazily inside the dispatch: `okf build` should not pay to import #: Imported lazily inside the dispatch: `okf build` should not pay to import
#: the ranker, and `okf consume` should not pay to import the proposer. #: the ranker, and `okf consume` should not pay to import the proposer.
DELEGATED = ("consume", "check", "skill", "project", "quality") DELEGATED = ("consume", "check", "skill", "project", "quality", "card", "mcp")
def _delegate(command: str, argv: list[str]) -> int: def _delegate(command: str, argv: list[str]) -> int:
@ -117,6 +117,10 @@ def _delegate(command: str, argv: list[str]) -> int:
from .skill import main as run from .skill import main as run
elif command == "quality": elif command == "quality":
from .quality import main as run from .quality import main as run
elif command == "card":
from .skill import card_main as run
elif command == "mcp":
from .mcp_server import main as run
else: else:
from .project import main as run from .project import main as run
return run(argv) return run(argv)
@ -648,6 +652,8 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
("skill", "instantiate the consumption skill template for one bundle"), ("skill", "instantiate the consumption skill template for one bundle"),
("project", "folder in, bundle plus skill out: build and skill in one step"), ("project", "folder in, bundle plus skill out: build and skill in one step"),
("quality", "judge one bundle per file type, with the denominator"), ("quality", "judge one bundle per file type, with the denominator"),
("card", "print one bundle's own identity, counts and denominators as JSON"),
("mcp", "serve one bundle, or every bundle under a root, over MCP on stdio"),
): ):
subcommands.add_parser(delegated, help=blurb, add_help=False) subcommands.add_parser(delegated, help=blurb, add_help=False)
build_parser = subcommands.add_parser( build_parser = subcommands.add_parser(

737
src/llm_ingestion_okf/mcp_server.py Normal file
View file

@ -0,0 +1,737 @@
"""Expose OKF bundles over the Model Context Protocol, in two shapes.
Beside `skill.py` because it belongs to the same class: a way to put a bundle
in front of an agent. The skill hands a consumer a document telling it which
command to run; this hands it a set of tools a client calls. Neither ranks
anything of its own -- both reach `consume.build_payload`, which stays the one
reading direction this library has.
TWO SHAPES, ONE IMPLEMENTATION.
* `--bundle PATH` serves exactly ONE bundle, fixed at startup. The bundle
tools take no bundle argument, because there is nothing to choose.
* `--root PATH` (repeatable) serves every bundle found under the roots, and
knows NONE of them by name. Discovery happens per call, so a bundle added,
removed or rebuilt while the process runs is seen by the next call without a
restart, a configuration edit or a code change.
NOTHING IS CACHED ACROSS CALLS, AND THAT IS THE DESIGN RATHER THAN AN
OVERSIGHT. A server that read the bundle list once at startup would keep
answering after the bundle was rebuilt, with an identity that no longer
describes the bytes -- and an answer from yesterday's bundle is the one
failure a consumer cannot see from the outside. Every call re-walks the roots
and recomputes `bundle_ref`, so the identity in an answer is always a fact
about the bytes on disk at the moment of the call. The cost is real: the
identity is a sha256 over the whole concept tree, and it is paid per call.
WHY THE PROTOCOL IS WRITTEN HERE AND NOT TAKEN FROM AN SDK. This package
declares exactly one runtime dependency, the security guard, and
`tests/test_packaging.py::test_the_only_runtime_dependency_is_the_security_boundary`
pins that list literally. An MCP SDK would be the second, on the DEFAULT
install path, for four JSON-RPC methods and a newline framing -- so the
protocol is written narrowly, with stdlib only, and the packaging invariant
stays a fact rather than an intention. Chosen over the SDK because the surface
needed is `initialize`, `notifications/initialized`, `tools/list` and
`tools/call`, and nothing here needs resources, prompts, sampling or progress.
CONTAINMENT IS TWO INDEPENDENT CHECKS, NEVER ONE. A concept is reachable only
if the bundle's own index names it (`consume.enumerate_concepts`, which
refuses a target climbing above the root) AND its resolved path is inside the
bundle (`connectors.safe_resolve`, on canonical paths). Either alone would be
defensible; the pair is what makes a defect in one of them survivable.
"""
from __future__ import annotations
import argparse
import json
import sys
from collections.abc import Iterator, Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Any, TextIO
from . import consume as okf_consume
from . import materialize
from .connectors import safe_resolve
from .errors import SourceError
from .profiles import BundleProfile
#: The revision this server implements. A client asking for another is
#: answered with this one, which the specification permits: the client then
#: decides whether it can proceed.
PROTOCOL_VERSION = "2025-06-18"
SERVER_NAME = "okf"
#: How deep a root is walked looking for bundles. A bundle is a directory with
#: an `index.md` carrying a `bundle_id`, and the walk does NOT descend into one
#: it has found -- a bundle inside a bundle is the door's own collision case,
#: not a second bundle. Bounded rather than unbounded because a root is given
#: by an operator and may be a home directory by accident.
MAX_DISCOVERY_DEPTH = 3
#: The largest concept `okf_fetch` will hand over whole. A concept is a
#: section of a document; this is two orders of magnitude above the largest in
#: any bundle measured here, and it exists so that a bundle carrying a file
#: that is not a concept cannot turn one tool call into a memory cost the
#: caller never asked for. Refused with its own code, never truncated: a
#: truncated concept read as whole is a wrong answer that looks right.
MAX_CONCEPT_BYTES = 1024 * 1024
#: Default breadth of an `okf_ask`. The library's own default, restated here
#: rather than imported implicitly, because a tool's default is part of its
#: contract.
DEFAULT_K = okf_consume.DEFAULT_K
class ToolError(Exception):
"""A refusal a client can act on. Always loud: it leaves the server as a
JSON-RPC error, never as a plausible-looking empty answer."""
def __init__(self, message: str, *, code: str) -> None:
super().__init__(message)
self.code = code
# --------------------------------------------------------------------------
# Discovery
# --------------------------------------------------------------------------
@dataclass(frozen=True)
class Served:
bundle_id: str
root: Path
@dataclass(frozen=True)
class Unreadable:
"""A directory that looks like a bundle and cannot be read as one.
Reported rather than skipped. A broken manifest that simply vanishes from
the list is an absence with no denominator, and the caller cannot tell it
from a bundle that was never there.
"""
path: str
reason: str
@dataclass(frozen=True)
class Discovery:
bundles: tuple[Served, ...]
unreadable: tuple[Unreadable, ...]
def _declared_bundle_id(index: Path) -> str:
frontmatter = materialize.parse_frontmatter(index)
return str(frontmatter.get("bundle_id", "")).strip()
def _walk(root: Path, depth: int) -> Iterator[Path]:
"""Directories under `root`, breadth-first, to `MAX_DISCOVERY_DEPTH`.
A symlink is never descended and never yielded: a link inside a served
root pointing outside it is exactly how a root boundary is escaped, and
refusing to follow one is cheaper than proving each target is contained.
"""
if depth > MAX_DISCOVERY_DEPTH:
return
try:
entries = sorted(root.iterdir(), key=lambda path: path.name)
except OSError:
return
for entry in entries:
if entry.is_symlink() or not entry.is_dir():
continue
yield entry
if not (entry / "index.md").is_file():
yield from _walk(entry, depth + 1)
def _candidates(roots: Sequence[Path], *, include_roots: bool) -> Iterator[Path]:
"""Directories to test for being a bundle.
`include_roots` is the whole difference between the two shapes at this
level: `--bundle` points AT a bundle, `--root` points at a directory that
holds them. Without it the one-to-one server discovers its own children and
never itself -- which is how the first build of this module answered every
call with "the bundle this server was started on is no longer readable".
"""
for root in roots:
if include_roots:
yield root
else:
yield from _walk(root, 1)
def discover(roots: Sequence[Path], *, include_roots: bool = False) -> Discovery:
"""Every bundle under the roots, recomputed on every call."""
bundles: dict[str, Served] = {}
unreadable: list[Unreadable] = []
for candidate in _candidates(roots, include_roots=include_roots):
index = candidate / "index.md"
if not index.is_file():
continue
try:
bundle_id = _declared_bundle_id(index)
except (OSError, UnicodeDecodeError, ValueError) as error:
unreadable.append(Unreadable(candidate.name, f"index.md unreadable: {error}"))
continue
if not bundle_id:
unreadable.append(Unreadable(candidate.name, "index.md declares no bundle_id"))
continue
if bundle_id in bundles:
unreadable.append(
Unreadable(candidate.name, f"a second bundle claims the id `{bundle_id}`")
)
continue
bundles[bundle_id] = Served(bundle_id, candidate)
return Discovery(
tuple(bundles[key] for key in sorted(bundles)),
tuple(sorted(unreadable, key=lambda entry: entry.path)),
)
@dataclass(frozen=True)
class Surface:
"""What the two shapes have in common, with the difference in one flag."""
roots: tuple[Path, ...]
fixed: str | None
profile: BundleProfile
@property
def one_to_many(self) -> bool:
return self.fixed is None
def discovery(self) -> Discovery:
"""Re-read on every call, in both shapes. The one-to-one server tests
its own root; the one-to-many server tests what is under its roots."""
return discover(self.roots, include_roots=not self.one_to_many)
def resolve(self, bundle_id: str | None) -> Served:
"""The bundle a call names, or the fixed one. Never a guess.
A one-to-many call that names no bundle is a usage error and not a
default: picking one would make the answer's provenance depend on
directory order.
"""
found = self.discovery()
served = {entry.bundle_id: entry for entry in found.bundles}
if not self.one_to_many:
assert self.fixed is not None
if self.fixed not in served:
raise ToolError(
f"the bundle this server was started on is no longer readable: {self.fixed}",
code="bundle_unreadable",
)
return served[self.fixed]
if not bundle_id:
raise ToolError(
"this server serves several bundles; name one with `bundle_id` "
f"({', '.join(sorted(served)) or 'none served'})",
code="bundle_id_required",
)
if bundle_id in served:
return served[bundle_id]
for entry in found.unreadable:
if entry.path == bundle_id:
raise ToolError(
f"`{bundle_id}` looks like a bundle and cannot be read as one: {entry.reason}",
code="bundle_unreadable",
)
raise ToolError(
f"no bundle named `{bundle_id}` is served "
f"({', '.join(sorted(served)) or 'none served'})",
code="bundle_unknown",
)
# --------------------------------------------------------------------------
# The card: everything about ONE bundle that a generic consumer needs
# --------------------------------------------------------------------------
def card(bundle_root: Path, *, profile: BundleProfile, concept_sample: int = 50) -> dict[str, Any]:
"""The per-bundle numbers a generic reader needs, DERIVED on demand.
This is the half of a generated consumption skill that differs between
bundles -- identity, concept count, which conditional fields are written on
how many concepts, what the whole bundle costs. Today `okf skill` bakes
those numbers into a document, which is what makes the document go stale
when the bundle is rebuilt.
Derived rather than written into the bundle. Writing a card file into every
bundle would move the bytes of all six `examples/*/expected-bundle` trees
(23 files compared byte-for-byte) and of the pinned reference bundle, to
store something recomputable from the bundle in under a second. A stored
card would also be one more artefact that can be stale, which is the defect
it was meant to remove.
"""
from . import skill as okf_skill
bundle_id = okf_consume.root_bundle_id_of(bundle_root, profile=profile)
concepts = okf_consume.link_parents(
[
okf_consume.read_concept(
bundle_root / f"{concept_id}{profile.paths.concept_suffix}",
bundle_root=bundle_root,
root_bundle_id=bundle_id,
)
for concept_id in okf_consume.enumerate_concepts(bundle_root, profile=profile)
]
)
counts = okf_skill.field_counts(concepts)
return {
"bundle_id": bundle_id,
"ref": okf_consume.bundle_ref(bundle_root, profile=profile),
"ref_algorithm": okf_consume.REF_ALGORITHM,
"profile": okf_skill.PROFILE_NAME,
"concept_count": len(concepts),
"concepts": [concept.concept_id for concept in concepts[:concept_sample]],
"concepts_truncated": len(concepts) > concept_sample,
"source_files": sorted(
{concept.source_file for concept in concepts if concept.source_file}
),
"conditional_fields": {
field: counts.get(field, 0) for field in okf_skill.CONDITIONAL_FIELDS
},
"whole_bundle_bytes": okf_skill.whole_bundle_cost(concepts),
"budget_unit": okf_consume.BUDGET_UNIT,
"default_limit": okf_consume.DEFAULT_LIMIT,
}
# --------------------------------------------------------------------------
# Tools. Each one has a reason, and the reason is the description a client reads.
# --------------------------------------------------------------------------
@dataclass(frozen=True)
class Tool:
name: str
description: str
schema: dict[str, Any]
_BUNDLE_ARGUMENT = {
"bundle_id": {
"type": "string",
"description": "the bundle to act on; omit on a server started with --bundle",
}
}
def tools(surface: Surface) -> tuple[Tool, ...]:
"""The minimum set that answers the questions a bundle exists to answer.
`okf_list` only on a server that serves more than one: on a one-to-one
server there is nothing to list, and a tool that always returns the same
single row invites a client to treat discovery as available when the
deployment does not have it.
"""
bundle = _BUNDLE_ARGUMENT if surface.one_to_many else {}
listing = (
Tool(
"okf_list",
"Every OKF bundle this server can currently reach, with its content "
"identity and concept count. Re-read from disk on every call, so a "
"bundle added, removed or rebuilt since the last call is reflected "
"without restarting anything. Exists because a client that cannot "
"discover bundles must be told their names out of band, which is the "
"configuration this shape is meant to remove.",
{"type": "object", "properties": {}, "additionalProperties": False},
),
)
common = (
Tool(
"okf_describe",
"What one bundle is: its id, its content identity, how many concepts "
"it holds, which source documents it was built from, and which "
"conditionally-written fields are present on how many concepts. "
"Exists because an answer must be attributable -- a claim from a "
"bundle whose identity the caller cannot state is a claim with no "
"provenance -- and because a reader needs the denominators before it "
"can read an absence.",
{
"type": "object",
"properties": dict(bundle),
"additionalProperties": False,
},
),
Tool(
"okf_ask",
"One question, one bounded payload of excerpts, each carrying its "
"bundle id, concept id, title and provenance locators, plus what was "
"withheld and why. This is the library's only reading direction and "
"it calls no model. On a multi-bundle server, omitting `bundle_id` "
"asks every served bundle and splits the budget between them. Exists "
"because handing a client the whole bundle is not an answer, and "
"letting it choose files by name is the enumeration the consumption "
"contract forbids.",
{
"type": "object",
"properties": {
"question": {"type": "string", "description": "the question, in prose"},
**bundle,
"k": {
"type": "integer",
"description": f"how many concepts to consider (default {DEFAULT_K})",
},
"limit": {"type": "integer", "description": "payload budget in utf-8 bytes"},
},
"required": ["question"],
"additionalProperties": False,
},
),
Tool(
"okf_fetch",
"One named concept, verbatim, with its frontmatter and its source "
"locators. Exists because a ranked payload is a SELECTION: an arm "
"that has been told a concept id -- by `okf_ask`, by a parent "
"pointer, or by a citation it is checking -- needs the bytes "
"themselves, and must not have to guess them from an excerpt.",
{
"type": "object",
"properties": {
"concept_id": {
"type": "string",
"description": "a bundle-relative concept id, as `okf_ask` reports it",
},
**bundle,
},
"required": ["concept_id"],
"additionalProperties": False,
},
),
)
return (listing + common) if surface.one_to_many else common
def call_list(surface: Surface, _arguments: Mapping[str, Any]) -> dict[str, Any]:
found = surface.discovery()
entries: list[dict[str, Any]] = []
for served in found.bundles:
entries.append(
{
"bundle_id": served.bundle_id,
"ref": okf_consume.bundle_ref(served.root, profile=surface.profile),
"concept_count": len(
okf_consume.enumerate_concepts(served.root, profile=surface.profile)
),
"directory": served.root.name,
}
)
return {
"bundles": entries,
"unreadable": [
{"directory": entry.path, "reason": entry.reason} for entry in found.unreadable
],
"shape": "one-to-many" if surface.one_to_many else "one-to-one",
}
def call_describe(surface: Surface, arguments: Mapping[str, Any]) -> dict[str, Any]:
served = surface.resolve(_string(arguments, "bundle_id"))
return card(served.root, profile=surface.profile)
def call_ask(surface: Surface, arguments: Mapping[str, Any]) -> dict[str, Any]:
question = _string(arguments, "question")
if not question:
raise ToolError("`question` is required and may not be empty", code="question_missing")
k = int(arguments.get("k") or DEFAULT_K)
limit = int(arguments.get("limit") or okf_consume.DEFAULT_LIMIT)
named = _string(arguments, "bundle_id")
if named or not surface.one_to_many:
targets = [surface.resolve(named)]
else:
targets = list(surface.discovery().bundles)
if not targets:
raise ToolError("no bundle is served under the given roots", code="bundle_none_served")
share = max(1, limit // len(targets))
if share < okf_consume.DEFAULT_LIMIT // 100:
raise ToolError(
f"the budget splits to {share} bytes across {len(targets)} bundles, which "
"cannot carry an excerpt; name one bundle or raise `limit`",
code="budget_too_thin",
)
answers = []
for served in targets:
try:
payload = okf_consume.build_payload(
served.root, question=question, k=k, limit=share, profile=surface.profile
)
except okf_consume.ConsumeError as error:
raise ToolError(
f"{served.bundle_id}: {error}", code=getattr(error, "code", "consume_refused")
) from error
answers.append({"bundle_id": served.bundle_id, "payload": payload})
return {
"question": question,
"asked": [served.bundle_id for served in targets],
"budget_per_bundle": share,
"answers": answers,
}
def call_fetch(surface: Surface, arguments: Mapping[str, Any]) -> dict[str, Any]:
concept_id = _string(arguments, "concept_id")
if not concept_id:
raise ToolError("`concept_id` is required", code="concept_id_missing")
served = surface.resolve(_string(arguments, "bundle_id"))
known = okf_consume.enumerate_concepts(served.root, profile=surface.profile)
if concept_id not in known:
raise ToolError(
f"`{concept_id}` is not a concept the bundle's index names",
code="concept_unknown",
)
suffix = surface.profile.paths.concept_suffix
try:
path = safe_resolve(served.root, f"{concept_id}{suffix}")
except SourceError as error:
raise ToolError(str(error), code="path_escape") from error
size = path.stat().st_size
if size > MAX_CONCEPT_BYTES:
raise ToolError(
f"`{concept_id}` is {size} bytes, above this server's {MAX_CONCEPT_BYTES}-byte "
"ceiling for one concept; it is refused whole rather than truncated",
code="concept_too_large",
)
concept = okf_consume.read_concept(
path,
bundle_root=served.root,
root_bundle_id=okf_consume.root_bundle_id_of(served.root, profile=surface.profile),
)
return {
"bundle_id": concept.bundle_id,
"ref": okf_consume.bundle_ref(served.root, profile=surface.profile),
"concept": {
"concept_id": concept.concept_id,
"title": concept.title,
"sha256": concept.sha256,
"adjudication": concept.adjudication,
"req_number": concept.req_number,
"source_file": concept.source_file,
"sources": [dict(entry) for entry in concept.sources],
"locators": dict(concept.locators),
"text": concept.body,
},
}
def _string(arguments: Mapping[str, Any], key: str) -> str:
value = arguments.get(key)
if value is None:
return ""
if not isinstance(value, str):
raise ToolError(
f"`{key}` must be a string, not {type(value).__name__}", code="argument_type"
)
return value
HANDLERS = {
"okf_list": call_list,
"okf_describe": call_describe,
"okf_ask": call_ask,
"okf_fetch": call_fetch,
}
# --------------------------------------------------------------------------
# The protocol: four methods, newline-delimited JSON-RPC 2.0 over stdio
# --------------------------------------------------------------------------
METHOD_NOT_FOUND = -32601
INVALID_PARAMS = -32602
INTERNAL_ERROR = -32603
def _tool_result(payload: Mapping[str, Any]) -> dict[str, Any]:
"""Both forms, on purpose.
`structuredContent` is what a client with a schema reads; the text block is
what one without a schema reads, and a client that got only the first would
see an empty message. The text is the SAME object, serialised -- two
renderings of one answer, never two answers.
"""
text = json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=False)
return {
"content": [{"type": "text", "text": text}],
"structuredContent": dict(payload),
"isError": False,
}
def _tool_refusal(message: str, code: str) -> dict[str, Any]:
return {
"content": [{"type": "text", "text": f"refused ({code}): {message}"}],
"isError": True,
}
def handle(surface: Surface, method: str, params: Mapping[str, Any]) -> dict[str, Any]:
"""One request to one result. Raises `ToolError` only through the envelope."""
if method == "initialize":
return {
"protocolVersion": PROTOCOL_VERSION,
"capabilities": {"tools": {"listChanged": False}},
"serverInfo": {"name": SERVER_NAME, "version": _version()},
"instructions": (
"Bundles are read-only. Ask `okf_ask` a question in prose rather "
"than fetching concepts by name: every excerpt it returns carries "
"the bundle id and concept id a claim must be attributed to, and "
"the payload states what it withheld and why."
),
}
if method == "ping":
return {}
if method == "tools/list":
return {
"tools": [
{"name": tool.name, "description": tool.description, "inputSchema": tool.schema}
for tool in tools(surface)
]
}
if method == "tools/call":
name = params.get("name")
arguments = params.get("arguments") or {}
if not isinstance(arguments, Mapping):
return _tool_refusal("`arguments` must be an object", "argument_type")
available = {tool.name for tool in tools(surface)}
if not isinstance(name, str) or name not in available:
return _tool_refusal(
f"no tool named {name!r} on this server ({', '.join(sorted(available))})",
"tool_unknown",
)
try:
return _tool_result(HANDLERS[name](surface, arguments))
except ToolError as error:
return _tool_refusal(str(error), error.code)
except okf_consume.ConsumeError as error:
return _tool_refusal(str(error), getattr(error, "code", "consume_refused"))
except SourceError as error:
return _tool_refusal(str(error), getattr(error, "code", "path_escape"))
# Broad on purpose: a traceback on stdout would break the framing, and
# a server that dies on one bad argument takes every other bundle with
# it. The refusal is still loud, and it still carries a code.
except Exception as error:
return _tool_refusal(f"{type(error).__name__}: {error}", "tool_failed")
raise LookupError(method)
def _version() -> str:
from . import __version__
return __version__
def serve(surface: Surface, *, stdin: TextIO, stdout: TextIO) -> int:
"""Read requests until stdin closes. One JSON object per line, both ways."""
for line in stdin:
line = line.strip()
if not line:
continue
try:
message = json.loads(line)
except json.JSONDecodeError:
continue # unframeable input: there is no id to answer it under
if not isinstance(message, dict):
continue
method = str(message.get("method", ""))
identifier = message.get("id")
params = message.get("params") or {}
if not isinstance(params, Mapping):
params = {}
if identifier is None:
continue # a notification: acknowledged by doing nothing
try:
result: dict[str, Any] = {
"jsonrpc": "2.0",
"id": identifier,
"result": handle(surface, method, params),
}
except LookupError:
result = {
"jsonrpc": "2.0",
"id": identifier,
"error": {"code": METHOD_NOT_FOUND, "message": f"no method {method!r}"},
}
except Exception as error:
result = {
"jsonrpc": "2.0",
"id": identifier,
"error": {
"code": INTERNAL_ERROR,
"message": f"{type(error).__name__}: {error}",
},
}
stdout.write(json.dumps(result, ensure_ascii=False) + "\n")
stdout.flush()
return 0
def build_surface(
*,
bundle: Path | None,
roots: Sequence[Path],
profile: BundleProfile = okf_consume.DEFAULT_PROFILE,
) -> Surface:
if bundle is not None:
index = bundle / "index.md"
if not index.is_file():
raise ToolError(
f"{bundle} carries no index.md, so it is not a bundle", code="not_a_bundle"
)
bundle_id = _declared_bundle_id(index)
if not bundle_id:
raise ToolError(f"{index} declares no bundle_id", code="not_a_bundle")
return Surface((bundle.resolve(),), bundle_id, profile)
if not roots:
raise ToolError("give either --bundle or at least one --root", code="no_target")
return Surface(tuple(root.resolve() for root in roots), None, profile)
def parse_args(argv: Sequence[str] | None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
prog="okf mcp",
description=(
"Serve OKF bundles over the Model Context Protocol on stdio. "
"`--bundle` serves one bundle and takes no bundle argument on its "
"tools; `--root` serves every bundle found under the given "
"directories and knows none of them by name."
),
)
parser.add_argument("--bundle", type=Path, help="serve exactly this bundle")
parser.add_argument(
"--root",
type=Path,
action="append",
default=[],
help="serve every bundle under this directory (repeatable)",
)
return parser.parse_args(list(argv) if argv is not None else None)
def main(argv: Sequence[str] | None = None) -> int:
args = parse_args(argv)
if args.bundle is not None and args.root:
print("okf mcp: --bundle and --root are two shapes; give one", file=sys.stderr)
return 2
try:
surface = build_surface(bundle=args.bundle, roots=args.root)
except ToolError as error:
print(f"okf mcp: refused ({error.code}): {error}", file=sys.stderr)
return 2
# Line-buffered both ways: a client blocks on our answer, and a block
# buffer would hold it until the buffer filled or the process exited.
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(line_buffering=True)
return serve(surface, stdin=sys.stdin, stdout=sys.stdout)
if __name__ == "__main__":
raise SystemExit(main())

240
src/llm_ingestion_okf/skill.py
View file

@ -54,6 +54,7 @@ from __future__ import annotations
import argparse import argparse
import json import json
import re import re
import sys
from collections import Counter from collections import Counter
from pathlib import Path from pathlib import Path
@ -179,6 +180,10 @@ TEMPLATE_ENUMERATION = (
TEMPLATE_OUTPUT = "Write to `<OUT>`. It must carry: the bundle ref; the findings, each with a" TEMPLATE_OUTPUT = "Write to `<OUT>`. It must carry: the bundle ref; the findings, each with a"
#: Every per-corpus hole the template carries. A generic skill that left one
#: would be the unfilled template with better manners, so it is refused.
_PLACEHOLDER = re.compile(r"<[A-Z][A-Z_]*>")
REPLACED_BLOCKS = ( REPLACED_BLOCKS = (
TEMPLATE_HEADER, TEMPLATE_HEADER,
TEMPLATE_PRE_PASS, TEMPLATE_PRE_PASS,
@ -664,7 +669,12 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
) )
parser.add_argument("bundle", type=Path, help="the OKF bundle to instantiate a skill for") parser.add_argument(
"bundle",
type=Path,
nargs="?",
help="the OKF bundle to instantiate a skill for (unused with --generic)",
)
parser.add_argument( parser.add_argument(
"--out", type=Path, required=True, help="the skill directory to write (SKILL.md inside)" "--out", type=Path, required=True, help="the skill directory to write (SKILL.md inside)"
) )
@ -677,15 +687,31 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
parser.add_argument( parser.add_argument(
"--force", action="store_true", help="replace an existing SKILL.md at --out" "--force", action="store_true", help="replace an existing SKILL.md at --out"
) )
parser.add_argument(
"--generic",
action="store_true",
help=(
"write the one-to-many skill instead: one installable document for ANY "
"bundle, carrying no bundle's identity or numbers. `bundle` is then "
"unused, and the reader is told to run `okf card <bundle>` at run time"
),
)
return parser.parse_args(argv) return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int: def main(argv: list[str] | None = None) -> int:
args = parse_args(argv) args = parse_args(argv)
try: try:
written = generate( if args.bundle is None and not args.generic:
print("refused (bundle_missing): name a bundle, or pass --generic", file=sys.stderr)
return 2
written = (
generate_generic(out=args.out, force=args.force)
if args.generic
else generate(
args.bundle, out=args.out, question=args.example_question, force=args.force args.bundle, out=args.out, question=args.example_question, force=args.force
) )
)
except okf_consume.ConsumeError as exc: except okf_consume.ConsumeError as exc:
print(f"refused ({exc.code}): {exc}") print(f"refused ({exc.code}): {exc}")
return 1 return 1
@ -701,3 +727,213 @@ def main(argv: list[str] | None = None) -> int:
if __name__ == "__main__": if __name__ == "__main__":
raise SystemExit(main()) raise SystemExit(main())
# --- The one-to-many candidate ------------------------------------------------
#: The name the generic skill carries. Claude Code takes a project skill's
#: command from its DIRECTORY name and uses `name` only as a display label, so
#: this is the label and not the command.
GENERIC_NAME = "okf-consume-any"
#: The command that hands a reader the per-bundle numbers this skill does not
#: carry. It has to exist for the skill to be honest: a generic document that
#: told a reader to "check the denominators somewhere" would be the unfilled
#: template with better manners.
CARD_COMMAND = "okf card"
GENERIC_BUNDLE = "<the bundle you were pointed at>"
def render_generic() -> str:
"""One installable skill for ANY bundle, carrying no bundle's numbers.
The measured fact this answers: two skills generated for two different
bundles are identical on 280 of 312 and 310 lines (measured 2026-09-20 on
this machine, over `examples/ingest-golden-segmented-okf-v0-2` and
`tests/fixtures/consume-bundle`; the order's own 227 of 285 is a different
pair of bundles and neither number contradicts the other). The 30-odd lines
that differ are identity, concept count, the conditional-field table, the
whole-bundle cost and the breaking point -- all of them recomputable from
the bundle in under a second, and all of them what makes a generated skill
go stale the moment its bundle is rebuilt.
So this text carries NONE of them, and says where to read each one instead.
The property that makes that claim checkable is that this function takes no
argument: there is no bundle it could have read, and two calls return the
same bytes.
"""
text = template_path().read_text(encoding="utf-8")
text = text.split("---\n", 2)[2]
replacements: list[tuple[str, str]] = [
(
TEMPLATE_HEADER,
"**This file is generic: it carries no bundle's identity and no bundle's\n"
"numbers,** and it is therefore never stale. It serves whichever bundle you\n"
"are pointed at. Before answering, read that bundle's own card:\n\n"
"```sh\n"
f"{CARD_COMMAND} {GENERIC_BUNDLE}\n"
"```\n\n"
"The card is DERIVED from the bundle on every run, never stored in it, so\n"
"there is no second artefact that can disagree with the bytes. Its\n"
"`bundle_id` and `ref` are the identity to carry into your output; its\n"
"`concept_count`, `conditional_fields` and `whole_bundle_bytes` are the\n"
"denominators the sections below ask for. The section headings are fixed:\n"
"the contract checker reads them by name.",
),
(
TEMPLATE_PRE_PASS,
"```sh\n"
f"{PRE_PASS_COMMAND} \\\n"
f" {GENERIC_BUNDLE} \\\n"
' --question "your question" \\\n'
" --ref THE_REF \\\n"
" --out /tmp/payload.json\n"
"```\n\n"
"`--ref` is an **assertion**, never an override: the identity is computed\n"
"from the bytes either way, and a mismatch refuses. Read the pre-pass's\n"
"own exit status, which carries three values: **0** a payload was written,\n"
"**1** the run happened and refused, **2** the run did not happen at all.",
),
(
TEMPLATE_CHECK,
f"```sh\n{CHECKER_COMMAND} --skill <this file> --payload /tmp/payload.json\n```",
),
(
TEMPLATE_CONTRACT_LINE,
f"The contract this skill is held to is `{CONTRACT}`. Where this",
),
(
TEMPLATE_EXTENSIONS,
"**Extensions.** This skill declares none. A corpus needing one declares it\n"
"in its own documentation; the five markings below are never extended here,\n"
"because a marking invented for one bundle would travel to every other.",
),
(
TEMPLATE_CONDITIONAL,
"**Conditionally-written fields.** Read `conditional_fields` from the card:\n"
"it gives, per field, how many of the bundle's concepts carry it. A field\n"
"written on some concepts and not others means its ABSENCE on one concept\n"
"is a measurement about that concept, never a fact about the world — so\n"
"report the count beside any claim that rests on an absence. The fields\n"
f"this profile can write are: {', '.join(f'`{field}`' for field in CONDITIONAL_FIELDS)}.",
),
(
TEMPLATE_SCALING,
"**Scaling.** Cost tracks the QUESTION, not the corpus: the payload is cut\n"
f"to {okf_consume.DEFAULT_LIMIT} {okf_consume.BUDGET_UNIT} whatever the bundle's size. What\n"
"does track the corpus is the bookkeeping — one `withheld` entry per\n"
"considered-and-not-delivered concept — so the point at which this strategy\n"
"stops fitting is a property of the bundle. Read `whole_bundle_bytes` from\n"
"the card and compare it with the budget: a bundle costing less than the\n"
"budget could have been handed over whole, and the pre-pass is then a\n"
"convenience rather than a necessity.",
),
(
TEMPLATE_DENOMINATORS,
"The payload reports three counts — `considered`, `withheld`, `delivered` —\n"
"and `considered == withheld + delivered`. Carry them into your output, and\n"
"carry the card's `concept_count` beside them: `considered` is what the cut\n"
"looked at, and the card says how much of the bundle that was.",
),
(
TEMPLATE_ENUMERATION,
f"- **No directory enumeration** unless the profile (`{PROFILE_NAME}`) says the\n"
" index is derived. The payload's own `bundle.entries_match_directory` says\n"
" whether it does, for the bundle in front of you.",
),
(
TEMPLATE_OUTPUT,
"Write to the path the caller names, or to your answer if none was named.\n"
"It must carry: the bundle ref; the findings, each with a",
),
("`<CORPUS>` bundle", "bundle you were pointed at"),
("# <CORPUS> consumption", "# OKF bundle consumption"),
]
# The blocks are STRICT: a template that stopped carrying one has drifted,
# and rewriting the rest would ship a skill missing a whole section.
for old, new in replacements:
if old not in text:
raise SkillError(
f"the template no longer carries the block this generator rewrites: {old[:70]!r}",
code="template_drift",
)
text = text.replace(old, new)
# The tokens are LENIENT, and the sweep below is what makes that safe: a
# token may already have been consumed by the block that carried it, and a
# strict check here would only measure the order of this list.
for old, new in (
("<PROFILE_NAME>", PROFILE_NAME),
("<PRE_PASS_COMMAND>", PRE_PASS_COMMAND),
("<BUDGET_LIMIT>", str(okf_consume.DEFAULT_LIMIT)),
("<BUDGET_UNIT>", okf_consume.BUDGET_UNIT),
("<BUDGET_INSTRUMENT>", okf_consume.BUDGET_INSTRUMENT),
("<KNOWN_POSITIVE_CASE>", okf_consume.KNOWN_POSITIVE_CASE),
("<KNOWN_POSITIVE_EXPECTED>", str(okf_consume.KNOWN_POSITIVE_EXPECTED)),
("<BUNDLE_ROOT>", GENERIC_BUNDLE),
("<PAYLOAD_PATH>", "/tmp/payload.json"),
("<SKILL_PATH>", "this file"),
("<REF>", "the card's `ref`"),
("<OUT>", "the path the caller named"),
):
text = text.replace(old, new)
left = sorted(set(_PLACEHOLDER.findall(text)))
if left:
raise SkillError(
f"the generic skill still carries a per-corpus hole: {', '.join(left)}. A hole "
"left in a generic document is a number the reader is invited to invent",
code="placeholder_unfilled",
)
description = block_scalar(
"Answer one question about ANY OKF bundle from a bounded payload assembled "
"by a deterministic pre-pass, marking every claim with its source, its title "
"and its provenance locator. Carries no bundle's identity: read the bundle's "
f"own card with `{CARD_COMMAND}` first. Use when the user asks a question of, "
"or states a hypothesis about, a corpus held as an OKF bundle."
)
header = f"---\nname: {block_scalar(GENERIC_NAME)}\ndescription: {description}\n---\n"
return header + text
def generate_generic(*, out: Path, force: bool = False) -> Path:
"""Write the generic skill. Takes no bundle, by construction."""
target = out / "SKILL.md"
if target.exists() and not force:
raise SkillError(
f"{target} already exists; pass --force to replace it",
code="target_occupied",
)
out.mkdir(parents=True, exist_ok=True)
target.write_text(render_generic(), encoding="utf-8")
return target
def card_main(argv: list[str] | None = None) -> int:
"""`okf card <bundle>` -- the per-bundle half of a consumption skill, as JSON.
The generic skill above tells its reader to run this. It is DERIVED on every
run and never stored in the bundle: a stored card is one more artefact that
can disagree with the bytes beside it, which is the defect the generic skill
exists to remove.
"""
parser = argparse.ArgumentParser(
prog="okf card",
description=(
"Print one bundle's identity, concept count, conditional-field counts "
"and whole-bundle cost as JSON. Derived from the bundle on every run."
),
)
parser.add_argument("bundle", type=Path, help="the OKF bundle to describe")
args = parser.parse_args(argv)
from .mcp_server import card as build_card
try:
payload = build_card(args.bundle.resolve(), profile=okf_consume.DEFAULT_PROFILE)
except okf_consume.ConsumeError as exc:
print(f"refused ({exc.code}): {exc}", file=sys.stderr)
return 1
except OSError as exc:
print(f"the run did not happen: {exc}", file=sys.stderr)
return 2
print(json.dumps(payload, ensure_ascii=False, indent=2))
return 0

77
tests/test_mcp_gate.py
View file

@ -67,7 +67,7 @@ def test_every_pinned_denominator_is_recounted_here() -> None:
) )
assert gate.DISCOVERY_BUNDLES * len(gate.DISCOVERY_CHECKS) == 3 * 3 assert gate.DISCOVERY_BUNDLES * len(gate.DISCOVERY_CHECKS) == 3 * 3
assert len(gate.CROSS_CHECKS) == 3 assert len(gate.CROSS_CHECKS) == 3
assert gate.HOSTILE_CASES == ( assert tuple(gate.HOSTILE_CASES) == (
"traversal-in-bundle-id", "traversal-in-bundle-id",
"traversal-in-concept-id", "traversal-in-concept-id",
"symlink-out-of-root", "symlink-out-of-root",
@ -75,6 +75,9 @@ def test_every_pinned_denominator_is_recounted_here() -> None:
"oversized-concept", "oversized-concept",
"unknown-bundle-id", "unknown-bundle-id",
) )
# The 10 MB case has exactly one acceptable refusal. Anything else means
# the ceiling did not run -- the false green this row shipped with once.
assert gate.HOSTILE_CASES["oversized-concept"] == frozenset({"concept_too_large"})
assert gate.OVERSIZED_BYTES == 10 * 1024 * 1024 assert gate.OVERSIZED_BYTES == 10 * 1024 * 1024
@ -128,7 +131,7 @@ def test_a_question_set_whose_bytes_moved_is_a_usage_error_not_a_red_row(
payload = { payload = {
"sporsmal": [ "sporsmal": [
{ {
"bundle": "bridge-notes", "bundles": ["bridge-notes"],
"atomer": [{"kilde_anker": "spennvidde", "kilde_sitat": "24 meter"}], "atomer": [{"kilde_anker": "spennvidde", "kilde_sitat": "24 meter"}],
} }
] ]
@ -171,6 +174,15 @@ def test_an_older_freeze_version_is_refused_rather_than_measured(scratch: Path)
gate.read_anchor_set(questions, freeze, want_version=4) gate.read_anchor_set(questions, freeze, want_version=4)
def test_a_refusal_code_is_read_from_the_message_a_client_sees() -> None:
"""Driven from both sides: a coded refusal yields its code, an uncoded one
yields the empty string rather than a plausible guess."""
assert gate.refusal_code(gate.RpcError(-32000, "refused (concept_too_large): 10 MB")) == (
"concept_too_large"
)
assert gate.refusal_code(gate.RpcError(-32000, "something went wrong")) == ""
def test_the_gate_reads_bundle_and_concept_ids_at_any_depth() -> None: def test_the_gate_reads_bundle_and_concept_ids_at_any_depth() -> None:
"""An `ask` answer carries one id per excerpt and a `list` answer one per """An `ask` answer carries one id per excerpt and a `list` answer one per
bundle. A rule reading the top level only would score the shape.""" bundle. A rule reading the top level only would score the shape."""
@ -222,3 +234,64 @@ def test_one_documented_sequence_answers_from_two_bundles(scratch: Path) -> None
def test_hostile_input_is_refused_out_loud(scratch: Path) -> None: def test_hostile_input_is_refused_out_loud(scratch: Path) -> None:
row = _row(6, scratch) row = _row(6, scratch)
assert row.k == row.m == 6, f"{row.k} of {row.m}: {row.details}" assert row.k == row.m == 6, f"{row.k} of {row.m}: {row.details}"
def test_row_twos_denominator_is_the_sets_and_not_what_the_surface_reached(
scratch: Path,
) -> None:
"""The defect this repository has already met twice: a row that counts
against what the run produced closes by producing less.
Three pairs are declared and one of them names an anchor no bundle carries,
so a denominator taken from the run would read `2 of 2` and call the gate
satisfied. Driven from both sides -- the reachable pairs really are
reachable, so a `0 of 3` would be a different defect.
"""
import hashlib
bundles = gate.corpus(scratch / "base")
questions = scratch / "sporsmal.json"
questions.write_text(
json.dumps(
{
"sporsmal": [
{
"bundles": ["bridge-notes"],
"atomer": [
{"kilde_anker": "spennvidde", "kilde_sitat": "spennvidde 24 meter"},
{"kilde_anker": "rekkverk", "kilde_sitat": "1,2 meter hoeyt"},
{
"kilde_anker": "finnes-ikke",
"kilde_sitat": "dette staar ingen steder",
},
],
}
]
}
),
encoding="utf-8",
)
freeze = scratch / "frys.json"
freeze.write_text(
json.dumps(
{
"versjon": 4,
"sha256": {"sporsmal.json": hashlib.sha256(questions.read_bytes()).hexdigest()},
}
),
encoding="utf-8",
)
anchors = gate.read_anchor_set(questions, freeze, want_version=4)
assert len(anchors.pairs) == 3
rows = {
row.number: row
for row in gate.evaluate(
scratch / "run",
anchors=anchors,
real={"bridge-notes": bundles["bridge-notes"]},
)
}
row = rows[2]
assert row.m == 3, f"the denominator came from the run: {row.k} of {row.m}"
assert row.k == 2, f"{row.k} of {row.m}: {row.details}"
assert row.status == gate.RED

196
tests/test_mcp_server.py Normal file
View file

@ -0,0 +1,196 @@
"""The MCP surface itself (`src/llm_ingestion_okf/mcp_server.py`).
`tools/okf_mcp_gate.py` measures this module over a real protocol and is the
eval it was built against. These tests hold the pieces the gate reaches only
through a verdict: what discovery does with a symlink, what the card does NOT
do to the bundle, and the property that makes the generic skill's whole claim
checkable.
"""
from __future__ import annotations
import hashlib
import json
import subprocess
import sys
from pathlib import Path
import pytest
from llm_ingestion_okf import mcp_server, skill
TOOLS = Path(__file__).resolve().parents[1] / "tools"
sys.path.insert(0, str(TOOLS))
import okf_mcp_gate as gate # noqa: E402
GOLDEN = (
Path(__file__).resolve().parents[1]
/ "examples"
/ "ingest-golden-segmented-okf-v0-2"
/ "expected-bundle"
)
def _tree(root: Path) -> dict[str, str]:
return {
str(path.relative_to(root)): hashlib.sha256(path.read_bytes()).hexdigest()
for path in sorted(root.rglob("*"))
if path.is_file()
}
def test_the_card_is_derived_and_writes_nothing_into_the_bundle(tmp_path: Path) -> None:
"""The whole reason the card is not a file in the bundle.
A stored card would move the bytes of all six `examples/*/expected-bundle`
trees and of the pinned reference bundle, and would be one more artefact
that can disagree with what is beside it.
"""
bundle = tmp_path / "b"
bundle.mkdir()
for source in GOLDEN.rglob("*"):
target = bundle / source.relative_to(GOLDEN)
if source.is_dir():
target.mkdir(parents=True, exist_ok=True)
else:
target.write_bytes(source.read_bytes())
before = _tree(bundle)
first = mcp_server.card(bundle, profile=mcp_server.okf_consume.DEFAULT_PROFILE)
second = mcp_server.card(bundle, profile=mcp_server.okf_consume.DEFAULT_PROFILE)
assert _tree(bundle) == before
assert first == second
assert first["concept_count"] == 3
assert first["ref"].startswith("sha256-tree:")
def test_the_card_command_prints_json_a_reader_can_parse() -> None:
"""The generic skill tells its reader to run this. A command whose output
could not be read back would make that instruction decorative."""
run = subprocess.run(
[sys.executable, "-m", "llm_ingestion_okf.cli", "card", str(GOLDEN)],
capture_output=True,
text=True,
check=False,
)
assert run.returncode == 0, run.stderr
payload = json.loads(run.stdout)
assert payload["bundle_id"] == "b-golden-segmented-okf-v0-2"
assert set(payload) >= {"ref", "concept_count", "conditional_fields", "whole_bundle_bytes"}
def test_discovery_never_descends_a_symlink(tmp_path: Path) -> None:
"""Driven from both sides: the real directory IS found, the link to it is
not. Without the control, a rule that found nothing at all would pass."""
outside = tmp_path / "outside"
gate.write_bundle(outside / "secret", "secret-notes", [("x", "X", "y")])
root = tmp_path / "root"
gate.write_bundle(root / "real", "real-notes", [("x", "X", "y")])
(root / "linked").symlink_to(outside / "secret", target_is_directory=True)
found = mcp_server.discover([root])
assert [served.bundle_id for served in found.bundles] == ["real-notes"]
assert mcp_server.discover([outside]).bundles[0].bundle_id == "secret-notes"
def test_a_directory_that_cannot_be_read_as_a_bundle_is_reported_not_skipped(
tmp_path: Path,
) -> None:
"""An absence with no denominator is not a boundary. A broken manifest that
simply vanished from the list would be indistinguishable from a bundle that
was never there."""
root = tmp_path / "root"
gate.write_bundle(root / "good", "good-notes", [("x", "X", "y")])
gate.write_bundle(root / "bad", "bad-notes", [("x", "X", "y")])
(root / "bad" / "index.md").write_bytes(b"\xff\xfe\x00")
found = mcp_server.discover([root])
assert [served.bundle_id for served in found.bundles] == ["good-notes"]
assert [entry.path for entry in found.unreadable] == ["bad"]
def test_the_one_to_one_server_finds_the_bundle_it_was_started_on(tmp_path: Path) -> None:
"""The defect the gate found on this module's first build: `--bundle` points
AT a bundle, and discovery that only looked at children found none."""
gate.write_bundle(tmp_path / "b", "solo-notes", [("x", "X", "y")])
surface = mcp_server.build_surface(bundle=tmp_path / "b", roots=[])
assert surface.resolve(None).bundle_id == "solo-notes"
def test_a_one_to_many_call_naming_no_bundle_is_refused_rather_than_guessed(
tmp_path: Path,
) -> None:
"""Picking one would make an answer's provenance depend on directory order."""
root = tmp_path / "root"
gate.write_bundle(root / "a", "a-notes", [("x", "X", "y")])
gate.write_bundle(root / "b", "b-notes", [("x", "X", "y")])
surface = mcp_server.build_surface(bundle=None, roots=[root])
with pytest.raises(mcp_server.ToolError) as raised:
surface.resolve(None)
assert raised.value.code == "bundle_id_required"
def test_the_one_to_one_server_offers_no_listing_tool(tmp_path: Path) -> None:
"""A tool that always returns the same single row invites a client to treat
discovery as available where the deployment does not have it."""
gate.write_bundle(tmp_path / "b", "solo-notes", [("x", "X", "y")])
one = mcp_server.build_surface(bundle=tmp_path / "b", roots=[])
many = mcp_server.build_surface(bundle=None, roots=[tmp_path])
assert [tool.name for tool in mcp_server.tools(one)] == list(gate.REQUIRED_TOOLS["one-to-one"])
assert [tool.name for tool in mcp_server.tools(many)] == list(
gate.REQUIRED_TOOLS["one-to-many"]
)
def test_every_tool_carries_a_written_reason() -> None:
"""The order's rule: a tool with no reason written down is a tool nobody
has to justify keeping."""
surface = mcp_server.Surface(
(Path("/nonexistent"),), None, mcp_server.okf_consume.DEFAULT_PROFILE
)
for tool in mcp_server.tools(surface):
assert "Exists because" in tool.description, tool.name
assert len(tool.description) > 200, tool.name
# --- the one-to-many skill candidate ------------------------------------------
def test_the_generic_skill_carries_no_bundles_identity(tmp_path: Path) -> None:
"""The property that makes the candidate's whole claim checkable: it takes
no argument, so there is no bundle it could have read.
Controlled against a per-bundle skill, which must carry exactly what this
one does not -- without that control a test asserting an absence would pass
on an empty string.
"""
generic = skill.render_generic()
skill.generate(GOLDEN, out=tmp_path / "per-bundle", question="krav", force=True)
per_bundle = (tmp_path / "per-bundle" / "SKILL.md").read_text(encoding="utf-8")
bundle_id = "b-golden-segmented-okf-v0-2"
ref = mcp_server.okf_consume.bundle_ref(GOLDEN)
assert bundle_id in per_bundle and ref in per_bundle
assert bundle_id not in generic and ref not in generic
assert skill.CARD_COMMAND in generic
def test_the_generic_skill_is_the_same_bytes_for_anyone(tmp_path: Path) -> None:
"""Two calls, and a written file, all identical. A rebuild of any bundle
cannot make this artefact wrong, which is stronger than refusing loudly."""
first = skill.render_generic()
written = skill.generate_generic(out=tmp_path / "g")
assert written.read_text(encoding="utf-8") == first == skill.render_generic()
def test_the_generic_skill_leaves_no_per_corpus_hole() -> None:
"""A hole left in a generic document is a number the reader is invited to
invent -- which is the unfilled template's own defect."""
assert skill._PLACEHOLDER.findall(skill.render_generic()) == []
def test_the_generic_skill_keeps_every_section_the_checker_reads() -> None:
from llm_ingestion_okf import contract_check
text = skill.render_generic()
for section in contract_check.REQUIRED_SECTIONS:
assert f"## {section}" in text, section

287
tools/okf_mcp_gate.py
View file

@ -42,6 +42,7 @@ import argparse
import hashlib import hashlib
import json import json
import os import os
import re
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
@ -104,14 +105,20 @@ CROSS_CHECKS: tuple[str, ...] = (
"documented-sequence", "documented-sequence",
) )
HOSTILE_CASES: tuple[str, ...] = ( #: Each hostile case with the refusal CODES that count as the right refusal.
"traversal-in-bundle-id", #: A code set rather than a bare "was refused": the first run of this gate had
"traversal-in-concept-id", #: the 10 MB concept refused as `concept_unknown`, because the fixture wrote
"symlink-out-of-root", #: the file without naming it in the index -- the size ceiling never ran, and
"broken-manifest", #: the row was green for a reason that had nothing to do with the attack. Two
"oversized-concept", #: checks giving the same verdict are not the same guarantee.
"unknown-bundle-id", HOSTILE_CASES: Mapping[str, frozenset[str]] = {
) "traversal-in-bundle-id": frozenset({"bundle_unknown", "path_escape", "bundle_id_invalid"}),
"traversal-in-concept-id": frozenset({"concept_unknown", "path_escape"}),
"symlink-out-of-root": frozenset({"bundle_unknown", "path_escape"}),
"broken-manifest": frozenset({"bundle_unreadable"}),
"oversized-concept": frozenset({"concept_too_large"}),
"unknown-bundle-id": frozenset({"bundle_unknown"}),
}
# A concept large enough that reading it whole is a decision rather than an # A concept large enough that reading it whole is a decision rather than an
# accident. The order names 10 MB; the gate writes exactly that. # accident. The order names 10 MB; the gate writes exactly that.
@ -492,6 +499,20 @@ def _source_ids(answer: Mapping[str, Any]) -> set[str]:
return found return found
_REFUSAL_CODE = re.compile(r"refused \(([a-z_]+)\)")
def refusal_code(error: RpcError) -> str:
"""The code a refusal carries, or `""`.
Read from the message because that is where a client sees it: a tool-level
refusal travels in the result envelope, and its JSON-RPC number is the same
for every one of them.
"""
match = _REFUSAL_CODE.search(error.message)
return match.group(1) if match else ""
def _probe_arguments(tool: str, bundle_id: str, concept_id: str, *, named: bool) -> dict[str, Any]: def _probe_arguments(tool: str, bundle_id: str, concept_id: str, *, named: bool) -> dict[str, Any]:
"""One representative call per tool. `named` is False for the one-to-one """One representative call per tool. `named` is False for the one-to-one
variant, whose bundle is fixed at startup and takes no bundle argument.""" variant, whose bundle is fixed at startup and takes no bundle argument."""
@ -615,22 +636,24 @@ def read_anchor_set(questions: Path, freeze: Path, *, want_version: int) -> Anch
document = json.loads(questions.read_text(encoding="utf-8")) document = json.loads(questions.read_text(encoding="utf-8"))
pairs: dict[tuple[str, str], str] = {} pairs: dict[tuple[str, str], str] = {}
for question in document.get("sporsmal", []): for question in document.get("sporsmal", []):
bundle = str(question.get("bundle") or question.get("kilde") or "") # The set names its bundles as a LIST per question, and a question may
# name several. A pair is (bundle, anchor), so a question naming two
# bundles and one anchor is two pairs: an anchor reachable in one
# bundle and not the other is two different facts.
named = [str(entry) for entry in (question.get("bundles") or []) if entry]
for atom in question.get("atomer", []) or []: for atom in question.get("atomer", []) or []:
anchor = atom.get("kilde_anker") anchor = atom.get("kilde_anker")
quote = atom.get("kilde_sitat") quote = str(atom.get("kilde_sitat") or "")
if isinstance(anchor, str) and anchor: if isinstance(anchor, str) and anchor:
pairs.setdefault((bundle, anchor), str(quote or "")) for bundle in named:
if pairs.get((bundle, anchor), "") == "":
pairs[(bundle, anchor)] = quote
for cite in question.get("must_cite", []) or []: for cite in question.get("must_cite", []) or []:
if isinstance(cite, str) and cite: entries = cite if isinstance(cite, list) else [cite]
pairs.setdefault((bundle, cite), "") for entry in entries:
elif isinstance(cite, Mapping): if isinstance(entry, str) and entry:
anchor = cite.get("anker") or cite.get("kilde_anker") for bundle in named:
if isinstance(anchor, str) and anchor: pairs.setdefault((bundle, entry), "")
pairs.setdefault(
(str(cite.get("bundle") or bundle), anchor),
str(cite.get("kilde_sitat") or ""),
)
return AnchorSet( return AnchorSet(
label=f"{questions.parent.name}/{questions.name}", label=f"{questions.parent.name}/{questions.name}",
pairs=tuple((bundle, anchor, quote) for (bundle, anchor), quote in sorted(pairs.items())), pairs=tuple((bundle, anchor, quote) for (bundle, anchor), quote in sorted(pairs.items())),
@ -651,12 +674,50 @@ SYNTHETIC_ANCHORS: tuple[tuple[str, str, str], ...] = (
) )
def _fetch_carries(client: Stdio, bundle_id: str, anchor: str, quote: str, *, named: bool) -> bool: _WHITESPACE = re.compile(r"\s+")
"""Can the surface hand back the text carrying `quote`, verbatim?
This is a CEILING, never a hit rate: it asks whether the bytes are
reachable at all, not whether a ranker would choose them. def fold(text: str) -> str:
"""The one normalisation both sides of a quote comparison get.
Two removals and nothing else. U+00AD, because `okf build` strips soft
hyphens from extracted text (`extract.normalise_extracted`) while the
publisher's own JSON keeps them, so a quote carrying one could never match
text that is otherwise identical. And whitespace runs, because a quote cut
out of a paragraph carries the line breaks of wherever it was cut. Case is
NOT folded and no character is transliterated: a quote is a quote.
""" """
return _WHITESPACE.sub(" ", text.replace("\u00ad", "")).strip()
def _reach_anchor(client: Stdio, bundle_id: str, anchor: str, quote: str) -> tuple[bool, str]:
"""A documented two-step sequence for one anchor, with the route recorded.
Step one asks whether the anchor IS a concept id -- the cheap case, and a
true ceiling. Step two asks the surface the anchor as a question and reads
the delivered excerpts. Step two goes through the ranker, so a pair met
only there is a FLOOR on the ceiling and never the ceiling itself: the row
prints both counts rather than one.
"""
needle = fold(quote)
try:
answer = client.call("okf_fetch", {BUNDLE_KEY: bundle_id, "concept_id": anchor})
except (RpcError, ServerGone):
pass
else:
if not needle or needle in fold(json.dumps(answer, ensure_ascii=False)):
return True, "fetch"
if not needle:
return False, "no quote to look for"
try:
answer = client.call("okf_ask", {BUNDLE_KEY: bundle_id, "question": anchor})
except (RpcError, ServerGone) as error:
return False, f"ask refused: {error}"
return (needle in fold(json.dumps(answer, ensure_ascii=False))), "ask"
def _fetch_carries(client: Stdio, bundle_id: str, anchor: str, quote: str, *, named: bool) -> bool:
"""The synthetic known-positive's route: fetch by id, nothing else."""
arguments: dict[str, Any] = {"concept_id": anchor} arguments: dict[str, Any] = {"concept_id": anchor}
if named: if named:
arguments[BUNDLE_KEY] = bundle_id arguments[BUNDLE_KEY] = bundle_id
@ -668,8 +729,30 @@ def _fetch_carries(client: Stdio, bundle_id: str, anchor: str, quote: str, *, na
return quote in text if quote else bool(_source_ids(answer)) return quote in text if quote else bool(_source_ids(answer))
def present_in_bundle(bundle_root: Path, quotes: Sequence[str]) -> list[bool]:
"""Is the quote anywhere in the bundle's concept bodies?
Read with the LIBRARY rather than the surface, on purpose: this separates
"the bundle does not carry it" from "the surface could not reach it", and
the order asks for the first to be reported as a fact about the bundle
rather than a defect in the server.
"""
from llm_ingestion_okf import consume as okf_consume
haystack = fold(
"\n".join(
(bundle_root / f"{concept_id}.md").read_text(encoding="utf-8", errors="replace")
for concept_id in okf_consume.enumerate_concepts(bundle_root)
)
)
return [bool(quote) and fold(quote) in haystack for quote in quotes]
def row_two( def row_two(
bundles: Mapping[str, Path], reachable: tuple[bool, str], anchors: AnchorSet | None bundles: Mapping[str, Path],
reachable: tuple[bool, str],
anchors: AnchorSet | None,
real: Mapping[str, Path] | None = None,
) -> Row: ) -> Row:
details: list[str] = [] details: list[str] = []
known_positive = 0 known_positive = 0
@ -687,10 +770,11 @@ def row_two(
f"known-positive (synthetic, this file's own text): " f"known-positive (synthetic, this file's own text): "
f"{known_positive} of {len(SYNTHETIC_ANCHORS)} anchors fetched verbatim" f"{known_positive} of {len(SYNTHETIC_ANCHORS)} anchors fetched verbatim"
) )
name = "coverage ceiling: every anchor the frozen set points at, fetched verbatim"
if anchors is None: if anchors is None:
return _row( return _row(
2, 2,
"coverage ceiling: every anchor the frozen set points at, fetched verbatim", name,
0, 0,
0, 0,
"no frozen question set supplied (--sett/--frys); the denominator is " "no frozen question set supplied (--sett/--frys); the denominator is "
@ -699,42 +783,85 @@ def row_two(
) )
m = len(anchors.pairs) m = len(anchors.pairs)
details.append( details.append(
f"set {anchors.label}, freeze version {anchors.version}, M = {m} (bundle, anchor) pairs" f"set {anchors.label}, freeze version {anchors.version}, "
f"M = {m} (bundle, anchor) pairs counted from the file"
) )
if not reachable[0]: if not real:
return _row( return _row(
2, 2,
"coverage ceiling: every anchor the frozen set points at, fetched verbatim", name,
0, 0,
m, m,
f"`python -c 'import {SERVER_MODULE}'` fails: {reachable[1]}", "no bundle root supplied (--bundle-root); the set's bundles are not "
"committed here and the row cannot be measured without them",
details, details,
) )
if not reachable[0]:
return _row(2, name, 0, m, f"server missing: {reachable[1]}", details)
by_bundle: dict[str, list[tuple[str, str]]] = {}
for bundle_id, anchor, quote in anchors.pairs:
by_bundle.setdefault(bundle_id, []).append((anchor, quote))
root = next(iter(real.values())).parent
k = 0 k = 0
missing_bundles: set[str] = set() carried = 0
root = next(iter(bundles.values())).parent routes = {"fetch": 0, "ask": 0}
with server(variant_argv("one-to-many", root)) as client: with server(variant_argv("one-to-many", root)) as client:
client.handshake() client.handshake()
served = {str(entry) for entry in _bundle_ids(client.call("okf_list", {}))} served = {str(entry) for entry in _bundle_ids(client.call("okf_list", {}))}
for bundle_id, anchor, quote in anchors.pairs: for bundle_id in sorted(by_bundle):
if bundle_id not in served: entries = by_bundle[bundle_id]
missing_bundles.add(bundle_id) target = real.get(bundle_id)
continue if target is None:
if _fetch_carries(client, bundle_id, anchor, quote, named=True):
k += 1
if missing_bundles:
details.append( details.append(
"red for the BUNDLE, not the server: no bundle served under this root is " f"{bundle_id}: RED for the BUNDLE -- no root supplied holds it "
f"named {', '.join(sorted(missing_bundles))}" f"({len(entries)} of {m} pairs)"
) )
return _row( continue
2, served_id = _served_name(target, served)
"coverage ceiling: every anchor the frozen set points at, fetched verbatim", if served_id is None:
k, details.append(
m, f"{bundle_id}: RED for the BUNDLE -- {target.name} is not served "
"the ceiling an arm can reach, per (bundle, anchor) pair", f"({len(entries)} of {m} pairs)"
details,
) )
continue
here = present_in_bundle(target, [quote for _anchor, quote in entries])
carried += sum(here)
hit = 0
for (anchor, quote), _in_bundle in zip(entries, here, strict=True):
met, route = _reach_anchor(client, served_id, anchor, quote)
if met:
hit += 1
routes[route] = routes.get(route, 0) + 1
k += hit
details.append(
f"{bundle_id} (served as `{served_id}`): {hit} of {len(entries)} reached; "
f"{sum(here)} of {len(entries)} present in the bundle at all"
)
details.append(
f"routes: {routes.get('fetch', 0)} met by `okf_fetch` on the anchor as a concept id "
f"(a true ceiling), {routes.get('ask', 0)} met only through `okf_ask` (a FLOOR on the "
"ceiling: that route runs the ranker)"
)
details.append(
f"present in some bundle at all: {carried} of {m} -- a pair the bundle does not "
"carry is red for the bundle, not for the server"
)
return _row(2, name, k, m, "the ceiling an arm can reach, per (bundle, anchor) pair", details)
def _served_name(target: Path, served: set[str]) -> str | None:
"""The id the server calls this directory, read from the bundle itself.
Never guessed from the directory name: the frozen set names `n100-2023`
and the bundle declares `vegnormal-n100-2023`, so a rule matching names
would have to invent an alias rule of its own.
"""
from llm_ingestion_okf import consume as okf_consume
declared = okf_consume.root_bundle_id_of(target)
return declared if declared in served else None
# -------------------------------------------------------------------------- # --------------------------------------------------------------------------
@ -1031,6 +1158,14 @@ def row_six(scratch: Path) -> Row:
), ),
encoding="utf-8", encoding="utf-8",
) )
# Named in the index, or it is not a concept and the ceiling never runs:
# the bundle's own index is what makes a file reachable at all.
index = root / "bridge-notes" / "index.md"
index.write_text(
index.read_text(encoding="utf-8").rstrip("\n")
+ "\n- [Svulmende](svulmende.md) \u2014 adjudication: proposed\n",
encoding="utf-8",
)
details: list[str] = [] details: list[str] = []
k = 0 k = 0
@ -1053,8 +1188,16 @@ def row_six(scratch: Path) -> Row:
try: try:
answer = client.call(tool, arguments) answer = client.call(tool, arguments)
except RpcError as error: except RpcError as error:
code = refusal_code(error)
if code in HOSTILE_CASES[case]:
k += 1 k += 1
details.append(f"{case}: refused loudly ({error})") details.append(f"{case}: refused loudly ({code})")
else:
details.append(
f"{case}: refused as `{code or error.code}`, which is not the "
f"check this case attacks ({'/'.join(sorted(HOSTILE_CASES[case]))}) "
f"-- {error}"
)
continue continue
except ServerGone as error: except ServerGone as error:
details.append(f"{case}: the server died instead of refusing ({error})") details.append(f"{case}: the server died instead of refusing ({error})")
@ -1099,12 +1242,13 @@ def evaluate(
scratch: Path, scratch: Path,
*, *,
anchors: AnchorSet | None = None, anchors: AnchorSet | None = None,
real: Mapping[str, Path] | None = None,
) -> list[Row]: ) -> list[Row]:
reachable = server_exists() reachable = server_exists()
bundles = corpus(scratch / "base") bundles = corpus(scratch / "base")
return [ return [
row_one(bundles, reachable), row_one(bundles, reachable),
row_two(bundles, reachable, anchors), row_two(bundles, reachable, anchors, real),
row_three(scratch), row_three(scratch),
row_four(scratch), row_four(scratch),
row_five(scratch), row_five(scratch),
@ -1131,6 +1275,35 @@ def render(rows: Sequence[Row]) -> str:
return "\n".join(lines) + "\n" return "\n".join(lines) + "\n"
def _real_bundles(roots: Sequence[Path], anchors: AnchorSet | None) -> dict[str, Path]:
"""Map every bundle the set names to a directory under the given roots.
The match is on the SET's name against the directory name's leading
segment, and the served id is then read from the bundle itself. A set name
matching two directories is a usage error: choosing one would make the
measurement depend on directory order.
"""
if not roots or anchors is None:
return {}
wanted = sorted({bundle for bundle, _anchor, _quote in anchors.pairs})
found: dict[str, Path] = {}
for name in wanted:
hits = [
child
for root in roots
for child in sorted(root.iterdir())
if child.is_dir() and child.name.startswith(name)
]
if len(hits) > 1:
raise GateUsage(
f"`{name}` matches {len(hits)} directories under the given roots: "
f"{', '.join(hit.name for hit in hits)}"
)
if hits:
found[name] = hits[0]
return found
def main(argv: list[str] | None = None) -> int: def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0]) parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0])
parser.add_argument("--json", action="store_true", help="emit the rows as JSON") parser.add_argument("--json", action="store_true", help="emit the rows as JSON")
@ -1144,6 +1317,17 @@ def main(argv: list[str] | None = None) -> int:
type=Path, type=Path,
help="the freeze file that pins --sett by sha256 and declares its version", help="the freeze file that pins --sett by sha256 and declares its version",
) )
parser.add_argument(
"--bundle-root",
type=Path,
action="append",
default=[],
help=(
"a directory holding the set's bundles, for row 2 (repeatable). The "
"bundles are never committed here; the row maps a set name to a served "
"bundle by reading the bundle's own declared id"
),
)
parser.add_argument( parser.add_argument(
"--sett-versjon", "--sett-versjon",
type=int, type=int,
@ -1159,8 +1343,9 @@ def main(argv: list[str] | None = None) -> int:
) )
if (args.sett is None) != (args.frys is None): if (args.sett is None) != (args.frys is None):
raise GateUsage("--sett and --frys are given together or not at all") raise GateUsage("--sett and --frys are given together or not at all")
real = _real_bundles(args.bundle_root, anchors)
with tempfile.TemporaryDirectory(prefix="okf-mcp-gate-") as scratch: with tempfile.TemporaryDirectory(prefix="okf-mcp-gate-") as scratch:
rows = evaluate(Path(scratch), anchors=anchors) rows = evaluate(Path(scratch), anchors=anchors, real=real)
except GateUsage as error: except GateUsage as error:
print(f"okf-mcp-gate: {error}", file=sys.stderr) print(f"okf-mcp-gate: {error}", file=sys.stderr)
return 2 return 2