open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(mcp): the bundle's map, and a working method that reads it first

Browse source 
C5. `bundlemap.build_map` lists a bundle in its own words: one line per
source document -- its name, then the titles of its concepts in document
order -- and documents whose names differ only in their numbers (a changelog
per release, a note per week) as ONE line: the name with every number as `#`,
the count, the first and last by natural order, and the titles across the
series that are words. `SERIES_MIN` = 5, at most `TITLES_PER_LINE` = 24 titles
a line, the lines capped at `MAP_MAX_BYTES` = 48 000 together with
`lines_truncated` counting the rest. Derived on every call, never stored.

The card (`okf card`, `okf_describe`) carries it as `map` and no longer
carries `source_files`: that list named every document a second time with no
series collapsed, a quarter of the reply on a large bundle, for names the map
already carries. Chose removal over keeping both because the describe reply
has to fit a client's tool-reply limit and the map says more.

The working method now reads: take the map first (`okf card`, or
`okf_describe`), write two to four sub-questions in its words, and send them
in ONE call (`--question` repeated, or `okf_ask` `questions`). Changed in the
skill template, the generated `skills/okf-consume`, and the server
instructions (held under the 2 KB a client keeps). The regeneration recipe for
`skills/okf-consume` gains `--for-bundle`: since v1.1 the generator writes the
generic skill by default, so the recipe as published produced the other file.

A test holds a four-sub-question `okf_ask` over concepts far over the passage
size under 50 000 bytes of reply text (25 000 tokens at a pessimistic two
bytes a token). The real-collection measurements are kept in local state.

Suite on a clean tree after `git add`: 2429 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-21 08:37:12 +02:00
commit da6faf8776
9 changed files with 450 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

60
src/llm_ingestion_okf/mcp_server.py
View file

@ -84,22 +84,21 @@ CLIENT_TRUNCATION_BYTES = 2048
#: under the cap by a test, with a control so the assertion is a measurement.
SERVER_INSTRUCTIONS = (
"Bundles are read-only and no call here runs a model.\n\n"
"HOW TO USE THIS SERVER. Read the bundle's map first with `okf_describe`, "
"then put the question into the bundle's own words -- its documents may be "
"HOW TO USE THIS SERVER. Read the bundle's `map` first with `okf_describe`: "
"one line per document with its section titles -- the bundle's own words. "
"Then write two to four sub-questions in THOSE words (its documents may be "
"written in another language than the question, and the ranking matches "
"words. Split a broad question into two to four sub-questions and call "
"`okf_ask` once per sub-question. After each call read BOTH what came back "
"and what lay just outside the cut: `withheld.nearest` names the "
"best-ranked concepts that missed, with their titles. If one of them is "
"what you wanted, that is a fact about the WORDS, not a closed door -- ask "
"again with that concept's own words, or fetch it by name with "
"`okf_fetch`. Several calls are normal and expected; there is no limit and "
"no penalty. When `coverage.weak` is true, rephrase in the bundle's words, "
"and if it stays weak say the bundle does not cover the question. Then "
"write ONE answer, ordered by sub-question, in the "
"questioner's language and in ordinary prose, citing the document and the "
"section (and the bundle, when you read more than one). Say plainly what "
"the bundles do not cover.\n\n"
"words) and send them in ONE call: `okf_ask` with `questions`. Each excerpt "
"names the sub-questions it answered. Read BOTH what came back and what lay "
"just outside the cut: `withheld.nearest` names the best-ranked concepts "
"that missed, with their titles. If one of them is what you wanted, that is "
"a fact about the WORDS, not a closed door -- ask again with that concept's "
"own words, or fetch it by name with `okf_fetch`. Asking again is normal and "
"expected. When `coverage.weak` is true, rephrase in the bundle's words, and "
"if it stays weak say the bundle does not cover the question. Then write ONE "
"answer, ordered by sub-question, in the questioner's language and in "
"ordinary prose, citing the document and the section (and the bundle, when "
"you read more than one). Say plainly what the bundles do not cover.\n\n"
"Every excerpt carries the bundle id and concept id a claim must be "
"attributed to; the payload states what it withheld and why."
)
@ -310,21 +309,11 @@ def card(bundle_root: Path, *, profile: BundleProfile, concept_sample: int = 50)
card would also be one more artefact that can be stale, which is the defect
it was meant to remove.
"""
from . import bundlemap
from . import skill as okf_skill
bundle_id = okf_consume.root_bundle_id_of(bundle_root, profile=profile)
concepts = okf_consume.link_parents(
[
okf_consume.read_concept(
okf_consume.read_path_in_bundle(
bundle_root, f"{concept_id}{profile.paths.concept_suffix}"
),
bundle_root=bundle_root,
root_bundle_id=bundle_id,
)
for concept_id in okf_consume.enumerate_concepts(bundle_root, profile=profile)
]
)
concepts = bundlemap.read_concepts(bundle_root, profile=profile)
counts = okf_skill.field_counts(concepts)
return {
"bundle_id": bundle_id,
@ -334,15 +323,17 @@ def card(bundle_root: Path, *, profile: BundleProfile, concept_sample: int = 50)
"concept_count": len(concepts),
"concepts": [concept.concept_id for concept in concepts[:concept_sample]],
"concepts_truncated": len(concepts) > concept_sample,
"source_files": sorted(
{concept.source_file for concept in concepts if concept.source_file}
),
"conditional_fields": {
field: counts.get(field, 0) for field in okf_skill.CONDITIONAL_FIELDS
},
"whole_bundle_bytes": okf_skill.whole_bundle_cost(concepts),
"budget_unit": okf_consume.BUDGET_UNIT,
"default_limit": okf_consume.DEFAULT_LIMIT,
# v1.1 C5: the bundle's own words, to write sub-questions in. It
# replaces the flat `source_files` list, which named every document a
# second time with no series collapsed -- a quarter of the reply on a
# large bundle, for names the map already carries.
"map": bundlemap.build_map(concepts),
}
@ -379,7 +370,7 @@ def tools(surface: Surface) -> tuple[Tool, ...]:
Tool(
"okf_list",
"Every OKF bundle this server can currently reach, with its content "
"identity and concept count. Re-read from disk on every call, so a "
"identity and concept count; `okf_describe` gives each one's map. Re-read from disk on every call, so a "
"bundle added, removed or rebuilt since the last call is reflected "
"without restarting anything. Exists because a client that cannot "
"discover bundles must be told their names out of band, which is the "
@ -391,9 +382,10 @@ def tools(surface: Surface) -> tuple[Tool, ...]:
Tool(
"okf_describe",
"What one bundle is: its id, its content identity, how many concepts "
"it holds, which source documents it was built from, and which "
"conditionally-written fields are present on how many concepts. "
"Read it BEFORE asking, so the question can be put into the "
"it holds, which conditionally-written fields are present on how "
"many concepts, and its `map` -- one line per source document with "
"its section titles, a series of like-named documents as one line. "
"Read it BEFORE asking, so the sub-questions can be put into the "
"bundle's own words. On a multi-bundle server, omitting `bundle_id` "
"describes every served bundle, as `okf_ask` does. "
"Exists because an answer must be attributable -- a claim from a "