open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(mcp): the bundle's map, and a working method that reads it first

Browse source 
C5. `bundlemap.build_map` lists a bundle in its own words: one line per
source document -- its name, then the titles of its concepts in document
order -- and documents whose names differ only in their numbers (a changelog
per release, a note per week) as ONE line: the name with every number as `#`,
the count, the first and last by natural order, and the titles across the
series that are words. `SERIES_MIN` = 5, at most `TITLES_PER_LINE` = 24 titles
a line, the lines capped at `MAP_MAX_BYTES` = 48 000 together with
`lines_truncated` counting the rest. Derived on every call, never stored.

The card (`okf card`, `okf_describe`) carries it as `map` and no longer
carries `source_files`: that list named every document a second time with no
series collapsed, a quarter of the reply on a large bundle, for names the map
already carries. Chose removal over keeping both because the describe reply
has to fit a client's tool-reply limit and the map says more.

The working method now reads: take the map first (`okf card`, or
`okf_describe`), write two to four sub-questions in its words, and send them
in ONE call (`--question` repeated, or `okf_ask` `questions`). Changed in the
skill template, the generated `skills/okf-consume`, and the server
instructions (held under the 2 KB a client keeps). The regeneration recipe for
`skills/okf-consume` gains `--for-bundle`: since v1.1 the generator writes the
generic skill by default, so the recipe as published produced the other file.

A test holds a four-sub-question `okf_ask` over concepts far over the passage
size under 50 000 bytes of reply text (25 000 tokens at a pessimistic two
bytes a token). The real-collection measurements are kept in local state.

Suite on a clean tree after `git add`: 2429 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-21 08:37:12 +02:00
commit da6faf8776
9 changed files with 450 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

181
src/llm_ingestion_okf/bundlemap.py Normal file
View file

@ -0,0 +1,181 @@
"""The map of a bundle: its documents and their titles, in its own words (v1.1 C5).
WHY IT EXISTS. The ranking matches words, and a question put in words the
collection does not use finds little however good the ranking is -- a question
asked in one language of a collection written in another most of all. The
reader closes that gap by rewriting the question into two to four
sub-questions in the collection's OWN words, and the one place those words
are listed is the collection itself. This module lists them, compactly enough
to be read before the first question: one line per source document, its name
and then the titles of its concepts in document order.
A SERIES IS ONE LINE. Documents whose names differ only in their numbers -- a
changelog per release, a note per week -- are one kind of document, and four
hundred lines saying so crowd out everything else a reader needs. They are
written as one line: the name with every number as `#`, how many documents,
the first and the last by natural order, and the titles across the series that
are words (a title that is only a version number names nothing).
DERIVED, NEVER STORED, like the card that carries it: the map is recomputed
from the bundle on every call, so it cannot disagree with the bytes beside it.
Deterministic: every order is by name, by position or by a count with the
name breaking ties.
"""
from __future__ import annotations
import re
from collections import Counter
from collections.abc import Sequence
from pathlib import Path
from .consume import (
Concept,
enumerate_concepts,
inherit_table_titles,
link_parents,
read_concept,
read_path_in_bundle,
root_bundle_id_of,
)
from .profiles import BundleProfile
#: How many documents sharing one name template make a series. Below it the
#: documents are listed one by one: two or three dated notes are still worth
#: their own lines, and a template shared by chance should not hide them.
SERIES_MIN = 5
#: The most titles one line lists before it says how many it left out. A
#: document is a handful of sections as a rule; a few are hundreds, and one
#: of those must not cost the whole map its room.
TITLES_PER_LINE = 24
#: The most bytes the map's lines take, together. A client keeps a tool reply
#: of 25 000 tokens (Claude Code's MCP output limit); at a pessimistic two
#: bytes a token that is 50 000 bytes, and the rest of the card needs a few
#: thousand. The largest bundle this was measured on stays under it, so the
#: ceiling is a guard for a larger one. Lines past it are counted in
#: `lines_truncated`, never dropped silently.
MAP_MAX_BYTES = 48_000
_DIGITS = re.compile(r"\d+")
_SPLIT = re.compile(r"(\d+)")
_LETTER = re.compile(r"[^\W\d_]")
#: The locators a concept's place in its document is read off, one per
#: document and never mixed (`consume.inherit_table_titles` reads the same).
_POSITION_KEYS = ("source_offset", "source_lines")
_FIRST_NUMBER = re.compile(r"\s*\[\s*(\d+)")
def _stem(source_file: str) -> str:
return source_file.removesuffix(".md")
def _natural(name: str) -> tuple[tuple[int, str], ...]:
"""Numbers compared as numbers: `v1-2` before `v1-13`."""
return tuple(
(int(part), "") if part.isdigit() else (-1, part) for part in _SPLIT.split(name) if part
)
def _position(concept: Concept, key: str) -> int | None:
match = _FIRST_NUMBER.match(concept.locators.get(key, ""))
return int(match.group(1)) if match else None
def _in_document_order(concepts: Sequence[Concept]) -> list[Concept]:
for key in _POSITION_KEYS:
positions = [_position(concept, key) for concept in concepts]
if all(position is not None for position in positions):
return [
concept
for _, _, concept in sorted(
(position, index, concept)
for index, (position, concept) in enumerate(
zip(positions, concepts, strict=True)
)
)
]
return list(concepts)
def _titled(titles: Sequence[str]) -> str:
kept = titles[:TITLES_PER_LINE]
text = " · ".join(kept)
if len(titles) > len(kept):
text += f" · (+{len(titles) - len(kept)} more)"
return text
def build_map(concepts: Sequence[Concept]) -> dict[str, object]:
"""The map of `concepts`: one line per document, one per series."""
by_document: dict[str, list[Concept]] = {}
for concept in concepts:
by_document.setdefault(_stem(concept.source_file), []).append(concept)
by_template: dict[str, list[str]] = {}
for document in by_document:
by_template.setdefault(_DIGITS.sub("#", document), []).append(document)
entries: list[tuple[str, str]] = []
for template, documents in by_template.items():
if len(documents) >= SERIES_MIN:
ordered = sorted(documents, key=_natural)
counts: Counter[str] = Counter(
title
for document in documents
for title in dict.fromkeys(concept.title for concept in by_document[document])
if _LETTER.search(title)
)
titles = sorted(counts, key=lambda title: (-counts[title], title))
line = f"{template} ({len(documents)} documents: {ordered[0]} … {ordered[-1]})"
if titles:
line += f": {_titled(titles)}"
entries.append((template, line))
continue
for document in documents:
titles = list(
dict.fromkeys(
concept.title for concept in _in_document_order(by_document[document])
)
)
name = document or "(no source file)"
entries.append((document, f"{name}: {_titled(titles)}"))
lines = [line for _, line in sorted(entries, key=lambda entry: (_natural(entry[0]), entry[0]))]
kept: list[str] = []
spent = 0
for line in lines:
size = len(line.encode("utf-8"))
if spent + size > MAP_MAX_BYTES:
break
kept.append(line)
spent += size
return {
"documents": len(by_document),
"concepts": len(concepts),
"lines_count": len(lines),
"lines_truncated": len(lines) - len(kept),
"lines": kept,
}
def read_concepts(bundle_root: Path, *, profile: BundleProfile) -> list[Concept]:
"""Every concept of the bundle, as `okf consume` reads them -- parents
linked and a table fragment named by the heading above it."""
bundle_id = root_bundle_id_of(bundle_root, profile=profile)
return inherit_table_titles(
link_parents(
[
read_concept(
read_path_in_bundle(bundle_root, f"{concept_id}{profile.paths.concept_suffix}"),
bundle_root=bundle_root,
root_bundle_id=bundle_id,
)
for concept_id in enumerate_concepts(bundle_root, profile=profile)
]
)
)
def bundle_map(bundle_root: Path, *, profile: BundleProfile) -> dict[str, object]:
return build_map(read_concepts(bundle_root, profile=profile))

60
src/llm_ingestion_okf/mcp_server.py
View file

@ -84,22 +84,21 @@ CLIENT_TRUNCATION_BYTES = 2048
#: under the cap by a test, with a control so the assertion is a measurement.
SERVER_INSTRUCTIONS = (
"Bundles are read-only and no call here runs a model.\n\n"
"HOW TO USE THIS SERVER. Read the bundle's map first with `okf_describe`, "
"then put the question into the bundle's own words -- its documents may be "
"HOW TO USE THIS SERVER. Read the bundle's `map` first with `okf_describe`: "
"one line per document with its section titles -- the bundle's own words. "
"Then write two to four sub-questions in THOSE words (its documents may be "
"written in another language than the question, and the ranking matches "
"words. Split a broad question into two to four sub-questions and call "
"`okf_ask` once per sub-question. After each call read BOTH what came back "
"and what lay just outside the cut: `withheld.nearest` names the "
"best-ranked concepts that missed, with their titles. If one of them is "
"what you wanted, that is a fact about the WORDS, not a closed door -- ask "
"again with that concept's own words, or fetch it by name with "
"`okf_fetch`. Several calls are normal and expected; there is no limit and "
"no penalty. When `coverage.weak` is true, rephrase in the bundle's words, "
"and if it stays weak say the bundle does not cover the question. Then "
"write ONE answer, ordered by sub-question, in the "
"questioner's language and in ordinary prose, citing the document and the "
"section (and the bundle, when you read more than one). Say plainly what "
"the bundles do not cover.\n\n"
"words) and send them in ONE call: `okf_ask` with `questions`. Each excerpt "
"names the sub-questions it answered. Read BOTH what came back and what lay "
"just outside the cut: `withheld.nearest` names the best-ranked concepts "
"that missed, with their titles. If one of them is what you wanted, that is "
"a fact about the WORDS, not a closed door -- ask again with that concept's "
"own words, or fetch it by name with `okf_fetch`. Asking again is normal and "
"expected. When `coverage.weak` is true, rephrase in the bundle's words, and "
"if it stays weak say the bundle does not cover the question. Then write ONE "
"answer, ordered by sub-question, in the questioner's language and in "
"ordinary prose, citing the document and the section (and the bundle, when "
"you read more than one). Say plainly what the bundles do not cover.\n\n"
"Every excerpt carries the bundle id and concept id a claim must be "
"attributed to; the payload states what it withheld and why."
)
@ -310,21 +309,11 @@ def card(bundle_root: Path, *, profile: BundleProfile, concept_sample: int = 50)
card would also be one more artefact that can be stale, which is the defect
it was meant to remove.
"""
from . import bundlemap
from . import skill as okf_skill
bundle_id = okf_consume.root_bundle_id_of(bundle_root, profile=profile)
concepts = okf_consume.link_parents(
[
okf_consume.read_concept(
okf_consume.read_path_in_bundle(
bundle_root, f"{concept_id}{profile.paths.concept_suffix}"
),
bundle_root=bundle_root,
root_bundle_id=bundle_id,
)
for concept_id in okf_consume.enumerate_concepts(bundle_root, profile=profile)
]
)
concepts = bundlemap.read_concepts(bundle_root, profile=profile)
counts = okf_skill.field_counts(concepts)
return {
"bundle_id": bundle_id,
@ -334,15 +323,17 @@ def card(bundle_root: Path, *, profile: BundleProfile, concept_sample: int = 50)
"concept_count": len(concepts),
"concepts": [concept.concept_id for concept in concepts[:concept_sample]],
"concepts_truncated": len(concepts) > concept_sample,
"source_files": sorted(
{concept.source_file for concept in concepts if concept.source_file}
),
"conditional_fields": {
field: counts.get(field, 0) for field in okf_skill.CONDITIONAL_FIELDS
},
"whole_bundle_bytes": okf_skill.whole_bundle_cost(concepts),
"budget_unit": okf_consume.BUDGET_UNIT,
"default_limit": okf_consume.DEFAULT_LIMIT,
# v1.1 C5: the bundle's own words, to write sub-questions in. It
# replaces the flat `source_files` list, which named every document a
# second time with no series collapsed -- a quarter of the reply on a
# large bundle, for names the map already carries.
"map": bundlemap.build_map(concepts),
}
@ -379,7 +370,7 @@ def tools(surface: Surface) -> tuple[Tool, ...]:
Tool(
"okf_list",
"Every OKF bundle this server can currently reach, with its content "
"identity and concept count. Re-read from disk on every call, so a "
"identity and concept count; `okf_describe` gives each one's map. Re-read from disk on every call, so a "
"bundle added, removed or rebuilt since the last call is reflected "
"without restarting anything. Exists because a client that cannot "
"discover bundles must be told their names out of band, which is the "
@ -391,9 +382,10 @@ def tools(surface: Surface) -> tuple[Tool, ...]:
Tool(
"okf_describe",
"What one bundle is: its id, its content identity, how many concepts "
"it holds, which source documents it was built from, and which "
"conditionally-written fields are present on how many concepts. "
"Read it BEFORE asking, so the question can be put into the "
"it holds, which conditionally-written fields are present on how "
"many concepts, and its `map` -- one line per source document with "
"its section titles, a series of like-named documents as one line. "
"Read it BEFORE asking, so the sub-questions can be put into the "
"bundle's own words. On a multi-bundle server, omitting `bundle_id` "
"describes every served bundle, as `okf_ask` does. "
"Exists because an answer must be attributable -- a claim from a "