feat(mcp): the bundle's map, and a working method that reads it first
C5. `bundlemap.build_map` lists a bundle in its own words: one line per source document -- its name, then the titles of its concepts in document order -- and documents whose names differ only in their numbers (a changelog per release, a note per week) as ONE line: the name with every number as `#`, the count, the first and last by natural order, and the titles across the series that are words. `SERIES_MIN` = 5, at most `TITLES_PER_LINE` = 24 titles a line, the lines capped at `MAP_MAX_BYTES` = 48 000 together with `lines_truncated` counting the rest. Derived on every call, never stored. The card (`okf card`, `okf_describe`) carries it as `map` and no longer carries `source_files`: that list named every document a second time with no series collapsed, a quarter of the reply on a large bundle, for names the map already carries. Chose removal over keeping both because the describe reply has to fit a client's tool-reply limit and the map says more. The working method now reads: take the map first (`okf card`, or `okf_describe`), write two to four sub-questions in its words, and send them in ONE call (`--question` repeated, or `okf_ask` `questions`). Changed in the skill template, the generated `skills/okf-consume`, and the server instructions (held under the 2 KB a client keeps). The regeneration recipe for `skills/okf-consume` gains `--for-bundle`: since v1.1 the generator writes the generic skill by default, so the recipe as published produced the other file. A test holds a four-sub-question `okf_ask` over concepts far over the passage size under 50 000 bytes of reply text (25 000 tokens at a pessimistic two bytes a token). The real-collection measurements are kept in local state. Suite on a clean tree after `git add`: 2429 passed, 2 skipped, 4 xfailed. ruff, ruff format, mypy --strict clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
f7cd84c5e6
commit
da6faf8776
9 changed files with 450 additions and 64 deletions
|
|
@ -1556,6 +1556,13 @@ R761 **8** (S1-S6 + KP + KN), vegnormal **32** questions / **43**
|
||||||
nearest concept above it in its document (`inherit_table_titles`, ordered by
|
nearest concept above it in its document (`inherit_table_titles`, ordered by
|
||||||
`source_offset` else `source_lines`), in ranking, excerpt and near misses;
|
`source_offset` else `source_lines`), in ranking, excerpt and near misses;
|
||||||
the excerpt carries `own_title`. A reading only -- no bundle bytes move.
|
the excerpt carries `own_title`. A reading only -- no bundle bytes move.
|
||||||
|
**C5 the map:** `bundlemap.build_map` -- one line per source document with
|
||||||
|
its concept titles in document order, a series (names differing only in
|
||||||
|
numbers, `SERIES_MIN` = 5) as one line with count and span, at most
|
||||||
|
`TITLES_PER_LINE` = 24 titles a line, lines capped at `MAP_MAX_BYTES` =
|
||||||
|
48 000 (`lines_truncated`). The card (`okf card`, `okf_describe`) carries it
|
||||||
|
as `map` and no longer carries `source_files`. The skill and the server
|
||||||
|
instructions say: read the map, 2-4 sub-questions in its words, ONE call.
|
||||||
What follows describes the fusion.
|
What follows describes the fusion.
|
||||||
- Consume a bundle: `okf consume <bundle> --question "<q>"
|
- Consume a bundle: `okf consume <bundle> --question "<q>"
|
||||||
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
|
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
|
||||||
|
|
|
||||||
18
README.md
18
README.md
|
|
@ -1331,15 +1331,15 @@ says why it exists:
|
||||||
| tool | what it answers |
|
| tool | what it answers |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `okf_list` | which bundles are reachable right now, with each one's content identity and concept count (multi-bundle servers only) |
|
| `okf_list` | which bundles are reachable right now, with each one's content identity and concept count (multi-bundle servers only) |
|
||||||
| `okf_describe` | what one bundle is: id, ref, concept count, source documents, and how many concepts carry each conditionally-written field. Omitting `bundle_id` on a multi-bundle server describes them all, as `okf_ask` does |
|
| `okf_describe` | what one bundle is: id, ref, concept count, how many concepts carry each conditionally-written field, and its `map` — one line per source document with its section titles, a series of like-named documents as one line. Omitting `bundle_id` on a multi-bundle server describes them all, as `okf_ask` does |
|
||||||
| `okf_ask` | one question, one bounded payload of excerpts, each with its bundle id, concept id, title and provenance locators. Omitting `bundle_id` on a multi-bundle server asks them all and splits the budget |
|
| `okf_ask` | one question, or two to four sub-questions in `questions`, and one bounded payload of excerpts, each with its bundle id, concept id, title and provenance locators. Omitting `bundle_id` on a multi-bundle server asks them all and splits the budget |
|
||||||
| `okf_fetch` | one named concept, verbatim, with its frontmatter and locators |
|
| `okf_fetch` | one named concept, verbatim, with its frontmatter and locators |
|
||||||
|
|
||||||
**The server carries the working method, because a subagent inherits MCP tools
|
**The server carries the working method, because a subagent inherits MCP tools
|
||||||
and not skills.** Its `instructions` and the `okf_ask` description state the
|
and not skills.** Its `instructions` and the `okf_ask` description state the
|
||||||
short form — read the map, put the question into the bundle's own words, split
|
short form — read the map, write two to four sub-questions in the bundle's own
|
||||||
it into sub-questions, read what lay just outside the cut and ask again with
|
words and send them in ONE call, read what lay just outside the cut and ask
|
||||||
its words, then write one answer in the questioner's language. Claude Code
|
again with its words, then write one answer in the questioner's language. Claude Code
|
||||||
truncates both at 2 KB, so the long form stays in the skill, which has no such
|
truncates both at 2 KB, so the long form stays in the skill, which has no such
|
||||||
cap; a test holds the short one under the limit with a control, because a
|
cap; a test holds the short one under the limit with a control, because a
|
||||||
truncated method is worse than a missing one.
|
truncated method is worse than a missing one.
|
||||||
|
|
@ -1387,6 +1387,14 @@ never written into the bundle**, so there is no second artefact that can
|
||||||
disagree with the bytes beside it. It is therefore never stale, and one skill
|
disagree with the bytes beside it. It is therefore never stale, and one skill
|
||||||
serves every bundle a project holds.
|
serves every bundle a project holds.
|
||||||
|
|
||||||
|
**The card carries the bundle's map** (`map`): one line per source document —
|
||||||
|
its name, then the titles of its concepts in document order — and documents
|
||||||
|
whose names differ only in their numbers (a changelog per release) as one line
|
||||||
|
with the count and the span. It is the bundle's own words, to write
|
||||||
|
sub-questions in. The lines are capped at 48 000 bytes together
|
||||||
|
(`lines_truncated` counts what a larger bundle leaves out), and a line lists at
|
||||||
|
most 24 titles. The map replaced the card's flat `source_files` list.
|
||||||
|
|
||||||
`okf skill <bundle> --for-bundle` still writes the per-bundle form, with the
|
`okf skill <bundle> --for-bundle` still writes the per-bundle form, with the
|
||||||
identity and the numbers measured into the text — which is exactly what makes
|
identity and the numbers measured into the text — which is exactly what makes
|
||||||
that file stale the moment the bundle is rebuilt. It refuses out loud when it
|
that file stale the moment the bundle is rebuilt. It refuses out loud when it
|
||||||
|
|
|
||||||
|
|
@ -57,28 +57,31 @@ and the cut; it decides nothing about the question.
|
||||||
Five steps, in this order. The pre-pass is step three, not step one: a question
|
Five steps, in this order. The pre-pass is step three, not step one: a question
|
||||||
asked in the wrong words reaches the wrong concepts however good the ranking is.
|
asked in the wrong words reaches the wrong concepts however good the ranking is.
|
||||||
|
|
||||||
**1. Understand the question first.** Read the bundle's map before you search
|
**1. Understand the question first.** Read the bundle's `map` before you search
|
||||||
it — what it holds, how many concepts, what the documents are called. Then put
|
it — `okf card <BUNDLE_ROOT>` prints it: one line per document with its section
|
||||||
the question into the bundle's own words: a bundle written in one language and
|
titles, a series of like-named documents as one line. Then put the question
|
||||||
a question asked in another share few tokens, and the pre-pass matches tokens.
|
into the bundle's own words: a bundle written in one language and a question
|
||||||
Take the terms from the bundle's own titles, not from your vocabulary.
|
asked in another share few tokens, and the pre-pass matches tokens. Take the
|
||||||
|
terms from the map's titles, not from your vocabulary.
|
||||||
|
|
||||||
**2. Split a broad question into 2–4 sub-questions.** One search for a question
|
**2. Split a broad question into 2–4 sub-questions.** One search for a question
|
||||||
with four parts returns the best eight concepts for the average of the four,
|
with four parts returns the best eight concepts for the average of the four,
|
||||||
which is often the best eight for none of them. Write the sub-questions down;
|
which is often the best eight for none of them. Write the sub-questions down;
|
||||||
they are also the shape of the answer.
|
they are also the shape of the answer.
|
||||||
|
|
||||||
**3. Search per sub-question. Several searches are normal, and searching again
|
**3. Search all sub-questions in ONE run. Several searches are normal, and
|
||||||
is allowed and expected.** Run the pre-pass once per sub-question:
|
searching again is allowed and expected.** Give the pre-pass every sub-question
|
||||||
|
at once:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
okf consume <BUNDLE_ROOT> --question "one sub-question" --out /tmp/p1.json
|
okf consume <BUNDLE_ROOT> --question "first sub-question" --question "second sub-question" --out /tmp/p1.json
|
||||||
```
|
```
|
||||||
|
|
||||||
After each run, read two things: what came back, and what lay just outside the
|
Each sub-question is ranked alone and the answers are interleaved; every
|
||||||
cut. `withheld.nearest` names the best-ranked concepts that missed, with their
|
excerpt names the `subquestions` it answered. After each run, read two things:
|
||||||
titles — if one of them is what you were after, that is a signal about the
|
what came back, and what lay just outside the cut. `withheld.nearest` names
|
||||||
WORDS, not a closed door. Search again with the words that concept uses, or
|
the best-ranked concepts that missed, with their titles — if one of them is
|
||||||
|
what you were after, that is a signal about the WORDS, not a closed door. Search again with the words that concept uses, or
|
||||||
ask for it by name. There is no limit on runs and no penalty for a run that
|
ask for it by name. There is no limit on runs and no penalty for a run that
|
||||||
found nothing; a run that found nothing is a measurement, and its denominator
|
found nothing; a run that found nothing is a measurement, and its denominator
|
||||||
is worth carrying. When `coverage.weak` is true — a word of yours the bundle
|
is worth carrying. When `coverage.weak` is true — a word of yours the bundle
|
||||||
|
|
|
||||||
|
|
@ -68,28 +68,31 @@ and the cut; it decides nothing about the question.
|
||||||
Five steps, in this order. The pre-pass is step three, not step one: a question
|
Five steps, in this order. The pre-pass is step three, not step one: a question
|
||||||
asked in the wrong words reaches the wrong concepts however good the ranking is.
|
asked in the wrong words reaches the wrong concepts however good the ranking is.
|
||||||
|
|
||||||
**1. Understand the question first.** Read the bundle's map before you search
|
**1. Understand the question first.** Read the bundle's `map` before you search
|
||||||
it — what it holds, how many concepts, what the documents are called. Then put
|
it — `okf card examples/ingest-golden-segmented-okf-v0-2/expected-bundle` prints it: one line per document with its section
|
||||||
the question into the bundle's own words: a bundle written in one language and
|
titles, a series of like-named documents as one line. Then put the question
|
||||||
a question asked in another share few tokens, and the pre-pass matches tokens.
|
into the bundle's own words: a bundle written in one language and a question
|
||||||
Take the terms from the bundle's own titles, not from your vocabulary.
|
asked in another share few tokens, and the pre-pass matches tokens. Take the
|
||||||
|
terms from the map's titles, not from your vocabulary.
|
||||||
|
|
||||||
**2. Split a broad question into 2–4 sub-questions.** One search for a question
|
**2. Split a broad question into 2–4 sub-questions.** One search for a question
|
||||||
with four parts returns the best eight concepts for the average of the four,
|
with four parts returns the best eight concepts for the average of the four,
|
||||||
which is often the best eight for none of them. Write the sub-questions down;
|
which is often the best eight for none of them. Write the sub-questions down;
|
||||||
they are also the shape of the answer.
|
they are also the shape of the answer.
|
||||||
|
|
||||||
**3. Search per sub-question. Several searches are normal, and searching again
|
**3. Search all sub-questions in ONE run. Several searches are normal, and
|
||||||
is allowed and expected.** Run the pre-pass once per sub-question:
|
searching again is allowed and expected.** Give the pre-pass every sub-question
|
||||||
|
at once:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
okf consume examples/ingest-golden-segmented-okf-v0-2/expected-bundle --question "one sub-question" --out /tmp/p1.json
|
okf consume examples/ingest-golden-segmented-okf-v0-2/expected-bundle --question "first sub-question" --question "second sub-question" --out /tmp/p1.json
|
||||||
```
|
```
|
||||||
|
|
||||||
After each run, read two things: what came back, and what lay just outside the
|
Each sub-question is ranked alone and the answers are interleaved; every
|
||||||
cut. `withheld.nearest` names the best-ranked concepts that missed, with their
|
excerpt names the `subquestions` it answered. After each run, read two things:
|
||||||
titles — if one of them is what you were after, that is a signal about the
|
what came back, and what lay just outside the cut. `withheld.nearest` names
|
||||||
WORDS, not a closed door. Search again with the words that concept uses, or
|
the best-ranked concepts that missed, with their titles — if one of them is
|
||||||
|
what you were after, that is a signal about the WORDS, not a closed door. Search again with the words that concept uses, or
|
||||||
ask for it by name. There is no limit on runs and no penalty for a run that
|
ask for it by name. There is no limit on runs and no penalty for a run that
|
||||||
found nothing; a run that found nothing is a measurement, and its denominator
|
found nothing; a run that found nothing is a measurement, and its denominator
|
||||||
is worth carrying. When `coverage.weak` is true — a word of yours the bundle
|
is worth carrying. When `coverage.weak` is true — a word of yours the bundle
|
||||||
|
|
|
||||||
|
|
@ -12,7 +12,8 @@ Regenerate them from the repository root rather than editing either file, with
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
okf skill examples/ingest-golden-segmented-okf-v0-2/expected-bundle \
|
okf skill examples/ingest-golden-segmented-okf-v0-2/expected-bundle \
|
||||||
--out skills/okf-consume --force --example-question "Hva sier veiledningen om krav?"
|
--out skills/okf-consume --force --for-bundle \
|
||||||
|
--example-question "Hva sier veiledningen om krav?"
|
||||||
python3 -c 'import os, pathlib; p = pathlib.Path("skills/okf-consume/SKILL.md"); p.write_text(p.read_text(encoding="utf-8").replace(os.path.realpath(".") + "/", ""), encoding="utf-8")'
|
python3 -c 'import os, pathlib; p = pathlib.Path("skills/okf-consume/SKILL.md"); p.write_text(p.read_text(encoding="utf-8").replace(os.path.realpath(".") + "/", ""), encoding="utf-8")'
|
||||||
okf check --skill skills/okf-consume/SKILL.md \
|
okf check --skill skills/okf-consume/SKILL.md \
|
||||||
--payload skills/okf-consume/references/example-payload.json
|
--payload skills/okf-consume/references/example-payload.json
|
||||||
|
|
@ -20,6 +21,8 @@ okf check --skill skills/okf-consume/SKILL.md \
|
||||||
|
|
||||||
Why each part is there:
|
Why each part is there:
|
||||||
|
|
||||||
|
- **`--for-bundle`**: since v1.1 the generator writes the GENERIC skill by
|
||||||
|
default; this copy is the instantiated one, for this bundle.
|
||||||
- **`--force`**: the generator refuses to replace an existing `SKILL.md`
|
- **`--force`**: the generator refuses to replace an existing `SKILL.md`
|
||||||
(`refused (target_occupied)`), because a silent overwrite would destroy a
|
(`refused (target_occupied)`), because a silent overwrite would destroy a
|
||||||
hand-edited copy.
|
hand-edited copy.
|
||||||
|
|
|
||||||
181
src/llm_ingestion_okf/bundlemap.py
Normal file
181
src/llm_ingestion_okf/bundlemap.py
Normal file
|
|
@ -0,0 +1,181 @@
|
||||||
|
"""The map of a bundle: its documents and their titles, in its own words (v1.1 C5).
|
||||||
|
|
||||||
|
WHY IT EXISTS. The ranking matches words, and a question put in words the
|
||||||
|
collection does not use finds little however good the ranking is -- a question
|
||||||
|
asked in one language of a collection written in another most of all. The
|
||||||
|
reader closes that gap by rewriting the question into two to four
|
||||||
|
sub-questions in the collection's OWN words, and the one place those words
|
||||||
|
are listed is the collection itself. This module lists them, compactly enough
|
||||||
|
to be read before the first question: one line per source document, its name
|
||||||
|
and then the titles of its concepts in document order.
|
||||||
|
|
||||||
|
A SERIES IS ONE LINE. Documents whose names differ only in their numbers -- a
|
||||||
|
changelog per release, a note per week -- are one kind of document, and four
|
||||||
|
hundred lines saying so crowd out everything else a reader needs. They are
|
||||||
|
written as one line: the name with every number as `#`, how many documents,
|
||||||
|
the first and the last by natural order, and the titles across the series that
|
||||||
|
are words (a title that is only a version number names nothing).
|
||||||
|
|
||||||
|
DERIVED, NEVER STORED, like the card that carries it: the map is recomputed
|
||||||
|
from the bundle on every call, so it cannot disagree with the bytes beside it.
|
||||||
|
Deterministic: every order is by name, by position or by a count with the
|
||||||
|
name breaking ties.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections import Counter
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from .consume import (
|
||||||
|
Concept,
|
||||||
|
enumerate_concepts,
|
||||||
|
inherit_table_titles,
|
||||||
|
link_parents,
|
||||||
|
read_concept,
|
||||||
|
read_path_in_bundle,
|
||||||
|
root_bundle_id_of,
|
||||||
|
)
|
||||||
|
from .profiles import BundleProfile
|
||||||
|
|
||||||
|
#: How many documents sharing one name template make a series. Below it the
|
||||||
|
#: documents are listed one by one: two or three dated notes are still worth
|
||||||
|
#: their own lines, and a template shared by chance should not hide them.
|
||||||
|
SERIES_MIN = 5
|
||||||
|
|
||||||
|
#: The most titles one line lists before it says how many it left out. A
|
||||||
|
#: document is a handful of sections as a rule; a few are hundreds, and one
|
||||||
|
#: of those must not cost the whole map its room.
|
||||||
|
TITLES_PER_LINE = 24
|
||||||
|
|
||||||
|
#: The most bytes the map's lines take, together. A client keeps a tool reply
|
||||||
|
#: of 25 000 tokens (Claude Code's MCP output limit); at a pessimistic two
|
||||||
|
#: bytes a token that is 50 000 bytes, and the rest of the card needs a few
|
||||||
|
#: thousand. The largest bundle this was measured on stays under it, so the
|
||||||
|
#: ceiling is a guard for a larger one. Lines past it are counted in
|
||||||
|
#: `lines_truncated`, never dropped silently.
|
||||||
|
MAP_MAX_BYTES = 48_000
|
||||||
|
|
||||||
|
_DIGITS = re.compile(r"\d+")
|
||||||
|
_SPLIT = re.compile(r"(\d+)")
|
||||||
|
_LETTER = re.compile(r"[^\W\d_]")
|
||||||
|
|
||||||
|
#: The locators a concept's place in its document is read off, one per
|
||||||
|
#: document and never mixed (`consume.inherit_table_titles` reads the same).
|
||||||
|
_POSITION_KEYS = ("source_offset", "source_lines")
|
||||||
|
_FIRST_NUMBER = re.compile(r"\s*\[\s*(\d+)")
|
||||||
|
|
||||||
|
|
||||||
|
def _stem(source_file: str) -> str:
|
||||||
|
return source_file.removesuffix(".md")
|
||||||
|
|
||||||
|
|
||||||
|
def _natural(name: str) -> tuple[tuple[int, str], ...]:
|
||||||
|
"""Numbers compared as numbers: `v1-2` before `v1-13`."""
|
||||||
|
return tuple(
|
||||||
|
(int(part), "") if part.isdigit() else (-1, part) for part in _SPLIT.split(name) if part
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _position(concept: Concept, key: str) -> int | None:
|
||||||
|
match = _FIRST_NUMBER.match(concept.locators.get(key, ""))
|
||||||
|
return int(match.group(1)) if match else None
|
||||||
|
|
||||||
|
|
||||||
|
def _in_document_order(concepts: Sequence[Concept]) -> list[Concept]:
|
||||||
|
for key in _POSITION_KEYS:
|
||||||
|
positions = [_position(concept, key) for concept in concepts]
|
||||||
|
if all(position is not None for position in positions):
|
||||||
|
return [
|
||||||
|
concept
|
||||||
|
for _, _, concept in sorted(
|
||||||
|
(position, index, concept)
|
||||||
|
for index, (position, concept) in enumerate(
|
||||||
|
zip(positions, concepts, strict=True)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
]
|
||||||
|
return list(concepts)
|
||||||
|
|
||||||
|
|
||||||
|
def _titled(titles: Sequence[str]) -> str:
|
||||||
|
kept = titles[:TITLES_PER_LINE]
|
||||||
|
text = " · ".join(kept)
|
||||||
|
if len(titles) > len(kept):
|
||||||
|
text += f" · (+{len(titles) - len(kept)} more)"
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def build_map(concepts: Sequence[Concept]) -> dict[str, object]:
|
||||||
|
"""The map of `concepts`: one line per document, one per series."""
|
||||||
|
by_document: dict[str, list[Concept]] = {}
|
||||||
|
for concept in concepts:
|
||||||
|
by_document.setdefault(_stem(concept.source_file), []).append(concept)
|
||||||
|
by_template: dict[str, list[str]] = {}
|
||||||
|
for document in by_document:
|
||||||
|
by_template.setdefault(_DIGITS.sub("#", document), []).append(document)
|
||||||
|
|
||||||
|
entries: list[tuple[str, str]] = []
|
||||||
|
for template, documents in by_template.items():
|
||||||
|
if len(documents) >= SERIES_MIN:
|
||||||
|
ordered = sorted(documents, key=_natural)
|
||||||
|
counts: Counter[str] = Counter(
|
||||||
|
title
|
||||||
|
for document in documents
|
||||||
|
for title in dict.fromkeys(concept.title for concept in by_document[document])
|
||||||
|
if _LETTER.search(title)
|
||||||
|
)
|
||||||
|
titles = sorted(counts, key=lambda title: (-counts[title], title))
|
||||||
|
line = f"{template} ({len(documents)} documents: {ordered[0]} … {ordered[-1]})"
|
||||||
|
if titles:
|
||||||
|
line += f": {_titled(titles)}"
|
||||||
|
entries.append((template, line))
|
||||||
|
continue
|
||||||
|
for document in documents:
|
||||||
|
titles = list(
|
||||||
|
dict.fromkeys(
|
||||||
|
concept.title for concept in _in_document_order(by_document[document])
|
||||||
|
)
|
||||||
|
)
|
||||||
|
name = document or "(no source file)"
|
||||||
|
entries.append((document, f"{name}: {_titled(titles)}"))
|
||||||
|
lines = [line for _, line in sorted(entries, key=lambda entry: (_natural(entry[0]), entry[0]))]
|
||||||
|
kept: list[str] = []
|
||||||
|
spent = 0
|
||||||
|
for line in lines:
|
||||||
|
size = len(line.encode("utf-8"))
|
||||||
|
if spent + size > MAP_MAX_BYTES:
|
||||||
|
break
|
||||||
|
kept.append(line)
|
||||||
|
spent += size
|
||||||
|
return {
|
||||||
|
"documents": len(by_document),
|
||||||
|
"concepts": len(concepts),
|
||||||
|
"lines_count": len(lines),
|
||||||
|
"lines_truncated": len(lines) - len(kept),
|
||||||
|
"lines": kept,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def read_concepts(bundle_root: Path, *, profile: BundleProfile) -> list[Concept]:
|
||||||
|
"""Every concept of the bundle, as `okf consume` reads them -- parents
|
||||||
|
linked and a table fragment named by the heading above it."""
|
||||||
|
bundle_id = root_bundle_id_of(bundle_root, profile=profile)
|
||||||
|
return inherit_table_titles(
|
||||||
|
link_parents(
|
||||||
|
[
|
||||||
|
read_concept(
|
||||||
|
read_path_in_bundle(bundle_root, f"{concept_id}{profile.paths.concept_suffix}"),
|
||||||
|
bundle_root=bundle_root,
|
||||||
|
root_bundle_id=bundle_id,
|
||||||
|
)
|
||||||
|
for concept_id in enumerate_concepts(bundle_root, profile=profile)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def bundle_map(bundle_root: Path, *, profile: BundleProfile) -> dict[str, object]:
|
||||||
|
return build_map(read_concepts(bundle_root, profile=profile))
|
||||||
|
|
@ -84,22 +84,21 @@ CLIENT_TRUNCATION_BYTES = 2048
|
||||||
#: under the cap by a test, with a control so the assertion is a measurement.
|
#: under the cap by a test, with a control so the assertion is a measurement.
|
||||||
SERVER_INSTRUCTIONS = (
|
SERVER_INSTRUCTIONS = (
|
||||||
"Bundles are read-only and no call here runs a model.\n\n"
|
"Bundles are read-only and no call here runs a model.\n\n"
|
||||||
"HOW TO USE THIS SERVER. Read the bundle's map first with `okf_describe`, "
|
"HOW TO USE THIS SERVER. Read the bundle's `map` first with `okf_describe`: "
|
||||||
"then put the question into the bundle's own words -- its documents may be "
|
"one line per document with its section titles -- the bundle's own words. "
|
||||||
|
"Then write two to four sub-questions in THOSE words (its documents may be "
|
||||||
"written in another language than the question, and the ranking matches "
|
"written in another language than the question, and the ranking matches "
|
||||||
"words. Split a broad question into two to four sub-questions and call "
|
"words) and send them in ONE call: `okf_ask` with `questions`. Each excerpt "
|
||||||
"`okf_ask` once per sub-question. After each call read BOTH what came back "
|
"names the sub-questions it answered. Read BOTH what came back and what lay "
|
||||||
"and what lay just outside the cut: `withheld.nearest` names the "
|
"just outside the cut: `withheld.nearest` names the best-ranked concepts "
|
||||||
"best-ranked concepts that missed, with their titles. If one of them is "
|
"that missed, with their titles. If one of them is what you wanted, that is "
|
||||||
"what you wanted, that is a fact about the WORDS, not a closed door -- ask "
|
"a fact about the WORDS, not a closed door -- ask again with that concept's "
|
||||||
"again with that concept's own words, or fetch it by name with "
|
"own words, or fetch it by name with `okf_fetch`. Asking again is normal and "
|
||||||
"`okf_fetch`. Several calls are normal and expected; there is no limit and "
|
"expected. When `coverage.weak` is true, rephrase in the bundle's words, and "
|
||||||
"no penalty. When `coverage.weak` is true, rephrase in the bundle's words, "
|
"if it stays weak say the bundle does not cover the question. Then write ONE "
|
||||||
"and if it stays weak say the bundle does not cover the question. Then "
|
"answer, ordered by sub-question, in the questioner's language and in "
|
||||||
"write ONE answer, ordered by sub-question, in the "
|
"ordinary prose, citing the document and the section (and the bundle, when "
|
||||||
"questioner's language and in ordinary prose, citing the document and the "
|
"you read more than one). Say plainly what the bundles do not cover.\n\n"
|
||||||
"section (and the bundle, when you read more than one). Say plainly what "
|
|
||||||
"the bundles do not cover.\n\n"
|
|
||||||
"Every excerpt carries the bundle id and concept id a claim must be "
|
"Every excerpt carries the bundle id and concept id a claim must be "
|
||||||
"attributed to; the payload states what it withheld and why."
|
"attributed to; the payload states what it withheld and why."
|
||||||
)
|
)
|
||||||
|
|
@ -310,21 +309,11 @@ def card(bundle_root: Path, *, profile: BundleProfile, concept_sample: int = 50)
|
||||||
card would also be one more artefact that can be stale, which is the defect
|
card would also be one more artefact that can be stale, which is the defect
|
||||||
it was meant to remove.
|
it was meant to remove.
|
||||||
"""
|
"""
|
||||||
|
from . import bundlemap
|
||||||
from . import skill as okf_skill
|
from . import skill as okf_skill
|
||||||
|
|
||||||
bundle_id = okf_consume.root_bundle_id_of(bundle_root, profile=profile)
|
bundle_id = okf_consume.root_bundle_id_of(bundle_root, profile=profile)
|
||||||
concepts = okf_consume.link_parents(
|
concepts = bundlemap.read_concepts(bundle_root, profile=profile)
|
||||||
[
|
|
||||||
okf_consume.read_concept(
|
|
||||||
okf_consume.read_path_in_bundle(
|
|
||||||
bundle_root, f"{concept_id}{profile.paths.concept_suffix}"
|
|
||||||
),
|
|
||||||
bundle_root=bundle_root,
|
|
||||||
root_bundle_id=bundle_id,
|
|
||||||
)
|
|
||||||
for concept_id in okf_consume.enumerate_concepts(bundle_root, profile=profile)
|
|
||||||
]
|
|
||||||
)
|
|
||||||
counts = okf_skill.field_counts(concepts)
|
counts = okf_skill.field_counts(concepts)
|
||||||
return {
|
return {
|
||||||
"bundle_id": bundle_id,
|
"bundle_id": bundle_id,
|
||||||
|
|
@ -334,15 +323,17 @@ def card(bundle_root: Path, *, profile: BundleProfile, concept_sample: int = 50)
|
||||||
"concept_count": len(concepts),
|
"concept_count": len(concepts),
|
||||||
"concepts": [concept.concept_id for concept in concepts[:concept_sample]],
|
"concepts": [concept.concept_id for concept in concepts[:concept_sample]],
|
||||||
"concepts_truncated": len(concepts) > concept_sample,
|
"concepts_truncated": len(concepts) > concept_sample,
|
||||||
"source_files": sorted(
|
|
||||||
{concept.source_file for concept in concepts if concept.source_file}
|
|
||||||
),
|
|
||||||
"conditional_fields": {
|
"conditional_fields": {
|
||||||
field: counts.get(field, 0) for field in okf_skill.CONDITIONAL_FIELDS
|
field: counts.get(field, 0) for field in okf_skill.CONDITIONAL_FIELDS
|
||||||
},
|
},
|
||||||
"whole_bundle_bytes": okf_skill.whole_bundle_cost(concepts),
|
"whole_bundle_bytes": okf_skill.whole_bundle_cost(concepts),
|
||||||
"budget_unit": okf_consume.BUDGET_UNIT,
|
"budget_unit": okf_consume.BUDGET_UNIT,
|
||||||
"default_limit": okf_consume.DEFAULT_LIMIT,
|
"default_limit": okf_consume.DEFAULT_LIMIT,
|
||||||
|
# v1.1 C5: the bundle's own words, to write sub-questions in. It
|
||||||
|
# replaces the flat `source_files` list, which named every document a
|
||||||
|
# second time with no series collapsed -- a quarter of the reply on a
|
||||||
|
# large bundle, for names the map already carries.
|
||||||
|
"map": bundlemap.build_map(concepts),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -379,7 +370,7 @@ def tools(surface: Surface) -> tuple[Tool, ...]:
|
||||||
Tool(
|
Tool(
|
||||||
"okf_list",
|
"okf_list",
|
||||||
"Every OKF bundle this server can currently reach, with its content "
|
"Every OKF bundle this server can currently reach, with its content "
|
||||||
"identity and concept count. Re-read from disk on every call, so a "
|
"identity and concept count; `okf_describe` gives each one's map. Re-read from disk on every call, so a "
|
||||||
"bundle added, removed or rebuilt since the last call is reflected "
|
"bundle added, removed or rebuilt since the last call is reflected "
|
||||||
"without restarting anything. Exists because a client that cannot "
|
"without restarting anything. Exists because a client that cannot "
|
||||||
"discover bundles must be told their names out of band, which is the "
|
"discover bundles must be told their names out of band, which is the "
|
||||||
|
|
@ -391,9 +382,10 @@ def tools(surface: Surface) -> tuple[Tool, ...]:
|
||||||
Tool(
|
Tool(
|
||||||
"okf_describe",
|
"okf_describe",
|
||||||
"What one bundle is: its id, its content identity, how many concepts "
|
"What one bundle is: its id, its content identity, how many concepts "
|
||||||
"it holds, which source documents it was built from, and which "
|
"it holds, which conditionally-written fields are present on how "
|
||||||
"conditionally-written fields are present on how many concepts. "
|
"many concepts, and its `map` -- one line per source document with "
|
||||||
"Read it BEFORE asking, so the question can be put into the "
|
"its section titles, a series of like-named documents as one line. "
|
||||||
|
"Read it BEFORE asking, so the sub-questions can be put into the "
|
||||||
"bundle's own words. On a multi-bundle server, omitting `bundle_id` "
|
"bundle's own words. On a multi-bundle server, omitting `bundle_id` "
|
||||||
"describes every served bundle, as `okf_ask` does. "
|
"describes every served bundle, as `okf_ask` does. "
|
||||||
"Exists because an answer must be attributable -- a claim from a "
|
"Exists because an answer must be attributable -- a claim from a "
|
||||||
|
|
|
||||||
153
tests/test_bundle_map.py
Normal file
153
tests/test_bundle_map.py
Normal file
|
|
@ -0,0 +1,153 @@
|
||||||
|
"""The map a reader writes sub-questions from (v1.1 order C, C5).
|
||||||
|
|
||||||
|
A question is best put in the collection's OWN words, and the one place those
|
||||||
|
words are listed is the collection itself. `bundlemap.build_map` lists them:
|
||||||
|
one line per source document -- its name, then the titles of its concepts in
|
||||||
|
document order -- and a SERIES of documents whose names differ only in their
|
||||||
|
numbers (a changelog per release, a note per week) as ONE line with the span,
|
||||||
|
because four hundred lines saying the same thing crowd out the rest.
|
||||||
|
|
||||||
|
`okf card` and `okf_describe` carry it as `map`.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from llm_ingestion_okf import bundlemap, consume, mcp_server
|
||||||
|
|
||||||
|
TOOLS = Path(__file__).resolve().parent.parent / "tools"
|
||||||
|
if str(TOOLS) not in sys.path:
|
||||||
|
sys.path.insert(0, str(TOOLS))
|
||||||
|
|
||||||
|
import okf_retrieval_gate as retrieval # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def _doc(name: str, *titles: str) -> retrieval.DocumentSpec:
|
||||||
|
return retrieval.DocumentSpec(
|
||||||
|
name,
|
||||||
|
f"{name}.md",
|
||||||
|
tuple(
|
||||||
|
retrieval.ConceptSpec(slug=f"s{position}", title=title, body=f"About {title}.")
|
||||||
|
for position, title in enumerate(titles, start=1)
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
RELEASES = tuple(_doc(f"changes-1-{minor}", f"1.{minor}") for minor in range(2, 14))
|
||||||
|
WEEKLY = tuple(
|
||||||
|
_doc(f"notes-2026-w{week}", "Highlights", f"Week {week} fixes") for week in (1, 2, 3, 4, 5)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def bundle(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||||
|
spec = retrieval.BundleSpec(
|
||||||
|
"map-synthetic",
|
||||||
|
(
|
||||||
|
_doc("guide-setup", "Setup", "Install the tool", "Configure a project"),
|
||||||
|
_doc("guide-hooks", "Hooks", "Hook events", "Hook events"),
|
||||||
|
*RELEASES,
|
||||||
|
*WEEKLY,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return retrieval.build_bundle(tmp_path_factory.mktemp("map") / "bundle", spec)
|
||||||
|
|
||||||
|
|
||||||
|
def _map(bundle: Path) -> dict[str, object]:
|
||||||
|
return bundlemap.bundle_map(bundle, profile=consume.DEFAULT_PROFILE)
|
||||||
|
|
||||||
|
|
||||||
|
def test_one_line_per_document_with_its_own_titles_in_order(bundle: Path) -> None:
|
||||||
|
lines = _map(bundle)["lines"]
|
||||||
|
assert isinstance(lines, list)
|
||||||
|
assert "guide-setup: Setup · Install the tool · Configure a project" in lines
|
||||||
|
# A title the document repeats is listed once.
|
||||||
|
assert "guide-hooks: Hooks · Hook events" in lines
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_series_is_one_line_with_its_span(bundle: Path) -> None:
|
||||||
|
lines = _map(bundle)["lines"]
|
||||||
|
assert isinstance(lines, list)
|
||||||
|
series = [line for line in lines if line.startswith("changes-#-#")]
|
||||||
|
assert series == ["changes-#-# (12 documents: changes-1-2 … changes-1-13)"]
|
||||||
|
assert not any(line.startswith("changes-1-") for line in lines)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_series_keeps_the_titles_that_are_words(bundle: Path) -> None:
|
||||||
|
lines = _map(bundle)["lines"]
|
||||||
|
assert isinstance(lines, list)
|
||||||
|
(line,) = [line for line in lines if line.startswith("notes-#-w#")]
|
||||||
|
assert line.startswith("notes-#-w# (5 documents: notes-2026-w1 … notes-2026-w5): Highlights")
|
||||||
|
assert "Week 1 fixes" in line
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_map_states_its_denominators(bundle: Path) -> None:
|
||||||
|
built = _map(bundle)
|
||||||
|
assert built["documents"] == 2 + 12 + 5
|
||||||
|
assert built["concepts"] == 3 + 3 + 12 + 10
|
||||||
|
assert built["lines_count"] == 4
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_long_document_is_cut_and_says_so(tmp_path: Path) -> None:
|
||||||
|
titles = [f"Section {n}" for n in range(bundlemap.TITLES_PER_LINE + 5)]
|
||||||
|
spec = retrieval.BundleSpec("long", (_doc("big", *titles),))
|
||||||
|
bundle = retrieval.build_bundle(tmp_path / "bundle", spec)
|
||||||
|
(line,) = _map(bundle)["lines"] # type: ignore[misc]
|
||||||
|
assert line.endswith("· (+5 more)")
|
||||||
|
assert line.count(" · ") == bundlemap.TITLES_PER_LINE
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_map_is_deterministic(bundle: Path) -> None:
|
||||||
|
assert json.dumps(_map(bundle)) == json.dumps(_map(bundle))
|
||||||
|
|
||||||
|
|
||||||
|
def test_describe_and_the_card_carry_the_map(bundle: Path) -> None:
|
||||||
|
surface = mcp_server.build_surface(bundle=bundle, roots=())
|
||||||
|
described = mcp_server.call_describe(surface, {})
|
||||||
|
assert described["map"] == _map(bundle)
|
||||||
|
assert mcp_server.card(bundle, profile=consume.DEFAULT_PROFILE)["map"] == _map(bundle)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_card_names_documents_through_the_map_alone(bundle: Path) -> None:
|
||||||
|
"""`source_files` listed every document a second time, one name per line
|
||||||
|
and no series collapsed; the map names every document or series already."""
|
||||||
|
assert "source_files" not in mcp_server.card(bundle, profile=consume.DEFAULT_PROFILE)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_working_method_is_map_first_then_one_call() -> None:
|
||||||
|
from llm_ingestion_okf import skill as okf_skill
|
||||||
|
|
||||||
|
generic = okf_skill.render_generic()
|
||||||
|
assert "`map`" in generic
|
||||||
|
assert "ONE run" in generic
|
||||||
|
# The example command itself carries more than one sub-question.
|
||||||
|
command = next(
|
||||||
|
line
|
||||||
|
for line in generic.splitlines()
|
||||||
|
if line.startswith("okf consume ") and "sub-question" in line
|
||||||
|
)
|
||||||
|
assert command.count("--question ") >= 2
|
||||||
|
instructions = mcp_server.SERVER_INSTRUCTIONS
|
||||||
|
assert "map" in instructions
|
||||||
|
assert "`questions`" in instructions
|
||||||
|
assert "ONE call" in instructions
|
||||||
|
assert len(instructions.encode("utf-8")) <= 2048
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_map_has_a_ceiling_and_says_what_it_left_out(
|
||||||
|
bundle: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
whole = _map(bundle)
|
||||||
|
assert whole["lines_truncated"] == 0
|
||||||
|
lines = whole["lines"]
|
||||||
|
assert isinstance(lines, list)
|
||||||
|
monkeypatch.setattr(bundlemap, "MAP_MAX_BYTES", len(lines[0].encode("utf-8")) + 1)
|
||||||
|
cut = _map(bundle)
|
||||||
|
assert cut["lines"] == lines[:1]
|
||||||
|
assert cut["lines_truncated"] == len(lines) - 1
|
||||||
|
assert cut["lines_count"] == len(lines)
|
||||||
|
|
@ -247,3 +247,39 @@ def test_okf_ask_refuses_both_forms_at_once(bundle: Path) -> None:
|
||||||
with pytest.raises(mcp_server.ToolError) as raised:
|
with pytest.raises(mcp_server.ToolError) as raised:
|
||||||
mcp_server.call_ask(surface, {"question": HEATING, "questions": [ENGINE]})
|
mcp_server.call_ask(surface, {"question": HEATING, "questions": [ENGINE]})
|
||||||
assert raised.value.code == "question_ambiguous"
|
assert raised.value.code == "question_ambiguous"
|
||||||
|
|
||||||
|
|
||||||
|
def test_four_subquestions_over_large_concepts_stay_under_a_tool_reply(tmp_path: Path) -> None:
|
||||||
|
"""The worst case the passage cut exists for: every delivered concept is far
|
||||||
|
over `PASSAGE_CHARS`. A client keeps a tool reply of 25 000 tokens; at a
|
||||||
|
pessimistic two bytes a token that is 50 000 bytes of text."""
|
||||||
|
words = ("stove", "engine", "apples", "roof")
|
||||||
|
spec = retrieval.BundleSpec(
|
||||||
|
"large-concepts",
|
||||||
|
tuple(
|
||||||
|
retrieval.DocumentSpec(
|
||||||
|
f"doc-{word}",
|
||||||
|
f"doc-{word}.md",
|
||||||
|
tuple(
|
||||||
|
retrieval.ConceptSpec(
|
||||||
|
slug=f"part-{n}",
|
||||||
|
title=f"{word.title()} part {n}",
|
||||||
|
body=f"The {word} is described here in detail. ",
|
||||||
|
repeat=400,
|
||||||
|
)
|
||||||
|
for n in range(3)
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for word in words
|
||||||
|
),
|
||||||
|
)
|
||||||
|
bundle = retrieval.build_bundle(tmp_path / "bundle", spec)
|
||||||
|
surface = mcp_server.build_surface(bundle=bundle, roots=())
|
||||||
|
result = mcp_server.call_ask(
|
||||||
|
surface, {"questions": [f"How is the {w} described?" for w in words]}
|
||||||
|
)
|
||||||
|
payload = result["answers"][0]["payload"]
|
||||||
|
assert len(payload["excerpts"]) == consume.DEFAULT_K
|
||||||
|
assert all(len(excerpt["text"]) > consume.PASSAGE_CHARS // 2 for excerpt in payload["excerpts"])
|
||||||
|
text = mcp_server._tool_result(result)["content"][0]["text"]
|
||||||
|
assert len(text.encode("utf-8")) < 50_000
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue