open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(consume): the withheld set is counts plus names, not one entry per concept

Browse source 
Measured 2026-09-20 on a 2313-concept bundle of one project's own
documentation: `withheld` held 2 305 entries = 186 440 B of compact JSON =
**65.5 % of the 284 850-byte payload**, and not one of those bytes counted
against the budget the same payload reports (`spent` was 45 192). A reader was
handed 239 658 bytes the budget line did not know about, to learn 2 305 concept
ids with nothing beside them -- the title being exactly what `--withheld-titles`
existed to buy, and which was off because buying it for 2 305 entries cost
another 37.9 %.

`withheld` is now a mapping: `total` (equal to `denominators.withheld`, so
SS 5.2's identity is unmoved and closes on the NUMBERS), `by_rule` (the same
total decomposed over the closed rule set, so "what kind of drop" is answerable
without the list), `nearest` (the best-ranked drops BY NAME, with title and
source document, so a reader who sees a near miss can ask for it) and
`complete`. The near misses are read off the ranking, not off `cut`'s output:
`cut` sorts by id so the partition is comparable, and that order says nothing
about which concept a reader might want next.

Same question, same bundle, after: **52 421 bytes, 18.4 % of the old file**.
The whole list stays reachable behind `--withheld-full`, and the two
instruments that classify EVERY miss by its rule -- the retrieval gate and
`okf_consume_measure` -- now ask for it explicitly and assert `complete`
rather than assuming it. `--withheld-nearest N` sets the cap (default 20,
which is `k` plus the next twelve). `--withheld-titles` is retired: a flag
whose only remaining effect would be to STRIP the title from a list the caller
asked for in full names no decision worth two shapes for one list.

`CONTRACT_REVISION` moves to `okf-consumption/2`, because a consumer indexing
the old key as a list would otherwise break silently. Three checker rules move
with it, and one of them is the interesting case: `parent_unfollowable` used
`excerpts` + `withheld` as the bundle's own denominator, which a truncated
block is not -- so that clause now runs only where the payload SAYS it is
complete, stated in SS 8.6 rather than left as a silence, with the other two
clauses (shape, self-reference) running either way. `Report` carries both
denominators, because a report claiming it examined 2 305 entries it never saw
is the same defect one level up.

The generated skill's "breaking point" section goes with it: it extrapolated a
concept count from the cost of ONE withheld entry, and there is no such slope
any more. It now states what this bundle's bookkeeping cost and that the block
is bounded by the cap rather than by the bundle -- an extrapolation from a
slope the code no longer has would be a measurement of the previous revision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-20 23:32:22 +02:00
commit e3169ec50c
15 changed files with 708 additions and 251 deletions
Download patch file Download diff file Expand all files Collapse all files

50
tests/test_okf_skill.py
View file

@ -366,42 +366,46 @@ def test_a_project_skill_still_passes_the_contract_checker(tmp_path: Path) -> No
@pytest.mark.parametrize("bundle", BUNDLES, ids=lambda path: path.name)
def test_the_breaking_point_is_a_measurement_or_it_is_withheld(
def test_the_payload_cost_section_is_measured_on_the_bundle_it_names(
bundle: Path, tmp_path: Path
) -> None:
"""`0 concepts` was a division that never happened, printed as a number.
"""The section that used to extrapolate a breaking point now states cost.
The figure is EXTRAPOLATED from what one `withheld` entry costs, so a
generation run that withheld nothing has no slope to extrapolate from:
`per_withheld` was `0.0`, the guard returned the literal `0`, and the
document told its reader the bundle's bookkeeping fills a 120000-byte
budget at zero concepts -- before the bundle holds anything at all.
`0 concepts` was a division that never happened, printed as a number: the
figure was extrapolated from what ONE `withheld` entry cost, and a run
that withheld nothing had no slope. Since `okf-consumption/2` there is no
slope at all -- the block is counts plus a capped sample -- so the section
states what this bundle's bookkeeping cost and no extrapolated count, and
a generator that printed one would be describing the previous revision.
Driven from both sides so a generator that simply stopped stating the
figure would fail: the bundle that withholds nothing must say it could not
measure it, and a bundle that withholds something must still print a
positive count.
Driven from both sides: the numbers must be the payload's own, and the
retired figure must not come back.
"""
written = _generate(bundle, tmp_path / "out")
text = written.read_text(encoding="utf-8")
payload = json.loads((tmp_path / "out" / "references" / "example-payload.json").read_text())
assert payload["withheld"], "the known-positive arm withheld nothing to extrapolate from"
assert "**0 concepts**" not in text
stated = re.search(r"At roughly\s+\*\*(\d+) concepts\*\*", text)
assert stated is not None, "a bundle that withheld something states no figure"
block = payload["withheld"]
assert block["total"], "the known-positive arm withheld nothing to report"
assert "At roughly" not in text and "**0 concepts**" not in text
stated = re.search(r"\*\*(\d+) bytes\*\* — (\d+) withheld of (\d+) concepts", text)
assert stated is not None, "the section states no measured cost"
assert int(stated.group(2)) == block["total"]
assert int(stated.group(1)) > 0
named = re.search(r"of which \*\*(\d+)\*\* are named", text)
assert named is not None and int(named.group(1)) == len(block["nearest"])
def test_a_generation_that_withheld_nothing_says_so_instead_of_printing_zero(
def test_a_generation_that_withheld_nothing_states_a_zero_it_measured(
tmp_path: Path,
) -> None:
"""The arm the SHIPPED skill is on, and the one that was wrong.
"""The arm the SHIPPED skill is on.
`okf skill --example-question "Hva sier veiledningen om krav?"` delivers
all three concepts of the golden bundle, so `withheld` is empty and there
is no per-entry cost. The question is part of what the shipped file is
(`skills/okf-consume/references/README.md`), which is why the defect was
in the repository rather than only reachable in theory.
all three concepts of the golden bundle, so nothing is withheld. Under the
flat list that left no per-entry cost and the guard printed `0 concepts`;
under counts-plus-names the zero is a count the run actually made, so it
is stated rather than withheld -- and the retired figure must still be
absent.
"""
written = okf_skill.generate(
GOLDEN,
@ -411,7 +415,7 @@ def test_a_generation_that_withheld_nothing_says_so_instead_of_printing_zero(
)
text = written.read_text(encoding="utf-8")
payload = json.loads((tmp_path / "out" / "references" / "example-payload.json").read_text())
assert payload["withheld"] == [], "the premise of this arm no longer holds"
assert payload["withheld"]["total"] == 0, "the premise of this arm no longer holds"
assert "**0 concepts**" not in text
assert "breaking point could not be measured" in text
assert "At roughly" not in text
assert "0 withheld of 3 concepts" in text