open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(consume): the withheld set is counts plus names, not one entry per concept

Browse source 
Measured 2026-09-20 on a 2313-concept bundle of one project's own
documentation: `withheld` held 2 305 entries = 186 440 B of compact JSON =
**65.5 % of the 284 850-byte payload**, and not one of those bytes counted
against the budget the same payload reports (`spent` was 45 192). A reader was
handed 239 658 bytes the budget line did not know about, to learn 2 305 concept
ids with nothing beside them -- the title being exactly what `--withheld-titles`
existed to buy, and which was off because buying it for 2 305 entries cost
another 37.9 %.

`withheld` is now a mapping: `total` (equal to `denominators.withheld`, so
SS 5.2's identity is unmoved and closes on the NUMBERS), `by_rule` (the same
total decomposed over the closed rule set, so "what kind of drop" is answerable
without the list), `nearest` (the best-ranked drops BY NAME, with title and
source document, so a reader who sees a near miss can ask for it) and
`complete`. The near misses are read off the ranking, not off `cut`'s output:
`cut` sorts by id so the partition is comparable, and that order says nothing
about which concept a reader might want next.

Same question, same bundle, after: **52 421 bytes, 18.4 % of the old file**.
The whole list stays reachable behind `--withheld-full`, and the two
instruments that classify EVERY miss by its rule -- the retrieval gate and
`okf_consume_measure` -- now ask for it explicitly and assert `complete`
rather than assuming it. `--withheld-nearest N` sets the cap (default 20,
which is `k` plus the next twelve). `--withheld-titles` is retired: a flag
whose only remaining effect would be to STRIP the title from a list the caller
asked for in full names no decision worth two shapes for one list.

`CONTRACT_REVISION` moves to `okf-consumption/2`, because a consumer indexing
the old key as a list would otherwise break silently. Three checker rules move
with it, and one of them is the interesting case: `parent_unfollowable` used
`excerpts` + `withheld` as the bundle's own denominator, which a truncated
block is not -- so that clause now runs only where the payload SAYS it is
complete, stated in SS 8.6 rather than left as a silence, with the other two
clauses (shape, self-reference) running either way. `Report` carries both
denominators, because a report claiming it examined 2 305 entries it never saw
is the same defect one level up.

The generated skill's "breaking point" section goes with it: it extrapolated a
concept count from the cost of ONE withheld entry, and there is no such slope
any more. It now states what this bundle's bookkeeping cost and that the block
is bounded by the cap rather than by the bundle -- an extrapolation from a
slope the code no longer has would be a measurement of the previous revision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-20 23:32:22 +02:00
commit e3169ec50c
15 changed files with 708 additions and 251 deletions
Download patch file Download diff file Expand all files Collapse all files

3
tests/test_contract_check.py
View file

@ -96,6 +96,7 @@ def test_the_report_carries_its_denominators() -> None:
assert report.rules_evaluated > 0
assert report.excerpts_examined == 3
assert report.withheld_examined == 2
assert report.withheld_total == 2
rendered = report.render()
assert str(report.rules_evaluated) in rendered
assert "3" in rendered and "2" in rendered
@ -150,7 +151,7 @@ def test_a_bundle_without_a_ref_is_named() -> None:
def test_a_withheld_concept_without_a_rule_is_an_undeclared_cut() -> None:
payload = load_example()
del payload["withheld"][0]["rule"]
del payload["withheld"]["nearest"][0]["rule"]
assert "cut_undeclared" in codes(instantiated(), payload)

319
tests/test_okf_consume.py
View file

@ -511,6 +511,22 @@ def test_document_scores_are_identical_across_two_calls() -> None:
# --- Step 6: stage-two concept ranking, fused by RRF --------------------------
def _withheld_entries(payload: Mapping[str, Any]) -> list[Any]:
"""Every withheld entry of a payload whose block says it names them all.
Since `okf-consumption/2` `withheld` is counts plus a CAPPED sample, so a
test reading entries has to say which it is reading. `complete` is
asserted rather than assumed: over a truncated block these tests would be
measuring the first twenty of something and reporting it as the set.
"""
block = payload["withheld"]
assert isinstance(block, Mapping)
assert block["complete"] is True, "the block is a sample here, not the set"
nearest = block["nearest"]
assert isinstance(nearest, list)
return nearest
def _fixture_concepts() -> list[okf_consume.Concept]:
return [
okf_consume.read_concept(
@ -638,51 +654,6 @@ def test_a_concept_whose_verified_cannot_be_read_is_withheld_by_name() -> None:
assert dict(withheld)["dyp/nivaa/blokkform-verifisert"] == "verified_unreadable"
def test_a_withheld_entry_names_what_was_dropped_under_the_flag() -> None:
# A reader who is told 262 concepts were withheld, by id and rule alone,
# cannot tell WHAT was withheld without reading the bundle -- which SS 2.2
# forbids. The title closes that, and it is emitted only where the concept
# carries one.
payload = okf_consume.build_payload(
FIXTURE, question="Hvordan skal prisene fylles ut?", withheld_titles=True
)
entries = payload["withheld"]
assert isinstance(entries, list) and entries
titled = [entry for entry in entries if "title" in entry]
assert titled, "no withheld entry carried a title, so the rule measures nothing"
concepts = {concept.concept_id: concept for concept in _fixture_concepts()}
for entry in entries:
concept = concepts[str(entry["concept_id"])]
if concept.title:
assert entry["title"] == concept.title
else:
assert "title" not in entry
def test_no_withheld_entry_names_anything_without_the_flag() -> None:
# The default is what every consumer already runs, and this is the
# measurement that keeps it theirs: a title on every withheld entry grew a
# 270-concept payload by 37.9 % and pushed a 629-concept bundle's
# bookkeeping past the budget limit itself.
payload = okf_consume.build_payload(FIXTURE, question="Hvordan skal prisene fylles ut?")
entries = payload["withheld"]
assert isinstance(entries, list) and entries
assert all(set(entry) == {"concept_id", "rule"} for entry in entries)
def test_the_withheld_title_flag_costs_bytes_and_the_default_pays_none() -> None:
question = "Hvordan skal prisene fylles ut?"
off = okf_consume.serialise(okf_consume.build_payload(FIXTURE, question=question))
explicit_off = okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, withheld_titles=False)
)
on = okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, withheld_titles=True)
)
assert off == explicit_off
assert len(on.encode("utf-8")) > len(off.encode("utf-8"))
def test_delivered_and_withheld_partition_the_considered_set() -> None:
delivered, withheld, considered = _cut_fixture()
delivered_ids = {excerpt["concept_id"] for excerpt in delivered}
@ -757,7 +728,7 @@ def test_the_payload_passes_the_checker_against_a_skill_for_its_own_bundle() ->
def test_the_payload_carries_every_section_eight_member() -> None:
payload = _payload()
assert payload["contract"] == "okf-consumption/1"
assert payload["contract"] == "okf-consumption/2"
assert set(payload) >= {
"contract",
"bundle",
@ -801,8 +772,9 @@ def test_spent_moves_when_an_excerpt_moves_and_holds_when_withheld_grows() -> No
def test_the_counts_and_the_lists_are_two_statements_of_one_fact() -> None:
payload = _payload()
counts, excerpts, withheld = payload["denominators"], payload["excerpts"], payload["withheld"]
assert isinstance(counts, dict) and isinstance(excerpts, list) and isinstance(withheld, list)
counts, excerpts = payload["denominators"], payload["excerpts"]
withheld = _withheld_entries(payload)
assert isinstance(counts, dict) and isinstance(excerpts, list)
assert counts["delivered"] == len(excerpts)
assert counts["withheld"] == len(withheld)
assert counts["considered"] == counts["delivered"] + counts["withheld"]
@ -858,14 +830,17 @@ def test_a_question_with_no_answer_returns_a_measured_empty_set_not_a_guess() ->
# still closes and the skill can say "measured, nothing cleared the bar"
# rather than "nothing was found".
payload = _payload(question="Hva er reglene for sveising av titan i vakuum?")
counts, excerpts, withheld = payload["denominators"], payload["excerpts"], payload["withheld"]
assert isinstance(counts, dict) and isinstance(excerpts, list) and isinstance(withheld, list)
counts, excerpts = payload["denominators"], payload["excerpts"]
assert isinstance(counts, dict) and isinstance(excerpts, list)
assert excerpts == []
assert counts["delivered"] == 0
assert (
counts["withheld"] == counts["considered"] == len(okf_consume.enumerate_concepts(FIXTURE))
)
assert {entry["rule"] for entry in withheld} == {"no_lexical_match", "verdict_layer_excluded"}
assert {entry["rule"] for entry in _withheld_entries(payload)} == {
"no_lexical_match",
"verdict_layer_excluded",
}
# And the control: the SAME payload builder returns a non-empty set for a
# question this bundle does answer, so the zero is a measurement.
answered = _payload()
@ -930,7 +905,16 @@ def test_spent_is_the_delivered_set_where_the_whole_payload_reading_would_refuse
# from the plan: the plan predicted 101 576 B for this excerpt and 188 758 B
# for the payload, both taken before per-line trailing-whitespace stripping
# landed. What this build actually produces is recorded instead.
payload = okf_consume.build_payload(K2_BUNDLE, question="Hvordan skal prisene fylles ut?")
# Run with `--withheld-full`, because the DEFAULT no longer reaches this
# case and that is the point of `okf-consumption/2`: the whole payload of
# this bundle came to 117 561 B under the default -- inside the limit --
# where under the flat list it did not. The distinction between the two
# readings of SS 7.2 is still real and still reachable, so it is measured
# where it is reachable rather than deleted with the shape that produced
# it. The control below fires if that stops being true.
payload = okf_consume.build_payload(
K2_BUNDLE, question="Hvordan skal prisene fylles ut?", withheld_full=True
)
budget, excerpts = payload["budget"], payload["excerpts"]
assert isinstance(budget, dict) and isinstance(excerpts, list)
whole_payload = len(okf_consume.serialise(payload).encode("utf-8"))
@ -940,6 +924,11 @@ def test_spent_is_the_delivered_set_where_the_whole_payload_reading_would_refuse
)
assert int(budget["spent"]) <= int(budget["limit"])
# And the default is now the other side of that line, on the same bundle
# and the same question: the bookkeeping stopped dwarfing the content.
default = okf_consume.build_payload(K2_BUNDLE, question="Hvordan skal prisene fylles ut?")
assert len(okf_consume.serialise(default).encode("utf-8")) < whole_payload
#: The gold set is LOCAL-ONLY: it names corpus documents, which never reach a
#: tracked file here. The test reads it rather than restating it, so this file
@ -1481,8 +1470,9 @@ def test_a_cost_question_reaches_no_price_concept_without_the_flag() -> None:
# The known-negative this whole step is measured against. Without it, the
# flag's effect below would have no denominator.
payload = _payload(question="Hvor kan vi kutte kostnader?")
counts, withheld = payload["denominators"], payload["withheld"]
assert isinstance(counts, dict) and isinstance(withheld, list)
counts = payload["denominators"]
withheld = _withheld_entries(payload)
assert isinstance(counts, dict)
assert counts["delivered"] == 0
# By the two fields this test is about, not by the whole entry: the entry
# also carries the concept's title, and pinning the exact dict here would
@ -1683,10 +1673,9 @@ def test_the_knapsack_evicts_the_top_ranked_candidate_that_costs_half_the_budget
}
assert EVICTION_TOP not in weights
assert (
dict(
(entry["concept_id"], entry["rule"])
for entry in payload["withheld"] # type: ignore[union-attr]
)[EVICTION_TOP]
dict((entry["concept_id"], entry["rule"]) for entry in _withheld_entries(payload))[
EVICTION_TOP
]
== "over_budget_after_knapsack"
)
# The shape itself, stated as numbers rather than assumed: the top candidate
@ -1714,10 +1703,7 @@ def test_reserving_the_top_rank_delivers_the_candidate_the_knapsack_evicted(
assert isinstance(excerpts, list)
assert excerpts[0]["concept_id"] == EVICTION_TOP
assert excerpts[0]["rank"] == 1
assert EVICTION_TOP not in {
entry["concept_id"]
for entry in payload["withheld"] # type: ignore[union-attr]
}
assert EVICTION_TOP not in {entry["concept_id"] for entry in _withheld_entries(payload)}
def test_the_reservation_is_off_by_default_and_the_default_payload_is_byte_identical(
@ -1747,10 +1733,7 @@ def test_a_top_candidate_that_alone_exceeds_the_budget_is_still_refused_by_name(
root = _eviction_bundle(tmp_path / "bundle")
limit = _eviction_bundle_top_weight(root) - 1
payload = _eviction_payload(root, limit=limit, reserve_top_rank=True)
rules = dict(
(entry["concept_id"], entry["rule"])
for entry in payload["withheld"] # type: ignore[union-attr]
)
rules = dict((entry["concept_id"], entry["rule"]) for entry in _withheld_entries(payload))
assert rules[EVICTION_TOP] == "over_budget_alone"
spent = payload["budget"]["spent"] # type: ignore[index]
assert isinstance(spent, int)
@ -1778,7 +1761,7 @@ def test_the_reservation_displaces_lower_ranked_excerpts_under_the_rule_that_exi
assert len(with_reservation["excerpts"]) < len(without["excerpts"]) # type: ignore[arg-type]
displaced = {
entry["concept_id"]
for entry in with_reservation["withheld"] # type: ignore[union-attr]
for entry in _withheld_entries(with_reservation)
if entry["rule"] == "over_budget_after_knapsack"
}
delivered_before = {
@ -1786,10 +1769,9 @@ def test_the_reservation_displaces_lower_ranked_excerpts_under_the_rule_that_exi
for excerpt in without["excerpts"] # type: ignore[union-attr]
}
assert displaced & delivered_before
assert {
entry["rule"]
for entry in with_reservation["withheld"] # type: ignore[union-attr]
} <= set(okf_consume.WITHHOLDING_RULES)
assert {entry["rule"] for entry in _withheld_entries(with_reservation)} <= set(
okf_consume.WITHHOLDING_RULES
)
def test_the_payload_declares_which_concept_the_reservation_took_and_what_it_cost(
@ -2718,3 +2700,198 @@ def test_a_question_the_bundle_answers_leaves_the_unanswered_lists_short() -> No
]
assert in_payload, "the delivered excerpts answer nothing of a question they were cut for"
assert len(coverage["unanswered_in_bundle"]) < len(terms) # type: ignore[arg-type]
# --- A2: the withheld list is bookkeeping, not the payload --------------------
def _mapping_of(value: object) -> Mapping[str, Any]:
assert isinstance(value, Mapping)
return value
def _sequence_of(value: object) -> list[Any]:
assert isinstance(value, list)
return value
def _ranked_fixture(question: str) -> list[tuple[okf_consume.Concept, float, int]]:
"""The same ranking `build_payload` runs on, with the same defaults.
Rebuilt here rather than exported: a helper that took the payload's own
order would be comparing the order against itself.
"""
concepts = _fixture_concepts()
texts = okf_consume.searchable_text(concepts, link_in_signal=okf_consume.DEFAULT_LINK_IN_SIGNAL)
stems = (
frozenset(token for text in texts for token in okf_consume.normalise(text))
if okf_consume.DEFAULT_STEM_PREFIX
else None
)
return list(
okf_consume.concept_scores(
concepts,
question,
okf_consume.document_scores(FIXTURE, question, stems=stems),
tie_shared_rank=okf_consume.DEFAULT_TIE_SHARED_RANK,
title_covered=okf_consume.DEFAULT_TITLE_COVERED,
stems=stems,
link_in_signal=okf_consume.DEFAULT_LINK_IN_SIGNAL,
)
)
def test_the_default_payload_reports_the_withheld_as_counts_and_near_misses() -> None:
"""Measured 2026-09-20 on a 2313-concept bundle: `withheld` held 2 305
entries = 186 440 B = 65.5 % of the written file, and NONE of it counted
against the budget the payload reports. A reader was handed 239 658 bytes
the budget line did not know about, to learn 2 305 ids they could do
nothing with.
The replacement states the same facts in the units a reader can act on:
the total, the count per rule, and the near misses BY NAME.
"""
payload = okf_consume.build_payload(FIXTURE, question="Hvordan skal prisene fylles ut?")
withheld = payload["withheld"]
assert isinstance(withheld, Mapping)
assert set(withheld) == {"total", "by_rule", "nearest", "complete"}
counts = payload["denominators"]
assert isinstance(counts, Mapping)
assert withheld["total"] == counts["withheld"]
by_rule = withheld["by_rule"]
assert isinstance(by_rule, Mapping)
assert sum(by_rule.values()) == withheld["total"]
assert set(by_rule) <= set(okf_consume.WITHHOLDING_RULES)
nearest = withheld["nearest"]
assert isinstance(nearest, list)
assert len(nearest) <= okf_consume.WITHHELD_NEAREST_DEFAULT
def test_a_near_miss_is_named_so_a_reader_can_ask_for_it() -> None:
"""Every entry carries the title and the document, not just the id.
`--withheld-titles` bought the title for all 2 305 entries and was off by
measurement (+37.9 % on N500). At twenty entries the same field costs
nothing and is the one thing that makes the list usable: a reader who sees
a near miss by name can ask for it.
"""
payload = okf_consume.build_payload(FIXTURE, question="Hvordan skal prisene fylles ut?")
withheld = payload["withheld"]
assert isinstance(withheld, Mapping)
nearest = withheld["nearest"]
assert isinstance(nearest, list) and nearest
concepts = {concept.concept_id: concept for concept in _fixture_concepts()}
named = 0
for entry in nearest:
assert isinstance(entry, Mapping)
assert set(entry) <= {"concept_id", "rule", "title", "source_file"}
assert entry["concept_id"] and entry["rule"]
concept = concepts[str(entry["concept_id"])]
if concept.title:
assert entry["title"] == concept.title
named += 1
else:
assert "title" not in entry
assert named, "no near miss carried a title, so the rule measures nothing"
def test_the_near_misses_are_the_best_ranked_of_the_withheld() -> None:
"""Rank order, not id order.
`cut` sorts its drops by id so the partition is comparable between runs;
that order is alphabetical and says nothing about which concept a reader
might want next. Driven directly rather than through `build_payload`,
because the synthetic fixture's six drops come back in the SAME order
either way -- a bundle where the two orders agree cannot tell them apart,
which the control below asserts before the rest is believed.
"""
concepts = _fixture_concepts()
assert len(concepts) >= 4
# Rank order deliberately the reverse of id order.
ranked = [(concept, 1.0, 1) for concept in sorted(concepts, key=lambda c: c.concept_id)[::-1]]
withheld = sorted((concept.concept_id, "below_k") for concept, _, _ in ranked)
rank_order = [concept.concept_id for concept, _, _ in ranked]
assert rank_order != [concept_id for concept_id, _ in withheld], (
"the two orders agree here, so the assertion below would measure nothing"
)
block = okf_consume.withheld_block(
withheld,
ranked,
titles_by_id={c.concept_id: c.title for c in concepts},
sources_by_id={c.concept_id: c.source_file for c in concepts},
nearest=3,
)
assert [str(entry["concept_id"]) for entry in _sequence_of(block["nearest"])] == rank_order[:3]
assert block["total"] == len(withheld)
assert block["complete"] is False
def test_every_withheld_rule_is_counted_even_when_it_is_not_named() -> None:
"""`by_rule` is what makes the truncation honest: the near misses are a
sample, the counts are the whole set."""
payload = okf_consume.build_payload(
FIXTURE, question="Hvordan skal prisene fylles ut?", withheld_nearest=0
)
block = _mapping_of(payload["withheld"])
by_rule = _mapping_of(block["by_rule"])
assert block["nearest"] == []
assert (
sum(by_rule.values()) == block["total"] == _mapping_of(payload["denominators"])["withheld"]
)
assert len(by_rule) > 1, "one rule only, so the decomposition measures nothing"
def test_the_whole_list_is_reachable_behind_one_explicit_switch() -> None:
"""The bookkeeping is not deleted; it is moved off the default path."""
question = "Hvordan skal prisene fylles ut?"
full = okf_consume.build_payload(FIXTURE, question=question, withheld_full=True)
block = full["withheld"]
assert isinstance(block, Mapping)
assert block["complete"] is True
nearest = block["nearest"]
assert isinstance(nearest, list)
assert len(nearest) == block["total"]
# The bytes the switch buys, measured against the same payload with no
# near misses named at all -- this fixture holds 7 concepts, so its
# DEFAULT is already complete and cannot show the difference.
none_named = okf_consume.build_payload(FIXTURE, question=question, withheld_nearest=0)
assert _mapping_of(none_named["withheld"])["complete"] is False
assert len(okf_consume.serialise(full)) > len(okf_consume.serialise(none_named))
def test_the_near_miss_cap_is_a_number_the_caller_may_set() -> None:
question = "Hvordan skal prisene fylles ut?"
payload = okf_consume.build_payload(FIXTURE, question=question, withheld_nearest=1)
block = payload["withheld"]
assert isinstance(block, Mapping)
nearest = block["nearest"]
assert isinstance(nearest, list)
assert len(nearest) == min(1, int(str(block["total"])))
none_at_all = okf_consume.build_payload(FIXTURE, question=question, withheld_nearest=0)
empty = none_at_all["withheld"]
assert isinstance(empty, Mapping)
assert empty["nearest"] == []
def test_the_cli_carries_both_switches() -> None:
parsed = okf_consume.parse_args(["b", "--question", "q", "--withheld-full"])
assert parsed.withheld_full is True
assert parsed.withheld_nearest == okf_consume.WITHHELD_NEAREST_DEFAULT
assert okf_consume.parse_args(["b", "--question", "q"]).withheld_full is False
assert (
okf_consume.parse_args(["b", "--question", "q", "--withheld-nearest", "3"]).withheld_nearest
== 3
)
def test_the_payload_declares_the_revision_whose_shape_it_has() -> None:
"""`withheld` went from a list to a mapping. A consumer reading the old
revision string and indexing it as a list would break silently, so the
string moves with the shape.
"""
assert okf_consume.CONTRACT_REVISION == "okf-consumption/2"
payload = okf_consume.build_payload(FIXTURE, question="Hvordan skal prisene fylles ut?")
assert payload["contract"] == "okf-consumption/2"

50
tests/test_okf_skill.py
View file

@ -366,42 +366,46 @@ def test_a_project_skill_still_passes_the_contract_checker(tmp_path: Path) -> No
@pytest.mark.parametrize("bundle", BUNDLES, ids=lambda path: path.name)
def test_the_breaking_point_is_a_measurement_or_it_is_withheld(
def test_the_payload_cost_section_is_measured_on_the_bundle_it_names(
bundle: Path, tmp_path: Path
) -> None:
"""`0 concepts` was a division that never happened, printed as a number.
"""The section that used to extrapolate a breaking point now states cost.
The figure is EXTRAPOLATED from what one `withheld` entry costs, so a
generation run that withheld nothing has no slope to extrapolate from:
`per_withheld` was `0.0`, the guard returned the literal `0`, and the
document told its reader the bundle's bookkeeping fills a 120000-byte
budget at zero concepts -- before the bundle holds anything at all.
`0 concepts` was a division that never happened, printed as a number: the
figure was extrapolated from what ONE `withheld` entry cost, and a run
that withheld nothing had no slope. Since `okf-consumption/2` there is no
slope at all -- the block is counts plus a capped sample -- so the section
states what this bundle's bookkeeping cost and no extrapolated count, and
a generator that printed one would be describing the previous revision.
Driven from both sides so a generator that simply stopped stating the
figure would fail: the bundle that withholds nothing must say it could not
measure it, and a bundle that withholds something must still print a
positive count.
Driven from both sides: the numbers must be the payload's own, and the
retired figure must not come back.
"""
written = _generate(bundle, tmp_path / "out")
text = written.read_text(encoding="utf-8")
payload = json.loads((tmp_path / "out" / "references" / "example-payload.json").read_text())
assert payload["withheld"], "the known-positive arm withheld nothing to extrapolate from"
assert "**0 concepts**" not in text
stated = re.search(r"At roughly\s+\*\*(\d+) concepts\*\*", text)
assert stated is not None, "a bundle that withheld something states no figure"
block = payload["withheld"]
assert block["total"], "the known-positive arm withheld nothing to report"
assert "At roughly" not in text and "**0 concepts**" not in text
stated = re.search(r"\*\*(\d+) bytes\*\* — (\d+) withheld of (\d+) concepts", text)
assert stated is not None, "the section states no measured cost"
assert int(stated.group(2)) == block["total"]
assert int(stated.group(1)) > 0
named = re.search(r"of which \*\*(\d+)\*\* are named", text)
assert named is not None and int(named.group(1)) == len(block["nearest"])
def test_a_generation_that_withheld_nothing_says_so_instead_of_printing_zero(
def test_a_generation_that_withheld_nothing_states_a_zero_it_measured(
tmp_path: Path,
) -> None:
"""The arm the SHIPPED skill is on, and the one that was wrong.
"""The arm the SHIPPED skill is on.
`okf skill --example-question "Hva sier veiledningen om krav?"` delivers
all three concepts of the golden bundle, so `withheld` is empty and there
is no per-entry cost. The question is part of what the shipped file is
(`skills/okf-consume/references/README.md`), which is why the defect was
in the repository rather than only reachable in theory.
all three concepts of the golden bundle, so nothing is withheld. Under the
flat list that left no per-entry cost and the guard printed `0 concepts`;
under counts-plus-names the zero is a count the run actually made, so it
is stated rather than withheld -- and the retired figure must still be
absent.
"""
written = okf_skill.generate(
GOLDEN,
@ -411,7 +415,7 @@ def test_a_generation_that_withheld_nothing_says_so_instead_of_printing_zero(
)
text = written.read_text(encoding="utf-8")
payload = json.loads((tmp_path / "out" / "references" / "example-payload.json").read_text())
assert payload["withheld"] == [], "the premise of this arm no longer holds"
assert payload["withheld"]["total"] == 0, "the premise of this arm no longer holds"
assert "**0 concepts**" not in text
assert "breaking point could not be measured" in text
assert "At roughly" not in text
assert "0 withheld of 3 concepts" in text

10
tests/test_shared_id_prefix.py
View file

@ -83,9 +83,15 @@ def _leaf(concept_id: str) -> str:
def _read(bundle: Path) -> tuple[list[str], dict[str, str]]:
payload = okf_consume.build_payload(bundle, question=QUESTION, k=10)
# `withheld_full`: this test names the rule for EVERY concept, so it asks
# for the whole set rather than the nearest N a reader is handed.
payload = okf_consume.build_payload(bundle, question=QUESTION, k=10, withheld_full=True)
delivered = [_leaf(e["concept_id"]) for e in payload["excerpts"] if isinstance(e, dict)]
withheld = {_leaf(w["concept_id"]): w["rule"] for w in payload["withheld"]}
block = payload["withheld"]
assert isinstance(block, dict) and block["complete"] is True
entries = block["nearest"]
assert isinstance(entries, list)
withheld = {_leaf(w["concept_id"]): w["rule"] for w in entries}
return delivered, withheld

9
tests/test_tie_shared_rank.py
View file

@ -183,7 +183,8 @@ def test_the_cli_exposes_the_flag_and_defaults_it_on(tmp_path: Path) -> None:
assert parsed.tie_shared_rank is True
parsed_off = okf_consume.parse_args([str(root), "--question", QUESTION, "--no-tie-shared-rank"])
assert parsed_off.tie_shared_rank is False
# `--withheld-titles` did NOT move with it, asserted here so the two are
# one measurement rather than two files' worth of trust: it is off for a
# reason of BYTES, which nothing this round touched.
assert parsed.withheld_titles is False
# The withheld cap did NOT move with it, asserted here so the two are one
# measurement rather than two files' worth of trust: it is a number chosen
# for reasons of BYTES, which nothing this round touched.
assert parsed.withheld_nearest == okf_consume.WITHHELD_NEAREST_DEFAULT
assert parsed.withheld_full is False