open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(check): the contract and the checker state the payload's v1.1 forms

Browse source 
C6. `docs/consumption-contract.md` now says what the pre-pass emits:

- SS 2.5 point 2: sub-questions are written in the bundle's terms and given to
  the pre-pass in ONE run where it takes several;
- SS 8's example is `okf-consumption/2` as shipped: `withheld` is the
  `{total, by_rule, nearest, complete}` mapping, `coverage` carries
  `absent_terms` and `weak`, an excerpt carries `text`, `text_sha256`,
  `passage` and `own_title`; point 1 closes the identity on `withheld.total`;
- point 7 keeps the falsified verdicts on record and states the one reading
  carried since v1.1, `weak`, with its rule: a word held in no form, or
  nothing delivered -- whether a word exists at all, which does not move with
  question style or corpus size the way the two shares did;
- point 8 the passage (`{start, end, of}`, `text_sha256` over the passage,
  `sha256` still the file's), point 9 several sub-questions in one run (one
  cut, `questions`, `subquestions`, per-sub-question coverage), point 10 a
  title inherited from the heading above, with `own_title`;
- SS 10 names this library's default ranking (BM25) and `--ranking fusion`.

`okf check` holds the two new forms a reader acts on: `passage_malformed`
(not whole numbers with 0 <= start < end <= of) and `subquestions_unindexed`
(not distinct indices into `questions`, or indices in a payload listing none).
17 -> 19 rules; the two tests that pin the published count move with it.
Each rule is held against a real payload (0 findings) and against that
payload broken six ways.

Editing the contract moved the SS 7.4 known-positive, measured once after the
edit: 19 837 -> 23 672 encoded, 19 358 -> 23 092 raw, delta 479 -> 580. The
example payload and `skills/okf-consume` are regenerated by the published
recipe and `okf check` reports 19 rules, 0 findings on them.

Suite on a clean tree after `git add`: 2443 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-21 08:47:46 +02:00
commit 977040f575
11 changed files with 308 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

8
CLAUDE.md
View file

@ -1563,6 +1563,11 @@ R761 **8** (S1-S6 + KP + KN), vegnormal **32** questions / **43**
48 000 (`lines_truncated`). The card (`okf card`, `okf_describe`) carries it
as `map` and no longer carries `source_files`. The skill and the server
instructions say: read the map, 2-4 sub-questions in its words, ONE call.
**C6:** the contract's SS 8 example and points 1, 7-10 state the current
form (`okf-consumption/2`'s `withheld` mapping, `absent_terms`/`weak`,
`passage`, `questions`/`subquestions`, `own_title`); `okf check` gains
`passage_malformed` and `subquestions_unindexed` (19 rules). Editing the
contract moved the known-positive to 23 672 / delta 580.
What follows describes the fusion.
- Consume a bundle: `okf consume <bundle> --question "<q>"
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
@ -1657,7 +1662,8 @@ R761 **8** (S1-S6 + KP + KN), vegnormal **32** questions / **43**
veiledningen om krav?"` are both required (the payload test asserts bytes for
that question), and the checkout prefix is then stripped, because `okf skill`
writes the bundle root and the skill path ABSOLUTE when `--out` is not under
`.claude/skills/`. The pair is rc 0, 17 rules (16 before K3-21), 0 findings,
`.claude/skills/`. The pair is rc 0, 19 rules since v1.1 C6 (17 before, 16
before K3-21), 0 findings,
and a test holds
the shipped bytes to the generator's. Its frontmatter `name` is now
`b-golden-segmented-okf-v0-2-consume`: Claude Code takes a project skill's

7
README.md
View file

@ -852,6 +852,13 @@ payload then carries `questions` instead of `question`, every excerpt names the
`weak` true only when every sub-question is weak. One question gives exactly
the payload it always did.
The consumption contract (`docs/consumption-contract.md` § 8.8–8.10) states
the three new forms — a passage, several sub-questions, a heading-inherited
title — and `okf check` holds the first two, with nineteen rules:
`passage_malformed` refuses a `passage` that is not whole numbers with
`0 <= start < end <= of`, and `subquestions_unindexed` refuses an excerpt whose
`subquestions` are not distinct indices into the payload's `questions`.
`--cost-vocabulary` is off by default and widens one question class: it lets a
declared list of cost/price/quantity terms bridge a question and a document that
name money with different words. The gate is the question — one naming no such

77
docs/consumption-contract.md
View file

@ -70,8 +70,9 @@ searches — and MUST NOT state one that stops at a single run.
to put the question into the bundle's terms. A bundle in one language and a
question in another share few tokens, and a pre-pass matching tokens then
ranks on almost nothing.
2. It MUST tell its reader to split a broad question into sub-questions and to
run the pre-pass per sub-question.
2. It MUST tell its reader to split a broad question into sub-questions written
in the bundle's own terms, and to give every sub-question to the pre-pass —
in ONE run where the pre-pass takes several (§ 8.9), otherwise one run each.
3. It MUST say that several runs are permitted and expected, and MUST NOT
contain a sentence forbidding a second run with other terms. Both the
`withheld` near misses (§ 5.3) and § 2.2 exist so that the second run can
@ -203,7 +204,7 @@ are permitted; the checker reads only the members this section names.
```json
{
"contract": "okf-consumption/1",
"contract": "okf-consumption/2",
"bundle": { "bundle_id": "<id>", "ref": "<commit or content identity>" },
"budget": {
"unit": "<named unit>",
@ -213,10 +214,13 @@ are permitted; the checker reads only the members this section names.
"known_positive": { "case": "<name>", "expected": 10406, "measured": 10406 }
},
"denominators": { "considered": 439, "withheld": 401, "delivered": 38 },
"question": "<the question asked>",
"coverage": {
"question_terms": ["<the terms the pre-pass read the question as>"],
"unanswered_in_bundle": ["<those no concept in the bundle answers>"],
"unanswered_in_payload": ["<those no delivered excerpt answers>"]
"unanswered_in_payload": ["<those no delivered excerpt answers>"],
"absent_terms": ["<those the bundle holds in no form at all>"],
"weak": false
},
"excerpts": [
{
@ -225,20 +229,31 @@ are permitted; the checker reads only the members this section names.
"sha256": "<hex digest of the concept file>",
"adjudication": "proposed",
"trust_tier": "machine-confirmed",
"title": "<the concept's own title>",
"title": "<the concept's own title, or the heading it stands under (point 10)>",
"own_title": "<the file's title, only where `title` was inherited>",
"req_number": "<the identifier the producer wrote, when there is one>",
"sources": [ { "resource": "<what the concept came from>" } ],
"source_pages": "<a locator into that resource, when the producer wrote one>",
"parent": { "concept_id": "<the concept enclosing this one>", "title": "<its title>" }
"parent": { "concept_id": "<the concept enclosing this one>", "title": "<its title>" },
"text_sha256": "<hex digest of `text`>",
"text": "<the concept body, or the passage of it that answers (point 8)>",
"passage": { "start": 8000, "end": 12000, "of": 35000 }
}
],
"withheld": [ { "concept_id": "<bundle-local id>", "rule": "<why it was cut>" } ]
"withheld": {
"total": 401,
"by_rule": { "below_k": 390, "no_lexical_match": 11 },
"nearest": [ { "concept_id": "<bundle-local id>", "rule": "below_k", "title": "<its title>" } ],
"complete": false
}
}
```
1. `len(excerpts)` MUST equal `denominators.delivered`, and `len(withheld)` MUST
equal `denominators.withheld`. The counts and the lists are two statements of
the same fact, and a payload where they disagree is refused.
1. `len(excerpts)` MUST equal `denominators.delivered`, and `withheld.total` MUST
equal `denominators.withheld`, with `withheld.by_rule` summing to it. The
counts and the lists are two statements of the same fact, and a payload where
they disagree is refused. `withheld.nearest` names the best-ranked drops and
is the whole withheld set only where `withheld.complete` is true.
2. `contract` MUST be present so a reader can tell which revision it is holding.
3. Every excerpt MUST carry a non-empty `title`. An excerpt a reader cannot name
is one an answer cannot cite, whatever its rank. This is a measurement, not a
@ -306,8 +321,45 @@ are permitted; the checker reads only the members this section names.
across corpora what was measured on one. Where a bar belongs is in the
consumer's own judgement or in a gate that publishes its sweep.
**One reading is carried, with its rule in the open (v1.1).** `absent_terms`
lists the question's words the bundle holds in NO form — not as written, and
not through a relative the pre-pass reads the word as — and `weak` is true
when nothing was delivered or `absent_terms` is not empty. That is not the
verdict falsified above: it asks whether a word exists in the bundle at all,
which does not move with question style or corpus size the way the two
shares did. A reader seeing `weak` rephrases in the bundle's own words, and
if it stays weak, says the bundle does not cover the question.
The checker does not read this member, for the same reason: it holds a
payload's FORM, and these lists carry no claim a form can be held to.
8. A pre-pass MAY deliver an excerpt of a large concept as the PASSAGE that
answers the question rather than the whole body. It then MUST say so, as
`passage: {start, end, of}` — whole numbers, `0 <= start < end <= of`,
character offsets of the passage in the concept's body and that body's
length — and `text_sha256` digests the delivered `text` while `sha256` stays
the concept file's, so the whole remains one fetch away by `concept_id`. The
delivered `text` MAY carry the nearest heading above the passage and markers
where text was left out, so its length is not `end - start`. The checker
refuses a `passage` that is not such a place (`passage_malformed`): a place
that is not a place sends a reader who fetches the concept to the wrong
characters of it.
9. A pre-pass MAY take several sub-questions in one run. Its payload then
carries `questions` — the list, in the order asked — in place of
`question`; every excerpt carries `subquestions`, the indices into
`questions` of the sub-questions it answered, the one whose text it carries
first; and `coverage` carries one block per sub-question under
`subquestions`, plus `weak_subquestions` and a `weak` that is true only when
every sub-question is weak. The cut is ONE: the same budget and the same `k`
one question gets, so asking several times in one run does not buy a larger
payload, and point 1's identities hold over the whole. The checker refuses
an excerpt whose `subquestions` are not distinct indices into `questions`,
or that carries them in a payload listing no `questions`
(`subquestions_unindexed`). A run of one question is the single form.
10. An excerpt whose concept's own title is a position rather than a name — a
table block a producer could only name by the line it starts on — MAY carry
as `title` the title of the concept it stands under in the same document,
and then MUST carry the file's own title as `own_title`, so the name shown
is never mistaken for the one in the file.
## 9. Prohibitions
@ -327,7 +379,10 @@ are permitted; the checker reads only the members this section names.
## 10. What this does not decide
- **No engine, ranker or cutter is designed here.** The contract binds a payload
and a document, not a retrieval algorithm.
and a document, not a retrieval algorithm. This library's pre-pass ranks by
BM25 over passages and titles by default and keeps its older three-signal
fusion reachable (`--ranking fusion`); a payload is conformant or not
whichever produced it.
- **No instrument is blessed.** § 7 requires that one be named and validated;
which one is the profile's choice.
- **Bundle shape is the producer's question.** Whether a corpus is nested, split

2
skills/okf-consume/SKILL.md
View file

@ -290,7 +290,7 @@ carries its denominator.
| Limit | `120000` |
| Unit | `utf-8 bytes of emitted JSON` |
| Instrument | `okf_consume.measure (len of the ensure_ascii=False JSON encoding, utf-8)` |
| Known-positive | `docs/consumption-contract.md, encoded as a JSON string` at `19837` |
| Known-positive | `docs/consumption-contract.md, encoded as a JSON string` at `23672` |
The instrument reproduces the known-positive figure before any of its own
numbers are believed. Report what the run actually spent.

2
skills/okf-consume/references/README.md
View file

@ -37,7 +37,7 @@ Why each part is there:
checkout on one machine. The line strips that checkout's prefix and nothing
else, and a test holds the shipped `SKILL.md` to the generator's output with
exactly that prefix removed.
- **`okf check`** should report `conformant: 17 rules over 3 excerpts and 0
- **`okf check`** should report `conformant: 19 rules over 3 excerpts and 0
withheld entries, 0 findings` and exit 0.
The generated `name` is `b-golden-segmented-okf-v0-2-consume` while this

8
skills/okf-consume/references/example-payload.json
View file

@ -11,10 +11,10 @@
"spent": 2289,
"known_positive": {
"case": "docs/consumption-contract.md, encoded as a JSON string",
"expected": 19837,
"measured": 19837,
"raw_bytes": 19358,
"encoding_delta": 479
"expected": 23672,
"measured": 23672,
"raw_bytes": 23092,
"encoding_delta": 580
}
},
"denominators": {

6
src/llm_ingestion_okf/consume.py
View file

@ -785,14 +785,14 @@ KNOWN_POSITIVE_CASE = "docs/consumption-contract.md, encoded as a JSON string"
#: `measure()`'s own answer for that file. Vacuous ALONE -- which is why the
#: delta below exists.
KNOWN_POSITIVE_EXPECTED = 19_837
KNOWN_POSITIVE_EXPECTED = 23_672
#: The second, independent route. `wc -c` reports 19 358 raw bytes for the same
#: The second, independent route. `wc -c` reports 23 092 raw bytes for the same
#: file; the difference is this file's JSON quoting and escaping overhead. A
#: reader can derive it without running `measure()` at all, and it moves the
#: moment `measure()` changes what it counts -- which is what stops
#: `expected == measured` from proving nothing.
KNOWN_POSITIVE_ENCODING_DELTA = 479
KNOWN_POSITIVE_ENCODING_DELTA = 580
#: The two places that file can be, resolved in this order.
#:

84
src/llm_ingestion_okf/contract_check.py
View file

@ -421,6 +421,88 @@ def rule_excerpt_parent(ctx: Context) -> list[Finding]:
return findings
def rule_subquestions_indexed(ctx: Context) -> list[Finding]:
"""SS 8.9: a payload asking several sub-questions indexes them.
`questions` is the list the caller asked, and every excerpt names the
sub-questions it answered as `subquestions`: distinct whole-number indices
into that list, at least one. An index a reader cannot look up names no
question, and an excerpt carrying indices in a payload that lists no
questions answers something the payload never states.
"""
if not ctx.payload_is_mapping:
return []
raw = ctx.payload.get("questions")
questions = _sequence(raw)
excerpts = _sequence(ctx.payload.get("excerpts"))
carried = [
position for position, excerpt in enumerate(excerpts) if "subquestions" in _mapping(excerpt)
]
if raw is None:
return [
Finding(
"subquestions_unindexed",
f"excerpt {position} names `subquestions` but the payload lists no "
"`questions` to look them up in (SS 8.9)",
)
for position in carried
]
findings: list[Finding] = []
count = len(questions)
for position, excerpt in enumerate(excerpts):
indices = _mapping(excerpt).get("subquestions")
values = [_whole(value) for value in _sequence(indices)]
if (
not isinstance(indices, list)
or not values
or any(value is None or not 0 <= value < count for value in values)
or len(set(values)) != len(values)
):
findings.append(
Finding(
"subquestions_unindexed",
f"excerpt {position} does not name the sub-questions it answered as "
f"distinct indices into the payload's {count} `questions` (SS 8.9)",
)
)
return findings
def rule_passage_placed(ctx: Context) -> list[Finding]:
"""SS 8.8: an excerpt delivered as a passage says where it lies.
`passage` is `{start, end, of}`: whole numbers with `0 <= start < end <=
of`, character offsets into the concept's body. A place that is not a
place sends a reader who fetches the whole concept to the wrong
characters of it.
"""
if not ctx.payload_is_mapping:
return []
findings: list[Finding] = []
for position, raw in enumerate(_sequence(ctx.payload.get("excerpts"))):
excerpt = _mapping(raw)
if "passage" not in excerpt:
continue
passage = excerpt["passage"]
place = _mapping(passage)
start, end, total = (_whole(place.get(key)) for key in ("start", "end", "of"))
if (
not isinstance(passage, Mapping)
or start is None
or end is None
or total is None
or not 0 <= start < end <= total
):
findings.append(
Finding(
"passage_malformed",
f"excerpt {position} carries a `passage` that is not whole numbers "
"with 0 <= start < end <= of (SS 8.8)",
)
)
return findings
def rule_excerpt_states(ctx: Context) -> list[Finding]:
if not ctx.payload_is_mapping:
return []
@ -736,6 +818,8 @@ RULES: tuple[Callable[[Context], list[Finding]], ...] = (
rule_excerpt_named,
rule_excerpt_parent,
rule_excerpt_states,
rule_subquestions_indexed,
rule_passage_placed,
rule_denominator_identity,
rule_denominator_lists,
rule_withheld_rules,

9
tests/test_bundle_identity.py
View file

@ -156,13 +156,14 @@ def test_a_payload_declaring_no_identity_is_left_to_its_own_rule(tmp_path: Path)
def test_the_rule_count_is_the_denominator_the_report_quotes() -> None:
"""The nevner moved 15 -> 16 with this rule and 16 -> 17 with
`parent_unfollowable` (K3-21), and every consumer quoting the old number is
"""The nevner moved 15 -> 16 with this rule, 16 -> 17 with
`parent_unfollowable` (K3-21) and 17 -> 19 with `subquestions_unindexed`
and `passage_malformed` (v1.1 C6), and every consumer quoting the old number is
quoting a number that has changed."""
text, payload = instantiate(BUNDLE_A)
report = contract_check.check(text, payload)
assert report.rules_evaluated == len(contract_check.RULES) == 17
assert "17 rules" in report.render()
assert report.rules_evaluated == len(contract_check.RULES) == 19
assert "19 rules" in report.render()
def test_the_generator_writes_an_identity_the_checker_can_read() -> None:

127
tests/test_contract_subquestions_passage.py Normal file
View file

@ -0,0 +1,127 @@
"""`okf check` holds the two forms v1.1 order C added to the payload (C6).
- `subquestions_unindexed`: a payload asking several sub-questions carries
`questions`, and every excerpt then names the sub-questions it answered as
indices into that list. An index a reader cannot look up names nothing.
- `passage_malformed`: an excerpt delivered as a passage of a larger concept
carries `passage: {start, end, of}`, and a place that is not a place --
backwards, past the end, not whole numbers -- sends a reader to the wrong
characters of the concept it fetches.
Each rule is held against a payload the pre-pass really produced (0 findings)
and against that payload broken one way at a time.
"""
from __future__ import annotations
import copy
import sys
from pathlib import Path
from typing import Any
import pytest
from llm_ingestion_okf import consume, contract_check
from llm_ingestion_okf import skill as okf_skill
TOOLS = Path(__file__).resolve().parent.parent / "tools"
if str(TOOLS) not in sys.path:
sys.path.insert(0, str(TOOLS))
import okf_retrieval_gate as retrieval # noqa: E402
@pytest.fixture(scope="module")
def bundle(tmp_path_factory: pytest.TempPathFactory) -> Path:
spec = retrieval.BundleSpec(
"contract-new-form",
(
retrieval.DocumentSpec(
"cabin",
"cabin.md",
(
retrieval.ConceptSpec(
slug="stove",
title="Stove",
body="The stove is lit with birch and kept burning all night. ",
repeat=200,
),
retrieval.ConceptSpec(
slug="well", title="Well", body="The well is drained in autumn."
),
),
),
),
)
return retrieval.build_bundle(tmp_path_factory.mktemp("contract") / "bundle", spec)
def _codes(payload: dict[str, Any]) -> list[str]:
report = contract_check.check(okf_skill.render_generic(), payload)
return [finding.code for finding in report.findings]
def _multi(bundle: Path) -> dict[str, Any]:
return consume.build_multi_payload(
bundle, questions=["How is the stove lit?", "When is the well drained?"]
)
def _passage(bundle: Path) -> dict[str, Any]:
payload = consume.build_payload(bundle, question="How is the stove lit with birch?")
assert any("passage" in excerpt for excerpt in payload["excerpts"]), "the premise"
return payload
def test_the_checker_has_nineteen_rules() -> None:
assert len(contract_check.RULES) == 19
assert contract_check.rule_subquestions_indexed in contract_check.RULES
assert contract_check.rule_passage_placed in contract_check.RULES
def test_real_payloads_of_both_forms_are_conformant(bundle: Path) -> None:
assert _codes(_multi(bundle)) == []
assert _codes(_passage(bundle)) == []
@pytest.mark.parametrize(
"break_it",
[
lambda p: p["excerpts"][0].__setitem__("subquestions", [2]),
lambda p: p["excerpts"][0].__setitem__("subquestions", []),
lambda p: p["excerpts"][0].__setitem__("subquestions", [0, 0]),
lambda p: p["excerpts"][0].__setitem__("subquestions", ["0"]),
lambda p: p["excerpts"][0].pop("subquestions"),
lambda p: p.pop("questions"),
],
ids=["out-of-range", "empty", "repeated", "not-a-number", "missing", "no-questions"],
)
def test_a_subquestion_index_a_reader_cannot_look_up_is_refused(
bundle: Path, break_it: Any
) -> None:
payload = copy.deepcopy(_multi(bundle))
break_it(payload)
codes = _codes(payload)
assert codes and set(codes) == {"subquestions_unindexed"}
def _passage_excerpt(payload: dict[str, Any]) -> dict[str, Any]:
return next(excerpt for excerpt in payload["excerpts"] if "passage" in excerpt)
@pytest.mark.parametrize(
"passage",
[
{"start": 10, "end": 5, "of": 100},
{"start": 0, "end": 101, "of": 100},
{"start": -1, "end": 5, "of": 100},
{"start": 0, "end": 5},
{"start": "0", "end": 5, "of": 100},
"0-5",
],
ids=["backwards", "past-the-end", "negative", "no-of", "not-a-number", "not-a-mapping"],
)
def test_a_passage_that_is_not_a_place_is_refused(bundle: Path, passage: object) -> None:
payload = copy.deepcopy(_passage(bundle))
_passage_excerpt(payload)["passage"] = passage
assert _codes(payload) == ["passage_malformed"]

6
tests/test_parent_reaches_reader.py
View file

@ -186,13 +186,13 @@ def _pair(tmp_path: Path) -> tuple[str, dict[str, Any]]:
return text, dict(payload)
def test_the_checker_has_seventeen_rules_and_accepts_a_followable_parent(tmp_path: Path) -> None:
def test_the_checker_has_nineteen_rules_and_accepts_a_followable_parent(tmp_path: Path) -> None:
text, payload = _pair(tmp_path)
assert any("parent" in excerpt for excerpt in payload["excerpts"])
report = contract_check.check(text, payload)
assert report.findings == ()
assert report.rules_evaluated == len(contract_check.RULES) == 17
assert "17 rules" in report.render()
assert report.rules_evaluated == len(contract_check.RULES) == 19
assert "19 rules" in report.render()
@pytest.mark.parametrize(