open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): parent reaches the reader -- excerpt field, body link, checker rule

Browse source 
K3-21 A. `okf consume` resolves a concept's `parent:` pointer -- a
`segment_id`, unique only inside one document's plan -- among the concepts
sharing its `source_file` (`consume.link_parents`, one pass, no file opened
again) and an excerpt carries `parent: { concept_id, title }`. Conditional
like `req_number`: a concept with no `parent` key moves no byte. A pointer
that lands nowhere is named `parent_unresolved: true`, never dropped.

The door writes ONE line into a heading-only body whose entry has a parent:
`Enclosing section: [<title>](/<bundle-relative path>)` (SPEC SS 5.1 lineage
through links, SS 6.1 the recommended absolute form and the kind in the
prose). Only such a body, so the segmented goldens' declared parents -- bodies
holding text -- are untouched. Appended AFTER structure derivation and
screened on its own (`_screened`, the `description` rule): read as body text
the link was derived into a second, unresolved `references` edge, measured on
the fixture. `segmentation.heading_only` is the one predicate the proposer and
the door share.

`okf check` gains its seventeenth rule, `parent_unfollowable`: a `parent`
that is not a concept_id and title, names its own excerpt, or names a concept
in neither `excerpts` nor `withheld` (together every considered concept).
Contract SS 8 point 6 added, the figure carries `parent`, and "additional
members are not read by the checker" now says the checker reads only the
members SS 8 names. The template tells the reader what `parent` is and that
SS 2.2 lets it read that one concept; `skill.CONDITIONAL_FIELDS` gains
`parent`. README and CLAUDE.md say what consume now reads.

Moved on purpose, each named: the SS 7.4 known-positive IS the contract
document, so `budget.known_positive` moves in every payload (13 238 / 12 893
/ 345 -> 14 455 / 14 083 / 372); `skills/okf-consume/` regenerated from the
segmented golden, whose plan declares s1 and s2 under s0 -- its example
payload now carries both parents; `test_bundle_identity` 16 -> 17 rules;
`test_shell_parent`'s byte test also accounts for the link line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-11 12:36:23 +02:00
commit 4f7bd61500
16 changed files with 319 additions and 58 deletions
Download patch file Download diff file Expand all files Collapse all files

89
src/llm_ingestion_okf/consume.py
View file

@ -50,7 +50,7 @@ import re
import sys
import unicodedata
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Literal
@ -287,6 +287,58 @@ class Concept:
locators: Mapping[str, str]
frontmatter: Mapping[str, str]
body: str
#: The concept this one's `parent` pointer resolves to, as its `concept_id`
#: and `title`, or `None`. Set by :func:`link_parents` over the whole set
#: and never by :func:`read_concept`: whether a pointer lands is a fact
#: about the bundle, and one file does not know its bundle.
parent: Mapping[str, str] | None = None
#: `True` when the concept carries a `parent` key that resolves to no
#: concept of its own document -- the third state, as for `sources`.
parent_unresolved: bool = False
#: The key a concept names its enclosing section under, and the key that
#: section answers to. Both are the segmented profiles' (`inbox.py` writes a
#: plan entry's `parent_id` as `parent:`, naming the other entry's `segment_id`).
PARENT_KEY = "parent"
SEGMENT_ID_KEY = "segment_id"
def link_parents(concepts: Sequence[Concept]) -> list[Concept]:
"""Every concept, with its `parent` pointer resolved or named unresolved.
A `parent` value is a `segment_id`, and a segment id is unique only inside
the plan of ONE document -- `p1` exists in every document of a
multi-document bundle -- so it is resolved among the concepts sharing this
concept's `source_file`, never across the bundle. An id two concepts of one
document claim resolves to neither, the rule `structure._lookup` applies: an
ambiguous pointer that silently picks a winner is worse than one reported
unresolved.
One pass over concepts already read; no file is opened again. A concept
carrying no `parent` comes back as it went in, which is what keeps every
payload of a bundle without the key byte-identical.
"""
claims: dict[tuple[str, str], list[Concept]] = {}
for concept in concepts:
segment_id = concept.frontmatter.get(SEGMENT_ID_KEY, "").strip()
if segment_id:
claims.setdefault((concept.source_file, segment_id), []).append(concept)
linked: list[Concept] = []
for concept in concepts:
pointer = concept.frontmatter.get(PARENT_KEY, "").strip()
if not pointer:
linked.append(concept)
continue
owners = claims.get((concept.source_file, pointer), [])
if len(owners) == 1 and owners[0] is not concept:
target = owners[0]
linked.append(
replace(concept, parent={"concept_id": target.concept_id, "title": target.title})
)
else:
linked.append(replace(concept, parent_unresolved=True))
return linked
def read_concept(path: Path, *, bundle_root: Path, root_bundle_id: str) -> Concept:
@ -617,14 +669,14 @@ KNOWN_POSITIVE_CASE = "docs/consumption-contract.md, encoded as a JSON string"
#: `measure()`'s own answer for that file. Vacuous ALONE -- which is why the
#: delta below exists.
KNOWN_POSITIVE_EXPECTED = 13_238
KNOWN_POSITIVE_EXPECTED = 14_455
#: The second, independent route. `wc -c` reports 12 893 raw bytes for the same
#: The second, independent route. `wc -c` reports 14 083 raw bytes for the same
#: file; the difference is this file's JSON quoting and escaping overhead. A
#: reader can derive it without running `measure()` at all, and it moves the
#: moment `measure()` changes what it counts -- which is what stops
#: `expected == measured` from proving nothing.
KNOWN_POSITIVE_ENCODING_DELTA = 345
KNOWN_POSITIVE_ENCODING_DELTA = 372
#: The two places that file can be, resolved in this order.
#:
@ -1585,6 +1637,13 @@ def excerpt_for(concept: Concept) -> dict[str, object] | None:
1 on three bundles while the model could not name it: the ranking found the
document and the delivery dropped the key.
Since K3-21, `parent` too, on the same terms: the concept that encloses
this one, as its `concept_id` and `title`, written only when the concept
names one and the pointer resolves (`link_parents`). Never the raw
`segment_id` the producer wrote -- a reader holding `p1977` can open
nothing -- and `parent_unresolved` where the pointer lands nowhere, the
`sources_unreadable` rule for the same reason.
Trailing whitespace is stripped per line: a spreadsheet render is padded to
hundreds of trailing spaces per line, and unstripped, most of a budget goes
on padding.
@ -1613,6 +1672,10 @@ def excerpt_for(concept: Concept) -> dict[str, object] | None:
# HAS an address and this reader could not decode it.
excerpt["sources_unreadable"] = True
excerpt.update(concept.locators)
if concept.parent is not None:
excerpt["parent"] = dict(concept.parent)
elif concept.parent_unresolved:
excerpt["parent_unresolved"] = True
excerpt["text_sha256"] = hashlib.sha256(text.encode("utf-8")).hexdigest()
excerpt["text"] = text
return excerpt
@ -1864,14 +1927,16 @@ def build_payload(
)
root_bundle_id = root_bundle_id_of(bundle_root, profile=profile)
concept_ids = enumerate_concepts(bundle_root, profile=profile)
concepts = [
read_concept(
bundle_root / f"{concept_id}{profile.paths.concept_suffix}",
bundle_root=bundle_root,
root_bundle_id=root_bundle_id,
)
for concept_id in concept_ids
]
concepts = link_parents(
[
read_concept(
bundle_root / f"{concept_id}{profile.paths.concept_suffix}",
bundle_root=bundle_root,
root_bundle_id=root_bundle_id,
)
for concept_id in concept_ids
]
)
# The bundle's OWN vocabulary, and the reason the rule is a set rather than
# a threshold: `pris` is a word here and `bila` is not, which is what
# separates a Norwegian compound from four coincidental characters. One

56
src/llm_ingestion_okf/contract_check.py
View file

@ -301,6 +301,61 @@ def rule_excerpt_named(ctx: Context) -> list[Finding]:
]
def rule_excerpt_parent(ctx: Context) -> list[Finding]:
"""SS 8.6: an excerpt's `parent`, when it carries one, is a pointer a
reader can follow.
Added 2026-09-11 (K3-21). Round 20 wrote `parent:` onto 675 of 710
heading-only sections of one standard as a `segment_id`, a value a reader
can open nothing with. The pre-pass now resolves it, and this rule holds
the resolved form: a `concept_id` and a `title`, naming a concept other
than the excerpt itself.
**The payload is its own denominator.** `excerpts` and `withheld` together
name every concept the pre-pass considered, which is every concept of the
bundle (SS 5.2), so a `parent.concept_id` in neither names nothing in the
bundle -- and the rule sees that without opening the bundle, the boundary
`rule_bundle_identity` keeps too. A payload lying about both lists at once
passes here and fails `denominator_identity` instead.
**Conditional, like SS 8.4's fields.** An excerpt with no `parent` meets
this rule as it always did. `parent_unresolved` is not a finding: SPEC SS
6.1, "Consumers MUST tolerate broken links", and a pointer named as
unresolved is the honest form of one.
"""
if not ctx.payload_is_mapping:
return []
excerpts = [_mapping(raw) for raw in _sequence(ctx.payload.get("excerpts"))]
considered = {_text(excerpt.get("concept_id")) for excerpt in excerpts} | {
_text(_mapping(raw).get("concept_id")) for raw in _sequence(ctx.payload.get("withheld"))
}
considered.discard("")
findings = []
for position, excerpt in enumerate(excerpts):
if "parent" not in excerpt:
continue
parent = excerpt.get("parent")
target = _text(_mapping(parent).get("concept_id"))
if not isinstance(parent, Mapping) or not target or not _text(parent.get("title")):
reason = "is not a `concept_id` and a `title`, so a reader can neither open nor cite it"
elif target == _text(excerpt.get("concept_id")):
reason = f"names the excerpt itself ({target!r})"
elif target not in considered:
reason = (
f"names {target!r}, which is in neither `excerpts` nor `withheld` and so is "
"no concept of this bundle"
)
else:
continue
findings.append(
Finding(
"parent_unfollowable",
f"excerpt {position}'s `parent` {reason} (SS 8.6)",
)
)
return findings
def rule_excerpt_states(ctx: Context) -> list[Finding]:
if not ctx.payload_is_mapping:
return []
@ -523,6 +578,7 @@ RULES: tuple[Callable[[Context], list[Finding]], ...] = (
rule_bundle_identity,
rule_excerpt_source_marking,
rule_excerpt_named,
rule_excerpt_parent,
rule_excerpt_states,
rule_denominator_identity,
rule_denominator_lists,

50
src/llm_ingestion_okf/inbox.py
View file

@ -52,6 +52,7 @@ from .segmentation import (
SegmentationPlan,
SegmentEntry,
assert_plan_applies,
heading_only,
observed_extractor_version,
slice_segments,
)
@ -101,6 +102,33 @@ def inbox_filename(slug: str, *, profile: BundleProfile = DEFAULT) -> str:
)
#: The words around the one link a heading-only body gains. SPEC SS 6.1: the
#: kind of relationship "is conveyed by the surrounding prose, not by the link
#: itself", so the line names the relation, in two words, because generic code
#: writes them into a body in the source's own language.
ENCLOSING_SECTION = "Enclosing section"
def _link_enclosing(body: str, parent: SegmentEntry, gate: Gate) -> str:
"""`body` plus one line linking the section that encloses it.
SPEC SS 5.1: "Lineage is expressed through links, not a dedicated field";
the `parent:` key stays beside it as a SS 4.1 extension. Bundle-relative
and absolute, SS 6.1's "recommended form", because the parent sits in
another directory and a relative link would count `..` across a layout the
next round may change.
The line carries the parent's title, which is document text, and joins a
body the gate has already judged -- so it is screened on its own, the rule
`_screened` applies to a `description`, and dropped rather than refused
when the gate would not persist it.
"""
line = _screened(gate, f"{ENCLOSING_SECTION}: [{parent.title}](/{parent.path})")
if line is None:
return body
return f"{body.rstrip(chr(10))}\n\n{line}\n"
def _normalize_body(text: str) -> str:
# LF-only with exactly one trailing newline is a byte-level guarantee, and
# dropped files legitimately arrive with CRLF — normalising is the
@ -820,6 +848,16 @@ def _render_segments(
extractor_version=observed_extractor_version(extractor_id),
)
sliced = slice_segments(text, plan)
# A body that is its heading alone gains ONE line linking the section its
# `parent` names. Only such a body: one holding text already has something
# to read, and the segmented goldens' declared parents are bodies holding
# text. Decided on the SLICE, the window the proposer's own predicate read.
enclosing = {entry.segment_id: entry for entry in plan.entries}
linked = {
entry.segment_id
for entry, body in sliced
if entry.parent_id is not None and heading_only(body)
}
decisions = [(entry, gate(body)) for entry, body in sliced]
refused = [
@ -837,11 +875,21 @@ def _render_segments(
if profile.index.facets is not None:
structure = derive_document_structure(decision.sanitized_text, source_file=source_file)
_validate_facets(structure, profile)
body = decision.sanitized_text
if entry.segment_id in linked:
# AFTER the structure is derived, and that order is the rule: read
# as body text, the link is a bundle-local target, so derivation
# would restate the `parent` relation as a `references` edge -- one
# relation under two kinds, the second rendered unresolved because
# nothing resolves the absolute form. Measured on the fixture
# before this order was chosen.
assert entry.parent_id is not None
body = _link_enclosing(body, enclosing[entry.parent_id], gate)
outputs.append(
(
entry.path,
render_inbox_concept(
decision.sanitized_text,
body,
okf_type=entry.okf_type,
# DECLARED by the adjudicator, never derived from the
# segment's own first line: the plan is the record of the

5
src/llm_ingestion_okf/propose.py
View file

@ -65,7 +65,7 @@ from .errors import IngestError
from .extract import OutlineMark, extract_text, strip_converter_attribute, xml_outline
from .extract import pdf_outline as extract_pdf_outline
from .materialize import reduce_to_id_grammar
from .segmentation import observed_extractor_version
from .segmentation import heading_only, observed_extractor_version
#: Stamped into every entry's `derived` list. The marker is what keeps a
#: proposal from being mistaken for the judgement the run path replays.
@ -1488,8 +1488,7 @@ def _link_shells(entries: list[dict[str, Any]], levels: Sequence[int], text: str
grew the excerpts past the budget.
"""
bodied = [
any(line.strip() and not line.startswith("#") for line in text[start:end].split("\n"))
for start, end in (entry["span"] for entry in entries)
not heading_only(text[start:end]) for start, end in (entry["span"] for entry in entries)
]
for index, entry in enumerate(entries):
if bodied[index]:

12
src/llm_ingestion_okf/segmentation.py
View file

@ -45,6 +45,18 @@ from typing import Any
from .errors import SegmentationError
from .materialize import reduce_to_id_grammar
def heading_only(text: str) -> bool:
"""`True` when no line of `text` holds anything but a markdown heading.
ONE predicate for two readers that must agree: the proposer gives such a
span a parent (`propose._link_shells`), and the door writes a link into
exactly those bodies. Both read the same window on the same extracted text,
so a second copy of the rule could only ever disagree with the first.
"""
return not any(line.strip() and not line.startswith("#") for line in text.split("\n"))
#: The top-level keys a plan payload must carry. Every one is required: a plan
#: missing its extractor identity would still parse, and would then be replayed
#: against an extraction nobody checked it against.

22
src/llm_ingestion_okf/skill.py
View file

@ -13,7 +13,8 @@ passes the UNFILLED template against a real payload (exit 0, 15 rules, 0
findings), and passes a skill built for a different bundle against this one's
payload. `contract_check.rule_bundle_identity` now compares the identity a
skill declares with the identity its payload declares, so all three measured
pairs are refused at exit 1 with one `bundle_mismatch` finding over 16 rules:
pairs are refused at exit 1 with one `bundle_mismatch` finding over 16 rules
(17 since K3-21's `parent_unfollowable`):
a skill against another bundle's payload, the unfilled template against a real
payload, and -- the arm an id comparison would miss -- a payload sharing the
skill's `bundle_id` at a foreign `ref`. The right pair is untouched at exit 0
@ -99,6 +100,7 @@ CONDITIONAL_FIELDS = (
"verified",
"req_number",
"sources",
"parent",
)
#: Tokens too short to carry a question. The same floor the pre-pass's own
@ -337,14 +339,16 @@ def render(
bundle_id = okf_consume.root_bundle_id_of(bundle_root, profile=profile)
ref = okf_consume.bundle_ref(bundle_root, profile=profile)
concept_ids = okf_consume.enumerate_concepts(bundle_root, profile=profile)
concepts = [
okf_consume.read_concept(
bundle_root / f"{concept_id}{profile.paths.concept_suffix}",
bundle_root=bundle_root,
root_bundle_id=bundle_id,
)
for concept_id in concept_ids
]
concepts = okf_consume.link_parents(
[
okf_consume.read_concept(
bundle_root / f"{concept_id}{profile.paths.concept_suffix}",
bundle_root=bundle_root,
root_bundle_id=bundle_id,
)
for concept_id in concept_ids
]
)
if not concepts:
raise SkillError(
f"{bundle_root} has an index but no concept under it; a skill for an "