open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): parent reaches the reader -- excerpt field, body link, checker rule

Browse source 
K3-21 A. `okf consume` resolves a concept's `parent:` pointer -- a
`segment_id`, unique only inside one document's plan -- among the concepts
sharing its `source_file` (`consume.link_parents`, one pass, no file opened
again) and an excerpt carries `parent: { concept_id, title }`. Conditional
like `req_number`: a concept with no `parent` key moves no byte. A pointer
that lands nowhere is named `parent_unresolved: true`, never dropped.

The door writes ONE line into a heading-only body whose entry has a parent:
`Enclosing section: [<title>](/<bundle-relative path>)` (SPEC SS 5.1 lineage
through links, SS 6.1 the recommended absolute form and the kind in the
prose). Only such a body, so the segmented goldens' declared parents -- bodies
holding text -- are untouched. Appended AFTER structure derivation and
screened on its own (`_screened`, the `description` rule): read as body text
the link was derived into a second, unresolved `references` edge, measured on
the fixture. `segmentation.heading_only` is the one predicate the proposer and
the door share.

`okf check` gains its seventeenth rule, `parent_unfollowable`: a `parent`
that is not a concept_id and title, names its own excerpt, or names a concept
in neither `excerpts` nor `withheld` (together every considered concept).
Contract SS 8 point 6 added, the figure carries `parent`, and "additional
members are not read by the checker" now says the checker reads only the
members SS 8 names. The template tells the reader what `parent` is and that
SS 2.2 lets it read that one concept; `skill.CONDITIONAL_FIELDS` gains
`parent`. README and CLAUDE.md say what consume now reads.

Moved on purpose, each named: the SS 7.4 known-positive IS the contract
document, so `budget.known_positive` moves in every payload (13 238 / 12 893
/ 345 -> 14 455 / 14 083 / 372); `skills/okf-consume/` regenerated from the
segmented golden, whose plan declares s1 and s2 under s0 -- its example
payload now carries both parents; `test_bundle_identity` 16 -> 17 rules;
`test_shell_parent`'s byte test also accounts for the link line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-11 12:36:23 +02:00
commit 4f7bd61500
16 changed files with 319 additions and 58 deletions
Download patch file Download diff file Expand all files Collapse all files

25
CLAUDE.md
View file

@ -625,8 +625,18 @@ and fixtures, never code.
The route reads the PLAN, never the row: on R761 it names the ancestor the
`<sec>` nesting names on **708 of 710** shells (the 2 misses sit at depth 7,
clipped to 6), where reading section numbers gets 686 (`12` begins with
`1`). 35 of 710 have no ancestor holding text and get none. Off because
`okf consume` reads no `parent` key. **The index renders every such parent
`1`). 35 of 710 have no ancestor holding text and get none. **Since K3-21
`okf consume` reads the key** (`consume.link_parents`, resolved among the
concepts of the concept's OWN `source_file`, because `p1` exists in every
document): an excerpt carries `parent: { concept_id, title }`, conditional
like `req_number`, or `parent_unresolved: true` where the pointer lands
nowhere, and a heading-only body gains ONE line `Enclosing section:
[title](/path)` (SPEC SS 5.1 lineage through links, SS 6.1 the recommended
absolute form), appended AFTER structure derivation -- read as body text it
became a second, unresolved `references` edge -- and screened on its own.
`okf check`'s seventeenth rule, `parent_unfollowable`, holds the form.
Whether the build default moves is K3-21 B's measurement, not this line's.
**The index renders every such parent
UNRESOLVED** (`parent: p1?`): `structure` reads `parent` as a document
NUMBER, and a segment id answers to none -- the one key with two meanings
`inbox.py` already names for a declared parent. Report:
@ -745,7 +755,10 @@ and fixtures, never code.
allowlist, because a list names the producers its author thought of and one
bundle locates by `source_element_id` on 269 of 274 concepts. A prefix, never
a substring (`resource_owner` is not a locator). An absent key stays absent
and an undecodable address is named (`sources_unreadable`). `sources` is READ
and an undecodable address is named (`sources_unreadable`). Since K3-21 an
excerpt also carries `parent`, the enclosing concept's `concept_id` and
`title` -- never the raw `segment_id` -- or `parent_unresolved`, and the
contract's SS 8 point 6 says the checker reads it. `sources` is READ
in both YAML forms because the two real bundles disagree (flow 629/629 on one,
block 270/270 on the other) — reading block is not a licence to write it, the
emission rule is unchanged. Contract § 8 makes `title` a MUST (checker code
@ -769,7 +782,8 @@ and fixtures, never code.
numbers a generic skill can only leave as holes or state falsely, and that
argument NEVER rested on conformance, so closing the measurement leaves it
standing. **That half is CLOSED 2026-09-10 by `bundle_mismatch`, the checker's
SIXTEENTH rule** (`RULES` is a tuple and `Report.rules_evaluated` is
SIXTEENTH rule** (seventeen since K3-21 added `parent_unfollowable`;
`RULES` is a tuple and `Report.rules_evaluated` is
`len(RULES)`, so every published «15 rules» line is now «16 rules» — a
contract change for anyone quoting it). It compares the identity a skill
DECLARES against the identity its payload declares, **both `bundle_id` AND
@ -809,7 +823,8 @@ and fixtures, never code.
veiledningen om krav?"` are both required (the payload test asserts bytes for
that question), and the checkout prefix is then stripped, because `okf skill`
writes the bundle root and the skill path ABSOLUTE when `--out` is not under
`.claude/skills/`. The pair is rc 0, 16 rules, 0 findings, and a test holds
`.claude/skills/`. The pair is rc 0, 17 rules (16 before K3-21), 0 findings,
and a test holds
the shipped bytes to the generator's. Its frontmatter `name` is now
`b-golden-segmented-okf-v0-2-consume`: Claude Code takes a project skill's
COMMAND from the directory name and uses `name` only as a display label, and

14
README.md
View file

@ -170,9 +170,12 @@ lets every nested section inherit them: measured on one process code, **710 of
2 761** concepts are heading-only, and the plan's level and order name the same
ancestor as the document's own nesting on **708 of 710** (the two others sit at
depth 7, which a markdown heading clips to 6, and point one level too high). It
is off because `okf consume` does not read `parent`, so no payload ranks
differently with it, while it moves the bytes of every bundle holding a
heading-only section. One cost is known and not repaired: the index projects
was off because `okf consume` did not read `parent`. Since K3-21 it does: an
excerpt carries `parent` as the enclosing concept's `concept_id` and `title`
(resolved inside the concept's own document, never the raw segment id), and a
heading-only body gains one line, `Enclosing section: [title](/path)`, in the
bundle-relative form SPEC § 6.1 recommends. Whether that moves the default is
measured separately. One cost is known and not repaired: the index projects
`parent` as a document number, so a segment id always renders there as
unresolved (`parent: p1?`), even though the concept it names is in the bundle.
@ -518,7 +521,10 @@ locators its author thought of, and one real bundle locates by
`source_element_id` on 269 of its 274 concepts. A key the producer did not write
stays absent rather than arriving empty, and an address this reader cannot
decode is named (`sources_unreadable`) rather than dropped into the same
silence. The reason is a measurement: with `concept_id` and body text alone, a
silence. A concept naming the section that encloses it delivers `parent` — that
concept's `concept_id` and `title` — or `parent_unresolved` when the pointer
lands nowhere; `okf check` (seventeen rules) refuses a `parent` a reader could
not follow. The reason is a measurement: with `concept_id` and body text alone, a
delivered gold concept at rank 1 still left the answer unable to name the
document it was quoting.
`considered == withheld + delivered` closes by construction, and the payload is

19
docs/consumption-contract.md
View file

@ -146,7 +146,7 @@ carries them.
## 8. The payload shape
The pre-pass MUST emit one JSON object with these members. Additional members
are permitted and are not read by the checker.
are permitted; the checker reads only the members this section names.
```json
{
@ -170,7 +170,8 @@ are permitted and are not read by the checker.
"title": "<the concept's own title>",
"req_number": "<the identifier the producer wrote, when there is one>",
"sources": [ { "resource": "<what the concept came from>" } ],
"source_pages": "<a locator into that resource, when the producer wrote one>"
"source_pages": "<a locator into that resource, when the producer wrote one>",
"parent": { "concept_id": "<the concept enclosing this one>", "title": "<its title>" }
}
],
"withheld": [ { "concept_id": "<bundle-local id>", "rule": "<why it was cut>" } ]
@ -206,6 +207,20 @@ are permitted and are not read by the checker.
by that producer under the same rule — and any allowlist drops it, leaving an
excerpt that names a document without naming the place in it. The prefix is a
prefix and never a substring: `resource_owner` is not a locator.
6. An excerpt SHOULD carry `parent` when its concept names the section that
encloses it, as the `concept_id` and `title` of the concept that pointer
resolves to in the same bundle — never the producer's own pointer, which a
reader cannot open without enumerating the bundle (§ 9.2). It is conditional
on the producer exactly as point 4's fields are, and § 6.4 forbids reading
its absence as a claim that the source has no enclosing section. A pointer
the pre-pass cannot resolve is named rather than dropped, as
`parent_unresolved` set to `true`, for point 4's reason, and is not a defect:
SPEC § 6.1, "Consumers MUST tolerate broken links". A `parent` names that one concept as reachable
in § 2.2's sense and names nothing else. The checker refuses
(`parent_unfollowable`) a `parent` that is not both members, that names its
own excerpt, or whose `concept_id` is in neither `excerpts` nor `withheld` —
together those name every concept the pre-pass considered (§ 5.2), so a
pointer outside them names nothing in the bundle.
## 9. Prohibitions

12
skills/okf-consume-template/SKILL.md
View file

@ -121,6 +121,18 @@ because the producer wrote none — never because the source has none, and never
something to fill in. An excerpt carrying `sources_unreadable` has an address
this reader could not decode: say so rather than reporting no address.
**An excerpt carrying `parent` names the section that encloses it** — the
`concept_id` and `title` of another concept in this bundle. The payload names
that one concept as reachable (§ 2.2), so it is the one file outside the
delivered excerpts you may read: when an excerpt's `text` is its heading alone,
what that section inherits stands in the enclosing concept, whose file is its
`concept_id` plus `.md` under the bundle root. The text links it too, on a line
`Enclosing section: [title](/path)`, where `/` is the bundle root. Cite what you
take from it by that concept's own `(bundle_id, concept_id)`, never by the
excerpt that pointed to it. An excerpt carrying `parent_unresolved` names a
parent this reader could not find in the bundle: say so rather than reporting
that it has none.
| Marking | Use when |
|---|---|
| `extracted` | the bundle states it directly |

21
skills/okf-consume/SKILL.md
View file

@ -132,6 +132,18 @@ because the producer wrote none — never because the source has none, and never
something to fill in. An excerpt carrying `sources_unreadable` has an address
this reader could not decode: say so rather than reporting no address.
**An excerpt carrying `parent` names the section that encloses it** — the
`concept_id` and `title` of another concept in this bundle. The payload names
that one concept as reachable (§ 2.2), so it is the one file outside the
delivered excerpts you may read: when an excerpt's `text` is its heading alone,
what that section inherits stands in the enclosing concept, whose file is its
`concept_id` plus `.md` under the bundle root. The text links it too, on a line
`Enclosing section: [title](/path)`, where `/` is the bundle root. Cite what you
take from it by that concept's own `(bundle_id, concept_id)`, never by the
excerpt that pointed to it. An excerpt carrying `parent_unresolved` names a
parent this reader could not find in the bundle: say so rather than reporting
that it has none.
| Marking | Use when |
|---|---|
| `extracted` | the bundle states it directly |
@ -179,6 +191,7 @@ measurement about the producer, never a fact about the source.
| `verified` | **0 of 3** | no concept in this bundle carries it | that the source document lacks what the field asserts |
| `req_number` | **0 of 3** | no concept in this bundle carries it | that the source document lacks what the field asserts |
| `sources` | **3 of 3** | the producer wrote none for that concept | that the source document lacks what the field asserts |
| `parent` | **2 of 3** | the producer wrote none for that concept | that the source document lacks what the field asserts |
| `source_file` | **3 of 3** | the producer wrote none for that concept | that the source document lacks what the field asserts |
| `source_lines` | **3 of 3** | the producer wrote none for that concept | that the source document lacks what the field asserts |
| `source_offset` | **3 of 3** | the producer wrote none for that concept | that the source document lacks what the field asserts |
@ -195,7 +208,7 @@ carries its denominator.
| Limit | `120000` |
| Unit | `utf-8 bytes of emitted JSON` |
| Instrument | `okf_consume.measure (len of the ensure_ascii=False JSON encoding, utf-8)` |
| Known-positive | `docs/consumption-contract.md, encoded as a JSON string` at `13238` |
| Known-positive | `docs/consumption-contract.md, encoded as a JSON string` at `14455` |
The instrument reproduces the known-positive figure before any of its own
numbers are believed. Report what the run actually spent.
@ -206,10 +219,10 @@ finding requiring a decision — not something to retry with a narrower question
**Scaling. Cost tracks the question, not the corpus.** Measured on this
bundle at generation time, with the question `Hva sier veiledningen om krav?`: the delivered set
was **3 excerpts** costing **2125 utf-8 bytes of emitted JSON**,
against a whole bundle that would cost **2092** by the same instrument if
was **3 excerpts** costing **2289 utf-8 bytes of emitted JSON**,
against a whole bundle that would cost **2256** by the same instrument if
one answer delivered all 3 concepts — so that answer was about
**101.6 %** of the corpus. One question is one measurement: a
**101.5 %** of the corpus. One question is one measurement: a
different question moves `spent` and this figure with it.
**The breaking point, stated so it can be observed to have been passed.**

2
skills/okf-consume/references/README.md
View file

@ -34,7 +34,7 @@ Why each part is there:
checkout on one machine. The line strips that checkout's prefix and nothing
else, and a test holds the shipped `SKILL.md` to the generator's output with
exactly that prefix removed.
- **`okf check`** should report `conformant: 16 rules over 3 excerpts and 0
- **`okf check`** should report `conformant: 17 rules over 3 excerpts and 0
withheld entries, 0 findings` and exit 0.
The generated `name` is `b-golden-segmented-okf-v0-2-consume` while this

18
skills/okf-consume/references/example-payload.json
View file

@ -8,13 +8,13 @@
"unit": "utf-8 bytes of emitted JSON",
"instrument": "okf_consume.measure (len of the ensure_ascii=False JSON encoding, utf-8)",
"limit": 120000,
"spent": 2125,
"spent": 2289,
"known_positive": {
"case": "docs/consumption-contract.md, encoded as a JSON string",
"expected": 13238,
"measured": 13238,
"raw_bytes": 12893,
"encoding_delta": 345
"expected": 14455,
"measured": 14455,
"raw_bytes": 14083,
"encoding_delta": 372
}
},
"denominators": {
@ -42,6 +42,10 @@
"source_sha256": "6906ec0acbcfc246e825bda9863c716eb5611b465020e8204eeb448c32343f7d",
"source_lines": "[5, 8]",
"source_offset": "[94, 176]",
"parent": {
"concept_id": "veiledning",
"title": "Veiledning for eksempelbundel"
},
"text_sha256": "08447c690087834883a78fb9f40d474c68b2526ed57d0dafe01c4868db3c04f0",
"text": "\n## 1.1 Foerste krav\n\nEt krav som beskriver den foerste betingelsen i eksempelet.",
"rank": 1
@ -64,6 +68,10 @@
"source_sha256": "6906ec0acbcfc246e825bda9863c716eb5611b465020e8204eeb448c32343f7d",
"source_lines": "[9, 11]",
"source_offset": "[176, 253]",
"parent": {
"concept_id": "veiledning",
"title": "Veiledning for eksempelbundel"
},
"text_sha256": "4c181e1debb141d3cb1a7bd47249fdac1c0b2c5db1b47cf255ef8f04f83ec840",
"text": "\n## 1.2 Andre krav\n\nEt krav som beskriver den andre betingelsen i eksempelet.",
"rank": 2

89
src/llm_ingestion_okf/consume.py
View file

@ -50,7 +50,7 @@ import re
import sys
import unicodedata
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Literal
@ -287,6 +287,58 @@ class Concept:
locators: Mapping[str, str]
frontmatter: Mapping[str, str]
body: str
#: The concept this one's `parent` pointer resolves to, as its `concept_id`
#: and `title`, or `None`. Set by :func:`link_parents` over the whole set
#: and never by :func:`read_concept`: whether a pointer lands is a fact
#: about the bundle, and one file does not know its bundle.
parent: Mapping[str, str] | None = None
#: `True` when the concept carries a `parent` key that resolves to no
#: concept of its own document -- the third state, as for `sources`.
parent_unresolved: bool = False
#: The key a concept names its enclosing section under, and the key that
#: section answers to. Both are the segmented profiles' (`inbox.py` writes a
#: plan entry's `parent_id` as `parent:`, naming the other entry's `segment_id`).
PARENT_KEY = "parent"
SEGMENT_ID_KEY = "segment_id"
def link_parents(concepts: Sequence[Concept]) -> list[Concept]:
"""Every concept, with its `parent` pointer resolved or named unresolved.
A `parent` value is a `segment_id`, and a segment id is unique only inside
the plan of ONE document -- `p1` exists in every document of a
multi-document bundle -- so it is resolved among the concepts sharing this
concept's `source_file`, never across the bundle. An id two concepts of one
document claim resolves to neither, the rule `structure._lookup` applies: an
ambiguous pointer that silently picks a winner is worse than one reported
unresolved.
One pass over concepts already read; no file is opened again. A concept
carrying no `parent` comes back as it went in, which is what keeps every
payload of a bundle without the key byte-identical.
"""
claims: dict[tuple[str, str], list[Concept]] = {}
for concept in concepts:
segment_id = concept.frontmatter.get(SEGMENT_ID_KEY, "").strip()
if segment_id:
claims.setdefault((concept.source_file, segment_id), []).append(concept)
linked: list[Concept] = []
for concept in concepts:
pointer = concept.frontmatter.get(PARENT_KEY, "").strip()
if not pointer:
linked.append(concept)
continue
owners = claims.get((concept.source_file, pointer), [])
if len(owners) == 1 and owners[0] is not concept:
target = owners[0]
linked.append(
replace(concept, parent={"concept_id": target.concept_id, "title": target.title})
)
else:
linked.append(replace(concept, parent_unresolved=True))
return linked
def read_concept(path: Path, *, bundle_root: Path, root_bundle_id: str) -> Concept:
@ -617,14 +669,14 @@ KNOWN_POSITIVE_CASE = "docs/consumption-contract.md, encoded as a JSON string"
#: `measure()`'s own answer for that file. Vacuous ALONE -- which is why the
#: delta below exists.
KNOWN_POSITIVE_EXPECTED = 13_238
KNOWN_POSITIVE_EXPECTED = 14_455
#: The second, independent route. `wc -c` reports 12 893 raw bytes for the same
#: The second, independent route. `wc -c` reports 14 083 raw bytes for the same
#: file; the difference is this file's JSON quoting and escaping overhead. A
#: reader can derive it without running `measure()` at all, and it moves the
#: moment `measure()` changes what it counts -- which is what stops
#: `expected == measured` from proving nothing.
KNOWN_POSITIVE_ENCODING_DELTA = 345
KNOWN_POSITIVE_ENCODING_DELTA = 372
#: The two places that file can be, resolved in this order.
#:
@ -1585,6 +1637,13 @@ def excerpt_for(concept: Concept) -> dict[str, object] | None:
1 on three bundles while the model could not name it: the ranking found the
document and the delivery dropped the key.
Since K3-21, `parent` too, on the same terms: the concept that encloses
this one, as its `concept_id` and `title`, written only when the concept
names one and the pointer resolves (`link_parents`). Never the raw
`segment_id` the producer wrote -- a reader holding `p1977` can open
nothing -- and `parent_unresolved` where the pointer lands nowhere, the
`sources_unreadable` rule for the same reason.
Trailing whitespace is stripped per line: a spreadsheet render is padded to
hundreds of trailing spaces per line, and unstripped, most of a budget goes
on padding.
@ -1613,6 +1672,10 @@ def excerpt_for(concept: Concept) -> dict[str, object] | None:
# HAS an address and this reader could not decode it.
excerpt["sources_unreadable"] = True
excerpt.update(concept.locators)
if concept.parent is not None:
excerpt["parent"] = dict(concept.parent)
elif concept.parent_unresolved:
excerpt["parent_unresolved"] = True
excerpt["text_sha256"] = hashlib.sha256(text.encode("utf-8")).hexdigest()
excerpt["text"] = text
return excerpt
@ -1864,14 +1927,16 @@ def build_payload(
)
root_bundle_id = root_bundle_id_of(bundle_root, profile=profile)
concept_ids = enumerate_concepts(bundle_root, profile=profile)
concepts = [
read_concept(
bundle_root / f"{concept_id}{profile.paths.concept_suffix}",
bundle_root=bundle_root,
root_bundle_id=root_bundle_id,
)
for concept_id in concept_ids
]
concepts = link_parents(
[
read_concept(
bundle_root / f"{concept_id}{profile.paths.concept_suffix}",
bundle_root=bundle_root,
root_bundle_id=root_bundle_id,
)
for concept_id in concept_ids
]
)
# The bundle's OWN vocabulary, and the reason the rule is a set rather than
# a threshold: `pris` is a word here and `bila` is not, which is what
# separates a Norwegian compound from four coincidental characters. One

56
src/llm_ingestion_okf/contract_check.py
View file

@ -301,6 +301,61 @@ def rule_excerpt_named(ctx: Context) -> list[Finding]:
]
def rule_excerpt_parent(ctx: Context) -> list[Finding]:
"""SS 8.6: an excerpt's `parent`, when it carries one, is a pointer a
reader can follow.
Added 2026-09-11 (K3-21). Round 20 wrote `parent:` onto 675 of 710
heading-only sections of one standard as a `segment_id`, a value a reader
can open nothing with. The pre-pass now resolves it, and this rule holds
the resolved form: a `concept_id` and a `title`, naming a concept other
than the excerpt itself.
**The payload is its own denominator.** `excerpts` and `withheld` together
name every concept the pre-pass considered, which is every concept of the
bundle (SS 5.2), so a `parent.concept_id` in neither names nothing in the
bundle -- and the rule sees that without opening the bundle, the boundary
`rule_bundle_identity` keeps too. A payload lying about both lists at once
passes here and fails `denominator_identity` instead.
**Conditional, like SS 8.4's fields.** An excerpt with no `parent` meets
this rule as it always did. `parent_unresolved` is not a finding: SPEC SS
6.1, "Consumers MUST tolerate broken links", and a pointer named as
unresolved is the honest form of one.
"""
if not ctx.payload_is_mapping:
return []
excerpts = [_mapping(raw) for raw in _sequence(ctx.payload.get("excerpts"))]
considered = {_text(excerpt.get("concept_id")) for excerpt in excerpts} | {
_text(_mapping(raw).get("concept_id")) for raw in _sequence(ctx.payload.get("withheld"))
}
considered.discard("")
findings = []
for position, excerpt in enumerate(excerpts):
if "parent" not in excerpt:
continue
parent = excerpt.get("parent")
target = _text(_mapping(parent).get("concept_id"))
if not isinstance(parent, Mapping) or not target or not _text(parent.get("title")):
reason = "is not a `concept_id` and a `title`, so a reader can neither open nor cite it"
elif target == _text(excerpt.get("concept_id")):
reason = f"names the excerpt itself ({target!r})"
elif target not in considered:
reason = (
f"names {target!r}, which is in neither `excerpts` nor `withheld` and so is "
"no concept of this bundle"
)
else:
continue
findings.append(
Finding(
"parent_unfollowable",
f"excerpt {position}'s `parent` {reason} (SS 8.6)",
)
)
return findings
def rule_excerpt_states(ctx: Context) -> list[Finding]:
if not ctx.payload_is_mapping:
return []
@ -523,6 +578,7 @@ RULES: tuple[Callable[[Context], list[Finding]], ...] = (
rule_bundle_identity,
rule_excerpt_source_marking,
rule_excerpt_named,
rule_excerpt_parent,
rule_excerpt_states,
rule_denominator_identity,
rule_denominator_lists,

50
src/llm_ingestion_okf/inbox.py
View file

@ -52,6 +52,7 @@ from .segmentation import (
SegmentationPlan,
SegmentEntry,
assert_plan_applies,
heading_only,
observed_extractor_version,
slice_segments,
)
@ -101,6 +102,33 @@ def inbox_filename(slug: str, *, profile: BundleProfile = DEFAULT) -> str:
)
#: The words around the one link a heading-only body gains. SPEC SS 6.1: the
#: kind of relationship "is conveyed by the surrounding prose, not by the link
#: itself", so the line names the relation, in two words, because generic code
#: writes them into a body in the source's own language.
ENCLOSING_SECTION = "Enclosing section"
def _link_enclosing(body: str, parent: SegmentEntry, gate: Gate) -> str:
"""`body` plus one line linking the section that encloses it.
SPEC SS 5.1: "Lineage is expressed through links, not a dedicated field";
the `parent:` key stays beside it as a SS 4.1 extension. Bundle-relative
and absolute, SS 6.1's "recommended form", because the parent sits in
another directory and a relative link would count `..` across a layout the
next round may change.
The line carries the parent's title, which is document text, and joins a
body the gate has already judged -- so it is screened on its own, the rule
`_screened` applies to a `description`, and dropped rather than refused
when the gate would not persist it.
"""
line = _screened(gate, f"{ENCLOSING_SECTION}: [{parent.title}](/{parent.path})")
if line is None:
return body
return f"{body.rstrip(chr(10))}\n\n{line}\n"
def _normalize_body(text: str) -> str:
# LF-only with exactly one trailing newline is a byte-level guarantee, and
# dropped files legitimately arrive with CRLF — normalising is the
@ -820,6 +848,16 @@ def _render_segments(
extractor_version=observed_extractor_version(extractor_id),
)
sliced = slice_segments(text, plan)
# A body that is its heading alone gains ONE line linking the section its
# `parent` names. Only such a body: one holding text already has something
# to read, and the segmented goldens' declared parents are bodies holding
# text. Decided on the SLICE, the window the proposer's own predicate read.
enclosing = {entry.segment_id: entry for entry in plan.entries}
linked = {
entry.segment_id
for entry, body in sliced
if entry.parent_id is not None and heading_only(body)
}
decisions = [(entry, gate(body)) for entry, body in sliced]
refused = [
@ -837,11 +875,21 @@ def _render_segments(
if profile.index.facets is not None:
structure = derive_document_structure(decision.sanitized_text, source_file=source_file)
_validate_facets(structure, profile)
body = decision.sanitized_text
if entry.segment_id in linked:
# AFTER the structure is derived, and that order is the rule: read
# as body text, the link is a bundle-local target, so derivation
# would restate the `parent` relation as a `references` edge -- one
# relation under two kinds, the second rendered unresolved because
# nothing resolves the absolute form. Measured on the fixture
# before this order was chosen.
assert entry.parent_id is not None
body = _link_enclosing(body, enclosing[entry.parent_id], gate)
outputs.append(
(
entry.path,
render_inbox_concept(
decision.sanitized_text,
body,
okf_type=entry.okf_type,
# DECLARED by the adjudicator, never derived from the
# segment's own first line: the plan is the record of the

5
src/llm_ingestion_okf/propose.py
View file

@ -65,7 +65,7 @@ from .errors import IngestError
from .extract import OutlineMark, extract_text, strip_converter_attribute, xml_outline
from .extract import pdf_outline as extract_pdf_outline
from .materialize import reduce_to_id_grammar
from .segmentation import observed_extractor_version
from .segmentation import heading_only, observed_extractor_version
#: Stamped into every entry's `derived` list. The marker is what keeps a
#: proposal from being mistaken for the judgement the run path replays.
@ -1488,8 +1488,7 @@ def _link_shells(entries: list[dict[str, Any]], levels: Sequence[int], text: str
grew the excerpts past the budget.
"""
bodied = [
any(line.strip() and not line.startswith("#") for line in text[start:end].split("\n"))
for start, end in (entry["span"] for entry in entries)
not heading_only(text[start:end]) for start, end in (entry["span"] for entry in entries)
]
for index, entry in enumerate(entries):
if bodied[index]:

12
src/llm_ingestion_okf/segmentation.py
View file

@ -45,6 +45,18 @@ from typing import Any
from .errors import SegmentationError
from .materialize import reduce_to_id_grammar
def heading_only(text: str) -> bool:
"""`True` when no line of `text` holds anything but a markdown heading.
ONE predicate for two readers that must agree: the proposer gives such a
span a parent (`propose._link_shells`), and the door writes a link into
exactly those bodies. Both read the same window on the same extracted text,
so a second copy of the rule could only ever disagree with the first.
"""
return not any(line.strip() and not line.startswith("#") for line in text.split("\n"))
#: The top-level keys a plan payload must carry. Every one is required: a plan
#: missing its extractor identity would still parse, and would then be replayed
#: against an extraction nobody checked it against.

22
src/llm_ingestion_okf/skill.py
View file

@ -13,7 +13,8 @@ passes the UNFILLED template against a real payload (exit 0, 15 rules, 0
findings), and passes a skill built for a different bundle against this one's
payload. `contract_check.rule_bundle_identity` now compares the identity a
skill declares with the identity its payload declares, so all three measured
pairs are refused at exit 1 with one `bundle_mismatch` finding over 16 rules:
pairs are refused at exit 1 with one `bundle_mismatch` finding over 16 rules
(17 since K3-21's `parent_unfollowable`):
a skill against another bundle's payload, the unfilled template against a real
payload, and -- the arm an id comparison would miss -- a payload sharing the
skill's `bundle_id` at a foreign `ref`. The right pair is untouched at exit 0
@ -99,6 +100,7 @@ CONDITIONAL_FIELDS = (
"verified",
"req_number",
"sources",
"parent",
)
#: Tokens too short to carry a question. The same floor the pre-pass's own
@ -337,14 +339,16 @@ def render(
bundle_id = okf_consume.root_bundle_id_of(bundle_root, profile=profile)
ref = okf_consume.bundle_ref(bundle_root, profile=profile)
concept_ids = okf_consume.enumerate_concepts(bundle_root, profile=profile)
concepts = [
okf_consume.read_concept(
bundle_root / f"{concept_id}{profile.paths.concept_suffix}",
bundle_root=bundle_root,
root_bundle_id=bundle_id,
)
for concept_id in concept_ids
]
concepts = okf_consume.link_parents(
[
okf_consume.read_concept(
bundle_root / f"{concept_id}{profile.paths.concept_suffix}",
bundle_root=bundle_root,
root_bundle_id=bundle_id,
)
for concept_id in concept_ids
]
)
if not concepts:
raise SkillError(
f"{bundle_root} has an index but no concept under it; a skill for an "

7
tests/test_bundle_identity.py
View file

@ -156,12 +156,13 @@ def test_a_payload_declaring_no_identity_is_left_to_its_own_rule(tmp_path: Path)
def test_the_rule_count_is_the_denominator_the_report_quotes() -> None:
"""The nevner moves 15 -> 16, and every consumer quoting "15 rules" is
"""The nevner moved 15 -> 16 with this rule and 16 -> 17 with
`parent_unfollowable` (K3-21), and every consumer quoting the old number is
quoting a number that has changed."""
text, payload = instantiate(BUNDLE_A)
report = contract_check.check(text, payload)
assert report.rules_evaluated == len(contract_check.RULES) == 16
assert "16 rules" in report.render()
assert report.rules_evaluated == len(contract_check.RULES) == 17
assert "17 rules" in report.render()
def test_the_generator_writes_an_identity_the_checker_can_read() -> None:

7
tests/test_parent_reaches_reader.py
View file

@ -170,9 +170,10 @@ def test_a_pointer_resolves_inside_its_own_document(tmp_path: Path) -> None:
documents = {str(e["concept_id"]).split("/")[0] for e in carried}
assert documents == {"a", "b"}
for excerpt in carried:
assert str(excerpt["parent"]["concept_id"]).split("/")[0] == str(
excerpt["concept_id"]
).split("/")[0]
assert (
str(excerpt["parent"]["concept_id"]).split("/")[0]
== str(excerpt["concept_id"]).split("/")[0]
)
# --- The checker ---------------------------------------------------------------

18
tests/test_shell_parent.py
View file

@ -168,16 +168,22 @@ def test_a_build_without_the_flag_is_the_build_with_the_opt_out(tmp_path: Path)
assert files == sorted(p.relative_to(opted_out) for p in opted_out.rglob("*") if p.is_file())
for relative in files:
assert (default / relative).read_bytes() == (opted_out / relative).read_bytes()
# The flag adds one `parent:` line to each shell with an ancestor, and the
# index -- a projection of the frontmatter -- shows the same key as a facet
# on that shell's entry. Nothing else moves.
added = 0
# The flag adds one `parent:` line to each shell with an ancestor, and --
# since K3-21 -- one link line to that shell's body
# (`tests/test_parent_reaches_reader.py`). The index, a projection of the
# frontmatter, shows the same key as a facet on that shell's entry. Nothing
# else moves.
added = linked = 0
for relative in files:
before = (default / relative).read_text(encoding="utf-8").split("\n")
after = (flagged / relative).read_text(encoding="utf-8").split("\n")
text = (flagged / relative).read_text(encoding="utf-8")
if relative.name == "index.md":
after = text.split("\n")
assert [re.sub(r"parent: p\d+\?; ", "", line) for line in after] == before
continue
text, links = re.subn(r"\n\nEnclosing section: \[[^\]]+\]\(/[^)\s]+\)\n\Z", "\n", text)
linked += links
after = text.split("\n")
assert [line for line in after if not line.startswith("parent: ")] == before
added += sum(1 for line in after if line.startswith("parent: "))
assert added == sum(1 for ancestor in EXPECTED.values() if ancestor is not None)
assert added == linked == sum(1 for ancestor in EXPECTED.values() if ancestor is not None)