llm-ingestion-okf/docs/plan/concept-granularity.md

Door B concept granularity: one document becomes many concepts

Record of what was decided and why, written after the work landed. The repo's pattern: cf65635 after structure derivation, d35bcb2 after Door C.

The defect this closes

process_inbox emitted exactly one flat concept per dropped file. OKF v0.2 §2 defines a concept as "a single unit of knowledge within a bundle" and a concept ID as "the path of the concept's file within the bundle" — neither ties a concept to a source file, and Appendix A presents v0.1 → v0.2 as a de-monolithization. Door B therefore implemented the shape the SPEC names as the one being migrated away from.

No conformance test caught it, and none could. §11 checks that every non-reserved .md has parsable frontmatter with a non-empty type, so a bundle of one giant concept is fully conformant. Conformance is the floor, not the proof. The measurement that found it was a reading of §2's terminology, not a count of keys.

What was built

A new profile, SEGMENTED_V1, and a caller-owned, human-adjudicated segmentation plan. DEFAULT, STRICT_V1, STRUCTURED_V1 and OKF_V0_2 are byte-identical; their goldens did not move, which is asserted against the baseline commit rather than assumed.

The splitting decision is made once, written down as data, adjudicated by a human, and thereafter replayed deterministically. The run path still makes zero model calls, and a from-scratch rebuild is byte-identical to an incremental update.

Piece	Where
Plan model, parser, cache key, slicing	segmentation.py
SegmentationPolicy + SEGMENTED_V1	profiles.py
Root frontmatter, per-concept mirroring, 1→N, collision gate, ownership, per-directory indexes	inbox.py
Proposer (outside the package)	tools/okf_propose_segments.py
Golden	examples/ingest-golden-segmented/

The identity decision, and what it costs

Order …2527032751 settled the form: (c) a root-frontmatter bundle identifier consumers join on, combined with (a) sources[].resource as an absolute URL wherever the source has a stable one. (c) carries identity, (a) carries provenance.

No step implemented form (a), and that is the correct outcome. It is already emitted, at materialize.py:205,214, from the manifest locator — Door A's surface. Door B ingests dropped files, which have no stable URL by construction.

S4b is resolved as ONE branch. The root index is the SOURCE of bundle_id — the caller supplies it exactly once, so D5 stays intact — and every concept MIRRORS it. Two bundles built from identical inputs therefore hold concepts whose paths collide by construction and whose identity values are disjoint. The collision is the expected behaviour under form (c), not a defect.

Two identity mechanisms, deliberately. Within a bundle, Door C keys on a content hash; across bundles, identity is this caller-assigned bundle_id. A bundle is a collection the caller delimits, so hashing it would hand it a new identity every time a document was added. The cost is stated rather than hidden: a consumer meets both mechanisms.

Decisions worth their own line

• The capability is profile.segmentation is not None, never IndexPolicy.per_directory. STRICT_V1 already sets per_directory=True while Door B ignores it, so keying there would have changed a shipped profile's output and broken its byte-stability pin.
• Offsets index the CANONICAL EXTRACTED TEXT, never source bytes. A .csv is re-rendered before it becomes a concept body, and a .pdf has no sliceable prose at all. The golden and the .csv fixture both pin this; on a .md fixture bytes and text are identical and the assertion would pass for the wrong reason.
• The adjudication cache key is the triple (source_sha256, extractor_id, extractor_version), not the hash alone (S5b). Source bytes cannot see an extractor swap or a version bump, either of which silently invalidates every stored offset while the hash still matches. Door B observes the hash and the extractor id; the parser version is the caller's to declare, for the same reason bundle_id's value is — the library names the key, whoever knows the value owns it.
• All N segments are gated before any is written. Gating and writing one at a time would leave a half-screened document on disk the moment segment 3 of 5 quarantines. A refusal is reported once, for the document.
• Ownership asks "which concepts are owned by source_file X?", recursively, and only under the capability. Keyed on the concept filename, a round naming fewer segments orphaned the rest: they survived incrementally, were absent from a scratch rebuild, and the two diverged silently.
• A retired directory loses its index too. A scratch rebuild writes an index only where a concept lives, so an orphaned one is a one-sided difference. Found by S7 on 2026-09-01, once its fixture was made to actually retire a segment — with round 2 a superset of round 1, S7 stayed green with retirement disabled entirely.
• Frontmatter values are FLOW form, never block. source_offset: [12, 48]. This library's parser round-trips a flow value as an opaque string and cannot read a block one at all, so emitting block would produce bundles we cannot read back.
• Index ordering has one seam, _index_sort_key. A consumer-controlled ordering is then a parameter, not a refactor.
• The proposer proposes; a human adjudicates. Every entry it emits carries PROPOSED and the rule that produced it. Research measured the reason: the OKF reference agent's four semantic gates need a model this run path does not have, and heading derivation is inert on most of the K2 corpus — 23 of 33 PDFs carry no outline, 95 % of outline entries are CAD metadata, and the scoreable denominator is one document.

Answers to the brief's open questions

• Where does the plan live? At a caller-supplied path, parsed by the caller and passed in. The library loads nothing from a fixed location — the same division D5 makes for values.
• Is source_offset readable? Not this round. It is write-only until the structured reader (D1b) lands. Nothing built here has to be torn out when it does.
• Does a plan cover a whole run? No — one plan, selected by content hash. Every other dropped file in the same run keeps today's one-concept rule.

Known gaps, stated rather than left to be discovered

1. derived marks a declared title as inferred. Closed 2026-09-01. A segmented concept's title is DECLARED by the adjudicator, but structure derivation ran over the segment body and added title to derived. The marker pointed the wrong way in the one direction this repo cares about — a consumer that distrusts derived fields would have distrusted a fact the producer stated. title is now dropped from derived when a segment is present, and only then. The fix is scoped to title alone and pinned that way: an inferred number still marks itself on a segment, because nothing about segmentation makes a derived document number declared, and blunting the marker would be the opposite defect. The SEGMENTED_V1 golden moved with it; the four others were measured byte-identical to baseline.
2. A plan matching no dropped file is silent. Closed 2026-09-01. A typo'd source_sha256 segmented nothing and reported nothing, which is the silent skip this library refuses elsewhere. process_inbox now refuses fail-fast with segmentation_plan_unmatched before any disk mutation. Two sharper questions had to be settled to make the refusal correct rather than merely loud. It asks whether a covering plan was found, not whether every file was examined, so an unreadable drop cannot suppress it. And coverage is recorded at selection, not after path validation: the first cut asked the later question, and a plan whose hash matched but whose entry paths were then refused looked identical to a plan that matched nothing — the operator would have been sent to check a hash that was already right. An existing collision test caught it; the case now has its own test, verified red against the earlier form.
3. status: deprecated has no producer. status is named as a facet key and nothing writes it, so a deprecated parent cannot carry one.
4. An unchanged round still rewrites. write_bytes writes unconditionally; the bytes are identical, so no contract is broken, but "an unchanged round writes nothing" is not true today.
5. Six new error codes live in tests/test_segmentation.py, not tests/test_error_codes.py. Closed 2026-09-01. The baseline pin forbade editing the registry file while it was the voyage's verification mechanism. The gate run above released it, so the registry convention — one test per code, this file IS the conformance suite — applies again, and the codes were migrated. A conformance suite split across two files stops being one quietly, which is why this was carried as an obligation rather than a preference.

The end-to-end gate run, and its one failure

/trekexecute verifies per-step manifests; the plan's Verification section holds twelve gates that cross step boundaries and none of them had been run. They were run 2026-09-01 against the tree at 63565bd with the baseline at 770d8d4, before any new commit — measuring the tree afterwards would have let the gates score the very work they exist to bound.

Denominator: twelve gates, eleven passed, one failed. The failure is gate 7, "no existing test weakened": git diff against baseline returns one hunk, in tests/test_profile.py::test_a_profile_is_assembled_from_its_policies, whose closed field set on BundleProfile gained "segmentation". The plan's assumption A4 — that no existing test file would be edited — was falsified during execution and the edit was ratified by the operator. The gate's mechanism is byte-equality against baseline; the property it stands for is that no assertion was loosened, and that property was measured directly rather than inferred: removing "segmentation" from the set makes the test fail. The assertion is a closed set with one more legitimate member. The gate stays recorded as failed rather than rewritten to pass, because a gate edited until it is green has stopped answering the question it was written for.

Two of the plan's mandatory known-positive controls fired, and both would otherwise have produced a false absence. git cat-file -e "$SHA:examples/..." lost its leading e to the zsh :e history modifier and reported the wrong error for every path; an unquoted for f in $NEW did not word-split under zsh and collapsed ten separate checks into one that could only ever fail. Neither was visible in the result — each looked like an answer. The remaining controls confirmed that git diff can find a change, comm can report a missing test id, the sed count pattern can match, pytest -rs prints SKIPPED against an injected mark, and grep returns 2 on a path error and 0 on a planted match.

Measured across the suite: 853 tests collected against a baseline of 728, 125 of them in the ten new modules, none skipped, no baseline test id missing, comm's stderr empty, four golden directories byte-identical to baseline, and the run path free of any model or subprocess reference across thirteen source files.

Relation to the two orders

• …2527032751 (identity) — executed. Form (c) built, form (a) measured as already present at Door A.
• …3790625395 (index ordering) — untouched by design. _index_sort_key is the single named seat a consumer-controlled ordering would occupy, so that decision stays a parameter change rather than a refactor.