llm-ingestion-okf/docs/plan/concept-granularity.md

Door B concept granularity: one document becomes many concepts

Record of what was decided and why, written after the work landed. The repo's pattern: cf65635 after structure derivation, d35bcb2 after Door C.

The defect this closes

process_inbox emitted exactly one flat concept per dropped file. OKF v0.2 §2 defines a concept as "a single unit of knowledge within a bundle" and a concept ID as "the path of the concept's file within the bundle" — neither ties a concept to a source file, and Appendix A presents v0.1 → v0.2 as a de-monolithization. Door B therefore implemented the shape the SPEC names as the one being migrated away from.

No conformance test caught it, and none could. §11 checks that every non-reserved .md has parsable frontmatter with a non-empty type, so a bundle of one giant concept is fully conformant. Conformance is the floor, not the proof. The measurement that found it was a reading of §2's terminology, not a count of keys.

What was built

A new profile, SEGMENTED_V1, and a caller-owned, human-adjudicated segmentation plan. DEFAULT, STRICT_V1, STRUCTURED_V1 and OKF_V0_2 are byte-identical; their goldens did not move, which is asserted against the baseline commit rather than assumed.

The splitting decision is made once, written down as data, adjudicated by a human, and thereafter replayed deterministically. The run path still makes zero model calls, and a from-scratch rebuild is byte-identical to an incremental update.

Piece	Where
Plan model, parser, cache key, slicing	segmentation.py
SegmentationPolicy + SEGMENTED_V1	profiles.py
Root frontmatter, per-concept mirroring, 1→N, collision gate, ownership, per-directory indexes	inbox.py
Proposer (outside the package)	tools/okf_propose_segments.py
Golden	examples/ingest-golden-segmented/

The identity decision, and what it costs

Order …2527032751 settled the form: (c) a root-frontmatter bundle identifier consumers join on, combined with (a) sources[].resource as an absolute URL wherever the source has a stable one. (c) carries identity, (a) carries provenance.

No step implemented form (a), and that is the correct outcome. It is already emitted, at materialize.py:205,214, from the manifest locator — Door A's surface. Door B ingests dropped files, which have no stable URL by construction.

S4b is resolved as ONE branch. The root index is the SOURCE of bundle_id — the caller supplies it exactly once, so D5 stays intact — and every concept MIRRORS it. Two bundles built from identical inputs therefore hold concepts whose paths collide by construction and whose identity values are disjoint. The collision is the expected behaviour under form (c), not a defect.

Two identity mechanisms, deliberately. Within a bundle, Door C keys on a content hash; across bundles, identity is this caller-assigned bundle_id. A bundle is a collection the caller delimits, so hashing it would hand it a new identity every time a document was added. The cost is stated rather than hidden: a consumer meets both mechanisms.

Decisions worth their own line

• The capability is profile.segmentation is not None, never IndexPolicy.per_directory. STRICT_V1 already sets per_directory=True while Door B ignores it, so keying there would have changed a shipped profile's output and broken its byte-stability pin.
• Offsets index the CANONICAL EXTRACTED TEXT, never source bytes. A .csv is re-rendered before it becomes a concept body, and a .pdf has no sliceable prose at all. The golden and the .csv fixture both pin this; on a .md fixture bytes and text are identical and the assertion would pass for the wrong reason.
• The adjudication cache key is the triple (source_sha256, extractor_id, extractor_version), not the hash alone (S5b). Source bytes cannot see an extractor swap or a version bump, either of which silently invalidates every stored offset while the hash still matches. Door B observes the hash and the extractor id; the parser version is the caller's to declare, for the same reason bundle_id's value is — the library names the key, whoever knows the value owns it.
• All N segments are gated before any is written. Gating and writing one at a time would leave a half-screened document on disk the moment segment 3 of 5 quarantines. A refusal is reported once, for the document.
• Ownership asks "which concepts are owned by source_file X?", recursively, and only under the capability. Keyed on the concept filename, a round naming fewer segments orphaned the rest: they survived incrementally, were absent from a scratch rebuild, and the two diverged silently.
• A retired directory loses its index too. A scratch rebuild writes an index only where a concept lives, so an orphaned one is a one-sided difference. Found by S7 on 2026-09-01, once its fixture was made to actually retire a segment — with round 2 a superset of round 1, S7 stayed green with retirement disabled entirely.
• Frontmatter values are FLOW form, never block. source_offset: [12, 48]. This library's parser round-trips a flow value as an opaque string and cannot read a block one at all, so emitting block would produce bundles we cannot read back.
• Index ordering has one seam, _index_sort_key. A consumer-controlled ordering is then a parameter, not a refactor.
• The proposer proposes; a human adjudicates. Every entry it emits carries PROPOSED and the rule that produced it. Research measured the reason: the OKF reference agent's four semantic gates need a model this run path does not have, and heading derivation is inert on most of the K2 corpus — 23 of 33 PDFs carry no outline, 95 % of outline entries are CAD metadata, and the scoreable denominator is one document.

Answers to the brief's open questions

• Where does the plan live? At a caller-supplied path, parsed by the caller and passed in. The library loads nothing from a fixed location — the same division D5 makes for values.
• Is source_offset readable? Not this round. It is write-only until the structured reader (D1b) lands. Nothing built here has to be torn out when it does.
• Does a plan cover a whole run? No — one plan, selected by content hash. Every other dropped file in the same run keeps today's one-concept rule.

Known gaps, stated rather than left to be discovered

1. derived marks a declared title as inferred. A segmented concept's title is DECLARED by the adjudicator, but structure derivation runs over the segment body and adds title to derived. The marker is therefore wrong in the one direction this repo cares about — a consumer that distrusts derived fields will distrust a fact the producer stated. The golden pins today's behaviour; the fix belongs with the code that composes the two.
2. A plan matching no dropped file is silent. A typo'd source_sha256 segments nothing and reports nothing, which is the silent skip this library refuses elsewhere.
3. status: deprecated has no producer. status is named as a facet key and nothing writes it, so a deprecated parent cannot carry one.
4. An unchanged round still rewrites. write_bytes writes unconditionally; the bytes are identical, so no contract is broken, but "an unchanged round writes nothing" is not true today.
5. Six new error codes live in tests/test_segmentation.py, not tests/test_error_codes.py. The baseline pin forbade editing the registry file. Migrate once the pin is released.

The end-to-end gate run, and its one failure

/trekexecute verifies per-step manifests; the plan's Verification section holds twelve gates that cross step boundaries and none of them had been run. They were run 2026-09-01 against the tree at 63565bd with the baseline at 770d8d4, before any new commit — measuring the tree afterwards would have let the gates score the very work they exist to bound.

Denominator: twelve gates, eleven passed, one failed. The failure is gate 7, "no existing test weakened": git diff against baseline returns one hunk, in tests/test_profile.py::test_a_profile_is_assembled_from_its_policies, whose closed field set on BundleProfile gained "segmentation". The plan's assumption A4 — that no existing test file would be edited — was falsified during execution and the edit was ratified by the operator. The gate's mechanism is byte-equality against baseline; the property it stands for is that no assertion was loosened, and that property was measured directly rather than inferred: removing "segmentation" from the set makes the test fail. The assertion is a closed set with one more legitimate member. The gate stays recorded as failed rather than rewritten to pass, because a gate edited until it is green has stopped answering the question it was written for.

Two of the plan's mandatory known-positive controls fired, and both would otherwise have produced a false absence. git cat-file -e "$SHA:examples/..." lost its leading e to the zsh :e history modifier and reported the wrong error for every path; an unquoted for f in $NEW did not word-split under zsh and collapsed ten separate checks into one that could only ever fail. Neither was visible in the result — each looked like an answer. The remaining controls confirmed that git diff can find a change, comm can report a missing test id, the sed count pattern can match, pytest -rs prints SKIPPED against an injected mark, and grep returns 2 on a path error and 0 on a planted match.

Measured across the suite: 853 tests collected against a baseline of 728, 125 of them in the ten new modules, none skipped, no baseline test id missing, comm's stderr empty, four golden directories byte-identical to baseline, and the run path free of any model or subprocess reference across thirteen source files.

Relation to the two orders

• …2527032751 (identity) — executed. Form (c) built, form (a) measured as already present at Door A.
• …3790625395 (index ordering) — untouched by design. _index_sort_key is the single named seat a consumer-controlled ordering would occupy, so that decision stays a parameter change rather than a refactor.