llm-ingestion-okf/docs/plan/concept-granularity.md

# Door B concept granularity: one document becomes many concepts

Record of what was decided and why, written after the work landed. The repo's
pattern: `cf65635` after structure derivation, `d35bcb2` after Door C.

## The defect this closes

`process_inbox` emitted exactly one flat concept per dropped file. OKF v0.2 §2
defines a concept as "a single unit of knowledge within a bundle" and a concept
ID as "the path of the concept's file within the bundle" — neither ties a
concept to a source file, and Appendix A presents v0.1 → v0.2 as a
de-monolithization. Door B therefore implemented the shape the SPEC names as
the one being migrated away from.

No conformance test caught it, and none could. §11 checks that every
non-reserved `.md` has parsable frontmatter with a non-empty `type`, so a
bundle of one giant concept is fully conformant. **Conformance is the floor,
not the proof.** The measurement that found it was a reading of §2's
terminology, not a count of keys.

## What was built

A new profile, `SEGMENTED_V1`, and a caller-owned, human-adjudicated
segmentation plan. `DEFAULT`, `STRICT_V1`, `STRUCTURED_V1` and `OKF_V0_2` are
byte-identical; their goldens did not move, which is asserted against the
baseline commit rather than assumed.

The splitting decision is made once, written down as data, adjudicated by a
human, and thereafter replayed deterministically. The run path still makes
**zero model calls**, and a from-scratch rebuild is byte-identical to an
incremental update.

| Piece | Where |
|---|---|
| Plan model, parser, cache key, slicing | `segmentation.py` |
| `SegmentationPolicy` + `SEGMENTED_V1` | `profiles.py` |
| Root frontmatter, per-concept mirroring, 1→N, collision gate, ownership, per-directory indexes | `inbox.py` |
| Proposer (outside the package) | `tools/okf_propose_segments.py` |
| Golden | `examples/ingest-golden-segmented/` |

## The identity decision, and what it costs

Order `…2527032751` settled the form: **(c) a root-frontmatter bundle
identifier consumers join on**, combined with **(a) `sources[].resource` as an
absolute URL wherever the source has a stable one**. (c) carries identity, (a)
carries provenance.

**No step implemented form (a), and that is the correct outcome.** It is
already emitted, at `materialize.py:205,214`, from the manifest locator —
Door A's surface. Door B ingests dropped files, which have no stable URL by
construction.

**S4b is resolved as ONE branch.** The root index is the SOURCE of `bundle_id`
— the caller supplies it exactly once, so D5 stays intact — and every concept
MIRRORS it. Two bundles built from identical inputs therefore hold concepts
whose paths **collide by construction** and whose identity values are disjoint.
The collision is the expected behaviour under form (c), not a defect.

**Two identity mechanisms, deliberately.** Within a bundle, Door C keys on a
content hash; across bundles, identity is this caller-assigned `bundle_id`. A
bundle is a collection the caller delimits, so hashing it would hand it a new
identity every time a document was added. The cost is stated rather than
hidden: a consumer meets both mechanisms.

## Decisions worth their own line

- **The capability is `profile.segmentation is not None`, never
  `IndexPolicy.per_directory`.** `STRICT_V1` already sets `per_directory=True`
  while Door B ignores it, so keying there would have changed a shipped
  profile's output and broken its byte-stability pin.
- **Offsets index the CANONICAL EXTRACTED TEXT, never source bytes.** A `.csv`
  is re-rendered before it becomes a concept body, and a `.pdf` has no
  sliceable prose at all. The golden and the `.csv` fixture both pin this; on a
  `.md` fixture bytes and text are identical and the assertion would pass for
  the wrong reason.
- **The adjudication cache key is the triple `(source_sha256, extractor_id,
  extractor_version)`, not the hash alone** (S5b). Source bytes cannot see an
  extractor swap or a version bump, either of which silently invalidates every
  stored offset while the hash still matches. Door B observes the hash and the
  extractor id; the parser *version* is the caller's to declare, for the same
  reason `bundle_id`'s value is — the library names the key, whoever knows the
  value owns it.
- **All N segments are gated before any is written.** Gating and writing one at
  a time would leave a half-screened document on disk the moment segment 3 of 5
  quarantines. A refusal is reported once, for the document.
- **Ownership asks "which concepts are owned by source_file X?"**, recursively,
  and only under the capability. Keyed on the concept filename, a round naming
  fewer segments orphaned the rest: they survived incrementally, were absent
  from a scratch rebuild, and the two diverged silently.
- **A retired directory loses its index too.** A scratch rebuild writes an
  index only where a concept lives, so an orphaned one is a one-sided
  difference. Found by S7 on 2026-09-01, once its fixture was made to actually
  retire a segment — with round 2 a superset of round 1, S7 stayed green with
  retirement disabled entirely.
- **Frontmatter values are FLOW form, never block.** `source_offset: [12, 48]`.
  This library's parser round-trips a flow value as an opaque string and cannot
  read a block one at all, so emitting block would produce bundles we cannot
  read back.
- **Index ordering has one seam, `_index_sort_key`.** A consumer-controlled
  ordering is then a parameter, not a refactor.
- **The proposer proposes; a human adjudicates.** Every entry it emits carries
  `PROPOSED` and the rule that produced it. Research measured the reason: the
  OKF reference agent's four semantic gates need a model this run path does not
  have, and heading derivation is inert on most of the K2 corpus — 23 of 33
  PDFs carry no outline, 95 % of outline entries are CAD metadata, and the
  scoreable denominator is **one document**.

## Answers to the brief's open questions

- *Where does the plan live?* At a caller-supplied path, parsed by the caller
  and passed in. The library loads nothing from a fixed location — the same
  division D5 makes for values.
- *Is `source_offset` readable?* Not this round. It is write-only until the
  structured reader (D1b) lands. Nothing built here has to be torn out when it
  does.
- *Does a plan cover a whole run?* No — one plan, selected by content hash.
  Every other dropped file in the same run keeps today's one-concept rule.

## Known gaps, stated rather than left to be discovered

1. ~~**`derived` marks a declared title as inferred.**~~ **Closed 2026-09-01.**
   A segmented concept's `title` is DECLARED by the adjudicator, but structure
   derivation ran over the segment body and added `title` to `derived`. The
   marker pointed the wrong way in the one direction this repo cares about — a
   consumer that distrusts derived fields would have distrusted a fact the
   producer stated. `title` is now dropped from `derived` when a segment is
   present, and only then. The fix is scoped to `title` alone and pinned that
   way: an inferred `number` still marks itself on a segment, because nothing
   about segmentation makes a derived document number declared, and blunting
   the marker would be the opposite defect. The `SEGMENTED_V1` golden moved
   with it; the four others were measured byte-identical to baseline.
2. ~~**A plan matching no dropped file is silent.**~~ **Closed 2026-09-01.**
   A typo'd `source_sha256` segmented nothing and reported nothing, which is
   the silent skip this library refuses elsewhere. `process_inbox` now refuses
   fail-fast with `segmentation_plan_unmatched` before any disk mutation. Two
   sharper questions had to be settled to make the refusal correct rather than
   merely loud. It asks whether a covering plan was **found**, not whether
   every file was examined, so an unreadable drop cannot suppress it. And
   coverage is recorded at **selection**, not after path validation: the first
   cut asked the later question, and a plan whose hash matched but whose entry
   paths were then refused looked identical to a plan that matched nothing —
   the operator would have been sent to check a hash that was already right.
   An existing collision test caught it; the case now has its own test,
   verified red against the earlier form.
3. **`status: deprecated` has no producer.** `status` is named as a facet key
   and nothing writes it, so a deprecated parent cannot carry one.
4. **An unchanged round still rewrites.** `write_bytes` writes
   unconditionally; the bytes are identical, so no contract is broken, but
   "an unchanged round writes nothing" is not true today.
5. ~~**Six new error codes live in `tests/test_segmentation.py`, not
   `tests/test_error_codes.py`.**~~ **Closed 2026-09-01.** The baseline pin
   forbade editing the registry file while it was the voyage's verification
   mechanism. The gate run above released it, so the registry convention — one
   test per code, this file IS the conformance suite — applies again, and the
   codes were migrated. A conformance suite split across two files stops being
   one quietly, which is why this was carried as an obligation rather than a
   preference.

## The end-to-end gate run, and its one failure

`/trekexecute` verifies per-step manifests; the plan's `Verification` section
holds twelve gates that cross step boundaries and none of them had been run.
They were run 2026-09-01 against the tree at `63565bd` with the baseline at
`770d8d4`, before any new commit — measuring the tree afterwards would have let
the gates score the very work they exist to bound.

**Denominator: twelve gates, eleven passed, one failed.** The failure is gate 7,
"no existing test weakened": `git diff` against baseline returns one hunk, in
`tests/test_profile.py::test_a_profile_is_assembled_from_its_policies`, whose
closed field set on `BundleProfile` gained `"segmentation"`. The plan's
assumption A4 — that no existing test file would be edited — was falsified during
execution and the edit was ratified by the operator. The gate's mechanism is
byte-equality against baseline; the property it stands for is that no assertion
was loosened, and that property was measured directly rather than inferred:
removing `"segmentation"` from the set makes the test fail. The assertion is a
closed set with one more legitimate member. The gate stays recorded as failed
rather than rewritten to pass, because a gate edited until it is green has
stopped answering the question it was written for.

Two of the plan's mandatory known-positive controls fired, and both would
otherwise have produced a false absence. `git cat-file -e "$SHA:examples/..."`
lost its leading `e` to the zsh `:e` history modifier and reported the wrong
error for every path; an unquoted `for f in $NEW` did not word-split under zsh
and collapsed ten separate checks into one that could only ever fail. Neither
was visible in the result — each looked like an answer. The remaining controls
confirmed that `git diff` can find a change, `comm` can report a missing test id,
the `sed` count pattern can match, `pytest -rs` prints `SKIPPED` against an
injected mark, and `grep` returns 2 on a path error and 0 on a planted match.

Measured across the suite: 853 tests collected against a baseline of 728, 125 of
them in the ten new modules, none skipped, no baseline test id missing, `comm`'s
stderr empty, four golden directories byte-identical to baseline, and the run
path free of any model or subprocess reference across thirteen source files.

## Relation to the two orders

- `…2527032751` (identity) — executed. Form (c) built, form (a) measured as
  already present at Door A.
- `…3790625395` (index ordering) — untouched by design. `_index_sort_key` is
  the single named seat a consumer-controlled ordering would occupy, so that
  decision stays a parameter change rather than a refactor.