open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(examples): SEGMENTED_V1 golden fixture and decision record

Browse source
This commit is contained in:
Kjell Tore Guttormsen 2026-09-01 00:31:15 +02:00
commit 63565bde98
13 changed files with 416 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

145
docs/plan/concept-granularity.md Normal file
View file

@ -0,0 +1,145 @@
# Door B concept granularity: one document becomes many concepts
Record of what was decided and why, written after the work landed. The repo's
pattern: `cf65635` after structure derivation, `d35bcb2` after Door C.
## The defect this closes
`process_inbox` emitted exactly one flat concept per dropped file. OKF v0.2 §2
defines a concept as "a single unit of knowledge within a bundle" and a concept
ID as "the path of the concept's file within the bundle" — neither ties a
concept to a source file, and Appendix A presents v0.1 → v0.2 as a
de-monolithization. Door B therefore implemented the shape the SPEC names as
the one being migrated away from.
No conformance test caught it, and none could. §11 checks that every
non-reserved `.md` has parsable frontmatter with a non-empty `type`, so a
bundle of one giant concept is fully conformant. **Conformance is the floor,
not the proof.** The measurement that found it was a reading of §2's
terminology, not a count of keys.
## What was built
A new profile, `SEGMENTED_V1`, and a caller-owned, human-adjudicated
segmentation plan. `DEFAULT`, `STRICT_V1`, `STRUCTURED_V1` and `OKF_V0_2` are
byte-identical; their goldens did not move, which is asserted against the
baseline commit rather than assumed.
The splitting decision is made once, written down as data, adjudicated by a
human, and thereafter replayed deterministically. The run path still makes
**zero model calls**, and a from-scratch rebuild is byte-identical to an
incremental update.
| Piece | Where |
|---|---|
| Plan model, parser, cache key, slicing | `segmentation.py` |
| `SegmentationPolicy` + `SEGMENTED_V1` | `profiles.py` |
| Root frontmatter, per-concept mirroring, 1→N, collision gate, ownership, per-directory indexes | `inbox.py` |
| Proposer (outside the package) | `tools/okf_propose_segments.py` |
| Golden | `examples/ingest-golden-segmented/` |
## The identity decision, and what it costs
Order `…2527032751` settled the form: **(c) a root-frontmatter bundle
identifier consumers join on**, combined with **(a) `sources[].resource` as an
absolute URL wherever the source has a stable one**. (c) carries identity, (a)
carries provenance.
**No step implemented form (a), and that is the correct outcome.** It is
already emitted, at `materialize.py:205,214`, from the manifest locator —
Door A's surface. Door B ingests dropped files, which have no stable URL by
construction.
**S4b is resolved as ONE branch.** The root index is the SOURCE of `bundle_id`
— the caller supplies it exactly once, so D5 stays intact — and every concept
MIRRORS it. Two bundles built from identical inputs therefore hold concepts
whose paths **collide by construction** and whose identity values are disjoint.
The collision is the expected behaviour under form (c), not a defect.
**Two identity mechanisms, deliberately.** Within a bundle, Door C keys on a
content hash; across bundles, identity is this caller-assigned `bundle_id`. A
bundle is a collection the caller delimits, so hashing it would hand it a new
identity every time a document was added. The cost is stated rather than
hidden: a consumer meets both mechanisms.
## Decisions worth their own line
- **The capability is `profile.segmentation is not None`, never
`IndexPolicy.per_directory`.** `STRICT_V1` already sets `per_directory=True`
while Door B ignores it, so keying there would have changed a shipped
profile's output and broken its byte-stability pin.
- **Offsets index the CANONICAL EXTRACTED TEXT, never source bytes.** A `.csv`
is re-rendered before it becomes a concept body, and a `.pdf` has no
sliceable prose at all. The golden and the `.csv` fixture both pin this; on a
`.md` fixture bytes and text are identical and the assertion would pass for
the wrong reason.
- **The adjudication cache key is the triple `(source_sha256, extractor_id,
extractor_version)`, not the hash alone** (S5b). Source bytes cannot see an
extractor swap or a version bump, either of which silently invalidates every
stored offset while the hash still matches. Door B observes the hash and the
extractor id; the parser *version* is the caller's to declare, for the same
reason `bundle_id`'s value is — the library names the key, whoever knows the
value owns it.
- **All N segments are gated before any is written.** Gating and writing one at
a time would leave a half-screened document on disk the moment segment 3 of 5
quarantines. A refusal is reported once, for the document.
- **Ownership asks "which concepts are owned by source_file X?"**, recursively,
and only under the capability. Keyed on the concept filename, a round naming
fewer segments orphaned the rest: they survived incrementally, were absent
from a scratch rebuild, and the two diverged silently.
- **A retired directory loses its index too.** A scratch rebuild writes an
index only where a concept lives, so an orphaned one is a one-sided
difference. Found by S7 on 2026-09-01, once its fixture was made to actually
retire a segment — with round 2 a superset of round 1, S7 stayed green with
retirement disabled entirely.
- **Frontmatter values are FLOW form, never block.** `source_offset: [12, 48]`.
This library's parser round-trips a flow value as an opaque string and cannot
read a block one at all, so emitting block would produce bundles we cannot
read back.
- **Index ordering has one seam, `_index_sort_key`.** A consumer-controlled
ordering is then a parameter, not a refactor.
- **The proposer proposes; a human adjudicates.** Every entry it emits carries
`PROPOSED` and the rule that produced it. Research measured the reason: the
OKF reference agent's four semantic gates need a model this run path does not
have, and heading derivation is inert on most of the K2 corpus — 23 of 33
PDFs carry no outline, 95 % of outline entries are CAD metadata, and the
scoreable denominator is **one document**.
## Answers to the brief's open questions
- *Where does the plan live?* At a caller-supplied path, parsed by the caller
and passed in. The library loads nothing from a fixed location — the same
division D5 makes for values.
- *Is `source_offset` readable?* Not this round. It is write-only until the
structured reader (D1b) lands. Nothing built here has to be torn out when it
does.
- *Does a plan cover a whole run?* No — one plan, selected by content hash.
Every other dropped file in the same run keeps today's one-concept rule.
## Known gaps, stated rather than left to be discovered
1. **`derived` marks a declared title as inferred.** A segmented concept's
`title` is DECLARED by the adjudicator, but structure derivation runs over
the segment body and adds `title` to `derived`. The marker is therefore
wrong in the one direction this repo cares about — a consumer that distrusts
derived fields will distrust a fact the producer stated. The golden pins
today's behaviour; the fix belongs with the code that composes the two.
2. **A plan matching no dropped file is silent.** A typo'd `source_sha256`
segments nothing and reports nothing, which is the silent skip this library
refuses elsewhere.
3. **`status: deprecated` has no producer.** `status` is named as a facet key
and nothing writes it, so a deprecated parent cannot carry one.
4. **An unchanged round still rewrites.** `write_bytes` writes
unconditionally; the bytes are identical, so no contract is broken, but
"an unchanged round writes nothing" is not true today.
5. **Six new error codes live in `tests/test_segmentation.py`, not
`tests/test_error_codes.py`.** The baseline pin forbade editing the
registry file. Migrate once the pin is released.
## Relation to the two orders
- `…2527032751` (identity) — executed. Form (c) built, form (a) measured as
already present at Door A.
- `…3790625395` (index ordering) — untouched by design. `_index_sort_key` is
the single named seat a consumer-controlled ordering would occupy, so that
decision stays a parameter change rather than a refactor.