feat(inbox): point every concept at the document it came from, with a locator per format

A concept named its source file by basename and, when segmented, carried a
`source_offset` into the text THIS LIBRARY extracted. Following that pointer
needed the corpus directory, the extractor and its exact transitive version --
none of which the bundle carries. Hand-walked on a real K2 concept: six steps,
four of them requiring knowledge from outside the bundle, to learn that a
requirement sits on pages 12-13 of a 20-page document.

The address is spec's: `sources: [{ resource, title }]`, where `resource` is
the dropped file's inbox-relative path (SPEC v0.2 5.1:303-306 -- "an absolute
URL, a bundle-relative path, or a path into a `references/` subdirectory").
The locator is ours, and it has to be: 5.1 has no field for a place within a
resource, and the pinned guard (1.3.0) rejects every route to putting one
inside a `sources` entry -- a non-allowlisted key by name, a nested flow list
as "scalar leaves only", and quoting as an unsupported form. So the locator is
top-level keys shaped like `source_offset`, and a path carrying a flow
terminator is refused fail-fast rather than mangled.

The unit table is built AT EXTRACTION, where the extracted text and the
original's structure are known to agree: pdf -> `source_pages` from
pdfplumber's own page numbers (a page that yielded no text does not renumber
the ones after it), xlsx -> `source_sheet` + `source_rows`, everything else ->
`source_lines`. `source_offset` stays.

Two measurements changed the design before it shipped. A `paragraphs` key for
docx would name a number the document does not have: `<w:p>` counts of
108/27/65/176/57 against converted-markdown lines of 75/33/67/144/63, not one
pair agreeing -- so the key is `source_lines` and says what it indexes. And an
empty spreadsheet row renders exactly like a table separator: the content-based
rule ate 8 empty rows on the K2 price sheet and reported its last row as 92
against a workbook that says 100. The separator is now found by position, and
`tomrad.xlsx` keeps that red.

One profile moves. `provenance` is a policy object, `None` everywhere but
`SEGMENTED_OKF_V0_2`; the other five shipped profiles are byte-identical.

K2 rebuilt from a frozen src copy: 629 concepts, 1108 files, name set identical,
0 ids moved, 479 files byte-identical, 629 changed and 0 lines removed anywhere.
629/629 now carry an address and a locator. New ref
`sha256-tree:665563a2f74423fcbcc8e4f0b0954ee73b73985ac0418de4f6987bd162a1f7c8`;
`2f82fcfe...` is stale. The pre-pass payload does not grow by one byte
(209 092 B before and after, 18 changed lines: the ref and eight per-concept
digests) -- because an excerpt carries the body, not the frontmatter, which is
also why the consumer still cannot cite "file X page 12" from a payload alone.

Report: docs/2026-09-08-proveniens-k2.md. 1339 tests, ruff and mypy clean.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 14:39:24 +02:00
commit b6a8c8bd89
16 changed files with 1301 additions and 26 deletions

View file

@ -905,6 +905,39 @@ class SegmentationPolicy:
adjudication_key: str | None = None
@dataclass(frozen=True)
class ProvenancePolicy:
"""Whether a concept carries an address back to the document it came from.
Two layers, and the split is load-bearing rather than tidy.
The ADDRESS is SPEC's. §5.1:303-306 makes `sources[].resource` REQUIRED
within an entry and lets it be "an absolute URL, a bundle-relative path, or
a path into a `references/` subdirectory (§6)" -- which is exactly what a
dropped file's inbox-relative path is. No new key is invented where the
spec already has one.
The LOCATOR is OURS, and it has to be. §5.1 has no field for a page, a
sheet row or a line, and the guard's frontmatter grammar (1.3.0, measured)
refuses every route to putting one inside a `sources` entry: a key outside
its `sources` allowlist is rejected by name, and a nested flow list is
rejected as "a flow mapping admits scalar leaves only". So a locator inside
the entry would be a bundle we emit and could never read back through Door
C. Top-level keys, in the shape `source_offset` already uses.
Every field NAMES a key and none supplies a value, like every other policy
here. The presence of this object IS the capability: a profile that names
no provenance writes none, which is what keeps the five shipped profiles
that do not name it byte-identical.
"""
sources_key: str = "sources"
pages_key: str = "source_pages"
sheet_key: str = "source_sheet"
rows_key: str = "source_rows"
lines_key: str = "source_lines"
@dataclass(frozen=True)
class BundleProfile:
"""One bundle contract: types, frontmatter, filenames, index."""
@ -926,6 +959,11 @@ class BundleProfile:
# package; writing one is a Non-Goal and is named here as unassigned so the
# absence is deliberate rather than an oversight.
renderers: Mapping[str, str] | None = None
# Defaulted to `None` for the same reason `segmentation` is: `None` is not
# "provenance off", it is the profile not having the capability, which is
# what the door's `is not None` check reads. Five of the six shipped
# profiles leave it unset and keep their bytes.
provenance: ProvenancePolicy | None = None
# The ingest-spec + Phase 2 contract. Every value here was a constant in
@ -1281,6 +1319,15 @@ SEGMENTED_OKF_V0_2 = BundleProfile(
# attribute is typed `| None`, and the equality is asserted in the suite so
# this stays a fresh copy of the same policy plus the discriminator.
segmentation=SegmentationPolicy(adjudication_key="adjudication"),
# O3, and set on THIS profile alone. `sources` is a v0.2 key, so a profile
# stating v0.1 must not name it; `DEFAULT` and `STRICT_V1` state contracts
# owned in other repositories, so adding a key to either from here would be
# this repository editing someone else's contract (O2); and `OKF_V0_2` is
# Door A's, where `sources` is already written from the manifest. What is
# left is the segmented v0.2 profile -- the one whose concepts come from a
# dropped binary document and therefore the only one with an original to
# point at.
provenance=ProvenancePolicy(),
)