open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(inbox): point every concept at the document it came from, with a locator per format

Browse source 
A concept named its source file by basename and, when segmented, carried a
`source_offset` into the text THIS LIBRARY extracted. Following that pointer
needed the corpus directory, the extractor and its exact transitive version --
none of which the bundle carries. Hand-walked on a real K2 concept: six steps,
four of them requiring knowledge from outside the bundle, to learn that a
requirement sits on pages 12-13 of a 20-page document.

The address is spec's: `sources: [{ resource, title }]`, where `resource` is
the dropped file's inbox-relative path (SPEC v0.2 5.1:303-306 -- "an absolute
URL, a bundle-relative path, or a path into a `references/` subdirectory").
The locator is ours, and it has to be: 5.1 has no field for a place within a
resource, and the pinned guard (1.3.0) rejects every route to putting one
inside a `sources` entry -- a non-allowlisted key by name, a nested flow list
as "scalar leaves only", and quoting as an unsupported form. So the locator is
top-level keys shaped like `source_offset`, and a path carrying a flow
terminator is refused fail-fast rather than mangled.

The unit table is built AT EXTRACTION, where the extracted text and the
original's structure are known to agree: pdf -> `source_pages` from
pdfplumber's own page numbers (a page that yielded no text does not renumber
the ones after it), xlsx -> `source_sheet` + `source_rows`, everything else ->
`source_lines`. `source_offset` stays.

Two measurements changed the design before it shipped. A `paragraphs` key for
docx would name a number the document does not have: `<w:p>` counts of
108/27/65/176/57 against converted-markdown lines of 75/33/67/144/63, not one
pair agreeing -- so the key is `source_lines` and says what it indexes. And an
empty spreadsheet row renders exactly like a table separator: the content-based
rule ate 8 empty rows on the K2 price sheet and reported its last row as 92
against a workbook that says 100. The separator is now found by position, and
`tomrad.xlsx` keeps that red.

One profile moves. `provenance` is a policy object, `None` everywhere but
`SEGMENTED_OKF_V0_2`; the other five shipped profiles are byte-identical.

K2 rebuilt from a frozen src copy: 629 concepts, 1108 files, name set identical,
0 ids moved, 479 files byte-identical, 629 changed and 0 lines removed anywhere.
629/629 now carry an address and a locator. New ref
`sha256-tree:665563a2f74423fcbcc8e4f0b0954ee73b73985ac0418de4f6987bd162a1f7c8`;
`2f82fcfe...` is stale. The pre-pass payload does not grow by one byte
(209 092 B before and after, 18 changed lines: the ref and eight per-concept
digests) -- because an excerpt carries the body, not the frontmatter, which is
also why the consumer still cannot cite "file X page 12" from a payload alone.

Report: docs/2026-09-08-proveniens-k2.md. 1339 tests, ruff and mypy clean.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 14:39:24 +02:00
commit b6a8c8bd89
16 changed files with 1301 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

420
docs/2026-09-08-proveniens-k2.md Normal file
View file

@ -0,0 +1,420 @@
# Provenance to the original: an address and a locator per format, measured on 629 concepts
**Date:** 2026-09-08 · **Order:** `20260908T104812Z-6143923560-from-.claude` ·
**Instrument:** `src/llm_ingestion_okf/{extract,inbox,profiles}.py` plus the
`SEGMENTED_OKF_V0_2` profile · **Corpus:** the same 43-document public tender
folder every K2 measurement in this repository runs on.
A consuming agent is expected to falsify a hypothesis against the ORIGINAL
document. Before this step it could not: a concept named its source file by
basename, digested it, and — when segmented — carried a `source_offset` that
indexes the text THIS LIBRARY extracted, not the document anybody wrote. The
corpus directory, the extractor and its exact version were all required to
follow that pointer, and none of the three is in the bundle.
The corpus is external and no document body from it appears here. Documents are
named by shape, and everything else is a count.
---
## 0. What IS measured, and what is NOT
**Measured.** The provenance keys present on all 629 K2 concepts before and
after, per source format with a denominator; the three questions the order asks
per format ("can a consumer find the file, verify it, open the right place?");
one hand-walked known-positive from `source_offset` back to a page of the
original, counted in steps; the guard's verdict on seven candidate `sources`
forms, known-positive first; a full K2 rebuild with the file-set,
byte-diff and content identity; the payload a consumer receives; and the
viewer's own route table.
**Known-positive, run first.** The rebuild reproduces the bundle this
repository has been measuring on all week before anything else is read from it:
`merged + coded rejections = 39 + 4 = 43 = N`, exit 0; **629 concepts**;
**1 108 files, and the file-set is identical name for name** — 0 concept ids
moved. The 479 files that are not concepts (478 `index.md` plus `log.md`) are
byte-identical to the previous bundle.
**NOT measured.**
- **That a locator is CORRECT for any document not in this corpus.** The page
numbers are the extractor's own page numbers and the sheet rows were checked
against the workbook XML; the line numbers for the converted formats are
lines of OUR extraction and are labelled as such (§ 4).
- **That any model reads the new keys.** Whether an agent handed a payload
actually opens the original is `portfolio-optimiser`'s measurement, not this
one — and it cannot be, because the payload does not carry these keys at all
today (§ 6).
- **The viewer's per-requirement anchor.** One route table and one JS bundle
were read (§ 7). Whether `?reference=` positions a requirement needs a
browser, and the app lazy-loads chunks this measurement did not fetch.
- **A `.doc`, `.zip` or `.smc` row.** Those three types are rejected before any
concept exists, so they have no provenance to carry and no denominator here.
---
## 1. Before: what a concept said, per format, with denominators
Measured on the bundle built at `56c1205` (`/tmp/okf-kravnummer-20260908/k2-new`),
which is what HEAD produced — the following commit `116d3e1` changed the
consumption side only.
```
python3 - <<'PY' # frontmatter key presence over every non-index .md
... # the script is /tmp/o3/measure1.py; it parses the line-oriented block
PY
```
| | concepts | `source_file` | `source_sha256` | `source_offset` | `sources` |
|---|---|---|---|---|---|
| `.pdf` (32 of 33 files) | 586 | 586 | 586 | 576 | **0** |
| `.docx` (5 of 5) | 41 | 41 | 41 | 40 | **0** |
| `.xlsx` (2 of 2) | 2 | 2 | 2 | 2 | **0** |
| `.doc` / `.zip` / `.smc` | 0 | — | — | — | — |
| **total** | **629** | 629 | 629 | 618 | **0** |
The three questions the order asks, per format:
| format | (a) find the original file? | (b) verify it? | (c) open the right PLACE? |
|---|---|---|---|
| `.pdf` | **no** — a bare basename, no directory and no URL | **yes** — `source_sha256` over the dropped bytes | **no** — `source_offset` is a character position in the joined page text; `_extract_pdf` joins pages with a blank line and drops empty ones, so the page number is gone |
| `.docx` | **no** — same | **yes** | **no** — offset only; no paragraph, no heading path |
| `.xlsx` | **no** — same | **yes** | **partly** — the sheet name survives inside the concept TITLE (`Prissammenstilling {#sheet-1}`), as prose; no row, and no key a consumer can read |
Column (b) is a yes with a caveat worth stating: the digest verifies a file you
have already found by other means. It cannot help you find it.
### The known-positive: one hand-walk from an offset back to a page
Taken on a real segmented concept — a fire-safety requirement, `source_offset:
[24966, 25869]`, `source_sha256: 973e9490…` — using a script that shares no
code with the bundle builder:
1. Know the corpus directory. **Not in the bundle.** (`log.md` names it once,
for the run, not per concept — and `log.md` is a run log, not a concept.)
2. Read the file's bytes; confirm `sha256` matches. ✔
3. Know that the extractor is `pdfplumber` behind `[extract]`. **Not in the
bundle.**
4. Know its transitive text pin, `pdfminer.six==20260107`. **Not in the
bundle**, and the extracted text is pinned to it by a frozen literal in
`tests/test_extract.py`.
5. Re-extract, slice `[24966:25869]`. The requirement text comes back. ✔
6. To learn the PAGE, re-open the PDF page by page, rebuild the same join, and
bisect the offset. **Pages 12–13.**
**Six steps, four of which need knowledge the bundle does not carry.** The
document has 20 pages; nothing in the concept says which two.
---
## 2. What the spec has, and what it does not
SPEC v0.2 § 5.1:303-306, verbatim:
> `resource`: REQUIRED within an entry. Names either a concrete artifact a
> consumer can follow (an absolute URL, a bundle-relative path, or a path
> into a `references/` subdirectory, §6) or a population or scope descriptor
> it cannot (for example `all queries in BigQuery project X`).
A dropped file's inbox-relative path is precisely "a bundle-relative path", so
the ADDRESS needs no key of ours. `docs/okf-nokkelinventar.md` had already
named that re-alignment for `source_file` (Del B, `sources[].resource` as a
bundle-relative or `references/` path, § 5.1:303-305 + § 6.3); this step
executes it.
§ 5.1 lists exactly six entry keys — `resource`, `id`, `title`, `author`,
`usage_count`, `last_modified` — and **none of them is a location within the
resource.** A page, a sheet row and a line have no spec key, at any level.
### Why the locator cannot live inside a `sources` entry
Not a preference — a measured gate. Against the pinned guard (1.3.0), with a
known-positive first:
```
.venv/bin/python -c 'from llm_ingestion_guard import okf; okf.parse_frontmatter(DOC)'
```
| candidate | verdict |
|---|---|
| `sources: [{ id: a, resource: fixture }]` (our Door A golden) | **PASS** — known-positive |
| `sources: [{ resource: a.pdf, title: A }]` | **PASS** |
| `sources: [{ resource: a.pdf, sha256: abc }]` | REJECT — `mapping key 'sha256' is not on the OKF mapping allowlist under 'sources'` |
| `sources: [{ resource: a.pdf, pages: [1, 2] }]` | REJECT — `a flow mapping admits scalar leaves only, not '['` |
| `sources: [{ resource: a.xlsx, sheet: Ark1, rows: [1, 9] }]` | REJECT — same |
| `sources: [{ resource: "Del II, Bilag.pdf" }]` | REJECT — `a quoted scalar inside a flow mapping is not a supported form` |
| `sources: [{ resource: Del II Bilag 3.3.1 - Brannkonsept.pdf }]` | **PASS** — spaces are fine unquoted |
Two of those rows decide the design. A locator inside the entry has **no
expressible form** that survives the gate, so it would produce bundles this
project emits and could never read back through Door C. And because quoting is
also refused, a path containing a flow terminator cannot be escaped — it is
refused fail-fast instead (`inbox_source_file_unaddressable`), which is this
library's standing rule of validation over repair.
**The spec deviation, in four parts.**
| | |
|---|---|
| **spec says** | `sources[].resource` is the address (§ 5.1:303-306); there is no field for a place within it |
| **we do** | write the address in spec form, and the locator as top-level keys of ours — `source_pages`, `source_sheet`, `source_rows`, `source_lines` — in the shape `source_offset` already had |
| **measurement** | the guard rejects every route to putting a locator inside the entry (table above), and § 5.1's six keys contain no locator to reuse |
| **re-align** | if upstream ever gives `sources` a location field, these four keys move into the entry and become a rename. Until then they are ours, named as ours, and they say what they index |
---
## 3. The rule
**A unit table is built AT EXTRACTION**, where the extracted text and the
original's structure are the only moment they are known to agree, and it is
stored as `(start offset, original unit number, scope)` triples. A locator is
then a lookup, never a guess about text whose page breaks are already gone.
- `.pdf` → **`source_pages: [from, to]`**, from `pdfplumber`'s own page numbers.
A page that yielded no text leaves the table without renumbering the pages
after it, so the number is the document's, not "the n-th page that produced
text".
- `.xlsx` → **`source_sheet: <name>` + `source_rows: [from, to]`**. The sheet
comes from the heading the converter writes per sheet; the row restarts at
every sheet and skips the separator line **by position, never by content**
(§ 4).
- `.docx` and every text type → **`source_lines: [from, to]`**, lines of the
extracted text (§ 4).
- `source_offset` **stays**, unchanged. It is what an existing consumer joins
on, and a locator that replaced it would break them in order to help them.
`sources` is written whether or not a locator can be computed: the address
answers "which document", the locator answers "where in it", and a consumer is
owed the first even when the second is unavailable.
**One profile moves.** `provenance` is a policy object on `BundleProfile`,
defaulted to `None`, and set on `SEGMENTED_OKF_V0_2` alone. `sources` is a v0.2
key so a v0.1 profile must not name it; `DEFAULT` and `STRICT_V1` state
contracts owned in other repositories; `OKF_V0_2` is Door A's, where `sources`
already comes from the manifest. Five shipped profiles are byte-identical, and
a test asserts the field set so a sixth cannot arrive unnoticed.
---
## 4. Two things the measurement CHANGED before they shipped
Both were found by running the rule against the real corpus, and both would
have shipped a confident wrong number.
**A `paragraphs` key for `.docx` would have named a number the document does
not have.** The order asks for a paragraph or heading index. Measured on the
five K2 `.docx` files — `<w:p>` elements against converted-markdown lines:
| document | `<w:p>` | md lines | headings |
|---|---|---|---|
| 1 | 108 | 75 | 2 |
| 2 | 27 | 33 | 0 |
| 3 | 65 | 67 | 4 |
| 4 | 176 | 144 | 2 |
| 5 | 57 | 63 | 0 |
Not one pair agrees, and the heading-path alternative is empty for 2 of the 5.
So the key is `source_lines` and it says what it indexes: lines of the text
this library extracted. For `md`/`txt` that text IS the dropped file, and the
number is the original's own line.
**An empty spreadsheet row renders exactly like a table separator.** The first
implementation skipped the separator by matching the LINE (`^\|[-|: ]+\|$`).
On the K2 price sheet that rule also ate **8 empty rows**, and reported the
sheet's last row as **92** against a workbook that says **100** — every row
after the first blank one shifted, silently, for the whole sheet. The rule is
now positional: the second table line of a sheet is the separator, and nothing
else is. A new hand-laid fixture (`tomrad.xlsx`, four rows with the third
empty) is what keeps that red.
After the fix, against the workbook XML:
| spreadsheet | locator rows | `<row>` elements | contiguous from 1 |
|---|---|---|---|
| K2 document list | 1–39 | 39 | yes |
| K2 price sheet | 1–100 | 100 | yes |
Exact, both. The honesty limit: this holds as far as one converted line per
`<row>` holds. A sheet whose XML omits a row entirely would be numbered from
the converted table, and nothing here can see that.
---
## 5. After: the rebuild
`okf build` over the same 43-document folder, same arguments
(`--ingested-at 2026-09-03T00:00:00Z --bundle-id k2-trinn1-20260903
--okf-version 0.2`), run from a frozen copy of `src/` on `PYTHONPATH` so no
later edit could reach a measurement in flight — the log's first line names
that path.
| | before | after |
|---|---|---|
| `merged + coded rejections == N` | `39 + 4 = 43` | `39 + 4 = 43`, exit 0 |
| concepts | 629 | **629** |
| files in the bundle | 1 108 | **1 108**, identical name set |
| concept ids that moved | — | **0** |
| files byte-identical | — | **479 of 1 108** (every `index.md`, plus `log.md`) |
| files changed | — | **629 — every concept, and nothing else** |
| lines REMOVED across all 629 | — | **0** |
The diff is additive by measurement, not by intention: every changed file gained
`sources` plus exactly one locator shape, and no file lost a line or changed a
body.
| added keys | concepts | matches |
|---|---|---|
| `sources`, `source_pages` | 586 | every `.pdf` concept |
| `sources`, `source_lines` | 41 | every `.docx` concept |
| `sources`, `source_sheet`, `source_rows` | 2 | every `.xlsx` concept |
**The three questions, after:**
| format | (a) find | (b) verify | (c) open the right place |
|---|---|---|---|
| `.pdf` | **yes** | **yes** | **yes** — `source_pages` |
| `.docx` | **yes** | **yes** | **partly** — `source_lines` locates the extraction, not the document |
| `.xlsx` | **yes** | **yes** | **yes** — `source_sheet` + `source_rows` |
**629 of 629 carry an address and a locator.** For `.pdf` and `.xlsx`, all
three columns are yes; for `.docx` the third is honest rather than yes.
**The known-positive, walked again.** The same fire-safety concept now reads
`source_pages: [12, 13]` — the two pages the six-step hand-walk in § 1 arrived
at, computed by a script that shares no code with the builder. The walk is now
zero steps and needs nothing the bundle does not carry.
### The new content identity
**`sha256-tree:665563a2f74423fcbcc8e4f0b0954ee73b73985ac0418de4f6987bd162a1f7c8`.**
The previous ref, `sha256-tree:2f82fcfe…`, is superseded and stale. It was
recomputed from the previous bundle in the same run, as the known-positive, and
it matched — an instrument that has not reproduced a known figure has not been
shown to count.
The bundle on disk grows **97 011 B, 1 943 406 → 2 040 417 (+4.99 %)**, which is
**154 B per concept** — one address line plus one locator line.
### The payload regression
The order asks whether the pre-pass payload's token count moves now that the
frontmatter is bigger. Measured, same question, same `k`, both bundles through
the same tool:
| | before | after |
|---|---|---|
| payload bytes | 209 092 | **209 092** |
| delivered excerpts | 8 | 8 |
| budget spent (`BUDGET_UNIT`) | 118 409 | 118 409 |
| concept ids delivered | — | identical |
| excerpt texts, `text_sha256`, `withheld`, `denominators` | — | identical |
| changed lines in the whole payload | — | **18** |
The 18 are the bundle `ref` and the eight per-concept `sha256` values. **The
payload does not grow by one byte**, and the reason is § 6's finding stated from
the other side: an excerpt carries the concept's BODY, so frontmatter growth is
invisible to it. The frontmatter got bigger and the thing a consumer is billed
for did not move.
There is no o200k figure here. `tiktoken` is not installed on this machine, so
the byte count and the payload digest are the measurement; repeating a published
token number would be quoting, not measuring.
---
## 6. The consumer side: measured, not fixed
`tools/okf_consume.py` builds one excerpt per delivered concept, and
`excerpt_for` names its members explicitly. Measured on a real payload:
```
python3 -c "import json; d=json.load(open('…/k2-new-default.json')); \
print(sorted({k for x in d['excerpts'] for k in x}))"
# -> ['adjudication', 'bundle_id', 'bundle_id_inherited', 'concept_id',
# 'rank', 'sha256', 'text', 'text_sha256', 'trust_tier']
```
**No. Nine members, and not one of them is a provenance field** — not
`source_file`, not `sources`, not a locator, not even the concept's `title`.
An agent holding this payload cannot say "file X, page 12". It can say
`concept_id`, which is a path INTO THE BUNDLE, so the provenance is one file
read away — but only for a consumer that also has the bundle.
What it would take: `excerpt_for` carries `sources` and the locator keys
through from the concept's frontmatter. That is a payload-shape change, the
consumption contract § 8 enumerates an excerpt's members, and the shape is
`portfolio-optimiser`'s to decide. **Not done here**, deliberately, and this
paragraph is the whole of the fix's cost.
---
## 7. The viewer anchor, for `vegnormal-okf`
`https://viewers.vegnorm.vegvesen.no/product/859990/nb` answers **200** with an
**863-byte SPA shell** — no content, so the page itself decides nothing. The
route table and the query keys come from the app's own bundle
(`/assets/index-D1Z6iWLz.js`, 3 407 504 B):
- The route is **`/product/:id/:languageCode?/:parameter?`**. Read in the app,
`:parameter` is compared against exactly two literals — `"selection"` and
`"preview"`. **It is not a per-requirement anchor.**
- The app parses a QUERY STRING into `filePath`, `organization`,
`publicationUrn`, `tenantId`, **`reference`**, `isPrint`, `theme`, and passes
them to its content API call. **`?reference=` is the only candidate** for a
per-requirement address.
- **0 occurrences of `scrollIntoView`** in that bundle, so there is no
fragment-anchor scroll in it. `location.hash` appears 3 times, all inside the
router.
**Denominator, stated:** ONE bundle file, fetched once. The app lazy-loads
further chunks (`_mapDeps([20, 21])`) that this measurement did not fetch, and
what `reference` does to the rendered document needs a browser. So: a
per-requirement URL is **plausible via `?reference=<value>`** and **not
demonstrated**. Either way the value is the producer's to choose, and
`sources[].resource` is where it belongs — `vegnormal-okf` owns `sources` for
those bundles.
---
## 8. Honesty limits
1. **`source_lines` is not a paragraph number**, and for `.docx`, `.pptx`,
`.odt`, `.rtf`, `.csv`, `.json` and `.html` it indexes our extraction rather
than the document. § 4 has the numbers that make that the honest answer
rather than a shortcut.
2. **A row number holds only while one converted line means one `<row>`.**
Verified exact on both K2 spreadsheets and all three fixtures, all
contiguous from row 1. A sheet with a genuinely missing row would number
from the converted table.
3. **The address is inbox-relative, not absolute.** It resolves against the
folder that was dropped, and the bundle still does not say where that folder
is. Naming an absolute path would leak an operator's filesystem into a
published bundle; naming a URL is available whenever the source HAS one, and
for a dropped file it does not.
4. **The original is still not in the bundle.** § 6.3's `references/`
convention would put it there; copying binaries into a bundle is a decision
about size and licensing that nobody has taken, and it is not this order's.
5. **The rebuild is one corpus.** Every number above is 43 documents of one
public tender.
6. **`sources` carries no `id`.** § 5.1 says it SHOULD be present when the body
cites the source, and no body here cites with a `[^id]` footnote — measured
0 across the corpus in `docs/okf-nokkelinventar.md`. When one does, the key
is available and unwritten.
---
## 9. Verification log
| claim | command |
|---|---|
| 629 concepts, per-format key counts, before and after | `python3 /tmp/o3/measure1.py <bundle>` |
| the file set and the byte diff | a `rglob` set comparison plus `read_bytes()` equality over 1 108 files |
| every diff is additive | `difflib.unified_diff` per concept; 0 removed lines over 629 files |
| the guard's verdict on seven `sources` forms | `okf.parse_frontmatter` on each, known-positive first |
| docx paragraphs ≠ markdown lines | `<w:p>` count from `word/document.xml` against `extract_text(...).split("\n")` |
| spreadsheet rows against the workbook | `<row>` elements from `xl/worksheets/sheet1.xml` against `source_units(...).covering(0, len(text))` |
| the known-positive page walk | an independent script re-extracting and bisecting page offsets: pages 12–13, matching the emitted `source_pages` |
| the viewer's route table | `curl` the SPA shell and its one JS asset; `path:"…"` and query-key extraction |
| the payload members | `json.load` over a shipped K2 payload |
| suite, lint, types | `pytest -q`, `ruff check src tests tools`, `ruff format --check`, `mypy --strict src` |