open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(inbox): point every concept at the document it came from, with a locator per format

Browse source 
A concept named its source file by basename and, when segmented, carried a
`source_offset` into the text THIS LIBRARY extracted. Following that pointer
needed the corpus directory, the extractor and its exact transitive version --
none of which the bundle carries. Hand-walked on a real K2 concept: six steps,
four of them requiring knowledge from outside the bundle, to learn that a
requirement sits on pages 12-13 of a 20-page document.

The address is spec's: `sources: [{ resource, title }]`, where `resource` is
the dropped file's inbox-relative path (SPEC v0.2 5.1:303-306 -- "an absolute
URL, a bundle-relative path, or a path into a `references/` subdirectory").
The locator is ours, and it has to be: 5.1 has no field for a place within a
resource, and the pinned guard (1.3.0) rejects every route to putting one
inside a `sources` entry -- a non-allowlisted key by name, a nested flow list
as "scalar leaves only", and quoting as an unsupported form. So the locator is
top-level keys shaped like `source_offset`, and a path carrying a flow
terminator is refused fail-fast rather than mangled.

The unit table is built AT EXTRACTION, where the extracted text and the
original's structure are known to agree: pdf -> `source_pages` from
pdfplumber's own page numbers (a page that yielded no text does not renumber
the ones after it), xlsx -> `source_sheet` + `source_rows`, everything else ->
`source_lines`. `source_offset` stays.

Two measurements changed the design before it shipped. A `paragraphs` key for
docx would name a number the document does not have: `<w:p>` counts of
108/27/65/176/57 against converted-markdown lines of 75/33/67/144/63, not one
pair agreeing -- so the key is `source_lines` and says what it indexes. And an
empty spreadsheet row renders exactly like a table separator: the content-based
rule ate 8 empty rows on the K2 price sheet and reported its last row as 92
against a workbook that says 100. The separator is now found by position, and
`tomrad.xlsx` keeps that red.

One profile moves. `provenance` is a policy object, `None` everywhere but
`SEGMENTED_OKF_V0_2`; the other five shipped profiles are byte-identical.

K2 rebuilt from a frozen src copy: 629 concepts, 1108 files, name set identical,
0 ids moved, 479 files byte-identical, 629 changed and 0 lines removed anywhere.
629/629 now carry an address and a locator. New ref
`sha256-tree:665563a2f74423fcbcc8e4f0b0954ee73b73985ac0418de4f6987bd162a1f7c8`;
`2f82fcfe...` is stale. The pre-pass payload does not grow by one byte
(209 092 B before and after, 18 changed lines: the ref and eight per-concept
digests) -- because an excerpt carries the body, not the frontmatter, which is
also why the consumer still cannot cite "file X page 12" from a payload alone.

Report: docs/2026-09-08-proveniens-k2.md. 1339 tests, ruff and mypy clean.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 14:39:24 +02:00
commit b6a8c8bd89
16 changed files with 1301 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

15
CLAUDE.md
View file

@ -35,7 +35,20 @@ one boundary rule:
title (leading heading → `title` key → `path.stem`), document number,
hierarchy, and cross-references — writes it into the concept frontmatter, and
projects it into a faceted index entry. Every inferred field is named in a
`derived` list; an unmarked heuristic is worse than none. The index is a
`derived` list; an unmarked heuristic is worse than none.
Under the SEGMENTED v0.2 profile a concept additionally POINTS BACK at the
original: `sources: [{ resource, title }]` in SPEC §5.1's form (`resource` is
the inbox-relative path), plus a locator per format — `source_pages`,
`source_sheet`+`source_rows`, else `source_lines`. The locator keys are OURS
and must stay top-level: §5.1 has no field for a place within a resource, and
the pinned guard rejects every route to putting one inside a `sources` entry
(non-allowlisted key, nested flow list, quoted scalar), so a locator in the
entry would emit bundles Door C could never read back. The unit table is
built AT EXTRACTION — a page number cannot be recovered from joined text —
and `source_offset` stays. `source_lines` indexes the EXTRACTED text, never
the original's paragraphs: measured, docx `<w:p>` counts and converted-line
counts do not agree on a single one of five documents. Record:
`docs/2026-09-08-proveniens-k2.md`. The index is a
PROJECTION recomputed from the whole bundle each round, which is what makes
rebuild-from-scratch equal an incremental update byte for byte. `DEFAULT` is
untouched and byte-identical. Record: `docs/plan/structure-derivation.md`.

10
README.md
View file

@ -201,6 +201,16 @@ bundle:
root as its `source_file`, while dot-directories and a bundle directory
sitting inside the inbox are skipped with a reported code.
Under the segmented v0.2 profile a concept also points back at the document
it was extracted from, so an agent citing it can open the original at the
right place: `sources: [{ resource, title }]` in the spec's own §5.1 form,
where `resource` is the inbox-relative path, plus a locator per format —
`source_pages` for a PDF, `source_sheet` and `source_rows` for a
spreadsheet, `source_lines` otherwise. The locator keys are this library's
own, because §5.1 has no field for a place *within* a resource; the line
numbers index the extracted text and say so. Measurements:
[`docs/2026-09-08-proveniens-k2.md`](docs/2026-09-08-proveniens-k2.md).
<!-- extract-formats: .md, .txt, .csv, .json, .html, .htm, .pdf, .docx, .xlsx, .pptx, .odt, .rtf -->
3. **External bundle import.** Import and merge of third-party OKF bundles:
each concept is assessed via the security gate, and only concepts that

420
docs/2026-09-08-proveniens-k2.md Normal file
View file

@ -0,0 +1,420 @@
# Provenance to the original: an address and a locator per format, measured on 629 concepts
**Date:** 2026-09-08 · **Order:** `20260908T104812Z-6143923560-from-.claude` ·
**Instrument:** `src/llm_ingestion_okf/{extract,inbox,profiles}.py` plus the
`SEGMENTED_OKF_V0_2` profile · **Corpus:** the same 43-document public tender
folder every K2 measurement in this repository runs on.
A consuming agent is expected to falsify a hypothesis against the ORIGINAL
document. Before this step it could not: a concept named its source file by
basename, digested it, and — when segmented — carried a `source_offset` that
indexes the text THIS LIBRARY extracted, not the document anybody wrote. The
corpus directory, the extractor and its exact version were all required to
follow that pointer, and none of the three is in the bundle.
The corpus is external and no document body from it appears here. Documents are
named by shape, and everything else is a count.
---
## 0. What IS measured, and what is NOT
**Measured.** The provenance keys present on all 629 K2 concepts before and
after, per source format with a denominator; the three questions the order asks
per format ("can a consumer find the file, verify it, open the right place?");
one hand-walked known-positive from `source_offset` back to a page of the
original, counted in steps; the guard's verdict on seven candidate `sources`
forms, known-positive first; a full K2 rebuild with the file-set,
byte-diff and content identity; the payload a consumer receives; and the
viewer's own route table.
**Known-positive, run first.** The rebuild reproduces the bundle this
repository has been measuring on all week before anything else is read from it:
`merged + coded rejections = 39 + 4 = 43 = N`, exit 0; **629 concepts**;
**1 108 files, and the file-set is identical name for name** — 0 concept ids
moved. The 479 files that are not concepts (478 `index.md` plus `log.md`) are
byte-identical to the previous bundle.
**NOT measured.**
- **That a locator is CORRECT for any document not in this corpus.** The page
numbers are the extractor's own page numbers and the sheet rows were checked
against the workbook XML; the line numbers for the converted formats are
lines of OUR extraction and are labelled as such (§ 4).
- **That any model reads the new keys.** Whether an agent handed a payload
actually opens the original is `portfolio-optimiser`'s measurement, not this
one — and it cannot be, because the payload does not carry these keys at all
today (§ 6).
- **The viewer's per-requirement anchor.** One route table and one JS bundle
were read (§ 7). Whether `?reference=` positions a requirement needs a
browser, and the app lazy-loads chunks this measurement did not fetch.
- **A `.doc`, `.zip` or `.smc` row.** Those three types are rejected before any
concept exists, so they have no provenance to carry and no denominator here.
---
## 1. Before: what a concept said, per format, with denominators
Measured on the bundle built at `56c1205` (`/tmp/okf-kravnummer-20260908/k2-new`),
which is what HEAD produced — the following commit `116d3e1` changed the
consumption side only.
```
python3 - <<'PY' # frontmatter key presence over every non-index .md
... # the script is /tmp/o3/measure1.py; it parses the line-oriented block
PY
```
| | concepts | `source_file` | `source_sha256` | `source_offset` | `sources` |
|---|---|---|---|---|---|
| `.pdf` (32 of 33 files) | 586 | 586 | 586 | 576 | **0** |
| `.docx` (5 of 5) | 41 | 41 | 41 | 40 | **0** |
| `.xlsx` (2 of 2) | 2 | 2 | 2 | 2 | **0** |
| `.doc` / `.zip` / `.smc` | 0 | — | — | — | — |
| **total** | **629** | 629 | 629 | 618 | **0** |
The three questions the order asks, per format:
| format | (a) find the original file? | (b) verify it? | (c) open the right PLACE? |
|---|---|---|---|
| `.pdf` | **no** — a bare basename, no directory and no URL | **yes** — `source_sha256` over the dropped bytes | **no** — `source_offset` is a character position in the joined page text; `_extract_pdf` joins pages with a blank line and drops empty ones, so the page number is gone |
| `.docx` | **no** — same | **yes** | **no** — offset only; no paragraph, no heading path |
| `.xlsx` | **no** — same | **yes** | **partly** — the sheet name survives inside the concept TITLE (`Prissammenstilling {#sheet-1}`), as prose; no row, and no key a consumer can read |
Column (b) is a yes with a caveat worth stating: the digest verifies a file you
have already found by other means. It cannot help you find it.
### The known-positive: one hand-walk from an offset back to a page
Taken on a real segmented concept — a fire-safety requirement, `source_offset:
[24966, 25869]`, `source_sha256: 973e9490…` — using a script that shares no
code with the bundle builder:
1. Know the corpus directory. **Not in the bundle.** (`log.md` names it once,
for the run, not per concept — and `log.md` is a run log, not a concept.)
2. Read the file's bytes; confirm `sha256` matches. ✔
3. Know that the extractor is `pdfplumber` behind `[extract]`. **Not in the
bundle.**
4. Know its transitive text pin, `pdfminer.six==20260107`. **Not in the
bundle**, and the extracted text is pinned to it by a frozen literal in
`tests/test_extract.py`.
5. Re-extract, slice `[24966:25869]`. The requirement text comes back. ✔
6. To learn the PAGE, re-open the PDF page by page, rebuild the same join, and
bisect the offset. **Pages 12–13.**
**Six steps, four of which need knowledge the bundle does not carry.** The
document has 20 pages; nothing in the concept says which two.
---
## 2. What the spec has, and what it does not
SPEC v0.2 § 5.1:303-306, verbatim:
> `resource`: REQUIRED within an entry. Names either a concrete artifact a
> consumer can follow (an absolute URL, a bundle-relative path, or a path
> into a `references/` subdirectory, §6) or a population or scope descriptor
> it cannot (for example `all queries in BigQuery project X`).
A dropped file's inbox-relative path is precisely "a bundle-relative path", so
the ADDRESS needs no key of ours. `docs/okf-nokkelinventar.md` had already
named that re-alignment for `source_file` (Del B, `sources[].resource` as a
bundle-relative or `references/` path, § 5.1:303-305 + § 6.3); this step
executes it.
§ 5.1 lists exactly six entry keys — `resource`, `id`, `title`, `author`,
`usage_count`, `last_modified` — and **none of them is a location within the
resource.** A page, a sheet row and a line have no spec key, at any level.
### Why the locator cannot live inside a `sources` entry
Not a preference — a measured gate. Against the pinned guard (1.3.0), with a
known-positive first:
```
.venv/bin/python -c 'from llm_ingestion_guard import okf; okf.parse_frontmatter(DOC)'
```
| candidate | verdict |
|---|---|
| `sources: [{ id: a, resource: fixture }]` (our Door A golden) | **PASS** — known-positive |
| `sources: [{ resource: a.pdf, title: A }]` | **PASS** |
| `sources: [{ resource: a.pdf, sha256: abc }]` | REJECT — `mapping key 'sha256' is not on the OKF mapping allowlist under 'sources'` |
| `sources: [{ resource: a.pdf, pages: [1, 2] }]` | REJECT — `a flow mapping admits scalar leaves only, not '['` |
| `sources: [{ resource: a.xlsx, sheet: Ark1, rows: [1, 9] }]` | REJECT — same |
| `sources: [{ resource: "Del II, Bilag.pdf" }]` | REJECT — `a quoted scalar inside a flow mapping is not a supported form` |
| `sources: [{ resource: Del II Bilag 3.3.1 - Brannkonsept.pdf }]` | **PASS** — spaces are fine unquoted |
Two of those rows decide the design. A locator inside the entry has **no
expressible form** that survives the gate, so it would produce bundles this
project emits and could never read back through Door C. And because quoting is
also refused, a path containing a flow terminator cannot be escaped — it is
refused fail-fast instead (`inbox_source_file_unaddressable`), which is this
library's standing rule of validation over repair.
**The spec deviation, in four parts.**
| | |
|---|---|
| **spec says** | `sources[].resource` is the address (§ 5.1:303-306); there is no field for a place within it |
| **we do** | write the address in spec form, and the locator as top-level keys of ours — `source_pages`, `source_sheet`, `source_rows`, `source_lines` — in the shape `source_offset` already had |
| **measurement** | the guard rejects every route to putting a locator inside the entry (table above), and § 5.1's six keys contain no locator to reuse |
| **re-align** | if upstream ever gives `sources` a location field, these four keys move into the entry and become a rename. Until then they are ours, named as ours, and they say what they index |
---
## 3. The rule
**A unit table is built AT EXTRACTION**, where the extracted text and the
original's structure are the only moment they are known to agree, and it is
stored as `(start offset, original unit number, scope)` triples. A locator is
then a lookup, never a guess about text whose page breaks are already gone.
- `.pdf` → **`source_pages: [from, to]`**, from `pdfplumber`'s own page numbers.
A page that yielded no text leaves the table without renumbering the pages
after it, so the number is the document's, not "the n-th page that produced
text".
- `.xlsx` → **`source_sheet: <name>` + `source_rows: [from, to]`**. The sheet
comes from the heading the converter writes per sheet; the row restarts at
every sheet and skips the separator line **by position, never by content**
(§ 4).
- `.docx` and every text type → **`source_lines: [from, to]`**, lines of the
extracted text (§ 4).
- `source_offset` **stays**, unchanged. It is what an existing consumer joins
on, and a locator that replaced it would break them in order to help them.
`sources` is written whether or not a locator can be computed: the address
answers "which document", the locator answers "where in it", and a consumer is
owed the first even when the second is unavailable.
**One profile moves.** `provenance` is a policy object on `BundleProfile`,
defaulted to `None`, and set on `SEGMENTED_OKF_V0_2` alone. `sources` is a v0.2
key so a v0.1 profile must not name it; `DEFAULT` and `STRICT_V1` state
contracts owned in other repositories; `OKF_V0_2` is Door A's, where `sources`
already comes from the manifest. Five shipped profiles are byte-identical, and
a test asserts the field set so a sixth cannot arrive unnoticed.
---
## 4. Two things the measurement CHANGED before they shipped
Both were found by running the rule against the real corpus, and both would
have shipped a confident wrong number.
**A `paragraphs` key for `.docx` would have named a number the document does
not have.** The order asks for a paragraph or heading index. Measured on the
five K2 `.docx` files — `<w:p>` elements against converted-markdown lines:
| document | `<w:p>` | md lines | headings |
|---|---|---|---|
| 1 | 108 | 75 | 2 |
| 2 | 27 | 33 | 0 |
| 3 | 65 | 67 | 4 |
| 4 | 176 | 144 | 2 |
| 5 | 57 | 63 | 0 |
Not one pair agrees, and the heading-path alternative is empty for 2 of the 5.
So the key is `source_lines` and it says what it indexes: lines of the text
this library extracted. For `md`/`txt` that text IS the dropped file, and the
number is the original's own line.
**An empty spreadsheet row renders exactly like a table separator.** The first
implementation skipped the separator by matching the LINE (`^\|[-|: ]+\|$`).
On the K2 price sheet that rule also ate **8 empty rows**, and reported the
sheet's last row as **92** against a workbook that says **100** — every row
after the first blank one shifted, silently, for the whole sheet. The rule is
now positional: the second table line of a sheet is the separator, and nothing
else is. A new hand-laid fixture (`tomrad.xlsx`, four rows with the third
empty) is what keeps that red.
After the fix, against the workbook XML:
| spreadsheet | locator rows | `<row>` elements | contiguous from 1 |
|---|---|---|---|
| K2 document list | 1–39 | 39 | yes |
| K2 price sheet | 1–100 | 100 | yes |
Exact, both. The honesty limit: this holds as far as one converted line per
`<row>` holds. A sheet whose XML omits a row entirely would be numbered from
the converted table, and nothing here can see that.
---
## 5. After: the rebuild
`okf build` over the same 43-document folder, same arguments
(`--ingested-at 2026-09-03T00:00:00Z --bundle-id k2-trinn1-20260903
--okf-version 0.2`), run from a frozen copy of `src/` on `PYTHONPATH` so no
later edit could reach a measurement in flight — the log's first line names
that path.
| | before | after |
|---|---|---|
| `merged + coded rejections == N` | `39 + 4 = 43` | `39 + 4 = 43`, exit 0 |
| concepts | 629 | **629** |
| files in the bundle | 1 108 | **1 108**, identical name set |
| concept ids that moved | — | **0** |
| files byte-identical | — | **479 of 1 108** (every `index.md`, plus `log.md`) |
| files changed | — | **629 — every concept, and nothing else** |
| lines REMOVED across all 629 | — | **0** |
The diff is additive by measurement, not by intention: every changed file gained
`sources` plus exactly one locator shape, and no file lost a line or changed a
body.
| added keys | concepts | matches |
|---|---|---|
| `sources`, `source_pages` | 586 | every `.pdf` concept |
| `sources`, `source_lines` | 41 | every `.docx` concept |
| `sources`, `source_sheet`, `source_rows` | 2 | every `.xlsx` concept |
**The three questions, after:**
| format | (a) find | (b) verify | (c) open the right place |
|---|---|---|---|
| `.pdf` | **yes** | **yes** | **yes** — `source_pages` |
| `.docx` | **yes** | **yes** | **partly** — `source_lines` locates the extraction, not the document |
| `.xlsx` | **yes** | **yes** | **yes** — `source_sheet` + `source_rows` |
**629 of 629 carry an address and a locator.** For `.pdf` and `.xlsx`, all
three columns are yes; for `.docx` the third is honest rather than yes.
**The known-positive, walked again.** The same fire-safety concept now reads
`source_pages: [12, 13]` — the two pages the six-step hand-walk in § 1 arrived
at, computed by a script that shares no code with the builder. The walk is now
zero steps and needs nothing the bundle does not carry.
### The new content identity
**`sha256-tree:665563a2f74423fcbcc8e4f0b0954ee73b73985ac0418de4f6987bd162a1f7c8`.**
The previous ref, `sha256-tree:2f82fcfe…`, is superseded and stale. It was
recomputed from the previous bundle in the same run, as the known-positive, and
it matched — an instrument that has not reproduced a known figure has not been
shown to count.
The bundle on disk grows **97 011 B, 1 943 406 → 2 040 417 (+4.99 %)**, which is
**154 B per concept** — one address line plus one locator line.
### The payload regression
The order asks whether the pre-pass payload's token count moves now that the
frontmatter is bigger. Measured, same question, same `k`, both bundles through
the same tool:
| | before | after |
|---|---|---|
| payload bytes | 209 092 | **209 092** |
| delivered excerpts | 8 | 8 |
| budget spent (`BUDGET_UNIT`) | 118 409 | 118 409 |
| concept ids delivered | — | identical |
| excerpt texts, `text_sha256`, `withheld`, `denominators` | — | identical |
| changed lines in the whole payload | — | **18** |
The 18 are the bundle `ref` and the eight per-concept `sha256` values. **The
payload does not grow by one byte**, and the reason is § 6's finding stated from
the other side: an excerpt carries the concept's BODY, so frontmatter growth is
invisible to it. The frontmatter got bigger and the thing a consumer is billed
for did not move.
There is no o200k figure here. `tiktoken` is not installed on this machine, so
the byte count and the payload digest are the measurement; repeating a published
token number would be quoting, not measuring.
---
## 6. The consumer side: measured, not fixed
`tools/okf_consume.py` builds one excerpt per delivered concept, and
`excerpt_for` names its members explicitly. Measured on a real payload:
```
python3 -c "import json; d=json.load(open('…/k2-new-default.json')); \
print(sorted({k for x in d['excerpts'] for k in x}))"
# -> ['adjudication', 'bundle_id', 'bundle_id_inherited', 'concept_id',
# 'rank', 'sha256', 'text', 'text_sha256', 'trust_tier']
```
**No. Nine members, and not one of them is a provenance field** — not
`source_file`, not `sources`, not a locator, not even the concept's `title`.
An agent holding this payload cannot say "file X, page 12". It can say
`concept_id`, which is a path INTO THE BUNDLE, so the provenance is one file
read away — but only for a consumer that also has the bundle.
What it would take: `excerpt_for` carries `sources` and the locator keys
through from the concept's frontmatter. That is a payload-shape change, the
consumption contract § 8 enumerates an excerpt's members, and the shape is
`portfolio-optimiser`'s to decide. **Not done here**, deliberately, and this
paragraph is the whole of the fix's cost.
---
## 7. The viewer anchor, for `vegnormal-okf`
`https://viewers.vegnorm.vegvesen.no/product/859990/nb` answers **200** with an
**863-byte SPA shell** — no content, so the page itself decides nothing. The
route table and the query keys come from the app's own bundle
(`/assets/index-D1Z6iWLz.js`, 3 407 504 B):
- The route is **`/product/:id/:languageCode?/:parameter?`**. Read in the app,
`:parameter` is compared against exactly two literals — `"selection"` and
`"preview"`. **It is not a per-requirement anchor.**
- The app parses a QUERY STRING into `filePath`, `organization`,
`publicationUrn`, `tenantId`, **`reference`**, `isPrint`, `theme`, and passes
them to its content API call. **`?reference=` is the only candidate** for a
per-requirement address.
- **0 occurrences of `scrollIntoView`** in that bundle, so there is no
fragment-anchor scroll in it. `location.hash` appears 3 times, all inside the
router.
**Denominator, stated:** ONE bundle file, fetched once. The app lazy-loads
further chunks (`_mapDeps([20, 21])`) that this measurement did not fetch, and
what `reference` does to the rendered document needs a browser. So: a
per-requirement URL is **plausible via `?reference=<value>`** and **not
demonstrated**. Either way the value is the producer's to choose, and
`sources[].resource` is where it belongs — `vegnormal-okf` owns `sources` for
those bundles.
---
## 8. Honesty limits
1. **`source_lines` is not a paragraph number**, and for `.docx`, `.pptx`,
`.odt`, `.rtf`, `.csv`, `.json` and `.html` it indexes our extraction rather
than the document. § 4 has the numbers that make that the honest answer
rather than a shortcut.
2. **A row number holds only while one converted line means one `<row>`.**
Verified exact on both K2 spreadsheets and all three fixtures, all
contiguous from row 1. A sheet with a genuinely missing row would number
from the converted table.
3. **The address is inbox-relative, not absolute.** It resolves against the
folder that was dropped, and the bundle still does not say where that folder
is. Naming an absolute path would leak an operator's filesystem into a
published bundle; naming a URL is available whenever the source HAS one, and
for a dropped file it does not.
4. **The original is still not in the bundle.** § 6.3's `references/`
convention would put it there; copying binaries into a bundle is a decision
about size and licensing that nobody has taken, and it is not this order's.
5. **The rebuild is one corpus.** Every number above is 43 documents of one
public tender.
6. **`sources` carries no `id`.** § 5.1 says it SHOULD be present when the body
cites the source, and no body here cites with a `[^id]` footnote — measured
0 across the corpus in `docs/okf-nokkelinventar.md`. When one does, the key
is available and unwritten.
---
## 9. Verification log
| claim | command |
|---|---|
| 629 concepts, per-format key counts, before and after | `python3 /tmp/o3/measure1.py <bundle>` |
| the file set and the byte diff | a `rglob` set comparison plus `read_bytes()` equality over 1 108 files |
| every diff is additive | `difflib.unified_diff` per concept; 0 removed lines over 629 files |
| the guard's verdict on seven `sources` forms | `okf.parse_frontmatter` on each, known-positive first |
| docx paragraphs ≠ markdown lines | `<w:p>` count from `word/document.xml` against `extract_text(...).split("\n")` |
| spreadsheet rows against the workbook | `<row>` elements from `xl/worksheets/sheet1.xml` against `source_units(...).covering(0, len(text))` |
| the known-positive page walk | an independent script re-extracting and bisecting page offsets: pages 12–13, matching the emitted `source_pages` |
| the viewer's route table | `curl` the SPA shell and its one JS asset; `path:"…"` and query-key extraction |
| the payload members | `json.load` over a shipped K2 payload |
| suite, lint, types | `pytest -q`, `ruff check src tests tools`, `ruff format --check`, `mypy --strict src` |

2
examples/ingest-golden-segmented-okf-v0-2/expected-bundle/krav/1-1/foerste-krav.md
View file

@ -5,10 +5,12 @@ source_file: veiledning.md
source_sha256: 6906ec0acbcfc246e825bda9863c716eb5611b465020e8204eeb448c32343f7d
ingested_at: 2026-08-30T09:00:00Z
generated: true
sources: [{ resource: veiledning.md, title: veiledning.md }]
adjudication: proposed
bundle_id: b-golden-segmented-okf-v0-2
parent: s0
segment_id: s1
source_lines: [5, 8]
source_offset: [94, 176]
---

2
examples/ingest-golden-segmented-okf-v0-2/expected-bundle/krav/1-2/andre-krav.md
View file

@ -5,10 +5,12 @@ source_file: veiledning.md
source_sha256: 6906ec0acbcfc246e825bda9863c716eb5611b465020e8204eeb448c32343f7d
ingested_at: 2026-08-30T09:00:00Z
generated: true
sources: [{ resource: veiledning.md, title: veiledning.md }]
adjudication: proposed
bundle_id: b-golden-segmented-okf-v0-2
parent: s0
segment_id: s2
source_lines: [9, 11]
source_offset: [176, 253]
---

2
examples/ingest-golden-segmented-okf-v0-2/expected-bundle/veiledning.md
View file

@ -5,9 +5,11 @@ source_file: veiledning.md
source_sha256: 6906ec0acbcfc246e825bda9863c716eb5611b465020e8204eeb448c32343f7d
ingested_at: 2026-08-30T09:00:00Z
generated: true
sources: [{ resource: veiledning.md, title: veiledning.md }]
adjudication: proposed
bundle_id: b-golden-segmented-okf-v0-2
segment_id: s0
source_lines: [1, 4]
source_offset: [0, 94]
---

8
skills/okf-consume/references/example-payload.json
View file

@ -2,7 +2,7 @@
"contract": "okf-consumption/1",
"bundle": {
"bundle_id": "b-golden-segmented-okf-v0-2",
"ref": "sha256-tree:e272be79295bdeaa19127cd7d8ee76b8a02f05b7f6b2feeb1c48ffb55e87b230"
"ref": "sha256-tree:f85646c1f77982a69b97afca08209f369b79cd39c217f6f2c074c2dc5bac23fe"
},
"budget": {
"unit": "utf-8 bytes of emitted JSON",
@ -27,7 +27,7 @@
{
"bundle_id": "b-golden-segmented-okf-v0-2",
"concept_id": "krav/1-1/foerste-krav",
"sha256": "29ac5dfe604b0d7bd50182a68b18f614f1388913c7487adcd95c4997f5ef4b35",
"sha256": "84145512284a8bbc1a8905c73d0de3f0e0cb7081cf517f778fe88a610d2b4f62",
"adjudication": "proposed",
"trust_tier": "unverified",
"bundle_id_inherited": false,
@ -38,7 +38,7 @@
{
"bundle_id": "b-golden-segmented-okf-v0-2",
"concept_id": "veiledning",
"sha256": "85d54e499a9d6499418994dbdbc472568cf5a14fea1ecc9179fd8983b61a31c1",
"sha256": "8f1f4d940887b96b39496d0aceb7a4fa212553510165917427e4f7e9efb98b03",
"adjudication": "proposed",
"trust_tier": "unverified",
"bundle_id_inherited": false,
@ -49,7 +49,7 @@
{
"bundle_id": "b-golden-segmented-okf-v0-2",
"concept_id": "krav/1-2/andre-krav",
"sha256": "c40b32d60f2488d22c73c06966174eb01a0971d4c7e0f8267b2d942e70f23038",
"sha256": "4f8a869aa4e898ec1a30595951b3e198e09091d76c41078a52e821000eb9e22c",
"adjudication": "proposed",
"trust_tier": "unverified",
"bundle_id_inherited": false,

247
src/llm_ingestion_okf/extract.py
View file

@ -32,6 +32,7 @@ import warnings
import zipfile
from xml.etree import ElementTree
from collections.abc import Callable, Sequence
from dataclasses import dataclass
from html.parser import HTMLParser
from pathlib import Path
@ -236,6 +237,45 @@ def _extra_missing(suffix: str) -> ExtractionError:
)
# How `_extract_pdf` joins its pages, named because the locator below has to
# reproduce the exact same arithmetic to turn a character offset back into a
# page number. Two constants that must agree, written once.
_PDF_PAGE_SEPARATOR = "\n\n"
@functools.lru_cache(maxsize=1)
def _pdf_pages(data: bytes) -> tuple[tuple[int, str], ...]:
"""Every page that produced text, as `(page number, text)`, in page order.
The page NUMBER is 1-based and comes from the document, so a page that
yielded nothing removes itself from the sequence without renumbering the
ones after it -- which is the difference between "the third page that
produced text" and "page 3", and the whole reason a locator is worth
writing down.
Memoised on the bytes with room for exactly one document: extraction and
location are two calls about the same file, back to back, and parsing it
twice would double the PDF cost of every corpus run for nothing. Anything
larger would hold whole documents in memory for no gain, since the caller
never returns to an earlier file.
"""
try:
import pdfplumber
except ImportError as exc:
raise _extra_missing(".pdf") from exc
try:
with pdfplumber.open(io.BytesIO(data)) as pdf:
pages = [(page.extract_text() or "").rstrip() for page in pdf.pages]
except ExtractionError:
raise
except Exception as exc: # noqa: BLE001 - third-party parser, wrapped never leaked
raise ExtractionError(
f"the PDF parser failed on this file: {exc}", code="extractor_pdf_error"
) from exc
return tuple((number, page) for number, page in enumerate(pages, start=1) if page)
def _extract_pdf(data: bytes) -> str:
"""`pdf`: page text via `pdfplumber`, in page order, pages separated by a
blank line.
@ -252,22 +292,8 @@ def _extract_pdf(data: bytes) -> str:
and pymupdf each emit all labels then all values. Re-pairing those is
guesswork, and in a requirements document a wrong pairing looks right.
"""
try:
import pdfplumber
except ImportError as exc:
raise _extra_missing(".pdf") from exc
try:
with pdfplumber.open(io.BytesIO(data)) as pdf:
pages = [(page.extract_text() or "").rstrip() for page in pdf.pages]
except ExtractionError:
raise
except Exception as exc: # noqa: BLE001 - third-party parser, wrapped never leaked
raise ExtractionError(
f"the PDF parser failed on this file: {exc}", code="extractor_pdf_error"
) from exc
text = "\n\n".join(page for page in pages if page)
pages = _pdf_pages(data)
text = _PDF_PAGE_SEPARATOR.join(page for _, page in pages)
if not text:
raise ExtractionError(
"the PDF yielded no text on any page; a scanned or image-only "
@ -420,6 +446,195 @@ _OPTIONAL_EXTRACTORS: dict[str, Callable[[bytes], str]] = {
}
# --- provenance: a character range of the extracted text -> a place in the
# original document ---------------------------------------------------------
#
# `source_offset` alone is a position in OUR extraction, so following it back
# needs the corpus directory, the extractor and its exact version -- none of
# which a bundle carries. A unit table is that mapping, saved AT EXTRACTION
# where the two are known to agree, rather than guessed afterwards from text
# whose page breaks are gone.
#
# THE UNIT IS PER FORMAT AND IS NAMED, never assumed:
#
# pages a PDF page number, from the document itself.
# rows a spreadsheet row, within the sheet named by `scope_of`.
# lines a line of the EXTRACTED text. For `md`/`txt` that text is the
# dropped file, so the number is the original's own line; for the
# converted formats it is not, and the key says `lines` rather than
# `paragraphs` for exactly that reason. Measured on the five K2
# `.docx` documents: `<w:p>` counts 108/27/65/176/57 against
# converted-markdown line counts 75/33/67/144/63 -- not one pair
# agrees, so a `paragraphs` key would name a number the original does
# not have.
#
# The heading a spreadsheet's sheet becomes, as the converter writes it:
# `## <sheet name> {#sheet-<n>}`. Anchored to the line start so a pipe cell
# containing a `#` cannot be read as a sheet.
_SHEET_HEADING = re.compile(r"^#{1,6} (?P<name>.*?) \{#sheet-\d+\}$")
# A line the converter wrote as part of a pipe table. Whether one of them is
# the table's SEPARATOR is decided by POSITION, never by content: an empty
# spreadsheet row renders as `| | |` and a separator as `|----|----|`, and
# every content rule that tells those apart also swallows a data row that
# happens to hold only dashes. Measured on the K2 price sheet: a content rule
# ate 8 empty rows and reported the sheet's last row as 92 against a workbook
# that says 100.
_TABLE_LINE = "|"
@dataclass(frozen=True)
class SourceUnits:
"""Where in the ORIGINAL each stretch of the extracted text came from.
`starts[i]` is the character offset in the extracted text at which unit
`numbers[i]` begins, and `scopes[i]` is the sheet that unit belongs to (or
`None` for a format that has no sheets). The three tuples are parallel and
`starts` ascends, which is what lets `covering` be a bisection rather than
a scan.
`numbers` is separate from the index on purpose. A PDF page that yielded no
text is not in this table, and a pipe table's separator line is a row of
nothing -- in both cases the position in the table and the number in the
original have already parted company, and an index standing in for a number
is the off-by-one this whole object exists to prevent.
"""
unit: str
starts: tuple[int, ...]
numbers: tuple[int, ...]
scopes: tuple[str | None, ...] = ()
def __post_init__(self) -> None:
if len(self.starts) != len(self.numbers):
raise ValueError("a unit table needs one number per start offset")
if self.scopes and len(self.scopes) != len(self.starts):
raise ValueError("a unit table needs one scope per start offset, or none at all")
def _index(self, offset: int) -> int:
"""The table row covering `offset`, clamped to the table's own ends."""
low, high = 0, len(self.starts) - 1
while low < high:
middle = (low + high + 1) // 2
if self.starts[middle] <= offset:
low = middle
else:
high = middle - 1
return low
def covering(self, start: int, end: int) -> tuple[int, int]:
"""The first and last original unit the half-open `[start, end)` touches.
`end` is exclusive, so a range ending exactly where the next unit
begins does not claim that unit -- a segment that stops at a page
boundary is on the page it was written on.
"""
if not self.starts:
raise ValueError("an empty unit table locates nothing")
first = self._index(start)
last = self._index(max(start, end - 1))
return self.numbers[first], self.numbers[last]
def scope_of(self, offset: int) -> str | None:
"""The sheet `offset` falls in, or `None` for a format without sheets."""
if not self.scopes:
return None
return self.scopes[self._index(offset)]
def scopes_covering(self, start: int, end: int) -> tuple[str | None, ...]:
"""Every distinct scope the range touches, in order, without repeats."""
if not self.scopes:
return ()
first = self._index(start)
last = self._index(max(start, end - 1))
seen: list[str | None] = []
for scope in self.scopes[first : last + 1]:
if not seen or seen[-1] != scope:
seen.append(scope)
return tuple(seen)
def _line_units(text: str) -> SourceUnits:
starts: list[int] = []
offset = 0
for line in text.split("\n"):
starts.append(offset)
offset += len(line) + 1
return SourceUnits("lines", tuple(starts), tuple(range(1, len(starts) + 1)))
def _pdf_units(data: bytes) -> SourceUnits:
starts: list[int] = []
numbers: list[int] = []
offset = 0
for number, page in _pdf_pages(data):
starts.append(offset)
numbers.append(number)
offset += len(page) + len(_PDF_PAGE_SEPARATOR)
return SourceUnits("pages", tuple(starts), tuple(numbers))
def _spreadsheet_units(text: str) -> SourceUnits | None:
"""Sheet and row for a converted spreadsheet, or `None` if it is not one.
The converter writes one heading per sheet and then one pipe-table line per
source row, with a separator line after the first. Row numbering therefore
restarts at every heading and skips that one line by POSITION.
The row number is the ORIGINAL sheet's, and that holds exactly as far as
one converted line per `<row>` element holds. Measured on the two K2
spreadsheets and both fixtures: 39 rows for 39, 100 for 100, 4 for 4, 6 for
6 and 3 for 3 -- every one contiguous from row 1. A sheet whose XML omits a
row entirely would number from the converted table instead, and nothing
here can see that.
"""
starts: list[int] = []
numbers: list[int] = []
scopes: list[str | None] = []
sheet: str | None = None
seen = 0
offset = 0
for line in text.split("\n"):
heading = _SHEET_HEADING.match(line)
if heading is not None:
sheet = heading.group("name")
seen = 0
elif sheet is not None and line.startswith(_TABLE_LINE):
seen += 1
# The SECOND table line of a sheet is the separator the converter
# writes under the header, and it is a row of no spreadsheet. Every
# line after it is one row further on than its position suggests.
if seen != 2:
starts.append(offset)
numbers.append(seen if seen == 1 else seen - 1)
scopes.append(sheet)
offset += len(line) + 1
if not starts:
return None
return SourceUnits("rows", tuple(starts), tuple(numbers), tuple(scopes))
def source_units(filename: str, data: bytes, text: str) -> SourceUnits | None:
"""The unit table for one dropped file, or `None` when it has none.
`text` must be what `extract_text` returned for these exact bytes: the
table indexes that string, and a table built against a different rendering
would point a consumer at the wrong place with full confidence.
`None` is a measurement, not a failure -- a spreadsheet the converter wrote
no table for has no rows to name, and the caller writes the address without
a locator rather than inventing one.
"""
suffix = Path(filename).suffix.lower()
if suffix == ".pdf":
return _pdf_units(data)
if suffix == ".xlsx":
return _spreadsheet_units(text)
if suffix in _CORE_EXTRACTORS or suffix in _PANDOC_FORMATS:
return _line_units(text)
return None
def extract_text(
filename: str, data: bytes, *, renderer: Callable[[str], str] | None = None
) -> str:

100
src/llm_ingestion_okf/inbox.py
View file

@ -27,7 +27,7 @@ from dataclasses import dataclass, replace
from pathlib import Path, PurePosixPath
from .errors import IngestError, MaterializationError, SegmentationError, SourceError
from .extract import extract_text
from .extract import SourceUnits, extract_text, source_units
from .materialize import (
_render_root_frontmatter,
check_filename_length,
@ -37,7 +37,7 @@ from .materialize import (
validate_ingested_at,
write_bytes,
)
from .profiles import DEFAULT, BundleProfile, IndexEntry
from .profiles import DEFAULT, BundleProfile, IndexEntry, ProvenancePolicy
from .segmentation import (
SegmentationPlan,
SegmentEntry,
@ -111,6 +111,8 @@ def render_inbox_concept(
structure: DocumentStructure | None = None,
segment: SegmentEntry | None = None,
bundle_id: str | None = None,
units: SourceUnits | None = None,
span: tuple[int, int] | None = None,
) -> str:
"""Frame extracted text as an inbox concept file with its provenance layer.
@ -119,6 +121,12 @@ def render_inbox_concept(
`ingested_at`, on the reserved verdict layer, and on a title or
`source_file` that would break an index link or inject frontmatter lines.
`units` and `span` carry the provenance locator and are read ONLY when the
profile declares that capability. `span` defaults to the segment's own when
the concept is segmented; a whole-document concept must supply it, because
the text arriving here is the SANITIZED text and its length is not
necessarily the extracted text's.
`segment` and `bundle_id` carry the 1-to-N identity layer and are read ONLY
when the profile declares the segmentation capability. A concept the plan
does not cover keeps today's rule verbatim, and the four shipped profiles
@ -208,9 +216,78 @@ def render_inbox_concept(
frontmatter["adjudicated_by"] = verdict.adjudicated_by
frontmatter["adjudicated_at"] = verdict.adjudicated_at
frontmatter["adjudication_dwell_s"] = str(verdict.adjudication_dwell_s)
if profile.provenance is not None:
# A segment's own span is the default, but only where the segment is
# being READ -- `segmented` is the same discriminator the identity
# layer above uses, so a profile with provenance and no segmentation
# cannot silently locate by a span it is ignoring everywhere else.
located = span
if located is None and segmented:
assert segment is not None
located = segment.span
frontmatter.update(
_provenance_frontmatter(
profile.provenance,
source_file=source_file,
units=units,
span=located,
)
)
return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{_normalize_body(text)}"
# The characters that would end a YAML flow mapping early, so a path carrying
# one would produce a `sources` list that parses as something other than what
# was written. The guard refuses a quoted scalar inside a flow mapping (1.3.0,
# measured), so escaping is not on the table -- validation is.
_FLOW_TERMINATORS = ",{}[]"
def _provenance_frontmatter(
policy: ProvenancePolicy,
*,
source_file: str,
units: SourceUnits | None,
span: tuple[int, int] | None,
) -> dict[str, str]:
"""The address, and the locator when one is available.
The address is written whether or not a locator is: `sources` answers
"which document", the locator answers "where in it", and a consumer is owed
the first even when the second cannot be computed.
"""
bad = [char for char in _FLOW_TERMINATORS if char in source_file]
if bad:
raise MaterializationError(
f"source_file {source_file!r} contains {bad[0]!r}, which would end the "
"`sources` flow mapping early; this profile writes an address a "
"consumer can follow, and a path it cannot express is refused rather "
"than mangled",
code="inbox_source_file_unaddressable",
)
values = {
policy.sources_key: (
f"[{{ resource: {source_file}, title: {PurePosixPath(source_file).name} }}]"
)
}
if units is None or span is None:
return values
first, last = units.covering(*span)
if units.unit == "pages":
values[policy.pages_key] = _render_flow_list([str(first), str(last)])
elif units.unit == "rows":
scopes = units.scopes_covering(*span)
# A row number means nothing until a sheet is named, so a range that
# crosses sheets gets neither key. An absence, never a first-sheet
# guess: a guess here reads exactly like a fact.
if len(scopes) == 1 and scopes[0] is not None:
values[policy.sheet_key] = scopes[0]
values[policy.rows_key] = _render_flow_list([str(first), str(last)])
else:
values[policy.lines_key] = _render_flow_list([str(first), str(last)])
return values
# --- the guard seam -------------------------------------------------------
# The guard's non-blocking floor. `Disposition` is a `str, Enum` in
@ -542,6 +619,7 @@ def _render_segments(
profile: BundleProfile,
bundle_id: str,
source_file: str,
units: SourceUnits | None,
) -> BlockedFile | None:
"""Render every segment, or refuse the WHOLE document.
@ -607,6 +685,7 @@ def _render_segments(
structure=structure,
segment=entry,
bundle_id=bundle_id,
units=units,
),
decision.reasons,
)
@ -879,6 +958,16 @@ def process_inbox(
source_bytes,
renderer=_resolve_renderer(profile, path.name),
)
# Computed from the SAME text the plan's offsets index, so the
# locator and the offset can never disagree about which rendering
# they describe. `None` when the profile names no provenance:
# building a unit table nobody writes would re-parse every PDF for
# a key that is never emitted.
units = (
source_units(source_name(path), source_bytes, text)
if profile.provenance is not None
else None
)
covering = _plan_covering(plans, source_bytes)
if covering is not None:
blocked = _render_segments(
@ -891,6 +980,7 @@ def process_inbox(
profile=profile,
bundle_id=(root_frontmatter_values or {})[_bundle_id_key(profile)],
source_file=source_name(path),
units=units,
)
if blocked is not None:
if blocked.disposition == _DISPOSITION_QUARANTINE:
@ -937,6 +1027,12 @@ def process_inbox(
ingested_at=ingested_at,
profile=profile,
structure=structure,
units=units,
# The EXTRACTED text's span, never the sanitized
# text's: the unit table indexes the former, and a
# gate that removed a character would shift every
# unit boundary after it.
span=(0, len(text)),
),
decision.reasons,
)

47
src/llm_ingestion_okf/profiles.py
View file

@ -905,6 +905,39 @@ class SegmentationPolicy:
adjudication_key: str | None = None
@dataclass(frozen=True)
class ProvenancePolicy:
"""Whether a concept carries an address back to the document it came from.
Two layers, and the split is load-bearing rather than tidy.
The ADDRESS is SPEC's. §5.1:303-306 makes `sources[].resource` REQUIRED
within an entry and lets it be "an absolute URL, a bundle-relative path, or
a path into a `references/` subdirectory (§6)" -- which is exactly what a
dropped file's inbox-relative path is. No new key is invented where the
spec already has one.
The LOCATOR is OURS, and it has to be. §5.1 has no field for a page, a
sheet row or a line, and the guard's frontmatter grammar (1.3.0, measured)
refuses every route to putting one inside a `sources` entry: a key outside
its `sources` allowlist is rejected by name, and a nested flow list is
rejected as "a flow mapping admits scalar leaves only". So a locator inside
the entry would be a bundle we emit and could never read back through Door
C. Top-level keys, in the shape `source_offset` already uses.
Every field NAMES a key and none supplies a value, like every other policy
here. The presence of this object IS the capability: a profile that names
no provenance writes none, which is what keeps the five shipped profiles
that do not name it byte-identical.
"""
sources_key: str = "sources"
pages_key: str = "source_pages"
sheet_key: str = "source_sheet"
rows_key: str = "source_rows"
lines_key: str = "source_lines"
@dataclass(frozen=True)
class BundleProfile:
"""One bundle contract: types, frontmatter, filenames, index."""
@ -926,6 +959,11 @@ class BundleProfile:
# package; writing one is a Non-Goal and is named here as unassigned so the
# absence is deliberate rather than an oversight.
renderers: Mapping[str, str] | None = None
# Defaulted to `None` for the same reason `segmentation` is: `None` is not
# "provenance off", it is the profile not having the capability, which is
# what the door's `is not None` check reads. Five of the six shipped
# profiles leave it unset and keep their bytes.
provenance: ProvenancePolicy | None = None
# The ingest-spec + Phase 2 contract. Every value here was a constant in
@ -1281,6 +1319,15 @@ SEGMENTED_OKF_V0_2 = BundleProfile(
# attribute is typed `| None`, and the equality is asserted in the suite so
# this stays a fresh copy of the same policy plus the discriminator.
segmentation=SegmentationPolicy(adjudication_key="adjudication"),
# O3, and set on THIS profile alone. `sources` is a v0.2 key, so a profile
# stating v0.1 must not name it; `DEFAULT` and `STRICT_V1` state contracts
# owned in other repositories, so adding a key to either from here would be
# this repository editing someone else's contract (O2); and `OKF_V0_2` is
# Door A's, where `sources` is already written from the manifest. What is
# left is the segmented v0.2 profile -- the one whose concepts come from a
# dropped binary document and therefore the only one with an original to
# point at.
provenance=ProvenancePolicy(),
)

2
tests/fixtures/README.md vendored
View file

@ -17,6 +17,7 @@ and reproducible from that one file.
|---|---|
| `two-line-krav.pdf` | One heading plus one requirement row with label and value on the **same line**. That pairing is the property `pdfplumber` was chosen for. |
| `no-text-layer.pdf` | A structurally valid page with no text operators — the shape a scanned or image-only PDF presents. Must fail fast (`extractor_empty_pdf`), never persist as an empty concept. |
| `three-page-krav.pdf` | Three pages, one line of text each, and **the middle page carries no text operators**. The extractor drops empty pages, so the last page's text belongs to page 3 — which is what separates a page NUMBER from a count of the pages that produced text. Two pages could not tell those apart. |
## The office fixtures
@ -35,6 +36,7 @@ symmetric — which is most of them.
| `two-line-krav.docx` | A heading plus one requirement row with label and value on the **same line** — the docx mirror of `two-line-krav.pdf`. |
| `no-styles-krav.docx` | The **same document without `word/styles.xml`**. A negative control: the body survives and the heading marker does not, which is what proves the styles part is load-bearing rather than decoration. |
| `two-line-krav.xlsx` | A sheet name that becomes a heading, plus a label/value pair on one row. |
| `tomrad.xlsx` | Four rows with the **third one empty**. The converter renders an empty row as a pipe line of nothing but spaces, which is what a pipe table's own separator line also looks like — so a rule that reads the line rather than its position swallows the row and renumbers every row after it. Found on the K2 price sheet (8 empty rows, last row reported as 92 against a workbook that says 100); this fixture is what keeps it red. |
Two things were measured while building these, and both are the same shape —
structurally valid input, silently reduced output, exit code 0 and no warning:

87
tests/fixtures/make_fixtures.py vendored
View file

@ -37,16 +37,57 @@ KRAV_CONTENT = (
NO_TEXT_CONTENT = b"20 20 160 160 re S\n"
# One line of text per page, so the page a stretch of extracted text came from
# is decidable by reading the text alone. THREE pages rather than two, and the
# MIDDLE one carries no text operators: a two-page fixture cannot tell a page
# INDEX from a page NUMBER, and without a blank page in the middle it cannot
# tell either of them from a count of the pages that produced text. The
# extractor drops empty pages, so the third page's text belongs to page 3 and
# to no other number.
PAGED_CONTENTS = (
b"BT /F1 12 Tf 20 160 Td (Side en om helning) Tj ET\n",
b"20 20 160 160 re S\n",
b"BT /F1 12 Tf 20 160 Td (Side tre om utkiling) Tj ET\n",
)
def build_pdf(content: bytes) -> bytes:
"""Assemble a one-page PDF around `content` as the page content stream."""
return build_paged_pdf((content,))
def build_paged_pdf(contents: tuple[bytes, ...]) -> bytes:
"""Assemble a PDF with one page per entry of `contents`.
The object numbering is laid out first and the xref offsets computed from
the emitted bytes, exactly as the single-page form did -- the fixture stays
a hand-written statement about the format rather than a library's output.
"""
count = len(contents)
# 1 catalog, 2 pages, then one page object and one content stream per page,
# and the shared font last.
page_numbers = [3 + 2 * index for index in range(count)]
font_number = 3 + 2 * count
kids = b" ".join(str(number).encode() + b" 0 R" for number in page_numbers)
objects = [
b"<< /Type /Catalog /Pages 2 0 R >>",
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] "
b"/Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>",
b"<< /Length " + str(len(content)).encode() + b" >>\nstream\n" + content + b"endstream",
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>",
b"<< /Type /Pages /Kids [" + kids + b"] /Count " + str(count).encode() + b" >>",
]
for index, content in enumerate(contents):
stream_number = page_numbers[index] + 1
objects.append(
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents "
+ str(stream_number).encode()
+ b" 0 R /Resources << /Font << /F1 "
+ str(font_number).encode()
+ b" 0 R >> >> >>"
)
objects.append(
b"<< /Length " + str(len(content)).encode() + b" >>\nstream\n" + content + b"endstream"
)
objects.append(
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>"
)
out = bytearray(b"%PDF-1.4\n")
offsets = []
@ -264,6 +305,38 @@ _PRISARK_PARTS = {
}
# A sheet with an EMPTY ROW IN THE MIDDLE, which the converter renders as a
# pipe row of nothing but spaces. That row looks exactly like a table's own
# separator line to any rule that reads the line rather than its position --
# and swallowing it renumbers every row after it, silently, for the whole
# sheet. Measured on the K2 price sheet before this fixture existed: 8 empty
# rows, and the last row reported as 92 when the workbook says 100.
_TOMRAD_STRINGS = ("Rad en", "Rad to", "Rad fire")
_TOMRAD_SHEET = (
'<row r="1"><c r="A1" t="s"><v>0</v></c></row>'
'<row r="2"><c r="A2" t="s"><v>1</v></c></row>'
'<row r="3"><c r="A3"/></row>'
'<row r="4"><c r="A4" t="s"><v>2</v></c></row>'
)
_TOMRAD_PARTS = {
"[Content_Types].xml": _XLSX_PARTS["[Content_Types].xml"],
"_rels/.rels": _XLSX_PARTS["_rels/.rels"],
"xl/_rels/workbook.xml.rels": _XLSX_PARTS["xl/_rels/workbook.xml.rels"],
"xl/workbook.xml": _XML
+ '<workbook xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"'
+ ' xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">'
+ '<sheets><sheet name="Tomrad" sheetId="1" r:id="rId1"/></sheets></workbook>',
"xl/sharedStrings.xml": _XML
+ '<sst xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"'
+ f' count="{len(_TOMRAD_STRINGS)}" uniqueCount="{len(_TOMRAD_STRINGS)}">'
+ "".join(f"<si><t>{value}</t></si>" for value in _TOMRAD_STRINGS)
+ "</sst>",
"xl/worksheets/sheet1.xml": _sheet("A1:A4", _TOMRAD_SHEET),
}
def build_ooxml(parts: dict[str, str]) -> bytes:
"""Zip the parts with a fixed timestamp and no compression variance.
@ -289,11 +362,15 @@ if __name__ == "__main__":
(HERE / name).write_bytes(build_pdf(content))
print(f"wrote {name}")
(HERE / "three-page-krav.pdf").write_bytes(build_paged_pdf(PAGED_CONTENTS))
print("wrote three-page-krav.pdf")
for name, parts in (
("two-line-krav.docx", _DOCX_PARTS),
("no-styles-krav.docx", _DOCX_NO_STYLES_PARTS),
("two-line-krav.xlsx", _XLSX_PARTS),
("prisark.xlsx", _PRISARK_PARTS),
("tomrad.xlsx", _TOMRAD_PARTS),
):
(HERE / name).write_bytes(build_ooxml(parts))
print(f"wrote {name}")

54
tests/fixtures/three-page-krav.pdf vendored Normal file
View file

@ -0,0 +1,54 @@
%PDF-1.4
1 0 obj
<< /Type /Catalog /Pages 2 0 R >>
endobj
2 0 obj
<< /Type /Pages /Kids [3 0 R 5 0 R 7 0 R] /Count 3 >>
endobj
3 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 4 0 R /Resources << /Font << /F1 9 0 R >> >> >>
endobj
4 0 obj
<< /Length 50 >>
stream
BT /F1 12 Tf 20 160 Td (Side en om helning) Tj ET
endstream
endobj
5 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 6 0 R /Resources << /Font << /F1 9 0 R >> >> >>
endobj
6 0 obj
<< /Length 19 >>
stream
20 20 160 160 re S
endstream
endobj
7 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 8 0 R /Resources << /Font << /F1 9 0 R >> >> >>
endobj
8 0 obj
<< /Length 52 >>
stream
BT /F1 12 Tf 20 160 Td (Side tre om utkiling) Tj ET
endstream
endobj
9 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>
endobj
xref
0 10
0000000000 65535 f
0000000009 00000 n
0000000058 00000 n
0000000127 00000 n
0000000253 00000 n
0000000352 00000 n
0000000478 00000 n
0000000546 00000 n
0000000672 00000 n
0000000773 00000 n
trailer
<< /Size 10 /Root 1 0 R >>
startxref
870
%%EOF

BIN
tests/fixtures/tomrad.xlsx vendored Normal file
View file

Binary file not shown.

6
tests/test_profile.py
View file

@ -234,4 +234,10 @@ def test_a_profile_is_assembled_from_its_policies() -> None:
# profile does not have the capability; every shipped profile still
# constructs unchanged and no golden moved.
"renderers",
# O3's capability, and the same deliberate update again: a concept that
# points back at the document it came from is a contract question, so
# the alternative to a policy on the object is a door branching on which
# profile it was handed. Set on `SEGMENTED_OKF_V0_2` alone; `None`
# everywhere else, so five shipped profiles construct unchanged.
"provenance",
}

329
tests/test_provenance.py Normal file
View file

@ -0,0 +1,329 @@
"""Provenance to the original: `sources` plus a per-format locator (O3).
A concept says what it was extracted FROM. Before this step it said so with a
`source_file` basename, a `source_sha256`, and — when segmented — a
`source_offset` that indexes the EXTRACTED text rather than the original, so a
consumer could not open the original at the right place without knowing the
corpus directory and re-running the extractor.
Two layers, and the split is the whole design:
- the ADDRESS is spec's, `sources[].resource` (SPEC v0.2 §5.1:303-306: "an
absolute URL, a bundle-relative path, or a path into a `references/`
subdirectory"), emitted in flow form because this library's parser cannot
read a block one back;
- the LOCATOR is ours, because §5.1 has no field for a page, a sheet row or a
line, and the guard's frontmatter grammar refuses both a non-allowlisted key
inside a `sources` entry and a nested flow list — measured here, so the
choice is a recorded constraint rather than a preference.
"""
from __future__ import annotations
import importlib.util
import warnings
from pathlib import Path
import pytest
from llm_ingestion_okf.extract import extract_text, source_units
from llm_ingestion_okf.inbox import render_inbox_concept
from llm_ingestion_okf.profiles import (
DEFAULT,
SEGMENTED_OKF_V0_2,
SEGMENTED_V1,
STRICT_V1,
STRUCTURED_V1,
)
from llm_ingestion_okf.segmentation import SegmentEntry
FIXTURES = Path(__file__).parent / "fixtures"
requires_extract = pytest.mark.skipif(
importlib.util.find_spec("pdfplumber") is None,
reason="the optional [extract] extra is not installed",
)
def _extract(name: str) -> tuple[bytes, str]:
data = (FIXTURES / name).read_bytes()
with warnings.catch_warnings():
warnings.simplefilter("ignore")
return data, extract_text(name, data)
def _segment(span: tuple[int, int]) -> SegmentEntry:
return SegmentEntry(
segment_id="p1",
path="a/b.md",
title="A segment",
okf_type="reference",
span=span,
ingested_at="2026-09-08T00:00:00Z",
)
# --- the unit table: a character range -> a place in the ORIGINAL ---------
@requires_extract
def test_a_pdf_reports_the_page_a_character_range_came_from() -> None:
# The middle page carries no text and the extractor drops it, so the third
# page's text is page 3 and not page 2. A fixture without that gap could
# not tell a page NUMBER from a count of the pages that produced text.
data, text = _extract("three-page-krav.pdf")
units = source_units("three-page-krav.pdf", data, text)
assert units is not None
assert units.unit == "pages"
assert units.covering(0, len("Side en om helning")) == (1, 1)
assert units.covering(text.index("Side tre"), len(text)) == (3, 3)
assert units.covering(0, len(text)) == (1, 3)
# The BOUNDARY, and it is the assertion that has to exist: the blank line
# the extractor joins pages with belongs to neither page's text, so an
# offset inside it is still the page before. A table built without the
# separator's own length passes every assertion above and fails this one,
# because the drift is two characters per page and only shows up where a
# page begins.
last_page_begins = text.index("Side tre")
assert units.covering(last_page_begins - 1, last_page_begins) == (1, 1)
@requires_extract
def test_a_spreadsheet_reports_the_sheet_and_the_rows_of_the_original() -> None:
# `prisark.xlsx` is hand-laid: sheet 1 spans A1:C6, sheet 2 spans A1:A3.
# The converter renders each sheet as a heading plus a pipe table, and
# numbering restarts per sheet — so the last row of the file is row 3 of
# sheet 2, not row 9 of the document.
data, text = _extract("prisark.xlsx")
units = source_units("prisark.xlsx", data, text)
assert units is not None
assert units.unit == "rows"
first_row = text.index("| Prisskjema")
assert units.covering(first_row, first_row + 1) == (1, 1)
assert units.scope_of(first_row) == "Prisark"
assert units.covering(0, len(text)) == (1, 3)
assert units.scopes_covering(0, len(text)) == ("Prisark", "Enkeltkolonne")
@requires_extract
def test_a_row_locator_counts_the_separator_line_as_no_row() -> None:
# The pipe table's `|----|` line is a row of no spreadsheet. Counting it
# would push every row after it up by one, silently, for the whole sheet.
data, text = _extract("prisark.xlsx")
units = source_units("prisark.xlsx", data, text)
assert units is not None
header = text.index("| Prisskjema")
second = text.index("| Post ")
assert units.covering(header, second + 1) == (1, 2)
def test_an_empty_row_still_counts_as_a_row_of_the_sheet() -> None:
# `tomrad.xlsx` is four rows with the THIRD empty. The converter renders an
# empty row as a pipe line of nothing but spaces, which is what a table's
# own separator line also looks like — so a rule that reads the LINE rather
# than its POSITION swallows the empty row and renumbers every row after
# it, for the whole sheet, silently.
#
# Found on the K2 price sheet, not here: 8 empty rows, and the last row
# reported as 92 against a workbook that says 100.
data, text = _extract("tomrad.xlsx")
units = source_units("tomrad.xlsx", data, text)
assert units is not None
assert units.numbers == (1, 2, 3, 4)
assert units.covering(text.index("Rad fire"), len(text)) == (4, 4)
def test_a_text_file_reports_line_numbers_of_the_original() -> None:
data = b"first\nsecond\nthird\n"
text = extract_text("note.md", data)
units = source_units("note.md", data, text)
assert units is not None
assert units.unit == "lines"
assert units.covering(0, 5) == (1, 1)
assert units.covering(text.index("third"), len(text)) == (3, 3)
assert units.covering(0, len(text)) == (1, 3)
def test_a_docx_locator_is_lines_because_paragraphs_do_not_survive() -> None:
# Measured on the five K2 `.docx` documents: `<w:p>` counts of 108, 27, 65,
# 176 and 57 against converted-markdown line counts of 75, 33, 67, 144 and
# 63. Not one pair agrees, so a `paragraphs` key would name a number the
# original does not have. `lines` says what it indexes.
data, text = _extract("two-line-krav.docx") if _has_converter() else (b"", "")
if not text:
pytest.skip("the optional [extract] extra is not installed")
units = source_units("two-line-krav.docx", data, text)
assert units is not None
assert units.unit == "lines"
def _has_converter() -> bool:
return importlib.util.find_spec("pypandoc") is not None
# --- the frontmatter a concept carries ------------------------------------
@requires_extract
def test_a_segmented_concept_points_at_the_original_and_its_pages() -> None:
data, text = _extract("three-page-krav.pdf")
units = source_units("three-page-krav.pdf", data, text)
document = render_inbox_concept(
text[20:],
okf_type="reference",
title="Side tre",
source_file="mappe/three-page-krav.pdf",
source_bytes=data,
ingested_at="2026-09-08T00:00:00Z",
profile=SEGMENTED_OKF_V0_2,
segment=_segment((20, len(text))),
bundle_id="b1",
units=units,
)
assert "sources: [{ resource: mappe/three-page-krav.pdf, title: three-page-krav.pdf }]\n" in (
document
)
assert "source_pages: [3, 3]\n" in document
# The offset stays: it is what an existing consumer joins on, and a
# locator that replaced it would break them to fix them.
assert "source_offset: [20, 40]\n" in document
@requires_extract
def test_a_whole_document_concept_gets_the_pages_it_spans() -> None:
# The order's known-negative: a concept the plan does not cover is the
# WHOLE document, so its locator is every page that produced text.
data, text = _extract("three-page-krav.pdf")
units = source_units("three-page-krav.pdf", data, text)
document = render_inbox_concept(
text,
okf_type="reference",
title="Hele",
source_file="three-page-krav.pdf",
source_bytes=data,
ingested_at="2026-09-08T00:00:00Z",
profile=SEGMENTED_OKF_V0_2,
units=units,
span=(0, len(text)),
)
assert "source_pages: [1, 3]\n" in document
assert "source_offset" not in document
@requires_extract
def test_a_spreadsheet_concept_names_the_sheet_and_its_rows() -> None:
data, text = _extract("prisark.xlsx")
units = source_units("prisark.xlsx", data, text)
document = render_inbox_concept(
text,
okf_type="reference",
title="Hele arket",
source_file="prisark.xlsx",
source_bytes=data,
ingested_at="2026-09-08T00:00:00Z",
profile=SEGMENTED_OKF_V0_2,
units=units,
span=(0, text.index("| Post ")),
)
assert "source_sheet: Prisark\n" in document
assert "source_rows: [1, 1]\n" in document
@requires_extract
def test_a_range_spanning_two_sheets_names_no_sheet_and_no_rows() -> None:
# A row number is only a place in the original once a sheet is named. A
# range covering two sheets has no single sheet, so it gets no row
# locator either — an absence, never a first-sheet guess.
data, text = _extract("prisark.xlsx")
units = source_units("prisark.xlsx", data, text)
document = render_inbox_concept(
text,
okf_type="reference",
title="Begge ark",
source_file="prisark.xlsx",
source_bytes=data,
ingested_at="2026-09-08T00:00:00Z",
profile=SEGMENTED_OKF_V0_2,
units=units,
span=(0, len(text)),
)
assert "source_sheet" not in document
assert "source_rows" not in document
assert "sources: [{ resource: prisark.xlsx, title: prisark.xlsx }]\n" in document
def test_the_address_is_written_even_when_no_locator_is_available() -> None:
# `sources` answers "which file", the locator answers "where in it". The
# first must not depend on the second: a caller with no unit table still
# owes a consumer the address.
document = render_inbox_concept(
"body\n",
okf_type="reference",
title="T",
source_file="sub/dir/note.md",
source_bytes=b"body\n",
ingested_at="2026-09-08T00:00:00Z",
profile=SEGMENTED_OKF_V0_2,
)
assert "sources: [{ resource: sub/dir/note.md, title: note.md }]\n" in document
assert "source_pages" not in document
assert "source_lines" not in document
# --- what must not move ----------------------------------------------------
@pytest.mark.parametrize("profile", [DEFAULT, STRUCTURED_V1, STRICT_V1, SEGMENTED_V1])
def test_a_profile_that_names_no_provenance_writes_none(profile: object) -> None:
# Support is additive: five shipped profiles keep their bytes, and only the
# profile the order targets moves. A key written unconditionally here would
# churn every golden in the suite.
assert getattr(profile, "provenance") is None
def test_the_shipped_profiles_that_move_are_exactly_one() -> None:
assert SEGMENTED_OKF_V0_2.provenance is not None
for profile in (DEFAULT, STRUCTURED_V1, STRICT_V1, SEGMENTED_V1):
assert profile.provenance is None
def test_a_source_file_that_would_break_the_flow_mapping_is_refused() -> None:
# Validation, not repair, and not silence: the emitted value is a YAML flow
# mapping, so a comma or a brace in the path would terminate the entry
# early and produce a `sources` list that parses as something else.
from llm_ingestion_okf.errors import MaterializationError
with pytest.raises(MaterializationError) as excinfo:
render_inbox_concept(
"body\n",
okf_type="reference",
title="T",
source_file="Del II, Bilag.pdf",
source_bytes=b"body\n",
ingested_at="2026-09-08T00:00:00Z",
profile=SEGMENTED_OKF_V0_2,
)
assert excinfo.value.code == "inbox_source_file_unaddressable"
def test_the_guard_parses_the_sources_form_this_door_emits() -> None:
# The published promise this test exists to keep red-able: what Door B
# writes must survive the guard's own frontmatter grammar, or a bundle we
# emit could never be read back through Door C.
okf = pytest.importorskip("llm_ingestion_guard.okf")
document = render_inbox_concept(
"body\n",
okf_type="reference",
title="T",
source_file="Del II Bilag 3.3.1 - Brannkonsept.pdf",
source_bytes=b"body\n",
ingested_at="2026-09-08T00:00:00Z",
profile=SEGMENTED_OKF_V0_2,
)
frontmatter, _ = okf.parse_frontmatter(document)
assert frontmatter["sources"] == [
{
"resource": "Del II Bilag 3.3.1 - Brannkonsept.pdf",
"title": "Del II Bilag 3.3.1 - Brannkonsept.pdf",
}
]