llm-ingestion-okf/tests
Kjell Tore Guttormsen bc39e8091f feat(assets): a bundle carries the images its sources declare (0.10.0)
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 10:01:31 +02:00
..
fixtures feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_adjudicate.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_asset_bundle.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_asset_gate.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_asset_import.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_asset_structure.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_assets.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_attested_computation.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_block_sources_flat_readers.py fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped 2026-09-12 16:42:22 +02:00
test_bold_title.py feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word 2026-09-09 23:05:45 +02:00
test_bundle_identity.py feat(consume): parent reaches the reader -- excerpt field, body link, checker rule 2026-09-11 12:36:23 +02:00
test_cid_measure.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_cli_build.py feat(cli,consume): the first span, and the fusion that punished fine-graining 2026-09-09 15:40:01 +02:00
test_cli_gate.py fix(gate,propose): okf build runs a real guard; a code fence declares no structure 2026-09-16 00:19:50 +02:00
test_code_fence.py fix(gate,propose): okf build runs a real guard; a code fence declares no structure 2026-09-16 00:19:50 +02:00
test_contents_name_part.py feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
test_contract_check.py feat(check): a skill and a payload naming different bundles is a finding 2026-09-10 23:42:06 +02:00
test_converter_attribute_titles.py feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
test_corpus_run.py fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index 2026-09-08 10:06:58 +02:00
test_default_bundle_pin.py feat(consume): one source document took 8 of 8 delivered places, so cap it 2026-09-10 00:29:02 +02:00
test_depth_seven.py test(extract): a section below markdown's sixth level keeps its own level in the plan 2026-09-11 14:02:49 +02:00
test_docs_promises.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_document_identity.py feat(identity): an STS document's doc-number names its directory and its title the address 2026-09-11 03:08:30 +02:00
test_document_prior.py feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment 2026-09-09 14:17:00 +02:00
test_error_codes.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_exception_chaining.py test(errors): pin __cause__ preservation on every fail-fast wrap 2026-07-23 07:02:47 +02:00
test_extract.py test(extract): the file that IS the product had no reader, so state what one owes 2026-09-10 03:30:48 +02:00
test_faceted_index.py feat(profiles): a faceted index policy and the additive STRUCTURED_V1 profile 2026-08-27 00:30:42 +02:00
test_fidelity.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_file_connector.py feat(connectors): add the file connector with fail-closed path boundary 2026-07-16 19:55:08 +02:00
test_first_span.py feat(cli,consume): the first span, and the fusion that punished fine-graining 2026-09-09 15:40:01 +02:00
test_frontmatter_nesting.py fix(frontmatter): a nested key must not substitute for a top-level one 2026-08-31 23:31:53 +02:00
test_golden.py feat(okf-v0.2): D5 — the v0.2 golden fixture, with okf_version in root frontmatter 2026-07-31 17:27:48 +02:00
test_guard_adapter.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_http_connector.py feat(connectors): add the http connector and wire the network gate 2026-07-16 20:06:21 +02:00
test_import_consumable.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_import_facets.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_import_flow.py feat(import): Door C flow against an injected import gate (Phase 2 step 5) 2026-07-25 06:57:25 +02:00
test_inbox.py feat(inbox): fail-fast on filenames over the 255-byte limit 2026-07-25 06:27:43 +02:00
test_inbox_flow.py feat(inbox): walk the drop directory recursively 2026-09-07 04:11:00 +02:00
test_inbox_recursion.py feat(inbox): walk the drop directory recursively 2026-09-07 04:11:00 +02:00
test_inbox_structure.py feat(inbox): Door B derives structure and reprojects the index additively 2026-08-27 00:37:12 +02:00
test_index.py feat(index): maintain existing indexes on re-materialization (spec §6) 2026-07-16 20:04:21 +02:00
test_index_policy.py fix(profiles): P1-F1 — a permitted root key is not a required one 2026-07-31 18:16:13 +02:00
test_index_sort.py test(index): pin Door C's cross-run ordering bound 2026-09-01 19:58:19 +02:00
test_index_star_row.py feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment 2026-09-09 14:17:00 +02:00
test_k2_office_fixtures.py feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
test_link_line_signal.py feat(consume): the door's link line is out of the default reading 2026-09-12 18:56:53 +02:00
test_load_bearing.py feat(profiles): DEFAULT stamps commons' O2 generated, V1 executed 2026-08-09 12:29:05 +02:00
test_manifest.py feat(manifest): reject [/] in extraction title (ingest-spec §4) 2026-07-23 21:33:47 +02:00
test_materialize.py feat(profiles): DEFAULT stamps commons' O2 generated, V1 executed 2026-08-09 12:29:05 +02:00
test_md_declared_headings.py feat(propose,cli): the coverage gap had one cause, and round 7's own decomposition did not reproduce 2026-09-09 17:43:56 +02:00
test_multi_source_provenance.py fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped 2026-09-12 16:42:22 +02:00
test_okf_consume.py fix(skill): regenerate skills/okf-consume from the golden bundle it ships a payload for 2026-09-11 02:21:35 +02:00
test_okf_consume_shim.py feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word 2026-09-09 23:05:45 +02:00
test_okf_project.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_okf_skill.py feat(check): a skill and a payload naming different bundles is a finding 2026-09-10 23:42:06 +02:00
test_okf_v0_2_characterization.py fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped 2026-09-12 16:42:22 +02:00
test_okf_v0_2_profile.py feat(profiles): DEFAULT stamps commons' O2 generated, V1 executed 2026-08-09 12:29:05 +02:00
test_okf_watch.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_outline_gate.py feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment 2026-09-09 14:17:00 +02:00
test_outline_measure.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_packaging.py feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
test_pandoc_binary.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_parent_index.py test(structure): the index resolves a parent naming a segment of its own document 2026-09-11 13:48:11 +02:00
test_parent_reaches_reader.py feat(consume): parent reaches the reader -- excerpt field, body link, checker rule 2026-09-11 12:36:23 +02:00
test_parent_text.py docs(consume,build): both parent defaults stay off, and the reason is now a measurement 2026-09-11 13:39:50 +02:00
test_pdf_font_and_ocr.py feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
test_pdf_heading_reserve.py feat(propose,cli): typography as a reserve, and the two of our own numbers it took to measure it 2026-09-09 00:25:51 +02:00
test_pdf_outline.py test(pdf): two bookmarks on one line, and nothing counts the one that is lost 2026-09-10 03:16:14 +02:00
test_profile.py feat(inbox): point every concept at the document it came from, with a locator per format 2026-09-08 14:39:24 +02:00
test_profile_threading.py test(profiles): pin the re-run promise V1 was published under 2026-08-09 12:39:23 +02:00
test_propose_segments.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_propose_sheet_sections.py feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
test_provenance.py fix(extract): HTML collapsed to one line, so 828 of 828 sections had no boundary 2026-09-09 23:57:47 +02:00
test_quality.py feat(quality): okf quality --fasit, boundary recall against a declared structure 2026-09-13 07:27:33 +02:00
test_render.py feat(render): add §5 body renderers as pure functions 2026-07-16 19:52:21 +02:00
test_render_hook.py feat(profiles): a profile may name a per-suffix renderer 2026-09-02 14:19:18 +02:00
test_root_frontmatter_emission.py feat(okf-v0.2): D5 — the v0.2 golden fixture, with okf_version in root frontmatter 2026-07-31 17:27:48 +02:00
test_round7_defaults.py feat(cli,consume): the first span, and the fusion that punished fine-graining 2026-09-09 15:40:01 +02:00
test_run_frontmatter.py fix(frontmatter): write a value a YAML reader reads back, and read both forms 2026-09-11 11:09:49 +02:00
test_segmentation.py feat(segmentation): parse the adjudication state a plan already carries 2026-09-02 14:44:06 +02:00
test_segmented_collisions.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_segmented_golden.py feat(examples): SEGMENTED_V1 golden fixture and decision record 2026-09-01 00:31:15 +02:00
test_segmented_identity.py fix(segmentation): hash the extracted text and let the plan key fire 2026-09-02 14:38:20 +02:00
test_segmented_inbox.py feat(inbox): accept one segmentation plan per document 2026-09-02 14:51:53 +02:00
test_segmented_index.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_segmented_okf_v0_2_golden.py test(examples): byte-pinned golden for SEGMENTED_OKF_V0_2 2026-09-02 14:24:05 +02:00
test_segmented_profile.py feat(profiles): SEGMENTED_OKF_V0_2 so a segmented bundle can declare its spec 2026-09-02 14:21:15 +02:00
test_segmented_rebuild.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_segmented_rounds.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_shared_id_prefix.py test(consume): a directory every concept shares must not order them 2026-09-11 04:43:37 +02:00
test_shell_parent.py fix(structure): the index resolves a parent naming a segment of its own document 2026-09-11 13:55:03 +02:00
test_span_gaps.py feat(propose,cli): the coverage gap had one cause, and round 7's own decomposition did not reproduce 2026-09-09 17:43:56 +02:00
test_sql_connector.py feat(connectors): add the sql connector (read-only sqlite, env-resolved) 2026-07-16 19:56:41 +02:00
test_stem_prefix.py feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word 2026-09-09 23:05:45 +02:00
test_strict_v1.py feat(phase-3): the index policy becomes configurable, with the reader that judges it 2026-07-25 20:30:54 +02:00
test_structure.py fix(structure): a derived reference needs a cue, not just a number shape 2026-08-29 09:26:57 +02:00
test_sts_description.py test(description): a spec point a YAML reader cannot read verbatim is not written 2026-09-11 03:46:11 +02:00
test_table_measure.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_tie_shared_rank.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_title_covered.py test(consume): the lookup partition still wins, and the CLI defaults agree with the signature 2026-09-10 22:20:42 +02:00
test_title_covered_rise.py test(consume): a short title the question happens to cover passes the section it names 2026-09-11 01:23:14 +02:00
test_xml_declared_route.py test(propose): a declared XML structure does not take the route declared structure takes 2026-09-10 06:55:00 +02:00
test_xml_sts.py test(extract): an exact invariant does not get to be 0.999998 2026-09-10 04:07:35 +02:00
test_yaml_frontmatter.py fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped 2026-09-12 16:42:22 +02:00