llm-ingestion-okf/tests
Kjell Tore Guttormsen cd7b792aaf feat(extract): five office formats through a table-driven converter seam
`_PANDOC_FORMATS` names the rows and no others: docx, xlsx, pptx, odt, rtf.
`.html` stays on its stdlib extractor -- routing it through the converter would
buy nothing and would add CVE-2025-51591 (SSRF via an iframe in HTML input),
unpatched in every converter version. `.epub` is out on the "no gain" half of
that.

`_EVIDENCE` records what each row rests on, asserted in the suite rather than
written in a comment: docx and xlsx are `measured`, and pptx, odt and rtf are
`unmeasured` because the corpus contains ZERO files of those types. Three of
five rows therefore leave this step working by construction and never checked
against a document anyone wrote, and the assertion is what keeps that visible.

Three converter arguments, all measured and none of them hygiene:
`--eol=lf --wrap=none` because the defaults produce different bytes (max line
length 75 against 447), and `-t markdown` never `-t plain` because plain
destroys the headings the segment proposer reads -- 15 entries with two real
headings become 13 with none.

`_UNPARSED_OPTIONAL_EXTENSIONS` is now empty and kept rather than deleted: the
branch still raises, and a future type arriving before its reader belongs there
rather than in a new mechanism. This is what the first step was for -- both
tests for `extractor_extra_missing` were repointed at the import probe before
the set emptied under them.

The converter call is isolated behind `_convert_bytes` so the seam's own logic
is testable without the binary; the conversion itself is pinned by frozen-text
fixtures in the next step. Checked live against a hand-laid docx through the
real vendored binary: heading and body both survive.

Suite 895 -> 908.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-02 14:09:49 +02:00
..
fixtures feat(extract): implement pdf behind the [extract] extra with pdfplumber 2026-08-21 20:22:39 +02:00
test_attested_computation.py fix(frontmatter): a nested key must not substitute for a top-level one 2026-08-31 23:31:53 +02:00
test_error_codes.py feat(errors): register the four converter codes 2026-09-02 14:01:49 +02:00
test_exception_chaining.py test(errors): pin __cause__ preservation on every fail-fast wrap 2026-07-23 07:02:47 +02:00
test_extract.py feat(extract): five office formats through a table-driven converter seam 2026-09-02 14:09:49 +02:00
test_faceted_index.py feat(profiles): a faceted index policy and the additive STRUCTURED_V1 profile 2026-08-27 00:30:42 +02:00
test_file_connector.py feat(connectors): add the file connector with fail-closed path boundary 2026-07-16 19:55:08 +02:00
test_frontmatter_nesting.py fix(frontmatter): a nested key must not substitute for a top-level one 2026-08-31 23:31:53 +02:00
test_golden.py feat(okf-v0.2): D5 — the v0.2 golden fixture, with okf_version in root frontmatter 2026-07-31 17:27:48 +02:00
test_guard_adapter.py build(deps): move the guard pin from >=0.3,<0.4 to >=1.2,<2.0 2026-08-23 21:05:18 +02:00
test_http_connector.py feat(connectors): add the http connector and wire the network gate 2026-07-16 20:06:21 +02:00
test_import_facets.py docs(plan): record the Door C facet decisions, the mutation result, and the harness that lied 2026-08-27 11:03:37 +02:00
test_import_flow.py feat(import): Door C flow against an injected import gate (Phase 2 step 5) 2026-07-25 06:57:25 +02:00
test_inbox.py feat(inbox): fail-fast on filenames over the 255-byte limit 2026-07-25 06:27:43 +02:00
test_inbox_flow.py feat(inbox): Door B flow against an injected guard gate (Phase 2 step 3) 2026-07-25 06:35:55 +02:00
test_inbox_structure.py feat(inbox): Door B derives structure and reprojects the index additively 2026-08-27 00:37:12 +02:00
test_index.py feat(index): maintain existing indexes on re-materialization (spec §6) 2026-07-16 20:04:21 +02:00
test_index_policy.py fix(profiles): P1-F1 — a permitted root key is not a required one 2026-07-31 18:16:13 +02:00
test_index_sort.py test(index): pin Door C's cross-run ordering bound 2026-09-01 19:58:19 +02:00
test_load_bearing.py feat(profiles): DEFAULT stamps commons' O2 generated, V1 executed 2026-08-09 12:29:05 +02:00
test_manifest.py feat(manifest): reject [/] in extraction title (ingest-spec §4) 2026-07-23 21:33:47 +02:00
test_materialize.py feat(profiles): DEFAULT stamps commons' O2 generated, V1 executed 2026-08-09 12:29:05 +02:00
test_okf_v0_2_characterization.py fix(frontmatter): a nested key must not substitute for a top-level one 2026-08-31 23:31:53 +02:00
test_okf_v0_2_profile.py feat(profiles): DEFAULT stamps commons' O2 generated, V1 executed 2026-08-09 12:29:05 +02:00
test_okf_watch.py feat(watch): weekly OKF upstream watch that can prove it found nothing 2026-08-23 20:38:37 +02:00
test_packaging.py build(extract): vendor the converter binary as a pinned wheel 2026-09-02 14:03:23 +02:00
test_pandoc_binary.py feat(extract): resolve the vendored converter and refuse an unpinned version 2026-09-02 14:07:20 +02:00
test_profile.py feat(profiles): SEGMENTED_V1 profile and SegmentationPolicy capability 2026-09-01 00:02:14 +02:00
test_profile_threading.py test(profiles): pin the re-run promise V1 was published under 2026-08-09 12:39:23 +02:00
test_propose_segments.py feat(tools): segmentation proposer emitting adjudicable PROPOSED entries 2026-09-01 00:28:41 +02:00
test_render.py feat(render): add §5 body renderers as pure functions 2026-07-16 19:52:21 +02:00
test_root_frontmatter_emission.py feat(okf-v0.2): D5 — the v0.2 golden fixture, with okf_version in root frontmatter 2026-07-31 17:27:48 +02:00
test_segmentation.py feat(segmentation): slice canonical extracted text by declared spans 2026-08-31 23:59:05 +02:00
test_segmented_collisions.py feat(inbox): collision gate over segment paths and additive concept reporting 2026-09-01 00:14:13 +02:00
test_segmented_golden.py feat(examples): SEGMENTED_V1 golden fixture and decision record 2026-09-01 00:31:15 +02:00
test_segmented_identity.py fix(inbox): a segment title is declared, never marked derived 2026-09-01 19:29:46 +02:00
test_segmented_inbox.py fix(inbox): refuse a segmentation plan that matches no dropped file 2026-09-01 19:27:27 +02:00
test_segmented_index.py feat(index): one ordering helper, called by both doors 2026-09-01 19:54:45 +02:00
test_segmented_profile.py feat(profiles): SEGMENTED_V1 profile and SegmentationPolicy capability 2026-09-01 00:02:14 +02:00
test_segmented_rebuild.py test(inbox): rebuild equals incremental across a re-split with varying timestamps 2026-09-01 00:25:35 +02:00
test_segmented_rounds.py feat(inbox): own concepts by source_file and retire stale segments 2026-09-01 00:17:04 +02:00
test_sql_connector.py feat(connectors): add the sql connector (read-only sqlite, env-resolved) 2026-07-16 19:56:41 +02:00
test_strict_v1.py feat(phase-3): the index policy becomes configurable, with the reader that judges it 2026-07-25 20:30:54 +02:00
test_structure.py fix(structure): a derived reference needs a cue, not just a number shape 2026-08-29 09:26:57 +02:00