open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

test(extract): the file that IS the product had no reader, so state what one owes

Browse source 
RED, 11 failing. Reproduced first, before any code: `okf build` on the folder
a publisher's own viewer delivers gives **110 of 110 unreadable, 0 plans, exit
2**, with `no extractor is registered for file extension '.xml'`. The
conservation identity `merged + coded rejections == N` is never written at all
-- the run aborts earlier on `FAILED - no segmentation plans` -- so the hole
was not even visible as a count.

The one xml file IS the whole product: R761 Prosesskoden:2025, the document
round 12 met as a 701-page PDF, in NISO-STS form. Measured on the file with
`xml.etree.ElementTree`: 7 715 `<sec>`, 2 761 with a `<title>`, 4 954 with a
`<label>` and no title, 10 `<table-wrap>`, 12 528 `<p>`, root `<standard>`, 0
`<!DOCTYPE` and 0 `<!ENTITY`. Its `<sec>`-nesting depths over the titled
sections are 28/118/500/1141/868/97/9 -- row for row the fasit's own
distribution. The ceiling is therefore structural rather than computed.

FOUR HAND-WRITTEN FIXTURES, none through `make_fixtures.py` and none
serialised by `ElementTree`: a library that writes and reads its own format
proves only that it agrees with itself. A known-positive STS mini, generic
non-STS xml, a `<!DOCTYPE` with an entity expansion, and a malformed file.

Two assertions that already existed are extended rather than duplicated: the
converter fence, because a file routed to the converter is read by a second
parser that never sees this reader's DTD refusal, and the evidence table,
because a row without a class is the failure that test exists for.

pytest -q: 11 failed, 1554 passed, 1 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 03:30:48 +02:00
commit 324cd0b1e5
7 changed files with 306 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

16
tests/test_extract.py
View file

@ -315,17 +315,24 @@ def test_every_office_row_names_its_reader() -> None:
}
def test_html_and_epub_are_excluded_from_the_converter_on_purpose() -> None:
def test_html_epub_and_xml_are_excluded_from_the_converter_on_purpose() -> None:
"""`.html` has a stdlib extractor; routing it through the converter would
buy nothing and would add CVE-2025-51591 (SSRF via an iframe in HTML
input), which is unpatched in every converter version. `.epub` is out for
the same "no gain" half of that reason.
`.xml` joins them on that same half PLUS one of its own: the XML reader
here refuses a DTD outright, and a file routed to the converter would be
read by a different parser that never sees that refusal. The hardening
would be true of the code and false of the file.
"""
from llm_ingestion_okf.extract import _CORE_EXTRACTORS, _PANDOC_FORMATS
assert ".html" not in _PANDOC_FORMATS
assert ".epub" not in _PANDOC_FORMATS
assert ".xml" not in _PANDOC_FORMATS
assert ".html" in _CORE_EXTRACTORS
assert ".xml" in _CORE_EXTRACTORS
def test_evidence_class_is_asserted_not_commented() -> None:
@ -345,12 +352,17 @@ def test_evidence_class_is_asserted_not_commented() -> None:
"""
from llm_ingestion_okf.extract import _EVIDENCE, _PANDOC_FORMATS
assert set(_EVIDENCE) == set(_PANDOC_FORMATS) | {".html", ".pdf"}, "every row needs a class"
assert set(_EVIDENCE) == set(_PANDOC_FORMATS) | {
".html",
".pdf",
".xml",
}, "every row needs a class"
assert {s for s, e in _EVIDENCE.items() if e == "measured"} == {
".pdf",
".docx",
".xlsx",
".html",
".xml",
}
# Since 2026-09-09 the three office rows are `constructed`, not
# `unmeasured`: each has now been put through end to end on a hand-built