test(extract): the file that IS the product had no reader, so state what one owes

RED, 11 failing. Reproduced first, before any code: `okf build` on the folder
a publisher's own viewer delivers gives **110 of 110 unreadable, 0 plans, exit
2**, with `no extractor is registered for file extension '.xml'`. The
conservation identity `merged + coded rejections == N` is never written at all
-- the run aborts earlier on `FAILED - no segmentation plans` -- so the hole
was not even visible as a count.

The one xml file IS the whole product: R761 Prosesskoden:2025, the document
round 12 met as a 701-page PDF, in NISO-STS form. Measured on the file with
`xml.etree.ElementTree`: 7 715 `<sec>`, 2 761 with a `<title>`, 4 954 with a
`<label>` and no title, 10 `<table-wrap>`, 12 528 `<p>`, root `<standard>`, 0
`<!DOCTYPE` and 0 `<!ENTITY`. Its `<sec>`-nesting depths over the titled
sections are 28/118/500/1141/868/97/9 -- row for row the fasit's own
distribution. The ceiling is therefore structural rather than computed.

FOUR HAND-WRITTEN FIXTURES, none through `make_fixtures.py` and none
serialised by `ElementTree`: a library that writes and reads its own format
proves only that it agrees with itself. A known-positive STS mini, generic
non-STS xml, a `<!DOCTYPE` with an entity expansion, and a malformed file.

Two assertions that already existed are extended rather than duplicated: the
converter fence, because a file routed to the converter is read by a second
parser that never sees this reader's DTD refusal, and the evidence table,
because a row without a class is the failure that test exists for.

pytest -q: 11 failed, 1554 passed, 1 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

tests/fixtures/README.md

@@ -20,6 +20,22 @@ and reproducible from that one file.
 | `outline-collision.pdf` | **Two bookmarks whose destinations resolve to the same line** — the tree's root node and a front-matter node, both on line 0, which is the shape R761 carries. The marks are collected in a dict keyed on the line index, so without this fixture the second node is dropped with nothing counting it: 2 763 nodes in, 2 762 marks out, `unresolved` at 0. A one-bookmark-per-line fixture cannot see that. |
 | `three-page-krav.pdf` | Three pages, one line of text each, and **the middle page carries no text operators**. The extractor drops empty pages, so the last page's text belongs to page 3 — which is what separates a page NUMBER from a count of the pages that produced text. Two pages could not tell those apart. |
 
+## The XML fixtures
+
+Hand-written text, directly in this directory — **not** through
+`make_fixtures.py`, which builds binary containers, and never serialised by
+`ElementTree`. The reason is the same one the OOXML policy states: a library
+that writes and then reads its own format proves only that it agrees with
+itself, and an `ElementTree`-written fixture would stay green through any
+round-trip-symmetric defect.
+
+| Fixture | What it is for |
+|---|---|
+| `sts-mini.xml` | The known-positive. A `<standard>` root with `<sec>` at three nesting levels carrying `<label>`+`<title>`, two lettered points (`a)`, `b)`) with a **label and no title**, one `<table-wrap>` with a label and two rows, `<p>` bodies, a `<list>`, and **one unnumbered section** (`Forord`, `<title>` with no `<label>`) mirroring the single such section in R761. The lettered points are what the 4 954 label-only `<sec>` in that document look like: promoted to headings they would bury its own 2 761. |
+| `generic-feed.xml` | Known-negative: XML that is **not** STS. It must produce text and ONE plan — never zero, never a crash, and never element names promoted to headings. |
+| `xml-doctype-bomb.xml` | Known-negative, security: a `<!DOCTYPE` with a small nested-entity expansion. It must be refused by `code`, and the test asserts the expansion appears in **no** output, including the error text. Small on purpose — the point is that it is never parsed, not that it detonates. |
+| `xml-malformed.xml` | Known-negative: an unterminated tag must raise a typed `ExtractionError`, not leak `ParseError` and not yield zero concepts in silence. |
+
 ## The office fixtures
 
 `two-line-krav.docx`, `no-styles-krav.docx` and `two-line-krav.xlsx` are

tests/fixtures/generic-feed.xml

@@ -0,0 +1,14 @@
+<rss version="2.0">
+  <channel>
+    <title>Driftsmeldinger</title>
+    <description>Meldinger om drift og vedlikehold.</description>
+    <item>
+      <title>Stengt tunnel</title>
+      <description>Tunnelen er stengt for vedlikehold natt til fredag.</description>
+    </item>
+    <item>
+      <title>Redusert framkommelighet</title>
+      <description>Ett felt er stengt i anleggsperioden.</description>
+    </item>
+  </channel>
+</rss>

tests/fixtures/sts-mini.xml

@@ -0,0 +1,54 @@
+<standard>
+  <front>
+    <std-doc-meta>
+      <title-wrap><full>Testnormal for fiksturbruk</full></title-wrap>
+    </std-doc-meta>
+  </front>
+  <body>
+    <sec id="s-forord" sec-type="foreword">
+      <title>Forord</title>
+      <p>Dette forordet baerer ingen nummerering.</p>
+    </sec>
+    <sec id="s-1">
+      <label>1</label>
+      <title>Bruksomraade</title>
+      <p>Normalen gjelder for testing av ekstraktoren.</p>
+      <sec id="s-1-1">
+        <label>1.1</label>
+        <title>Omfang</title>
+        <p>Omfanget dekker hele arbeidet.</p>
+        <sec id="s-1-1-1">
+          <label>1.1.1</label>
+          <title>Materialer</title>
+          <p>Materialene skal vaere godkjente.</p>
+          <sec id="s-1-1-1-a" sec-type="spec">
+            <label>a)</label>
+            <p>Betong skal ha fasthetsklasse B35.</p>
+          </sec>
+          <sec id="s-1-1-1-b" sec-type="spec">
+            <label>b)</label>
+            <p>Armering skal vaere av klasse B500NC.</p>
+          </sec>
+        </sec>
+      </sec>
+    </sec>
+    <sec id="s-2">
+      <label>2</label>
+      <title>Tabeller</title>
+      <p>Tabellen under er en enkelt tabellblokk.</p>
+      <table-wrap id="tab-2-1">
+        <label>Tabell 2-1</label>
+        <table>
+          <tbody>
+            <tr><td>Prosess</td><td>Enhet</td></tr>
+            <tr><td>Sprengning</td><td>kubikkmeter</td></tr>
+          </tbody>
+        </table>
+      </table-wrap>
+      <list list-type="alpha-lower">
+        <list-item><p>Foerste punkt i lista.</p></list-item>
+        <list-item><p>Andre punkt i lista.</p></list-item>
+      </list>
+    </sec>
+  </body>
+</standard>

tests/fixtures/xml-doctype-bomb.xml

@@ -0,0 +1,14 @@
+<?xml version="1.0"?>
+<!DOCTYPE lolz [
+  <!ENTITY lol "SPRENGSTOFF">
+  <!ENTITY lol2 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
+  <!ENTITY lol3 "&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;">
+  <!ENTITY lol4 "&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;">
+]>
+<standard>
+  <body>
+    <sec id="s-1"><label>1</label><title>Uskyldig tittel</title>
+      <p>&lol4;</p>
+    </sec>
+  </body>
+</standard>

tests/fixtures/xml-malformed.xml

@@ -0,0 +1,9 @@
+<standard>
+  <body>
+    <sec id="s-1">
+      <label>1</label>
+      <title>Uavsluttet
+      <p>Denne taggen lukkes aldri.</p>
+    </sec>
+  </body>
+</standard>

tests/test_extract.py

@@ -315,17 +315,24 @@ def test_every_office_row_names_its_reader() -> None:
     }
 
 
-def test_html_and_epub_are_excluded_from_the_converter_on_purpose() -> None:
+def test_html_epub_and_xml_are_excluded_from_the_converter_on_purpose() -> None:
     """`.html` has a stdlib extractor; routing it through the converter would
     buy nothing and would add CVE-2025-51591 (SSRF via an iframe in HTML
     input), which is unpatched in every converter version. `.epub` is out for
     the same "no gain" half of that reason.
+
+    `.xml` joins them on that same half PLUS one of its own: the XML reader
+    here refuses a DTD outright, and a file routed to the converter would be
+    read by a different parser that never sees that refusal. The hardening
+    would be true of the code and false of the file.
     """
     from llm_ingestion_okf.extract import _CORE_EXTRACTORS, _PANDOC_FORMATS
 
     assert ".html" not in _PANDOC_FORMATS
     assert ".epub" not in _PANDOC_FORMATS
+    assert ".xml" not in _PANDOC_FORMATS
     assert ".html" in _CORE_EXTRACTORS
+    assert ".xml" in _CORE_EXTRACTORS
 
 
 def test_evidence_class_is_asserted_not_commented() -> None:
@@ -345,12 +352,17 @@ def test_evidence_class_is_asserted_not_commented() -> None:
     """
     from llm_ingestion_okf.extract import _EVIDENCE, _PANDOC_FORMATS
 
-    assert set(_EVIDENCE) == set(_PANDOC_FORMATS) | {".html", ".pdf"}, "every row needs a class"
+    assert set(_EVIDENCE) == set(_PANDOC_FORMATS) | {
+        ".html",
+        ".pdf",
+        ".xml",
+    }, "every row needs a class"
     assert {s for s, e in _EVIDENCE.items() if e == "measured"} == {
         ".pdf",
         ".docx",
         ".xlsx",
         ".html",
+        ".xml",
     }
     # Since 2026-09-09 the three office rows are `constructed`, not
     # `unmeasured`: each has now been put through end to end on a hand-built

tests/test_xml_sts.py

@@ -0,0 +1,185 @@
+"""`.xml`: a NISO-STS document carries its own structure, and it was unreadable.
+
+MEASURED OUTSIDE THIS REPOSITORY AND REPRODUCED HERE: the zip a publisher's
+own viewer delivers as "Html" holds 0 html, 1 xml and 109 images, and `okf
+build` on it gave **110 of 110 unreadable, 0 plans, exit 2** -- `.xml` was in
+neither extractor registry. The one xml file IS the whole product: R761
+Prosesskoden:2025, the same document round 12 met as a 701-page PDF, in
+NISO-STS form. It carries 7 715 `<sec>`, **2 761 of them with a `<title>`**,
+and a `<sec>`-nesting depth distribution of 28/118/500/1141/868/97/9 that is
+row-for-row identical to the fasit built from that same file.
+
+THE CEILING IS THEREFORE STRUCTURAL AND NOT COMPUTED: 2 761 of 2 761. The PDF
+arm reached 2 759 of 2 761 by bridging from (page, y) to a line index; here the
+publisher states the structure in elements, so nothing is recovered and nothing
+is guessed.
+
+WHAT THESE TESTS PIN, each because the measurement said it could go wrong:
+
+- `<label>` carries the number and `<title>` carries the text. Emitting only
+  `<title>` scores 0 of 2 761 while every line of the code looks right, because
+  the number is what okf reduces to a directory name.
+- a `<sec>` with a `<label>` and NO `<title>` is never a heading. 4 954 of the
+  7 715 are lettered points (`a)`, `c)`) inside a process description, and one
+  heading each would bury the document's own 2 761.
+- the ONE titled section with no label -- `Forord` -- emitted without a
+  numbering token, because the corrected instrument key looks it up by title.
+- a `<table-wrap>` as ONE markdown table block. The PDF path delivered 0 of 10.
+- XML that is NOT STS producing text and ONE plan, never zero and never an
+  invented structure from element names.
+- a DTD REFUSED rather than parsed. Measured on this machine's interpreter
+  (3.14.0, pyexpat 2.7.3): an external entity is refused by the stdlib, but the
+  billion-laughs limit comes from libexpat >= 2.4.0 and NOT from Python, while
+  `pyproject.toml` requires only `>=3.10`. Refusing every DTD is a guarantee
+  about this code; relying on the parser is a guarantee about the machine.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from llm_ingestion_okf import extract, propose
+from llm_ingestion_okf.errors import IngestError
+
+FIXTURES = Path(__file__).parent / "fixtures"
+STS = FIXTURES / "sts-mini.xml"
+GENERIC = FIXTURES / "generic-feed.xml"
+BOMB = FIXTURES / "xml-doctype-bomb.xml"
+MALFORMED = FIXTURES / "xml-malformed.xml"
+
+STS_EXPECTED = """\
+Testnormal for fiksturbruk
+# Forord
+Dette forordet baerer ingen nummerering.
+# 1 Bruksomraade
+Normalen gjelder for testing av ekstraktoren.
+## 1.1 Omfang
+Omfanget dekker hele arbeidet.
+### 1.1.1 Materialer
+Materialene skal vaere godkjente.
+a) Betong skal ha fasthetsklasse B35.
+b) Armering skal vaere av klasse B500NC.
+# 2 Tabeller
+Tabellen under er en enkelt tabellblokk.
+Tabell 2-1
+| Prosess | Enhet |
+| --- | --- |
+| Sprengning | kubikkmeter |
+Foerste punkt i lista.
+Andre punkt i lista."""
+
+
+def test_an_sts_document_becomes_the_markdown_the_office_rows_produce() -> None:
+    """The whole emission, asserted as one string rather than in pieces.
+
+    A per-property test would pass on an output whose LINES were right and
+    whose order was not, and order is what every boundary grammar reads.
+    """
+    assert extract.extract_text(STS.name, STS.read_bytes()) == STS_EXPECTED
+
+
+def test_the_label_and_the_title_reach_the_proposer_as_one_heading() -> None:
+    """Title, LEVEL and OFFSET -- not "it found more than one"."""
+    text = extract.extract_text(STS.name, STS.read_bytes())
+    candidates = propose.find_candidates(text)
+
+    assert [(c.number, c.title, c.level) for c in candidates] == [
+        (None, "Forord", 1),
+        (None, "1 Bruksomraade", 1),
+        ("1.1", "Omfang", 2),
+        ("1.1.1", "Materialer", 3),
+        (None, "2 Tabeller", 1),
+    ]
+    # The offsets, stated against the text itself: a candidate naming the right
+    # title at the wrong offset would pass the list above.
+    for candidate, line in zip(
+        candidates,
+        ["# Forord", "# 1 Bruksomraade", "## 1.1 Omfang", "### 1.1.1 Materialer", "# 2 Tabeller"],
+        strict=True,
+    ):
+        assert text[candidate.start : candidate.start + len(line)] == line
+
+
+def test_a_lettered_point_is_a_body_line_and_never_a_heading() -> None:
+    """4 954 of R761's 7 715 `<sec>` are these. One heading each buries 2 761."""
+    text = extract.extract_text(STS.name, STS.read_bytes())
+
+    assert "a) Betong skal ha fasthetsklasse B35." in text.split("\n")
+    assert not any(line.startswith("#") and "Betong" in line for line in text.split("\n"))
+    assert [c.title for c in propose.find_candidates(text) if "Betong" in c.title] == []
+
+
+def test_a_table_wrap_is_one_table_block_with_its_label_above_it() -> None:
+    """The separator line is what makes it a BLOCK and not two pipe lines."""
+    lines = extract.extract_text(STS.name, STS.read_bytes()).split("\n")
+    at = lines.index("Tabell 2-1")
+
+    assert lines[at + 1 : at + 4] == [
+        "| Prosess | Enhet |",
+        "| --- | --- |",
+        "| Sprengning | kubikkmeter |",
+    ]
+
+
+def test_the_text_survives_the_markup_exactly() -> None:
+    """EXACT, not a percentage: strip what the emission added and compare."""
+    import xml.etree.ElementTree as ET
+
+    text = extract.extract_text(STS.name, STS.read_bytes())
+    source = ET.fromstring(STS.read_text(encoding="utf-8"))
+
+    kept = []
+    for line in text.split("\n"):
+        if set(line) <= set("| -:"):
+            continue  # the table separator, which no source character produced
+        kept.append(line.lstrip("#").replace("|", ""))
+    emitted = "".join("".join(kept).split())
+    original = "".join("".join(source.itertext()).split())
+    assert emitted == original
+
+
+def test_generic_xml_keeps_its_text_and_invents_no_structure() -> None:
+    """One plan with the content preserved -- never zero, never element names
+    promoted to headings. The schema test is NAMED (`<standard>` root, or any
+    `<sec>`), so a document that is not STS is not guessed at."""
+    text = extract.extract_text(GENERIC.name, GENERIC.read_bytes())
+
+    assert "Tunnelen er stengt for vedlikehold natt til fredag." in text
+    assert "Redusert framkommelighet" in text
+    assert not any(line.startswith("#") for line in text.split("\n"))
+    assert propose.find_candidates(text) == []
+
+
+def test_a_document_type_declaration_is_refused_and_never_expanded() -> None:
+    """A guarantee about this code, not about the machine's libexpat."""
+    data = BOMB.read_bytes()
+    assert b"<!DOCTYPE" in data  # the fixture really carries one
+
+    with pytest.raises(IngestError) as caught:
+        extract.extract_text(BOMB.name, data)
+    assert caught.value.code == "extractor_xml_doctype"
+    # The expansion must not appear ANYWHERE, including in the error text.
+    assert "SPRENGSTOFF" not in str(caught.value)
+
+
+def test_malformed_xml_raises_a_typed_error_and_not_a_parse_error() -> None:
+    """Not a leaked `ParseError`, and not zero concepts in silence."""
+    with pytest.raises(IngestError) as caught:
+        extract.extract_text(MALFORMED.name, MALFORMED.read_bytes())
+    assert caught.value.code == "extractor_xml_parse_error"
+
+
+def test_xml_is_a_core_type_and_never_reaches_the_converter() -> None:
+    """Stdlib parsing, so `[extract]` would make a stdlib type binary-dependent
+    -- and the converter is a second parser this hardening never sees."""
+    from llm_ingestion_okf.extract import (
+        _CORE_EXTRACTORS,
+        _OPTIONAL_EXTRACTORS,
+        _PANDOC_FORMATS,
+    )
+
+    assert ".xml" in _CORE_EXTRACTORS
+    assert ".xml" not in _OPTIONAL_EXTRACTORS
+    assert ".xml" not in _PANDOC_FORMATS