open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(extract): the file that IS the product now reads, as a core stdlib type

Browse source 
GREEN on the twelve tests from the two commits before it. `.xml` joins
`_CORE_EXTRACTORS` beside `.html`, and 110 of 110 unreadable becomes a bundle.

A CORE type, not an `[extract]` one, because the parser is stdlib: putting it
behind the extra would make a pure-stdlib file type binary-dependent for no
gain. And never through the converter, which the existing fence test now says
in one more line: a file routed there is read by a second parser that never
sees this reader's DTD refusal, so the hardening would be true of the code and
false of the file.

TWO PATHS, NAMED RATHER THAN GUESSED. STS when the root is `<standard>` or the
document holds any `<sec>`; generic otherwise. Generic XML keeps its text in
document order and gets NO structure -- an element name is never promoted to a
heading, because an RSS feed is not a numbered standard.

THE GRAMMAR IS MARKDOWN, the same markdown the office rows and round 11's HTML
row reach the proposer through, and `propose.py` is untouched. `<sec>` with a
`<title>` becomes one ATX line carrying `<label>` + space + `<title>` at its
own nesting depth; `<sec>` with only a `<label>` becomes a body line with the
label in front, the way `li` is treated in HTML -- 4 954 of R761's 7 715 are
lettered points and one heading each would bury its own 2 761; `<table-wrap>`
becomes its label plus ONE markdown table through this package's own
`render_table`, separator line included, which is what makes it a block.

TWO CHOICES THAT ARE MEASUREMENTS, not preferences:

- Inline by ALLOWLIST, block by default -- the inverse of the HTML reader,
  because block-versus-inline is a property of HTML and XML has no such
  universal. Assuming block is safe (an extra break never removes text and a
  boundary needs a line that matches a grammar); assuming inline is not. The
  allowlist is load-bearing: that document carries 1 701 `<italic>` and
  1 396 `<bold>` inside its prose.
- The ATX ceiling is 6 and STS nesting reaches 7. The depth is CLIPPED, not
  dropped: 9 of the 2 761 titled sections sit at depth 7 and `#######` matches
  nothing, so dropping loses the section while clipping keeps the boundary and
  states the nesting one level too shallow.

A DTD IS REFUSED UNPARSED, and that is a guarantee about this code rather than
about the machine. Measured on this interpreter (3.14.0, pyexpat 2.7.3): an
external SYSTEM entity is refused by the stdlib and never fetched, but the
amplification limit that stops a billion-laughs comes from libexpat >= 2.4.0
and NOT from Python -- five levels still expand -- while `pyproject.toml`
requires only `>=3.10` and no lockfile pins an interpreter. `XMLParser` exposes
no `.parser` attribute on the C accelerator, so the handler route is not
portable either. NO new dependency: `defusedxml` and `lxml` both occur 0 times
in `uv.lock` and still do.

pytest -q: 1566 passed, 1 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 03:41:02 +02:00
commit 567a63d455
4 changed files with 276 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

228
src/llm_ingestion_okf/extract.py
View file

@ -37,6 +37,7 @@ from dataclasses import dataclass
from html.parser import HTMLParser
from pathlib import Path
from xml.etree import ElementTree
from xml.etree.ElementTree import Element
from .errors import ExtractionError, ExtractionWarning
from .render import render_fenced_block, render_table
@ -124,6 +125,7 @@ _EVIDENCE: dict[str, str] = {
".odt": "constructed",
".rtf": "constructed",
".html": "measured",
".xml": "measured",
}
# Load-bearing, all three, and none of them hygiene:
@ -255,6 +257,68 @@ _BLOCK_TAGS = frozenset(
)
# --- NISO-STS, and the two facts the whole reader turns on ------------------
#
# ONE: `<label>` carries the number and `<title>` carries the text. Measured on
# the 701-page process code, 2 of its 2 761 `<title>` strings begin with a
# digit -- the number is a sibling element, never glued on. A fasit that shows
# `"2.1Hovedprosesser"` is its BUILDER joining the two. Emitting `<title>`
# alone therefore scores 0 of 2 761 while every line of this file looks right,
# because the number is what okf reduces to a directory name.
#
# TWO: a `<sec>` with a `<label>` and no `<title>` is not a section heading. It
# is a lettered point (`a)`, `c)`, `sec-type="spec"`) inside a process
# description, and there are 4 954 of them against the document's own 2 761.
# One heading each and the document's structure is the minority of its own
# outline.
_STS_ROOT = "standard"
# Inline by allowlist, block by default -- the INVERSE of the HTML reader, and
# for the reason that reader gives for its own direction. Block versus inline
# is a property of HTML; XML has no such universal, so an unknown element
# cannot be assumed inline without fusing two paragraphs into one line. It can
# safely be assumed block: an extra line break never removes text and never
# invents a boundary, because a boundary needs a line that MATCHES a grammar.
#
# The members are NISO-STS's own inline set, and they are load-bearing rather
# than decorative: that document carries 1 701 `<italic>` and 1 396 `<bold>`
# inside its prose, so breaking on them would shred a paragraph into fragments
# that are individually true and collectively unreadable.
_XML_INLINE_TAGS = frozenset(
{
"italic",
"bold",
"underline",
"sup",
"sub",
"sc",
"monospace",
"roman",
"sans-serif",
"overline",
"strike",
"xref",
"ext-link",
"uri",
"std-ref",
"inline-formula",
"styled-content",
"named-content",
"break",
}
)
# The maximum ATX level `propose._ATX` can read (`#{1,6}`), which `_HEADING_TAGS`
# stops at for the same reason. STS nesting goes DEEPER: 9 of the 2 761 titled
# sections in that document sit at depth 7, and `#######` matches nothing at
# all. The depth is CLIPPED rather than dropped -- a clipped heading still sets
# its boundary and states its nesting one level too shallow, where a dropped
# one loses the section entirely. The cost shows up in frontmatter nesting, not
# in the depth row, because that row is the source's own depth and not the ATX
# level we emitted.
_ATX_MAX_LEVEL = 6
def decode_text(data: bytes) -> str:
"""Decode file bytes as UTF-8 (BOM-stripping), typed on failure.
@ -369,6 +433,169 @@ def _extract_html(data: bytes) -> str:
return parser.text()
class _XmlTextExtractor:
"""Collect an XML document's text as LINES of markdown.
THE OUTPUT GRAMMAR IS MARKDOWN, and deliberately the same markdown the
office rows and the HTML row reach the proposer through. Nothing in
`propose.py` knows this format exists: `_ATX` reads the heading lines and
`_TABLE_ROW` reads the table ones, exactly as they read a converted
`.docx`. That is the whole reason this is an extractor and not a
segmentation arm.
TWO PATHS, and which one runs is NAMED rather than guessed:
STS the root is `<standard>` or the document contains a `<sec>`.
A `<sec>` with a `<title>` becomes one ATX line whose level is
its `<sec>`-nesting depth and whose text is `<label>` + space +
`<title>`; a `<sec>` with only a `<label>` becomes a body line
with the label in front of it, the way `li` is treated in HTML;
a `<table-wrap>` becomes its label on a line and its rows as one
markdown table block.
GENERIC everything else. Text content in document order, one line per
block-like element, and NO element name is ever promoted to a
heading. An RSS feed is not a numbered standard, and reading it
as one would state a structure its author did not.
TEXT IS PRESERVED EXACTLY. The only characters added are the ATX markers
and the table pipes; strip those and the non-whitespace sequence is
identical to `"".join(root.itertext())`.
"""
def __init__(self, *, sts: bool) -> None:
self._sts = sts
self._lines: list[str] = []
self._current: list[str] = []
self._prefix = ""
def _break(self, prefix: str = "") -> None:
"""Close the line being accumulated and open the next one.
A pending prefix SURVIVES a break that emitted nothing -- a label-only
`<sec>` holds its `a)` until the first line that has words in it, which
may be several empty elements later.
"""
line = " ".join("".join(self._current).split())
self._current = []
if line:
self._lines.append(f"{self._prefix}{line}")
self._prefix = ""
if prefix:
self._prefix = prefix
def _emit(self, line: str) -> None:
"""Put a whole line out, ahead of whatever is being accumulated."""
self._break()
self._lines.append(line)
def _text_of(self, element: Element) -> str:
"""An element's whole text, whitespace collapsed."""
return " ".join("".join(element.itertext()).split())
def _table(self, element: Element) -> bool:
"""A `<table-wrap>`: its label on a line, its rows as ONE table block.
The separator line is what makes it a block rather than two pipe lines
-- `--table-grid` and `--keep-table-heading` read the block, and the
PDF path delivered 0 of this document's 10 tables as one.
"""
rows = [
[self._text_of(cell) for cell in row if cell.tag in ("td", "th")]
for row in element.iter("tr")
]
rows = [row for row in rows if row]
if not rows:
return False
label = element.find("label")
if label is not None:
self._emit(self._text_of(label))
caption = element.find("caption")
if caption is not None:
self._emit(self._text_of(caption))
self._break()
self._lines.extend(render_table(rows[0], rows[1:]).rstrip("\n").split("\n"))
return True
def _walk(self, element: Element, depth: int) -> None:
tag = _local_name(element.tag)
if self._sts and tag == "table-wrap" and self._table(element):
return
skip: set[int] = set()
if self._sts and tag == "sec":
depth += 1
label = element.find("label")
title = element.find("title")
if title is not None:
level = min(depth, _ATX_MAX_LEVEL)
parts = [self._text_of(label)] if label is not None else []
parts.append(self._text_of(title))
self._emit("#" * level + " " + " ".join(part for part in parts if part))
skip = {id(title)} | ({id(label)} if label is not None else set())
elif label is not None:
# NEVER a heading. The label goes in FRONT of the body line the
# way `li` is treated in the HTML reader.
self._break(self._text_of(label) + " ")
skip = {id(label)}
inline = _local_name(element.tag) in _XML_INLINE_TAGS
if not inline:
self._break()
else:
self._current.append(" ")
if element.text:
self._current.append(element.text)
for child in element:
if id(child) not in skip:
self._walk(child, depth)
if child.tail:
self._current.append(child.tail)
if not inline:
self._break()
def text(self, root: Element) -> str:
self._walk(root, 0)
self._break()
return "\n".join(self._lines)
def _extract_xml(data: bytes) -> str:
"""`xml`: NISO-STS structure as markdown, any other schema as its text.
A DTD IS REFUSED RATHER THAN PARSED, and that is a guarantee about this
code instead of one about the machine. Measured on this interpreter
(3.14.0, `pyexpat.version_info` 2.7.3): an external `SYSTEM` entity is
refused by the stdlib and never fetched, but the entity-amplification limit
that stops a billion-laughs comes from libexpat >= 2.4.0 and NOT from
Python -- five levels still expanded, six and seven were refused -- while
`pyproject.toml` requires only `>=3.10` and no lockfile pins an
interpreter. `XMLParser` exposes no `.parser` attribute on the C
accelerator either, so the handler route is not portable. Refusing every
document type declaration costs nothing here (0 of 1 file carries one) and
holds on every interpreter.
"""
text = decode_text(data)
prologue = text[: text.find("<", text.find("<") + 1) + 1] if "<" in text else text
if "<!DOCTYPE" in prologue or "<!DOCTYPE" in text[:4096]:
raise ExtractionError(
"XML carrying a document type declaration is refused unparsed: a DTD can "
"define entities, and the parser's amplification limit is a property of "
"the installed libexpat rather than of this package",
code="extractor_xml_doctype",
)
try:
root = ElementTree.fromstring(text)
except ElementTree.ParseError as exc:
raise ExtractionError(
f"the XML parser failed on this file: {exc}", code="extractor_xml_parse_error"
) from exc
sts = _local_name(root.tag) == _STS_ROOT or next(root.iter("sec"), None) is not None
return _XmlTextExtractor(sts=sts).text(root)
def _local_name(tag: str) -> str:
"""`{ns}sec` -> `sec`. A namespaced document names the same elements."""
return tag.rsplit("}", 1)[-1]
def _extra_missing(suffix: str) -> ExtractionError:
"""The one rejection for a `[extract]` type without the extra installed.
@ -1024,6 +1251,7 @@ _CORE_EXTRACTORS: dict[str, Callable[[bytes], str]] = {
".json": _extract_json,
".html": _extract_html,
".htm": _extract_html,
".xml": _extract_xml,
}
# Types the `[extract]` extra ships a parser for. Kept separate from the core