open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(extract): the file that IS the product now reads, as a core stdlib type

Browse source 
GREEN on the twelve tests from the two commits before it. `.xml` joins
`_CORE_EXTRACTORS` beside `.html`, and 110 of 110 unreadable becomes a bundle.

A CORE type, not an `[extract]` one, because the parser is stdlib: putting it
behind the extra would make a pure-stdlib file type binary-dependent for no
gain. And never through the converter, which the existing fence test now says
in one more line: a file routed there is read by a second parser that never
sees this reader's DTD refusal, so the hardening would be true of the code and
false of the file.

TWO PATHS, NAMED RATHER THAN GUESSED. STS when the root is `<standard>` or the
document holds any `<sec>`; generic otherwise. Generic XML keeps its text in
document order and gets NO structure -- an element name is never promoted to a
heading, because an RSS feed is not a numbered standard.

THE GRAMMAR IS MARKDOWN, the same markdown the office rows and round 11's HTML
row reach the proposer through, and `propose.py` is untouched. `<sec>` with a
`<title>` becomes one ATX line carrying `<label>` + space + `<title>` at its
own nesting depth; `<sec>` with only a `<label>` becomes a body line with the
label in front, the way `li` is treated in HTML -- 4 954 of R761's 7 715 are
lettered points and one heading each would bury its own 2 761; `<table-wrap>`
becomes its label plus ONE markdown table through this package's own
`render_table`, separator line included, which is what makes it a block.

TWO CHOICES THAT ARE MEASUREMENTS, not preferences:

- Inline by ALLOWLIST, block by default -- the inverse of the HTML reader,
  because block-versus-inline is a property of HTML and XML has no such
  universal. Assuming block is safe (an extra break never removes text and a
  boundary needs a line that matches a grammar); assuming inline is not. The
  allowlist is load-bearing: that document carries 1 701 `<italic>` and
  1 396 `<bold>` inside its prose.
- The ATX ceiling is 6 and STS nesting reaches 7. The depth is CLIPPED, not
  dropped: 9 of the 2 761 titled sections sit at depth 7 and `#######` matches
  nothing, so dropping loses the section while clipping keeps the boundary and
  states the nesting one level too shallow.

A DTD IS REFUSED UNPARSED, and that is a guarantee about this code rather than
about the machine. Measured on this interpreter (3.14.0, pyexpat 2.7.3): an
external SYSTEM entity is refused by the stdlib and never fetched, but the
amplification limit that stops a billion-laughs comes from libexpat >= 2.4.0
and NOT from Python -- five levels still expand -- while `pyproject.toml`
requires only `>=3.10` and no lockfile pins an interpreter. `XMLParser` exposes
no `.parser` attribute on the C accelerator, so the handler route is not
portable either. NO new dependency: `defusedxml` and `lxml` both occur 0 times
in `uv.lock` and still do.

pytest -q: 1566 passed, 1 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 03:41:02 +02:00
commit 567a63d455
4 changed files with 276 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

39
CLAUDE.md
View file

@ -46,7 +46,44 @@ one boundary rule:
`measured` since 2026-09-10, the row with the most measurement behind it and
no entry in the table at all -- with the limit
that travels with it -- one product, one format, one publisher, and a
generator's cut, not 828 documents anyone wrote. `pdf`/`docx`/`xlsx` only via
generator's cut, not 828 documents anyone wrote.
**`.xml` became a CORE type 2026-09-11** and it is the first row whose
ceiling is structural rather than recovered. A NISO-STS zip from a
publisher's own viewer was **110 of 110 unreadable, 0 plans, exit 2** --
`.xml` was in neither registry -- and the one xml file in it IS the whole
product: R761 Prosesskoden:2025, the document round 12 met as a 701-page
PDF, carrying 7 715 `<sec>`, **2 761 with a `<title>`**, 4 954 with a
`<label>` and no title, 10 `<table-wrap>`, root `<standard>`, 0 `<!DOCTYPE`.
Its `<sec>`-nesting depths over the titled sections are
28/118/500/1141/868/97/9, row for row the fasit's own. The reader is stdlib
(`xml.etree.ElementTree`) and adds NO dependency -- `defusedxml` and `lxml`
are 0 occurrences in `uv.lock` -- so it is core beside `.html` rather than
behind `[extract]`, which would make a pure-stdlib type binary-dependent.
**The output grammar is MARKDOWN**, the same the office rows reach the
proposer through, and `propose.py` is untouched: `<sec>` with a `<title>`
becomes one ATX line carrying `<label>` + space + `<title>` at its own
nesting depth, `<sec>` with only a `<label>` becomes a body line with the
label in front (never a heading -- 4 954 of 7 715 are lettered points and one
heading each buries the document's own 2 761), and `<table-wrap>` becomes its
label plus one markdown table through `render_table`. **`<label>` carries the
number and `<title>` carries the text** -- 2 of 2 761 titles begin with a
digit -- so emitting `<title>` alone scores 0 of 2 761 with nothing in the
code looking wrong. Inline is an ALLOWLIST and block is the default, the
inverse of the HTML reader, because block-versus-inline is a property of HTML
and XML has no such universal; the allowlist is load-bearing at 1 701
`<italic>` and 1 396 `<bold>` inside that document's prose. The ATX ceiling is
6 and STS nesting reaches 7, so the depth is CLIPPED and not dropped: 9 of
2 761 sit at depth 7 and `#######` matches nothing. A `<!DOCTYPE` is REFUSED
unparsed with its own code, a guarantee about the code rather than about the
machine -- measured on 3.14.0 with pyexpat 2.7.3, an external SYSTEM entity
is refused by the stdlib but the billion-laughs limit comes from libexpat
>= 2.4.0 and not from Python, while `pyproject.toml` requires only `>=3.10`.
XML that is not STS keeps its text in document order and gets NO invented
structure, and `.xml` never routes through the converter -- a second parser
that would never see that refusal. The registries are COUPLED: a row in
`_CORE_EXTRACTORS` and not in `segmentation._STDLIB_EXTRACTOR_IDS` refuses
every proposal for the type, two layers away from the extractor.
`pdf`/`docx`/`xlsx` only via
the optional `[extract]` extra; without it those types are rejected
fail-fast. The extra ships `pdfplumber` for `pdf` (chosen on ONE measured
property: it keeps a requirement table's label and value on the same line

11
README.md
View file

@ -537,10 +537,17 @@ bundle:
run path; output is reproducible byte-for-byte against golden fixtures.
2. **Bundle inbox.** A drop directory where common file types are converted
to OKF concept files. All file-type→text extraction lives in this library:
`md`, `txt`, `csv`, `json`, and `html` are handled by the stdlib core;
`md`, `txt`, `csv`, `json`, `html` and `xml` are handled by the stdlib core;
`pdf` and the five office formats (`docx`, `xlsx`, `pptx`, `odt`, `rtf`)
require the optional `[extract]` extra and are rejected fail-fast without
it. Extracted text passes the security gate before anything is persisted.
An `xml` file that declares NISO-STS structure (`<standard>` root, or any
`<sec>`) becomes one heading per titled section, at the section's own
nesting depth, with the section's `<label>` and `<title>` on one line; a
`<sec>` carrying only a label is a body line and never a heading, and a
`<table-wrap>` becomes one markdown table. Any other XML keeps its text in
document order and gets no invented structure. XML carrying a
`<!DOCTYPE` is refused unparsed.
The drop directory is walked **recursively**, in sorted relative-path order:
a file at any depth is ingested and records its path relative to the inbox
root as its `source_file`, while dot-directories and a bundle directory
@ -556,7 +563,7 @@ bundle:
numbers index the extracted text and say so. Measurements:
[`docs/2026-09-08-proveniens-k2.md`](docs/2026-09-08-proveniens-k2.md).
<!-- extract-formats: .md, .txt, .csv, .json, .html, .htm, .pdf, .docx, .xlsx, .pptx, .odt, .rtf -->
<!-- extract-formats: .md, .txt, .csv, .json, .html, .htm, .xml, .pdf, .docx, .xlsx, .pptx, .odt, .rtf -->
3. **External bundle import.** Import and merge of third-party OKF bundles:
each concept is assessed via the security gate, and only concepts that
pass are merged, materialized, and linked into the index.

228
src/llm_ingestion_okf/extract.py
View file

@ -37,6 +37,7 @@ from dataclasses import dataclass
from html.parser import HTMLParser
from pathlib import Path
from xml.etree import ElementTree
from xml.etree.ElementTree import Element
from .errors import ExtractionError, ExtractionWarning
from .render import render_fenced_block, render_table
@ -124,6 +125,7 @@ _EVIDENCE: dict[str, str] = {
".odt": "constructed",
".rtf": "constructed",
".html": "measured",
".xml": "measured",
}
# Load-bearing, all three, and none of them hygiene:
@ -255,6 +257,68 @@ _BLOCK_TAGS = frozenset(
)
# --- NISO-STS, and the two facts the whole reader turns on ------------------
#
# ONE: `<label>` carries the number and `<title>` carries the text. Measured on
# the 701-page process code, 2 of its 2 761 `<title>` strings begin with a
# digit -- the number is a sibling element, never glued on. A fasit that shows
# `"2.1Hovedprosesser"` is its BUILDER joining the two. Emitting `<title>`
# alone therefore scores 0 of 2 761 while every line of this file looks right,
# because the number is what okf reduces to a directory name.
#
# TWO: a `<sec>` with a `<label>` and no `<title>` is not a section heading. It
# is a lettered point (`a)`, `c)`, `sec-type="spec"`) inside a process
# description, and there are 4 954 of them against the document's own 2 761.
# One heading each and the document's structure is the minority of its own
# outline.
_STS_ROOT = "standard"
# Inline by allowlist, block by default -- the INVERSE of the HTML reader, and
# for the reason that reader gives for its own direction. Block versus inline
# is a property of HTML; XML has no such universal, so an unknown element
# cannot be assumed inline without fusing two paragraphs into one line. It can
# safely be assumed block: an extra line break never removes text and never
# invents a boundary, because a boundary needs a line that MATCHES a grammar.
#
# The members are NISO-STS's own inline set, and they are load-bearing rather
# than decorative: that document carries 1 701 `<italic>` and 1 396 `<bold>`
# inside its prose, so breaking on them would shred a paragraph into fragments
# that are individually true and collectively unreadable.
_XML_INLINE_TAGS = frozenset(
{
"italic",
"bold",
"underline",
"sup",
"sub",
"sc",
"monospace",
"roman",
"sans-serif",
"overline",
"strike",
"xref",
"ext-link",
"uri",
"std-ref",
"inline-formula",
"styled-content",
"named-content",
"break",
}
)
# The maximum ATX level `propose._ATX` can read (`#{1,6}`), which `_HEADING_TAGS`
# stops at for the same reason. STS nesting goes DEEPER: 9 of the 2 761 titled
# sections in that document sit at depth 7, and `#######` matches nothing at
# all. The depth is CLIPPED rather than dropped -- a clipped heading still sets
# its boundary and states its nesting one level too shallow, where a dropped
# one loses the section entirely. The cost shows up in frontmatter nesting, not
# in the depth row, because that row is the source's own depth and not the ATX
# level we emitted.
_ATX_MAX_LEVEL = 6
def decode_text(data: bytes) -> str:
"""Decode file bytes as UTF-8 (BOM-stripping), typed on failure.
@ -369,6 +433,169 @@ def _extract_html(data: bytes) -> str:
return parser.text()
class _XmlTextExtractor:
"""Collect an XML document's text as LINES of markdown.
THE OUTPUT GRAMMAR IS MARKDOWN, and deliberately the same markdown the
office rows and the HTML row reach the proposer through. Nothing in
`propose.py` knows this format exists: `_ATX` reads the heading lines and
`_TABLE_ROW` reads the table ones, exactly as they read a converted
`.docx`. That is the whole reason this is an extractor and not a
segmentation arm.
TWO PATHS, and which one runs is NAMED rather than guessed:
STS the root is `<standard>` or the document contains a `<sec>`.
A `<sec>` with a `<title>` becomes one ATX line whose level is
its `<sec>`-nesting depth and whose text is `<label>` + space +
`<title>`; a `<sec>` with only a `<label>` becomes a body line
with the label in front of it, the way `li` is treated in HTML;
a `<table-wrap>` becomes its label on a line and its rows as one
markdown table block.
GENERIC everything else. Text content in document order, one line per
block-like element, and NO element name is ever promoted to a
heading. An RSS feed is not a numbered standard, and reading it
as one would state a structure its author did not.
TEXT IS PRESERVED EXACTLY. The only characters added are the ATX markers
and the table pipes; strip those and the non-whitespace sequence is
identical to `"".join(root.itertext())`.
"""
def __init__(self, *, sts: bool) -> None:
self._sts = sts
self._lines: list[str] = []
self._current: list[str] = []
self._prefix = ""
def _break(self, prefix: str = "") -> None:
"""Close the line being accumulated and open the next one.
A pending prefix SURVIVES a break that emitted nothing -- a label-only
`<sec>` holds its `a)` until the first line that has words in it, which
may be several empty elements later.
"""
line = " ".join("".join(self._current).split())
self._current = []
if line:
self._lines.append(f"{self._prefix}{line}")
self._prefix = ""
if prefix:
self._prefix = prefix
def _emit(self, line: str) -> None:
"""Put a whole line out, ahead of whatever is being accumulated."""
self._break()
self._lines.append(line)
def _text_of(self, element: Element) -> str:
"""An element's whole text, whitespace collapsed."""
return " ".join("".join(element.itertext()).split())
def _table(self, element: Element) -> bool:
"""A `<table-wrap>`: its label on a line, its rows as ONE table block.
The separator line is what makes it a block rather than two pipe lines
-- `--table-grid` and `--keep-table-heading` read the block, and the
PDF path delivered 0 of this document's 10 tables as one.
"""
rows = [
[self._text_of(cell) for cell in row if cell.tag in ("td", "th")]
for row in element.iter("tr")
]
rows = [row for row in rows if row]
if not rows:
return False
label = element.find("label")
if label is not None:
self._emit(self._text_of(label))
caption = element.find("caption")
if caption is not None:
self._emit(self._text_of(caption))
self._break()
self._lines.extend(render_table(rows[0], rows[1:]).rstrip("\n").split("\n"))
return True
def _walk(self, element: Element, depth: int) -> None:
tag = _local_name(element.tag)
if self._sts and tag == "table-wrap" and self._table(element):
return
skip: set[int] = set()
if self._sts and tag == "sec":
depth += 1
label = element.find("label")
title = element.find("title")
if title is not None:
level = min(depth, _ATX_MAX_LEVEL)
parts = [self._text_of(label)] if label is not None else []
parts.append(self._text_of(title))
self._emit("#" * level + " " + " ".join(part for part in parts if part))
skip = {id(title)} | ({id(label)} if label is not None else set())
elif label is not None:
# NEVER a heading. The label goes in FRONT of the body line the
# way `li` is treated in the HTML reader.
self._break(self._text_of(label) + " ")
skip = {id(label)}
inline = _local_name(element.tag) in _XML_INLINE_TAGS
if not inline:
self._break()
else:
self._current.append(" ")
if element.text:
self._current.append(element.text)
for child in element:
if id(child) not in skip:
self._walk(child, depth)
if child.tail:
self._current.append(child.tail)
if not inline:
self._break()
def text(self, root: Element) -> str:
self._walk(root, 0)
self._break()
return "\n".join(self._lines)
def _extract_xml(data: bytes) -> str:
"""`xml`: NISO-STS structure as markdown, any other schema as its text.
A DTD IS REFUSED RATHER THAN PARSED, and that is a guarantee about this
code instead of one about the machine. Measured on this interpreter
(3.14.0, `pyexpat.version_info` 2.7.3): an external `SYSTEM` entity is
refused by the stdlib and never fetched, but the entity-amplification limit
that stops a billion-laughs comes from libexpat >= 2.4.0 and NOT from
Python -- five levels still expanded, six and seven were refused -- while
`pyproject.toml` requires only `>=3.10` and no lockfile pins an
interpreter. `XMLParser` exposes no `.parser` attribute on the C
accelerator either, so the handler route is not portable. Refusing every
document type declaration costs nothing here (0 of 1 file carries one) and
holds on every interpreter.
"""
text = decode_text(data)
prologue = text[: text.find("<", text.find("<") + 1) + 1] if "<" in text else text
if "<!DOCTYPE" in prologue or "<!DOCTYPE" in text[:4096]:
raise ExtractionError(
"XML carrying a document type declaration is refused unparsed: a DTD can "
"define entities, and the parser's amplification limit is a property of "
"the installed libexpat rather than of this package",
code="extractor_xml_doctype",
)
try:
root = ElementTree.fromstring(text)
except ElementTree.ParseError as exc:
raise ExtractionError(
f"the XML parser failed on this file: {exc}", code="extractor_xml_parse_error"
) from exc
sts = _local_name(root.tag) == _STS_ROOT or next(root.iter("sec"), None) is not None
return _XmlTextExtractor(sts=sts).text(root)
def _local_name(tag: str) -> str:
"""`{ns}sec` -> `sec`. A namespaced document names the same elements."""
return tag.rsplit("}", 1)[-1]
def _extra_missing(suffix: str) -> ExtractionError:
"""The one rejection for a `[extract]` type without the extra installed.
@ -1024,6 +1251,7 @@ _CORE_EXTRACTORS: dict[str, Callable[[bytes], str]] = {
".json": _extract_json,
".html": _extract_html,
".htm": _extract_html,
".xml": _extract_xml,
}
# Types the `[extract]` extra ships a parser for. Kept separate from the core

2
src/llm_ingestion_okf/segmentation.py
View file

@ -85,7 +85,7 @@ STDLIB_EXTRACTOR_VERSION = "stdlib-1"
#: Extractor ids answered by the stdlib registry. `none` is a dropped file with
#: no suffix, which the proposer and the run path both reduce to that literal.
_STDLIB_EXTRACTOR_IDS = frozenset({"md", "txt", "csv", "json", "html", "htm", "none"})
_STDLIB_EXTRACTOR_IDS = frozenset({"md", "txt", "csv", "json", "html", "htm", "xml", "none"})
#: Extractor ids answered by the vendored converter. Held here rather than
#: imported from the extraction registry, which must not be made to depend on