open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity
llm-ingestion-okf/src/llm_ingestion_okf/extract.py
Kjell Tore Guttormsen bc39e8091f feat(assets): a bundle carries the images its sources declare (0.10.0) 
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 10:01:31 +02:00

2411 lines
105 KiB
Python
Raw Blame History

"""Door B extraction registry: dropped file bytes -> text, per file type.
All file-type -> text extraction lives here (the guard is text-only). The core
registry is stdlib-only and deterministic: `md`/`txt` pass through, `csv` renders
the Phase 1 markdown table, `json` is fenced verbatim, and `html`/`htm` are
reduced to text with `html.parser`. Binary types are `[extract]`-gated and each
now has a reader: `pdf` through `pdfplumber`, and the five office rows
(`docx`/`xlsx`/`pptx`/`odt`/`rtf`) through a table-driven converter seam over
the vendored binary. Every one of those gates is an IMPORT PROBE rather than a
membership test, so an absent extra is rejected with the same typed error
whatever the type. Never a silent skip and never a bundled parser in core.
Two of the five office rows are `measured` and three are `unmeasured` -- the
corpus this arm was built on contains zero `pptx`, `odt` or `rtf` files, so
those rows work by construction and have never met a document anyone wrote.
`_EVIDENCE` carries that per row and the suite asserts it, because an
unmeasured row must not read as a supported one.
`extract_text` returns the extracted text *content*; final LF framing and the
concept frontmatter are the materializer's concern (Phase 2 step 2), not this
registry's. No guard call and no model call anywhere in this module.
"""
from __future__ import annotations
import base64
import binascii
import collections
import csv
import functools
import io
import re
import statistics
import tempfile
import urllib.parse
import warnings
import zipfile
from collections.abc import Callable, Sequence
from dataclasses import dataclass
from html.parser import HTMLParser
from pathlib import Path
from xml.etree import ElementTree
from xml.etree.ElementTree import Element
from .assets import (
AssetRejection,
ExtractedImage,
encode_png,
read_image,
render_block,
render_missing,
sniff,
)
from .connectors import safe_resolve
from .errors import ExtractionError, ExtractionWarning, SourceError
from .render import render_fenced_block, render_table
# Binary types gated behind the optional `[extract]` extra that it ships no
# parser for. EMPTY, and kept rather than deleted: the dispatch branch it feeds
# still raises `extractor_extra_missing`, and a later type that arrives before
# its reader belongs here rather than in a new mechanism. Every type the extra
# names now has a reader -- `.pdf` through the import probe in `_extract_pdf`,
# the five office rows through the converter seam below -- so the gate for all
# of them is an import probe, which is why the two tests for that code reach it
# that way.
_UNPARSED_OPTIONAL_EXTENSIONS: frozenset[str] = frozenset()
# The office rows: suffix -> the converter's reader name. THESE ROWS AND NO
# OTHERS. `.html` is excluded although the converter can read it: it already
# has a stdlib extractor here, so routing it through the converter would buy
# nothing and would add CVE-2025-51591 (SSRF via an iframe in HTML input),
# unpatched in every converter version. `.epub` is excluded on the "no gain"
# half of the same reason.
_PANDOC_FORMATS: dict[str, str] = {
".docx": "docx",
".xlsx": "xlsx",
".pptx": "pptx",
".odt": "odt",
".rtf": "rtf",
}
# What each row's behaviour actually rests on, asserted in the suite rather
# than written in a comment that rots. THREE classes, and the third exists
# because the first two could not tell the truth about these rows:
#
# measured real CORPUS files and a hand-counted fasit. Someone wrote the
# document for their own purposes and we counted what we got.
# constructed hand-built or generator-built documents with a hand-written
# fasit, and no corpus file at all. The row has now met a
# document end to end -- it is not `unmeasured` -- but a
# document written to exercise it is not a document anyone
# wrote, so it is not `measured` either.
# unmeasured the corpus contains ZERO files of the type AND no document
# has ever been put through the row. It works by construction.
#
# An unmeasured row must not read as a supported one, and neither must a
# constructed one.
#
# THE THREE OFFICE ROWS MOVED unmeasured -> constructed ON 2026-09-09, each on
# its own hand-built document (N = 1, except `pptx` at N = 2):
#
# .odt 1 of 1 declared headings recovered, 1 concept, 0 characters in no
# segment.
# .pptx 2 of 2 declared slide titles recovered on a deck that DECLARES them
# (`<p:ph type="title"/>`), 0 of 2 on one that does not -- the latter
# lands as `Slide 1`/`Slide 2`, which is the converter naming a slide
# it has no title for, not a segmentation failure.
# .rtf 0 declared headings, because the container has no heading style and
# the author's title is bold text. The proposer therefore proposes
# nothing and the document reaches Door B's INBOX as one concept --
# content preserved, structure zero. That is the row's honest result
# and it is the one open finding of the three.
#
# `.html` JOINED THE TABLE 2026-09-09, as `measured`, and the class was chosen
# against the definitions above rather than assumed: the 828 files are a
# consumer's own export of a real published handbook, produced for their
# ingestion and not to exercise this row, with a fasit written before any
# lookup -- which is `measured`'s test, "someone wrote the document for their
# own purposes and we counted what we got". What that class does NOT claim, and
# the honesty limit that travels with it: the 828 files are ONE product in ONE
# format from ONE publisher, and the file boundaries and `<h1>`s are a
# generator's cut of that document, not 828 documents anyone wrote.
#
# `.xml` JOINED THE TABLE 2026-09-11, as `measured`, and the class was read off
# the definitions above rather than inherited: the one file is a publisher's own
# NISO-STS delivery of R761, written for their purposes years before any lookup
# of ours, and its 2 761 titled `<sec>` are a fasit nobody here authored. The
# honesty limit that travels with it and does NOT move when the build reaches
# the reader's ceiling: the denominator is ONE file, ONE publisher, ONE schema.
# `.xml` as a file type is far wider than NISO-STS, and a document in any other
# schema keeps its text in document order and gets no structure at all -- which
# is measured on fixtures, not on a corpus.
#
# `.pdf` JOINED THE TABLE 2026-09-10, as `measured`, and it enters on the
# strongest evidence of any row here: eight real corpus PDFs with a fasit the
# operator hand-counted document by document, plus a 701-page process code
# whose PUBLISHER also ships a NISO-STS structure for it -- 2 761 titled
# sections, written for their own purposes and long before any lookup of ours.
# The honesty limit that travels with it: those 2 761 rows are ONE product in
# ONE format from ONE publisher, its structure is a strict numbered hierarchy
# on 2 739 of 2 761 titles, and a running prose document would measure
# something else entirely.
_EVIDENCE: dict[str, str] = {
".pdf": "measured",
".docx": "measured",
".xlsx": "measured",
".pptx": "constructed",
".odt": "constructed",
".rtf": "constructed",
".html": "measured",
".xml": "measured",
}
# Load-bearing, all three, and none of them hygiene:
#
# --eol=lf the defaults produce DIFFERENT BYTES (maximum line length 75
# --wrap=none against 447), which a byte-pinned golden registers as a change
# nobody made.
# -t markdown never `-t plain`: plain destroys the headings the segment
# proposer reads. Measured -- a document yielding 15 entries
# including two real headings yields 13 with none under `plain`,
# so the writer choice silently sets the ceiling for the arm
# downstream of it.
_PANDOC_WRITER = "markdown"
_PANDOC_ARGS = ("--eol=lf", "--wrap=none")
# The spreadsheet row writes PIPE tables, and it is the only row that does.
#
# The default writer prefers simple tables, which pad every cell out to the
# width of the widest cell in its column. Measured on the K2 price sheet: one
# 594-character prose cell turned every other row in that column into a run of
# up to 887 spaces between a label and its amount, 100 795 characters in all,
# and the header row named ONE column because only the first cell of the source
# row 1 is filled. The bytes reached the reader and the structure did not. The
# same sheet through this writer is 11 221 characters with no whitespace run
# longer than two, one row per line, each source column its own cell.
#
# `--columns=1` is load-bearing rather than cosmetic: the pipe writer pads cells
# out to the column width it computes from that setting, so at the default 72 a
# NARROW table gains runs of up to 45 spaces -- the same defect at a smaller
# scale. Measured across every office fixture and every K2 office file, the
# longest whitespace run with it is 2.
#
# SPREADSHEET-ONLY, deliberately. The other four rows have the same defect
# available to the same one-line fix (measured: the odt fixture 1366 -> 1105
# characters), but a spreadsheet IS a grid with no prose fallback, while moving
# the prose rows would move a corpus denominator that nothing has measured.
# `tests/test_extract.py` pins that scoping with three digests.
_SPREADSHEET_WRITER = "markdown-simple_tables-multiline_tables-grid_tables"
_SPREADSHEET_ARGS = (*_PANDOC_ARGS, "--columns=1")
# SpreadsheetML's namespace, needed to read the workbook's shared string table.
_SSML = "http://schemas.openxmlformats.org/spreadsheetml/2006/main"
# A table cell whose whole content is an integer with the converter's trailing
# `.0`. Bounded by unescaped pipes on both sides so a cell containing an escaped
# `\|` can never be split in the middle.
_INTEGRAL_CELL = re.compile(r"(?<!\\)\|(\s*)(-?\d+)\.0(\s*)(?=(?<!\\)\|)")
# Conversion recovers text, on the same terms as PDF extraction: a drawing has
# no text to recover. Said out loud on every conversion rather than detected
# per document, for the same reason.
_OFFICE_LOSSY_WARNING = (
"office-file conversion recovers text only: figures, diagrams, images and "
"drawn shapes are not represented in the output (their captions are). A "
"bundle built from drawn documents is incomplete by construction."
)
# Text extraction recovers text. A figure is a vector drawing with no text to
# recover — only its caption survives — so any bundle built from drawn
# documents is incomplete by construction. Said out loud on every PDF rather
# than detected per document: deciding "is there a figure here" is a layout
# heuristic this library does not own.
_PDF_LOSSY_WARNING = (
"PDF extraction recovers text only: figures, diagrams and images are not "
"represented in the output (their captions are). A bundle built from "
"drawn documents is incomplete by construction."
)
# Tags whose text content is never document prose.
#
# TWO MEMBERS, and it stays two in round 11. Dropping `nav`/`header`/`footer`
# as well is a DIFFERENT change with a different guarantee: the exact
# text-preservation invariant below holds only while nothing is dropped, and a
# quiet widening here would hide exactly how many characters left the document.
_SKIP_TAGS = frozenset({"script", "style"})
# The heading tags, and the ATX level each becomes. The level is the TAG's:
# a flat `#` for every heading would hand `propose._ATX` three top-level
# boundaries where the document declares one section and two subsections.
_HEADING_TAGS: dict[str, int] = {f"h{level}": level for level in range(1, 7)}
# Tags that open a line of their own. Everything NOT here is inline and stays a
# word boundary inside the current line, which is what `b`/`em`/`a`/`span` were
# already treated as.
#
# WHY THIS IS A SET AND NOT THE FIVE TAGS THE CORPUS EXERCISES. Block versus
# inline is a property of HTML, not of one corpus. The measured corpus writes
# its prose in `p`, `li` and `tr`; a `div`-structured page -- the ordinary shape
# of hand-written and exported HTML -- carries the same prose in containers this
# corpus never uses, and restricting the set to what was measured would leave
# that page collapsing into one line, which IS the defect. Adding a line break
# never removes text and never invents a boundary on its own: a boundary needs a
# line that MATCHES a grammar.
_BLOCK_TAGS = frozenset(
{
"p",
"li",
"tr",
"pre",
"div",
"section",
"article",
"header",
"footer",
"nav",
"main",
"aside",
"table",
"thead",
"tbody",
"tfoot",
"caption",
"ul",
"ol",
"dl",
"dt",
"dd",
"blockquote",
"figure",
"figcaption",
"hr",
"address",
"form",
"fieldset",
"legend",
"title",
"body",
}
)
# --- NISO-STS, and the two facts the whole reader turns on ------------------
#
# ONE: `<label>` carries the number and `<title>` carries the text. Measured on
# the 701-page process code, 2 of its 2 761 `<title>` strings begin with a
# digit -- the number is a sibling element, never glued on. A fasit that shows
# `"2.1Hovedprosesser"` is its BUILDER joining the two. Emitting `<title>`
# alone therefore scores 0 of 2 761 while every line of this file looks right,
# because the number is what okf reduces to a directory name.
#
# TWO: a `<sec>` with a `<label>` and no `<title>` is not a section heading. It
# is a lettered point (`a)`, `c)`, `sec-type="spec"`) inside a process
# description, and there are 4 954 of them against the document's own 2 761.
# One heading each and the document's structure is the minority of its own
# outline.
_STS_ROOT = "standard"
# Inline by allowlist, block by default -- the INVERSE of the HTML reader, and
# for the reason that reader gives for its own direction. Block versus inline
# is a property of HTML; XML has no such universal, so an unknown element
# cannot be assumed inline without fusing two paragraphs into one line. It can
# safely be assumed block: an extra line break never removes text and never
# invents a boundary, because a boundary needs a line that MATCHES a grammar.
#
# The members are NISO-STS's own inline set, and they are load-bearing rather
# than decorative: that document carries 1 701 `<italic>` and 1 396 `<bold>`
# inside its prose, so breaking on them would shred a paragraph into fragments
# that are individually true and collectively unreadable.
_XML_INLINE_TAGS = frozenset(
{
"italic",
"bold",
"underline",
"sup",
"sub",
"sc",
"monospace",
"roman",
"sans-serif",
"overline",
"strike",
"xref",
"ext-link",
"uri",
"std-ref",
"inline-formula",
"styled-content",
"named-content",
"break",
}
)
# The maximum ATX level `propose._ATX` can read (`#{1,6}`), which `_HEADING_TAGS`
# stops at for the same reason. STS nesting goes DEEPER: 9 of the 2 761 titled
# sections in that document sit at depth 7, and `#######` matches nothing at
# all. The depth is CLIPPED rather than dropped -- a clipped heading still sets
# its boundary, where a dropped one loses the section entirely. The clip is the
# HEADING's alone (K3-21): the `OutlineMark` beside it carries the declared
# depth, because the declared route builds its plan from the mark and a plan
# reading 7 as 6 gave a depth-7 section the ancestor one level too high.
_ATX_MAX_LEVEL = 6
# --- assets: the images a document carries beside its text ------------------
#
# ADDED IN 0.10.0, and off unless the caller asks. Until then no reader here
# fetched, named or copied an image: `<img>`'s attributes were never read, an
# STS `<graphic>` was walked past, a PDF was opened for `extract_text` alone,
# and the converter's markdown writer dropped every picture. The two lossiness
# warnings above said so on every run, which made the loss honest and did not
# make it smaller.
#
# THE READER PLACES, THE COLLECTOR DECIDES. Each reader knows where in its own
# document an image stands and what the document calls it; nothing else. What
# an image IS, what it is named in the bundle and how it is pointed at is
# `assets.py`'s, so the four readers cannot drift into four grammars, and
# "carried N of M" means one thing across all of them.
#: How a reader asks for bytes the document only POINTS at. The inbox supplies
#: one rooted at the dropped file's own directory and refuses to leave the drop
#: -- an `<img src="../../../etc/passwd">` is a path traversal written by
#: whoever wrote the document, which is exactly the class of input this package
#: treats as untrusted. `None` means the caller gave no resolver, and every
#: pointer then resolves to nothing rather than to a guess.
Resolver = Callable[[str], bytes | None]
#: A scheme this package will not open. Extraction opens no socket at all --
#: network access here is an explicit per-run opt-in and extraction is not on
#: that path -- so a remote image is carried as a POINTER and never as bytes.
_REMOTE_SOURCE = re.compile(r"^(?:[a-zA-Z][a-zA-Z0-9+.-]*:|//)")
_DATA_URI = re.compile(r"^data:(?P<media>[^;,]*)(?P<base64>;base64)?,(?P<payload>.*)$", re.DOTALL)
@dataclass(frozen=True)
class ExtractedDocument:
"""One dropped file's text, and the images that stand inside that text.
The text is what it always was when `images` and `rejected` are empty --
which is what the `assets=False` default guarantees byte for byte, so every
golden, every pinned bundle and every published digest is unmoved by this
capability existing.
"""
text: str
images: tuple[ExtractedImage, ...] = ()
rejected: tuple[AssetRejection, ...] = ()
class _AssetCollector:
"""What a reader hands an image to, and the only place a rejection is made.
A rejection is a ROW, never a failed document: one unreadable picture must
not cost the text around it. Every rejection also leaves a line in the
concept saying what was there and why it is not here -- an absence a reader
is never shown is an absence they cannot weigh, which is the defect this
whole capability closes one level up.
"""
def __init__(self, resolve: Resolver | None = None) -> None:
self._resolve = resolve
self.images: list[ExtractedImage] = []
self.rejected: list[AssetRejection] = []
def carry(self, data: bytes, *, name: str, label: str | None = None) -> str:
"""Bytes the reader already holds, as the block that stands in the text."""
try:
image = read_image(data, name=name, label=label)
except ExtractionError as exc:
return self.reject(name, code=exc.code, reason=str(exc), label=label)
self.images.append(image)
return render_block(image)
def reject(
self,
name: str,
*,
code: str,
reason: str,
label: str | None = None,
href: str | None = None,
) -> str:
self.rejected.append(AssetRejection(name=name, code=code, reason=reason))
return render_missing(name, reason=reason, label=label, href=href)
def local(self, source: str, *, label: str | None = None, sibling: str | None = None) -> str:
"""An href the DOCUMENT points at: a data URI, a local path, or remote.
`sibling` is a SECOND path to try, and it exists because one format's
convention is not another's: a NISO-STS delivery writes a bare file
name and ships the files in a `graphics/` directory beside the
document. Passing the candidate rather than teaching this method about
STS keeps the convention with the reader that has it, and the
containment rule with the resolver that owns it.
"""
data_uri = _DATA_URI.match(source)
if data_uri is not None:
return self._data_uri(data_uri, label=label)
if _REMOTE_SOURCE.match(source):
return self.reject(
source,
code="asset_remote",
reason="the source is off this machine and extraction opens no socket",
label=label,
href=source,
)
data = self._resolve(source) if self._resolve is not None else None
if data is None and sibling is not None and sibling != source and self._resolve is not None:
data = self._resolve(sibling)
if data is None:
return self.reject(
source,
code="asset_unresolved",
reason="the file the document points at was not found beside it",
label=label,
)
return self.carry(data, name=source, label=label)
def _data_uri(self, match: re.Match[str], *, label: str | None) -> str:
payload = match.group("payload")
name = f"data-uri{Path(match.group('media').split('/')[-1] or 'bin').suffix or ''}"
try:
raw = (
base64.b64decode(payload, validate=True)
if match.group("base64")
else urllib.parse.unquote_to_bytes(payload)
)
except (binascii.Error, ValueError) as exc:
return self.reject(
name,
code="asset_unresolved",
reason=f"the inline data URI could not be decoded: {exc}",
label=label,
)
return self.carry(raw, name=name, label=label)
def directory_resolver(root: Path) -> Resolver:
"""A resolver rooted at ONE directory, fail-closed, reading nothing else.
Containment is against the DOCUMENT'S OWN directory rather than against the
whole drop, and that is not caution for its own sake: the proposer reads a
file straight off disk and the door reads it out of the inbox, and a plan
indexes the exact string it was proposed against. One root both sides can
compute from the document alone is what makes the two renderings identical
without threading a second path through either. A reference above the
document's directory is refused (`asset_unresolved`) rather than followed;
the limit is stated in the concept, like every other rejection.
`safe_resolve` is Door A's own rule, reused verbatim: `..` traversal, an
absolute path, a symlink escape and a prefix-collision sibling all fail
closed. An `<img src="../../../etc/passwd">` is a path written by whoever
wrote the document, which is exactly the class of input this package treats
as untrusted.
"""
def resolve(relative: str) -> bytes | None:
try:
target = safe_resolve(root, relative)
except SourceError:
return None
try:
if not target.is_file():
return None
return target.read_bytes()
except OSError:
return None
return resolve
def decode_text(data: bytes) -> str:
"""Decode file bytes as UTF-8 (BOM-stripping), typed on failure.
utf-8-sig so a byte-order mark never leaks into the first character
(baseline parity with Door A's read_csv). A non-UTF-8 file is a corrupt
input: fail fast with a typed error rather than leaking UnicodeDecodeError.
"""
try:
return data.decode("utf-8-sig")
except UnicodeDecodeError as exc:
raise ExtractionError(
f"file bytes are not valid UTF-8: {exc}", code="extractor_decode_error"
) from exc
def _extract_passthrough(data: bytes) -> str:
"""`md`/`txt`: the decoded text verbatim."""
return decode_text(data)
def _extract_csv(data: bytes) -> str:
"""`csv`: parse with the stdlib reader, render the Phase 1 markdown table."""
reader = csv.reader(io.StringIO(decode_text(data)))
header = next(reader, None)
if header is None:
raise ExtractionError("CSV has no header row", code="extractor_empty_csv")
rows = list(reader)
return render_table(header, rows)
def _extract_json(data: bytes) -> str:
"""`json`: the decoded text verbatim inside a fenced block (Phase 1 renderer)."""
return render_fenced_block(decode_text(data))
class _HTMLTextExtractor(HTMLParser):
"""Collect document text as LINES, skipping `script`/`style`.
A block tag opens a line of its own, a heading tag opens one carrying the
ATX marker for its level, `br` breaks the current line, and every other tag
stays what it always was: a word boundary inside the line, so adjacent
inline text (``<b>a</b>b``) does not fuse. Runs of whitespace inside a line
collapse to single spaces.
THE OUTPUT GRAMMAR IS MARKDOWN, and deliberately the same markdown the
office rows reach the proposer through. `_ATX` and every other boundary
grammar is line-anchored, so this class decides -- alone -- whether an HTML
document can be segmented at all. It emitted one line for any input until
2026-09-09, which is why 828 of 828 real sections produced zero boundaries.
TEXT IS PRESERVED EXACTLY. The only characters this adds are the ATX
markers; strip those and the non-whitespace sequence is identical to the
one-line form. Nothing is ever dropped here beyond `_SKIP_TAGS`.
"""
def __init__(self, collector: _AssetCollector | None = None) -> None:
super().__init__(convert_charrefs=True)
self._lines: list[str] = []
self._current: list[str] = []
self._prefix = ""
self._skip_depth = 0
self._collector = collector
def _break(self, prefix: str = "") -> None:
"""Close the line being accumulated and open the next one."""
line = " ".join("".join(self._current).split())
self._current = []
if line:
self._lines.append(f"{self._prefix}{line}")
self._prefix = prefix
def _image(self, attrs: list[tuple[str, str | None]]) -> None:
"""An `<img>`, in the place it stands.
`attrs` was read by nothing here until 0.10.0, so `src` and `alt` were
both dropped -- a document's figures left no trace in the concept at
all, not even their alt text. The block goes on its own lines because
markdown wants a paragraph and because every boundary grammar in
`propose.py` is line-anchored.
"""
if self._collector is None or self._skip_depth:
return
values = {key: value or "" for key, value in attrs}
source = values.get("src", "").strip()
if not source:
return
self._break()
for line in self._collector.local(source, label=values.get("alt") or None).split("\n"):
self._lines.append(line)
self._break()
def _open(self, tag: str) -> bool:
"""Break for a block or heading tag; report whether it was one."""
level = _HEADING_TAGS.get(tag)
if level is not None:
self._break("#" * level + " ")
return True
if tag in _BLOCK_TAGS or tag == "br":
self._break()
return True
return False
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
if not self._open(tag):
self._current.append(" ")
if tag in _SKIP_TAGS:
self._skip_depth += 1
if tag == "img":
self._image(attrs)
def handle_startendtag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
if not self._open(tag):
self._current.append(" ")
if tag == "img":
self._image(attrs)
def handle_endtag(self, tag: str) -> None:
if tag in _SKIP_TAGS and self._skip_depth > 0:
self._skip_depth -= 1
if not self._open(tag):
self._current.append(" ")
def handle_data(self, data: str) -> None:
if self._skip_depth == 0:
self._current.append(data)
def text(self) -> str:
self._break()
return "\n".join(self._lines)
def _extract_html(data: bytes, collector: _AssetCollector | None = None) -> str:
"""`html`/`htm`: text via `html.parser`, script/style stripped (spec B3)."""
parser = _HTMLTextExtractor(collector)
parser.feed(decode_text(data))
parser.close()
return parser.text()
class _XmlTextExtractor:
"""Collect an XML document's text as LINES of markdown.
THE OUTPUT GRAMMAR IS MARKDOWN, and deliberately the same markdown the
office rows and the HTML row reach the proposer through. Nothing in
`propose.py` knows this format exists: `_ATX` reads the heading lines and
`_TABLE_ROW` reads the table ones, exactly as they read a converted
`.docx`. That is the whole reason this is an extractor and not a
segmentation arm.
TWO PATHS, and which one runs is NAMED rather than guessed:
STS the root is `<standard>` or the document contains a `<sec>`.
A `<sec>` with a `<title>` becomes one ATX line whose level is
its `<sec>`-nesting depth and whose text is `<label>` + space +
`<title>`; a `<sec>` with only a `<label>` becomes a body line
with the label in front of it, the way `li` is treated in HTML;
a `<table-wrap>` becomes its label on a line and its rows as one
markdown table block.
GENERIC everything else. Text content in document order, one line per
block-like element, and NO element name is ever promoted to a
heading. An RSS feed is not a numbered standard, and reading it
as one would state a structure its author did not.
TEXT IS PRESERVED EXACTLY. The only characters added are the ATX markers
and the table pipes; strip those and the non-whitespace sequence is
identical to `"".join(root.itertext())`.
"""
def __init__(self, *, sts: bool, collector: _AssetCollector | None = None) -> None:
self._sts = sts
self._collector = collector
self._lines: list[str] = []
self._current: list[str] = []
self._prefix = ""
# The declared structure, recorded WHERE it is written rather than
# recovered from the finished string. The PDF arm has to bridge from
# (page, `/XYZ` top) onto a line index and was wrong on 1 840 of 2 762
# nodes under the naive rule; here the reader appended the line, so the
# index is not a guess and carries no tolerance. Empty for a document
# that is not STS -- that is "this schema declares no section", and it
# must not collapse into "this document has no structure to state".
self.marks: list[OutlineMark] = []
def _break(self, prefix: str = "") -> None:
"""Close the line being accumulated and open the next one.
A pending prefix SURVIVES a break that emitted nothing -- a label-only
`<sec>` holds its `a)` until the first line that has words in it, which
may be several empty elements later.
"""
line = " ".join("".join(self._current).split())
self._current = []
if line:
self._lines.append(f"{self._prefix}{line}")
self._prefix = ""
if prefix:
# A prefix still pending here belongs to a section that turned out
# to have no body line at all, and REPLACING it would drop it from
# the document. Measured on R761: exactly one `x)`, two characters,
# which is the whole distance between 0.999998 and exact.
if self._prefix:
self._lines.append(self._prefix.rstrip())
self._prefix = prefix
def _emit(self, line: str) -> None:
"""Put a whole line out, ahead of whatever is being accumulated.
A prefix still pending is FLUSHED first rather than carried past a
heading: the section it belongs to is above this one, and holding it
would either attach it to the wrong body or lose it outright.
"""
self._break()
if self._prefix:
self._lines.append(self._prefix.rstrip())
self._prefix = ""
self._lines.append(line)
def _text_of(self, element: Element) -> str:
"""An element's whole text, whitespace collapsed."""
return " ".join("".join(element.itertext()).split())
def _spec_point(self, section: Element) -> str | None:
"""The FIRST `<p>` of the FIRST direct-child `sec-type="spec"`, whole.
The limit is this package's, not the spec's: SPEC SS 4.1 asks for "a
single sentence" and sets no length anywhere. It is STRUCTURAL rather
than a character count, because a cut inside a paragraph writes a
sentence the source never wrote. Measured on the one STS document this
row has: 2 026 of 2 761 titled sections carry a direct-child spec
point; 264 of those points hold more than one `<p>` and 2 hold none;
the first `<p>` runs 17 / 109 / 273 / 521 / 942 characters at min /
median / p90 / p99 / max.
A DIRECT child only: a spec point belongs to the section it opens under,
and a container borrowing its first child's would describe a section by
a sentence about another one.
"""
for child in section:
if _local_name(child.tag) == "sec" and child.get("sec-type") == "spec":
for paragraph in child:
if _local_name(paragraph.tag) == "p":
return self._text_of(paragraph) or None
return None
return None
def _table(self, element: Element) -> bool:
"""A `<table-wrap>`: its label on a line, its rows as ONE table block.
The separator line is what makes it a block rather than two pipe lines
-- `--table-grid` and `--keep-table-heading` read the block, and the
PDF path delivered 0 of this document's 10 tables as one.
"""
rows = [
[self._text_of(cell) for cell in row if cell.tag in ("td", "th")]
for row in element.iter("tr")
]
rows = [row for row in rows if row]
if not rows:
return False
label = element.find("label")
if label is not None:
self._emit(self._text_of(label))
caption = element.find("caption")
if caption is not None:
self._emit(self._text_of(caption))
self._break()
self._lines.extend(render_table(rows[0], rows[1:]).rstrip("\n").split("\n"))
return True
def _graphic(self, element: Element) -> bool:
"""A `<graphic>`, in the place it stands. Reports whether it was one.
Measured on the R761 delivery, 2026-09-16: 50 `<graphic>` elements, all
50 direct children of a `<sec>`, none inside a `<table-wrap>`, none
carrying a caption element of any kind -- the "Figur 11.1 ..." line a
human reads is a sibling `<p>` this reader already emits on its own
line. So the label falls back to the file name rather than being
guessed from the neighbourhood.
TWO RESOLUTION ROUTES, and the second is the delivery's own convention:
the href as written, and then `graphics/<name>`, because that publisher
writes a BARE file name and ships the files in a sibling directory.
Both are tried through the caller's resolver, which is what keeps the
containment rule in one place.
"""
if self._collector is None:
return False
href = next(
(value for key, value in element.attrib.items() if _local_name(key) == "href"),
None,
)
if not href:
return False
block = self._collector.local(href, sibling=f"graphics/{Path(href).name}")
self._emit_lines(block.split("\n"))
return True
def _emit_lines(self, lines: list[str]) -> None:
for line in lines:
self._emit(line)
def _walk(self, element: Element, depth: int) -> None:
tag = _local_name(element.tag)
if tag in ("graphic", "inline-graphic") and self._graphic(element):
return
if self._sts and tag == "table-wrap" and self._table(element):
return
skip: set[int] = set()
if self._sts and tag == "sec":
depth += 1
label = element.find("label")
title = element.find("title")
if title is not None:
level = min(depth, _ATX_MAX_LEVEL)
parts = [self._text_of(label)] if label is not None else []
parts.append(self._text_of(title))
heading = " ".join(part for part in parts if part)
self._emit("#" * level + " " + heading)
self.marks.append(
OutlineMark(
line=len(self._lines) - 1,
# The DECLARED depth, never the clipped one: the mark
# is not a markdown heading, and the plan the declared
# route builds from it reads nesting off this level.
level=depth,
title=heading,
description=self._spec_point(element),
)
)
skip = {id(title)} | ({id(label)} if label is not None else set())
elif label is not None:
# NEVER a heading. The label goes in FRONT of the body line the
# way `li` is treated in the HTML reader.
self._break(self._text_of(label) + " ")
skip = {id(label)}
inline = _local_name(element.tag) in _XML_INLINE_TAGS
if not inline:
self._break()
else:
self._current.append(" ")
if element.text:
self._current.append(element.text)
for child in element:
if id(child) not in skip:
self._walk(child, depth)
if child.tail:
self._current.append(child.tail)
if not inline:
self._break()
def text(self, root: Element) -> str:
self._walk(root, 0)
self._break()
if self._prefix:
self._lines.append(self._prefix.rstrip())
return "\n".join(self._lines)
def _xml_document(
data: bytes, collector: _AssetCollector | None = None
) -> tuple[str, tuple[OutlineMark, ...]]:
"""`xml`: NISO-STS structure as markdown, any other schema as its text.
A DTD IS REFUSED RATHER THAN PARSED, and that is a guarantee about this
code instead of one about the machine. Measured on this interpreter
(3.14.0, `pyexpat.version_info` 2.7.3): an external `SYSTEM` entity is
refused by the stdlib and never fetched, but the entity-amplification limit
that stops a billion-laughs comes from libexpat >= 2.4.0 and NOT from
Python -- five levels still expanded, six and seven were refused -- while
`pyproject.toml` requires only `>=3.10` and no lockfile pins an
interpreter. `XMLParser` exposes no `.parser` attribute on the C
accelerator either, so the handler route is not portable. Refusing every
document type declaration costs nothing here (0 of 1 file carries one) and
holds on every interpreter.
"""
root = _parse_xml(data)
reader = _XmlTextExtractor(sts=_is_sts(root), collector=collector)
return reader.text(root), tuple(reader.marks)
def _parse_xml(data: bytes) -> Element:
"""The one parse, with the DTD refusal in front of it (see `_xml_document`)."""
text = decode_text(data)
prologue = text[: text.find("<", text.find("<") + 1) + 1] if "<" in text else text
if "<!DOCTYPE" in prologue or "<!DOCTYPE" in text[:4096]:
raise ExtractionError(
"XML carrying a document type declaration is refused unparsed: a DTD can "
"define entities, and the parser's amplification limit is a property of "
"the installed libexpat rather than of this package",
code="extractor_xml_doctype",
)
try:
return ElementTree.fromstring(text)
except ElementTree.ParseError as exc:
raise ExtractionError(
f"the XML parser failed on this file: {exc}", code="extractor_xml_parse_error"
) from exc
def _is_sts(root: Element) -> bool:
"""The NAMED schema test: a `<standard>` root, or any `<sec>`."""
return _local_name(root.tag) == _STS_ROOT or next(root.iter("sec"), None) is not None
@dataclass(frozen=True)
class DeclaredIdentity:
"""What a document states about itself, read from its own elements.
Each field is `None` when the document does not state it, and ALSO when it
states it more than once: an adopted standard carries one `<std-ident>` per
body that issued it, and taking the first would be a guess dressed as a
reading. The caller falls back to the file name for whatever is `None`.
"""
doc_number: str | None
year: str | None
title: str | None
def declared_identity(name: str, data: bytes) -> DeclaredIdentity | None:
"""`xml`: the identity a NISO-STS document declares, or `None`.
MEASURED ON THE ONE STS DOCUMENT THIS ROW HAS: exactly one `<std-ident>`
(`<doc-number>R761 Prosesskoden</doc-number>` beside `<year>2025</year>`)
and one `<title-wrap>` whose `<full>` is the document's title -- while the
file carrying it was named for a delivery path, a UUID occurring 0 times in
the document. `<doc-type>` is read by nobody: it said `Innledning` there,
which is the name of a chapter and not a kind of document.
`None` for every other row and for XML that is not STS: a declaration is a
property of a schema, and a text that merely LOOKS like one declares
nothing. An unparseable file is `None` too, never an exception -- extracting
the same bytes refuses it with its own code, and an identity is not the
place a document is refused.
"""
if Path(name).suffix.lower() != ".xml":
return None
try:
root = _parse_xml(data)
except ExtractionError:
return None
if not _is_sts(root):
return None
declared = [
(_child_text(element, "doc-number"), _child_text(element, "year"))
for element in root.iter()
if _local_name(element.tag) == "std-ident"
]
declared = [pair for pair in declared if pair[0]]
doc_number, year = declared[0] if len(declared) == 1 else (None, None)
wraps = [element for element in root.iter() if _local_name(element.tag) == "title-wrap"]
title = (
(_child_text(wraps[0], "full") or _child_text(wraps[0], "main"))
if len(wraps) == 1
else None
)
if doc_number is None and title is None:
return None
return DeclaredIdentity(doc_number=doc_number, year=year, title=title)
def _child_text(element: Element, name: str) -> str | None:
"""A direct child's whole text, whitespace collapsed; `None` when absent or empty."""
for child in element:
if _local_name(child.tag) == name:
return " ".join("".join(child.itertext()).split()) or None
return None
def _extract_xml(data: bytes, collector: _AssetCollector | None = None) -> str:
return _xml_document(data, collector)[0]
def xml_outline(
name: str, data: bytes, *, assets: bool = False, resolve: Resolver | None = None
) -> tuple[OutlineMark, ...]:
"""`xml`: the sections the document DECLARES, as marks on the extracted text.
The counterpart of `pdf_outline`, and the difference between them is the
whole point of the row. A bookmark states a page and a y position, so that
arm has to BRIDGE onto a line and reports what did not bridge; an STS
`<sec><title>` is written into the output by this reader, so the line index
is the one it appended at -- nothing is recovered, nothing is unresolved,
and there is no tolerance constant to choose.
RE-READS the bytes rather than returning both from one call, for the same
reason `pdf_outline` does: `extract_text` has one signature that every
caller and every registry entry is keyed to, and a second return value
would change it for eight rows to serve one. The parse is stdlib and the
document is read twice; measured on a 2.4 MB NISO-STS file, that is the
smaller cost by a wide margin.
Empty for every schema that is not STS. That is a statement about the
document -- it declares no section -- and `find_candidates` reads an empty
list as "leave every rule untouched", never as a route.
"""
del name # the registry decides which reader runs; kept for `pdf_outline`'s shape
# `assets` and `resolve` are NOT options of this arm, exactly as
# `pdf_headings` is not one of `pdf_outline`'s: carrying an image inserts
# lines into the extracted text, so marks computed with the images off name
# the right sections at the wrong line numbers. They are threaded so both
# sides of the plan can be computed against ONE rendering.
return _xml_document(data, _AssetCollector(resolve) if assets else None)[1]
def _local_name(tag: str) -> str:
"""`{ns}sec` -> `sec`. A namespaced document names the same elements."""
return tag.rsplit("}", 1)[-1]
def _extra_missing(suffix: str) -> ExtractionError:
"""The one rejection for a `[extract]` type without the extra installed.
One constructor, one wording: the import probe and the still-unparsed
types must be indistinguishable to a consumer, because they are the same
fact — the extra is not installed.
"""
return ExtractionError(
f"extracting {suffix!r} requires the optional 'extract' extra "
f"(pip install 'llm-ingestion-okf[extract]'); it is not installed",
code="extractor_extra_missing",
)
def _ocr_group_missing() -> ExtractionError:
"""The one rejection for `--ocr` without the optional `ocr` group.
A DIFFERENT code from `extractor_extra_missing`, because it is a different
fact and a different remedy: the `[extract]` extra can be fully installed
-- the document parsed, the pages counted -- and the OCR engine still be
absent. One error naming both would send an operator to reinstall
something they already have.
"""
return ExtractionError(
"reading a PDF page with OCR requires the optional 'ocr' group "
"(pip install 'llm-ingestion-okf[extract,ocr]'), which ships rapidocr "
"on onnxruntime; it is not installed",
code="extractor_ocr_group_missing",
)
#: The literal placeholder `pdfminer.six` (behind `pdfplumber`) emits for a
#: glyph whose font carries no usable ToUnicode mapping. The text is present on
#: the page and unreadable in the extraction -- a failure that looks like
#: success, which is why it needs a measurement rather than an exception.
_CID_CODE = re.compile(r"\(cid:\d+\)")
#: The share of a page's extracted characters that must be `(cid:N)` codes
#: before `--ocr` reads the page as an image instead.
#:
#: MEASURED, not chosen: `docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md`
#: reports the per-page distribution over the K2 corpus, and it is bimodal
#: with nothing in between -- one document's pages sit near 1.0 and every other
#: page in the corpus sits at 0.0. Any value in that gap selects the same
#: pages, which is what makes 0.10 defensible and also what makes it
#: uninformative about a corpus that has intermediate pages. Stated rather than
#: implied: this threshold is bounded by the corpus, not by a property of the
#: format.
OCR_CID_SHARE = 0.10
#: The resolution a page is rendered at before it is read as an image.
#: 200 dpi is what the round-4 measurement was taken at; the engine's own
#: preprocessing rescales from there, so this is a floor on how much of the
#: page's detail reaches it rather than a tuning knob. It is part of the output
#: contract in the same way the parser version is: OCR text is deterministic
#: within one resolution and one model version, and across neither.
OCR_DPI = 200
def cid_share(text: str) -> float:
"""The share of `text` made of `(cid:N)` placeholder codes, 0.0 for empty.
Module level and importable: `tools/okf_cid_measure.py` answers the same
question at DOCUMENT level, and two definitions of one metric drift.
"""
if not text:
return 0.0
return sum(len(match.group(0)) for match in _CID_CODE.finditer(text)) / len(text)
def needs_ocr(text: str) -> bool:
"""Whether a page's extracted text is unusable enough to read the image.
TWO conditions, because there are two ways a page's text never arrives and
they look nothing alike: a page with no text layer extracts as the empty
string, and a page whose fonts carry no ToUnicode mapping extracts as a
full page of `(cid:N)`. A trigger written for one of them would leave the
other exactly where it was.
"""
return not text.strip() or cid_share(text) >= OCR_CID_SHARE
def _ocr_reader() -> Callable[[object], list[str]]:
"""The OCR engine, or the typed refusal. The import IS the gate.
Same shape as `_extract_pdf`'s probe and for the same reason: membership in
a suffix set cannot tell whether a package is importable, and this group is
the one a consumer is most likely not to have.
"""
try:
import rapidocr
except ImportError as exc:
raise _ocr_group_missing() from exc
if rapidocr is None: # pragma: no cover - the sys.modules probe in tests
raise _ocr_group_missing()
engine = rapidocr.RapidOCR()
def read(image: object) -> list[str]:
result = engine(image)
# `txts` is None when the detector found nothing at all, which is a
# legitimate answer for a blank page and not an error.
return [str(line) for line in (getattr(result, "txts", None) or ())]
return read
#: Bold as a PDF says it: in the font's NAME (`Helvetica-Bold`,
#: `ABCDEF+Arial-BoldMT`). There is no weight attribute on a character, so the
#: name is the only place a text extractor can read it.
_PDF_BOLD_MARKER = "bold"
#: The deepest ATX level the emitted markdown may use. `_ATX` in `propose.py`
#: reads one to six hashes, and a document with seven distinct heading sizes
#: would otherwise emit a line the proposer reads as body.
_PDF_MAX_HEADING_LEVEL = 6
def _dominant(values: list[str]) -> str:
"""The most frequent value, ties broken by first occurrence.
`Counter.most_common(1)` reduces to `max` over the items in insertion
order, so the tie-break is document order and the result is deterministic
for identical bytes -- which is the property everything downstream is
pinned to.
"""
return collections.Counter(values).most_common(1)[0][0]
def _typography(line: dict[str, object]) -> tuple[float, str] | None:
"""One line's dominant font size and font name, or `None` if it is blank.
Blank characters are excluded from both: a space carries a size and a font
like any other character, and a heading padded with body-sized spaces would
read as body.
"""
chars = [char for char in line["chars"] if str(char["text"]).strip()] # type: ignore[attr-defined]
if not chars:
return None
sizes = [f"{float(char['size']):.1f}" for char in chars]
fonts = [str(char["fontname"]) for char in chars]
return float(_dominant(sizes)), _dominant(fonts)
def _heading_levels(lines: list[tuple[str, float, str]]) -> dict[float, int]:
"""Which font sizes are headings in this document, and at what ATX level.
The rule is the CONJUNCTION this repository already measured: larger than
the body AND bold. `docs/2026-09-07-k3-arm-d.md`'s predecessor measured
size-and-bold from poppler at recall 1.000 / precision 0.846, and measured
that adding weight as a DISJUNCT made precision worse (0.786 -> 0.524). A
disjunction here would mark every emphasised phrase in the body.
The body size is the CHARACTER-weighted median over the whole document, not
the page: a title page is 100 % heading by line count, and a per-page
median would compare it with itself and mark nothing. Weighted by
characters rather than lines for the same reason in miniature -- a document
front-loaded with short lines has a line median that no paragraph shares.
The ATX LEVEL is the size's rank among the heading sizes, largest first, so
a document's own typographic hierarchy survives into the markdown instead
of flattening to one level. Deeper than six is clamped, because `_ATX`
reads six.
"""
weighted: list[float] = []
for text, size, _ in lines:
weighted.extend([size] * len(text.replace(" ", "")))
if not weighted:
return {}
body = statistics.median(weighted)
sizes = {size for _, size, font in lines if size > body and _PDF_BOLD_MARKER in font.lower()}
return {
size: min(rank, _PDF_MAX_HEADING_LEVEL)
for rank, size in enumerate(sorted(sizes, reverse=True), start=1)
}
def _mark_headings(lines: list[tuple[str, float, str]], levels: dict[float, int]) -> str:
"""One page's lines as markdown, the heading sizes carrying their hashes.
BOLD is checked again here rather than folded into the size map: a document
can set a caption in the same size as a heading without setting it bold,
and a map keyed on size alone would promote it.
"""
out: list[str] = []
for text, size, font in lines:
level = levels.get(size) if _PDF_BOLD_MARKER in font.lower() else None
out.append(f"{'#' * level} {text}" if level is not None and text else text)
return "\n".join(out)
# How `_extract_pdf` joins its pages, named because the locator below has to
# reproduce the exact same arithmetic to turn a character offset back into a
# page number. Two constants that must agree, written once.
_PDF_PAGE_SEPARATOR = "\n\n"
@dataclass(frozen=True)
class _PdfPage:
"""One page's text, and the images drawn on it, kept APART on purpose.
The pointer blocks are appended after the page's own lines by
`_pdf_page_text`, and the body is kept separately because `pdf_outline`
compares the page's line splitting against `page.extract_text_lines()` --
a per-page check that ships and that decides whether the primary bridge
route may be used at all. Appended lines are not in that geometry, so a
joined string would fail the check on every page carrying an image and
silently demote 2 762 bookmarks to the fallback route.
"""
number: int
text: str
images: tuple[ExtractedImage, ...] = ()
rejected: tuple[AssetRejection, ...] = ()
def _pdf_page_text(page: _PdfPage) -> str:
"""A page as it reaches the extracted text: its lines, then its pointers.
END OF PAGE, not the image's y position, and the reason is stated rather
than hidden: a PDF image has a bounding box and no place in the reading
order, so "where it stands" is the page. Inserting by y would reorder the
page's own lines against the geometry `pdf_outline` checks itself against,
and would put a pointer inside a sentence. A caption printed above a figure
therefore keeps its own line where the document put it, and the pointer
follows the page it was drawn on.
"""
blocks = [render_block(image) for image in page.images]
blocks += [
render_missing(rejection.name, reason=rejection.reason) for rejection in page.rejected
]
if not blocks:
return page.text
joined = "\n\n".join(blocks)
return f"{page.text}\n\n{joined}" if page.text else joined
#: The bits-per-component this encoder expresses. A PDF may store 1, 2, 4, 8 or
#: 16, and everything but 8 is REFUSED with a code rather than rescaled --
#: rescaling a 1-bit stencil to 8 bits is a decision about what black means, and
#: a wrong one looks exactly like a right one.
_PDF_SAMPLE_BITS = 8
def _pdf_colour(space: object) -> tuple[int, bytes | None] | None:
"""A PDF colour space as `(channels, palette)`, or `None` if not expressible.
`None` is the honest answer for CMYK, for a separation space and for
anything with a transfer function: converting those needs a colour model
this package does not carry, and a guess would be a picture that is
plausibly the wrong colour. It is counted and stated, never approximated.
"""
from pdfminer.pdftypes import PDFStream, resolve1
space = resolve1(space)
name = getattr(space, "name", None)
if name in ("DeviceGray", "CalGray", "G"):
return 1, None
if name in ("DeviceRGB", "CalRGB", "RGB"):
return 3, None
if not isinstance(space, list) or not space:
return None
head = getattr(resolve1(space[0]), "name", None)
if head == "ICCBased" and len(space) > 1:
profile = resolve1(space[1])
components = resolve1(profile.attrs.get("N")) if isinstance(profile, PDFStream) else None
return (int(components), None) if components in (1, 3) else None
if head in ("CalGray",):
return 1, None
if head in ("CalRGB", "Lab"):
return 3, None
if head in ("Indexed", "I") and len(space) >= 4:
base = _pdf_colour(space[1])
if base is None:
return None
lookup = resolve1(space[3])
if isinstance(lookup, PDFStream):
lookup = lookup.get_data()
if not isinstance(lookup, bytes):
return None
if base[0] == 3:
palette = lookup[: (len(lookup) // 3) * 3]
else:
# PNG's PLTE is RGB triples only, so a grey palette is widened
# rather than refused. Widening a grey to r=g=b is exact, not an
# approximation -- which is why this branch exists and the CMYK one
# does not.
palette = b"".join(bytes([value, value, value]) for value in lookup)
return (1, palette) if palette else None
return None
def _pdf_alpha(attrs: dict[str, object], width: int, height: int) -> bytes | None | bool:
"""A soft mask as one alpha byte per pixel, `None` for none, `False` to refuse.
An `SMask` this encoder cannot express is a REFUSAL rather than a dropped
channel: an image whose transparency is thrown away is composited against
nothing and reads as a black or white rectangle over the page, which is a
picture that is wrong in a way no consumer can detect.
"""
from pdfminer.pdftypes import PDFStream, resolve1
mask = resolve1(attrs.get("SMask"))
if mask is None:
return None
if not isinstance(mask, PDFStream):
return False
shape = mask.attrs
if (
resolve1(shape.get("Width")) != width
or resolve1(shape.get("Height")) != height
or resolve1(shape.get("BitsPerComponent")) != _PDF_SAMPLE_BITS
):
return False
try:
alpha = mask.get_data()
except Exception:
return False
return alpha if len(alpha) >= width * height else False
def _pdf_image(stream: object, name: str) -> ExtractedImage:
"""One image XObject, carried verbatim where it already is a file.
TWO ROUTES, and which one runs is decided by the BYTES rather than by the
filter name. `get_data()` applies every filter pdfminer knows and stops at
the image codecs, so a `DCTDecode` stream comes back as a finished JPEG and
a `FlateDecode` one comes back as raw samples. Sniffing the result is what
makes the first route exact: an embedded JPEG is written to the bundle as
the publisher's own bytes, unre-encoded, and its content-addressed name is
therefore stable for as long as the document is.
Measured on R761 (2026-09-16): 29 of 50 image objects are `DCTDecode` and
take the verbatim route; 21 are `FlateDecode` and are encoded here. Over
the 33-document K2 reference corpus the population is 4 828 objects, and
the filters are mixed enough (`FlateDecode`, `DCTDecode`, `JPXDecode`,
`ASCII85Decode` chains, `CCITTFaxDecode`) that guessing from the filter
name would have been wrong on several hundred.
RENDERING THE PAGE REGION WAS THE ALTERNATIVE AND IT WAS NOT TAKEN. A
rasterised crop would be one code path and would handle every filter, but
its bytes -- and therefore the asset's name and the bundle's digest --
would depend on the version of the rasteriser installed, which is the one
property `OCR_DPI`'s docstring already admits OCR text cannot have. An
embedded stream has no such dependency.
"""
from pdfminer.pdftypes import resolve1
try:
data = stream.get_data() # type: ignore[attr-defined]
except Exception as exc:
raise ExtractionError(
f"the PDF image stream behind {name!r} could not be decoded: {exc}",
code="asset_pdf_unsupported",
) from exc
if data and sniff(data) is not None:
return read_image(data, name=name)
attrs = dict(getattr(stream, "attrs", {}))
width = resolve1(attrs.get("Width"))
height = resolve1(attrs.get("Height"))
bits = resolve1(attrs.get("BitsPerComponent"))
if not isinstance(width, int) or not isinstance(height, int):
raise ExtractionError(
f"the PDF image {name!r} declares no usable size",
code="asset_pdf_unsupported",
)
if resolve1(attrs.get("ImageMask")):
raise ExtractionError(
f"the PDF image {name!r} is a stencil mask, which paints the current "
"fill colour rather than carrying one of its own",
code="asset_pdf_unsupported",
)
if bits != _PDF_SAMPLE_BITS:
raise ExtractionError(
f"the PDF image {name!r} stores {bits}-bit samples; this encoder writes "
f"{_PDF_SAMPLE_BITS}-bit ones and will not rescale, because rescaling a "
"stencil is a decision about what black means",
code="asset_pdf_unsupported",
)
if attrs.get("Decode") is not None:
raise ExtractionError(
f"the PDF image {name!r} carries a Decode array, which remaps every "
"sample; carrying it unmapped would invert the picture",
code="asset_pdf_unsupported",
)
colour = _pdf_colour(attrs.get("ColorSpace"))
if colour is None:
raise ExtractionError(
f"the PDF image {name!r} uses a colour space this encoder does not "
f"express ({attrs.get('ColorSpace')!r})",
code="asset_pdf_unsupported",
)
alpha = _pdf_alpha(attrs, width, height)
if alpha is False:
raise ExtractionError(
f"the PDF image {name!r} has a soft mask this encoder cannot express; "
"dropping transparency would composite the picture against nothing",
code="asset_pdf_unsupported",
)
channels, palette = colour
encoded = encode_png(
width,
height,
data,
channels=channels,
palette=palette,
alpha=alpha if isinstance(alpha, bytes) else None,
)
return read_image(encoded, name=name)
def _pdf_images(page: object) -> tuple[tuple[ExtractedImage, ...], tuple[AssetRejection, ...]]:
"""Every image drawn on one page, with the failures kept beside them."""
carried: list[ExtractedImage] = []
rejected: list[AssetRejection] = []
number = getattr(page, "page_number", 0)
for index, drawn in enumerate(getattr(page, "images", []) or [], start=1):
# The name a PDF image does NOT have. An XObject is reached through a
# resource name local to one page's dictionary, so it is not an
# identifier -- the page number in front of it is what makes the string
# readable, and the content-addressed digest is what makes it unique.
label = str(drawn.get("name") or index).lstrip("/")
name = f"page-{number}-{label}"
stream = drawn.get("stream")
if stream is None:
rejected.append(
AssetRejection(name, "asset_pdf_unsupported", "the image object has no stream")
)
continue
try:
carried.append(_pdf_image(stream, name))
except ExtractionError as exc:
rejected.append(AssetRejection(name, exc.code, str(exc)))
return tuple(carried), tuple(rejected)
@functools.lru_cache(maxsize=1)
def _pdf_pages(
data: bytes, headings: bool = False, ocr: bool = False, assets: bool = False
) -> tuple[_PdfPage, ...]:
"""Every page that produced content, as a `_PdfPage`, in page order.
The page NUMBER is 1-based and comes from the document, so a page that
yielded nothing removes itself from the sequence without renumbering the
ones after it -- which is the difference between "the third page that
produced text" and "page 3", and the whole reason a locator is worth
writing down.
Memoised on the bytes AND on the two options, with room for exactly one
entry: extraction and location are two calls about the same file with the
same options, back to back, and parsing it twice would double the PDF cost
of every corpus run for nothing. The options are part of the key because
two renderings of one document are two different strings, and a locator
built against the wrong one points at the wrong place with full confidence.
`headings` and `ocr` are INDEPENDENT and compose. With both off this is the
path every byte-pinned golden was measured on, unchanged: the default
branch still calls `page.extract_text()` rather than reassembling the page
from its lines. Measured, the two agree on 11 of 11 pages of a real tender
PDF -- but "agree on the document I tried" is not a contract, so the
default does not depend on it.
`assets` is the third, and with it off not one line below it runs: no
stream is decoded, no sample buffer is allocated, and the emitted pages are
the objects they always were. A page that produced no TEXT is still
dropped even when it carries an image, because `_extract_pdf` refuses a
document with no text at all (`extractor_empty_pdf`) and an image-only
document is `--ocr`'s question, not this one's.
"""
try:
import pdfplumber
except ImportError as exc:
raise _extra_missing(".pdf") from exc
read = _ocr_reader() if ocr else None
try:
with pdfplumber.open(io.BytesIO(data)) as pdf:
# PASS ONE. Nothing is emitted here, because the heading rule needs
# a fact about the WHOLE document -- the body's size -- and a page
# cannot supply it. A title page is 100 % heading, and a per-page
# median would compare it with itself and mark nothing.
recovered: list[str | list[tuple[str, float, str]]] = []
numbers: list[int] = []
drawn: list[tuple[tuple[ExtractedImage, ...], tuple[AssetRejection, ...]]] = []
for page in pdf.pages:
flat = (page.extract_text() or "").rstrip()
numbers.append(page.page_number)
drawn.append(_pdf_images(page) if assets else ((), ()))
if read is not None and needs_ocr(flat):
# The page's own text is unusable, so it is replaced
# WHOLESALE rather than merged with: a page of `(cid:N)`
# has nothing worth keeping, and interleaving two readings
# of one page would put a guess and a fact in one paragraph
# with no way to tell them apart. An OCR'd page carries no
# typography either -- the engine reports text, not fonts --
# so it is a finished string and never a heading candidate.
recovered.append("\n".join(read(page.to_image(resolution=OCR_DPI).original)))
elif not headings:
recovered.append(flat)
else:
recovered.append(
[
(str(line["text"]), *found)
for line in page.extract_text_lines()
if (found := _typography(line)) is not None
]
)
levels = _heading_levels(
[
line
for page_lines in recovered
if not isinstance(page_lines, str)
for line in page_lines
]
)
# PASS TWO.
pages = [
page_lines
if isinstance(page_lines, str)
else _mark_headings(page_lines, levels).rstrip()
for page_lines in recovered
]
except ExtractionError:
raise
except Exception as exc:
raise ExtractionError(
f"the PDF parser failed on this file: {exc}", code="extractor_pdf_error"
) from exc
return tuple(
_PdfPage(number=number, text=text, images=images, rejected=rejected)
for number, text, (images, rejected) in zip(numbers, pages, drawn)
if text
)
@dataclass(frozen=True)
class OutlineMark:
"""One `/Outlines` node, placed on a LINE of the string `extract_text` returns.
`level` is what the TREE declares, not a distance normalised against
anything: a document whose outline carries its own root node puts its
chapters at level 2, and rewriting that here would state a structure the
publisher did not. Measured on a 701-page process code -- the tree's levels
2..8 hold 28/118/500/1141/872/93/9 nodes against the publisher's own
NISO-STS depths 1..7 at 28/118/500/1141/868/97/9, so the mapping is level
minus one on five rows and the publisher disagrees with the publisher on
four nodes. That disagreement is data, and it survives only if the level is
reported rather than fixed up.
"""
line: int
level: int
title: str
#: The section's own first spec point, where the SOURCE declares one. Set
#: only by the NISO-STS reader; `None` on every bookmark mark, because a
#: bookmark declares a place and never a summary.
description: str | None = None
@dataclass(frozen=True)
class PdfOutline:
"""The bookmark tree, bridged onto lines -- with what did not bridge counted.
`unresolved` is not decoration. A `/Dest` that names an object which is not
a page, or a page that produced no text, has to be DROPPED: fabricating a
boundary from it would put a heading somewhere the document never had one,
and raising would refuse a file over a defect in one of its bookmarks.
Dropping silently is the third option this library refuses everywhere else,
so the count is part of the return value.
`collided` is the same principle applied to the OTHER way a node leaves
without a boundary. Two bookmarks can resolve to one line -- measured on
the 701-page process code, its tree's root node and `SVV - Forside` both
land on line 0 -- and only the first can become a mark, because two
candidates at one offset give the first an empty span that the orphan check
then deletes without a word. That was measured on that document and is why
keeping both was felled rather than argued. What the count buys is the
identity: NODES IN == len(marks) + unresolved + collided, so a document
that loses several nodes this way says so instead of returning a shorter
list that looks complete.
"""
marks: tuple[OutlineMark, ...]
unresolved: int
collided: int = 0
def _outline_page_and_top(doc: object, dest: object, action: object) -> tuple[object, float | None]:
"""`(page reference, /XYZ top)` from a bookmark's destination, or `(None, None)`.
Four shapes reach here and all four are in the wild: an explicit array, a
NAMED destination resolved through the document's name tree, a `GoTo`
action carrying either, and an indirect reference to any of them.
"""
target = dest
if target is None and action is not None:
resolved = action.resolve() if hasattr(action, "resolve") else action
if isinstance(resolved, dict):
target = resolved.get("D")
if isinstance(target, (bytes, str)) or hasattr(target, "name"):
name = target.name if hasattr(target, "name") else target
try:
target = doc.get_dest(name) # type: ignore[attr-defined]
except Exception:
return (None, None)
if hasattr(target, "resolve"):
try:
target = target.resolve()
except Exception:
return (None, None)
if isinstance(target, dict):
target = target.get("D")
if not isinstance(target, list) or not target:
return (None, None)
top: float | None = None
if len(target) > 3 and getattr(target[1], "name", None) == "XYZ":
candidate = target[3]
if isinstance(candidate, (int, float)):
top = float(candidate)
return (target[0], top)
def pdf_outline(
name: str,
data: bytes,
*,
pdf_headings: bool = False,
ocr: bool = False,
assets: bool = False,
) -> PdfOutline:
"""`pdf`: the file's own `/Outlines` tree, as marks on the extracted text.
THE BRIDGE IS THE WHOLE PROBLEM, and both routes are measured rather than
argued. A bookmark states a PAGE and a y position; a candidate needs a LINE
index. On the 701-page document this was built against, 2 706 of 2 761
bookmarks share a destination page with another bookmark, so the page alone
is never a cut point.
Y ROUTE (primary). `page.extract_text_lines()` carries a `top` per line,
and the mark takes the FIRST line at or below the destination. It needs
the line splitting to be the one `page.extract_text()` produced -- an
assumption, so it is CHECKED per page and the route is used only where
the two strings are identical. Measured: 701 of 701 pages, and the
resulting index agrees with the title route on 2 762 of 2 762 nodes,
flat from a 0 pt tolerance to 8 pt and collapsing at 12 (the line
spacing). It therefore ships with NO tolerance constant at all.
TITLE ROUTE (fallback). The bookmark's title, normalised, searched in the
destination page's own lines. It resolved 2 762 of 2 763 on that
document, and its weakness is real: a title like `Armering` occurs nine
times in that structure, so it is scoped to the destination page and is
never asked a question the y route already answered.
`pdf_headings` and `ocr` are passed through so the line indices address the
SAME rendering the caller extracted. They are not options of this arm: a
plan indexes one exact string, and marks computed against another one point
at the right words in the wrong places.
"""
if Path(name).suffix.lower() != ".pdf":
return PdfOutline((), 0)
try:
import pdfplumber
except ImportError as exc:
raise _extra_missing(".pdf") from exc
from pdfminer.pdfdocument import PDFNoOutlines
from pdfminer.pdfpage import PDFPage
rendered = _pdf_pages(data, pdf_headings, ocr, assets)
starts: dict[int, int] = {}
page_lines: dict[int, list[str]] = {}
# The page's OWN lines, without the pointer blocks appended after them.
# `starts` has to count the appended lines (they are in the text a mark
# indexes) while the geometry check must not see them, because
# `extract_text_lines` reports the page and knows nothing about what this
# package added underneath it.
body_lines: dict[int, list[str]] = {}
offset = 0
for rendered_page in rendered:
starts[rendered_page.number] = offset
page_lines[rendered_page.number] = _pdf_page_text(rendered_page).split("\n")
body_lines[rendered_page.number] = rendered_page.text.split("\n")
offset += len(page_lines[rendered_page.number]) + 1
unresolved = 0
collided = 0
placed: dict[int, OutlineMark] = {}
with pdfplumber.open(io.BytesIO(data)) as pdf:
try:
nodes = list(pdf.doc.get_outlines())
except PDFNoOutlines:
# NOT an error, and not zero concepts either: this file simply
# carries no index, which is the common case and the one the
# byte-identical guarantee below rests on.
return PdfOutline((), 0)
except Exception as exc:
raise ExtractionError(
f"the PDF parser failed reading /Outlines: {exc}",
code="extractor_pdf_error",
) from exc
numbers = {
page.pageid: index + 1 for index, page in enumerate(PDFPage.create_pages(pdf.doc))
}
wanted: dict[int, list[tuple[int, str, float | None]]] = {}
for level, title, dest, action, _ in nodes:
reference, top = _outline_page_and_top(pdf.doc, dest, action)
page_number = numbers.get(getattr(reference, "objid", None))
if page_number is None or page_number not in starts:
unresolved += 1
continue
wanted.setdefault(page_number, []).append((int(level), str(title), top))
# Geometry is read only for the pages that carry a bookmark, because
# `extract_text_lines` costs a second render of every page it is asked
# about -- 78 s over 701 pages, and nothing at all over the pages no
# bookmark points at.
for page in pdf.pages:
number = page.page_number
group = wanted.get(number)
if not group:
continue
lines = page_lines[number]
tops: list[float] | None = None
geometry = page.extract_text_lines()
if [str(entry["text"]) for entry in geometry] == body_lines[number]:
tops = [float(entry["top"]) for entry in geometry]
height = float(page.height)
for level, title, top in group:
index: int | None = None
if tops is not None and top is not None:
want = height - top
index = next(
(position for position, value in enumerate(tops) if value >= want),
len(tops) - 1,
)
else:
target = _normalise_outline(title)
joined = ""
bounds: list[int] = []
for rendered_line in lines:
bounds.append(len(joined))
joined += _normalise_outline(rendered_line)
found = joined.find(target)
if found >= 0:
index = max(position for position, at in enumerate(bounds) if at <= found)
if index is None:
unresolved += 1
continue
at = starts[number] + index
# FIRST in tree order wins a shared line. Two marks on one line
# would give the second an empty span, and the orphan check
# deletes an empty span silently -- the same trap the proposer
# documents for a second-pass candidate list. The loser is
# COUNTED rather than dropped: `setdefault` alone made a lost
# node indistinguishable from a node that was never there.
if at in placed:
collided += 1
continue
placed[at] = OutlineMark(line=at, level=level, title=title)
return PdfOutline(tuple(placed[at] for at in sorted(placed)), unresolved, collided)
def _normalise_outline(value: str) -> str:
"""Whitespace out, case folded -- the form the title route compares on."""
return re.sub(r"\s+", "", value).lower()
def _extract_pdf(
data: bytes, *, headings: bool = False, ocr: bool = False, assets: bool = False
) -> str:
"""`pdf`: page text via `pdfplumber`, in page order, pages separated by a
blank line.
The gate is this import, not a membership test: without the `[extract]`
extra the very same typed rejection is raised as for the types that ship
no parser at all. Text is returned VERBATIM — no Unicode normalization,
matching `md`/`txt` passthrough; normalizing would edit source content,
and NFC folding belongs to filenames and titles, not to document bodies.
`pdfplumber` was chosen on ONE measured property (2026-08-21,
docs/2026-08-21-g2-pdf-extraction-measurement.md): on a real requirement
table it keeps label and value on the same line, where pypdf, pdfminer.six
and pymupdf each emit all labels then all values. Re-pairing those is
guesswork, and in a requirements document a wrong pairing looks right.
"""
pages = _pdf_pages(data, headings, ocr, assets)
text = _PDF_PAGE_SEPARATOR.join(_pdf_page_text(page) for page in pages)
if not text:
raise ExtractionError(
"the PDF yielded no text on any page; a scanned or image-only "
"document needs OCR, which this registry does only behind the "
"optional 'ocr' group and only when asked",
code="extractor_empty_pdf",
)
# After the parse, not before: a run that produced no text has nothing to
# be lossy about, and warning there would just add noise to a failure.
warnings.warn(_PDF_LOSSY_WARNING, ExtractionWarning, stacklevel=3)
return text
def _convert_bytes(source: bytes, to: str, format: str, extra_args: Sequence[str]) -> str:
"""The one converter call, isolated so the seam above it is testable.
Separated for a reason beyond tidiness: every test of the seam's behaviour
would otherwise need the binary present and a real office document, which
would make the seam's own logic untestable on a machine without the extra.
The conversion itself is covered by the frozen-text fixtures instead.
THE INPUT GOES THROUGH A FILE, NOT THROUGH THE TEXT ENTRY POINT. Every
format here is a binary container, and the converter's text entry point
takes an `encoding` because it treats its source as text -- which corrupts
a zip. Measured: a hand-laid `.xlsx` that pandoc reads correctly from disk
fails through the text path with `Failed to unpack XLSX archive: not enough
bytes`. A `.docx` of the same shape happened to survive, which is what
makes this worth writing down: the defect is SILENT for some inputs and
fatal for others, so "it worked on the file I tried" is not evidence here.
The temporary directory is removed on every path, including the failure
one, and nothing outside it is written.
"""
import pypandoc
from ._pandoc import converter_path
with tempfile.TemporaryDirectory() as staging:
staged = Path(staging) / f"input.{format}"
staged.write_bytes(source)
with converter_path():
return str(
pypandoc.convert_file(str(staged), to, format=format, extra_args=list(extra_args))
)
#: A markdown image as the converter's own writer emits it. Measured against
#: pandoc 3.10.2 on hand-laid fixtures: a `.docx` picture arrives as
#: `![<descr>](<path>){width="..." height="..."}` and a `.pptx` one as
#: `![<descr>](<path> "<name>")`, so the title form and the attribute form are
#: both real and a regex written for one of them silently leaves the other's
#: link in the text.
_MEDIA_LINK = re.compile(
r"!\[(?P<alt>[^\]]*)\]\("
r"(?:<(?P<angle>[^>]*)>|(?P<plain>[^)\s]*))"
r'(?:\s+"(?P<title>[^"]*)")?\)'
r"(?P<attrs>\{[^}]*\})?"
)
def _convert_with_media(
source: bytes, to: str, format: str, extra_args: Sequence[str]
) -> tuple[str, dict[str, bytes]]:
"""The converter call again, with `--extract-media` and the files read back.
A separate function rather than a flag on `_convert_bytes` because the
media must be READ INSIDE the temporary directory's lifetime: the directory
is removed on every path, and a caller handed a rewritten markdown string
pointing into it would hold links to files that no longer exist. Returning
the bytes is what makes the seam closed.
The staging path is absolute, so the converter writes absolute links. That
is deliberate: every one of them is replaced below, and a link that somehow
survived would carry a temporary directory name into a concept -- a string
that differs on every run, which a byte-determinism rule would catch loudly
rather than never.
"""
import pypandoc
from ._pandoc import converter_path
with tempfile.TemporaryDirectory() as staging:
staged = Path(staging) / f"input.{format}"
staged.write_bytes(source)
media_root = Path(staging) / "extracted"
with converter_path():
text = str(
pypandoc.convert_file(
str(staged),
to,
format=format,
extra_args=[*extra_args, f"--extract-media={media_root}"],
)
)
media: dict[str, bytes] = {}
if media_root.is_dir():
for path in sorted(media_root.rglob("*")):
if path.is_file():
media[str(path)] = path.read_bytes()
return text, media
def _rewrite_media_links(text: str, media: dict[str, bytes], collector: _AssetCollector) -> str:
"""Every converter image link, replaced by this package's own pointer block.
UNCONDITIONAL, including the links that cannot be resolved. The converter
already emitted a markdown image before this existed -- measured on a
hand-laid `.docx`, today's output carries
`![Tabell 84-2](media/tabell-84-2.png)` with no such file anywhere, which
`structure._scan_references` reads as a cross-reference to a concept that
cannot exist. Leaving an unresolvable link in place would keep that defect
and add a temporary directory name to it.
"""
def replace(match: re.Match[str]) -> str:
target = match.group("angle") or match.group("plain") or ""
label = match.group("alt") or match.group("title") or None
data = media.get(target)
if data is not None:
# The name the CONTAINER gave it, not the staging path: pandoc
# preserves the part name under its own media directory, so
# `word/media/tabell-84-2.png` arrives as `media/tabell-84-2.png`.
inside = target.split("/extracted/", 1)[-1]
return collector.carry(data, name=inside, label=label)
if not target:
return collector.reject(
"image", code="asset_unresolved", reason="the converter emitted no target"
)
return collector.local(target, label=label)
return _MEDIA_LINK.sub(replace, text)
def _extract_office(suffix: str, data: bytes, collector: _AssetCollector | None = None) -> str:
"""The five office rows, converted through the vendored binary.
Shaped after `_extract_pdf`: the gate is an import probe rather than a
membership test, third-party failures are wrapped rather than leaked, empty
output is refused rather than persisted, and the lossiness is stated after
the parse rather than before it.
"""
try:
import pypandoc # noqa: F401
except ImportError as exc:
raise _extra_missing(suffix) from exc
spreadsheet = suffix == ".xlsx"
writer = _SPREADSHEET_WRITER if spreadsheet else _PANDOC_WRITER
args = _SPREADSHEET_ARGS if spreadsheet else _PANDOC_ARGS
try:
if collector is None:
text = _convert_bytes(data, writer, _PANDOC_FORMATS[suffix], args)
else:
text, media = _convert_with_media(data, writer, _PANDOC_FORMATS[suffix], args)
text = _rewrite_media_links(text, media, collector)
except ExtractionError:
raise
except Exception as exc:
raise ExtractionError(
f"the converter failed on this {suffix} file: {exc}",
code="extractor_convert_error",
) from exc
text = text.strip()
if not text:
raise ExtractionError(
f"the converter returned no text for this {suffix} file; refused "
"rather than persisted as an empty concept",
code="extractor_empty_conversion",
)
if spreadsheet:
text = _drop_converter_decimals(text, data)
# After the parse, not before: a run that produced no text has nothing to
# be lossy about, and warning there would just add noise to a failure.
warnings.warn(_OFFICE_LOSSY_WARNING, ExtractionWarning, stacklevel=3)
return text
def _shared_strings(data: bytes) -> frozenset[str]:
"""Every literal in a workbook's shared string table, or nothing.
Read for one purpose: to tell a NUMBER from TEXT THAT LOOKS LIKE ONE. The
converter renders a numeric cell as a double, so an integral value arrives
as `5647500.0` -- and a text cell reading `92.0` arrives as `92.0` too. The
output alone cannot separate them, and rewriting on the output alone would
silently edit somebody's authored text.
Shared strings are the only text the converter recovers from a sheet at
all: an inline string (`t="inlineStr"`) is read as an EMPTY cell, measured
while the first xlsx fixture was built (`tests/fixtures/README.md`). So a
`<digits>.0` that is not in this set did not come from text.
Every failure returns the empty set, which makes the rewrite a no-op rather
than a guess: a workbook this cannot read keeps its converter decimals.
"""
try:
with zipfile.ZipFile(io.BytesIO(data)) as archive:
raw = archive.read("xl/sharedStrings.xml")
root = ElementTree.fromstring(raw)
except (KeyError, OSError, zipfile.BadZipFile, ElementTree.ParseError):
return frozenset()
return frozenset(
"".join(node.text or "" for node in item.iter(f"{{{_SSML}}}t")) for item in root
)
def _drop_converter_decimals(text: str, data: bytes) -> str:
"""Undo the converter's `N.0` on cells the workbook stores as integers.
Cell-scoped and never applied to prose: the pattern is anchored between two
unescaped pipes, so only a cell whose ENTIRE content is an integer with a
trailing `.0` is rewritten, and only when that same literal is absent from
the shared string table.
"""
literals = _shared_strings(data)
def rewrite(match: re.Match[str]) -> str:
digits = match.group(2)
if f"{digits}.0" in literals:
return match.group(0)
return f"|{match.group(1)}{digits}{match.group(3)}"
return _INTEGRAL_CELL.sub(rewrite, text)
_CORE_EXTRACTORS: dict[str, Callable[[bytes], str]] = {
".md": _extract_passthrough,
".txt": _extract_passthrough,
".csv": _extract_csv,
".json": _extract_json,
".html": _extract_html,
".htm": _extract_html,
".xml": _extract_xml,
}
# Types the `[extract]` extra ships a parser for. Kept separate from the core
# registry so "adds no runtime dependency" stays readable at a glance.
_OPTIONAL_EXTRACTORS: dict[str, Callable[[bytes], str]] = {
".pdf": _extract_pdf,
**{suffix: functools.partial(_extract_office, suffix) for suffix in _PANDOC_FORMATS},
}
# --- provenance: a character range of the extracted text -> a place in the
# original document ---------------------------------------------------------
#
# `source_offset` alone is a position in OUR extraction, so following it back
# needs the corpus directory, the extractor and its exact version -- none of
# which a bundle carries. A unit table is that mapping, saved AT EXTRACTION
# where the two are known to agree, rather than guessed afterwards from text
# whose page breaks are gone.
#
# THE UNIT IS PER FORMAT AND IS NAMED, never assumed:
#
# pages a PDF page number, from the document itself.
# rows a spreadsheet row, within the sheet named by `scope_of`.
# lines a line of the EXTRACTED text. For `md`/`txt` that text is the
# dropped file, so the number is the original's own line; for the
# converted formats it is not, and the key says `lines` rather than
# `paragraphs` for exactly that reason. Measured on the five K2
# `.docx` documents: `<w:p>` counts 108/27/65/176/57 against
# converted-markdown line counts 75/33/67/144/63 -- not one pair
# agrees, so a `paragraphs` key would name a number the original does
# not have.
#
# The heading a spreadsheet's sheet becomes, as the converter writes it:
# `## <sheet name> {#sheet-<n>}`. Anchored to the line start so a pipe cell
# containing a `#` cannot be read as a sheet.
_SHEET_HEADING = re.compile(r"^#{1,6} (?P<name>.*?) \{#sheet-\d+\}$")
# Pandoc's ATTRIBUTE syntax at the end of a heading, which is what the sheet
# and slide anchors above are an instance of. Deliberately NARROW, because the
# known-negative is the whole point: an author writing `Mal for {kundenavn}` or
# `Feltet {"id": 4}` wrote a title, and stripping that would be this same
# defect pointed the other way.
#
# The three narrowings, each doing work: the block must be at the END of the
# title (`$`), it must OPEN with `#` (pandoc's identifier -- `{.class}` and
# `{key=val}` alone are not what any converter here emits, and matching them
# would reach further than measured), and the identifier is the restricted
# character set pandoc actually generates, so a brace holding a space, a quote
# or a colon is not an attribute.
_CONVERTER_ATTRIBUTE = re.compile(r"\s*\{#[A-Za-z0-9_.:-]+\}\s*$")
# A line the converter wrote as part of a pipe table. Whether one of them is
# the table's SEPARATOR is decided by POSITION, never by content: an empty
# spreadsheet row renders as `| | |` and a separator as `|----|----|`, and
# every content rule that tells those apart also swallows a data row that
# happens to hold only dashes. Measured on the K2 price sheet: a content rule
# ate 8 empty rows and reported the sheet's last row as 92 against a workbook
# that says 100.
_TABLE_LINE = "|"
def strip_converter_attribute(title: str) -> str:
"""Remove a trailing pandoc attribute anchor from a heading's title.
ONE definition, read by both title-forming sites: `propose` names a
segment from an ATX heading, `structure` derives a document title from its
leading heading, and a rule living in only one of them would strip the
attribute on one path and leave it on the other -- with the id and the
title then disagreeing about the same concept.
Lives HERE because the attribute is a CONVERTER artefact: `_SHEET_HEADING`
above is the same syntax read for a different purpose, and this module is
the one that knows what pandoc writes. That reading must keep its
attribute, which is why the strip is applied to a title downstream and
never to the extracted text.
RENAMES CONCEPT IDS, by design and with the operator's authorisation
(2026-09-09): a filename is reduced FROM the title, so the two move
together. Measured exposure at the time: 2 of 810 concepts on the default
K2 bundle and 2 of 1108 on Arm B.
"""
return _CONVERTER_ATTRIBUTE.sub("", title)
@dataclass(frozen=True)
class SourceUnits:
"""Where in the ORIGINAL each stretch of the extracted text came from.
`starts[i]` is the character offset in the extracted text at which unit
`numbers[i]` begins, and `scopes[i]` is the sheet that unit belongs to (or
`None` for a format that has no sheets). The three tuples are parallel and
`starts` ascends, which is what lets `covering` be a bisection rather than
a scan.
`numbers` is separate from the index on purpose. A PDF page that yielded no
text is not in this table, and a pipe table's separator line is a row of
nothing -- in both cases the position in the table and the number in the
original have already parted company, and an index standing in for a number
is the off-by-one this whole object exists to prevent.
"""
unit: str
starts: tuple[int, ...]
numbers: tuple[int, ...]
scopes: tuple[str | None, ...] = ()
def __post_init__(self) -> None:
if len(self.starts) != len(self.numbers):
raise ValueError("a unit table needs one number per start offset")
if self.scopes and len(self.scopes) != len(self.starts):
raise ValueError("a unit table needs one scope per start offset, or none at all")
def _index(self, offset: int) -> int:
"""The table row covering `offset`, clamped to the table's own ends."""
low, high = 0, len(self.starts) - 1
while low < high:
middle = (low + high + 1) // 2
if self.starts[middle] <= offset:
low = middle
else:
high = middle - 1
return low
def covering(self, start: int, end: int) -> tuple[int, int]:
"""The first and last original unit the half-open `[start, end)` touches.
`end` is exclusive, so a range ending exactly where the next unit
begins does not claim that unit -- a segment that stops at a page
boundary is on the page it was written on.
"""
if not self.starts:
raise ValueError("an empty unit table locates nothing")
first = self._index(start)
last = self._index(max(start, end - 1))
return self.numbers[first], self.numbers[last]
def scope_of(self, offset: int) -> str | None:
"""The sheet `offset` falls in, or `None` for a format without sheets."""
if not self.scopes:
return None
return self.scopes[self._index(offset)]
def scopes_covering(self, start: int, end: int) -> tuple[str | None, ...]:
"""Every distinct scope the range touches, in order, without repeats."""
if not self.scopes:
return ()
first = self._index(start)
last = self._index(max(start, end - 1))
seen: list[str | None] = []
for scope in self.scopes[first : last + 1]:
if not seen or seen[-1] != scope:
seen.append(scope)
return tuple(seen)
def _line_units(text: str) -> SourceUnits:
starts: list[int] = []
offset = 0
for line in text.split("\n"):
starts.append(offset)
offset += len(line) + 1
return SourceUnits("lines", tuple(starts), tuple(range(1, len(starts) + 1)))
def _pdf_units(data: bytes, headings: bool, ocr: bool, assets: bool = False) -> SourceUnits:
starts: list[int] = []
numbers: list[int] = []
offset = 0
for page in _pdf_pages(data, headings, ocr, assets):
starts.append(offset)
numbers.append(page.number)
# The page as it reaches the text, pointers included: a locator built
# from the body alone would drift by two lines per carried image and
# would name the wrong page from the first one onwards.
offset += len(_pdf_page_text(page)) + len(_PDF_PAGE_SEPARATOR)
return SourceUnits("pages", tuple(starts), tuple(numbers))
def _spreadsheet_units(text: str) -> SourceUnits | None:
"""Sheet and row for a converted spreadsheet, or `None` if it is not one.
The converter writes one heading per sheet and then one pipe-table line per
source row, with a separator line after the first. Row numbering therefore
restarts at every heading and skips that one line by POSITION.
The row number is the ORIGINAL sheet's, and that holds exactly as far as
one converted line per `<row>` element holds. Measured on the two K2
spreadsheets and both fixtures: 39 rows for 39, 100 for 100, 4 for 4, 6 for
6 and 3 for 3 -- every one contiguous from row 1. A sheet whose XML omits a
row entirely would number from the converted table instead, and nothing
here can see that.
"""
starts: list[int] = []
numbers: list[int] = []
scopes: list[str | None] = []
sheet: str | None = None
seen = 0
offset = 0
for line in text.split("\n"):
heading = _SHEET_HEADING.match(line)
if heading is not None:
sheet = heading.group("name")
seen = 0
elif sheet is not None and line.startswith(_TABLE_LINE):
seen += 1
# The SECOND table line of a sheet is the separator the converter
# writes under the header, and it is a row of no spreadsheet. Every
# line after it is one row further on than its position suggests.
if seen != 2:
starts.append(offset)
numbers.append(seen if seen == 1 else seen - 1)
scopes.append(sheet)
offset += len(line) + 1
if not starts:
return None
return SourceUnits("rows", tuple(starts), tuple(numbers), tuple(scopes))
def source_units(
filename: str,
data: bytes,
text: str,
*,
pdf_headings: bool = False,
ocr: bool = False,
assets: bool = False,
) -> SourceUnits | None:
"""The unit table for one dropped file, or `None` when it has none.
`text` must be what `extract_text` returned for these exact bytes: the
table indexes that string, and a table built against a different rendering
would point a consumer at the wrong place with full confidence.
`None` is a measurement, not a failure -- a spreadsheet the converter wrote
no table for has no rows to name, and the caller writes the address without
a locator rather than inventing one.
"""
suffix = Path(filename).suffix.lower()
if suffix == ".pdf":
return _pdf_units(data, pdf_headings, ocr, assets)
if suffix == ".xlsx":
return _spreadsheet_units(text)
if suffix in _CORE_EXTRACTORS or suffix in _PANDOC_FORMATS:
return _line_units(text)
return None
def extract_text(
filename: str,
data: bytes,
*,
renderer: Callable[[str], str] | None = None,
pdf_headings: bool = False,
ocr: bool = False,
assets: bool = False,
resolve: Resolver | None = None,
) -> str:
"""Convert one dropped file's bytes to OKF concept text, dispatched by type.
`filename` supplies the extension (case-insensitive); `data` is the raw
bytes. A core stdlib type is extracted; a `[extract]`-gated binary type
without the extra, and any unregistered extension, fail fast with a typed
:class:`ExtractionError`. Extracting a `pdf` also emits an
:class:`ExtractionWarning`: drawn content has no text to recover.
`renderer`, when given, is applied to the EXTRACTED TEXT before it is
returned -- after extraction, never instead of it, so a renderer never has
to re-implement a reader and the two cannot drift. It is a plain callable
rather than anything profile-shaped ON PURPOSE: this module is the
extraction registry and must not import the contract layer, or the
dependency would run backwards and the registry would stop standing on its
own. Resolving a profile's NAMED renderer to a function is the caller's
job, in the layer that already holds the profile.
The default is identity, which is what keeps every existing byte-pinned
golden byte-pinned.
`pdf_headings` and `ocr` are PDF-only and both default to off. They are
branched on here rather than expressed as two more registry rows because
the registry's contract is `bytes -> str`: a row per option combination
would be four rows for one reader, and a reader chosen by a suffix lookup
that also has to consult two flags is not a lookup. A non-PDF caller
passing either argument gets today's behaviour, silently, which is correct
-- the options describe a reader, not a policy for the run.
"""
return extract_document(
filename,
data,
renderer=renderer,
pdf_headings=pdf_headings,
ocr=ocr,
assets=assets,
resolve=resolve,
).text
#: The types that can carry an image, and the reader that places it. Kept apart
#: from `_CORE_EXTRACTORS` and `_OPTIONAL_EXTRACTORS` rather than folded into
#: them, and that separation is the byte-identity guarantee: with `assets=False`
#: not one entry below is consulted and the dispatch is the one every golden,
#: every pinned bundle and every published digest was measured on. `.csv`,
#: `.json`, `.md` and `.txt` are absent because the formats carry no image;
#: `.xlsx` is absent because its converter writes one pipe table per sheet and
#: a two-line block inside one would break the row locator `_spreadsheet_units`
#: reads back out of it -- measured 2026-09-16, 0 of 4 K2 workbooks hold any
#: media at all, so the row is a limit stated rather than a loss taken.
_ASSET_READERS: dict[str, Callable[[bytes, _AssetCollector], str]] = {
".html": _extract_html,
".htm": _extract_html,
".xml": _extract_xml,
**{
suffix: functools.partial(_extract_office, suffix)
for suffix in _PANDOC_FORMATS
if suffix != ".xlsx"
},
}
def extract_document(
filename: str,
data: bytes,
*,
renderer: Callable[[str], str] | None = None,
pdf_headings: bool = False,
ocr: bool = False,
assets: bool = False,
resolve: Resolver | None = None,
) -> ExtractedDocument:
"""One dropped file as text PLUS the images that stand inside that text.
The entry point :func:`extract_text` keeps for the eight callers that want
a string, and the one Door B uses since 0.10.0. With `assets=False` -- the
default, everywhere -- this runs exactly the dispatch that existed before
the asset layer did, and returns an :class:`ExtractedDocument` whose text is
byte-identical and whose two image tuples are empty.
`resolve` answers for the formats that POINT at a file instead of embedding
it (`html`, `xml`). Without one every pointer resolves to nothing and is
stated as such; with one, containment is that resolver's rule and not this
module's. `pdf` and the office rows embed their images and never consult it.
"""
suffix = Path(filename).suffix.lower()
extractor = _CORE_EXTRACTORS.get(suffix) or _OPTIONAL_EXTRACTORS.get(suffix)
if extractor is None:
if suffix in _UNPARSED_OPTIONAL_EXTENSIONS:
raise _extra_missing(suffix)
raise ExtractionError(
f"no extractor is registered for file extension {suffix!r} ({filename!r})",
code="extractor_unknown",
)
collector = _AssetCollector(resolve) if assets else None
if suffix == ".pdf":
if pdf_headings or ocr or assets:
text = _extract_pdf(data, headings=pdf_headings, ocr=ocr, assets=assets)
else:
text = extractor(data)
if collector is not None:
for page in _pdf_pages(data, pdf_headings, ocr, True):
collector.images.extend(page.images)
collector.rejected.extend(page.rejected)
elif collector is not None and suffix in _ASSET_READERS:
text = _ASSET_READERS[suffix](data, collector)
else:
text = extractor(data)
return ExtractedDocument(
text=renderer(text) if renderer is not None else text,
images=tuple(collector.images) if collector is not None else (),
rejected=tuple(collector.rejected) if collector is not None else (),
)
Reference in a new issue View git blame Copy permalink