open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(assets): a bundle carries the images its sources declare (0.10.0)

Browse source 
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-17 10:01:31 +02:00
commit bc39e8091f
33 changed files with 3638 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

775
src/llm_ingestion_okf/extract.py
View file

@ -23,6 +23,8 @@ registry's. No guard call and no model call anywhere in this module.
from __future__ import annotations
import base64
import binascii
import collections
import csv
import functools
@ -30,6 +32,7 @@ import io
import re
import statistics
import tempfile
import urllib.parse
import warnings
import zipfile
from collections.abc import Callable, Sequence
@ -39,7 +42,17 @@ from pathlib import Path
from xml.etree import ElementTree
from xml.etree.ElementTree import Element
from .errors import ExtractionError, ExtractionWarning
from .assets import (
AssetRejection,
ExtractedImage,
encode_png,
read_image,
render_block,
render_missing,
sniff,
)
from .connectors import safe_resolve
from .errors import ExtractionError, ExtractionWarning, SourceError
from .render import render_fenced_block, render_table
# Binary types gated behind the optional `[extract]` extra that it ships no
@ -329,6 +342,174 @@ _XML_INLINE_TAGS = frozenset(
_ATX_MAX_LEVEL = 6
# --- assets: the images a document carries beside its text ------------------
#
# ADDED IN 0.10.0, and off unless the caller asks. Until then no reader here
# fetched, named or copied an image: `<img>`'s attributes were never read, an
# STS `<graphic>` was walked past, a PDF was opened for `extract_text` alone,
# and the converter's markdown writer dropped every picture. The two lossiness
# warnings above said so on every run, which made the loss honest and did not
# make it smaller.
#
# THE READER PLACES, THE COLLECTOR DECIDES. Each reader knows where in its own
# document an image stands and what the document calls it; nothing else. What
# an image IS, what it is named in the bundle and how it is pointed at is
# `assets.py`'s, so the four readers cannot drift into four grammars, and
# "carried N of M" means one thing across all of them.
#: How a reader asks for bytes the document only POINTS at. The inbox supplies
#: one rooted at the dropped file's own directory and refuses to leave the drop
#: -- an `<img src="../../../etc/passwd">` is a path traversal written by
#: whoever wrote the document, which is exactly the class of input this package
#: treats as untrusted. `None` means the caller gave no resolver, and every
#: pointer then resolves to nothing rather than to a guess.
Resolver = Callable[[str], bytes | None]
#: A scheme this package will not open. Extraction opens no socket at all --
#: network access here is an explicit per-run opt-in and extraction is not on
#: that path -- so a remote image is carried as a POINTER and never as bytes.
_REMOTE_SOURCE = re.compile(r"^(?:[a-zA-Z][a-zA-Z0-9+.-]*:|//)")
_DATA_URI = re.compile(r"^data:(?P<media>[^;,]*)(?P<base64>;base64)?,(?P<payload>.*)$", re.DOTALL)
@dataclass(frozen=True)
class ExtractedDocument:
"""One dropped file's text, and the images that stand inside that text.
The text is what it always was when `images` and `rejected` are empty --
which is what the `assets=False` default guarantees byte for byte, so every
golden, every pinned bundle and every published digest is unmoved by this
capability existing.
"""
text: str
images: tuple[ExtractedImage, ...] = ()
rejected: tuple[AssetRejection, ...] = ()
class _AssetCollector:
"""What a reader hands an image to, and the only place a rejection is made.
A rejection is a ROW, never a failed document: one unreadable picture must
not cost the text around it. Every rejection also leaves a line in the
concept saying what was there and why it is not here -- an absence a reader
is never shown is an absence they cannot weigh, which is the defect this
whole capability closes one level up.
"""
def __init__(self, resolve: Resolver | None = None) -> None:
self._resolve = resolve
self.images: list[ExtractedImage] = []
self.rejected: list[AssetRejection] = []
def carry(self, data: bytes, *, name: str, label: str | None = None) -> str:
"""Bytes the reader already holds, as the block that stands in the text."""
try:
image = read_image(data, name=name, label=label)
except ExtractionError as exc:
return self.reject(name, code=exc.code, reason=str(exc), label=label)
self.images.append(image)
return render_block(image)
def reject(
self,
name: str,
*,
code: str,
reason: str,
label: str | None = None,
href: str | None = None,
) -> str:
self.rejected.append(AssetRejection(name=name, code=code, reason=reason))
return render_missing(name, reason=reason, label=label, href=href)
def local(self, source: str, *, label: str | None = None, sibling: str | None = None) -> str:
"""An href the DOCUMENT points at: a data URI, a local path, or remote.
`sibling` is a SECOND path to try, and it exists because one format's
convention is not another's: a NISO-STS delivery writes a bare file
name and ships the files in a `graphics/` directory beside the
document. Passing the candidate rather than teaching this method about
STS keeps the convention with the reader that has it, and the
containment rule with the resolver that owns it.
"""
data_uri = _DATA_URI.match(source)
if data_uri is not None:
return self._data_uri(data_uri, label=label)
if _REMOTE_SOURCE.match(source):
return self.reject(
source,
code="asset_remote",
reason="the source is off this machine and extraction opens no socket",
label=label,
href=source,
)
data = self._resolve(source) if self._resolve is not None else None
if data is None and sibling is not None and sibling != source and self._resolve is not None:
data = self._resolve(sibling)
if data is None:
return self.reject(
source,
code="asset_unresolved",
reason="the file the document points at was not found beside it",
label=label,
)
return self.carry(data, name=source, label=label)
def _data_uri(self, match: re.Match[str], *, label: str | None) -> str:
payload = match.group("payload")
name = f"data-uri{Path(match.group('media').split('/')[-1] or 'bin').suffix or ''}"
try:
raw = (
base64.b64decode(payload, validate=True)
if match.group("base64")
else urllib.parse.unquote_to_bytes(payload)
)
except (binascii.Error, ValueError) as exc:
return self.reject(
name,
code="asset_unresolved",
reason=f"the inline data URI could not be decoded: {exc}",
label=label,
)
return self.carry(raw, name=name, label=label)
def directory_resolver(root: Path) -> Resolver:
"""A resolver rooted at ONE directory, fail-closed, reading nothing else.
Containment is against the DOCUMENT'S OWN directory rather than against the
whole drop, and that is not caution for its own sake: the proposer reads a
file straight off disk and the door reads it out of the inbox, and a plan
indexes the exact string it was proposed against. One root both sides can
compute from the document alone is what makes the two renderings identical
without threading a second path through either. A reference above the
document's directory is refused (`asset_unresolved`) rather than followed;
the limit is stated in the concept, like every other rejection.
`safe_resolve` is Door A's own rule, reused verbatim: `..` traversal, an
absolute path, a symlink escape and a prefix-collision sibling all fail
closed. An `<img src="../../../etc/passwd">` is a path written by whoever
wrote the document, which is exactly the class of input this package treats
as untrusted.
"""
def resolve(relative: str) -> bytes | None:
try:
target = safe_resolve(root, relative)
except SourceError:
return None
try:
if not target.is_file():
return None
return target.read_bytes()
except OSError:
return None
return resolve
def decode_text(data: bytes) -> str:
"""Decode file bytes as UTF-8 (BOM-stripping), typed on failure.
@ -384,12 +565,13 @@ class _HTMLTextExtractor(HTMLParser):
one-line form. Nothing is ever dropped here beyond `_SKIP_TAGS`.
"""
def __init__(self) -> None:
def __init__(self, collector: _AssetCollector | None = None) -> None:
super().__init__(convert_charrefs=True)
self._lines: list[str] = []
self._current: list[str] = []
self._prefix = ""
self._skip_depth = 0
self._collector = collector
def _break(self, prefix: str = "") -> None:
"""Close the line being accumulated and open the next one."""
@ -399,6 +581,26 @@ class _HTMLTextExtractor(HTMLParser):
self._lines.append(f"{self._prefix}{line}")
self._prefix = prefix
def _image(self, attrs: list[tuple[str, str | None]]) -> None:
"""An `<img>`, in the place it stands.
`attrs` was read by nothing here until 0.10.0, so `src` and `alt` were
both dropped -- a document's figures left no trace in the concept at
all, not even their alt text. The block goes on its own lines because
markdown wants a paragraph and because every boundary grammar in
`propose.py` is line-anchored.
"""
if self._collector is None or self._skip_depth:
return
values = {key: value or "" for key, value in attrs}
source = values.get("src", "").strip()
if not source:
return
self._break()
for line in self._collector.local(source, label=values.get("alt") or None).split("\n"):
self._lines.append(line)
self._break()
def _open(self, tag: str) -> bool:
"""Break for a block or heading tag; report whether it was one."""
level = _HEADING_TAGS.get(tag)
@ -415,10 +617,14 @@ class _HTMLTextExtractor(HTMLParser):
self._current.append(" ")
if tag in _SKIP_TAGS:
self._skip_depth += 1
if tag == "img":
self._image(attrs)
def handle_startendtag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
if not self._open(tag):
self._current.append(" ")
if tag == "img":
self._image(attrs)
def handle_endtag(self, tag: str) -> None:
if tag in _SKIP_TAGS and self._skip_depth > 0:
@ -435,9 +641,9 @@ class _HTMLTextExtractor(HTMLParser):
return "\n".join(self._lines)
def _extract_html(data: bytes) -> str:
def _extract_html(data: bytes, collector: _AssetCollector | None = None) -> str:
"""`html`/`htm`: text via `html.parser`, script/style stripped (spec B3)."""
parser = _HTMLTextExtractor()
parser = _HTMLTextExtractor(collector)
parser.feed(decode_text(data))
parser.close()
return parser.text()
@ -472,8 +678,9 @@ class _XmlTextExtractor:
identical to `"".join(root.itertext())`.
"""
def __init__(self, *, sts: bool) -> None:
def __init__(self, *, sts: bool, collector: _AssetCollector | None = None) -> None:
self._sts = sts
self._collector = collector
self._lines: list[str] = []
self._current: list[str] = []
self._prefix = ""
@ -572,8 +779,42 @@ class _XmlTextExtractor:
self._lines.extend(render_table(rows[0], rows[1:]).rstrip("\n").split("\n"))
return True
def _graphic(self, element: Element) -> bool:
"""A `<graphic>`, in the place it stands. Reports whether it was one.
Measured on the R761 delivery, 2026-09-16: 50 `<graphic>` elements, all
50 direct children of a `<sec>`, none inside a `<table-wrap>`, none
carrying a caption element of any kind -- the "Figur 11.1 ..." line a
human reads is a sibling `<p>` this reader already emits on its own
line. So the label falls back to the file name rather than being
guessed from the neighbourhood.
TWO RESOLUTION ROUTES, and the second is the delivery's own convention:
the href as written, and then `graphics/<name>`, because that publisher
writes a BARE file name and ships the files in a sibling directory.
Both are tried through the caller's resolver, which is what keeps the
containment rule in one place.
"""
if self._collector is None:
return False
href = next(
(value for key, value in element.attrib.items() if _local_name(key) == "href"),
None,
)
if not href:
return False
block = self._collector.local(href, sibling=f"graphics/{Path(href).name}")
self._emit_lines(block.split("\n"))
return True
def _emit_lines(self, lines: list[str]) -> None:
for line in lines:
self._emit(line)
def _walk(self, element: Element, depth: int) -> None:
tag = _local_name(element.tag)
if tag in ("graphic", "inline-graphic") and self._graphic(element):
return
if self._sts and tag == "table-wrap" and self._table(element):
return
skip: set[int] = set()
@ -627,7 +868,9 @@ class _XmlTextExtractor:
return "\n".join(self._lines)
def _xml_document(data: bytes) -> tuple[str, tuple[OutlineMark, ...]]:
def _xml_document(
data: bytes, collector: _AssetCollector | None = None
) -> tuple[str, tuple[OutlineMark, ...]]:
"""`xml`: NISO-STS structure as markdown, any other schema as its text.
A DTD IS REFUSED RATHER THAN PARSED, and that is a guarantee about this
@ -643,7 +886,7 @@ def _xml_document(data: bytes) -> tuple[str, tuple[OutlineMark, ...]]:
holds on every interpreter.
"""
root = _parse_xml(data)
reader = _XmlTextExtractor(sts=_is_sts(root))
reader = _XmlTextExtractor(sts=_is_sts(root), collector=collector)
return reader.text(root), tuple(reader.marks)
@ -736,11 +979,13 @@ def _child_text(element: Element, name: str) -> str | None:
return None
def _extract_xml(data: bytes) -> str:
return _xml_document(data)[0]
def _extract_xml(data: bytes, collector: _AssetCollector | None = None) -> str:
return _xml_document(data, collector)[0]
def xml_outline(name: str, data: bytes) -> tuple[OutlineMark, ...]:
def xml_outline(
name: str, data: bytes, *, assets: bool = False, resolve: Resolver | None = None
) -> tuple[OutlineMark, ...]:
"""`xml`: the sections the document DECLARES, as marks on the extracted text.
The counterpart of `pdf_outline`, and the difference between them is the
@ -762,7 +1007,12 @@ def xml_outline(name: str, data: bytes) -> tuple[OutlineMark, ...]:
list as "leave every rule untouched", never as a route.
"""
del name # the registry decides which reader runs; kept for `pdf_outline`'s shape
return _xml_document(data)[1]
# `assets` and `resolve` are NOT options of this arm, exactly as
# `pdf_headings` is not one of `pdf_outline`'s: carrying an image inserts
# lines into the extracted text, so marks computed with the images off name
# the right sections at the wrong line numbers. They are threaded so both
# sides of the plan can be computed against ONE rendering.
return _xml_document(data, _AssetCollector(resolve) if assets else None)[1]
def _local_name(tag: str) -> str:
@ -966,11 +1216,251 @@ def _mark_headings(lines: list[tuple[str, float, str]], levels: dict[float, int]
_PDF_PAGE_SEPARATOR = "\n\n"
@dataclass(frozen=True)
class _PdfPage:
"""One page's text, and the images drawn on it, kept APART on purpose.
The pointer blocks are appended after the page's own lines by
`_pdf_page_text`, and the body is kept separately because `pdf_outline`
compares the page's line splitting against `page.extract_text_lines()` --
a per-page check that ships and that decides whether the primary bridge
route may be used at all. Appended lines are not in that geometry, so a
joined string would fail the check on every page carrying an image and
silently demote 2 762 bookmarks to the fallback route.
"""
number: int
text: str
images: tuple[ExtractedImage, ...] = ()
rejected: tuple[AssetRejection, ...] = ()
def _pdf_page_text(page: _PdfPage) -> str:
"""A page as it reaches the extracted text: its lines, then its pointers.
END OF PAGE, not the image's y position, and the reason is stated rather
than hidden: a PDF image has a bounding box and no place in the reading
order, so "where it stands" is the page. Inserting by y would reorder the
page's own lines against the geometry `pdf_outline` checks itself against,
and would put a pointer inside a sentence. A caption printed above a figure
therefore keeps its own line where the document put it, and the pointer
follows the page it was drawn on.
"""
blocks = [render_block(image) for image in page.images]
blocks += [
render_missing(rejection.name, reason=rejection.reason) for rejection in page.rejected
]
if not blocks:
return page.text
joined = "\n\n".join(blocks)
return f"{page.text}\n\n{joined}" if page.text else joined
#: The bits-per-component this encoder expresses. A PDF may store 1, 2, 4, 8 or
#: 16, and everything but 8 is REFUSED with a code rather than rescaled --
#: rescaling a 1-bit stencil to 8 bits is a decision about what black means, and
#: a wrong one looks exactly like a right one.
_PDF_SAMPLE_BITS = 8
def _pdf_colour(space: object) -> tuple[int, bytes | None] | None:
"""A PDF colour space as `(channels, palette)`, or `None` if not expressible.
`None` is the honest answer for CMYK, for a separation space and for
anything with a transfer function: converting those needs a colour model
this package does not carry, and a guess would be a picture that is
plausibly the wrong colour. It is counted and stated, never approximated.
"""
from pdfminer.pdftypes import PDFStream, resolve1
space = resolve1(space)
name = getattr(space, "name", None)
if name in ("DeviceGray", "CalGray", "G"):
return 1, None
if name in ("DeviceRGB", "CalRGB", "RGB"):
return 3, None
if not isinstance(space, list) or not space:
return None
head = getattr(resolve1(space[0]), "name", None)
if head == "ICCBased" and len(space) > 1:
profile = resolve1(space[1])
components = resolve1(profile.attrs.get("N")) if isinstance(profile, PDFStream) else None
return (int(components), None) if components in (1, 3) else None
if head in ("CalGray",):
return 1, None
if head in ("CalRGB", "Lab"):
return 3, None
if head in ("Indexed", "I") and len(space) >= 4:
base = _pdf_colour(space[1])
if base is None:
return None
lookup = resolve1(space[3])
if isinstance(lookup, PDFStream):
lookup = lookup.get_data()
if not isinstance(lookup, bytes):
return None
if base[0] == 3:
palette = lookup[: (len(lookup) // 3) * 3]
else:
# PNG's PLTE is RGB triples only, so a grey palette is widened
# rather than refused. Widening a grey to r=g=b is exact, not an
# approximation -- which is why this branch exists and the CMYK one
# does not.
palette = b"".join(bytes([value, value, value]) for value in lookup)
return (1, palette) if palette else None
return None
def _pdf_alpha(attrs: dict[str, object], width: int, height: int) -> bytes | None | bool:
"""A soft mask as one alpha byte per pixel, `None` for none, `False` to refuse.
An `SMask` this encoder cannot express is a REFUSAL rather than a dropped
channel: an image whose transparency is thrown away is composited against
nothing and reads as a black or white rectangle over the page, which is a
picture that is wrong in a way no consumer can detect.
"""
from pdfminer.pdftypes import PDFStream, resolve1
mask = resolve1(attrs.get("SMask"))
if mask is None:
return None
if not isinstance(mask, PDFStream):
return False
shape = mask.attrs
if (
resolve1(shape.get("Width")) != width
or resolve1(shape.get("Height")) != height
or resolve1(shape.get("BitsPerComponent")) != _PDF_SAMPLE_BITS
):
return False
try:
alpha = mask.get_data()
except Exception:
return False
return alpha if len(alpha) >= width * height else False
def _pdf_image(stream: object, name: str) -> ExtractedImage:
"""One image XObject, carried verbatim where it already is a file.
TWO ROUTES, and which one runs is decided by the BYTES rather than by the
filter name. `get_data()` applies every filter pdfminer knows and stops at
the image codecs, so a `DCTDecode` stream comes back as a finished JPEG and
a `FlateDecode` one comes back as raw samples. Sniffing the result is what
makes the first route exact: an embedded JPEG is written to the bundle as
the publisher's own bytes, unre-encoded, and its content-addressed name is
therefore stable for as long as the document is.
Measured on R761 (2026-09-16): 29 of 50 image objects are `DCTDecode` and
take the verbatim route; 21 are `FlateDecode` and are encoded here. Over
the 33-document K2 reference corpus the population is 4 828 objects, and
the filters are mixed enough (`FlateDecode`, `DCTDecode`, `JPXDecode`,
`ASCII85Decode` chains, `CCITTFaxDecode`) that guessing from the filter
name would have been wrong on several hundred.
RENDERING THE PAGE REGION WAS THE ALTERNATIVE AND IT WAS NOT TAKEN. A
rasterised crop would be one code path and would handle every filter, but
its bytes -- and therefore the asset's name and the bundle's digest --
would depend on the version of the rasteriser installed, which is the one
property `OCR_DPI`'s docstring already admits OCR text cannot have. An
embedded stream has no such dependency.
"""
from pdfminer.pdftypes import resolve1
try:
data = stream.get_data() # type: ignore[attr-defined]
except Exception as exc:
raise ExtractionError(
f"the PDF image stream behind {name!r} could not be decoded: {exc}",
code="asset_pdf_unsupported",
) from exc
if data and sniff(data) is not None:
return read_image(data, name=name)
attrs = dict(getattr(stream, "attrs", {}))
width = resolve1(attrs.get("Width"))
height = resolve1(attrs.get("Height"))
bits = resolve1(attrs.get("BitsPerComponent"))
if not isinstance(width, int) or not isinstance(height, int):
raise ExtractionError(
f"the PDF image {name!r} declares no usable size",
code="asset_pdf_unsupported",
)
if resolve1(attrs.get("ImageMask")):
raise ExtractionError(
f"the PDF image {name!r} is a stencil mask, which paints the current "
"fill colour rather than carrying one of its own",
code="asset_pdf_unsupported",
)
if bits != _PDF_SAMPLE_BITS:
raise ExtractionError(
f"the PDF image {name!r} stores {bits}-bit samples; this encoder writes "
f"{_PDF_SAMPLE_BITS}-bit ones and will not rescale, because rescaling a "
"stencil is a decision about what black means",
code="asset_pdf_unsupported",
)
if attrs.get("Decode") is not None:
raise ExtractionError(
f"the PDF image {name!r} carries a Decode array, which remaps every "
"sample; carrying it unmapped would invert the picture",
code="asset_pdf_unsupported",
)
colour = _pdf_colour(attrs.get("ColorSpace"))
if colour is None:
raise ExtractionError(
f"the PDF image {name!r} uses a colour space this encoder does not "
f"express ({attrs.get('ColorSpace')!r})",
code="asset_pdf_unsupported",
)
alpha = _pdf_alpha(attrs, width, height)
if alpha is False:
raise ExtractionError(
f"the PDF image {name!r} has a soft mask this encoder cannot express; "
"dropping transparency would composite the picture against nothing",
code="asset_pdf_unsupported",
)
channels, palette = colour
encoded = encode_png(
width,
height,
data,
channels=channels,
palette=palette,
alpha=alpha if isinstance(alpha, bytes) else None,
)
return read_image(encoded, name=name)
def _pdf_images(page: object) -> tuple[tuple[ExtractedImage, ...], tuple[AssetRejection, ...]]:
"""Every image drawn on one page, with the failures kept beside them."""
carried: list[ExtractedImage] = []
rejected: list[AssetRejection] = []
number = getattr(page, "page_number", 0)
for index, drawn in enumerate(getattr(page, "images", []) or [], start=1):
# The name a PDF image does NOT have. An XObject is reached through a
# resource name local to one page's dictionary, so it is not an
# identifier -- the page number in front of it is what makes the string
# readable, and the content-addressed digest is what makes it unique.
label = str(drawn.get("name") or index).lstrip("/")
name = f"page-{number}-{label}"
stream = drawn.get("stream")
if stream is None:
rejected.append(
AssetRejection(name, "asset_pdf_unsupported", "the image object has no stream")
)
continue
try:
carried.append(_pdf_image(stream, name))
except ExtractionError as exc:
rejected.append(AssetRejection(name, exc.code, str(exc)))
return tuple(carried), tuple(rejected)
@functools.lru_cache(maxsize=1)
def _pdf_pages(
data: bytes, headings: bool = False, ocr: bool = False
) -> tuple[tuple[int, str], ...]:
"""Every page that produced text, as `(page number, text)`, in page order.
data: bytes, headings: bool = False, ocr: bool = False, assets: bool = False
) -> tuple[_PdfPage, ...]:
"""Every page that produced content, as a `_PdfPage`, in page order.
The page NUMBER is 1-based and comes from the document, so a page that
yielded nothing removes itself from the sequence without renumbering the
@ -991,6 +1481,13 @@ def _pdf_pages(
from its lines. Measured, the two agree on 11 of 11 pages of a real tender
PDF -- but "agree on the document I tried" is not a contract, so the
default does not depend on it.
`assets` is the third, and with it off not one line below it runs: no
stream is decoded, no sample buffer is allocated, and the emitted pages are
the objects they always were. A page that produced no TEXT is still
dropped even when it carries an image, because `_extract_pdf` refuses a
document with no text at all (`extractor_empty_pdf`) and an image-only
document is `--ocr`'s question, not this one's.
"""
try:
import pdfplumber
@ -1005,8 +1502,12 @@ def _pdf_pages(
# cannot supply it. A title page is 100 % heading, and a per-page
# median would compare it with itself and mark nothing.
recovered: list[str | list[tuple[str, float, str]]] = []
numbers: list[int] = []
drawn: list[tuple[tuple[ExtractedImage, ...], tuple[AssetRejection, ...]]] = []
for page in pdf.pages:
flat = (page.extract_text() or "").rstrip()
numbers.append(page.page_number)
drawn.append(_pdf_images(page) if assets else ((), ()))
if read is not None and needs_ocr(flat):
# The page's own text is unusable, so it is replaced
# WHOLESALE rather than merged with: a page of `(cid:N)`
@ -1047,7 +1548,11 @@ def _pdf_pages(
raise ExtractionError(
f"the PDF parser failed on this file: {exc}", code="extractor_pdf_error"
) from exc
return tuple((number, page) for number, page in enumerate(pages, start=1) if page)
return tuple(
_PdfPage(number=number, text=text, images=images, rejected=rejected)
for number, text, (images, rejected) in zip(numbers, pages, drawn)
if text
)
@dataclass(frozen=True)
@ -1138,7 +1643,12 @@ def _outline_page_and_top(doc: object, dest: object, action: object) -> tuple[ob
def pdf_outline(
name: str, data: bytes, *, pdf_headings: bool = False, ocr: bool = False
name: str,
data: bytes,
*,
pdf_headings: bool = False,
ocr: bool = False,
assets: bool = False,
) -> PdfOutline:
"""`pdf`: the file's own `/Outlines` tree, as marks on the extracted text.
@ -1177,14 +1687,21 @@ def pdf_outline(
from pdfminer.pdfdocument import PDFNoOutlines
from pdfminer.pdfpage import PDFPage
rendered = _pdf_pages(data, pdf_headings, ocr)
rendered = _pdf_pages(data, pdf_headings, ocr, assets)
starts: dict[int, int] = {}
page_lines: dict[int, list[str]] = {}
# The page's OWN lines, without the pointer blocks appended after them.
# `starts` has to count the appended lines (they are in the text a mark
# indexes) while the geometry check must not see them, because
# `extract_text_lines` reports the page and knows nothing about what this
# package added underneath it.
body_lines: dict[int, list[str]] = {}
offset = 0
for number, page_text in rendered:
starts[number] = offset
page_lines[number] = page_text.split("\n")
offset += len(page_lines[number]) + 1
for rendered_page in rendered:
starts[rendered_page.number] = offset
page_lines[rendered_page.number] = _pdf_page_text(rendered_page).split("\n")
body_lines[rendered_page.number] = rendered_page.text.split("\n")
offset += len(page_lines[rendered_page.number]) + 1
unresolved = 0
collided = 0
@ -1225,7 +1742,7 @@ def pdf_outline(
lines = page_lines[number]
tops: list[float] | None = None
geometry = page.extract_text_lines()
if [str(entry["text"]) for entry in geometry] == lines:
if [str(entry["text"]) for entry in geometry] == body_lines[number]:
tops = [float(entry["top"]) for entry in geometry]
height = float(page.height)
for level, title, top in group:
@ -1268,7 +1785,9 @@ def _normalise_outline(value: str) -> str:
return re.sub(r"\s+", "", value).lower()
def _extract_pdf(data: bytes, *, headings: bool = False, ocr: bool = False) -> str:
def _extract_pdf(
data: bytes, *, headings: bool = False, ocr: bool = False, assets: bool = False
) -> str:
"""`pdf`: page text via `pdfplumber`, in page order, pages separated by a
blank line.
@ -1284,8 +1803,8 @@ def _extract_pdf(data: bytes, *, headings: bool = False, ocr: bool = False) -> s
and pymupdf each emit all labels then all values. Re-pairing those is
guesswork, and in a requirements document a wrong pairing looks right.
"""
pages = _pdf_pages(data, headings, ocr)
text = _PDF_PAGE_SEPARATOR.join(page for _, page in pages)
pages = _pdf_pages(data, headings, ocr, assets)
text = _PDF_PAGE_SEPARATOR.join(_pdf_page_text(page) for page in pages)
if not text:
raise ExtractionError(
"the PDF yielded no text on any page; a scanned or image-only "
@ -1332,7 +1851,94 @@ def _convert_bytes(source: bytes, to: str, format: str, extra_args: Sequence[str
)
def _extract_office(suffix: str, data: bytes) -> str:
#: A markdown image as the converter's own writer emits it. Measured against
#: pandoc 3.10.2 on hand-laid fixtures: a `.docx` picture arrives as
#: `![<descr>](<path>){width="..." height="..."}` and a `.pptx` one as
#: `![<descr>](<path> "<name>")`, so the title form and the attribute form are
#: both real and a regex written for one of them silently leaves the other's
#: link in the text.
_MEDIA_LINK = re.compile(
r"!\[(?P<alt>[^\]]*)\]\("
r"(?:<(?P<angle>[^>]*)>|(?P<plain>[^)\s]*))"
r'(?:\s+"(?P<title>[^"]*)")?\)'
r"(?P<attrs>\{[^}]*\})?"
)
def _convert_with_media(
source: bytes, to: str, format: str, extra_args: Sequence[str]
) -> tuple[str, dict[str, bytes]]:
"""The converter call again, with `--extract-media` and the files read back.
A separate function rather than a flag on `_convert_bytes` because the
media must be READ INSIDE the temporary directory's lifetime: the directory
is removed on every path, and a caller handed a rewritten markdown string
pointing into it would hold links to files that no longer exist. Returning
the bytes is what makes the seam closed.
The staging path is absolute, so the converter writes absolute links. That
is deliberate: every one of them is replaced below, and a link that somehow
survived would carry a temporary directory name into a concept -- a string
that differs on every run, which a byte-determinism rule would catch loudly
rather than never.
"""
import pypandoc
from ._pandoc import converter_path
with tempfile.TemporaryDirectory() as staging:
staged = Path(staging) / f"input.{format}"
staged.write_bytes(source)
media_root = Path(staging) / "extracted"
with converter_path():
text = str(
pypandoc.convert_file(
str(staged),
to,
format=format,
extra_args=[*extra_args, f"--extract-media={media_root}"],
)
)
media: dict[str, bytes] = {}
if media_root.is_dir():
for path in sorted(media_root.rglob("*")):
if path.is_file():
media[str(path)] = path.read_bytes()
return text, media
def _rewrite_media_links(text: str, media: dict[str, bytes], collector: _AssetCollector) -> str:
"""Every converter image link, replaced by this package's own pointer block.
UNCONDITIONAL, including the links that cannot be resolved. The converter
already emitted a markdown image before this existed -- measured on a
hand-laid `.docx`, today's output carries
`![Tabell 84-2](media/tabell-84-2.png)` with no such file anywhere, which
`structure._scan_references` reads as a cross-reference to a concept that
cannot exist. Leaving an unresolvable link in place would keep that defect
and add a temporary directory name to it.
"""
def replace(match: re.Match[str]) -> str:
target = match.group("angle") or match.group("plain") or ""
label = match.group("alt") or match.group("title") or None
data = media.get(target)
if data is not None:
# The name the CONTAINER gave it, not the staging path: pandoc
# preserves the part name under its own media directory, so
# `word/media/tabell-84-2.png` arrives as `media/tabell-84-2.png`.
inside = target.split("/extracted/", 1)[-1]
return collector.carry(data, name=inside, label=label)
if not target:
return collector.reject(
"image", code="asset_unresolved", reason="the converter emitted no target"
)
return collector.local(target, label=label)
return _MEDIA_LINK.sub(replace, text)
def _extract_office(suffix: str, data: bytes, collector: _AssetCollector | None = None) -> str:
"""The five office rows, converted through the vendored binary.
Shaped after `_extract_pdf`: the gate is an import probe rather than a
@ -1349,7 +1955,11 @@ def _extract_office(suffix: str, data: bytes) -> str:
writer = _SPREADSHEET_WRITER if spreadsheet else _PANDOC_WRITER
args = _SPREADSHEET_ARGS if spreadsheet else _PANDOC_ARGS
try:
text = _convert_bytes(data, writer, _PANDOC_FORMATS[suffix], args)
if collector is None:
text = _convert_bytes(data, writer, _PANDOC_FORMATS[suffix], args)
else:
text, media = _convert_with_media(data, writer, _PANDOC_FORMATS[suffix], args)
text = _rewrite_media_links(text, media, collector)
except ExtractionError:
raise
except Exception as exc:
@ -1594,14 +2204,17 @@ def _line_units(text: str) -> SourceUnits:
return SourceUnits("lines", tuple(starts), tuple(range(1, len(starts) + 1)))
def _pdf_units(data: bytes, headings: bool, ocr: bool) -> SourceUnits:
def _pdf_units(data: bytes, headings: bool, ocr: bool, assets: bool = False) -> SourceUnits:
starts: list[int] = []
numbers: list[int] = []
offset = 0
for number, page in _pdf_pages(data, headings, ocr):
for page in _pdf_pages(data, headings, ocr, assets):
starts.append(offset)
numbers.append(number)
offset += len(page) + len(_PDF_PAGE_SEPARATOR)
numbers.append(page.number)
# The page as it reaches the text, pointers included: a locator built
# from the body alone would drift by two lines per carried image and
# would name the wrong page from the first one onwards.
offset += len(_pdf_page_text(page)) + len(_PDF_PAGE_SEPARATOR)
return SourceUnits("pages", tuple(starts), tuple(numbers))
@ -1646,7 +2259,13 @@ def _spreadsheet_units(text: str) -> SourceUnits | None:
def source_units(
filename: str, data: bytes, text: str, *, pdf_headings: bool = False, ocr: bool = False
filename: str,
data: bytes,
text: str,
*,
pdf_headings: bool = False,
ocr: bool = False,
assets: bool = False,
) -> SourceUnits | None:
"""The unit table for one dropped file, or `None` when it has none.
@ -1660,7 +2279,7 @@ def source_units(
"""
suffix = Path(filename).suffix.lower()
if suffix == ".pdf":
return _pdf_units(data, pdf_headings, ocr)
return _pdf_units(data, pdf_headings, ocr, assets)
if suffix == ".xlsx":
return _spreadsheet_units(text)
if suffix in _CORE_EXTRACTORS or suffix in _PANDOC_FORMATS:
@ -1675,6 +2294,8 @@ def extract_text(
renderer: Callable[[str], str] | None = None,
pdf_headings: bool = False,
ocr: bool = False,
assets: bool = False,
resolve: Resolver | None = None,
) -> str:
"""Convert one dropped file's bytes to OKF concept text, dispatched by type.
@ -1704,17 +2325,87 @@ def extract_text(
passing either argument gets today's behaviour, silently, which is correct
-- the options describe a reader, not a policy for the run.
"""
return extract_document(
filename,
data,
renderer=renderer,
pdf_headings=pdf_headings,
ocr=ocr,
assets=assets,
resolve=resolve,
).text
#: The types that can carry an image, and the reader that places it. Kept apart
#: from `_CORE_EXTRACTORS` and `_OPTIONAL_EXTRACTORS` rather than folded into
#: them, and that separation is the byte-identity guarantee: with `assets=False`
#: not one entry below is consulted and the dispatch is the one every golden,
#: every pinned bundle and every published digest was measured on. `.csv`,
#: `.json`, `.md` and `.txt` are absent because the formats carry no image;
#: `.xlsx` is absent because its converter writes one pipe table per sheet and
#: a two-line block inside one would break the row locator `_spreadsheet_units`
#: reads back out of it -- measured 2026-09-16, 0 of 4 K2 workbooks hold any
#: media at all, so the row is a limit stated rather than a loss taken.
_ASSET_READERS: dict[str, Callable[[bytes, _AssetCollector], str]] = {
".html": _extract_html,
".htm": _extract_html,
".xml": _extract_xml,
**{
suffix: functools.partial(_extract_office, suffix)
for suffix in _PANDOC_FORMATS
if suffix != ".xlsx"
},
}
def extract_document(
filename: str,
data: bytes,
*,
renderer: Callable[[str], str] | None = None,
pdf_headings: bool = False,
ocr: bool = False,
assets: bool = False,
resolve: Resolver | None = None,
) -> ExtractedDocument:
"""One dropped file as text PLUS the images that stand inside that text.
The entry point :func:`extract_text` keeps for the eight callers that want
a string, and the one Door B uses since 0.10.0. With `assets=False` -- the
default, everywhere -- this runs exactly the dispatch that existed before
the asset layer did, and returns an :class:`ExtractedDocument` whose text is
byte-identical and whose two image tuples are empty.
`resolve` answers for the formats that POINT at a file instead of embedding
it (`html`, `xml`). Without one every pointer resolves to nothing and is
stated as such; with one, containment is that resolver's rule and not this
module's. `pdf` and the office rows embed their images and never consult it.
"""
suffix = Path(filename).suffix.lower()
extractor = _CORE_EXTRACTORS.get(suffix) or _OPTIONAL_EXTRACTORS.get(suffix)
if extractor is not None:
if suffix == ".pdf" and (pdf_headings or ocr):
text = _extract_pdf(data, headings=pdf_headings, ocr=ocr)
if extractor is None:
if suffix in _UNPARSED_OPTIONAL_EXTENSIONS:
raise _extra_missing(suffix)
raise ExtractionError(
f"no extractor is registered for file extension {suffix!r} ({filename!r})",
code="extractor_unknown",
)
collector = _AssetCollector(resolve) if assets else None
if suffix == ".pdf":
if pdf_headings or ocr or assets:
text = _extract_pdf(data, headings=pdf_headings, ocr=ocr, assets=assets)
else:
text = extractor(data)
return renderer(text) if renderer is not None else text
if suffix in _UNPARSED_OPTIONAL_EXTENSIONS:
raise _extra_missing(suffix)
raise ExtractionError(
f"no extractor is registered for file extension {suffix!r} ({filename!r})",
code="extractor_unknown",
if collector is not None:
for page in _pdf_pages(data, pdf_headings, ocr, True):
collector.images.extend(page.images)
collector.rejected.extend(page.rejected)
elif collector is not None and suffix in _ASSET_READERS:
text = _ASSET_READERS[suffix](data, collector)
else:
text = extractor(data)
return ExtractedDocument(
text=renderer(text) if renderer is not None else text,
images=tuple(collector.images) if collector is not None else (),
rejected=tuple(collector.rejected) if collector is not None else (),
)