feat(assets): a bundle carries the images its sources declare (0.10.0)
Until now no reader in this package fetched, named, described or copied a single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>` was walked past, a PDF was opened for its text alone, the converter's markdown writer dropped every picture, and the only writer into a bundle took `content: str`. The two lossiness warnings said so on every run, which made the loss honest and did not make it smaller. Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N` and 9 `Figur N-N` captions stand over nothing, because that publisher ships those tables as raster pictures in both. Process 84's "toleranseklasse ... er gitt i tabell 84-2" points at empty space. THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its denominator out of the source (`page.images`, `word/media/`, `ppt/media/`, `<img`, `<graphic`), never from a constant here. Measured at332961a, built from `git archive` and not from the editable tree: carried 0 of 8 local images across 5 documents (9 declared), and no `assets/` at all. After: 8 of 8, with the ninth a remote source carried as a pointer without a file. FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is (sniffed from the bytes, never from the claimed extension), what it is called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed at (one two-line block, one regex). `.xlsx` is deliberately not a row: a block inside its pipe tables would break the `source_rows` locator, and 0 of 4 K2 workbooks hold media. A PDF stream that is already a file is carried VERBATIM (29 of R761's 50 objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so no new dependency. Rendering the page region was the alternative and was felled on determinism: a rasterised crop's bytes, and therefore the asset's content-addressed name and the bundle's digest, would depend on the installed rasteriser. What the encoder cannot express exactly is refused with a code and counted, never approximated. NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of the K2 corpus the size distribution is a broad spread with no gap, unlike OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose. ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document reference corpus at332961aversus rebuilt at HEAD with `--no-assets`: 865 files on both sides, `diff -rq` reports ONE difference, the added `Images: NOT CARRIED` line in log.md. Every concept byte-identical. Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets (2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s -> 3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new concept has a measured cause: the pointers are body text, so a section holding 146 of that document's images grew from 19.0 % to 30.6 % of the extracted text and crossed `--outline-gate`'s 0.20 share clause. THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block passes the gate as body text, the picture beside it passes nothing, and log.md says so on every run. Also fixed, both found by measuring rather than by reading: - a markdown image is no longer read as a cross-reference. `structure._LINK` never looked at the character in front of the bracket, so every pointer would have arrived in the index as an edge to a concept that cannot exist. - Door C carries the assets its merged concepts point at. Before this, importing a bundle built with `--assets` merged 6 of 6 concepts and wrote no `assets/` at all, so every pointer named a missing file. Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md Spec proposal: docs/plan/okf-assets-section-6-4.md Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
332961a19c
commit
bc39e8091f
33 changed files with 3638 additions and 64 deletions
|
|
@ -76,7 +76,7 @@ from .manifest import (
|
|||
)
|
||||
from .materialize import IngestResult, materialize_bundle
|
||||
|
||||
__version__ = "0.9.0"
|
||||
__version__ = "0.10.0"
|
||||
|
||||
__all__ = [
|
||||
"BlockedFile",
|
||||
|
|
|
|||
482
src/llm_ingestion_okf/assets.py
Normal file
482
src/llm_ingestion_okf/assets.py
Normal file
|
|
@ -0,0 +1,482 @@
|
|||
"""Binary assets: what an image IS, what it is called, and how it is pointed at.
|
||||
|
||||
Until 0.10.0 this package had no image path at all. Every reader recovered
|
||||
text, every warning said so, and the only writer into a bundle was
|
||||
:func:`materialize.write_bytes`, whose signature is ``(bundle_dir, name,
|
||||
content: str)`` -- UTF-8 in, text out, no binary route anywhere. A document
|
||||
whose table is a raster picture therefore reached a concept as an absence with
|
||||
no denominator. Measured on R761 Prosesskoden:2025, the publisher's own
|
||||
NISO-STS delivery: the process text is carried in full, and 12 ``Tabell N-N``
|
||||
and 9 ``Figur N-N`` captions stand over nothing, so process 84's
|
||||
"toleranseklasse ... er gitt i tabell 84-2" points at empty space.
|
||||
|
||||
THIS MODULE IS THE ONE PLACE THAT DECIDES WHAT AN IMAGE IS. A reader hands it
|
||||
bytes and a name; it returns a carried image or raises a coded rejection. That
|
||||
is what makes "N carried of M found" mean the same thing for ``pdf``, ``docx``,
|
||||
``html`` and ``xml``, and it is what keeps each format's quirks out of the
|
||||
bundle layout.
|
||||
|
||||
THREE RULES, and each one exists because the alternative is a silent lie:
|
||||
|
||||
- **The type is SNIFFED, never claimed.** Measured on the R761 delivery, the
|
||||
graphics directory holds ``.bmp``, ``.jpg`` and ``.png`` side by side and the
|
||||
document's ``xlink:href`` values are whatever the publisher's tool wrote. A
|
||||
name is a claim; the magic bytes are the fact. A consumer dispatching on the
|
||||
extension of a name that lies reads the file wrong with full confidence.
|
||||
- **The name is CONTENT-ADDRESSED** -- ``<sha256[:12]>-<reduced original>``.
|
||||
Two drops of one image are one file, a rebuild of one corpus is one bundle,
|
||||
and the digest carries the uniqueness so the readable tail can be shortened
|
||||
without any risk of collision. That is the byte-determinism rule this package
|
||||
already holds for text, extended to the bytes beside it.
|
||||
- **The pointer is ONE GRAMMAR**, owned here. ``okf describe`` (step 2) has to
|
||||
find every pointer mechanically in order to write a transcription under it,
|
||||
so the block is a regex this module ships beside the writer rather than a
|
||||
shape each reader invents and each consumer re-derives.
|
||||
|
||||
WHAT THIS MODULE DOES NOT DO: it never looks at a picture. Classifying an image
|
||||
as a table or a figure, and reading what it says, is a model call, and the
|
||||
invariant "no model calls anywhere in the run path" is not negotiated here --
|
||||
step 2 is a separate command, outside the build path, and this module is
|
||||
importable without it.
|
||||
|
||||
**The image BYTES are not screened.** The guard is text-only (its own boundary,
|
||||
not ours), so what passes a persist gate is the pointer block, as body text,
|
||||
like every other line. The bytes of a carried image are written to the bundle
|
||||
unscreened. Stated here rather than implied, because a consumer weighing an
|
||||
untrusted drop needs to know which half of the concept was looked at.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import re
|
||||
import struct
|
||||
import unicodedata
|
||||
import zlib
|
||||
from dataclasses import dataclass
|
||||
|
||||
from .errors import ExtractionError
|
||||
|
||||
#: The one directory a bundle keeps its binary assets in, at the bundle root.
|
||||
#: Fixed rather than configurable: a consumer resolving `/assets/...` out of a
|
||||
#: concept has only the bundle, and a per-profile directory would make that
|
||||
#: pointer unresolvable without also shipping the profile that wrote it.
|
||||
ASSETS_DIR = "assets"
|
||||
|
||||
#: How much of the digest names the file. 12 hex characters is 48 bits; over
|
||||
#: the largest asset population measured here (4 828 image objects in one
|
||||
#: 33-document corpus) the birthday probability of a collision is about
|
||||
#: 4e-11. A collision would be caught anyway -- an occupied name is re-used
|
||||
#: only when the bytes there are already identical, the same content-identity
|
||||
#: rule Door C proves ownership with.
|
||||
DIGEST_PREFIX = 12
|
||||
|
||||
#: How much of the original name survives into the asset name. The digest
|
||||
#: carries uniqueness, so this is decoration and truncating it is safe -- which
|
||||
#: is the opposite of `materialize.check_filename_length`'s situation, where the
|
||||
#: name IS the identity and truncation would silently merge two documents.
|
||||
NAME_TAIL_MAX = 60
|
||||
|
||||
#: Magic bytes -> (media type, suffix). Sniffed in this order; the first match
|
||||
#: wins, and nothing here overlaps.
|
||||
_MAGIC: tuple[tuple[bytes, str, str], ...] = (
|
||||
(b"\x89PNG\r\n\x1a\n", "image/png", ".png"),
|
||||
(b"\xff\xd8\xff", "image/jpeg", ".jpg"),
|
||||
(b"GIF87a", "image/gif", ".gif"),
|
||||
(b"GIF89a", "image/gif", ".gif"),
|
||||
(b"BM", "image/bmp", ".bmp"),
|
||||
(b"II\x2a\x00", "image/tiff", ".tiff"),
|
||||
(b"MM\x00\x2a", "image/tiff", ".tiff"),
|
||||
)
|
||||
|
||||
#: JPEG 2000, in both the forms a PDF `JPXDecode` stream hands back: the JP2
|
||||
#: container and a bare codestream.
|
||||
_JP2_SIGNATURE = b"\x00\x00\x00\x0cjP \r\n\x87\n"
|
||||
_J2K_SIGNATURE = b"\xff\x4f\xff\x51"
|
||||
|
||||
#: The frame markers that carry a JPEG's dimensions. Every SOF except the four
|
||||
#: that are not frame headers at all (`DHT` 0xC4, `JPG` 0xC8, `DAC` 0xCC).
|
||||
_JPEG_SOF = frozenset(range(0xC0, 0xD0)) - {0xC4, 0xC8, 0xCC}
|
||||
|
||||
#: `materialize.reduce_to_id_grammar`'s rule, restated. Not imported: this
|
||||
#: module is reached from `extract.py`, whose registry must not import the
|
||||
#: contract layer, and `materialize` pulls in `manifest` and `profiles`.
|
||||
#: `tests/test_assets.py` holds the two forms equal on the same inputs, so the
|
||||
#: restatement cannot drift into a second grammar.
|
||||
_SEPARATOR_RUN = re.compile(r"[^a-z0-9]+")
|
||||
|
||||
#: One pointer block, as this module writes it. Group 1 is the label, group 2
|
||||
#: is the asset file name, group 3 is the whole second line. `okf describe`
|
||||
#: finds its work with this and writes under the match; a consumer wanting to
|
||||
#: strip pointers uses the same expression, so there is one definition of what
|
||||
#: a pointer looks like rather than one per reader.
|
||||
IMAGE_POINTER = re.compile(
|
||||
r"^!\[(?P<label>[^\]\n]*)\]\(/" + ASSETS_DIR + r"/(?P<asset>[^)\s]+)\)\n"
|
||||
r"(?P<detail>Image: [^\n]*)$",
|
||||
re.MULTILINE,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExtractedImage:
|
||||
"""One image a document carries, as this package will write it.
|
||||
|
||||
`name` is what the SOURCE calls the file -- a `xlink:href`, an `<img src>`,
|
||||
a media entry inside an OOXML container, or a synthesised name for a PDF
|
||||
image object, which has none. It is carried verbatim into the pointer's
|
||||
second line and reduced (never trusted) into the asset's own name.
|
||||
|
||||
`label` is a caption or alt text where the format HAS one, and `None`
|
||||
where it does not. Two of the four readers have none: a PDF image object
|
||||
and an STS `<graphic>` carry no caption element, and the caption a human
|
||||
sees is running text the extractor already emitted on its own line.
|
||||
Deriving a label from the nearest line would be an unmarked heuristic,
|
||||
which this package treats as worse than no heuristic at all.
|
||||
"""
|
||||
|
||||
data: bytes
|
||||
name: str
|
||||
media_type: str
|
||||
suffix: str
|
||||
width: int | None
|
||||
height: int | None
|
||||
label: str | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AssetRejection:
|
||||
"""An image that was FOUND and not carried, with the reason as a code.
|
||||
|
||||
A rejection is a row in the run log, never a failed document: one
|
||||
unreadable picture must not cost the 3 000 concepts of text around it. The
|
||||
denominator is what makes the log worth reading -- "51 carried" says
|
||||
nothing without "of 53 found".
|
||||
"""
|
||||
|
||||
name: str
|
||||
code: str
|
||||
reason: str
|
||||
|
||||
|
||||
def sniff(data: bytes) -> tuple[str, str] | None:
|
||||
"""`(media type, suffix)` read from the bytes, or `None` if not an image.
|
||||
|
||||
The claimed extension never participates. A `.jpg` that is really a PNG is
|
||||
carried as a PNG under a `.png` name, because the alternative is a bundle
|
||||
whose file names disagree with their contents.
|
||||
"""
|
||||
if data.startswith(_JP2_SIGNATURE) or data.startswith(_J2K_SIGNATURE):
|
||||
return "image/jp2", ".jp2"
|
||||
for magic, media_type, suffix in _MAGIC:
|
||||
if data.startswith(magic):
|
||||
return media_type, suffix
|
||||
return None
|
||||
|
||||
|
||||
def dimensions(data: bytes) -> tuple[int, int] | None:
|
||||
"""`(width, height)` in pixels from the file's own header, or `None`.
|
||||
|
||||
`None` is a MEASUREMENT: this reader does not read every format's geometry
|
||||
(JPEG 2000 and TIFF are absent), and a concept that printed `0x0 px` would
|
||||
be stating a number nobody took. The pointer says "dimensions unknown"
|
||||
instead.
|
||||
"""
|
||||
kind = sniff(data)
|
||||
if kind is None:
|
||||
return None
|
||||
suffix = kind[1]
|
||||
try:
|
||||
if suffix == ".png":
|
||||
if len(data) < 24 or data[12:16] != b"IHDR":
|
||||
return None
|
||||
width, height = struct.unpack(">II", data[16:24])
|
||||
return (width, height) if width and height else None
|
||||
if suffix == ".jpg":
|
||||
return _jpeg_dimensions(data)
|
||||
if suffix == ".gif":
|
||||
if len(data) < 10:
|
||||
return None
|
||||
width, height = struct.unpack("<HH", data[6:10])
|
||||
return (width, height) if width and height else None
|
||||
if suffix == ".bmp":
|
||||
return _bmp_dimensions(data)
|
||||
except (struct.error, IndexError):
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _jpeg_dimensions(data: bytes) -> tuple[int, int] | None:
|
||||
position = 2
|
||||
end = len(data)
|
||||
while position + 3 < end:
|
||||
if data[position] != 0xFF:
|
||||
position += 1
|
||||
continue
|
||||
marker = data[position + 1]
|
||||
if marker in (0xD8, 0x01) or 0xD0 <= marker <= 0xD7:
|
||||
position += 2
|
||||
continue
|
||||
if marker == 0xFF:
|
||||
position += 1
|
||||
continue
|
||||
length = struct.unpack(">H", data[position + 2 : position + 4])[0]
|
||||
if marker in _JPEG_SOF:
|
||||
if position + 9 > end:
|
||||
return None
|
||||
height, width = struct.unpack(">HH", data[position + 5 : position + 9])
|
||||
return (width, height) if width and height else None
|
||||
position += 2 + length
|
||||
return None
|
||||
|
||||
|
||||
def _bmp_dimensions(data: bytes) -> tuple[int, int] | None:
|
||||
if len(data) < 26:
|
||||
return None
|
||||
header_size = struct.unpack("<I", data[14:18])[0]
|
||||
if header_size == 12:
|
||||
width, height = struct.unpack("<hh", data[18:22])
|
||||
else:
|
||||
width, height = struct.unpack("<ii", data[18:26])
|
||||
# A negative height is a top-down BMP; the magnitude is the pixel count.
|
||||
return (abs(width), abs(height)) if width and height else None
|
||||
|
||||
|
||||
def read_image(data: bytes, *, name: str, label: str | None = None) -> ExtractedImage:
|
||||
"""One image, typed by its bytes, or a coded refusal.
|
||||
|
||||
Raises :class:`ExtractionError` with `asset_type_unknown` when the bytes
|
||||
are not an image this package recognises. The caller records that as an
|
||||
:class:`AssetRejection` and keeps going -- a document is not lost over one
|
||||
picture.
|
||||
"""
|
||||
kind = sniff(data)
|
||||
if kind is None:
|
||||
raise ExtractionError(
|
||||
f"the bytes behind {name!r} are not an image format this package "
|
||||
f"recognises (first bytes {data[:8]!r})",
|
||||
code="asset_type_unknown",
|
||||
)
|
||||
media_type, suffix = kind
|
||||
size = dimensions(data)
|
||||
return ExtractedImage(
|
||||
data=data,
|
||||
name=name,
|
||||
media_type=media_type,
|
||||
suffix=suffix,
|
||||
width=size[0] if size else None,
|
||||
height=size[1] if size else None,
|
||||
label=label,
|
||||
)
|
||||
|
||||
|
||||
def digest(data: bytes) -> str:
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def _reduce(text: str) -> str:
|
||||
return _SEPARATOR_RUN.sub("-", unicodedata.normalize("NFC", text).lower()).strip("-")
|
||||
|
||||
|
||||
def asset_name(image: ExtractedImage) -> str:
|
||||
"""`<sha256[:12]>-<reduced original BASENAME><sniffed suffix>`.
|
||||
|
||||
DEDUP IS ON CONTENT, and the digest is what makes it so: the same bytes
|
||||
arriving twice are one file, whichever document dropped them. The residual
|
||||
is stated rather than claimed away -- identical bytes shipped under two
|
||||
different base names are two files in the bundle, both correct and both
|
||||
holding the same picture. Measured on the fixture inbox and on the R761
|
||||
delivery, that case occurs 0 times; a rule that could not produce it at all
|
||||
would have to drop the readable tail entirely, and the tail is what makes
|
||||
`assets/` legible to the person checking a bundle by hand.
|
||||
|
||||
|
||||
The suffix comes from the bytes and the stem from the name, reduced to the
|
||||
same grammar every generated filename in this package uses. A link target
|
||||
sits inside `](...)`, which a space or a closing parenthesis terminates --
|
||||
and the R761 delivery's own hrefs carry both (`25-0143 - Tabeller -
|
||||
Prosesskoden (R761-R762).jpg`), so reducing is what makes the pointer
|
||||
followable rather than merely tidy.
|
||||
"""
|
||||
# The BASENAME, never the path the document reached it through. Measured
|
||||
# on the fixture inbox: one image pointed at as `graphics/figur-84-1.png`
|
||||
# from an HTML document and as `figur-84-1.png` from an STS one was written
|
||||
# twice, under two names, in one run -- with the digest in both announcing
|
||||
# that the bytes were identical. The path is a property of the pointer, not
|
||||
# of the picture, and the full original survives on the pointer's own line.
|
||||
base = image.name.rsplit("/", 1)[-1]
|
||||
stem = _reduce(base.rsplit(".", 1)[0] if "." in base else base)
|
||||
head = digest(image.data)[:DIGEST_PREFIX]
|
||||
if not stem:
|
||||
return f"{head}{image.suffix}"
|
||||
return f"{head}-{stem[:NAME_TAIL_MAX].rstrip('-')}{image.suffix}"
|
||||
|
||||
|
||||
def asset_href(image: ExtractedImage) -> str:
|
||||
"""The bundle-absolute path SPEC SS 6.2 allows.
|
||||
|
||||
Absolute rather than relative because a segmented bundle puts concepts at
|
||||
different depths: `assets/x.png` resolves to two different places from two
|
||||
concepts of one document, and `/assets/x.png` to one place from every
|
||||
concept in the bundle.
|
||||
"""
|
||||
return f"/{ASSETS_DIR}/{asset_name(image)}"
|
||||
|
||||
|
||||
def _inline(value: str) -> str:
|
||||
"""A label, made safe for the one line it is written on.
|
||||
|
||||
`[` and `]` are the link grammar's own delimiters and a newline would open
|
||||
a third line in a two-line block, so both are replaced rather than escaped:
|
||||
Door B refuses a title containing a bracket outright (`inbox_title_invalid`)
|
||||
and this text reaches a title through no route, but the pointer is body text
|
||||
a proposer reads, and a half-open link there is a pointer that resolves
|
||||
nowhere.
|
||||
"""
|
||||
collapsed = " ".join(value.split())
|
||||
return collapsed.replace("[", "(").replace("]", ")")
|
||||
|
||||
|
||||
def render_block(image: ExtractedImage) -> str:
|
||||
"""The two lines that stand where the image stands.
|
||||
|
||||
Line one is markdown, so a reader that renders the concept sees the picture
|
||||
and a reader that does not sees the label. Line two states what the first
|
||||
line cannot: the name the SOURCE gave the file, and the size in pixels --
|
||||
the two facts a person checking the bundle against the original needs, and
|
||||
the two a transcription in step 2 has to be judged against.
|
||||
"""
|
||||
label = image.label or image.name or asset_name(image)
|
||||
size = (
|
||||
f"{image.width}x{image.height} px"
|
||||
if image.width is not None and image.height is not None
|
||||
else "dimensions unknown"
|
||||
)
|
||||
detail = f"Image: {_inline(image.name or asset_name(image))} ({size})"
|
||||
if image.label:
|
||||
detail += f" -- {_inline(image.label)}"
|
||||
return f"})\n{detail}"
|
||||
|
||||
|
||||
def encode_png(
|
||||
width: int,
|
||||
height: int,
|
||||
samples: bytes,
|
||||
*,
|
||||
channels: int,
|
||||
palette: bytes | None = None,
|
||||
alpha: bytes | None = None,
|
||||
) -> bytes:
|
||||
"""8-bit samples as a PNG, using nothing but `zlib`.
|
||||
|
||||
A PDF image is usually not a file: `FlateDecode` hands back raw samples
|
||||
with the colour model in the dictionary beside them, so carrying one at all
|
||||
means encoding it. Doing that with the stdlib rather than with a renderer
|
||||
is what keeps the `pdf` row free of a new dependency AND keeps an asset's
|
||||
bytes -- and therefore its content-addressed name -- independent of which
|
||||
version of a rasteriser happened to be installed. `OCR_DPI`'s docstring
|
||||
states the opposite property for OCR text, and the difference is deliberate.
|
||||
|
||||
`channels` is 1 (grey, or indexed when `palette` is given) or 3 (RGB).
|
||||
`alpha` is one byte per pixel, from a PDF `SMask`; absent means opaque.
|
||||
"""
|
||||
if channels not in (1, 3):
|
||||
raise ExtractionError(
|
||||
f"PNG encoding supports 1 or 3 channels, not {channels}",
|
||||
code="asset_samples_invalid",
|
||||
)
|
||||
if width <= 0 or height <= 0:
|
||||
raise ExtractionError(
|
||||
f"PNG encoding needs positive dimensions, got {width}x{height}",
|
||||
code="asset_samples_invalid",
|
||||
)
|
||||
expected = width * height * channels
|
||||
if len(samples) < expected:
|
||||
raise ExtractionError(
|
||||
f"the sample buffer holds {len(samples)} bytes where {width}x{height} "
|
||||
f"at {channels} channels needs {expected} -- refusing to pad, because a "
|
||||
"short buffer means the image dictionary was read wrong",
|
||||
code="asset_samples_invalid",
|
||||
)
|
||||
if alpha is not None and len(alpha) < width * height:
|
||||
raise ExtractionError(
|
||||
f"the alpha buffer holds {len(alpha)} bytes where {width}x{height} needs "
|
||||
f"{width * height}",
|
||||
code="asset_samples_invalid",
|
||||
)
|
||||
if palette is not None:
|
||||
if channels != 1:
|
||||
raise ExtractionError(
|
||||
"a palette applies to single-channel samples only",
|
||||
code="asset_samples_invalid",
|
||||
)
|
||||
if not palette or len(palette) % 3:
|
||||
raise ExtractionError(
|
||||
f"a palette must be whole RGB triples, got {len(palette)} bytes",
|
||||
code="asset_samples_invalid",
|
||||
)
|
||||
colour_type = 3
|
||||
elif alpha is not None:
|
||||
colour_type = 6 if channels == 3 else 4
|
||||
else:
|
||||
colour_type = 2 if channels == 3 else 0
|
||||
|
||||
rows = bytearray()
|
||||
for row in range(height):
|
||||
# Filter type 0 (None) on every row. A filter would shrink the file and
|
||||
# would make the bytes depend on a heuristic; this encoder's output has
|
||||
# to be reproducible from the samples alone, for as long as the bundle
|
||||
# is quoted by its digest.
|
||||
rows.append(0)
|
||||
start = row * width * channels
|
||||
line = samples[start : start + width * channels]
|
||||
if alpha is None:
|
||||
rows += line
|
||||
else:
|
||||
for pixel in range(width):
|
||||
rows += line[pixel * channels : (pixel + 1) * channels]
|
||||
rows.append(alpha[row * width + pixel])
|
||||
|
||||
def chunk(kind: bytes, payload: bytes) -> bytes:
|
||||
return (
|
||||
len(payload).to_bytes(4, "big")
|
||||
+ kind
|
||||
+ payload
|
||||
+ zlib.crc32(kind + payload).to_bytes(4, "big")
|
||||
)
|
||||
|
||||
ihdr = struct.pack(">IIBBBBB", width, height, 8, colour_type, 0, 0, 0)
|
||||
body = b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", ihdr)
|
||||
if palette is not None:
|
||||
body += chunk(b"PLTE", palette)
|
||||
# Level 9 with no filtering: `zlib` is stdlib and its output for a given
|
||||
# level is stable within a Python build, which is the same guarantee the
|
||||
# rest of this package's byte-determinism rests on.
|
||||
body += chunk(b"IDAT", zlib.compress(bytes(rows), 9))
|
||||
return body + chunk(b"IEND", b"")
|
||||
|
||||
|
||||
def render_missing(
|
||||
name: str,
|
||||
*,
|
||||
reason: str,
|
||||
label: str | None = None,
|
||||
href: str | None = None,
|
||||
) -> str:
|
||||
"""A pointer to an image the bundle does NOT carry, and why.
|
||||
|
||||
An image this package found and could not carry is stated in the concept,
|
||||
not dropped. The reader of the bundle is the person who has to decide
|
||||
whether the missing picture mattered, and they cannot decide about an
|
||||
absence they were never shown -- which is precisely the shape of the defect
|
||||
this whole capability exists to close.
|
||||
|
||||
The href is kept when the source had one and it survives the link grammar,
|
||||
so a remote figure says WHERE it was. Extraction never opens a socket: the
|
||||
network gate is an explicit per-run opt-in and extraction is not on that
|
||||
path, so a remote source is carried as a name and never as bytes.
|
||||
"""
|
||||
shown = _inline(name or "image")
|
||||
detail = f"Image: {shown} (not carried: {_inline(reason)})"
|
||||
if href and not re.search(r"[\s()\[\]]", href):
|
||||
return f"\n{detail}"
|
||||
return detail
|
||||
|
|
@ -75,6 +75,7 @@ from functools import partial
|
|||
from pathlib import Path
|
||||
|
||||
from .corpus import (
|
||||
DEFAULT_ASSETS,
|
||||
GATE_GUARD_TRUSTED_SOURCE,
|
||||
GATE_NAMES,
|
||||
LOG_NAME,
|
||||
|
|
@ -231,6 +232,7 @@ DEFAULT_SHELL_PARENT = False
|
|||
#: reproduces published numbers with it; what changed is that reaching it is an
|
||||
#: act, and `log.md` records which gate screened the bundle either way.
|
||||
DEFAULT_GATE = GATE_GUARD_TRUSTED_SOURCE
|
||||
|
||||
#: Round 9. Clause 1 asked whether a title ENDED in a number, which is a
|
||||
#: question about the number rather than about the title: a drawing's dimension
|
||||
#: chain, a P&ID's schematic labels, a door schedule and a borehole log's
|
||||
|
|
@ -413,6 +415,7 @@ def _propose_plans(
|
|||
ocr: bool = False,
|
||||
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
|
||||
shell_parent: bool = DEFAULT_SHELL_PARENT,
|
||||
assets: bool = False,
|
||||
) -> tuple[int, int, int]:
|
||||
"""Propose a plan per dropped file. Returns (written, nothing, failed).
|
||||
|
||||
|
|
@ -454,6 +457,7 @@ def _propose_plans(
|
|||
ocr=ocr,
|
||||
pdf_outline=pdf_outline,
|
||||
shell_parent=shell_parent,
|
||||
assets=assets,
|
||||
)
|
||||
except ProposerError as exc:
|
||||
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
|
||||
|
|
@ -495,6 +499,7 @@ def build(
|
|||
shell_parent: bool = DEFAULT_SHELL_PARENT,
|
||||
frontmatter: Mapping[str, str] | None = None,
|
||||
gate: str = DEFAULT_GATE,
|
||||
assets: bool = DEFAULT_ASSETS,
|
||||
) -> CorpusReport:
|
||||
"""Folder in, bundle out. The whole command, minus argument parsing.
|
||||
|
||||
|
|
@ -535,6 +540,7 @@ def build(
|
|||
ocr=ocr,
|
||||
concept_frontmatter_values=concept_values,
|
||||
gate=gate,
|
||||
assets=assets,
|
||||
)
|
||||
_write_log(bundle, report, profile=STRUCTURED_V1)
|
||||
return report
|
||||
|
|
@ -576,6 +582,12 @@ def build(
|
|||
ocr=ocr,
|
||||
pdf_outline=pdf_outline,
|
||||
shell_parent=shell_parent,
|
||||
# THE SAME VALUE ON BOTH SIDES, and it cannot be otherwise: a plan
|
||||
# records `text_sha256` of the exact string it was proposed
|
||||
# against, and carrying an image inserts lines into that string.
|
||||
# Proposing with the images off and replaying with them on makes
|
||||
# `assert_plan_applies` refuse every document that has a figure.
|
||||
assets=assets,
|
||||
)
|
||||
print(
|
||||
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
|
||||
|
|
@ -595,6 +607,7 @@ def build(
|
|||
ocr=ocr,
|
||||
concept_frontmatter_values=concept_values,
|
||||
gate=gate,
|
||||
assets=assets,
|
||||
)
|
||||
_write_log(bundle, report, profile=SEGMENTED_OKF_V0_2)
|
||||
return report
|
||||
|
|
@ -897,6 +910,30 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
|||
"pre-2026-09-11 spans byte for byte"
|
||||
),
|
||||
)
|
||||
build_parser.add_argument(
|
||||
"--assets",
|
||||
action="store_true",
|
||||
default=DEFAULT_ASSETS,
|
||||
help=(
|
||||
"carry the images the sources declare into the bundle's assets/ "
|
||||
"directory and point at them from the concept text where they "
|
||||
"stand. ON since 0.10.0. Measured on the R761 process code: 12 "
|
||||
"Tabell N-N and 9 Figur N-N captions stood over nothing, because "
|
||||
"the publisher ships those tables as raster pictures. The image "
|
||||
"BYTES are not screened -- the gate reads text -- and log.md says so"
|
||||
),
|
||||
)
|
||||
build_parser.add_argument(
|
||||
"--no-assets",
|
||||
action="store_false",
|
||||
dest="assets",
|
||||
help=(
|
||||
"the explicit opt-out, reproducing the pre-0.10.0 bytes for a "
|
||||
"consumer pinned to them. log.md then says NOT CARRIED, so a "
|
||||
"bundle nobody looked for figures in cannot be mistaken for a "
|
||||
"bundle of documents that had none"
|
||||
),
|
||||
)
|
||||
build_parser.add_argument(
|
||||
"--gate",
|
||||
choices=GATE_NAMES,
|
||||
|
|
@ -1084,6 +1121,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
pdf_outline=args.pdf_outline,
|
||||
shell_parent=args.shell_parent,
|
||||
gate=args.gate,
|
||||
assets=args.assets,
|
||||
frontmatter=frontmatter_from_flags(args.frontmatter or ()),
|
||||
)
|
||||
except (IngestError, OSError, ValueError) as exc:
|
||||
|
|
|
|||
|
|
@ -40,6 +40,7 @@ from collections.abc import Callable, Mapping
|
|||
from dataclasses import dataclass, replace
|
||||
from pathlib import Path
|
||||
|
||||
from .assets import ASSETS_DIR
|
||||
from .errors import IngestError
|
||||
from .extract import extract_text
|
||||
from .inbox import (
|
||||
|
|
@ -87,6 +88,33 @@ LOG_TITLE = "Corpus run history"
|
|||
#: numbers with it and a caller measuring segmentation alone has a legitimate
|
||||
#: reason to take the gate out of the picture -- but reaching it is now an act,
|
||||
#: and the bundle's own log says so.
|
||||
#: Whether a build carries the images its sources declare (0.10.0).
|
||||
#:
|
||||
#: ON, and the reason is the premise the capability was ordered on: what ends
|
||||
#: up in a bundle after converting N sources has to BE the sources. Measured on
|
||||
#: R761 Prosesskoden:2025 -- 2 726 of 2 726 processes carry their text and 12
|
||||
#: `Tabell N-N` and 9 `Figur N-N` captions stand over nothing, because the
|
||||
#: publisher ships those tables as raster pictures in both the PDF and the
|
||||
#: NISO-STS delivery. A bundle that reads as complete and answers "toleranse-
|
||||
#: klasse ... er gitt i tabell 84-2" with an empty space is the defect; it is
|
||||
#: not a preference about bundle size.
|
||||
#:
|
||||
#: WHAT THIS COSTS IS PUBLISHED RATHER THAN IMPLIED. Measured over the K2
|
||||
#: reference corpus, 2026-09-16: 32 of its 33 PDFs carry image objects and the
|
||||
#: population is 4 828, of which 2 931 are 512x512 pixels or larger, 1 307 are
|
||||
#: smaller and 149 declare no size at all. A drawing set contributes 2 524 on
|
||||
#: its own. So this is a large change for a drawing-heavy corpus and none at
|
||||
#: all for a prose one, and `--no-assets` reproduces the pre-0.10.0 bytes
|
||||
#: exactly -- measured by `diff`, not asserted.
|
||||
#:
|
||||
#: NO SIZE FLOOR, and that is a measurement too. The obvious filter would be
|
||||
#: "ignore anything under N pixels", and the distribution above does not offer
|
||||
#: an N: it is a broad spread with no gap, unlike `OCR_CID_SHARE`'s, which is
|
||||
#: bimodal with nothing between the modes. A threshold read off no gap is a
|
||||
#: number this package chose, and it would silently drop somebody's small
|
||||
#: table. Stated as a limit instead of guessed at.
|
||||
DEFAULT_ASSETS = True
|
||||
|
||||
GATE_NONE = "none"
|
||||
#: `okf build`'s default, set by the operator 2026-09-15 on a measurement:
|
||||
#: over the 453 concept bodies of the pinned default bundle, this tier returns
|
||||
|
|
@ -200,6 +228,14 @@ class CorpusReport:
|
|||
#: stub so every existing construction site stays source-compatible; the
|
||||
#: two commands that build bundles both pass it explicitly.
|
||||
gate: str = GATE_NONE
|
||||
#: THE ASSET DENOMINATOR (0.10.0). `assets_carried` is what reached the
|
||||
#: bundle, `assets_found` is what the readers saw. Both, always: "51
|
||||
#: carried" is a measurement only beside "of 53 found", and a run whose
|
||||
#: figures were all refused must not read like a run over documents that had
|
||||
#: none. `assets` says whether they were looked for at all.
|
||||
assets: bool = False
|
||||
assets_carried: int = 0
|
||||
assets_found: int = 0
|
||||
|
||||
@property
|
||||
def merged(self) -> int:
|
||||
|
|
@ -291,11 +327,37 @@ class CorpusReport:
|
|||
f"* **Gate**: {self.gate} "
|
||||
f"({_GATE_DESCRIPTIONS.get(self.gate, 'unrecognised gate name')}). "
|
||||
"Every persisted byte of this bundle passed it.",
|
||||
self._assets_line(),
|
||||
]
|
||||
if self.unaccounted:
|
||||
lines.append("* **Unaccounted**: " + ", ".join(self.unaccounted) + " — K1b FAILED.")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
def _assets_line(self) -> str:
|
||||
"""What this run did about the images in its sources, stated either way.
|
||||
|
||||
An absent bullet would make "this corpus had no figures" and "this run
|
||||
never looked" the same artifact, which is the trust trap `--gate none`
|
||||
already has its own sentence for. So the OFF case is written out in
|
||||
capitals rather than left to an inference.
|
||||
|
||||
THE IMAGE BYTES ARE NOT SCREENED, and the log is where a consumer finds
|
||||
that out. The guard is text-only, so the pointer block passed the gate
|
||||
as body text and the picture beside it did not pass anything.
|
||||
"""
|
||||
if not self.assets:
|
||||
return (
|
||||
"* **Images**: NOT CARRIED — this run did not look for images, so a "
|
||||
"document whose table is a picture reached the bundle as text with a "
|
||||
"gap in it. Absence of an image here is not evidence the sources had none."
|
||||
)
|
||||
return (
|
||||
f"* **Images**: {self.assets_carried} carried of {self.assets_found} found, "
|
||||
f"written to `{ASSETS_DIR}/` and pointed at from the concepts they stand in. "
|
||||
"The image BYTES were not screened: the gate above reads text, and a "
|
||||
"picture is not text."
|
||||
)
|
||||
|
||||
|
||||
def load_plans(plans_dir: Path) -> dict[str, SegmentationPlan]:
|
||||
"""Every proposal artifact in a directory, keyed by filename.
|
||||
|
|
@ -357,6 +419,7 @@ def measure(
|
|||
ocr: bool = False,
|
||||
concept_frontmatter_values: Mapping[str, str] | None = None,
|
||||
gate: str = GATE_NONE,
|
||||
assets: bool = False,
|
||||
) -> CorpusReport:
|
||||
"""Run the corpus through the door and count what happened.
|
||||
|
||||
|
|
@ -381,6 +444,7 @@ def measure(
|
|||
pdf_headings=pdf_headings,
|
||||
heading_reserve=heading_reserve,
|
||||
ocr=ocr,
|
||||
assets=assets,
|
||||
concept_frontmatter_values=concept_frontmatter_values,
|
||||
)
|
||||
elapsed = time.monotonic() - started
|
||||
|
|
@ -416,6 +480,9 @@ def measure(
|
|||
codes=tuple(sorted(counts.items())),
|
||||
unaccounted=unaccounted_names(dropped=dropped, merged=merged_names, coded=coded_names),
|
||||
gate=gate,
|
||||
assets=assets,
|
||||
assets_carried=len(result.assets),
|
||||
assets_found=len(result.assets) + len(result.assets_rejected),
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -456,6 +523,24 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
|||
"would make that equality depend on which command you ran"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--assets",
|
||||
action="store_true",
|
||||
default=DEFAULT_ASSETS,
|
||||
help=(
|
||||
"carry the images the sources declare into the bundle. The same "
|
||||
"name and the same default as `okf build --assets`, for the same "
|
||||
"reason the gate carries them: the two paths are held byte-equal "
|
||||
"by a test, and two different defaults would make that equality "
|
||||
"depend on which command you ran"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-assets",
|
||||
action="store_false",
|
||||
dest="assets",
|
||||
help="the explicit opt-out, reproducing the pre-0.10.0 bytes",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--okf-version",
|
||||
default=None,
|
||||
|
|
@ -532,6 +617,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
report = measure(
|
||||
args.corpus,
|
||||
bundle,
|
||||
assets=args.assets,
|
||||
ingested_at=args.ingested_at,
|
||||
plans=plans,
|
||||
profile=profile,
|
||||
|
|
|
|||
|
|
@ -108,6 +108,31 @@ class ExtractionError(IngestError):
|
|||
`extractor_extra_missing`: the `[extract]` extra can be fully installed
|
||||
and the document parsed, with only the OCR engine absent, and one code
|
||||
for both would send an operator to reinstall what they already have
|
||||
|
||||
ASSET codes (0.10.0). None of these fails a document: an image a reader
|
||||
could not carry is a ROW in the run log and a line in the concept saying
|
||||
what was there, because one unreadable picture must not cost the three
|
||||
thousand concepts of text around it.
|
||||
|
||||
- `asset_type_unknown` — the bytes behind a pointer are not an image format
|
||||
this package recognises. Sniffed from the bytes, never from the claimed
|
||||
extension: a `.jpg` that is really a PNG would otherwise be written under
|
||||
a name whose extension lies
|
||||
- `asset_unresolved` — the file a document points at was not found beside
|
||||
it, or an inline data URI could not be decoded. Containment is the
|
||||
document's own directory, so a reference above it lands here rather than
|
||||
being followed
|
||||
- `asset_remote` — the source is off this machine. Extraction opens no
|
||||
socket: network access is an explicit per-run opt-in and extraction is not
|
||||
on that path, so a remote image is carried as a pointer and never as bytes
|
||||
- `asset_pdf_unsupported` — a PDF image whose samples this encoder does not
|
||||
express: a stencil mask, a `Decode` array, a colour space with no exact
|
||||
PNG form, a soft mask that cannot be carried, or anything but 8-bit
|
||||
samples. Refused rather than approximated, because a picture that is
|
||||
plausibly the wrong colour is wrong in a way no consumer can detect
|
||||
- `asset_samples_invalid` — the sample buffer does not fit the dimensions
|
||||
the image dictionary declares. Refused rather than padded: a short buffer
|
||||
means the dictionary was read wrong
|
||||
"""
|
||||
|
||||
|
||||
|
|
@ -130,6 +155,12 @@ class MaterializationError(IngestError):
|
|||
- `ingested_at_invalid` — ingested_at is not ISO-8601 UTC with a Z suffix
|
||||
- `collision_unstamped` — the §3 collision gate: a generated name is
|
||||
occupied by a file without the ingest stamp
|
||||
- `asset_collision` — two different images reduce to one asset name in one
|
||||
run, or a name in `assets/` is occupied by different bytes (Doors B and
|
||||
C, 0.10.0). An asset name carries the digest of its own contents, so this
|
||||
is a `sha256[:12]` collision; refused rather than resolved, because
|
||||
resolving it silently means one of two pictures is lost and every pointer
|
||||
to it shows the other
|
||||
- `source_reference_unquotable` — a manifest source's id or locator
|
||||
contains a character that would restructure the `sources` flow mapping
|
||||
(Door A, v0.2 profiles); refused rather than emitted, because the
|
||||
|
|
|
|||
|
|
@ -23,6 +23,8 @@ registry's. No guard call and no model call anywhere in this module.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import collections
|
||||
import csv
|
||||
import functools
|
||||
|
|
@ -30,6 +32,7 @@ import io
|
|||
import re
|
||||
import statistics
|
||||
import tempfile
|
||||
import urllib.parse
|
||||
import warnings
|
||||
import zipfile
|
||||
from collections.abc import Callable, Sequence
|
||||
|
|
@ -39,7 +42,17 @@ from pathlib import Path
|
|||
from xml.etree import ElementTree
|
||||
from xml.etree.ElementTree import Element
|
||||
|
||||
from .errors import ExtractionError, ExtractionWarning
|
||||
from .assets import (
|
||||
AssetRejection,
|
||||
ExtractedImage,
|
||||
encode_png,
|
||||
read_image,
|
||||
render_block,
|
||||
render_missing,
|
||||
sniff,
|
||||
)
|
||||
from .connectors import safe_resolve
|
||||
from .errors import ExtractionError, ExtractionWarning, SourceError
|
||||
from .render import render_fenced_block, render_table
|
||||
|
||||
# Binary types gated behind the optional `[extract]` extra that it ships no
|
||||
|
|
@ -329,6 +342,174 @@ _XML_INLINE_TAGS = frozenset(
|
|||
_ATX_MAX_LEVEL = 6
|
||||
|
||||
|
||||
# --- assets: the images a document carries beside its text ------------------
|
||||
#
|
||||
# ADDED IN 0.10.0, and off unless the caller asks. Until then no reader here
|
||||
# fetched, named or copied an image: `<img>`'s attributes were never read, an
|
||||
# STS `<graphic>` was walked past, a PDF was opened for `extract_text` alone,
|
||||
# and the converter's markdown writer dropped every picture. The two lossiness
|
||||
# warnings above said so on every run, which made the loss honest and did not
|
||||
# make it smaller.
|
||||
#
|
||||
# THE READER PLACES, THE COLLECTOR DECIDES. Each reader knows where in its own
|
||||
# document an image stands and what the document calls it; nothing else. What
|
||||
# an image IS, what it is named in the bundle and how it is pointed at is
|
||||
# `assets.py`'s, so the four readers cannot drift into four grammars, and
|
||||
# "carried N of M" means one thing across all of them.
|
||||
|
||||
#: How a reader asks for bytes the document only POINTS at. The inbox supplies
|
||||
#: one rooted at the dropped file's own directory and refuses to leave the drop
|
||||
#: -- an `<img src="../../../etc/passwd">` is a path traversal written by
|
||||
#: whoever wrote the document, which is exactly the class of input this package
|
||||
#: treats as untrusted. `None` means the caller gave no resolver, and every
|
||||
#: pointer then resolves to nothing rather than to a guess.
|
||||
Resolver = Callable[[str], bytes | None]
|
||||
|
||||
#: A scheme this package will not open. Extraction opens no socket at all --
|
||||
#: network access here is an explicit per-run opt-in and extraction is not on
|
||||
#: that path -- so a remote image is carried as a POINTER and never as bytes.
|
||||
_REMOTE_SOURCE = re.compile(r"^(?:[a-zA-Z][a-zA-Z0-9+.-]*:|//)")
|
||||
|
||||
_DATA_URI = re.compile(r"^data:(?P<media>[^;,]*)(?P<base64>;base64)?,(?P<payload>.*)$", re.DOTALL)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExtractedDocument:
|
||||
"""One dropped file's text, and the images that stand inside that text.
|
||||
|
||||
The text is what it always was when `images` and `rejected` are empty --
|
||||
which is what the `assets=False` default guarantees byte for byte, so every
|
||||
golden, every pinned bundle and every published digest is unmoved by this
|
||||
capability existing.
|
||||
"""
|
||||
|
||||
text: str
|
||||
images: tuple[ExtractedImage, ...] = ()
|
||||
rejected: tuple[AssetRejection, ...] = ()
|
||||
|
||||
|
||||
class _AssetCollector:
|
||||
"""What a reader hands an image to, and the only place a rejection is made.
|
||||
|
||||
A rejection is a ROW, never a failed document: one unreadable picture must
|
||||
not cost the text around it. Every rejection also leaves a line in the
|
||||
concept saying what was there and why it is not here -- an absence a reader
|
||||
is never shown is an absence they cannot weigh, which is the defect this
|
||||
whole capability closes one level up.
|
||||
"""
|
||||
|
||||
def __init__(self, resolve: Resolver | None = None) -> None:
|
||||
self._resolve = resolve
|
||||
self.images: list[ExtractedImage] = []
|
||||
self.rejected: list[AssetRejection] = []
|
||||
|
||||
def carry(self, data: bytes, *, name: str, label: str | None = None) -> str:
|
||||
"""Bytes the reader already holds, as the block that stands in the text."""
|
||||
try:
|
||||
image = read_image(data, name=name, label=label)
|
||||
except ExtractionError as exc:
|
||||
return self.reject(name, code=exc.code, reason=str(exc), label=label)
|
||||
self.images.append(image)
|
||||
return render_block(image)
|
||||
|
||||
def reject(
|
||||
self,
|
||||
name: str,
|
||||
*,
|
||||
code: str,
|
||||
reason: str,
|
||||
label: str | None = None,
|
||||
href: str | None = None,
|
||||
) -> str:
|
||||
self.rejected.append(AssetRejection(name=name, code=code, reason=reason))
|
||||
return render_missing(name, reason=reason, label=label, href=href)
|
||||
|
||||
def local(self, source: str, *, label: str | None = None, sibling: str | None = None) -> str:
|
||||
"""An href the DOCUMENT points at: a data URI, a local path, or remote.
|
||||
|
||||
`sibling` is a SECOND path to try, and it exists because one format's
|
||||
convention is not another's: a NISO-STS delivery writes a bare file
|
||||
name and ships the files in a `graphics/` directory beside the
|
||||
document. Passing the candidate rather than teaching this method about
|
||||
STS keeps the convention with the reader that has it, and the
|
||||
containment rule with the resolver that owns it.
|
||||
"""
|
||||
data_uri = _DATA_URI.match(source)
|
||||
if data_uri is not None:
|
||||
return self._data_uri(data_uri, label=label)
|
||||
if _REMOTE_SOURCE.match(source):
|
||||
return self.reject(
|
||||
source,
|
||||
code="asset_remote",
|
||||
reason="the source is off this machine and extraction opens no socket",
|
||||
label=label,
|
||||
href=source,
|
||||
)
|
||||
data = self._resolve(source) if self._resolve is not None else None
|
||||
if data is None and sibling is not None and sibling != source and self._resolve is not None:
|
||||
data = self._resolve(sibling)
|
||||
if data is None:
|
||||
return self.reject(
|
||||
source,
|
||||
code="asset_unresolved",
|
||||
reason="the file the document points at was not found beside it",
|
||||
label=label,
|
||||
)
|
||||
return self.carry(data, name=source, label=label)
|
||||
|
||||
def _data_uri(self, match: re.Match[str], *, label: str | None) -> str:
|
||||
payload = match.group("payload")
|
||||
name = f"data-uri{Path(match.group('media').split('/')[-1] or 'bin').suffix or ''}"
|
||||
try:
|
||||
raw = (
|
||||
base64.b64decode(payload, validate=True)
|
||||
if match.group("base64")
|
||||
else urllib.parse.unquote_to_bytes(payload)
|
||||
)
|
||||
except (binascii.Error, ValueError) as exc:
|
||||
return self.reject(
|
||||
name,
|
||||
code="asset_unresolved",
|
||||
reason=f"the inline data URI could not be decoded: {exc}",
|
||||
label=label,
|
||||
)
|
||||
return self.carry(raw, name=name, label=label)
|
||||
|
||||
|
||||
def directory_resolver(root: Path) -> Resolver:
|
||||
"""A resolver rooted at ONE directory, fail-closed, reading nothing else.
|
||||
|
||||
Containment is against the DOCUMENT'S OWN directory rather than against the
|
||||
whole drop, and that is not caution for its own sake: the proposer reads a
|
||||
file straight off disk and the door reads it out of the inbox, and a plan
|
||||
indexes the exact string it was proposed against. One root both sides can
|
||||
compute from the document alone is what makes the two renderings identical
|
||||
without threading a second path through either. A reference above the
|
||||
document's directory is refused (`asset_unresolved`) rather than followed;
|
||||
the limit is stated in the concept, like every other rejection.
|
||||
|
||||
`safe_resolve` is Door A's own rule, reused verbatim: `..` traversal, an
|
||||
absolute path, a symlink escape and a prefix-collision sibling all fail
|
||||
closed. An `<img src="../../../etc/passwd">` is a path written by whoever
|
||||
wrote the document, which is exactly the class of input this package treats
|
||||
as untrusted.
|
||||
"""
|
||||
|
||||
def resolve(relative: str) -> bytes | None:
|
||||
try:
|
||||
target = safe_resolve(root, relative)
|
||||
except SourceError:
|
||||
return None
|
||||
try:
|
||||
if not target.is_file():
|
||||
return None
|
||||
return target.read_bytes()
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
return resolve
|
||||
|
||||
|
||||
def decode_text(data: bytes) -> str:
|
||||
"""Decode file bytes as UTF-8 (BOM-stripping), typed on failure.
|
||||
|
||||
|
|
@ -384,12 +565,13 @@ class _HTMLTextExtractor(HTMLParser):
|
|||
one-line form. Nothing is ever dropped here beyond `_SKIP_TAGS`.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
def __init__(self, collector: _AssetCollector | None = None) -> None:
|
||||
super().__init__(convert_charrefs=True)
|
||||
self._lines: list[str] = []
|
||||
self._current: list[str] = []
|
||||
self._prefix = ""
|
||||
self._skip_depth = 0
|
||||
self._collector = collector
|
||||
|
||||
def _break(self, prefix: str = "") -> None:
|
||||
"""Close the line being accumulated and open the next one."""
|
||||
|
|
@ -399,6 +581,26 @@ class _HTMLTextExtractor(HTMLParser):
|
|||
self._lines.append(f"{self._prefix}{line}")
|
||||
self._prefix = prefix
|
||||
|
||||
def _image(self, attrs: list[tuple[str, str | None]]) -> None:
|
||||
"""An `<img>`, in the place it stands.
|
||||
|
||||
`attrs` was read by nothing here until 0.10.0, so `src` and `alt` were
|
||||
both dropped -- a document's figures left no trace in the concept at
|
||||
all, not even their alt text. The block goes on its own lines because
|
||||
markdown wants a paragraph and because every boundary grammar in
|
||||
`propose.py` is line-anchored.
|
||||
"""
|
||||
if self._collector is None or self._skip_depth:
|
||||
return
|
||||
values = {key: value or "" for key, value in attrs}
|
||||
source = values.get("src", "").strip()
|
||||
if not source:
|
||||
return
|
||||
self._break()
|
||||
for line in self._collector.local(source, label=values.get("alt") or None).split("\n"):
|
||||
self._lines.append(line)
|
||||
self._break()
|
||||
|
||||
def _open(self, tag: str) -> bool:
|
||||
"""Break for a block or heading tag; report whether it was one."""
|
||||
level = _HEADING_TAGS.get(tag)
|
||||
|
|
@ -415,10 +617,14 @@ class _HTMLTextExtractor(HTMLParser):
|
|||
self._current.append(" ")
|
||||
if tag in _SKIP_TAGS:
|
||||
self._skip_depth += 1
|
||||
if tag == "img":
|
||||
self._image(attrs)
|
||||
|
||||
def handle_startendtag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
|
||||
if not self._open(tag):
|
||||
self._current.append(" ")
|
||||
if tag == "img":
|
||||
self._image(attrs)
|
||||
|
||||
def handle_endtag(self, tag: str) -> None:
|
||||
if tag in _SKIP_TAGS and self._skip_depth > 0:
|
||||
|
|
@ -435,9 +641,9 @@ class _HTMLTextExtractor(HTMLParser):
|
|||
return "\n".join(self._lines)
|
||||
|
||||
|
||||
def _extract_html(data: bytes) -> str:
|
||||
def _extract_html(data: bytes, collector: _AssetCollector | None = None) -> str:
|
||||
"""`html`/`htm`: text via `html.parser`, script/style stripped (spec B3)."""
|
||||
parser = _HTMLTextExtractor()
|
||||
parser = _HTMLTextExtractor(collector)
|
||||
parser.feed(decode_text(data))
|
||||
parser.close()
|
||||
return parser.text()
|
||||
|
|
@ -472,8 +678,9 @@ class _XmlTextExtractor:
|
|||
identical to `"".join(root.itertext())`.
|
||||
"""
|
||||
|
||||
def __init__(self, *, sts: bool) -> None:
|
||||
def __init__(self, *, sts: bool, collector: _AssetCollector | None = None) -> None:
|
||||
self._sts = sts
|
||||
self._collector = collector
|
||||
self._lines: list[str] = []
|
||||
self._current: list[str] = []
|
||||
self._prefix = ""
|
||||
|
|
@ -572,8 +779,42 @@ class _XmlTextExtractor:
|
|||
self._lines.extend(render_table(rows[0], rows[1:]).rstrip("\n").split("\n"))
|
||||
return True
|
||||
|
||||
def _graphic(self, element: Element) -> bool:
|
||||
"""A `<graphic>`, in the place it stands. Reports whether it was one.
|
||||
|
||||
Measured on the R761 delivery, 2026-09-16: 50 `<graphic>` elements, all
|
||||
50 direct children of a `<sec>`, none inside a `<table-wrap>`, none
|
||||
carrying a caption element of any kind -- the "Figur 11.1 ..." line a
|
||||
human reads is a sibling `<p>` this reader already emits on its own
|
||||
line. So the label falls back to the file name rather than being
|
||||
guessed from the neighbourhood.
|
||||
|
||||
TWO RESOLUTION ROUTES, and the second is the delivery's own convention:
|
||||
the href as written, and then `graphics/<name>`, because that publisher
|
||||
writes a BARE file name and ships the files in a sibling directory.
|
||||
Both are tried through the caller's resolver, which is what keeps the
|
||||
containment rule in one place.
|
||||
"""
|
||||
if self._collector is None:
|
||||
return False
|
||||
href = next(
|
||||
(value for key, value in element.attrib.items() if _local_name(key) == "href"),
|
||||
None,
|
||||
)
|
||||
if not href:
|
||||
return False
|
||||
block = self._collector.local(href, sibling=f"graphics/{Path(href).name}")
|
||||
self._emit_lines(block.split("\n"))
|
||||
return True
|
||||
|
||||
def _emit_lines(self, lines: list[str]) -> None:
|
||||
for line in lines:
|
||||
self._emit(line)
|
||||
|
||||
def _walk(self, element: Element, depth: int) -> None:
|
||||
tag = _local_name(element.tag)
|
||||
if tag in ("graphic", "inline-graphic") and self._graphic(element):
|
||||
return
|
||||
if self._sts and tag == "table-wrap" and self._table(element):
|
||||
return
|
||||
skip: set[int] = set()
|
||||
|
|
@ -627,7 +868,9 @@ class _XmlTextExtractor:
|
|||
return "\n".join(self._lines)
|
||||
|
||||
|
||||
def _xml_document(data: bytes) -> tuple[str, tuple[OutlineMark, ...]]:
|
||||
def _xml_document(
|
||||
data: bytes, collector: _AssetCollector | None = None
|
||||
) -> tuple[str, tuple[OutlineMark, ...]]:
|
||||
"""`xml`: NISO-STS structure as markdown, any other schema as its text.
|
||||
|
||||
A DTD IS REFUSED RATHER THAN PARSED, and that is a guarantee about this
|
||||
|
|
@ -643,7 +886,7 @@ def _xml_document(data: bytes) -> tuple[str, tuple[OutlineMark, ...]]:
|
|||
holds on every interpreter.
|
||||
"""
|
||||
root = _parse_xml(data)
|
||||
reader = _XmlTextExtractor(sts=_is_sts(root))
|
||||
reader = _XmlTextExtractor(sts=_is_sts(root), collector=collector)
|
||||
return reader.text(root), tuple(reader.marks)
|
||||
|
||||
|
||||
|
|
@ -736,11 +979,13 @@ def _child_text(element: Element, name: str) -> str | None:
|
|||
return None
|
||||
|
||||
|
||||
def _extract_xml(data: bytes) -> str:
|
||||
return _xml_document(data)[0]
|
||||
def _extract_xml(data: bytes, collector: _AssetCollector | None = None) -> str:
|
||||
return _xml_document(data, collector)[0]
|
||||
|
||||
|
||||
def xml_outline(name: str, data: bytes) -> tuple[OutlineMark, ...]:
|
||||
def xml_outline(
|
||||
name: str, data: bytes, *, assets: bool = False, resolve: Resolver | None = None
|
||||
) -> tuple[OutlineMark, ...]:
|
||||
"""`xml`: the sections the document DECLARES, as marks on the extracted text.
|
||||
|
||||
The counterpart of `pdf_outline`, and the difference between them is the
|
||||
|
|
@ -762,7 +1007,12 @@ def xml_outline(name: str, data: bytes) -> tuple[OutlineMark, ...]:
|
|||
list as "leave every rule untouched", never as a route.
|
||||
"""
|
||||
del name # the registry decides which reader runs; kept for `pdf_outline`'s shape
|
||||
return _xml_document(data)[1]
|
||||
# `assets` and `resolve` are NOT options of this arm, exactly as
|
||||
# `pdf_headings` is not one of `pdf_outline`'s: carrying an image inserts
|
||||
# lines into the extracted text, so marks computed with the images off name
|
||||
# the right sections at the wrong line numbers. They are threaded so both
|
||||
# sides of the plan can be computed against ONE rendering.
|
||||
return _xml_document(data, _AssetCollector(resolve) if assets else None)[1]
|
||||
|
||||
|
||||
def _local_name(tag: str) -> str:
|
||||
|
|
@ -966,11 +1216,251 @@ def _mark_headings(lines: list[tuple[str, float, str]], levels: dict[float, int]
|
|||
_PDF_PAGE_SEPARATOR = "\n\n"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _PdfPage:
|
||||
"""One page's text, and the images drawn on it, kept APART on purpose.
|
||||
|
||||
The pointer blocks are appended after the page's own lines by
|
||||
`_pdf_page_text`, and the body is kept separately because `pdf_outline`
|
||||
compares the page's line splitting against `page.extract_text_lines()` --
|
||||
a per-page check that ships and that decides whether the primary bridge
|
||||
route may be used at all. Appended lines are not in that geometry, so a
|
||||
joined string would fail the check on every page carrying an image and
|
||||
silently demote 2 762 bookmarks to the fallback route.
|
||||
"""
|
||||
|
||||
number: int
|
||||
text: str
|
||||
images: tuple[ExtractedImage, ...] = ()
|
||||
rejected: tuple[AssetRejection, ...] = ()
|
||||
|
||||
|
||||
def _pdf_page_text(page: _PdfPage) -> str:
|
||||
"""A page as it reaches the extracted text: its lines, then its pointers.
|
||||
|
||||
END OF PAGE, not the image's y position, and the reason is stated rather
|
||||
than hidden: a PDF image has a bounding box and no place in the reading
|
||||
order, so "where it stands" is the page. Inserting by y would reorder the
|
||||
page's own lines against the geometry `pdf_outline` checks itself against,
|
||||
and would put a pointer inside a sentence. A caption printed above a figure
|
||||
therefore keeps its own line where the document put it, and the pointer
|
||||
follows the page it was drawn on.
|
||||
"""
|
||||
blocks = [render_block(image) for image in page.images]
|
||||
blocks += [
|
||||
render_missing(rejection.name, reason=rejection.reason) for rejection in page.rejected
|
||||
]
|
||||
if not blocks:
|
||||
return page.text
|
||||
joined = "\n\n".join(blocks)
|
||||
return f"{page.text}\n\n{joined}" if page.text else joined
|
||||
|
||||
|
||||
#: The bits-per-component this encoder expresses. A PDF may store 1, 2, 4, 8 or
|
||||
#: 16, and everything but 8 is REFUSED with a code rather than rescaled --
|
||||
#: rescaling a 1-bit stencil to 8 bits is a decision about what black means, and
|
||||
#: a wrong one looks exactly like a right one.
|
||||
_PDF_SAMPLE_BITS = 8
|
||||
|
||||
|
||||
def _pdf_colour(space: object) -> tuple[int, bytes | None] | None:
|
||||
"""A PDF colour space as `(channels, palette)`, or `None` if not expressible.
|
||||
|
||||
`None` is the honest answer for CMYK, for a separation space and for
|
||||
anything with a transfer function: converting those needs a colour model
|
||||
this package does not carry, and a guess would be a picture that is
|
||||
plausibly the wrong colour. It is counted and stated, never approximated.
|
||||
"""
|
||||
from pdfminer.pdftypes import PDFStream, resolve1
|
||||
|
||||
space = resolve1(space)
|
||||
name = getattr(space, "name", None)
|
||||
if name in ("DeviceGray", "CalGray", "G"):
|
||||
return 1, None
|
||||
if name in ("DeviceRGB", "CalRGB", "RGB"):
|
||||
return 3, None
|
||||
if not isinstance(space, list) or not space:
|
||||
return None
|
||||
head = getattr(resolve1(space[0]), "name", None)
|
||||
if head == "ICCBased" and len(space) > 1:
|
||||
profile = resolve1(space[1])
|
||||
components = resolve1(profile.attrs.get("N")) if isinstance(profile, PDFStream) else None
|
||||
return (int(components), None) if components in (1, 3) else None
|
||||
if head in ("CalGray",):
|
||||
return 1, None
|
||||
if head in ("CalRGB", "Lab"):
|
||||
return 3, None
|
||||
if head in ("Indexed", "I") and len(space) >= 4:
|
||||
base = _pdf_colour(space[1])
|
||||
if base is None:
|
||||
return None
|
||||
lookup = resolve1(space[3])
|
||||
if isinstance(lookup, PDFStream):
|
||||
lookup = lookup.get_data()
|
||||
if not isinstance(lookup, bytes):
|
||||
return None
|
||||
if base[0] == 3:
|
||||
palette = lookup[: (len(lookup) // 3) * 3]
|
||||
else:
|
||||
# PNG's PLTE is RGB triples only, so a grey palette is widened
|
||||
# rather than refused. Widening a grey to r=g=b is exact, not an
|
||||
# approximation -- which is why this branch exists and the CMYK one
|
||||
# does not.
|
||||
palette = b"".join(bytes([value, value, value]) for value in lookup)
|
||||
return (1, palette) if palette else None
|
||||
return None
|
||||
|
||||
|
||||
def _pdf_alpha(attrs: dict[str, object], width: int, height: int) -> bytes | None | bool:
|
||||
"""A soft mask as one alpha byte per pixel, `None` for none, `False` to refuse.
|
||||
|
||||
An `SMask` this encoder cannot express is a REFUSAL rather than a dropped
|
||||
channel: an image whose transparency is thrown away is composited against
|
||||
nothing and reads as a black or white rectangle over the page, which is a
|
||||
picture that is wrong in a way no consumer can detect.
|
||||
"""
|
||||
from pdfminer.pdftypes import PDFStream, resolve1
|
||||
|
||||
mask = resolve1(attrs.get("SMask"))
|
||||
if mask is None:
|
||||
return None
|
||||
if not isinstance(mask, PDFStream):
|
||||
return False
|
||||
shape = mask.attrs
|
||||
if (
|
||||
resolve1(shape.get("Width")) != width
|
||||
or resolve1(shape.get("Height")) != height
|
||||
or resolve1(shape.get("BitsPerComponent")) != _PDF_SAMPLE_BITS
|
||||
):
|
||||
return False
|
||||
try:
|
||||
alpha = mask.get_data()
|
||||
except Exception:
|
||||
return False
|
||||
return alpha if len(alpha) >= width * height else False
|
||||
|
||||
|
||||
def _pdf_image(stream: object, name: str) -> ExtractedImage:
|
||||
"""One image XObject, carried verbatim where it already is a file.
|
||||
|
||||
TWO ROUTES, and which one runs is decided by the BYTES rather than by the
|
||||
filter name. `get_data()` applies every filter pdfminer knows and stops at
|
||||
the image codecs, so a `DCTDecode` stream comes back as a finished JPEG and
|
||||
a `FlateDecode` one comes back as raw samples. Sniffing the result is what
|
||||
makes the first route exact: an embedded JPEG is written to the bundle as
|
||||
the publisher's own bytes, unre-encoded, and its content-addressed name is
|
||||
therefore stable for as long as the document is.
|
||||
|
||||
Measured on R761 (2026-09-16): 29 of 50 image objects are `DCTDecode` and
|
||||
take the verbatim route; 21 are `FlateDecode` and are encoded here. Over
|
||||
the 33-document K2 reference corpus the population is 4 828 objects, and
|
||||
the filters are mixed enough (`FlateDecode`, `DCTDecode`, `JPXDecode`,
|
||||
`ASCII85Decode` chains, `CCITTFaxDecode`) that guessing from the filter
|
||||
name would have been wrong on several hundred.
|
||||
|
||||
RENDERING THE PAGE REGION WAS THE ALTERNATIVE AND IT WAS NOT TAKEN. A
|
||||
rasterised crop would be one code path and would handle every filter, but
|
||||
its bytes -- and therefore the asset's name and the bundle's digest --
|
||||
would depend on the version of the rasteriser installed, which is the one
|
||||
property `OCR_DPI`'s docstring already admits OCR text cannot have. An
|
||||
embedded stream has no such dependency.
|
||||
"""
|
||||
from pdfminer.pdftypes import resolve1
|
||||
|
||||
try:
|
||||
data = stream.get_data() # type: ignore[attr-defined]
|
||||
except Exception as exc:
|
||||
raise ExtractionError(
|
||||
f"the PDF image stream behind {name!r} could not be decoded: {exc}",
|
||||
code="asset_pdf_unsupported",
|
||||
) from exc
|
||||
if data and sniff(data) is not None:
|
||||
return read_image(data, name=name)
|
||||
|
||||
attrs = dict(getattr(stream, "attrs", {}))
|
||||
width = resolve1(attrs.get("Width"))
|
||||
height = resolve1(attrs.get("Height"))
|
||||
bits = resolve1(attrs.get("BitsPerComponent"))
|
||||
if not isinstance(width, int) or not isinstance(height, int):
|
||||
raise ExtractionError(
|
||||
f"the PDF image {name!r} declares no usable size",
|
||||
code="asset_pdf_unsupported",
|
||||
)
|
||||
if resolve1(attrs.get("ImageMask")):
|
||||
raise ExtractionError(
|
||||
f"the PDF image {name!r} is a stencil mask, which paints the current "
|
||||
"fill colour rather than carrying one of its own",
|
||||
code="asset_pdf_unsupported",
|
||||
)
|
||||
if bits != _PDF_SAMPLE_BITS:
|
||||
raise ExtractionError(
|
||||
f"the PDF image {name!r} stores {bits}-bit samples; this encoder writes "
|
||||
f"{_PDF_SAMPLE_BITS}-bit ones and will not rescale, because rescaling a "
|
||||
"stencil is a decision about what black means",
|
||||
code="asset_pdf_unsupported",
|
||||
)
|
||||
if attrs.get("Decode") is not None:
|
||||
raise ExtractionError(
|
||||
f"the PDF image {name!r} carries a Decode array, which remaps every "
|
||||
"sample; carrying it unmapped would invert the picture",
|
||||
code="asset_pdf_unsupported",
|
||||
)
|
||||
colour = _pdf_colour(attrs.get("ColorSpace"))
|
||||
if colour is None:
|
||||
raise ExtractionError(
|
||||
f"the PDF image {name!r} uses a colour space this encoder does not "
|
||||
f"express ({attrs.get('ColorSpace')!r})",
|
||||
code="asset_pdf_unsupported",
|
||||
)
|
||||
alpha = _pdf_alpha(attrs, width, height)
|
||||
if alpha is False:
|
||||
raise ExtractionError(
|
||||
f"the PDF image {name!r} has a soft mask this encoder cannot express; "
|
||||
"dropping transparency would composite the picture against nothing",
|
||||
code="asset_pdf_unsupported",
|
||||
)
|
||||
channels, palette = colour
|
||||
encoded = encode_png(
|
||||
width,
|
||||
height,
|
||||
data,
|
||||
channels=channels,
|
||||
palette=palette,
|
||||
alpha=alpha if isinstance(alpha, bytes) else None,
|
||||
)
|
||||
return read_image(encoded, name=name)
|
||||
|
||||
|
||||
def _pdf_images(page: object) -> tuple[tuple[ExtractedImage, ...], tuple[AssetRejection, ...]]:
|
||||
"""Every image drawn on one page, with the failures kept beside them."""
|
||||
carried: list[ExtractedImage] = []
|
||||
rejected: list[AssetRejection] = []
|
||||
number = getattr(page, "page_number", 0)
|
||||
for index, drawn in enumerate(getattr(page, "images", []) or [], start=1):
|
||||
# The name a PDF image does NOT have. An XObject is reached through a
|
||||
# resource name local to one page's dictionary, so it is not an
|
||||
# identifier -- the page number in front of it is what makes the string
|
||||
# readable, and the content-addressed digest is what makes it unique.
|
||||
label = str(drawn.get("name") or index).lstrip("/")
|
||||
name = f"page-{number}-{label}"
|
||||
stream = drawn.get("stream")
|
||||
if stream is None:
|
||||
rejected.append(
|
||||
AssetRejection(name, "asset_pdf_unsupported", "the image object has no stream")
|
||||
)
|
||||
continue
|
||||
try:
|
||||
carried.append(_pdf_image(stream, name))
|
||||
except ExtractionError as exc:
|
||||
rejected.append(AssetRejection(name, exc.code, str(exc)))
|
||||
return tuple(carried), tuple(rejected)
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize=1)
|
||||
def _pdf_pages(
|
||||
data: bytes, headings: bool = False, ocr: bool = False
|
||||
) -> tuple[tuple[int, str], ...]:
|
||||
"""Every page that produced text, as `(page number, text)`, in page order.
|
||||
data: bytes, headings: bool = False, ocr: bool = False, assets: bool = False
|
||||
) -> tuple[_PdfPage, ...]:
|
||||
"""Every page that produced content, as a `_PdfPage`, in page order.
|
||||
|
||||
The page NUMBER is 1-based and comes from the document, so a page that
|
||||
yielded nothing removes itself from the sequence without renumbering the
|
||||
|
|
@ -991,6 +1481,13 @@ def _pdf_pages(
|
|||
from its lines. Measured, the two agree on 11 of 11 pages of a real tender
|
||||
PDF -- but "agree on the document I tried" is not a contract, so the
|
||||
default does not depend on it.
|
||||
|
||||
`assets` is the third, and with it off not one line below it runs: no
|
||||
stream is decoded, no sample buffer is allocated, and the emitted pages are
|
||||
the objects they always were. A page that produced no TEXT is still
|
||||
dropped even when it carries an image, because `_extract_pdf` refuses a
|
||||
document with no text at all (`extractor_empty_pdf`) and an image-only
|
||||
document is `--ocr`'s question, not this one's.
|
||||
"""
|
||||
try:
|
||||
import pdfplumber
|
||||
|
|
@ -1005,8 +1502,12 @@ def _pdf_pages(
|
|||
# cannot supply it. A title page is 100 % heading, and a per-page
|
||||
# median would compare it with itself and mark nothing.
|
||||
recovered: list[str | list[tuple[str, float, str]]] = []
|
||||
numbers: list[int] = []
|
||||
drawn: list[tuple[tuple[ExtractedImage, ...], tuple[AssetRejection, ...]]] = []
|
||||
for page in pdf.pages:
|
||||
flat = (page.extract_text() or "").rstrip()
|
||||
numbers.append(page.page_number)
|
||||
drawn.append(_pdf_images(page) if assets else ((), ()))
|
||||
if read is not None and needs_ocr(flat):
|
||||
# The page's own text is unusable, so it is replaced
|
||||
# WHOLESALE rather than merged with: a page of `(cid:N)`
|
||||
|
|
@ -1047,7 +1548,11 @@ def _pdf_pages(
|
|||
raise ExtractionError(
|
||||
f"the PDF parser failed on this file: {exc}", code="extractor_pdf_error"
|
||||
) from exc
|
||||
return tuple((number, page) for number, page in enumerate(pages, start=1) if page)
|
||||
return tuple(
|
||||
_PdfPage(number=number, text=text, images=images, rejected=rejected)
|
||||
for number, text, (images, rejected) in zip(numbers, pages, drawn)
|
||||
if text
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
|
@ -1138,7 +1643,12 @@ def _outline_page_and_top(doc: object, dest: object, action: object) -> tuple[ob
|
|||
|
||||
|
||||
def pdf_outline(
|
||||
name: str, data: bytes, *, pdf_headings: bool = False, ocr: bool = False
|
||||
name: str,
|
||||
data: bytes,
|
||||
*,
|
||||
pdf_headings: bool = False,
|
||||
ocr: bool = False,
|
||||
assets: bool = False,
|
||||
) -> PdfOutline:
|
||||
"""`pdf`: the file's own `/Outlines` tree, as marks on the extracted text.
|
||||
|
||||
|
|
@ -1177,14 +1687,21 @@ def pdf_outline(
|
|||
from pdfminer.pdfdocument import PDFNoOutlines
|
||||
from pdfminer.pdfpage import PDFPage
|
||||
|
||||
rendered = _pdf_pages(data, pdf_headings, ocr)
|
||||
rendered = _pdf_pages(data, pdf_headings, ocr, assets)
|
||||
starts: dict[int, int] = {}
|
||||
page_lines: dict[int, list[str]] = {}
|
||||
# The page's OWN lines, without the pointer blocks appended after them.
|
||||
# `starts` has to count the appended lines (they are in the text a mark
|
||||
# indexes) while the geometry check must not see them, because
|
||||
# `extract_text_lines` reports the page and knows nothing about what this
|
||||
# package added underneath it.
|
||||
body_lines: dict[int, list[str]] = {}
|
||||
offset = 0
|
||||
for number, page_text in rendered:
|
||||
starts[number] = offset
|
||||
page_lines[number] = page_text.split("\n")
|
||||
offset += len(page_lines[number]) + 1
|
||||
for rendered_page in rendered:
|
||||
starts[rendered_page.number] = offset
|
||||
page_lines[rendered_page.number] = _pdf_page_text(rendered_page).split("\n")
|
||||
body_lines[rendered_page.number] = rendered_page.text.split("\n")
|
||||
offset += len(page_lines[rendered_page.number]) + 1
|
||||
|
||||
unresolved = 0
|
||||
collided = 0
|
||||
|
|
@ -1225,7 +1742,7 @@ def pdf_outline(
|
|||
lines = page_lines[number]
|
||||
tops: list[float] | None = None
|
||||
geometry = page.extract_text_lines()
|
||||
if [str(entry["text"]) for entry in geometry] == lines:
|
||||
if [str(entry["text"]) for entry in geometry] == body_lines[number]:
|
||||
tops = [float(entry["top"]) for entry in geometry]
|
||||
height = float(page.height)
|
||||
for level, title, top in group:
|
||||
|
|
@ -1268,7 +1785,9 @@ def _normalise_outline(value: str) -> str:
|
|||
return re.sub(r"\s+", "", value).lower()
|
||||
|
||||
|
||||
def _extract_pdf(data: bytes, *, headings: bool = False, ocr: bool = False) -> str:
|
||||
def _extract_pdf(
|
||||
data: bytes, *, headings: bool = False, ocr: bool = False, assets: bool = False
|
||||
) -> str:
|
||||
"""`pdf`: page text via `pdfplumber`, in page order, pages separated by a
|
||||
blank line.
|
||||
|
||||
|
|
@ -1284,8 +1803,8 @@ def _extract_pdf(data: bytes, *, headings: bool = False, ocr: bool = False) -> s
|
|||
and pymupdf each emit all labels then all values. Re-pairing those is
|
||||
guesswork, and in a requirements document a wrong pairing looks right.
|
||||
"""
|
||||
pages = _pdf_pages(data, headings, ocr)
|
||||
text = _PDF_PAGE_SEPARATOR.join(page for _, page in pages)
|
||||
pages = _pdf_pages(data, headings, ocr, assets)
|
||||
text = _PDF_PAGE_SEPARATOR.join(_pdf_page_text(page) for page in pages)
|
||||
if not text:
|
||||
raise ExtractionError(
|
||||
"the PDF yielded no text on any page; a scanned or image-only "
|
||||
|
|
@ -1332,7 +1851,94 @@ def _convert_bytes(source: bytes, to: str, format: str, extra_args: Sequence[str
|
|||
)
|
||||
|
||||
|
||||
def _extract_office(suffix: str, data: bytes) -> str:
|
||||
#: A markdown image as the converter's own writer emits it. Measured against
|
||||
#: pandoc 3.10.2 on hand-laid fixtures: a `.docx` picture arrives as
|
||||
#: `{width="..." height="..."}` and a `.pptx` one as
|
||||
#: ``, so the title form and the attribute form are
|
||||
#: both real and a regex written for one of them silently leaves the other's
|
||||
#: link in the text.
|
||||
_MEDIA_LINK = re.compile(
|
||||
r"!\[(?P<alt>[^\]]*)\]\("
|
||||
r"(?:<(?P<angle>[^>]*)>|(?P<plain>[^)\s]*))"
|
||||
r'(?:\s+"(?P<title>[^"]*)")?\)'
|
||||
r"(?P<attrs>\{[^}]*\})?"
|
||||
)
|
||||
|
||||
|
||||
def _convert_with_media(
|
||||
source: bytes, to: str, format: str, extra_args: Sequence[str]
|
||||
) -> tuple[str, dict[str, bytes]]:
|
||||
"""The converter call again, with `--extract-media` and the files read back.
|
||||
|
||||
A separate function rather than a flag on `_convert_bytes` because the
|
||||
media must be READ INSIDE the temporary directory's lifetime: the directory
|
||||
is removed on every path, and a caller handed a rewritten markdown string
|
||||
pointing into it would hold links to files that no longer exist. Returning
|
||||
the bytes is what makes the seam closed.
|
||||
|
||||
The staging path is absolute, so the converter writes absolute links. That
|
||||
is deliberate: every one of them is replaced below, and a link that somehow
|
||||
survived would carry a temporary directory name into a concept -- a string
|
||||
that differs on every run, which a byte-determinism rule would catch loudly
|
||||
rather than never.
|
||||
"""
|
||||
import pypandoc
|
||||
|
||||
from ._pandoc import converter_path
|
||||
|
||||
with tempfile.TemporaryDirectory() as staging:
|
||||
staged = Path(staging) / f"input.{format}"
|
||||
staged.write_bytes(source)
|
||||
media_root = Path(staging) / "extracted"
|
||||
with converter_path():
|
||||
text = str(
|
||||
pypandoc.convert_file(
|
||||
str(staged),
|
||||
to,
|
||||
format=format,
|
||||
extra_args=[*extra_args, f"--extract-media={media_root}"],
|
||||
)
|
||||
)
|
||||
media: dict[str, bytes] = {}
|
||||
if media_root.is_dir():
|
||||
for path in sorted(media_root.rglob("*")):
|
||||
if path.is_file():
|
||||
media[str(path)] = path.read_bytes()
|
||||
return text, media
|
||||
|
||||
|
||||
def _rewrite_media_links(text: str, media: dict[str, bytes], collector: _AssetCollector) -> str:
|
||||
"""Every converter image link, replaced by this package's own pointer block.
|
||||
|
||||
UNCONDITIONAL, including the links that cannot be resolved. The converter
|
||||
already emitted a markdown image before this existed -- measured on a
|
||||
hand-laid `.docx`, today's output carries
|
||||
`` with no such file anywhere, which
|
||||
`structure._scan_references` reads as a cross-reference to a concept that
|
||||
cannot exist. Leaving an unresolvable link in place would keep that defect
|
||||
and add a temporary directory name to it.
|
||||
"""
|
||||
|
||||
def replace(match: re.Match[str]) -> str:
|
||||
target = match.group("angle") or match.group("plain") or ""
|
||||
label = match.group("alt") or match.group("title") or None
|
||||
data = media.get(target)
|
||||
if data is not None:
|
||||
# The name the CONTAINER gave it, not the staging path: pandoc
|
||||
# preserves the part name under its own media directory, so
|
||||
# `word/media/tabell-84-2.png` arrives as `media/tabell-84-2.png`.
|
||||
inside = target.split("/extracted/", 1)[-1]
|
||||
return collector.carry(data, name=inside, label=label)
|
||||
if not target:
|
||||
return collector.reject(
|
||||
"image", code="asset_unresolved", reason="the converter emitted no target"
|
||||
)
|
||||
return collector.local(target, label=label)
|
||||
|
||||
return _MEDIA_LINK.sub(replace, text)
|
||||
|
||||
|
||||
def _extract_office(suffix: str, data: bytes, collector: _AssetCollector | None = None) -> str:
|
||||
"""The five office rows, converted through the vendored binary.
|
||||
|
||||
Shaped after `_extract_pdf`: the gate is an import probe rather than a
|
||||
|
|
@ -1349,7 +1955,11 @@ def _extract_office(suffix: str, data: bytes) -> str:
|
|||
writer = _SPREADSHEET_WRITER if spreadsheet else _PANDOC_WRITER
|
||||
args = _SPREADSHEET_ARGS if spreadsheet else _PANDOC_ARGS
|
||||
try:
|
||||
text = _convert_bytes(data, writer, _PANDOC_FORMATS[suffix], args)
|
||||
if collector is None:
|
||||
text = _convert_bytes(data, writer, _PANDOC_FORMATS[suffix], args)
|
||||
else:
|
||||
text, media = _convert_with_media(data, writer, _PANDOC_FORMATS[suffix], args)
|
||||
text = _rewrite_media_links(text, media, collector)
|
||||
except ExtractionError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
|
|
@ -1594,14 +2204,17 @@ def _line_units(text: str) -> SourceUnits:
|
|||
return SourceUnits("lines", tuple(starts), tuple(range(1, len(starts) + 1)))
|
||||
|
||||
|
||||
def _pdf_units(data: bytes, headings: bool, ocr: bool) -> SourceUnits:
|
||||
def _pdf_units(data: bytes, headings: bool, ocr: bool, assets: bool = False) -> SourceUnits:
|
||||
starts: list[int] = []
|
||||
numbers: list[int] = []
|
||||
offset = 0
|
||||
for number, page in _pdf_pages(data, headings, ocr):
|
||||
for page in _pdf_pages(data, headings, ocr, assets):
|
||||
starts.append(offset)
|
||||
numbers.append(number)
|
||||
offset += len(page) + len(_PDF_PAGE_SEPARATOR)
|
||||
numbers.append(page.number)
|
||||
# The page as it reaches the text, pointers included: a locator built
|
||||
# from the body alone would drift by two lines per carried image and
|
||||
# would name the wrong page from the first one onwards.
|
||||
offset += len(_pdf_page_text(page)) + len(_PDF_PAGE_SEPARATOR)
|
||||
return SourceUnits("pages", tuple(starts), tuple(numbers))
|
||||
|
||||
|
||||
|
|
@ -1646,7 +2259,13 @@ def _spreadsheet_units(text: str) -> SourceUnits | None:
|
|||
|
||||
|
||||
def source_units(
|
||||
filename: str, data: bytes, text: str, *, pdf_headings: bool = False, ocr: bool = False
|
||||
filename: str,
|
||||
data: bytes,
|
||||
text: str,
|
||||
*,
|
||||
pdf_headings: bool = False,
|
||||
ocr: bool = False,
|
||||
assets: bool = False,
|
||||
) -> SourceUnits | None:
|
||||
"""The unit table for one dropped file, or `None` when it has none.
|
||||
|
||||
|
|
@ -1660,7 +2279,7 @@ def source_units(
|
|||
"""
|
||||
suffix = Path(filename).suffix.lower()
|
||||
if suffix == ".pdf":
|
||||
return _pdf_units(data, pdf_headings, ocr)
|
||||
return _pdf_units(data, pdf_headings, ocr, assets)
|
||||
if suffix == ".xlsx":
|
||||
return _spreadsheet_units(text)
|
||||
if suffix in _CORE_EXTRACTORS or suffix in _PANDOC_FORMATS:
|
||||
|
|
@ -1675,6 +2294,8 @@ def extract_text(
|
|||
renderer: Callable[[str], str] | None = None,
|
||||
pdf_headings: bool = False,
|
||||
ocr: bool = False,
|
||||
assets: bool = False,
|
||||
resolve: Resolver | None = None,
|
||||
) -> str:
|
||||
"""Convert one dropped file's bytes to OKF concept text, dispatched by type.
|
||||
|
||||
|
|
@ -1704,17 +2325,87 @@ def extract_text(
|
|||
passing either argument gets today's behaviour, silently, which is correct
|
||||
-- the options describe a reader, not a policy for the run.
|
||||
"""
|
||||
return extract_document(
|
||||
filename,
|
||||
data,
|
||||
renderer=renderer,
|
||||
pdf_headings=pdf_headings,
|
||||
ocr=ocr,
|
||||
assets=assets,
|
||||
resolve=resolve,
|
||||
).text
|
||||
|
||||
|
||||
#: The types that can carry an image, and the reader that places it. Kept apart
|
||||
#: from `_CORE_EXTRACTORS` and `_OPTIONAL_EXTRACTORS` rather than folded into
|
||||
#: them, and that separation is the byte-identity guarantee: with `assets=False`
|
||||
#: not one entry below is consulted and the dispatch is the one every golden,
|
||||
#: every pinned bundle and every published digest was measured on. `.csv`,
|
||||
#: `.json`, `.md` and `.txt` are absent because the formats carry no image;
|
||||
#: `.xlsx` is absent because its converter writes one pipe table per sheet and
|
||||
#: a two-line block inside one would break the row locator `_spreadsheet_units`
|
||||
#: reads back out of it -- measured 2026-09-16, 0 of 4 K2 workbooks hold any
|
||||
#: media at all, so the row is a limit stated rather than a loss taken.
|
||||
_ASSET_READERS: dict[str, Callable[[bytes, _AssetCollector], str]] = {
|
||||
".html": _extract_html,
|
||||
".htm": _extract_html,
|
||||
".xml": _extract_xml,
|
||||
**{
|
||||
suffix: functools.partial(_extract_office, suffix)
|
||||
for suffix in _PANDOC_FORMATS
|
||||
if suffix != ".xlsx"
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def extract_document(
|
||||
filename: str,
|
||||
data: bytes,
|
||||
*,
|
||||
renderer: Callable[[str], str] | None = None,
|
||||
pdf_headings: bool = False,
|
||||
ocr: bool = False,
|
||||
assets: bool = False,
|
||||
resolve: Resolver | None = None,
|
||||
) -> ExtractedDocument:
|
||||
"""One dropped file as text PLUS the images that stand inside that text.
|
||||
|
||||
The entry point :func:`extract_text` keeps for the eight callers that want
|
||||
a string, and the one Door B uses since 0.10.0. With `assets=False` -- the
|
||||
default, everywhere -- this runs exactly the dispatch that existed before
|
||||
the asset layer did, and returns an :class:`ExtractedDocument` whose text is
|
||||
byte-identical and whose two image tuples are empty.
|
||||
|
||||
`resolve` answers for the formats that POINT at a file instead of embedding
|
||||
it (`html`, `xml`). Without one every pointer resolves to nothing and is
|
||||
stated as such; with one, containment is that resolver's rule and not this
|
||||
module's. `pdf` and the office rows embed their images and never consult it.
|
||||
"""
|
||||
suffix = Path(filename).suffix.lower()
|
||||
extractor = _CORE_EXTRACTORS.get(suffix) or _OPTIONAL_EXTRACTORS.get(suffix)
|
||||
if extractor is not None:
|
||||
if suffix == ".pdf" and (pdf_headings or ocr):
|
||||
text = _extract_pdf(data, headings=pdf_headings, ocr=ocr)
|
||||
if extractor is None:
|
||||
if suffix in _UNPARSED_OPTIONAL_EXTENSIONS:
|
||||
raise _extra_missing(suffix)
|
||||
raise ExtractionError(
|
||||
f"no extractor is registered for file extension {suffix!r} ({filename!r})",
|
||||
code="extractor_unknown",
|
||||
)
|
||||
collector = _AssetCollector(resolve) if assets else None
|
||||
if suffix == ".pdf":
|
||||
if pdf_headings or ocr or assets:
|
||||
text = _extract_pdf(data, headings=pdf_headings, ocr=ocr, assets=assets)
|
||||
else:
|
||||
text = extractor(data)
|
||||
return renderer(text) if renderer is not None else text
|
||||
if suffix in _UNPARSED_OPTIONAL_EXTENSIONS:
|
||||
raise _extra_missing(suffix)
|
||||
raise ExtractionError(
|
||||
f"no extractor is registered for file extension {suffix!r} ({filename!r})",
|
||||
code="extractor_unknown",
|
||||
if collector is not None:
|
||||
for page in _pdf_pages(data, pdf_headings, ocr, True):
|
||||
collector.images.extend(page.images)
|
||||
collector.rejected.extend(page.rejected)
|
||||
elif collector is not None and suffix in _ASSET_READERS:
|
||||
text = _ASSET_READERS[suffix](data, collector)
|
||||
else:
|
||||
text = extractor(data)
|
||||
return ExtractedDocument(
|
||||
text=renderer(text) if renderer is not None else text,
|
||||
images=tuple(collector.images) if collector is not None else (),
|
||||
rejected=tuple(collector.rejected) if collector is not None else (),
|
||||
)
|
||||
|
|
|
|||
|
|
@ -27,11 +27,13 @@ obeys the verdict it returns.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Protocol
|
||||
|
||||
from .assets import ASSETS_DIR, IMAGE_POINTER
|
||||
from .connectors import safe_resolve
|
||||
from .errors import IngestError, MaterializationError, SourceError
|
||||
from .extract import decode_text
|
||||
from .materialize import (
|
||||
|
|
@ -355,6 +357,48 @@ def _read_bundle(
|
|||
return documents, failed
|
||||
|
||||
|
||||
def _carry_assets(source: Path, bundle: Path, merged: Sequence[MergedConcept]) -> None:
|
||||
"""Copy each asset a merged concept points at, by content identity.
|
||||
|
||||
The sender's bytes verbatim, exactly as the concept beside them: this door
|
||||
may not rewrite a merged concept, so it may not rewrite the pointer either,
|
||||
and the file therefore has to land under the name the pointer already
|
||||
names. An occupied name is re-used only when the bytes there are already
|
||||
identical -- Door C's ownership rule, and here the name carries the digest
|
||||
of those bytes, so a mismatch is a `sha256[:12]` collision and is refused
|
||||
rather than resolved.
|
||||
|
||||
A pointer whose asset the sender did not ship is left alone. SPEC SS 6.1
|
||||
requires a consumer to tolerate a broken link, and a pointer recording that
|
||||
the source had a figure nobody holds is information, not corruption.
|
||||
"""
|
||||
for entry in merged:
|
||||
try:
|
||||
text = entry.path.read_text(encoding="utf-8")
|
||||
except OSError:
|
||||
continue
|
||||
for match in IMAGE_POINTER.finditer(text):
|
||||
name = match.group("asset")
|
||||
try:
|
||||
origin = safe_resolve(source / ASSETS_DIR, name)
|
||||
target = safe_resolve(bundle / ASSETS_DIR, name)
|
||||
except SourceError:
|
||||
continue
|
||||
if not origin.is_file():
|
||||
continue
|
||||
data = origin.read_bytes()
|
||||
if target.exists():
|
||||
if target.read_bytes() != data:
|
||||
raise MaterializationError(
|
||||
f"the asset {name!r} already exists here with different bytes; "
|
||||
"refusing to overwrite content this import did not write",
|
||||
code="asset_collision",
|
||||
)
|
||||
continue
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_bytes(data)
|
||||
|
||||
|
||||
def import_bundle(
|
||||
source_dir: Path,
|
||||
bundle_dir: Path,
|
||||
|
|
@ -544,6 +588,19 @@ def import_bundle(
|
|||
MergedConcept(concept_path=concept_path, path=path, reasons=verdict.reasons)
|
||||
)
|
||||
|
||||
# THE ASSETS THE MERGED CONCEPTS POINT AT (0.10.0), read off the files
|
||||
# that actually landed. Measured before this existed: a bundle built
|
||||
# with `--assets` imported as 6 of 6 concepts and NO `assets/`
|
||||
# directory, so every pointer in the imported bundle named a file that
|
||||
# was not there -- the same "complete and not" defect the capability
|
||||
# exists to close, one door over.
|
||||
#
|
||||
# POINTED AT, never "every file in the sender's assets/". An asset
|
||||
# belonging to a concept the gate refused must not ride in on the back
|
||||
# of one it cleared, and an asset nothing names is a file no retirement
|
||||
# pass ever reaches.
|
||||
_carry_assets(source, bundle, merged)
|
||||
|
||||
# §10 pointers, surfaced over what actually landed. Read AFTER the merge
|
||||
# decision and never before it: this door's tolerance is structural —
|
||||
# it writes the sender's bytes verbatim and judges no shape — and a
|
||||
|
|
|
|||
|
|
@ -27,8 +27,23 @@ from collections.abc import Callable, Mapping, Sequence
|
|||
from dataclasses import dataclass, replace
|
||||
from pathlib import Path, PurePosixPath
|
||||
|
||||
from .assets import (
|
||||
ASSETS_DIR,
|
||||
IMAGE_POINTER,
|
||||
AssetRejection,
|
||||
ExtractedImage,
|
||||
asset_name,
|
||||
)
|
||||
from .connectors import safe_resolve
|
||||
from .errors import IngestError, MaterializationError, SegmentationError, SourceError
|
||||
from .extract import DeclaredIdentity, SourceUnits, declared_identity, extract_text, source_units
|
||||
from .extract import (
|
||||
DeclaredIdentity,
|
||||
SourceUnits,
|
||||
declared_identity,
|
||||
directory_resolver,
|
||||
extract_document,
|
||||
source_units,
|
||||
)
|
||||
from .materialize import (
|
||||
_render_root_frontmatter,
|
||||
check_filename_length,
|
||||
|
|
@ -39,6 +54,7 @@ from .materialize import (
|
|||
write_bytes,
|
||||
)
|
||||
from .profiles import (
|
||||
ASSET_COUNT_KEY,
|
||||
DEFAULT,
|
||||
BundleProfile,
|
||||
IndexEntry,
|
||||
|
|
@ -210,6 +226,19 @@ def render_inbox_concept(
|
|||
code="inbox_source_file_invalid",
|
||||
)
|
||||
|
||||
# HOW MANY ASSET POINTERS THIS CONCEPT CARRIES, counted out of the concept's
|
||||
# own text rather than threaded down from the extraction. Two reasons, and
|
||||
# the second is the load-bearing one: a segmented document's images belong
|
||||
# to the segments whose spans hold them, so a document-level total would be
|
||||
# written onto every segment and be wrong on all but one of them; and a
|
||||
# count a reader can verify from the file in front of them is a different
|
||||
# kind of fact from a count only the producer could have known.
|
||||
#
|
||||
# Named on this repository's own profiles only, so a bundle written under
|
||||
# `DEFAULT` or `STRICT_V1` keeps exactly the key set its contract names.
|
||||
assets_carried = (
|
||||
len(IMAGE_POINTER.findall(text)) if ASSET_COUNT_KEY in profile.frontmatter.order else 0
|
||||
)
|
||||
frontmatter = {
|
||||
"type": okf_type,
|
||||
"title": title,
|
||||
|
|
@ -287,6 +316,11 @@ def render_inbox_concept(
|
|||
title=source_title,
|
||||
)
|
||||
)
|
||||
if assets_carried:
|
||||
# Conditional, like `req_number`: absent is "this concept carries no
|
||||
# image", which is what every bundle written before 0.10.0 says, so a
|
||||
# corpus without pictures is byte-identical across the move.
|
||||
frontmatter[ASSET_COUNT_KEY] = str(assets_carried)
|
||||
if concept_frontmatter_values:
|
||||
# LAST, and the position is the precedence: a value the caller states
|
||||
# for the run beats what the document declares, which beats the file
|
||||
|
|
@ -592,6 +626,13 @@ class InboxResult:
|
|||
# consumer's four buckets keep their meaning: a skipped directory holds no
|
||||
# dropped FILE outcome, it explains a set of files that were never dropped.
|
||||
skipped: tuple[SkippedPath, ...] = ()
|
||||
# THE ASSET DENOMINATOR (0.10.0). `assets` is what reached the bundle;
|
||||
# `assets_rejected` is what was found and could not be. Both, or neither
|
||||
# number means anything: "51 carried" is a measurement only beside "of 53
|
||||
# found", and a run whose figures were all refused would otherwise look
|
||||
# exactly like a run over documents that had none.
|
||||
assets: tuple[str, ...] = ()
|
||||
assets_rejected: tuple[AssetRejection, ...] = ()
|
||||
|
||||
|
||||
def relative_source(path: Path, inbox: Path) -> str:
|
||||
|
|
@ -970,6 +1011,7 @@ def process_inbox(
|
|||
pdf_headings: bool = False,
|
||||
heading_reserve: Callable[[str], bool] | None = None,
|
||||
ocr: bool = False,
|
||||
assets: bool = False,
|
||||
concept_frontmatter_values: Mapping[str, str] | None = None,
|
||||
) -> InboxResult:
|
||||
"""Convert every file dropped in `inbox_dir` into an OKF concept.
|
||||
|
|
@ -1037,6 +1079,14 @@ def process_inbox(
|
|||
quarantined: list[BlockedFile] = []
|
||||
rejected: list[BlockedFile] = []
|
||||
failed: list[FailedFile] = []
|
||||
# Keyed by asset name, so one image dropped by two documents is one entry
|
||||
# and the bundle holds one file. The bytes are kept until the write, which
|
||||
# happens per document AFTER that document's gate decision -- an image
|
||||
# belonging to a document the guard refused must not be left behind in
|
||||
# `assets/`, where nothing would ever point at it and nothing would ever
|
||||
# retire it.
|
||||
carried_assets: dict[str, bytes] = {}
|
||||
refused_assets: list[AssetRejection] = []
|
||||
|
||||
# Phase 1: name every file BEFORE any gate call or write, so an intra-run
|
||||
# collision is caught while both files can still be refused together. Under
|
||||
|
|
@ -1184,13 +1234,23 @@ def process_inbox(
|
|||
continue
|
||||
outputs: list[tuple[str, str, tuple[str, ...]]] = []
|
||||
try:
|
||||
text = extract_text(
|
||||
# The resolver is rooted at the DOCUMENT's own directory, which is
|
||||
# the same root `propose.propose_segments` computes from the file it
|
||||
# reads off disk. One root both sides derive independently is what
|
||||
# makes the two renderings identical -- and a plan indexes the exact
|
||||
# string it was proposed against, so a resolver that disagreed would
|
||||
# turn every document carrying a pointer into a coded rejection.
|
||||
resolve = directory_resolver(path.parent) if assets else None
|
||||
document = extract_document(
|
||||
source_name(path),
|
||||
source_bytes,
|
||||
renderer=_resolve_renderer(profile, path.name),
|
||||
pdf_headings=pdf_headings,
|
||||
ocr=ocr,
|
||||
assets=assets,
|
||||
resolve=resolve,
|
||||
)
|
||||
text = document.text
|
||||
# The heading RESERVE, supplied as a predicate rather than decided
|
||||
# here: the condition is the proposer's outline grammar, and the
|
||||
# door does not own that grammar. A callable keeps the dependency
|
||||
|
|
@ -1201,13 +1261,16 @@ def process_inbox(
|
|||
reading_fonts = pdf_headings
|
||||
if heading_reserve is not None and not pdf_headings and heading_reserve(text):
|
||||
reading_fonts = True
|
||||
text = extract_text(
|
||||
document = extract_document(
|
||||
source_name(path),
|
||||
source_bytes,
|
||||
renderer=_resolve_renderer(profile, path.name),
|
||||
pdf_headings=True,
|
||||
ocr=ocr,
|
||||
assets=assets,
|
||||
resolve=resolve,
|
||||
)
|
||||
text = document.text
|
||||
# Computed from the SAME text the plan's offsets index, so the
|
||||
# locator and the offset can never disagree about which rendering
|
||||
# they describe. `None` when the profile names no provenance:
|
||||
|
|
@ -1220,6 +1283,7 @@ def process_inbox(
|
|||
text,
|
||||
pdf_headings=reading_fonts,
|
||||
ocr=ocr,
|
||||
assets=assets,
|
||||
)
|
||||
if profile.provenance is not None
|
||||
else None
|
||||
|
|
@ -1321,6 +1385,12 @@ def process_inbox(
|
|||
continue
|
||||
|
||||
bundle.mkdir(parents=True, exist_ok=True)
|
||||
if outputs and (document.images or document.rejected):
|
||||
# AFTER the gate, and only where the document actually produced
|
||||
# concepts. An asset written for a refused document would be an
|
||||
# orphan no pointer names and no retirement pass reaches.
|
||||
_write_assets(bundle, document.images, carried_assets)
|
||||
refused_assets.extend(document.rejected)
|
||||
for target_name, content, reasons in outputs:
|
||||
# `write_bytes` resolves a subpath through `safe_resolve` but never
|
||||
# creates one. Without this the very first hierarchical write fails.
|
||||
|
|
@ -1377,6 +1447,8 @@ def process_inbox(
|
|||
failed=tuple(sorted(failed, key=lambda entry: entry.source_file)),
|
||||
concepts=tuple(concepts),
|
||||
skipped=skipped,
|
||||
assets=tuple(sorted(carried_assets)),
|
||||
assets_rejected=tuple(refused_assets),
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -1397,6 +1469,46 @@ ADJUDICATION_STATES = (ADJUDICATION_PROPOSED, ADJUDICATION_ADJUDICATED)
|
|||
ADJUDICATION_COMPANION_KEYS = ("adjudicated_by", "adjudicated_at", "adjudication_dwell_s")
|
||||
|
||||
|
||||
def _write_assets(bundle: Path, images: Sequence[ExtractedImage], seen: dict[str, bytes]) -> None:
|
||||
"""Put one document's images in the bundle's `assets/` directory.
|
||||
|
||||
OWNERSHIP IS PROVEN BY CONTENT IDENTITY, which is Door C's rule reused
|
||||
verbatim: an occupied name is re-used only when the bytes there are already
|
||||
identical, and never overwritten otherwise. Here the name carries the
|
||||
digest of those very bytes, so an occupied name with different contents is
|
||||
a `sha256[:12]` collision -- refused loudly rather than resolved silently,
|
||||
because silently resolving it would mean one of two pictures is lost and
|
||||
every pointer to it shows the other.
|
||||
|
||||
A binary write, and the only one in this package. `materialize.write_bytes`
|
||||
takes `content: str` and encodes UTF-8, which is correct for every text
|
||||
guarantee it holds and cannot carry a JPEG.
|
||||
"""
|
||||
directory = bundle / ASSETS_DIR
|
||||
for image in images:
|
||||
name = asset_name(image)
|
||||
known = seen.get(name)
|
||||
if known is not None:
|
||||
if known != image.data:
|
||||
raise MaterializationError(
|
||||
f"two different images reduce to the asset name {name!r} in one run; "
|
||||
"refusing to overwrite the first, because every pointer to it would "
|
||||
"then show the second",
|
||||
code="asset_collision",
|
||||
)
|
||||
continue
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
target = safe_resolve(directory, name)
|
||||
if target.exists() and target.read_bytes() != image.data:
|
||||
raise MaterializationError(
|
||||
f"the asset {name!r} already exists in the bundle with different bytes; "
|
||||
"refusing to overwrite content this run did not write",
|
||||
code="asset_collision",
|
||||
)
|
||||
target.write_bytes(image.data)
|
||||
seen[name] = image.data
|
||||
|
||||
|
||||
def _validate_facets(structure: DocumentStructure, profile: BundleProfile) -> None:
|
||||
"""Refuse a document whose values cannot be rendered as index facets.
|
||||
|
||||
|
|
|
|||
|
|
@ -1414,10 +1414,23 @@ _STRUCTURE_KEYS = (
|
|||
# 2026-08-26 arm that lost on trap exposure was 6 031 characters against
|
||||
# 21 879 for the head-scan it lost to - the headroom for carrying the metadata
|
||||
# back into the index is most of that gap.
|
||||
#: How many binary assets a concept carries (0.10.0). NOT a facet: it is a fact
|
||||
#: about the extraction, not about the document's own structure, and projecting
|
||||
#: it into the index would put a count where a consumer expects a subject.
|
||||
#:
|
||||
#: Named on the profiles this repository owns and on no others. `DEFAULT` states
|
||||
#: commons' ingest-spec SS 5 layer and `STRICT_V1` the wiki's ratified contract,
|
||||
#: so naming it in either from here is this repository editing someone else's
|
||||
#: contract (O2) -- the same reason `sources` sits on the segmented v0.2 profile
|
||||
#: alone. Under those profiles Door B still CARRIES the images and still writes
|
||||
#: the pointer; only the count is absent, which is a smaller loss than a key
|
||||
#: appearing in a contract its owner never agreed to.
|
||||
ASSET_COUNT_KEY = "images"
|
||||
|
||||
STRUCTURED_V1 = BundleProfile(
|
||||
types=DEFAULT.types,
|
||||
frontmatter=FrontmatterSchema(
|
||||
order=(*DEFAULT.frontmatter.order, *_STRUCTURE_KEYS),
|
||||
order=(*DEFAULT.frontmatter.order, *_STRUCTURE_KEYS, ASSET_COUNT_KEY),
|
||||
collapsed_keys=DEFAULT.frontmatter.collapsed_keys,
|
||||
),
|
||||
paths=DEFAULT.paths,
|
||||
|
|
@ -1610,7 +1623,14 @@ assert _SEGMENTED_FACETS is not None
|
|||
# release.
|
||||
SEGMENTED_OKF_V0_2 = BundleProfile(
|
||||
types=OKF_V0_2.types,
|
||||
frontmatter=OKF_V0_2.frontmatter,
|
||||
# OKF_V0_2's schema plus the asset count, and the key is appended HERE for
|
||||
# the same reason `sources` is: `OKF_V0_2` is Door A's profile, where no
|
||||
# dropped binary document is ever read, and widening it would name a key in
|
||||
# a contract that has no producer for it.
|
||||
frontmatter=replace(
|
||||
OKF_V0_2.frontmatter,
|
||||
order=(*OKF_V0_2.frontmatter.order, ASSET_COUNT_KEY),
|
||||
),
|
||||
paths=SEGMENTED_V1.paths,
|
||||
# The facet tuple is EXTENDED here rather than shared, and that is the
|
||||
# discriminator doing its job: `SEGMENTED_V1.index.facets` is one object
|
||||
|
|
|
|||
|
|
@ -62,7 +62,14 @@ from pathlib import Path
|
|||
from typing import Any
|
||||
|
||||
from .errors import IngestError
|
||||
from .extract import OutlineMark, extract_text, strip_converter_attribute, xml_outline
|
||||
from .extract import (
|
||||
OutlineMark,
|
||||
Resolver,
|
||||
directory_resolver,
|
||||
extract_text,
|
||||
strip_converter_attribute,
|
||||
xml_outline,
|
||||
)
|
||||
from .extract import pdf_outline as extract_pdf_outline
|
||||
from .materialize import reduce_to_id_grammar
|
||||
from .segmentation import heading_only, observed_extractor_version
|
||||
|
|
@ -1613,6 +1620,8 @@ def build_plan(
|
|||
bold_title: bool = False,
|
||||
outline_marks: Sequence[OutlineMark] | None = None,
|
||||
shell_parent: bool = False,
|
||||
assets: bool = False,
|
||||
resolve: Resolver | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
|
||||
taken: set[str] = set()
|
||||
|
|
@ -1626,7 +1635,12 @@ def build_plan(
|
|||
# is the document.
|
||||
outline_rule = RULE_PDF_OUTLINE
|
||||
if extractor_id in DECLARED_STRUCTURE_IDS:
|
||||
outline_marks = xml_outline(source.name, source_bytes)
|
||||
# The SAME rendering the text was extracted under. A mark is a line
|
||||
# index into that exact string, and carrying an image inserts lines:
|
||||
# marks read with the images off would name the right sections at the
|
||||
# wrong line numbers, silently, on every document with a figure above a
|
||||
# heading.
|
||||
outline_marks = xml_outline(source.name, source_bytes, assets=assets, resolve=resolve)
|
||||
outline_rule = RULE_XML_SECTION
|
||||
entries: list[dict[str, Any]] = []
|
||||
candidates = find_candidates(
|
||||
|
|
@ -1755,6 +1769,7 @@ def run(
|
|||
ocr: bool = False,
|
||||
pdf_outline: bool = False,
|
||||
shell_parent: bool = False,
|
||||
assets: bool = False,
|
||||
) -> int:
|
||||
if max_segment_chars < 0:
|
||||
raise ProposerError(
|
||||
|
|
@ -1798,7 +1813,19 @@ def run(
|
|||
# exact string -- a plan proposed against one rendering and replayed
|
||||
# against another is refused by `assert_plan_applies`, which is the
|
||||
# right outcome and a confusing one to debug.
|
||||
text = extract_text(source.name, source_bytes, pdf_headings=pdf_headings, ocr=ocr)
|
||||
# Rooted at the document's own directory, which is what Door B derives
|
||||
# for the same file. The two sides never exchange the root; they compute
|
||||
# it, so a plan and the run that replays it read the same bytes for
|
||||
# every `<img src>` and every `<graphic xlink:href>`.
|
||||
resolve = directory_resolver(source.parent) if assets else None
|
||||
text = extract_text(
|
||||
source.name,
|
||||
source_bytes,
|
||||
pdf_headings=pdf_headings,
|
||||
ocr=ocr,
|
||||
assets=assets,
|
||||
resolve=resolve,
|
||||
)
|
||||
reading_fonts = pdf_headings
|
||||
# The reserve, and the reason it re-extracts rather than post-processes:
|
||||
# the font reader works on the PDF's glyph geometry, which the joined
|
||||
|
|
@ -1807,7 +1834,14 @@ def run(
|
|||
# never a pair to combine.
|
||||
if pdf_headings_reserve and not pdf_headings:
|
||||
if heading_reserve_applies(text, outline_run=outline_run):
|
||||
text = extract_text(source.name, source_bytes, pdf_headings=True, ocr=ocr)
|
||||
text = extract_text(
|
||||
source.name,
|
||||
source_bytes,
|
||||
pdf_headings=True,
|
||||
ocr=ocr,
|
||||
assets=assets,
|
||||
resolve=resolve,
|
||||
)
|
||||
reading_fonts = True
|
||||
# LAST, and against the text that is final: a plan indexes one exact
|
||||
# string, so marks bridged onto the pre-reserve rendering would name
|
||||
|
|
@ -1815,7 +1849,7 @@ def run(
|
|||
# reserve may have changed, and the marks follow it.
|
||||
marks = (
|
||||
extract_pdf_outline(
|
||||
source.name, source_bytes, pdf_headings=reading_fonts, ocr=ocr
|
||||
source.name, source_bytes, pdf_headings=reading_fonts, ocr=ocr, assets=assets
|
||||
).marks
|
||||
if pdf_outline
|
||||
else ()
|
||||
|
|
@ -1844,6 +1878,8 @@ def run(
|
|||
bold_title=bold_title,
|
||||
outline_marks=marks,
|
||||
shell_parent=shell_parent,
|
||||
assets=assets,
|
||||
resolve=resolve,
|
||||
)
|
||||
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
|
||||
# plan cannot be replayed -- `process_inbox` refuses one, because a plan
|
||||
|
|
|
|||
|
|
@ -235,6 +235,17 @@ def _scan_references(body: str, offset: int, own_number: str | None) -> tuple[st
|
|||
start, end = match.span(1)
|
||||
for position in range(start, end):
|
||||
masked[position] = " "
|
||||
# AN IMAGE IS NOT A CROSS-REFERENCE. `` embeds a resource
|
||||
# where a link points at a subject, and `_LINK` cannot tell them apart
|
||||
# because it never looks at the character in front of the bracket. Left
|
||||
# unhandled, every asset pointer written since 0.10.0 would arrive in
|
||||
# the index as an edge to a concept that cannot exist -- the same shape
|
||||
# as K3-21's `Enclosing section:` line becoming a second, unresolved
|
||||
# edge. Masked and then skipped, never merely skipped: the file name
|
||||
# carries digits (`tabell-84-2.png`) that the number scan below would
|
||||
# otherwise read as a document this one refers to.
|
||||
if match.start() > 0 and body[match.start() - 1] == "!":
|
||||
continue
|
||||
# A fragment-only target points inside THIS document, and a target
|
||||
# carrying a brace is a template placeholder from prose ABOUT links
|
||||
# (`reduce_to_id_grammar` cannot emit a brace). Neither can ever
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue