open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(assets): a bundle carries the images its sources declare (0.10.0)

Browse source 
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-17 10:01:31 +02:00
commit bc39e8091f
33 changed files with 3638 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

86
src/llm_ingestion_okf/corpus.py
View file

@ -40,6 +40,7 @@ from collections.abc import Callable, Mapping
from dataclasses import dataclass, replace
from pathlib import Path
from .assets import ASSETS_DIR
from .errors import IngestError
from .extract import extract_text
from .inbox import (
@ -87,6 +88,33 @@ LOG_TITLE = "Corpus run history"
#: numbers with it and a caller measuring segmentation alone has a legitimate
#: reason to take the gate out of the picture -- but reaching it is now an act,
#: and the bundle's own log says so.
#: Whether a build carries the images its sources declare (0.10.0).
#:
#: ON, and the reason is the premise the capability was ordered on: what ends
#: up in a bundle after converting N sources has to BE the sources. Measured on
#: R761 Prosesskoden:2025 -- 2 726 of 2 726 processes carry their text and 12
#: `Tabell N-N` and 9 `Figur N-N` captions stand over nothing, because the
#: publisher ships those tables as raster pictures in both the PDF and the
#: NISO-STS delivery. A bundle that reads as complete and answers "toleranse-
#: klasse ... er gitt i tabell 84-2" with an empty space is the defect; it is
#: not a preference about bundle size.
#:
#: WHAT THIS COSTS IS PUBLISHED RATHER THAN IMPLIED. Measured over the K2
#: reference corpus, 2026-09-16: 32 of its 33 PDFs carry image objects and the
#: population is 4 828, of which 2 931 are 512x512 pixels or larger, 1 307 are
#: smaller and 149 declare no size at all. A drawing set contributes 2 524 on
#: its own. So this is a large change for a drawing-heavy corpus and none at
#: all for a prose one, and `--no-assets` reproduces the pre-0.10.0 bytes
#: exactly -- measured by `diff`, not asserted.
#:
#: NO SIZE FLOOR, and that is a measurement too. The obvious filter would be
#: "ignore anything under N pixels", and the distribution above does not offer
#: an N: it is a broad spread with no gap, unlike `OCR_CID_SHARE`'s, which is
#: bimodal with nothing between the modes. A threshold read off no gap is a
#: number this package chose, and it would silently drop somebody's small
#: table. Stated as a limit instead of guessed at.
DEFAULT_ASSETS = True
GATE_NONE = "none"
#: `okf build`'s default, set by the operator 2026-09-15 on a measurement:
#: over the 453 concept bodies of the pinned default bundle, this tier returns
@ -200,6 +228,14 @@ class CorpusReport:
#: stub so every existing construction site stays source-compatible; the
#: two commands that build bundles both pass it explicitly.
gate: str = GATE_NONE
#: THE ASSET DENOMINATOR (0.10.0). `assets_carried` is what reached the
#: bundle, `assets_found` is what the readers saw. Both, always: "51
#: carried" is a measurement only beside "of 53 found", and a run whose
#: figures were all refused must not read like a run over documents that had
#: none. `assets` says whether they were looked for at all.
assets: bool = False
assets_carried: int = 0
assets_found: int = 0
@property
def merged(self) -> int:
@ -291,11 +327,37 @@ class CorpusReport:
f"* **Gate**: {self.gate} "
f"({_GATE_DESCRIPTIONS.get(self.gate, 'unrecognised gate name')}). "
"Every persisted byte of this bundle passed it.",
self._assets_line(),
]
if self.unaccounted:
lines.append("* **Unaccounted**: " + ", ".join(self.unaccounted) + " — K1b FAILED.")
return "\n".join(lines) + "\n"
def _assets_line(self) -> str:
"""What this run did about the images in its sources, stated either way.
An absent bullet would make "this corpus had no figures" and "this run
never looked" the same artifact, which is the trust trap `--gate none`
already has its own sentence for. So the OFF case is written out in
capitals rather than left to an inference.
THE IMAGE BYTES ARE NOT SCREENED, and the log is where a consumer finds
that out. The guard is text-only, so the pointer block passed the gate
as body text and the picture beside it did not pass anything.
"""
if not self.assets:
return (
"* **Images**: NOT CARRIED — this run did not look for images, so a "
"document whose table is a picture reached the bundle as text with a "
"gap in it. Absence of an image here is not evidence the sources had none."
)
return (
f"* **Images**: {self.assets_carried} carried of {self.assets_found} found, "
f"written to `{ASSETS_DIR}/` and pointed at from the concepts they stand in. "
"The image BYTES were not screened: the gate above reads text, and a "
"picture is not text."
)
def load_plans(plans_dir: Path) -> dict[str, SegmentationPlan]:
"""Every proposal artifact in a directory, keyed by filename.
@ -357,6 +419,7 @@ def measure(
ocr: bool = False,
concept_frontmatter_values: Mapping[str, str] | None = None,
gate: str = GATE_NONE,
assets: bool = False,
) -> CorpusReport:
"""Run the corpus through the door and count what happened.
@ -381,6 +444,7 @@ def measure(
pdf_headings=pdf_headings,
heading_reserve=heading_reserve,
ocr=ocr,
assets=assets,
concept_frontmatter_values=concept_frontmatter_values,
)
elapsed = time.monotonic() - started
@ -416,6 +480,9 @@ def measure(
codes=tuple(sorted(counts.items())),
unaccounted=unaccounted_names(dropped=dropped, merged=merged_names, coded=coded_names),
gate=gate,
assets=assets,
assets_carried=len(result.assets),
assets_found=len(result.assets) + len(result.assets_rejected),
)
@ -456,6 +523,24 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"would make that equality depend on which command you ran"
),
)
parser.add_argument(
"--assets",
action="store_true",
default=DEFAULT_ASSETS,
help=(
"carry the images the sources declare into the bundle. The same "
"name and the same default as `okf build --assets`, for the same "
"reason the gate carries them: the two paths are held byte-equal "
"by a test, and two different defaults would make that equality "
"depend on which command you ran"
),
)
parser.add_argument(
"--no-assets",
action="store_false",
dest="assets",
help="the explicit opt-out, reproducing the pre-0.10.0 bytes",
)
parser.add_argument(
"--okf-version",
default=None,
@ -532,6 +617,7 @@ def main(argv: list[str] | None = None) -> int:
report = measure(
args.corpus,
bundle,
assets=args.assets,
ingested_at=args.ingested_at,
plans=plans,
profile=profile,