feat(inbox): point every concept at the document it came from, with a locator per format
A concept named its source file by basename and, when segmented, carried a
`source_offset` into the text THIS LIBRARY extracted. Following that pointer
needed the corpus directory, the extractor and its exact transitive version --
none of which the bundle carries. Hand-walked on a real K2 concept: six steps,
four of them requiring knowledge from outside the bundle, to learn that a
requirement sits on pages 12-13 of a 20-page document.
The address is spec's: `sources: [{ resource, title }]`, where `resource` is
the dropped file's inbox-relative path (SPEC v0.2 5.1:303-306 -- "an absolute
URL, a bundle-relative path, or a path into a `references/` subdirectory").
The locator is ours, and it has to be: 5.1 has no field for a place within a
resource, and the pinned guard (1.3.0) rejects every route to putting one
inside a `sources` entry -- a non-allowlisted key by name, a nested flow list
as "scalar leaves only", and quoting as an unsupported form. So the locator is
top-level keys shaped like `source_offset`, and a path carrying a flow
terminator is refused fail-fast rather than mangled.
The unit table is built AT EXTRACTION, where the extracted text and the
original's structure are known to agree: pdf -> `source_pages` from
pdfplumber's own page numbers (a page that yielded no text does not renumber
the ones after it), xlsx -> `source_sheet` + `source_rows`, everything else ->
`source_lines`. `source_offset` stays.
Two measurements changed the design before it shipped. A `paragraphs` key for
docx would name a number the document does not have: `<w:p>` counts of
108/27/65/176/57 against converted-markdown lines of 75/33/67/144/63, not one
pair agreeing -- so the key is `source_lines` and says what it indexes. And an
empty spreadsheet row renders exactly like a table separator: the content-based
rule ate 8 empty rows on the K2 price sheet and reported its last row as 92
against a workbook that says 100. The separator is now found by position, and
`tomrad.xlsx` keeps that red.
One profile moves. `provenance` is a policy object, `None` everywhere but
`SEGMENTED_OKF_V0_2`; the other five shipped profiles are byte-identical.
K2 rebuilt from a frozen src copy: 629 concepts, 1108 files, name set identical,
0 ids moved, 479 files byte-identical, 629 changed and 0 lines removed anywhere.
629/629 now carry an address and a locator. New ref
`sha256-tree:665563a2f74423fcbcc8e4f0b0954ee73b73985ac0418de4f6987bd162a1f7c8`;
`2f82fcfe...` is stale. The pre-pass payload does not grow by one byte
(209 092 B before and after, 18 changed lines: the ref and eight per-concept
digests) -- because an excerpt carries the body, not the frontmatter, which is
also why the consumer still cannot cite "file X page 12" from a payload alone.
Report: docs/2026-09-08-proveniens-k2.md. 1339 tests, ruff and mypy clean.
Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
parent
d3bfe92acd
commit
b6a8c8bd89
16 changed files with 1301 additions and 26 deletions
|
|
@ -32,6 +32,7 @@ import warnings
|
|||
import zipfile
|
||||
from xml.etree import ElementTree
|
||||
from collections.abc import Callable, Sequence
|
||||
from dataclasses import dataclass
|
||||
from html.parser import HTMLParser
|
||||
from pathlib import Path
|
||||
|
||||
|
|
@ -236,6 +237,45 @@ def _extra_missing(suffix: str) -> ExtractionError:
|
|||
)
|
||||
|
||||
|
||||
# How `_extract_pdf` joins its pages, named because the locator below has to
|
||||
# reproduce the exact same arithmetic to turn a character offset back into a
|
||||
# page number. Two constants that must agree, written once.
|
||||
_PDF_PAGE_SEPARATOR = "\n\n"
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize=1)
|
||||
def _pdf_pages(data: bytes) -> tuple[tuple[int, str], ...]:
|
||||
"""Every page that produced text, as `(page number, text)`, in page order.
|
||||
|
||||
The page NUMBER is 1-based and comes from the document, so a page that
|
||||
yielded nothing removes itself from the sequence without renumbering the
|
||||
ones after it -- which is the difference between "the third page that
|
||||
produced text" and "page 3", and the whole reason a locator is worth
|
||||
writing down.
|
||||
|
||||
Memoised on the bytes with room for exactly one document: extraction and
|
||||
location are two calls about the same file, back to back, and parsing it
|
||||
twice would double the PDF cost of every corpus run for nothing. Anything
|
||||
larger would hold whole documents in memory for no gain, since the caller
|
||||
never returns to an earlier file.
|
||||
"""
|
||||
try:
|
||||
import pdfplumber
|
||||
except ImportError as exc:
|
||||
raise _extra_missing(".pdf") from exc
|
||||
|
||||
try:
|
||||
with pdfplumber.open(io.BytesIO(data)) as pdf:
|
||||
pages = [(page.extract_text() or "").rstrip() for page in pdf.pages]
|
||||
except ExtractionError:
|
||||
raise
|
||||
except Exception as exc: # noqa: BLE001 - third-party parser, wrapped never leaked
|
||||
raise ExtractionError(
|
||||
f"the PDF parser failed on this file: {exc}", code="extractor_pdf_error"
|
||||
) from exc
|
||||
return tuple((number, page) for number, page in enumerate(pages, start=1) if page)
|
||||
|
||||
|
||||
def _extract_pdf(data: bytes) -> str:
|
||||
"""`pdf`: page text via `pdfplumber`, in page order, pages separated by a
|
||||
blank line.
|
||||
|
|
@ -252,22 +292,8 @@ def _extract_pdf(data: bytes) -> str:
|
|||
and pymupdf each emit all labels then all values. Re-pairing those is
|
||||
guesswork, and in a requirements document a wrong pairing looks right.
|
||||
"""
|
||||
try:
|
||||
import pdfplumber
|
||||
except ImportError as exc:
|
||||
raise _extra_missing(".pdf") from exc
|
||||
|
||||
try:
|
||||
with pdfplumber.open(io.BytesIO(data)) as pdf:
|
||||
pages = [(page.extract_text() or "").rstrip() for page in pdf.pages]
|
||||
except ExtractionError:
|
||||
raise
|
||||
except Exception as exc: # noqa: BLE001 - third-party parser, wrapped never leaked
|
||||
raise ExtractionError(
|
||||
f"the PDF parser failed on this file: {exc}", code="extractor_pdf_error"
|
||||
) from exc
|
||||
|
||||
text = "\n\n".join(page for page in pages if page)
|
||||
pages = _pdf_pages(data)
|
||||
text = _PDF_PAGE_SEPARATOR.join(page for _, page in pages)
|
||||
if not text:
|
||||
raise ExtractionError(
|
||||
"the PDF yielded no text on any page; a scanned or image-only "
|
||||
|
|
@ -420,6 +446,195 @@ _OPTIONAL_EXTRACTORS: dict[str, Callable[[bytes], str]] = {
|
|||
}
|
||||
|
||||
|
||||
# --- provenance: a character range of the extracted text -> a place in the
|
||||
# original document ---------------------------------------------------------
|
||||
#
|
||||
# `source_offset` alone is a position in OUR extraction, so following it back
|
||||
# needs the corpus directory, the extractor and its exact version -- none of
|
||||
# which a bundle carries. A unit table is that mapping, saved AT EXTRACTION
|
||||
# where the two are known to agree, rather than guessed afterwards from text
|
||||
# whose page breaks are gone.
|
||||
#
|
||||
# THE UNIT IS PER FORMAT AND IS NAMED, never assumed:
|
||||
#
|
||||
# pages a PDF page number, from the document itself.
|
||||
# rows a spreadsheet row, within the sheet named by `scope_of`.
|
||||
# lines a line of the EXTRACTED text. For `md`/`txt` that text is the
|
||||
# dropped file, so the number is the original's own line; for the
|
||||
# converted formats it is not, and the key says `lines` rather than
|
||||
# `paragraphs` for exactly that reason. Measured on the five K2
|
||||
# `.docx` documents: `<w:p>` counts 108/27/65/176/57 against
|
||||
# converted-markdown line counts 75/33/67/144/63 -- not one pair
|
||||
# agrees, so a `paragraphs` key would name a number the original does
|
||||
# not have.
|
||||
#
|
||||
# The heading a spreadsheet's sheet becomes, as the converter writes it:
|
||||
# `## <sheet name> {#sheet-<n>}`. Anchored to the line start so a pipe cell
|
||||
# containing a `#` cannot be read as a sheet.
|
||||
_SHEET_HEADING = re.compile(r"^#{1,6} (?P<name>.*?) \{#sheet-\d+\}$")
|
||||
|
||||
# A line the converter wrote as part of a pipe table. Whether one of them is
|
||||
# the table's SEPARATOR is decided by POSITION, never by content: an empty
|
||||
# spreadsheet row renders as `| | |` and a separator as `|----|----|`, and
|
||||
# every content rule that tells those apart also swallows a data row that
|
||||
# happens to hold only dashes. Measured on the K2 price sheet: a content rule
|
||||
# ate 8 empty rows and reported the sheet's last row as 92 against a workbook
|
||||
# that says 100.
|
||||
_TABLE_LINE = "|"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SourceUnits:
|
||||
"""Where in the ORIGINAL each stretch of the extracted text came from.
|
||||
|
||||
`starts[i]` is the character offset in the extracted text at which unit
|
||||
`numbers[i]` begins, and `scopes[i]` is the sheet that unit belongs to (or
|
||||
`None` for a format that has no sheets). The three tuples are parallel and
|
||||
`starts` ascends, which is what lets `covering` be a bisection rather than
|
||||
a scan.
|
||||
|
||||
`numbers` is separate from the index on purpose. A PDF page that yielded no
|
||||
text is not in this table, and a pipe table's separator line is a row of
|
||||
nothing -- in both cases the position in the table and the number in the
|
||||
original have already parted company, and an index standing in for a number
|
||||
is the off-by-one this whole object exists to prevent.
|
||||
"""
|
||||
|
||||
unit: str
|
||||
starts: tuple[int, ...]
|
||||
numbers: tuple[int, ...]
|
||||
scopes: tuple[str | None, ...] = ()
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if len(self.starts) != len(self.numbers):
|
||||
raise ValueError("a unit table needs one number per start offset")
|
||||
if self.scopes and len(self.scopes) != len(self.starts):
|
||||
raise ValueError("a unit table needs one scope per start offset, or none at all")
|
||||
|
||||
def _index(self, offset: int) -> int:
|
||||
"""The table row covering `offset`, clamped to the table's own ends."""
|
||||
low, high = 0, len(self.starts) - 1
|
||||
while low < high:
|
||||
middle = (low + high + 1) // 2
|
||||
if self.starts[middle] <= offset:
|
||||
low = middle
|
||||
else:
|
||||
high = middle - 1
|
||||
return low
|
||||
|
||||
def covering(self, start: int, end: int) -> tuple[int, int]:
|
||||
"""The first and last original unit the half-open `[start, end)` touches.
|
||||
|
||||
`end` is exclusive, so a range ending exactly where the next unit
|
||||
begins does not claim that unit -- a segment that stops at a page
|
||||
boundary is on the page it was written on.
|
||||
"""
|
||||
if not self.starts:
|
||||
raise ValueError("an empty unit table locates nothing")
|
||||
first = self._index(start)
|
||||
last = self._index(max(start, end - 1))
|
||||
return self.numbers[first], self.numbers[last]
|
||||
|
||||
def scope_of(self, offset: int) -> str | None:
|
||||
"""The sheet `offset` falls in, or `None` for a format without sheets."""
|
||||
if not self.scopes:
|
||||
return None
|
||||
return self.scopes[self._index(offset)]
|
||||
|
||||
def scopes_covering(self, start: int, end: int) -> tuple[str | None, ...]:
|
||||
"""Every distinct scope the range touches, in order, without repeats."""
|
||||
if not self.scopes:
|
||||
return ()
|
||||
first = self._index(start)
|
||||
last = self._index(max(start, end - 1))
|
||||
seen: list[str | None] = []
|
||||
for scope in self.scopes[first : last + 1]:
|
||||
if not seen or seen[-1] != scope:
|
||||
seen.append(scope)
|
||||
return tuple(seen)
|
||||
|
||||
|
||||
def _line_units(text: str) -> SourceUnits:
|
||||
starts: list[int] = []
|
||||
offset = 0
|
||||
for line in text.split("\n"):
|
||||
starts.append(offset)
|
||||
offset += len(line) + 1
|
||||
return SourceUnits("lines", tuple(starts), tuple(range(1, len(starts) + 1)))
|
||||
|
||||
|
||||
def _pdf_units(data: bytes) -> SourceUnits:
|
||||
starts: list[int] = []
|
||||
numbers: list[int] = []
|
||||
offset = 0
|
||||
for number, page in _pdf_pages(data):
|
||||
starts.append(offset)
|
||||
numbers.append(number)
|
||||
offset += len(page) + len(_PDF_PAGE_SEPARATOR)
|
||||
return SourceUnits("pages", tuple(starts), tuple(numbers))
|
||||
|
||||
|
||||
def _spreadsheet_units(text: str) -> SourceUnits | None:
|
||||
"""Sheet and row for a converted spreadsheet, or `None` if it is not one.
|
||||
|
||||
The converter writes one heading per sheet and then one pipe-table line per
|
||||
source row, with a separator line after the first. Row numbering therefore
|
||||
restarts at every heading and skips that one line by POSITION.
|
||||
|
||||
The row number is the ORIGINAL sheet's, and that holds exactly as far as
|
||||
one converted line per `<row>` element holds. Measured on the two K2
|
||||
spreadsheets and both fixtures: 39 rows for 39, 100 for 100, 4 for 4, 6 for
|
||||
6 and 3 for 3 -- every one contiguous from row 1. A sheet whose XML omits a
|
||||
row entirely would number from the converted table instead, and nothing
|
||||
here can see that.
|
||||
"""
|
||||
starts: list[int] = []
|
||||
numbers: list[int] = []
|
||||
scopes: list[str | None] = []
|
||||
sheet: str | None = None
|
||||
seen = 0
|
||||
offset = 0
|
||||
for line in text.split("\n"):
|
||||
heading = _SHEET_HEADING.match(line)
|
||||
if heading is not None:
|
||||
sheet = heading.group("name")
|
||||
seen = 0
|
||||
elif sheet is not None and line.startswith(_TABLE_LINE):
|
||||
seen += 1
|
||||
# The SECOND table line of a sheet is the separator the converter
|
||||
# writes under the header, and it is a row of no spreadsheet. Every
|
||||
# line after it is one row further on than its position suggests.
|
||||
if seen != 2:
|
||||
starts.append(offset)
|
||||
numbers.append(seen if seen == 1 else seen - 1)
|
||||
scopes.append(sheet)
|
||||
offset += len(line) + 1
|
||||
if not starts:
|
||||
return None
|
||||
return SourceUnits("rows", tuple(starts), tuple(numbers), tuple(scopes))
|
||||
|
||||
|
||||
def source_units(filename: str, data: bytes, text: str) -> SourceUnits | None:
|
||||
"""The unit table for one dropped file, or `None` when it has none.
|
||||
|
||||
`text` must be what `extract_text` returned for these exact bytes: the
|
||||
table indexes that string, and a table built against a different rendering
|
||||
would point a consumer at the wrong place with full confidence.
|
||||
|
||||
`None` is a measurement, not a failure -- a spreadsheet the converter wrote
|
||||
no table for has no rows to name, and the caller writes the address without
|
||||
a locator rather than inventing one.
|
||||
"""
|
||||
suffix = Path(filename).suffix.lower()
|
||||
if suffix == ".pdf":
|
||||
return _pdf_units(data)
|
||||
if suffix == ".xlsx":
|
||||
return _spreadsheet_units(text)
|
||||
if suffix in _CORE_EXTRACTORS or suffix in _PANDOC_FORMATS:
|
||||
return _line_units(text)
|
||||
return None
|
||||
|
||||
|
||||
def extract_text(
|
||||
filename: str, data: bytes, *, renderer: Callable[[str], str] | None = None
|
||||
) -> str:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue