feat(propose): a section the source DECLARES takes the declared-structure route

`extract.xml_outline` reports the marks the XML reader wrote itself -- no
bridge, no tolerance constant, no unresolved bucket, because the reader
appended the line it is naming. `find_candidates` gains `outline_rule` so the
route it already had for a PDF bookmark tree can carry a second reader's name,
and `build_plan` chooses it by the ROW (`DECLARED_STRUCTURE_IDS`), never by the
text: the same markdown from a `.md` file is still a guess and still keeps
`rule:heading`.

`rule:xml-section` is orphan-exempt for the reason the bookmark arm is -- the
check asks whether a guess was a heading, and a container section is not a
false positive.

MEASURED on R761 at SHIPPED DEFAULTS, no flag: 23 concepts -> 2761, and
2761 of 2761 declared sections became a concept with the directory and the
title the source states (0 unmatched, 0 concepts matching no declaration).
`a)`-points as concepts 0 of 4954, table blocks 10 of 10.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 07:00:37 +02:00
commit ee12f644ca
2 changed files with 103 additions and 11 deletions

View file

@ -62,7 +62,7 @@ from pathlib import Path
from typing import Any
from .errors import IngestError
from .extract import OutlineMark, extract_text, strip_converter_attribute
from .extract import OutlineMark, extract_text, strip_converter_attribute, xml_outline
from .extract import pdf_outline as extract_pdf_outline
from .materialize import reduce_to_id_grammar
from .segmentation import observed_extractor_version
@ -142,6 +142,26 @@ RULE_BOLD_TITLE = "rule:bold-title"
#: a structure index the file already carries. A reader who cannot tell the two
#: apart in an artifact cannot tell a recovered heading from a declared one.
RULE_PDF_OUTLINE = "rule:pdf-outline"
#: A section the SOURCE FORMAT declares as an element, read by a reader in this
#: package. NOT `RULE_PDF_OUTLINE`: that one is a publisher's bookmark tree
#: BRIDGED from (page, y) onto a line, and its arm reports what did not bridge.
#: NOT `RULE_HEADING` either, and that distinction is the one this rule exists
#: for -- an ATX line reaching the proposer says nothing about who wrote it, so
#: a `<sec><title>` and a heading a converter guessed out of a font size were
#: indistinguishable in the artifact and were judged by the same two steps.
#: Measured on R761: the orphan check removed 710 of 2 761 declared sections
#: (710 of 710 removed are followed immediately by another heading -- they are
#: containers) and Arm F folded 2 066 more, 2 089 -> 23 at shipped defaults.
RULE_XML_SECTION = "rule:xml-section"
#: The extractor ids whose reader WRITES the heading line from an element the
#: source declared, so the structure is transcribed rather than recovered. One
#: row, and it is deliberately not "every type whose headings look declared":
#: an office document's headings arrive through an external converter's
#: rendering decisions, and moving that row is a measurement over the whole
#: unit worksheet, which this rule did not make.
DECLARED_STRUCTURE_IDS = frozenset({"xml"})
RULE_NAMES = (
RULE_HEADING,
RULE_TABLE_BLOCK,
@ -152,6 +172,7 @@ RULE_NAMES = (
RULE_SHEET_SECTION,
RULE_BOLD_TITLE,
RULE_PDF_OUTLINE,
RULE_XML_SECTION,
)
#: How many characters of context each side of a quote anchor carries. Enough
@ -606,10 +627,12 @@ def _sheet_section_rows(lines: list[str]) -> dict[int, tuple[str, str]]:
#: Rules the orphan check is not asked about. D3 because a sheet row carries
#: its content in its own cells, and the bookmark arm because the check judges
#: whether a GUESS was a heading -- a question a publisher's own tree has
#: already answered, and one that deletes every container section if asked.
_ORPHAN_EXEMPT = (RULE_SHEET_SECTION, RULE_PDF_OUTLINE)
#: its content in its own cells, and the two DECLARED-structure rules because
#: the check judges whether a GUESS was a heading -- a question a publisher's
#: own tree has already answered, and one that deletes every container section
#: if asked. Measured on one 701-page process code: 683 of 2 762 bookmark nodes
#: and 710 of 2 761 STS sections are containers.
_ORPHAN_EXEMPT = (RULE_SHEET_SECTION, RULE_PDF_OUTLINE, RULE_XML_SECTION)
def _split_outline_title(title: str) -> tuple[str | None, str]:
@ -648,6 +671,7 @@ def find_candidates(
contents_name: bool = False,
bold_title: bool = False,
outline_marks: Sequence[OutlineMark] | None = None,
outline_rule: str = RULE_PDF_OUTLINE,
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
@ -719,8 +743,12 @@ def find_candidates(
before spans are closed, so the text they opened is carried by the mark
above rather than lost.
`outline_marks` is the PDF bookmark arm, and it is the only input here that
REPLACES the rules rather than gating one of them. A non-empty list is the
`outline_marks` is the DECLARED-STRUCTURE route, and it is the only input
here that REPLACES the rules rather than gating one of them. Two readers
reach it -- `extract.pdf_outline` for a `/Outlines` tree and
`extract.xml_outline` for NISO-STS `<sec><title>` -- and `outline_rule`
names which, because an artifact that cannot tell a bridged bookmark from
an element the reader wrote itself cannot be audited. A non-empty list is the
publisher's own declaration of the document's structure, so nothing below
votes against it: the text heuristics, the two gates and Arm F's fold are
all skipped, and the orphan check is not applied to its marks. An EMPTY
@ -782,7 +810,7 @@ def find_candidates(
title=title,
level=mark.level,
number=number,
rule=RULE_PDF_OUTLINE,
rule=outline_rule,
start=offsets[mark.line],
end=end_of_text,
),
@ -1458,6 +1486,17 @@ def build_plan(
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
extractor_id = source.suffix.lower().lstrip(".") or "none"
# The declared-structure route, chosen by the ROW and never by the text.
# That is what keeps every other file type byte-identical: the same
# markdown arriving from a `.md` file carries a heading somebody guessed,
# and the reader that wrote a `<sec><title>` is the only witness that the
# source declared it. `--pdf-outline` stays a flag because a bookmark tree
# is a publisher's CLAIM about a document it also typeset; an STS element
# is the document.
outline_rule = RULE_PDF_OUTLINE
if extractor_id in DECLARED_STRUCTURE_IDS:
outline_marks = xml_outline(source.name, source_bytes)
outline_rule = RULE_XML_SECTION
entries: list[dict[str, Any]] = []
candidates = find_candidates(
text,
@ -1473,6 +1512,7 @@ def build_plan(
contents_name=contents_name,
bold_title=bold_title,
outline_marks=outline_marks,
outline_rule=outline_rule,
)
for candidate in subdivide(text, candidates, max_segment_chars):
entries.append(