feat(propose): a section the source DECLARES takes the declared-structure route
`extract.xml_outline` reports the marks the XML reader wrote itself -- no bridge, no tolerance constant, no unresolved bucket, because the reader appended the line it is naming. `find_candidates` gains `outline_rule` so the route it already had for a PDF bookmark tree can carry a second reader's name, and `build_plan` chooses it by the ROW (`DECLARED_STRUCTURE_IDS`), never by the text: the same markdown from a `.md` file is still a guess and still keeps `rule:heading`. `rule:xml-section` is orphan-exempt for the reason the bookmark arm is -- the check asks whether a guess was a heading, and a container section is not a false positive. MEASURED on R761 at SHIPPED DEFAULTS, no flag: 23 concepts -> 2761, and 2761 of 2761 declared sections became a concept with the directory and the title the source states (0 unmatched, 0 concepts matching no declaration). `a)`-points as concepts 0 of 4954, table blocks 10 of 10. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
958e9bc998
commit
ee12f644ca
2 changed files with 103 additions and 11 deletions
|
|
@ -108,6 +108,16 @@ _PANDOC_FORMATS: dict[str, str] = {
|
|||
# format from ONE publisher, and the file boundaries and `<h1>`s are a
|
||||
# generator's cut of that document, not 828 documents anyone wrote.
|
||||
#
|
||||
# `.xml` JOINED THE TABLE 2026-09-11, as `measured`, and the class was read off
|
||||
# the definitions above rather than inherited: the one file is a publisher's own
|
||||
# NISO-STS delivery of R761, written for their purposes years before any lookup
|
||||
# of ours, and its 2 761 titled `<sec>` are a fasit nobody here authored. The
|
||||
# honesty limit that travels with it and does NOT move when the build reaches
|
||||
# the reader's ceiling: the denominator is ONE file, ONE publisher, ONE schema.
|
||||
# `.xml` as a file type is far wider than NISO-STS, and a document in any other
|
||||
# schema keeps its text in document order and gets no structure at all -- which
|
||||
# is measured on fixtures, not on a corpus.
|
||||
#
|
||||
# `.pdf` JOINED THE TABLE 2026-09-10, as `measured`, and it enters on the
|
||||
# strongest evidence of any row here: eight real corpus PDFs with a fasit the
|
||||
# operator hand-counted document by document, plus a 701-page process code
|
||||
|
|
@ -467,6 +477,14 @@ class _XmlTextExtractor:
|
|||
self._lines: list[str] = []
|
||||
self._current: list[str] = []
|
||||
self._prefix = ""
|
||||
# The declared structure, recorded WHERE it is written rather than
|
||||
# recovered from the finished string. The PDF arm has to bridge from
|
||||
# (page, `/XYZ` top) onto a line index and was wrong on 1 840 of 2 762
|
||||
# nodes under the naive rule; here the reader appended the line, so the
|
||||
# index is not a guess and carries no tolerance. Empty for a document
|
||||
# that is not STS -- that is "this schema declares no section", and it
|
||||
# must not collapse into "this document has no structure to state".
|
||||
self.marks: list[OutlineMark] = []
|
||||
|
||||
def _break(self, prefix: str = "") -> None:
|
||||
"""Close the line being accumulated and open the next one.
|
||||
|
|
@ -543,7 +561,11 @@ class _XmlTextExtractor:
|
|||
level = min(depth, _ATX_MAX_LEVEL)
|
||||
parts = [self._text_of(label)] if label is not None else []
|
||||
parts.append(self._text_of(title))
|
||||
self._emit("#" * level + " " + " ".join(part for part in parts if part))
|
||||
heading = " ".join(part for part in parts if part)
|
||||
self._emit("#" * level + " " + heading)
|
||||
self.marks.append(
|
||||
OutlineMark(line=len(self._lines) - 1, level=level, title=heading)
|
||||
)
|
||||
skip = {id(title)} | ({id(label)} if label is not None else set())
|
||||
elif label is not None:
|
||||
# NEVER a heading. The label goes in FRONT of the body line the
|
||||
|
|
@ -573,7 +595,7 @@ class _XmlTextExtractor:
|
|||
return "\n".join(self._lines)
|
||||
|
||||
|
||||
def _extract_xml(data: bytes) -> str:
|
||||
def _xml_document(data: bytes) -> tuple[str, tuple[OutlineMark, ...]]:
|
||||
"""`xml`: NISO-STS structure as markdown, any other schema as its text.
|
||||
|
||||
A DTD IS REFUSED RATHER THAN PARSED, and that is a guarantee about this
|
||||
|
|
@ -604,7 +626,37 @@ def _extract_xml(data: bytes) -> str:
|
|||
f"the XML parser failed on this file: {exc}", code="extractor_xml_parse_error"
|
||||
) from exc
|
||||
sts = _local_name(root.tag) == _STS_ROOT or next(root.iter("sec"), None) is not None
|
||||
return _XmlTextExtractor(sts=sts).text(root)
|
||||
reader = _XmlTextExtractor(sts=sts)
|
||||
return reader.text(root), tuple(reader.marks)
|
||||
|
||||
|
||||
def _extract_xml(data: bytes) -> str:
|
||||
return _xml_document(data)[0]
|
||||
|
||||
|
||||
def xml_outline(name: str, data: bytes) -> tuple[OutlineMark, ...]:
|
||||
"""`xml`: the sections the document DECLARES, as marks on the extracted text.
|
||||
|
||||
The counterpart of `pdf_outline`, and the difference between them is the
|
||||
whole point of the row. A bookmark states a page and a y position, so that
|
||||
arm has to BRIDGE onto a line and reports what did not bridge; an STS
|
||||
`<sec><title>` is written into the output by this reader, so the line index
|
||||
is the one it appended at -- nothing is recovered, nothing is unresolved,
|
||||
and there is no tolerance constant to choose.
|
||||
|
||||
RE-READS the bytes rather than returning both from one call, for the same
|
||||
reason `pdf_outline` does: `extract_text` has one signature that every
|
||||
caller and every registry entry is keyed to, and a second return value
|
||||
would change it for eight rows to serve one. The parse is stdlib and the
|
||||
document is read twice; measured on a 2.4 MB NISO-STS file, that is the
|
||||
smaller cost by a wide margin.
|
||||
|
||||
Empty for every schema that is not STS. That is a statement about the
|
||||
document -- it declares no section -- and `find_candidates` reads an empty
|
||||
list as "leave every rule untouched", never as a route.
|
||||
"""
|
||||
del name # the registry decides which reader runs; kept for `pdf_outline`'s shape
|
||||
return _xml_document(data)[1]
|
||||
|
||||
|
||||
def _local_name(tag: str) -> str:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue