open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

test(extract): an exact invariant does not get to be 0.999998

Browse source 
RED. Measured on R761 after the reader landed: non-whitespace preservation was
1 283 393 characters against the source's 1 283 395. Two characters, and a
percentage would have let them pass -- a ratio of 0.999998 reads like rounding.
Located by a prefix/suffix scan rather than a diff: the whole loss is one `x)`
marking a `<sec>` that carries a label and nothing else.

The mechanism is this reader's own. A label-only section holds its label as a
PREFIX for the body line beneath it, the way `li` is handled in the HTML
reader; when no body line follows -- the next thing emitted is the following
section's heading -- the pending prefix is overwritten by the next one and the
label never reaches the text.

`sts-empty-label.xml` is the case, hand-written and small: a lettered point
with a body, then one with none, then the next titled section.

pytest -q: 1 failed, 1566 passed, 1 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 04:07:35 +02:00
commit fc238440d7
3 changed files with 44 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

23
tests/test_xml_sts.py
View file

@ -48,6 +48,7 @@ STS = FIXTURES / "sts-mini.xml"
GENERIC = FIXTURES / "generic-feed.xml"
BOMB = FIXTURES / "xml-doctype-bomb.xml"
MALFORMED = FIXTURES / "xml-malformed.xml"
EMPTY_LABEL = FIXTURES / "sts-empty-label.xml"
STS_EXPECTED = """\
Testnormal for fiksturbruk
@ -226,3 +227,25 @@ def test_a_bundle_is_built_end_to_end_from_an_sts_document(tmp_path: Path) -> No
)
assert titles, "an STS document must produce concepts"
assert "Forord" in titles
def test_a_label_with_no_body_under_it_still_reaches_the_text() -> None:
"""A pending label must not be overwritten by the next one, or by a heading.
MEASURED ON R761, and it is exactly two characters: one `x)` marks a
`<sec>` that carries a label and nothing else, and the label was held as a
prefix for a body line that never came -- the next thing emitted was the
following section's heading. Non-whitespace preservation was 1 283 393
against 1 283 395, ratio 0.999998. An EXACT invariant does not get to be
0.999998, and a percentage would have hidden which two characters they
were.
"""
import xml.etree.ElementTree as ET
text = extract.extract_text(EMPTY_LABEL.name, EMPTY_LABEL.read_bytes())
source = ET.fromstring(EMPTY_LABEL.read_text(encoding="utf-8"))
assert "x)" in text
emitted = "".join("".join(line.lstrip("#") for line in text.split("\n")).split())
original = "".join("".join(source.itertext()).split())
assert emitted == original