open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

test(extract): an exact invariant does not get to be 0.999998

Browse source 
RED. Measured on R761 after the reader landed: non-whitespace preservation was
1 283 393 characters against the source's 1 283 395. Two characters, and a
percentage would have let them pass -- a ratio of 0.999998 reads like rounding.
Located by a prefix/suffix scan rather than a diff: the whole loss is one `x)`
marking a `<sec>` that carries a label and nothing else.

The mechanism is this reader's own. A label-only section holds its label as a
PREFIX for the body line beneath it, the way `li` is handled in the HTML
reader; when no body line follows -- the next thing emitted is the following
section's heading -- the pending prefix is overwritten by the next one and the
label never reaches the text.

`sts-empty-label.xml` is the case, hand-written and small: a lettered point
with a body, then one with none, then the next titled section.

pytest -q: 1 failed, 1566 passed, 1 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 04:07:35 +02:00
commit fc238440d7
3 changed files with 44 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1
tests/fixtures/README.md vendored
View file

@ -32,6 +32,7 @@ round-trip-symmetric defect.
| Fixture | What it is for |
|---|---|
| `sts-mini.xml` | The known-positive. A `<standard>` root with `<sec>` at three nesting levels carrying `<label>`+`<title>`, two lettered points (`a)`, `b)`) with a **label and no title**, one `<table-wrap>` with a label and two rows, `<p>` bodies, a `<list>`, and **one unnumbered section** (`Forord`, `<title>` with no `<label>`) mirroring the single such section in R761. The lettered points are what the 4 954 label-only `<sec>` in that document look like: promoted to headings they would bury its own 2 761. |
| `sts-empty-label.xml` | A `<sec>` carrying a `<label>` and **nothing else**, between a lettered point that has a body and the next titled section. The label is held as a prefix for a body line that never arrives, so it was overwritten and lost: measured on R761 that is exactly one `x)`, two characters of 1 283 395, ratio 0.999998. An exact invariant does not get to be 0.999998. |
| `generic-feed.xml` | Known-negative: XML that is **not** STS. It must produce text and ONE plan — never zero, never a crash, and never element names promoted to headings. |
| `xml-doctype-bomb.xml` | Known-negative, security: a `<!DOCTYPE` with a small nested-entity expansion. It must be refused by `code`, and the test asserts the expansion appears in **no** output, including the error text. Small on purpose — the point is that it is never parsed, not that it detonates. |
| `xml-malformed.xml` | Known-negative: an unterminated tag must raise a typed `ExtractionError`, not leak `ParseError` and not yield zero concepts in silence. |

20
tests/fixtures/sts-empty-label.xml vendored Normal file
View file

@ -0,0 +1,20 @@
<standard>
<body>
<sec id="s-1">
<label>84.61</label>
<title>Beskyttelse av betong</title>
<sec id="s-1-a" sec-type="spec">
<label>a)</label>
<p>Omfatter beskyttelse mot kjemikalier.</p>
</sec>
<sec id="s-1-x" sec-type="spec">
<label>x)</label>
</sec>
</sec>
<sec id="s-2">
<label>84.62</label>
<title>Rengjoering av betongoverflate</title>
<p>Overflaten rengjoeres foer behandling.</p>
</sec>
</body>
</standard>

23
tests/test_xml_sts.py
View file

@ -48,6 +48,7 @@ STS = FIXTURES / "sts-mini.xml"
GENERIC = FIXTURES / "generic-feed.xml"
BOMB = FIXTURES / "xml-doctype-bomb.xml"
MALFORMED = FIXTURES / "xml-malformed.xml"
EMPTY_LABEL = FIXTURES / "sts-empty-label.xml"
STS_EXPECTED = """\
Testnormal for fiksturbruk
@ -226,3 +227,25 @@ def test_a_bundle_is_built_end_to_end_from_an_sts_document(tmp_path: Path) -> No
)
assert titles, "an STS document must produce concepts"
assert "Forord" in titles
def test_a_label_with_no_body_under_it_still_reaches_the_text() -> None:
"""A pending label must not be overwritten by the next one, or by a heading.
MEASURED ON R761, and it is exactly two characters: one `x)` marks a
`<sec>` that carries a label and nothing else, and the label was held as a
prefix for a body line that never came -- the next thing emitted was the
following section's heading. Non-whitespace preservation was 1 283 393
against 1 283 395, ratio 0.999998. An EXACT invariant does not get to be
0.999998, and a percentage would have hidden which two characters they
were.
"""
import xml.etree.ElementTree as ET
text = extract.extract_text(EMPTY_LABEL.name, EMPTY_LABEL.read_bytes())
source = ET.fromstring(EMPTY_LABEL.read_text(encoding="utf-8"))
assert "x)" in text
emitted = "".join("".join(line.lstrip("#") for line in text.split("\n")).split())
original = "".join("".join(source.itertext()).split())
assert emitted == original