open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(extract): a label with no body under it is emitted, not overwritten

Browse source 
GREEN. A pending label prefix is now flushed as its own line when the next
thing out is a heading, when a second prefix would replace it, and at the end
of the document. Measured on R761: non-whitespace preservation goes from
1 283 395 against 1 283 393 to **1 283 395 against 1 283 395, ratio
1.000000** -- exact, which is what the invariant claims. ATX heading lines
2 761 of 2 761, unchanged.

Known-negatives held: `~/okf-test/dokumenter` still byte-identical against the
frozen `e1f4faa` export (`diff -r` empty), and that folder has 0 `.xml` files
of its 5, so it is a real control and not merely an unchanged number.

pytest -q: 1567 passed, 1 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 04:09:55 +02:00
commit f280b14056
1 changed files with 17 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

18
src/llm_ingestion_okf/extract.py
View file

@ -481,11 +481,25 @@ class _XmlTextExtractor:
self._lines.append(f"{self._prefix}{line}") self._lines.append(f"{self._prefix}{line}")
self._prefix = "" self._prefix = ""
if prefix: if prefix:
# A prefix still pending here belongs to a section that turned out
# to have no body line at all, and REPLACING it would drop it from
# the document. Measured on R761: exactly one `x)`, two characters,
# which is the whole distance between 0.999998 and exact.
if self._prefix:
self._lines.append(self._prefix.rstrip())
self._prefix = prefix self._prefix = prefix
def _emit(self, line: str) -> None: def _emit(self, line: str) -> None:
"""Put a whole line out, ahead of whatever is being accumulated.""" """Put a whole line out, ahead of whatever is being accumulated.
A prefix still pending is FLUSHED first rather than carried past a
heading: the section it belongs to is above this one, and holding it
would either attach it to the wrong body or lose it outright.
"""
self._break() self._break()
if self._prefix:
self._lines.append(self._prefix.rstrip())
self._prefix = ""
self._lines.append(line) self._lines.append(line)
def _text_of(self, element: Element) -> str: def _text_of(self, element: Element) -> str:
@ -554,6 +568,8 @@ class _XmlTextExtractor:
def text(self, root: Element) -> str: def text(self, root: Element) -> str:
self._walk(root, 0) self._walk(root, 0)
self._break() self._break()
if self._prefix:
self._lines.append(self._prefix.rstrip())
return "\n".join(self._lines) return "\n".join(self._lines)