open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(extract): HTML collapsed to one line, so 828 of 828 sections had no boundary

Browse source 
`_HTMLTextExtractor.text()` was `" ".join("".join(parts).split())`. `str.split()`
with no argument splits on newlines too, so extraction of ANY HTML file returned
unconditionally one line. Every boundary grammar in `propose` is line-anchored
(`_ATX`, `_NUMBERED`, `_TABLE_ROW`, `_GRID_RULE`, `_OUTLINE`, each with `^`), and
on one line at most the first can match while a match at line 0 opens no interior
boundary. Measured outside this repo on a consumer's export of a published
handbook: 83 / 414 / 828 `.html` files gave 0 plans, N documents with no boundary
and exit 2 at every point, and a coarser 145-document cut gave 145 of 145. The
same sections as markdown gave 828 of 828 plans -- so the instrument was fine and
`.html` was the one core-supported type that had never met a real document.

Block tags now open lines of their own, `h1`-`h6` carry the ATX marker for their
own level (not a flat `#`, which would hand `_ATX` three top-level boundaries
where the document declares one section and two subsections), `br` breaks the
line, and every other tag stays the word boundary it already was. The output
grammar is MARKDOWN and deliberately the same markdown the office rows reach the
proposer through, so no HTML-only heading grammar exists.

NOT via the converter: `.html` stays out of `_PANDOC_FORMATS` because routing it
there would add CVE-2025-51591 (SSRF via an iframe in HTML input), unpatched in
every converter version. The test asserting that exclusion is untouched and green.

The block set is wider than the five tags the corpus exercises, on purpose:
block versus inline is a property of HTML, not of one corpus, and a `div`-
structured page carries its prose in containers this corpus never uses.

Measured after, all with denominators:
- 828 of 828 plans, exit 0, `merged + coded rejections = 828; N = 828`; 3206
  concepts / 6015 md files, which is the markdown path's count EXACTLY -- 0.0 %
  deviation against the +/-2 % bar, and the same at 50 % (1651) and 10 % (343).
  The coarser 145-document cut goes 145 of 145 with no boundary to 145 plans /
  953 concepts.
- Text preservation as an EXACT invariant, not a percentage: strip the added ATX
  markers and the non-whitespace sequence is identical to the old extractor's for
  the same bytes. 828 of 828 files exact, character ratio 1.000000 against the
  >= 99.8 % bar. 7600 markers added; 31 141 lines produced where the old
  extractor produced 828, one per file.
- `_SKIP_TAGS` unchanged at {script, style}. Dropping nav/header/footer is a
  different change with a different guarantee and is not made here.
- No other file type moved, measured rather than argued: 0 of 86 K2 corpus files
  and 0 of 5 smoke-folder files are HTML, and the smoke bundle is byte-identical
  before and after (`diff -r` empty, 52 md / 26 concepts, 0 of 5 rejected).
  `okf project` stays byte-equal to `okf build` (`diff -r` empty).

Provenance moves with it: `source_units` routed `.html` through `_line_units`
already, but the table was trivial -- every offset resolved to line 1. The
numbers now mean something, and what they mean is a line of OUR extraction (a
BLOCK), never a line of the original markup.

`_EVIDENCE` gains a `.html` row at `measured`, chosen against the class
definitions: the files are a consumer's own export of a real published handbook,
produced for their ingestion and not to exercise this row. What the class does
not claim travels with it -- one product, one format, one publisher, and a
generator's cut.

One existing test changed because the behaviour changed, and it says so:
`test_html_text_via_htmlparser` asserted the collapsed form. The other two
(`test_html_skips_script_and_style`, `test_htm_is_an_html_alias`) were re-read
and hold unchanged -- the order expected three to move; only one did.

The corpus-wide invariant runs in the suite behind `OKF_HTML_CORPUS`: a corpus
path names a consumer's export and this repository is public.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 23:57:47 +02:00
commit 732f84df6e
3 changed files with 310 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

127
src/llm_ingestion_okf/extract.py
View file

@ -96,12 +96,23 @@ _PANDOC_FORMATS: dict[str, str] = {
# nothing and the document reaches Door B's INBOX as one concept --
# content preserved, structure zero. That is the row's honest result
# and it is the one open finding of the three.
#
# `.html` JOINED THE TABLE 2026-09-09, as `measured`, and the class was chosen
# against the definitions above rather than assumed: the 828 files are a
# consumer's own export of a real published handbook, produced for their
# ingestion and not to exercise this row, with a fasit written before any
# lookup -- which is `measured`'s test, "someone wrote the document for their
# own purposes and we counted what we got". What that class does NOT claim, and
# the honesty limit that travels with it: the 828 files are ONE product in ONE
# format from ONE publisher, and the file boundaries and `<h1>`s are a
# generator's cut of that document, not 828 documents anyone wrote.
_EVIDENCE: dict[str, str] = {
".docx": "measured",
".xlsx": "measured",
".pptx": "constructed",
".odt": "constructed",
".rtf": "constructed",
".html": "measured",
}
# Load-bearing, all three, and none of them hygiene:
@ -171,8 +182,67 @@ _PDF_LOSSY_WARNING = (
)
# Tags whose text content is never document prose.
#
# TWO MEMBERS, and it stays two in round 11. Dropping `nav`/`header`/`footer`
# as well is a DIFFERENT change with a different guarantee: the exact
# text-preservation invariant below holds only while nothing is dropped, and a
# quiet widening here would hide exactly how many characters left the document.
_SKIP_TAGS = frozenset({"script", "style"})
# The heading tags, and the ATX level each becomes. The level is the TAG's:
# a flat `#` for every heading would hand `propose._ATX` three top-level
# boundaries where the document declares one section and two subsections.
_HEADING_TAGS: dict[str, int] = {f"h{level}": level for level in range(1, 7)}
# Tags that open a line of their own. Everything NOT here is inline and stays a
# word boundary inside the current line, which is what `b`/`em`/`a`/`span` were
# already treated as.
#
# WHY THIS IS A SET AND NOT THE FIVE TAGS THE CORPUS EXERCISES. Block versus
# inline is a property of HTML, not of one corpus. The measured corpus writes
# its prose in `p`, `li` and `tr`; a `div`-structured page -- the ordinary shape
# of hand-written and exported HTML -- carries the same prose in containers this
# corpus never uses, and restricting the set to what was measured would leave
# that page collapsing into one line, which IS the defect. Adding a line break
# never removes text and never invents a boundary on its own: a boundary needs a
# line that MATCHES a grammar.
_BLOCK_TAGS = frozenset(
{
"p",
"li",
"tr",
"pre",
"div",
"section",
"article",
"header",
"footer",
"nav",
"main",
"aside",
"table",
"thead",
"tbody",
"tfoot",
"caption",
"ul",
"ol",
"dl",
"dt",
"dd",
"blockquote",
"figure",
"figcaption",
"hr",
"address",
"form",
"fieldset",
"legend",
"title",
"body",
}
)
def decode_text(data: bytes) -> str:
"""Decode file bytes as UTF-8 (BOM-stripping), typed on failure.
@ -210,37 +280,74 @@ def _extract_json(data: bytes) -> str:
class _HTMLTextExtractor(HTMLParser):
"""Collect document text, skipping `script`/`style`, tags as word boundaries.
"""Collect document text as LINES, skipping `script`/`style`.
Tags contribute no text of their own but do separate words: a boundary
space is emitted at every tag so adjacent block text (``</h1><p>``) does not
fuse. Runs of whitespace collapse to single spaces in :meth:`text`.
A block tag opens a line of its own, a heading tag opens one carrying the
ATX marker for its level, `br` breaks the current line, and every other tag
stays what it always was: a word boundary inside the line, so adjacent
inline text (``<b>a</b>b``) does not fuse. Runs of whitespace inside a line
collapse to single spaces.
THE OUTPUT GRAMMAR IS MARKDOWN, and deliberately the same markdown the
office rows reach the proposer through. `_ATX` and every other boundary
grammar is line-anchored, so this class decides -- alone -- whether an HTML
document can be segmented at all. It emitted one line for any input until
2026-09-09, which is why 828 of 828 real sections produced zero boundaries.
TEXT IS PRESERVED EXACTLY. The only characters this adds are the ATX
markers; strip those and the non-whitespace sequence is identical to the
one-line form. Nothing is ever dropped here beyond `_SKIP_TAGS`.
"""
def __init__(self) -> None:
super().__init__(convert_charrefs=True)
self._parts: list[str] = []
self._lines: list[str] = []
self._current: list[str] = []
self._prefix = ""
self._skip_depth = 0
def _break(self, prefix: str = "") -> None:
"""Close the line being accumulated and open the next one."""
line = " ".join("".join(self._current).split())
self._current = []
if line:
self._lines.append(f"{self._prefix}{line}")
self._prefix = prefix
def _open(self, tag: str) -> bool:
"""Break for a block or heading tag; report whether it was one."""
level = _HEADING_TAGS.get(tag)
if level is not None:
self._break("#" * level + " ")
return True
if tag in _BLOCK_TAGS or tag == "br":
self._break()
return True
return False
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
self._parts.append(" ")
if not self._open(tag):
self._current.append(" ")
if tag in _SKIP_TAGS:
self._skip_depth += 1
def handle_startendtag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
self._parts.append(" ")
if not self._open(tag):
self._current.append(" ")
def handle_endtag(self, tag: str) -> None:
if tag in _SKIP_TAGS and self._skip_depth > 0:
self._skip_depth -= 1
self._parts.append(" ")
if not self._open(tag):
self._current.append(" ")
def handle_data(self, data: str) -> None:
if self._skip_depth == 0:
self._parts.append(data)
self._current.append(data)
def text(self) -> str:
return " ".join("".join(self._parts).split())
self._break()
return "\n".join(self._lines)
def _extract_html(data: bytes) -> str: