open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(extract): HTML collapsed to one line, so 828 of 828 sections had no boundary

Browse source 
`_HTMLTextExtractor.text()` was `" ".join("".join(parts).split())`. `str.split()`
with no argument splits on newlines too, so extraction of ANY HTML file returned
unconditionally one line. Every boundary grammar in `propose` is line-anchored
(`_ATX`, `_NUMBERED`, `_TABLE_ROW`, `_GRID_RULE`, `_OUTLINE`, each with `^`), and
on one line at most the first can match while a match at line 0 opens no interior
boundary. Measured outside this repo on a consumer's export of a published
handbook: 83 / 414 / 828 `.html` files gave 0 plans, N documents with no boundary
and exit 2 at every point, and a coarser 145-document cut gave 145 of 145. The
same sections as markdown gave 828 of 828 plans -- so the instrument was fine and
`.html` was the one core-supported type that had never met a real document.

Block tags now open lines of their own, `h1`-`h6` carry the ATX marker for their
own level (not a flat `#`, which would hand `_ATX` three top-level boundaries
where the document declares one section and two subsections), `br` breaks the
line, and every other tag stays the word boundary it already was. The output
grammar is MARKDOWN and deliberately the same markdown the office rows reach the
proposer through, so no HTML-only heading grammar exists.

NOT via the converter: `.html` stays out of `_PANDOC_FORMATS` because routing it
there would add CVE-2025-51591 (SSRF via an iframe in HTML input), unpatched in
every converter version. The test asserting that exclusion is untouched and green.

The block set is wider than the five tags the corpus exercises, on purpose:
block versus inline is a property of HTML, not of one corpus, and a `div`-
structured page carries its prose in containers this corpus never uses.

Measured after, all with denominators:
- 828 of 828 plans, exit 0, `merged + coded rejections = 828; N = 828`; 3206
  concepts / 6015 md files, which is the markdown path's count EXACTLY -- 0.0 %
  deviation against the +/-2 % bar, and the same at 50 % (1651) and 10 % (343).
  The coarser 145-document cut goes 145 of 145 with no boundary to 145 plans /
  953 concepts.
- Text preservation as an EXACT invariant, not a percentage: strip the added ATX
  markers and the non-whitespace sequence is identical to the old extractor's for
  the same bytes. 828 of 828 files exact, character ratio 1.000000 against the
  >= 99.8 % bar. 7600 markers added; 31 141 lines produced where the old
  extractor produced 828, one per file.
- `_SKIP_TAGS` unchanged at {script, style}. Dropping nav/header/footer is a
  different change with a different guarantee and is not made here.
- No other file type moved, measured rather than argued: 0 of 86 K2 corpus files
  and 0 of 5 smoke-folder files are HTML, and the smoke bundle is byte-identical
  before and after (`diff -r` empty, 52 md / 26 concepts, 0 of 5 rejected).
  `okf project` stays byte-equal to `okf build` (`diff -r` empty).

Provenance moves with it: `source_units` routed `.html` through `_line_units`
already, but the table was trivial -- every offset resolved to line 1. The
numbers now mean something, and what they mean is a line of OUR extraction (a
BLOCK), never a line of the original markup.

`_EVIDENCE` gains a `.html` row at `measured`, chosen against the class
definitions: the files are a consumer's own export of a real published handbook,
produced for their ingestion and not to exercise this row. What the class does
not claim travels with it -- one product, one format, one publisher, and a
generator's cut.

One existing test changed because the behaviour changed, and it says so:
`test_html_text_via_htmlparser` asserted the collapsed form. The other two
(`test_html_skips_script_and_style`, `test_htm_is_an_html_alias`) were re-read
and hold unchanged -- the order expected three to move; only one did.

The corpus-wide invariant runs in the suite behind `OKF_HTML_CORPUS`: a corpus
path names a consumer's export and this repository is public.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 23:57:47 +02:00
commit 732f84df6e
3 changed files with 310 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

127
src/llm_ingestion_okf/extract.py
View file

@ -96,12 +96,23 @@ _PANDOC_FORMATS: dict[str, str] = {
# nothing and the document reaches Door B's INBOX as one concept --
# content preserved, structure zero. That is the row's honest result
# and it is the one open finding of the three.
#
# `.html` JOINED THE TABLE 2026-09-09, as `measured`, and the class was chosen
# against the definitions above rather than assumed: the 828 files are a
# consumer's own export of a real published handbook, produced for their
# ingestion and not to exercise this row, with a fasit written before any
# lookup -- which is `measured`'s test, "someone wrote the document for their
# own purposes and we counted what we got". What that class does NOT claim, and
# the honesty limit that travels with it: the 828 files are ONE product in ONE
# format from ONE publisher, and the file boundaries and `<h1>`s are a
# generator's cut of that document, not 828 documents anyone wrote.
_EVIDENCE: dict[str, str] = {
".docx": "measured",
".xlsx": "measured",
".pptx": "constructed",
".odt": "constructed",
".rtf": "constructed",
".html": "measured",
}
# Load-bearing, all three, and none of them hygiene:
@ -171,8 +182,67 @@ _PDF_LOSSY_WARNING = (
)
# Tags whose text content is never document prose.
#
# TWO MEMBERS, and it stays two in round 11. Dropping `nav`/`header`/`footer`
# as well is a DIFFERENT change with a different guarantee: the exact
# text-preservation invariant below holds only while nothing is dropped, and a
# quiet widening here would hide exactly how many characters left the document.
_SKIP_TAGS = frozenset({"script", "style"})
# The heading tags, and the ATX level each becomes. The level is the TAG's:
# a flat `#` for every heading would hand `propose._ATX` three top-level
# boundaries where the document declares one section and two subsections.
_HEADING_TAGS: dict[str, int] = {f"h{level}": level for level in range(1, 7)}
# Tags that open a line of their own. Everything NOT here is inline and stays a
# word boundary inside the current line, which is what `b`/`em`/`a`/`span` were
# already treated as.
#
# WHY THIS IS A SET AND NOT THE FIVE TAGS THE CORPUS EXERCISES. Block versus
# inline is a property of HTML, not of one corpus. The measured corpus writes
# its prose in `p`, `li` and `tr`; a `div`-structured page -- the ordinary shape
# of hand-written and exported HTML -- carries the same prose in containers this
# corpus never uses, and restricting the set to what was measured would leave
# that page collapsing into one line, which IS the defect. Adding a line break
# never removes text and never invents a boundary on its own: a boundary needs a
# line that MATCHES a grammar.
_BLOCK_TAGS = frozenset(
{
"p",
"li",
"tr",
"pre",
"div",
"section",
"article",
"header",
"footer",
"nav",
"main",
"aside",
"table",
"thead",
"tbody",
"tfoot",
"caption",
"ul",
"ol",
"dl",
"dt",
"dd",
"blockquote",
"figure",
"figcaption",
"hr",
"address",
"form",
"fieldset",
"legend",
"title",
"body",
}
)
def decode_text(data: bytes) -> str:
"""Decode file bytes as UTF-8 (BOM-stripping), typed on failure.
@ -210,37 +280,74 @@ def _extract_json(data: bytes) -> str:
class _HTMLTextExtractor(HTMLParser):
"""Collect document text, skipping `script`/`style`, tags as word boundaries.
"""Collect document text as LINES, skipping `script`/`style`.
Tags contribute no text of their own but do separate words: a boundary
space is emitted at every tag so adjacent block text (``</h1><p>``) does not
fuse. Runs of whitespace collapse to single spaces in :meth:`text`.
A block tag opens a line of its own, a heading tag opens one carrying the
ATX marker for its level, `br` breaks the current line, and every other tag
stays what it always was: a word boundary inside the line, so adjacent
inline text (``<b>a</b>b``) does not fuse. Runs of whitespace inside a line
collapse to single spaces.
THE OUTPUT GRAMMAR IS MARKDOWN, and deliberately the same markdown the
office rows reach the proposer through. `_ATX` and every other boundary
grammar is line-anchored, so this class decides -- alone -- whether an HTML
document can be segmented at all. It emitted one line for any input until
2026-09-09, which is why 828 of 828 real sections produced zero boundaries.
TEXT IS PRESERVED EXACTLY. The only characters this adds are the ATX
markers; strip those and the non-whitespace sequence is identical to the
one-line form. Nothing is ever dropped here beyond `_SKIP_TAGS`.
"""
def __init__(self) -> None:
super().__init__(convert_charrefs=True)
self._parts: list[str] = []
self._lines: list[str] = []
self._current: list[str] = []
self._prefix = ""
self._skip_depth = 0
def _break(self, prefix: str = "") -> None:
"""Close the line being accumulated and open the next one."""
line = " ".join("".join(self._current).split())
self._current = []
if line:
self._lines.append(f"{self._prefix}{line}")
self._prefix = prefix
def _open(self, tag: str) -> bool:
"""Break for a block or heading tag; report whether it was one."""
level = _HEADING_TAGS.get(tag)
if level is not None:
self._break("#" * level + " ")
return True
if tag in _BLOCK_TAGS or tag == "br":
self._break()
return True
return False
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
self._parts.append(" ")
if not self._open(tag):
self._current.append(" ")
if tag in _SKIP_TAGS:
self._skip_depth += 1
def handle_startendtag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
self._parts.append(" ")
if not self._open(tag):
self._current.append(" ")
def handle_endtag(self, tag: str) -> None:
if tag in _SKIP_TAGS and self._skip_depth > 0:
self._skip_depth -= 1
self._parts.append(" ")
if not self._open(tag):
self._current.append(" ")
def handle_data(self, data: str) -> None:
if self._skip_depth == 0:
self._parts.append(data)
self._current.append(data)
def text(self) -> str:
return " ".join("".join(self._parts).split())
self._break()
return "\n".join(self._lines)
def _extract_html(data: bytes) -> str:

175
tests/test_extract.py
View file

@ -11,6 +11,7 @@ from __future__ import annotations
import hashlib
import importlib.util
import os
import re
import sys
from pathlib import Path
@ -58,20 +59,178 @@ def test_json_is_verbatim_inside_a_fenced_block() -> None:
def test_html_text_via_htmlparser() -> None:
# Block boundaries separate words; tags themselves contribute no text.
"""CHANGED 2026-09-09 BECAUSE THE BEHAVIOUR CHANGED, not to go green.
This asserted the collapsed one-line form -- the very form that made 828
of 828 real HTML sections unsegmentable. Block tags now open lines and
headings carry their ATX level; inline tags are still word boundaries,
which is the half of the old assertion that still holds and is still here.
"""
out = extract_text("page.html", b"<h1>Title</h1><p>Hello <b>world</b></p>")
assert out == "Title Hello world"
assert out == "# Title\nHello world"
def test_html_skips_script_and_style() -> None:
"""UNCHANGED across round 11, and re-read to confirm it: `_SKIP_TAGS` still
holds exactly `script` and `style`, and a single block still renders as a
single line, so the old expectation is still the right one."""
html = b"<style>.x{color:red}</style><p>Keep</p><script>evil()</script>"
assert extract_text("page.html", html) == "Keep"
def test_htm_is_an_html_alias() -> None:
"""UNCHANGED across round 11, same reason as the test above."""
assert extract_text("page.htm", b"<p>hi</p>") == "hi"
# --- html: block structure survives extraction (round 11) -------------------
#
# The defect this closes, measured OUTSIDE this repo on 828 real sections:
# `text()` used to be `" ".join("".join(parts).split())`, and `str.split()`
# with no argument splits on newlines too, so every HTML file extracted to
# UNCONDITIONALLY ONE LINE. Every boundary grammar in `propose` is
# line-anchored (`_ATX`, `_NUMBERED`, `_TABLE_ROW`, `_GRID_RULE`, `_OUTLINE`,
# each with `^`), so 828 of 828 documents got zero boundaries and the run
# exited 2. The same 828 sections as markdown gave 828 plans.
_ADDED_ATX = re.compile(r"(?m)^#{1,6} ")
_HTML_KNOWN_POSITIVE = b"""<!doctype html>
<html><head><title>Doc</title></head><body>
<h1>Top</h1>
<p>Intro <b>bold</b> word.</p>
<h2>First part</h2>
<ul><li>alpha</li><li>beta</li></ul>
<h2>Second part</h2>
<table><tr><td>cell one</td><td>cell two</td></tr><tr><td>r2c1</td><td>r2c2</td></tr></table>
</body></html>
"""
def _legacy_html_text(data: bytes) -> str:
"""The pre-round-11 extractor, kept HERE as the invariant's reference.
The text-preservation invariant is stated against what the old extractor
returned for the same bytes, so it needs that string. Reproducing the ten
lines in the test is the only way to keep both forms available from one
definition of "what the old code did" -- exporting a second renderer from
the module would put a function in the package that nothing ships uses.
"""
from html.parser import HTMLParser
from llm_ingestion_okf.extract import _SKIP_TAGS
class _Legacy(HTMLParser):
def __init__(self) -> None:
super().__init__(convert_charrefs=True)
self.parts: list[str] = []
self.skip = 0
def handle_starttag(self, tag: str, attrs: object) -> None: # type: ignore[override]
self.parts.append(" ")
if tag in _SKIP_TAGS:
self.skip += 1
def handle_startendtag(self, tag: str, attrs: object) -> None: # type: ignore[override]
self.parts.append(" ")
def handle_endtag(self, tag: str) -> None:
if tag in _SKIP_TAGS and self.skip > 0:
self.skip -= 1
self.parts.append(" ")
def handle_data(self, data: str) -> None:
if self.skip == 0:
self.parts.append(data)
parser = _Legacy()
parser.feed(data.decode("utf-8-sig"))
parser.close()
return " ".join("".join(parser.parts).split())
def test_html_headings_become_atx_lines_at_the_tag_s_own_level() -> None:
"""The LEVEL is the tag's, not a flat `#` for every heading.
A flat prefix would give `propose._ATX` three level-1 boundaries where the
document declares one section and two subsections, so the assertion is on
the line CONTENT, never on "there are now several lines".
"""
out = extract_text("page.html", _HTML_KNOWN_POSITIVE)
lines = out.split("\n")
assert "# Top" in lines
assert "## First part" in lines
assert "## Second part" in lines
# Blocks are their own lines; inline markup stays a word boundary.
assert "Intro bold word." in lines
assert "alpha" in lines
assert "beta" in lines
# A table row is one line, its cells separated by a space.
assert "cell one cell two" in lines
assert "r2c1 r2c2" in lines
def test_html_without_a_heading_is_one_concept_s_worth_of_text_never_zero() -> None:
"""KNOWN-NEGATIVE: no heading in, no fabricated heading out.
The `rtf` row's honest result, and the shape this must keep: a document
that declares nothing lands as ONE concept with its content preserved --
never zero concepts and never zero characters.
"""
out = extract_text("flat.html", b"<p>First para.</p><p>Second para.</p>")
assert not _ADDED_ATX.search(out)
assert out.split("\n") == ["First para.", "Second para."]
def test_html_line_structure_preserves_every_non_whitespace_character() -> None:
"""Text preservation is an EXACT invariant, not a percentage.
Strip the ATX markers this extractor added and the non-whitespace sequence
must be identical to what the old extractor returned for the same bytes.
"""
new = extract_text("page.html", _HTML_KNOWN_POSITIVE)
assert _ADDED_ATX.search(new), "known-positive: the query must be able to find a marker"
stripped = _ADDED_ATX.sub("", new)
assert "".join(stripped.split()) == "".join(_legacy_html_text(_HTML_KNOWN_POSITIVE).split())
@pytest.mark.skipif(
not os.environ.get("OKF_HTML_CORPUS"),
reason=(
"set OKF_HTML_CORPUS to a directory of .html files to run the "
"corpus-wide invariant. The path is an ENVIRONMENT VARIABLE and not a "
"constant here on purpose: a corpus path names a consumer's own "
"export, and this repository is public"
),
)
def test_the_invariant_holds_over_a_whole_corpus_not_one_document() -> None:
"""One document proves the invariant is reachable, not that it holds.
Run over the SK1 corpus (828 `.html` sections of a published handbook) this
was 828 of 828 exact and a character ratio of 1.000000 against SK1's own
>= 99.8 % bar, with 7600 ATX markers added and 31 141 lines produced where
the old extractor produced 828 -- one per file, which was the defect.
"""
root = Path(os.environ["OKF_HTML_CORPUS"])
files = sorted(root.rglob("*.html"))
assert files, f"no .html under {root}: the corpus pointer is wrong, not the code"
markers = 0
for path in files:
data = path.read_bytes()
new = extract_text(path.name, data)
markers += len(_ADDED_ATX.findall(new))
stripped = _ADDED_ATX.sub("", new)
assert "".join(stripped.split()) == "".join(_legacy_html_text(data).split()), (
f"text preservation broke on {path.name}"
)
assert markers > 0, "known-positive: a corpus of prose must yield some heading"
def test_html_br_is_a_line_break_and_inline_tags_are_not() -> None:
out = extract_text("page.html", b"<p>one<br>two <em>three</em> four</p>")
assert out.split("\n") == ["one", "two three four"]
def test_extension_dispatch_is_case_insensitive() -> None:
assert extract_text("NOTE.MD", b"x") == "x"
@ -175,11 +334,19 @@ def test_evidence_class_is_asserted_not_commented() -> None:
`.pptx`, `.odt` and `.rtf` have denominator ZERO in the corpus this arm was
measured on. A comment saying so rots; an assertion that names them keeps
an unmeasured row from quietly presenting as a supported one.
The table is NOT the converter's rows alone: `.html` is core-supported and
was the one core row that had never met a real document, so it carries a
class here too (2026-09-09, `measured`, 828 files).
"""
from llm_ingestion_okf.extract import _EVIDENCE, _PANDOC_FORMATS
assert set(_EVIDENCE) == set(_PANDOC_FORMATS), "every row needs an evidence class"
assert {s for s, e in _EVIDENCE.items() if e == "measured"} == {".docx", ".xlsx"}
assert set(_EVIDENCE) == set(_PANDOC_FORMATS) | {".html"}, "every row needs a class"
assert {s for s, e in _EVIDENCE.items() if e == "measured"} == {
".docx",
".xlsx",
".html",
}
# Since 2026-09-09 the three office rows are `constructed`, not
# `unmeasured`: each has now been put through end to end on a hand-built
# document with a hand-written fasit, and none of them has a corpus file.

21
tests/test_provenance.py
View file

@ -145,6 +145,27 @@ def test_a_text_file_reports_line_numbers_of_the_original() -> None:
assert units.covering(0, len(text)) == (1, 3)
def test_an_html_locator_is_lines_of_our_extraction_not_of_the_source_file() -> None:
"""The `.html` unit table stopped being trivial in round 11.
It always existed -- `.html` is in `_CORE_EXTRACTORS`, so `source_units`
routed it through `_line_units` -- but the extractor returned ONE line for
any input, so every offset in every HTML concept resolved to line 1. Now
the numbers mean something, and what they mean is a line of OUR extraction:
the source file below is four physical lines and the extraction is three,
because a line is a BLOCK here, not a line of the original markup.
"""
data = b"<html><body>\n<h1>Top</h1>\n<p>first para</p>\n<p>second para</p>\n</body></html>"
text = extract_text("page.html", data)
units = source_units("page.html", data, text)
assert units is not None
assert units.unit == "lines"
assert text.split("\n") == ["# Top", "first para", "second para"]
assert units.covering(0, 5) == (1, 1)
assert units.covering(text.index("second"), len(text)) == (3, 3)
assert units.covering(0, len(text)) == (1, 3)
def test_a_docx_locator_is_lines_because_paragraphs_do_not_survive() -> None:
# Measured on the five K2 `.docx` documents: `<w:p>` counts of 108, 27, 65,
# 176 and 57 against converted-markdown line counts of 75, 33, 67, 144 and