`_HTMLTextExtractor.text()` was `" ".join("".join(parts).split())`. `str.split()`
with no argument splits on newlines too, so extraction of ANY HTML file returned
unconditionally one line. Every boundary grammar in `propose` is line-anchored
(`_ATX`, `_NUMBERED`, `_TABLE_ROW`, `_GRID_RULE`, `_OUTLINE`, each with `^`), and
on one line at most the first can match while a match at line 0 opens no interior
boundary. Measured outside this repo on a consumer's export of a published
handbook: 83 / 414 / 828 `.html` files gave 0 plans, N documents with no boundary
and exit 2 at every point, and a coarser 145-document cut gave 145 of 145. The
same sections as markdown gave 828 of 828 plans -- so the instrument was fine and
`.html` was the one core-supported type that had never met a real document.
Block tags now open lines of their own, `h1`-`h6` carry the ATX marker for their
own level (not a flat `#`, which would hand `_ATX` three top-level boundaries
where the document declares one section and two subsections), `br` breaks the
line, and every other tag stays the word boundary it already was. The output
grammar is MARKDOWN and deliberately the same markdown the office rows reach the
proposer through, so no HTML-only heading grammar exists.
NOT via the converter: `.html` stays out of `_PANDOC_FORMATS` because routing it
there would add CVE-2025-51591 (SSRF via an iframe in HTML input), unpatched in
every converter version. The test asserting that exclusion is untouched and green.
The block set is wider than the five tags the corpus exercises, on purpose:
block versus inline is a property of HTML, not of one corpus, and a `div`-
structured page carries its prose in containers this corpus never uses.
Measured after, all with denominators:
- 828 of 828 plans, exit 0, `merged + coded rejections = 828; N = 828`; 3206
concepts / 6015 md files, which is the markdown path's count EXACTLY -- 0.0 %
deviation against the +/-2 % bar, and the same at 50 % (1651) and 10 % (343).
The coarser 145-document cut goes 145 of 145 with no boundary to 145 plans /
953 concepts.
- Text preservation as an EXACT invariant, not a percentage: strip the added ATX
markers and the non-whitespace sequence is identical to the old extractor's for
the same bytes. 828 of 828 files exact, character ratio 1.000000 against the
>= 99.8 % bar. 7600 markers added; 31 141 lines produced where the old
extractor produced 828, one per file.
- `_SKIP_TAGS` unchanged at {script, style}. Dropping nav/header/footer is a
different change with a different guarantee and is not made here.
- No other file type moved, measured rather than argued: 0 of 86 K2 corpus files
and 0 of 5 smoke-folder files are HTML, and the smoke bundle is byte-identical
before and after (`diff -r` empty, 52 md / 26 concepts, 0 of 5 rejected).
`okf project` stays byte-equal to `okf build` (`diff -r` empty).
Provenance moves with it: `source_units` routed `.html` through `_line_units`
already, but the table was trivial -- every offset resolved to line 1. The
numbers now mean something, and what they mean is a line of OUR extraction (a
BLOCK), never a line of the original markup.
`_EVIDENCE` gains a `.html` row at `measured`, chosen against the class
definitions: the files are a consumer's own export of a real published handbook,
produced for their ingestion and not to exercise this row. What the class does
not claim travels with it -- one product, one format, one publisher, and a
generator's cut.
One existing test changed because the behaviour changed, and it says so:
`test_html_text_via_htmlparser` asserted the collapsed form. The other two
(`test_html_skips_script_and_style`, `test_htm_is_an_html_alias`) were re-read
and hold unchanged -- the order expected three to move; only one did.
The corpus-wide invariant runs in the suite behind `OKF_HTML_CORPUS`: a corpus
path names a consumer's export and this repository is public.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Kjell Tore Guttormsen
732f84df6e