fix(extract): HTML collapsed to one line, so 828 of 828 sections had no boundary

`_HTMLTextExtractor.text()` was `" ".join("".join(parts).split())`. `str.split()`
with no argument splits on newlines too, so extraction of ANY HTML file returned
unconditionally one line. Every boundary grammar in `propose` is line-anchored
(`_ATX`, `_NUMBERED`, `_TABLE_ROW`, `_GRID_RULE`, `_OUTLINE`, each with `^`), and
on one line at most the first can match while a match at line 0 opens no interior
boundary. Measured outside this repo on a consumer's export of a published
handbook: 83 / 414 / 828 `.html` files gave 0 plans, N documents with no boundary
and exit 2 at every point, and a coarser 145-document cut gave 145 of 145. The
same sections as markdown gave 828 of 828 plans -- so the instrument was fine and
`.html` was the one core-supported type that had never met a real document.

Block tags now open lines of their own, `h1`-`h6` carry the ATX marker for their
own level (not a flat `#`, which would hand `_ATX` three top-level boundaries
where the document declares one section and two subsections), `br` breaks the
line, and every other tag stays the word boundary it already was. The output
grammar is MARKDOWN and deliberately the same markdown the office rows reach the
proposer through, so no HTML-only heading grammar exists.

NOT via the converter: `.html` stays out of `_PANDOC_FORMATS` because routing it
there would add CVE-2025-51591 (SSRF via an iframe in HTML input), unpatched in
every converter version. The test asserting that exclusion is untouched and green.

The block set is wider than the five tags the corpus exercises, on purpose:
block versus inline is a property of HTML, not of one corpus, and a `div`-
structured page carries its prose in containers this corpus never uses.

Measured after, all with denominators:
- 828 of 828 plans, exit 0, `merged + coded rejections = 828; N = 828`; 3206
  concepts / 6015 md files, which is the markdown path's count EXACTLY -- 0.0 %
  deviation against the +/-2 % bar, and the same at 50 % (1651) and 10 % (343).
  The coarser 145-document cut goes 145 of 145 with no boundary to 145 plans /
  953 concepts.
- Text preservation as an EXACT invariant, not a percentage: strip the added ATX
  markers and the non-whitespace sequence is identical to the old extractor's for
  the same bytes. 828 of 828 files exact, character ratio 1.000000 against the
  >= 99.8 % bar. 7600 markers added; 31 141 lines produced where the old
  extractor produced 828, one per file.
- `_SKIP_TAGS` unchanged at {script, style}. Dropping nav/header/footer is a
  different change with a different guarantee and is not made here.
- No other file type moved, measured rather than argued: 0 of 86 K2 corpus files
  and 0 of 5 smoke-folder files are HTML, and the smoke bundle is byte-identical
  before and after (`diff -r` empty, 52 md / 26 concepts, 0 of 5 rejected).
  `okf project` stays byte-equal to `okf build` (`diff -r` empty).

Provenance moves with it: `source_units` routed `.html` through `_line_units`
already, but the table was trivial -- every offset resolved to line 1. The
numbers now mean something, and what they mean is a line of OUR extraction (a
BLOCK), never a line of the original markup.

`_EVIDENCE` gains a `.html` row at `measured`, chosen against the class
definitions: the files are a consumer's own export of a real published handbook,
produced for their ingestion and not to exercise this row. What the class does
not claim travels with it -- one product, one format, one publisher, and a
generator's cut.

One existing test changed because the behaviour changed, and it says so:
`test_html_text_via_htmlparser` asserted the collapsed form. The other two
(`test_html_skips_script_and_style`, `test_htm_is_an_html_alias`) were re-read
and hold unchanged -- the order expected three to move; only one did.

The corpus-wide invariant runs in the suite behind `OKF_HTML_CORPUS`: a corpus
path names a consumer's export and this repository is public.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

tests/test_extract.py

@@ -11,6 +11,7 @@ from __future__ import annotations
 
 import hashlib
 import importlib.util
+import os
 import re
 import sys
 from pathlib import Path
@@ -58,20 +59,178 @@ def test_json_is_verbatim_inside_a_fenced_block() -> None:
 
 
 def test_html_text_via_htmlparser() -> None:
-    # Block boundaries separate words; tags themselves contribute no text.
+    """CHANGED 2026-09-09 BECAUSE THE BEHAVIOUR CHANGED, not to go green.
+
+    This asserted the collapsed one-line form -- the very form that made 828
+    of 828 real HTML sections unsegmentable. Block tags now open lines and
+    headings carry their ATX level; inline tags are still word boundaries,
+    which is the half of the old assertion that still holds and is still here.
+    """
     out = extract_text("page.html", b"<h1>Title</h1><p>Hello <b>world</b></p>")
-    assert out == "Title Hello world"
+    assert out == "# Title\nHello world"
 
 
 def test_html_skips_script_and_style() -> None:
+    """UNCHANGED across round 11, and re-read to confirm it: `_SKIP_TAGS` still
+    holds exactly `script` and `style`, and a single block still renders as a
+    single line, so the old expectation is still the right one."""
     html = b"<style>.x{color:red}</style><p>Keep</p><script>evil()</script>"
     assert extract_text("page.html", html) == "Keep"
 
 
 def test_htm_is_an_html_alias() -> None:
+    """UNCHANGED across round 11, same reason as the test above."""
     assert extract_text("page.htm", b"<p>hi</p>") == "hi"
 
 
+# --- html: block structure survives extraction (round 11) -------------------
+#
+# The defect this closes, measured OUTSIDE this repo on 828 real sections:
+# `text()` used to be `" ".join("".join(parts).split())`, and `str.split()`
+# with no argument splits on newlines too, so every HTML file extracted to
+# UNCONDITIONALLY ONE LINE. Every boundary grammar in `propose` is
+# line-anchored (`_ATX`, `_NUMBERED`, `_TABLE_ROW`, `_GRID_RULE`, `_OUTLINE`,
+# each with `^`), so 828 of 828 documents got zero boundaries and the run
+# exited 2. The same 828 sections as markdown gave 828 plans.
+
+_ADDED_ATX = re.compile(r"(?m)^#{1,6} ")
+
+_HTML_KNOWN_POSITIVE = b"""<!doctype html>
+<html><head><title>Doc</title></head><body>
+<h1>Top</h1>
+<p>Intro <b>bold</b> word.</p>
+<h2>First part</h2>
+<ul><li>alpha</li><li>beta</li></ul>
+<h2>Second part</h2>
+<table><tr><td>cell one</td><td>cell two</td></tr><tr><td>r2c1</td><td>r2c2</td></tr></table>
+</body></html>
+"""
+
+
+def _legacy_html_text(data: bytes) -> str:
+    """The pre-round-11 extractor, kept HERE as the invariant's reference.
+
+    The text-preservation invariant is stated against what the old extractor
+    returned for the same bytes, so it needs that string. Reproducing the ten
+    lines in the test is the only way to keep both forms available from one
+    definition of "what the old code did" -- exporting a second renderer from
+    the module would put a function in the package that nothing ships uses.
+    """
+    from html.parser import HTMLParser
+
+    from llm_ingestion_okf.extract import _SKIP_TAGS
+
+    class _Legacy(HTMLParser):
+        def __init__(self) -> None:
+            super().__init__(convert_charrefs=True)
+            self.parts: list[str] = []
+            self.skip = 0
+
+        def handle_starttag(self, tag: str, attrs: object) -> None:  # type: ignore[override]
+            self.parts.append(" ")
+            if tag in _SKIP_TAGS:
+                self.skip += 1
+
+        def handle_startendtag(self, tag: str, attrs: object) -> None:  # type: ignore[override]
+            self.parts.append(" ")
+
+        def handle_endtag(self, tag: str) -> None:
+            if tag in _SKIP_TAGS and self.skip > 0:
+                self.skip -= 1
+            self.parts.append(" ")
+
+        def handle_data(self, data: str) -> None:
+            if self.skip == 0:
+                self.parts.append(data)
+
+    parser = _Legacy()
+    parser.feed(data.decode("utf-8-sig"))
+    parser.close()
+    return " ".join("".join(parser.parts).split())
+
+
+def test_html_headings_become_atx_lines_at_the_tag_s_own_level() -> None:
+    """The LEVEL is the tag's, not a flat `#` for every heading.
+
+    A flat prefix would give `propose._ATX` three level-1 boundaries where the
+    document declares one section and two subsections, so the assertion is on
+    the line CONTENT, never on "there are now several lines".
+    """
+    out = extract_text("page.html", _HTML_KNOWN_POSITIVE)
+    lines = out.split("\n")
+    assert "# Top" in lines
+    assert "## First part" in lines
+    assert "## Second part" in lines
+    # Blocks are their own lines; inline markup stays a word boundary.
+    assert "Intro bold word." in lines
+    assert "alpha" in lines
+    assert "beta" in lines
+    # A table row is one line, its cells separated by a space.
+    assert "cell one cell two" in lines
+    assert "r2c1 r2c2" in lines
+
+
+def test_html_without_a_heading_is_one_concept_s_worth_of_text_never_zero() -> None:
+    """KNOWN-NEGATIVE: no heading in, no fabricated heading out.
+
+    The `rtf` row's honest result, and the shape this must keep: a document
+    that declares nothing lands as ONE concept with its content preserved --
+    never zero concepts and never zero characters.
+    """
+    out = extract_text("flat.html", b"<p>First para.</p><p>Second para.</p>")
+    assert not _ADDED_ATX.search(out)
+    assert out.split("\n") == ["First para.", "Second para."]
+
+
+def test_html_line_structure_preserves_every_non_whitespace_character() -> None:
+    """Text preservation is an EXACT invariant, not a percentage.
+
+    Strip the ATX markers this extractor added and the non-whitespace sequence
+    must be identical to what the old extractor returned for the same bytes.
+    """
+    new = extract_text("page.html", _HTML_KNOWN_POSITIVE)
+    assert _ADDED_ATX.search(new), "known-positive: the query must be able to find a marker"
+    stripped = _ADDED_ATX.sub("", new)
+    assert "".join(stripped.split()) == "".join(_legacy_html_text(_HTML_KNOWN_POSITIVE).split())
+
+
+@pytest.mark.skipif(
+    not os.environ.get("OKF_HTML_CORPUS"),
+    reason=(
+        "set OKF_HTML_CORPUS to a directory of .html files to run the "
+        "corpus-wide invariant. The path is an ENVIRONMENT VARIABLE and not a "
+        "constant here on purpose: a corpus path names a consumer's own "
+        "export, and this repository is public"
+    ),
+)
+def test_the_invariant_holds_over_a_whole_corpus_not_one_document() -> None:
+    """One document proves the invariant is reachable, not that it holds.
+
+    Run over the SK1 corpus (828 `.html` sections of a published handbook) this
+    was 828 of 828 exact and a character ratio of 1.000000 against SK1's own
+    >= 99.8 % bar, with 7600 ATX markers added and 31 141 lines produced where
+    the old extractor produced 828 -- one per file, which was the defect.
+    """
+    root = Path(os.environ["OKF_HTML_CORPUS"])
+    files = sorted(root.rglob("*.html"))
+    assert files, f"no .html under {root}: the corpus pointer is wrong, not the code"
+    markers = 0
+    for path in files:
+        data = path.read_bytes()
+        new = extract_text(path.name, data)
+        markers += len(_ADDED_ATX.findall(new))
+        stripped = _ADDED_ATX.sub("", new)
+        assert "".join(stripped.split()) == "".join(_legacy_html_text(data).split()), (
+            f"text preservation broke on {path.name}"
+        )
+    assert markers > 0, "known-positive: a corpus of prose must yield some heading"
+
+
+def test_html_br_is_a_line_break_and_inline_tags_are_not() -> None:
+    out = extract_text("page.html", b"<p>one<br>two <em>three</em> four</p>")
+    assert out.split("\n") == ["one", "two three four"]
+
+
 def test_extension_dispatch_is_case_insensitive() -> None:
     assert extract_text("NOTE.MD", b"x") == "x"
 
@@ -175,11 +334,19 @@ def test_evidence_class_is_asserted_not_commented() -> None:
     `.pptx`, `.odt` and `.rtf` have denominator ZERO in the corpus this arm was
     measured on. A comment saying so rots; an assertion that names them keeps
     an unmeasured row from quietly presenting as a supported one.
+
+    The table is NOT the converter's rows alone: `.html` is core-supported and
+    was the one core row that had never met a real document, so it carries a
+    class here too (2026-09-09, `measured`, 828 files).
     """
     from llm_ingestion_okf.extract import _EVIDENCE, _PANDOC_FORMATS
 
-    assert set(_EVIDENCE) == set(_PANDOC_FORMATS), "every row needs an evidence class"
-    assert {s for s, e in _EVIDENCE.items() if e == "measured"} == {".docx", ".xlsx"}
+    assert set(_EVIDENCE) == set(_PANDOC_FORMATS) | {".html"}, "every row needs a class"
+    assert {s for s, e in _EVIDENCE.items() if e == "measured"} == {
+        ".docx",
+        ".xlsx",
+        ".html",
+    }
     # Since 2026-09-09 the three office rows are `constructed`, not
     # `unmeasured`: each has now been put through end to end on a hand-built
     # document with a hand-written fasit, and none of them has a corpus file.

tests/test_provenance.py

@@ -145,6 +145,27 @@ def test_a_text_file_reports_line_numbers_of_the_original() -> None:
     assert units.covering(0, len(text)) == (1, 3)
 
 
+def test_an_html_locator_is_lines_of_our_extraction_not_of_the_source_file() -> None:
+    """The `.html` unit table stopped being trivial in round 11.
+
+    It always existed -- `.html` is in `_CORE_EXTRACTORS`, so `source_units`
+    routed it through `_line_units` -- but the extractor returned ONE line for
+    any input, so every offset in every HTML concept resolved to line 1. Now
+    the numbers mean something, and what they mean is a line of OUR extraction:
+    the source file below is four physical lines and the extraction is three,
+    because a line is a BLOCK here, not a line of the original markup.
+    """
+    data = b"<html><body>\n<h1>Top</h1>\n<p>first para</p>\n<p>second para</p>\n</body></html>"
+    text = extract_text("page.html", data)
+    units = source_units("page.html", data, text)
+    assert units is not None
+    assert units.unit == "lines"
+    assert text.split("\n") == ["# Top", "first para", "second para"]
+    assert units.covering(0, 5) == (1, 1)
+    assert units.covering(text.index("second"), len(text)) == (3, 3)
+    assert units.covering(0, len(text)) == (1, 3)
+
+
 def test_a_docx_locator_is_lines_because_paragraphs_do_not_survive() -> None:
     # Measured on the five K2 `.docx` documents: `<w:p>` counts of 108, 27, 65,
     # 176 and 57 against converted-markdown line counts of 75, 33, 67, 144 and