open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(fixtures): the STS fixtures and fixture codes are fictitious

Browse source 
Three STS fixtures still carried the section titles and labels of one real
reference document, and three identifiers were copies of its codes with a
letter or a word swapped. They now describe an invented kitchen counter and
cookbook series: the titles, labels and descriptions of sts-identity.xml,
sts-inherit.xml and sts-empty-label.xml, the P350/P351 document codes, the
99-0001 delivery prefix and chapter 7 of the image and accounting corpora.
Generated fixtures are regenerated and the witness inventory's per-document
totals are identical before and after; only names and text move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 14:50:44 +02:00
commit 88cf67f12e
65 changed files with 396 additions and 390 deletions
Download patch file Download diff file Expand all files Collapse all files

22
tests/test_extract.py
View file

@ -61,8 +61,8 @@ def test_json_is_verbatim_inside_a_fenced_block() -> None:
def test_html_text_via_htmlparser() -> None:
"""CHANGED 2026-09-09 BECAUSE THE BEHAVIOUR CHANGED, not to go green.
This asserted the collapsed one-line form -- the very form that made 828
of 828 real HTML sections unsegmentable. Block tags now open lines and
This asserted the collapsed one-line form -- the very form that made every
one of M real HTML sections unsegmentable. Block tags now open lines and
headings carry their ATX level; inline tags are still word boundaries,
which is the half of the old assertion that still holds and is still here.
"""
@ -85,13 +85,13 @@ def test_htm_is_an_html_alias() -> None:
# --- html: block structure survives extraction (round 11) -------------------
#
# The defect this closes, measured OUTSIDE this repo on 828 real sections:
# The defect this closes, measured OUTSIDE this repo on M real sections:
# `text()` used to be `" ".join("".join(parts).split())`, and `str.split()`
# with no argument splits on newlines too, so every HTML file extracted to
# UNCONDITIONALLY ONE LINE. Every boundary grammar in `propose` is
# line-anchored (`_ATX`, `_NUMBERED`, `_TABLE_ROW`, `_GRID_RULE`, `_OUTLINE`,
# each with `^`), so 828 of 828 documents got zero boundaries and the run
# exited 2. The same 828 sections as markdown gave 828 plans.
# each with `^`), so M of M documents got zero boundaries and the run
# exited 2. The same M sections as markdown gave M plans.
_ADDED_ATX = re.compile(r"(?m)^#{1,6} ")
@ -206,10 +206,10 @@ def test_html_line_structure_preserves_every_non_whitespace_character() -> None:
def test_the_invariant_holds_over_a_whole_corpus_not_one_document() -> None:
"""One document proves the invariant is reachable, not that it holds.
Run over the SK1 corpus (828 `.html` sections of a published handbook) this
was 828 of 828 exact and a character ratio of 1.000000 against SK1's own
Run over the SK1 corpus (M `.html` sections of a published handbook) this
was M of M exact and a character ratio of 1.000000 against SK1's own
>= 99.8 % bar, with 7600 ATX markers added and 31 141 lines produced where
the old extractor produced 828 -- one per file, which was the defect.
the old extractor produced M -- one per file, which was the defect.
"""
root = Path(os.environ["OKF_HTML_CORPUS"])
files = sorted(root.rglob("*.html"))
@ -344,9 +344,9 @@ def test_evidence_class_is_asserted_not_commented() -> None:
The table is NOT the converter's rows alone: `.html` and `.pdf` are
core-supported and carry a class here too -- `.html` since 2026-09-09
(`measured`, 828 files), `.pdf` since 2026-09-10 (`measured`, eight corpus
documents with a hand-counted fasit plus a publisher's own 2 761-section
structure for a 701-page one). `.pdf` was the row with the most measurement
(`measured`, M files), `.pdf` since 2026-09-10 (`measured`, eight corpus
documents with a hand-counted fasit plus a publisher's own section
structure for a long one). `.pdf` was the row with the most measurement
behind it and no row in the table at all, which is the one way a table like
this can mislead while every entry in it is true.
"""