open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(fixtures): the STS fixtures and fixture codes are fictitious

Browse source 
Three STS fixtures still carried the section titles and labels of one real
reference document, and three identifiers were copies of its codes with a
letter or a word swapped. They now describe an invented kitchen counter and
cookbook series: the titles, labels and descriptions of sts-identity.xml,
sts-inherit.xml and sts-empty-label.xml, the P350/P351 document codes, the
99-0001 delivery prefix and chapter 7 of the image and accounting corpora.
Generated fixtures are regenerated and the witness inventory's per-document
totals are identical before and after; only names and text move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 14:50:44 +02:00
commit 88cf67f12e
65 changed files with 396 additions and 390 deletions
Download patch file Download diff file Expand all files Collapse all files

22
tests/test_xml_sts.py
View file

@ -4,24 +4,24 @@ MEASURED OUTSIDE THIS REPOSITORY AND REPRODUCED HERE: the zip a publisher's
own viewer delivers as "Html" holds 0 html, 1 xml and 109 images, and `okf
build` on it gave **110 of 110 unreadable, 0 plans, exit 2** -- `.xml` was in
neither extractor registry. The one xml file IS the whole product: a large
regulatory reference document, the same one round 12 met as a 701-page PDF,
in NISO-STS form. It carries 7 715 `<sec>`, **2 761 of them with a `<title>`**,
and a `<sec>`-nesting depth distribution of 28/118/500/1141/868/97/9 that is
row-for-row identical to the fasit built from that same file.
regulatory reference document, the same one round 12 met as a PDF,
in NISO-STS form. It carries several thousand `<sec>`, **N (36 %) of them with a
`<title>`**, and a `<sec>`-nesting depth distribution that is row for row the
fasit's own, the fasit being built from that same file.
THE CEILING IS THEREFORE STRUCTURAL AND NOT COMPUTED: 2 761 of 2 761. The PDF
arm reached 2 759 of 2 761 by bridging from (page, y) to a line index; here the
THE CEILING IS THEREFORE STRUCTURAL AND NOT COMPUTED: N of N. The PDF
arm reached N - 2 of N (99.9 %) by bridging from (page, y) to a line index; here the
publisher states the structure in elements, so nothing is recovered and nothing
is guessed.
WHAT THESE TESTS PIN, each because the measurement said it could go wrong:
- `<label>` carries the number and `<title>` carries the text. Emitting only
`<title>` scores 0 of 2 761 while every line of the code looks right, because
`<title>` scores 0 of N while every line of the code looks right, because
the number is what okf reduces to a directory name.
- a `<sec>` with a `<label>` and NO `<title>` is never a heading. 4 954 of the
7 715 are lettered points (`a)`, `c)`) inside a process description, and one
heading each would bury the document's own 2 761.
- a `<sec>` with a `<label>` and NO `<title>` is never a heading. Most (64 %) of
the `<sec>` are lettered points (`a)`, `c)`) inside a process description, and one
heading each would bury the document's own N.
- the ONE titled section with no label -- `Forord` -- emitted without a
numbering token, because the corrected instrument key looks it up by title.
- a `<table-wrap>` as ONE markdown table block. The PDF path delivered 0 of 10.
@ -111,7 +111,7 @@ def test_the_label_and_the_title_reach_the_proposer_as_one_heading() -> None:
def test_a_lettered_point_is_a_body_line_and_never_a_heading() -> None:
"""4 954 of the reference standard's 7 715 `<sec>` are these. One heading each buries 2 761."""
"""64 % of the reference standard's `<sec>` are these. One heading each buries its N."""
text = extract.extract_text(STS.name, STS.read_bytes())
assert "a) Betong skal ha fasthetsklasse B35." in text.split("\n")