open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(fixtures): replace sector-specific example material with generic, fictitious examples — green

Browse source 
Every fixture, test document, tool example and document now uses an invented
kitchen-and-baking handbook series, written in this repository. The package's
behaviour is unchanged; src/ changes are comments and help text only.

- Generated fixtures are regenerated from their generators. Their structural
  counts are identical before and after: elements, images, rows, cells,
  headings, bookmarks and the witness inventory's per-document totals. The
  image-inbox and accounting documents are renamed kapittel-84-*.
- tools/okf_accounting_gate.py: the two options that named one real corpus
  each are replaced by a generic, repeatable --corpus PATH with no default.
  Row 5 compares the PDF pair alone. Gate verdict unchanged: RED rows 2, 3, 6.
- tools/okf_witness.py: the STS JSON reader for one publisher's delivery is
  removed, along with its three twins and five tests. The mutation harness
  loses W09.
- docs/: 13 dated reports that documented runs on a retired reference corpus
  are removed, and 40 are neutralized. Dead links are removed, and no new
  dangling path is introduced.
- The synthetic MCP-gate corpus and the residual probe words are neutral.

Valgt: keep the `okf quality --fasit` bar value (the measured fraction, one corpus) and
rewrite only its provenance, because the verdict stays unchanged and the
number names nothing.

Term check with the local list: 0 of 411 tracked files, 0 file names, 0 of
27 binary fixtures. Suite after git add: 2457 passed, 1 skipped. The base
tree had 2460 passed and 2 skipped; five tests went with the JSON reader and
four were added by the term check. ruff, ruff format and mypy --strict src/
are clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 13:54:57 +02:00
commit 9d1f4b14ed
174 changed files with 1889 additions and 6512 deletions
Download patch file Download diff file Expand all files Collapse all files

12
tests/test_stem_prefix.py
View file

@ -8,7 +8,7 @@ FIRST every time:
word equality prefix share
under (control) 79 172 38.0 %
undersjoisk 0 172 38.0 %
under… 0 172 38.0 %
bilateral 0 400 88.3 %
standhaftig 0 219 48.3 %
@ -39,11 +39,11 @@ WHAT DOES SEPARATE THEM. `pris` is a word; `bila` and `stan` are not. The
fourth candidate requires the shared prefix to occur as a token in the bundle's
own concepts, and it is the only one that clears both criteria at once:
bundle bilateral standhaftig undersjoisk control ranks
bundle bilateral standhaftig under… control ranks
default-20260912 400 -> 0 219 -> 56 172 -> 162 172 (1,1,1,1,1,None)
armB-20260903 512 -> 0 235 -> 33 174 -> 160 174 (1,1,1,1,1,None)
WHAT REMAINS, AND WHY IT IS NOT A CEILING BUT A DIFFERENT ANSWER. `undersjoisk`
WHAT REMAINS, AND WHY IT IS NOT A CEILING BUT A DIFFERENT ANSWER. `under…`
still reaches 162 concepts, because it shares `under` with them and `under` IS
a word in this corpus -- a productive Norwegian prefix. `standhaftig` still
reaches 56 through `stand`. Those are genuine shared morphemes, and a rule that
@ -98,13 +98,13 @@ def test_a_shared_prefix_that_is_a_word_still_matches() -> None:
def test_the_productive_norwegian_prefix_is_kept_and_that_is_the_answer() -> None:
"""`under` is a word, so `undersjoisk` keeps reaching `underbygning`.
"""`under` is a word, so `under…` keeps reaching `underholdning`.
Stated as a decision rather than left as a residual: this rule does not
claim to separate two words that genuinely share a morpheme, and the 162
concepts `undersjoisk` still reaches are that morpheme, not a defect.
concepts `under…` still reaches are that morpheme, not a defect.
"""
assert consume.tokens_match("undersjøisk", "underbygning", stems=STEMS) is True
assert consume.tokens_match("undertøy", "underholdning", stems=STEMS) is True
def test_an_identifier_is_untouched_by_the_stem_rule() -> None: