"""Clause 1 asks whether a title NAMES something, not whether a number ends it. `_TRAILING_PAGE_NUMBER` reads a run of consecutive titles ending in an integer as a contents listing and discards it. That predicate cannot tell a contents entry from a run of DATA ROWS, because a drawing's dimension chain, a P&ID's schematic labels, a door schedule and a borehole log's coordinate column all end in integers too. Measured on the 43-document corpus, clause 1 dropped 68 candidates over 11 of 39 readable documents, and 19 of those 68 over 5 of the 11 were data rows whose titles -- `40.23`, `300 1`, `3000 V 1`, `619649.097` -- name nothing at all. Round 8 removed the DAMAGE (`--close-span-gaps` carries a removed mark's text on the mark above), so what is left is classification. The repair is one clause: a title is a contents entry only if a NAME survives stripping the page number. WHERE THE THRESHOLD SITS IS MEASURED, NOT CHOSEN. Swept over the corpus at an alphabetic run of >= 1, >= 2 and >= 3 characters, and it collapses at both ends: at >= 1 three data rows keep a stray single letter (`3000 V`) and stay misclassified (13 of 19 rescued); at >= 3 a real contents list breaks, because `VA` is a two-letter section name and dropping it out of run membership takes `RIB`, `MMI` and `Tittelfelt` below `CONTENTS_RUN` with it -- one acronym costing four real entries. At >= 2 the rescue is 16 of 19 and the regression 0 of 49 real entries. This file holds BOTH sides, and the second is the one that matters: a rule that rescues a table by also rescuing contents lists has moved the defect rather than fixed it. """ from __future__ import annotations from llm_ingestion_okf import cli from llm_ingestion_okf.propose import find_candidates DEFAULT = dict( outline_run=cli.DEFAULT_OUTLINE_RUN, table_grid=cli.DEFAULT_TABLE_GRID, unit_fold=cli.DEFAULT_UNIT_FOLD, keep_table_heading=cli.DEFAULT_KEEP_TABLE_HEADING, sheet_section_rows=cli.DEFAULT_SHEET_SECTION_ROWS, drop_wrapped_outline=cli.DEFAULT_DROP_WRAPPED_OUTLINE, outline_gate=cli.DEFAULT_OUTLINE_GATE, first_span_from_zero=cli.DEFAULT_FIRST_SPAN_FROM_ZERO, close_span_gaps=cli.DEFAULT_CLOSE_SPAN_GAPS, contents_name=cli.DEFAULT_CONTENTS_NAME, ) #: The borehole-log shape, rebuilt from the corpus document's geometry: a #: coordinate column whose rows are `.` and end in digits. #: Five consecutive rows, comfortably over `CONTENTS_RUN`, each with a body so #: the orphan check does not remove them first. DATA_ROWS = """# Geoteknisk borelogg Testdokument Dette dokumentet har en kjent struktur: en koordinatkolonne der hver rad er et tall, og ingen av dem navngir noe som helst. ## 619482.244 Proevepunkt P20, naverboring, fire proever i sjiktet 0-4 meter. ## 619551.473 Proevepunkt P21, naverboring, tre proever i sjiktet 0-3 meter. ## 619575.846 Proevepunkt P22, naverboring, fem proever i sjiktet 0-5 meter. ## 619649.097 Proevepunkt P23, naverboring, to proever i sjiktet 0-2 meter. ## 619712.518 Proevepunkt P24, naverboring, seks proever i sjiktet 0-6 meter. """ #: The known-negative, and it carries the exact boundary the sweep found: a #: real contents listing whose shortest entry is a TWO-letter section name. #: Every one of these must still be discarded. REAL_CONTENTS = """# Modellansvar Testdokument Innhold ## Ansvar fagmodellansvarlig 5 ## Kontaktpersoner 7 ## RIB 9 ## VA 11 ## MMI 13 ## Tittelfelt 15 Dette avsnittet staar under listen og gir den en kropp. """ def _titles(document: str) -> list[str]: return [candidate.title for candidate in find_candidates(document, **DEFAULT)] def test_a_run_of_numeric_data_rows_is_not_a_contents_listing() -> None: """The known-positive: nothing survives stripping, so nothing is a name.""" titles = _titles(DATA_ROWS) for row in ("619482.244", "619551.473", "619575.846", "619649.097", "619712.518"): assert row in titles, f"clause 1 discarded the data row {row!r} as a contents entry" def test_a_real_contents_listing_is_still_discarded() -> None: """The known-negative, and the reason the threshold is 2 and not 3. `VA` is two letters. At an alphabetic run of >= 3 it stops being a name, falls out of run membership, and the run around it drops below `CONTENTS_RUN` -- so the whole listing survives and four real entries are emitted as concepts. Measured on the corpus, that is exactly what happened. """ titles = _titles(REAL_CONTENTS) for entry in ("Ansvar fagmodellansvarlig", "Kontaktpersoner", "RIB", "VA", "MMI", "Tittelfelt"): assert not any(title.startswith(entry) for title in titles), ( f"clause 1 emitted the contents entry {entry!r} as a concept" )