feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id
Round 9: the four rests in STATE's NESTE that needed no operator decision.
CLAUSE 1 CLASSIFIED BY THE NUMBER, NOT THE TITLE. `_TRAILING_PAGE_NUMBER`
admitted a candidate into a contents run by asking whether the title ended in
an integer -- a question about the number. A drawing's dimension chain, a
schematic's labels, a door schedule, a coordinate column and a soil-layer
table all end in integers and name nothing. Measured over the 43-document
corpus: 68 candidates discarded over 11 of 39 readable documents, of which
19 over 5 documents are data rows.
That corrects round 8's own decomposition. Its "four misclassified numeric
tables and seven real contents listings" needs each document on one side, and
two of the eleven are both. Read across all 68 titles rather than the
three-title sample: 5 documents carry a data row, 8 carry a real entry.
`--contents-name` requires a NAME to survive stripping the page number. The
threshold is SWEPT, not chosen, and collapses at both ends: at an alphabetic
run of 1 a door schedule keeps a stray `V` and 13 of 19 are rescued; at 3 the
two-letter section name `VA` stops being a name, falls out of run membership,
and takes `RIB`, `MMI` and `Tittelfelt` below `CONTENTS_RUN` with it -- one
acronym costing four REAL entries. At 2: 16 of 19 rescued, 0 of 49 regressed.
The three not rescued carry a real word and are named rather than rounded off.
THE CONVERTER'S ANCHOR WAS IN THE CONCEPT ID. Pandoc writes a sheet as
`## <name> {#sheet-N}` and a titled slide as `## <title> {#slide-N}`. Because
a filename is reduced FROM the title, the anchor reached both. Operator
authorised the strip 2026-09-09 after the exposure was counted: 2 of 810
concepts on the previous default bundle, 2 of 1108 on Arm B, 1 of 26 on the
operator's folder. Two ids renamed, one of which `portfolio-optimiser` has
cited in writing; both are in the report so that message can be sent.
One rule in one function, read by BOTH title-forming sites -- a rule in only
one would leave the id and the title naming the same concept differently. The
known-negative is the point: `Mal for {kundenavn}` is a title an author wrote.
odt/rtf/pptx MEASURED END TO END FOR THE FIRST TIME, on hand-built documents,
because the corpus denominator is genuinely zero (86 files: 66 pdf, 10 docx,
4 xlsx, 2 zip, 2 smc, 2 doc). `_EVIDENCE` gains a third class rather than
stretching an existing one: `constructed` means the row has met a document,
but not one anyone wrote for their own purposes. odt 1 of 1 declared headings;
pptx 2 of 2 on a deck that declares slide titles and 0 of 2 on one that does
not -- round 7's reading of pptx was a fixture property, not the format; rtf
0 segments, because the container has no heading style and the author's title
is bold text. rtf is the one open finding.
ACCEPTANCE, all four. The 12-position reference is label-identical in BOTH
readings (pdf 7/8, docx 3/3, xlsx 0/1 or 1/1, sheet 10/12 or 11/12). One K2
bundle carrying both changes: 453 concepts / 865 md, hit@8 [1,1,1,1,1,None]
on it AND on Arm B, with the known-negative still reproducing on the new
bytes. `okf project` byte-equal to `okf build`, `diff -r` empty. Consumer
cost is a re-run: 436/832 -> 453/865, digest 21af4a1aa98315cf.
Three published numbers corrected: README's 596 tests (1515), README's "15
concepts out" for `okf project` (that was the O6 defect; it is 26), and O6's
print-mode method, which does not reproduce without --allowedTools.
Report: docs/2026-09-09-k3-runde9-restene.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
fbad5b0eab
commit
6ff18fd703
15 changed files with 964 additions and 67 deletions
123
tests/test_contents_name_part.py
Normal file
123
tests/test_contents_name_part.py
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
"""Clause 1 asks whether a title NAMES something, not whether a number ends it.
|
||||
|
||||
`_TRAILING_PAGE_NUMBER` reads a run of consecutive titles ending in an integer
|
||||
as a contents listing and discards it. That predicate cannot tell a contents
|
||||
entry from a run of DATA ROWS, because a drawing's dimension chain, a P&ID's
|
||||
schematic labels, a door schedule and a borehole log's coordinate column all
|
||||
end in integers too. Measured on the 43-document corpus, clause 1 dropped 68
|
||||
candidates over 11 of 39 readable documents, and 19 of those 68 over 5 of the
|
||||
11 were data rows whose titles -- `40.23`, `300 1`, `3000 V 1`, `619649.097`
|
||||
-- name nothing at all.
|
||||
|
||||
Round 8 removed the DAMAGE (`--close-span-gaps` carries a removed mark's text
|
||||
on the mark above), so what is left is classification. The repair is one
|
||||
clause: a title is a contents entry only if a NAME survives stripping the page
|
||||
number.
|
||||
|
||||
WHERE THE THRESHOLD SITS IS MEASURED, NOT CHOSEN. Swept over the corpus at an
|
||||
alphabetic run of >= 1, >= 2 and >= 3 characters, and it collapses at both
|
||||
ends: at >= 1 three data rows keep a stray single letter (`3000 V`) and stay
|
||||
misclassified (13 of 19 rescued); at >= 3 a real contents list breaks, because
|
||||
`VA` is a two-letter section name and dropping it out of run membership takes
|
||||
`RIB`, `MMI` and `Tittelfelt` below `CONTENTS_RUN` with it -- one acronym
|
||||
costing four real entries. At >= 2 the rescue is 16 of 19 and the regression 0
|
||||
of 49 real entries. This file holds BOTH sides, and the second is the one that
|
||||
matters: a rule that rescues a table by also rescuing contents lists has moved
|
||||
the defect rather than fixed it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from llm_ingestion_okf import cli
|
||||
from llm_ingestion_okf.propose import find_candidates
|
||||
|
||||
DEFAULT = dict(
|
||||
outline_run=cli.DEFAULT_OUTLINE_RUN,
|
||||
table_grid=cli.DEFAULT_TABLE_GRID,
|
||||
unit_fold=cli.DEFAULT_UNIT_FOLD,
|
||||
keep_table_heading=cli.DEFAULT_KEEP_TABLE_HEADING,
|
||||
sheet_section_rows=cli.DEFAULT_SHEET_SECTION_ROWS,
|
||||
drop_wrapped_outline=cli.DEFAULT_DROP_WRAPPED_OUTLINE,
|
||||
outline_gate=cli.DEFAULT_OUTLINE_GATE,
|
||||
first_span_from_zero=cli.DEFAULT_FIRST_SPAN_FROM_ZERO,
|
||||
close_span_gaps=cli.DEFAULT_CLOSE_SPAN_GAPS,
|
||||
contents_name=cli.DEFAULT_CONTENTS_NAME,
|
||||
)
|
||||
|
||||
#: The borehole-log shape, rebuilt from the corpus document's geometry: a
|
||||
#: coordinate column whose rows are `<northing>.<decimals>` and end in digits.
|
||||
#: Five consecutive rows, comfortably over `CONTENTS_RUN`, each with a body so
|
||||
#: the orphan check does not remove them first.
|
||||
DATA_ROWS = """# Geoteknisk borelogg Testdokument
|
||||
|
||||
Dette dokumentet har en kjent struktur: en koordinatkolonne der hver rad er
|
||||
et tall, og ingen av dem navngir noe som helst.
|
||||
|
||||
## 619482.244
|
||||
|
||||
Proevepunkt P20, naverboring, fire proever i sjiktet 0-4 meter.
|
||||
|
||||
## 619551.473
|
||||
|
||||
Proevepunkt P21, naverboring, tre proever i sjiktet 0-3 meter.
|
||||
|
||||
## 619575.846
|
||||
|
||||
Proevepunkt P22, naverboring, fem proever i sjiktet 0-5 meter.
|
||||
|
||||
## 619649.097
|
||||
|
||||
Proevepunkt P23, naverboring, to proever i sjiktet 0-2 meter.
|
||||
|
||||
## 619712.518
|
||||
|
||||
Proevepunkt P24, naverboring, seks proever i sjiktet 0-6 meter.
|
||||
"""
|
||||
|
||||
#: The known-negative, and it carries the exact boundary the sweep found: a
|
||||
#: real contents listing whose shortest entry is a TWO-letter section name.
|
||||
#: Every one of these must still be discarded.
|
||||
REAL_CONTENTS = """# Modellansvar Testdokument
|
||||
|
||||
Innhold
|
||||
|
||||
## Ansvar fagmodellansvarlig 5
|
||||
|
||||
## Kontaktpersoner 7
|
||||
|
||||
## RIB 9
|
||||
|
||||
## VA 11
|
||||
|
||||
## MMI 13
|
||||
|
||||
## Tittelfelt 15
|
||||
|
||||
Dette avsnittet staar under listen og gir den en kropp.
|
||||
"""
|
||||
|
||||
|
||||
def _titles(document: str) -> list[str]:
|
||||
return [candidate.title for candidate in find_candidates(document, **DEFAULT)]
|
||||
|
||||
|
||||
def test_a_run_of_numeric_data_rows_is_not_a_contents_listing() -> None:
|
||||
"""The known-positive: nothing survives stripping, so nothing is a name."""
|
||||
titles = _titles(DATA_ROWS)
|
||||
for row in ("619482.244", "619551.473", "619575.846", "619649.097", "619712.518"):
|
||||
assert row in titles, f"clause 1 discarded the data row {row!r} as a contents entry"
|
||||
|
||||
|
||||
def test_a_real_contents_listing_is_still_discarded() -> None:
|
||||
"""The known-negative, and the reason the threshold is 2 and not 3.
|
||||
|
||||
`VA` is two letters. At an alphabetic run of >= 3 it stops being a name,
|
||||
falls out of run membership, and the run around it drops below
|
||||
`CONTENTS_RUN` -- so the whole listing survives and four real entries are
|
||||
emitted as concepts. Measured on the corpus, that is exactly what happened.
|
||||
"""
|
||||
titles = _titles(REAL_CONTENTS)
|
||||
for entry in ("Ansvar fagmodellansvarlig", "Kontaktpersoner", "RIB", "VA", "MMI", "Tittelfelt"):
|
||||
assert not any(title.startswith(entry) for title in titles), (
|
||||
f"clause 1 emitted the contents entry {entry!r} as a concept"
|
||||
)
|
||||
Loading…
Add table
Add a link
Reference in a new issue