feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id
Round 9: the four rests in STATE's NESTE that needed no operator decision.
CLAUSE 1 CLASSIFIED BY THE NUMBER, NOT THE TITLE. `_TRAILING_PAGE_NUMBER`
admitted a candidate into a contents run by asking whether the title ended in
an integer -- a question about the number. A drawing's dimension chain, a
schematic's labels, a door schedule, a coordinate column and a soil-layer
table all end in integers and name nothing. Measured over the 43-document
corpus: 68 candidates discarded over 11 of 39 readable documents, of which
19 over 5 documents are data rows.
That corrects round 8's own decomposition. Its "four misclassified numeric
tables and seven real contents listings" needs each document on one side, and
two of the eleven are both. Read across all 68 titles rather than the
three-title sample: 5 documents carry a data row, 8 carry a real entry.
`--contents-name` requires a NAME to survive stripping the page number. The
threshold is SWEPT, not chosen, and collapses at both ends: at an alphabetic
run of 1 a door schedule keeps a stray `V` and 13 of 19 are rescued; at 3 the
two-letter section name `VA` stops being a name, falls out of run membership,
and takes `RIB`, `MMI` and `Tittelfelt` below `CONTENTS_RUN` with it -- one
acronym costing four REAL entries. At 2: 16 of 19 rescued, 0 of 49 regressed.
The three not rescued carry a real word and are named rather than rounded off.
THE CONVERTER'S ANCHOR WAS IN THE CONCEPT ID. Pandoc writes a sheet as
`## <name> {#sheet-N}` and a titled slide as `## <title> {#slide-N}`. Because
a filename is reduced FROM the title, the anchor reached both. Operator
authorised the strip 2026-09-09 after the exposure was counted: 2 of 810
concepts on the previous default bundle, 2 of 1108 on Arm B, 1 of 26 on the
operator's folder. Two ids renamed, one of which `portfolio-optimiser` has
cited in writing; both are in the report so that message can be sent.
One rule in one function, read by BOTH title-forming sites -- a rule in only
one would leave the id and the title naming the same concept differently. The
known-negative is the point: `Mal for {kundenavn}` is a title an author wrote.
odt/rtf/pptx MEASURED END TO END FOR THE FIRST TIME, on hand-built documents,
because the corpus denominator is genuinely zero (86 files: 66 pdf, 10 docx,
4 xlsx, 2 zip, 2 smc, 2 doc). `_EVIDENCE` gains a third class rather than
stretching an existing one: `constructed` means the row has met a document,
but not one anyone wrote for their own purposes. odt 1 of 1 declared headings;
pptx 2 of 2 on a deck that declares slide titles and 0 of 2 on one that does
not -- round 7's reading of pptx was a fixture property, not the format; rtf
0 segments, because the container has no heading style and the author's title
is bold text. rtf is the one open finding.
ACCEPTANCE, all four. The 12-position reference is label-identical in BOTH
readings (pdf 7/8, docx 3/3, xlsx 0/1 or 1/1, sheet 10/12 or 11/12). One K2
bundle carrying both changes: 453 concepts / 865 md, hit@8 [1,1,1,1,1,None]
on it AND on Arm B, with the known-negative still reproducing on the new
bytes. `okf project` byte-equal to `okf build`, `diff -r` empty. Consumer
cost is a re-run: 436/832 -> 453/865, digest 21af4a1aa98315cf.
Three published numbers corrected: README's 596 tests (1515), README's "15
concepts out" for `okf project` (that was the O6 defect; it is 26), and O6's
print-mode method, which does not reproduce without --allowedTools.
Report: docs/2026-09-09-k3-runde9-restene.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
fbad5b0eab
commit
6ff18fd703
15 changed files with 964 additions and 67 deletions
169
tests/test_converter_attribute_titles.py
Normal file
169
tests/test_converter_attribute_titles.py
Normal file
|
|
@ -0,0 +1,169 @@
|
|||
"""A converter's own attribute is not part of a document's title.
|
||||
|
||||
Converting a spreadsheet, pandoc writes each sheet as
|
||||
`## <sheet name> {#sheet-<n>}`, and a deck whose slides carry title
|
||||
placeholders as `## <slide title> {#slide-<n>}`. The brace block is pandoc's
|
||||
attribute syntax -- an anchor it emits so its own HTML output can link to the
|
||||
section. It is not something the author of the spreadsheet or the deck wrote.
|
||||
|
||||
It reached the artifact twice over: into a concept's `title`, and from there
|
||||
into the concept's id, because a filename is REDUCED from the title. Measured
|
||||
2026-09-09 on the two bundles on disk, `2 of 810` concepts on the default K2
|
||||
bundle and `2 of 1108` on Arm B carried it, and the ids were
|
||||
`del-ii-bilag-7-prisskjema/prissammenstilling-sheet-1` and
|
||||
`del-ii-bilag-0-dokumentliste-del-ii/ark1-sheet-1`.
|
||||
|
||||
Round 7 measured this and left it, because the repair RENAMES concept ids
|
||||
another repository has cited in writing, and renaming those is not a
|
||||
measurement's call. The operator authorised the strip on 2026-09-09; the
|
||||
exposure is in the round-9 report with both ids side by side, before and
|
||||
after.
|
||||
|
||||
TWO TITLE SITES, ONE RULE. `propose` forms a segment's title from an ATX
|
||||
heading and `structure` derives a document title from its leading heading, so
|
||||
a rule living in one of them would strip the attribute on one path and leave
|
||||
it on the other. Both read `strip_converter_attribute`.
|
||||
|
||||
THE KNOWN-NEGATIVE IS THE POINT. The pattern must match pandoc's attribute
|
||||
form and nothing else: a heading that legitimately contains braces -- a
|
||||
placeholder in a template, a code identifier -- is a title the author wrote,
|
||||
and stripping it would be the same defect pointed the other way.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from llm_ingestion_okf import cli
|
||||
from llm_ingestion_okf.extract import extract_text, strip_converter_attribute
|
||||
from llm_ingestion_okf.propose import find_candidates
|
||||
from llm_ingestion_okf.structure import derive_document_structure
|
||||
|
||||
DEFAULT = dict(
|
||||
outline_run=cli.DEFAULT_OUTLINE_RUN,
|
||||
table_grid=cli.DEFAULT_TABLE_GRID,
|
||||
unit_fold=cli.DEFAULT_UNIT_FOLD,
|
||||
keep_table_heading=cli.DEFAULT_KEEP_TABLE_HEADING,
|
||||
sheet_section_rows=cli.DEFAULT_SHEET_SECTION_ROWS,
|
||||
drop_wrapped_outline=cli.DEFAULT_DROP_WRAPPED_OUTLINE,
|
||||
outline_gate=cli.DEFAULT_OUTLINE_GATE,
|
||||
first_span_from_zero=cli.DEFAULT_FIRST_SPAN_FROM_ZERO,
|
||||
close_span_gaps=cli.DEFAULT_CLOSE_SPAN_GAPS,
|
||||
contents_name=cli.DEFAULT_CONTENTS_NAME,
|
||||
)
|
||||
|
||||
SHEET_DOCUMENT = """## Prissammenstilling {#sheet-1}
|
||||
|
||||
Denne posten samler prisene for hele leveransen og er den posten et
|
||||
spoersmaal om pris maa naa.
|
||||
|
||||
## Ark1 {#sheet-2}
|
||||
|
||||
Denne posten lister dokumentene i del II og har sin egen kropp.
|
||||
|
||||
## Lysbilde med krav {#slide-3}
|
||||
|
||||
Dette lysbildet bar en tittelplassholder, saa konvertereren ga det et anker.
|
||||
"""
|
||||
|
||||
#: The known-negative. Braces the AUTHOR wrote, in the two shapes that occur:
|
||||
#: a template placeholder and a code identifier. Neither is pandoc's attribute
|
||||
#: form, and neither may be touched.
|
||||
AUTHORED_BRACES = """## Mal for {kundenavn} og leveransen
|
||||
|
||||
Denne overskriften har krollparenteser forfatteren skrev selv, og den staar
|
||||
paa niva 2 sammen med den neste -- clause 2 folds a level that occurs once
|
||||
into the level above, which would empty this fixture without saying so.
|
||||
|
||||
## Feltet {"id": 4} i nyttelasten
|
||||
|
||||
Dette avsnittet beskriver et felt og staar under sin egen overskrift.
|
||||
"""
|
||||
|
||||
|
||||
def test_the_attribute_is_stripped_from_a_segment_title() -> None:
|
||||
titles = [candidate.title for candidate in find_candidates(SHEET_DOCUMENT, **DEFAULT)]
|
||||
for leaked in ("{#sheet-1}", "{#sheet-2}", "{#slide-3}"):
|
||||
assert not any(leaked in title for title in titles), (
|
||||
f"the converter attribute {leaked} reached a concept title: {titles}"
|
||||
)
|
||||
assert "Prissammenstilling" in titles
|
||||
assert "Lysbilde med krav" in titles
|
||||
|
||||
|
||||
def test_the_attribute_is_stripped_from_a_derived_document_title() -> None:
|
||||
derived = derive_document_structure(
|
||||
"# Prissammenstilling {#sheet-1}\n\nEn kropp.\n", source_file="regneark.md"
|
||||
)
|
||||
title = derived.title
|
||||
assert title == "Prissammenstilling", f"structure derived the title {title!r}"
|
||||
|
||||
|
||||
def test_authored_braces_are_left_alone() -> None:
|
||||
"""The known-negative: only pandoc's attribute form is an attribute."""
|
||||
titles = [candidate.title for candidate in find_candidates(AUTHORED_BRACES, **DEFAULT)]
|
||||
assert any("{kundenavn}" in title for title in titles), titles
|
||||
assert any('{"id": 4}' in title for title in titles), titles
|
||||
assert strip_converter_attribute('Feltet {"id": 4} i nyttelasten') == (
|
||||
'Feltet {"id": 4} i nyttelasten'
|
||||
)
|
||||
assert strip_converter_attribute("Mal for {kundenavn}") == "Mal for {kundenavn}"
|
||||
|
||||
|
||||
def test_a_deck_that_declares_slide_titles_is_named_by_them() -> None:
|
||||
"""The slide form, on a REAL conversion rather than a hand-written string.
|
||||
|
||||
The committed `pptx` fixture emits `## Slide 1` / `## Slide 2`, and round 7
|
||||
read that as the format segmenting badly. Decomposed 2026-09-09 it is not:
|
||||
that deck's title shapes carry no `<p:ph type="title"/>` placeholder, so the
|
||||
converter has nothing to name a slide WITH. Given a deck that declares one,
|
||||
the converter writes `## <authored title> {#slide-N}` -- and the attribute
|
||||
is exactly what this module strips, so the concept ends up named by the
|
||||
title the author wrote.
|
||||
|
||||
Built in memory from the committed generator rather than added to
|
||||
`tests/fixtures/k2-office/`, which holds exactly three containers and
|
||||
asserts that it does.
|
||||
"""
|
||||
pytest.importorskip("pypandoc", reason="office conversion needs the [extract] extra")
|
||||
# Loaded BY PATH: the generator lives under `tests/fixtures/` and is not on
|
||||
# `sys.path`, so `importorskip` on its name skips silently -- a green run
|
||||
# over an empty set, which is the one outcome this file exists to prevent.
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"make_k2_office", Path(__file__).parent / "fixtures" / "make_k2_office.py"
|
||||
)
|
||||
assert spec is not None and spec.loader is not None
|
||||
make = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(make)
|
||||
|
||||
def titled(shape_id: int, name: str, text: str) -> str:
|
||||
return (
|
||||
"<p:sp><p:nvSpPr>"
|
||||
f'<p:cNvPr id="{shape_id}" name="{name}"/><p:cNvSpPr/>'
|
||||
'<p:nvPr><p:ph type="title"/></p:nvPr>'
|
||||
"</p:nvSpPr><p:spPr/>" + make._pptx_text_body(text) + "</p:sp>"
|
||||
)
|
||||
|
||||
parts = make.pptx_parts()
|
||||
parts["ppt/slides/slide1.xml"] = make._pptx_slide(
|
||||
titled(2, "Tittel", make.TITLE)
|
||||
+ make._pptx_shape(3, "Ingress", make.INTRO)
|
||||
+ make._pptx_table(4, "Kravtabell", tuple(make.PAIRS))
|
||||
)
|
||||
parts["ppt/slides/slide2.xml"] = make._pptx_slide(
|
||||
titled(2, "Undertittel", make.GRID_CAPTION)
|
||||
+ make._pptx_table(3, "Luminansmatrise", make.GRID)
|
||||
)
|
||||
data = make.build_container(parts, stored_first="[Content_Types].xml")
|
||||
|
||||
text = extract_text("tittelplassholder.pptx", data)
|
||||
headings = [line for line in text.splitlines() if line.startswith("#")]
|
||||
# The known-positive for the strip: the converter DOES write the anchor
|
||||
# here, so a green assertion below is not green over an empty set.
|
||||
assert any("{#slide-" in line for line in headings), headings
|
||||
|
||||
titles = [candidate.title for candidate in find_candidates(text, **DEFAULT)]
|
||||
assert titles == [make.TITLE, make.GRID_CAPTION], titles
|
||||
Loading…
Add table
Add a link
Reference in a new issue