feat(propose,cli): the coverage gap had one cause, and round 7's own decomposition did not reproduce
Round 7 named two open items: a table-block candidate displacing a declared
heading (26 041 characters between entries, `md` at 3 of 4 declared headings)
and 17 590 characters after the last entry, never examined. Measured on
`a364ef4`, the first premise does not reproduce and the second is not a
separate mechanism.
`md` recovers 4 of 4, not 3 of 4. D1 (`--keep-table-heading`) is the repair
for exactly the mechanism round 7 § 5 describes, and round 7 moved it into
the default in the same commit -- so § 5 is a pre-move measurement presented
as the post-move state. That is round 7's own trap, stated in its own report:
a number about a rule is a measurement of a configuration. Nothing held the
cell, which is why it could be wrong and stay wrong.
The remaining gap has ONE cause. Every rule closes a span against the NEXT
MARK; three steps then remove a mark after its neighbour's `end` was fixed
against it. The orphan check leaks 18 527 characters over 15 of 39 documents,
`fold_units` clause 1 leaks 7 514 between entries, and the same clause on the
last run leaks all 17 590 tail characters -- with `unit_fold=False` the corpus
tail gap is 0. Round 6 already established the principle (filter at admission,
let the mark above carry the text) and it was never applied post-filter.
`--close-span-gaps` states it once, after the fold, where every removal has
happened: a span runs to the next SURVIVOR, the last to the end of the text.
It adds no boundary. Measured: 43 631 characters (2.51 %) -> 0, entries
429 = 429, K2 concepts 436 = 436, the operator's folder 52 md = 52.
All four acceptance conditions hold at once. The 12-position reference is
label-identical -- 11 of 12 under |F|[3]=12 and 10 of 12 under |F|[3]=11,
both readings reported, not one position moved. hit@8 holds [1,1,1,1,1,-] on
the round 8 bundle (436), the round 7 default (436) and Arm B (629), and the
known-negative reproduces on the new bytes: `--no-tie-shared-rank` gives
[2,1,1,1,1,-].
The pin is rebuilt against the default that actually ships
(`K2-bundle-default-20260911`, 832 files, digest 8c93e5e3...); two independent
builds of it differ in nothing, `log.md` included.
Two measurement failures are recorded rather than hidden. The corpus identity
check first used `xargs` without `-print0`, every filename split on its
spaces, `shasum` read nothing, and both sides agreed on a digest of the
failure; a control against the digest of empty input separated them. And the
smoke build cannot prove the flag arrives -- the operator's folder has a
coverage gap of zero already, so `diff -rq` is 0 either way; the plumbing has
its own test on a document that has the defect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
a364ef4c7d
commit
956714594d
8 changed files with 592 additions and 22 deletions
|
|
@ -7,10 +7,10 @@ guarantee it looks like was never held by anything.
|
|||
|
||||
This file pins the CURRENT default: `--outline-run 3 --table-grid --unit-fold
|
||||
--drop-wrapped-outline --outline-gate --first-span-from-zero
|
||||
--sheet-section-rows --keep-table-heading`, plus the reading side's
|
||||
`tie_shared_rank`. Round 6 moved the first five on 2026-09-09 and round 7 moved
|
||||
the last four on 2026-09-10, each after measuring hit@8 on exactly the bundle
|
||||
its own default produces.
|
||||
--sheet-section-rows --keep-table-heading --close-span-gaps`, plus the reading
|
||||
side's `tie_shared_rank`. Round 6 moved the first five on 2026-09-09, round 7
|
||||
moved four more on 2026-09-10 and round 8 moved the last on 2026-09-11, each
|
||||
after measuring hit@8 on exactly the bundle its own default produces.
|
||||
|
||||
The gold set is LOCAL-ONLY and stays that way: no question and no
|
||||
`gold_document` is reproduced here, and a row is named by its INDEX, the way
|
||||
|
|
@ -37,10 +37,16 @@ import okf_consume_measure # noqa: E402
|
|||
|
||||
#: Built by:
|
||||
#: okf build <corpus>/K2/trinn1 \
|
||||
#: --bundle ~/corpora/okf-telling-20260829/K2-bundle-default-20260910 \
|
||||
#: --bundle ~/corpora/okf-telling-20260829/K2-bundle-default-20260911 \
|
||||
#: --bundle-id k2-trinn1-20260903 --okf-version 0.2
|
||||
#: with no arm flag at all -- the package default, which is the point.
|
||||
DEFAULT_BUNDLE = Path.home() / "corpora" / "okf-telling-20260829" / "K2-bundle-default-20260910"
|
||||
#:
|
||||
#: Rebuilt 2026-09-11 for `--close-span-gaps`. Digest, from inside the bundle:
|
||||
#: find . -type f -print0 | sort -z | xargs -0 shasum -a 256 | shasum -a 256
|
||||
#: -> 8c93e5e3222577a2b3352ca83af980e403d3a571c3a467b83c3d8170b1df2b69
|
||||
#: Two independent builds of it differ in NOTHING (`diff -rq`), including
|
||||
#: `log.md`, which carries the corpus path and never the bundle's own.
|
||||
DEFAULT_BUNDLE = Path.home() / "corpora" / "okf-telling-20260829" / "K2-bundle-default-20260911"
|
||||
GOLD_SET = PROJECT_ROOT / ".claude/projects/2026-09-07-okf-consume-prepass/hit-at-k-questions.json"
|
||||
|
||||
requires_default_bundle = pytest.mark.skipif(
|
||||
|
|
@ -52,10 +58,12 @@ requires_default_bundle = pytest.mark.skipif(
|
|||
),
|
||||
)
|
||||
|
||||
#: Measured 2026-09-10 on the bundle above. The count moved 425 -> 436 with
|
||||
#: `--sheet-section-rows --keep-table-heading`; `--first-span-from-zero` moved
|
||||
#: it by NOTHING, which is the point of that rule -- it adds no boundary and
|
||||
#: only moves one span's start.
|
||||
#: Measured 2026-09-11 on the bundle above. The count moved 425 -> 436 with
|
||||
#: `--sheet-section-rows --keep-table-heading`; `--first-span-from-zero` and
|
||||
#: `--close-span-gaps` each moved it by NOTHING, which is the point of both --
|
||||
#: they add no boundary, they only move a span's start or its end. Round 8's
|
||||
#: rule closed 43 631 characters (2.51 % of the corpus) that were in no
|
||||
#: segment, and the count is byte-for-byte the same 436.
|
||||
EXPECTED_CONCEPTS = 436
|
||||
EXPECTED_HITS = 5
|
||||
#: Rank per question INDEX, `None` for the row that misses on every bundle and
|
||||
|
|
|
|||
100
tests/test_md_declared_headings.py
Normal file
100
tests/test_md_declared_headings.py
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
"""`md` recovers every heading it declares, and D1 is what makes that true.
|
||||
|
||||
Round 7 reported this cell as **3 of 4**, with the mechanism decomposed: a
|
||||
`rule:table-block` candidate opens BELOW `## 3 Prising` instead of a
|
||||
`rule:heading` opening ON it, so the heading is bodiless, the orphan check
|
||||
drops it, and the fold then merges the nameless block into the section above.
|
||||
|
||||
Re-measured on `a364ef4` (round 7's own commit), the default recovers **4 of
|
||||
4**. The report's § 5 is a measurement of the configuration that existed
|
||||
BEFORE the same round moved `--keep-table-heading` into the default -- D1 is
|
||||
precisely the repair for that mechanism, and it shipped in the commit the
|
||||
report describes. Nothing in the tree said so, because the decomposition and
|
||||
the default move live in different files.
|
||||
|
||||
That is the round-7 trap restated: a number about a rule is a measurement of a
|
||||
CONFIGURATION. This file is the missing pin. The cell is asserted here, and so
|
||||
is its cause -- turn D1 off and the third heading becomes a table block again.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from llm_ingestion_okf import cli
|
||||
from llm_ingestion_okf.propose import RULE_HEADING, RULE_TABLE_BLOCK, find_candidates
|
||||
|
||||
DEFAULT = dict(
|
||||
outline_run=cli.DEFAULT_OUTLINE_RUN,
|
||||
table_grid=cli.DEFAULT_TABLE_GRID,
|
||||
unit_fold=cli.DEFAULT_UNIT_FOLD,
|
||||
keep_table_heading=cli.DEFAULT_KEEP_TABLE_HEADING,
|
||||
sheet_section_rows=cli.DEFAULT_SHEET_SECTION_ROWS,
|
||||
drop_wrapped_outline=cli.DEFAULT_DROP_WRAPPED_OUTLINE,
|
||||
outline_gate=cli.DEFAULT_OUTLINE_GATE,
|
||||
first_span_from_zero=cli.DEFAULT_FIRST_SPAN_FROM_ZERO,
|
||||
close_span_gaps=cli.DEFAULT_CLOSE_SPAN_GAPS,
|
||||
)
|
||||
|
||||
#: The same structure round 7 built: three chapters, the last one a heading
|
||||
#: with a table directly under it and a closing line after the table.
|
||||
DOCUMENT = """# Prosjektbeskrivelse Testdokument
|
||||
|
||||
Dette dokumentet har en kjent struktur: tre kapitler paa niva 1, hvert med
|
||||
en kort brodtekst, samt en innledning over det forste kapitlet.
|
||||
|
||||
## 1 Omfang
|
||||
|
||||
Leveransen omfatter tre delytelser. Hver delytelse har egen frist og egen
|
||||
akseptansetest. Teksten her er brodtekst og skal ikke bli en overskrift.
|
||||
|
||||
## 2 Krav til dokumentasjon
|
||||
|
||||
Dokumentasjon leveres i PDF. Tegninger leveres i DWG. Modeller leveres i IFC.
|
||||
Krav nummer 2.1 gjelder alle tre formatene.
|
||||
|
||||
## 3 Prising
|
||||
|
||||
| Post | Beskrivelse | Enhet | Mengde |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | Riving av eksisterende dekke | m2 | 420 |
|
||||
| 2 | Ny baerekonstruksjon | tonn | 38 |
|
||||
| 3 | Tekniske installasjoner | RS | 1 |
|
||||
|
||||
Prisene fylles ut av tilbyder ved tilbudsfrist.
|
||||
"""
|
||||
|
||||
DECLARED = (
|
||||
"Prosjektbeskrivelse Testdokument",
|
||||
"1 Omfang",
|
||||
"2 Krav til dokumentasjon",
|
||||
"3 Prising",
|
||||
)
|
||||
|
||||
|
||||
def _declared_headings(**kwargs: object) -> list[str]:
|
||||
candidates = find_candidates(DOCUMENT, **{**DEFAULT, **kwargs}) # type: ignore[arg-type]
|
||||
return [c.title for c in candidates if c.rule == RULE_HEADING]
|
||||
|
||||
|
||||
def test_the_default_recovers_all_four_declared_headings() -> None:
|
||||
assert _declared_headings() == list(DECLARED)
|
||||
|
||||
|
||||
def test_the_priced_section_is_a_heading_and_not_a_table_block() -> None:
|
||||
"""The specific cell: `## 3 Prising` keeps its own line and its own name."""
|
||||
candidates = find_candidates(DOCUMENT, **DEFAULT)
|
||||
prising = [c for c in candidates if c.title == "3 Prising"]
|
||||
assert len(prising) == 1
|
||||
assert prising[0].rule == RULE_HEADING
|
||||
assert DOCUMENT[prising[0].start : prising[0].end].startswith("## 3 Prising")
|
||||
|
||||
|
||||
def test_without_d1_the_cell_falls_back_to_three_of_four() -> None:
|
||||
"""The known-negative: the pin names its own cause instead of being green
|
||||
for an unstated reason. Without `--keep-table-heading` the heading is
|
||||
orphaned and a table block carries its name from a line below it."""
|
||||
assert _declared_headings(keep_table_heading=False) == list(DECLARED[:3])
|
||||
candidates = find_candidates(DOCUMENT, **{**DEFAULT, "keep_table_heading": False})
|
||||
carrier = [c for c in candidates if c.title == "3 Prising"]
|
||||
assert len(carrier) == 1
|
||||
assert carrier[0].rule == RULE_TABLE_BLOCK
|
||||
assert not DOCUMENT[carrier[0].start : carrier[0].end].startswith("## 3 Prising")
|
||||
153
tests/test_span_gaps.py
Normal file
153
tests/test_span_gaps.py
Normal file
|
|
@ -0,0 +1,153 @@
|
|||
"""No character stands BETWEEN two concepts, or after the last one.
|
||||
|
||||
Round 7 closed the text ABOVE the first concept (`--first-span-from-zero`,
|
||||
163 804 characters, 79 % of the whole gap) and named the remainder without
|
||||
opening it: 43 631 characters, 2.51 % of the corpus, over 8 of the 32
|
||||
documents that get a plan -- 26 041 between entries and 17 590 after the last.
|
||||
|
||||
Measured here, the remainder has ONE cause, not the two the shape suggests.
|
||||
Every rule in `find_candidates` closes a span against the NEXT MARK, and three
|
||||
separate steps then remove a mark after its neighbour's `end` was already
|
||||
fixed against it:
|
||||
|
||||
* the **orphan check** drops a heading with no body, and the heading line
|
||||
itself -- `## 3 Prising\n`, 14 characters -- is what the next candidate no
|
||||
longer starts at. Measured with the fold off: **18 527 characters over 15 of
|
||||
39 documents**;
|
||||
* `fold_units` **clause 1** discards a contents run, and the text those
|
||||
entries opened goes with them. It accounts for the remaining **7 514**
|
||||
characters between entries;
|
||||
* the same clause discarding the LAST run takes the tail with it. Measured:
|
||||
with `unit_fold=False` the tail gap over the corpus is **0**, so all
|
||||
**17 590** tail characters are clause 1's.
|
||||
|
||||
That is round 6's own principle -- the outline gate filters at ADMISSION,
|
||||
"before spans close, so the text a removed mark opened is carried by the mark
|
||||
above" -- not applied to the two steps that remove marks AFTER spans close.
|
||||
This rule states it once, at the end, where every removal has already
|
||||
happened: a candidate's span runs to the next candidate's start, and the last
|
||||
one runs to the end of the text.
|
||||
|
||||
It adds NO boundary and removes none. A plan's entry count is identical either
|
||||
way, which is the same property `--first-span-from-zero` has and the reason
|
||||
both can be measured by coverage rather than by count.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from llm_ingestion_okf import cli
|
||||
from llm_ingestion_okf.propose import find_candidates
|
||||
|
||||
ARMS = dict(outline_run=3, table_grid=True, unit_fold=True, first_span_from_zero=True)
|
||||
|
||||
#: The orphan check's gap: `## Tom` has no body, so it is dropped -- and the
|
||||
#: seven characters of its own heading line belong to no segment.
|
||||
ORPHAN = "## Forste\nInnhold under.\n## Tom\n## Andre\nInnhold under.\n"
|
||||
|
||||
#: Clause 1's gap: three page-numbered siblings are read as a contents list
|
||||
#: and discarded, and the body under them goes too. 130 of 159 characters.
|
||||
CONTENTS_TAIL = (
|
||||
"## Innledning\nBrodtekst her.\n"
|
||||
"## Kapittel en 3\nNoe innhold.\n"
|
||||
"## Kapittel to 5\nMer innhold.\n"
|
||||
"## Kapittel tre 9\nEn avsluttende brodtekst som ingen kandidat dekker.\n"
|
||||
)
|
||||
|
||||
#: The known-negative: nothing is removed, so nothing is carried.
|
||||
NO_GAP = "## Forste\nInnhold under.\n## Andre\nInnhold under.\n"
|
||||
|
||||
|
||||
def _uncovered(text: str, **kwargs: object) -> int:
|
||||
candidates = find_candidates(text, **ARMS, **kwargs) # type: ignore[arg-type]
|
||||
if not candidates:
|
||||
return len(text)
|
||||
between = sum(
|
||||
max(0, candidates[i].start - candidates[i - 1].end) for i in range(1, len(candidates))
|
||||
)
|
||||
return candidates[0].start + between + max(0, len(text) - candidates[-1].end)
|
||||
|
||||
|
||||
def test_the_default_still_loses_the_orphaned_heading_line() -> None:
|
||||
"""The defect, kept visible: the dropped mark's own line goes nowhere."""
|
||||
assert _uncovered(ORPHAN) == len("## Tom\n")
|
||||
|
||||
|
||||
def test_the_default_still_loses_the_tail_under_a_discarded_contents_run() -> None:
|
||||
"""82 % of this document is in no segment, and none of it is a listing."""
|
||||
assert _uncovered(CONTENTS_TAIL) == 130
|
||||
|
||||
|
||||
def test_no_character_lies_between_two_concepts() -> None:
|
||||
assert _uncovered(ORPHAN, close_span_gaps=True) == 0
|
||||
candidates = find_candidates(ORPHAN, **ARMS, close_span_gaps=True)
|
||||
carried = ORPHAN[candidates[0].start : candidates[0].end]
|
||||
assert "## Tom" in carried, "the removed mark's text is carried by the mark ABOVE"
|
||||
|
||||
|
||||
def test_no_character_lies_after_the_last_concept() -> None:
|
||||
assert _uncovered(CONTENTS_TAIL, close_span_gaps=True) == 0
|
||||
candidates = find_candidates(CONTENTS_TAIL, **ARMS, close_span_gaps=True)
|
||||
assert CONTENTS_TAIL[candidates[-1].start : candidates[-1].end].endswith(
|
||||
"En avsluttende brodtekst som ingen kandidat dekker.\n"
|
||||
)
|
||||
|
||||
|
||||
def test_the_rule_adds_and_removes_no_boundary() -> None:
|
||||
"""Only `end` moves. Titles, rules and starts are identical."""
|
||||
for text in (ORPHAN, CONTENTS_TAIL, NO_GAP):
|
||||
plain = find_candidates(text, **ARMS)
|
||||
closed = find_candidates(text, **ARMS, close_span_gaps=True)
|
||||
assert [c.title for c in plain] == [c.title for c in closed]
|
||||
assert [c.rule for c in plain] == [c.rule for c in closed]
|
||||
assert [c.start for c in plain] == [c.start for c in closed]
|
||||
|
||||
|
||||
def test_a_document_with_no_gap_is_untouched() -> None:
|
||||
"""The known-negative, as IDENTICAL objects rather than an equal count."""
|
||||
before = find_candidates(NO_GAP, **ARMS)
|
||||
after = find_candidates(NO_GAP, **ARMS, close_span_gaps=True)
|
||||
assert before == after
|
||||
assert _uncovered(NO_GAP) == 0
|
||||
|
||||
|
||||
def test_it_is_on_by_default_in_the_build_command() -> None:
|
||||
assert cli.DEFAULT_CLOSE_SPAN_GAPS is True
|
||||
|
||||
|
||||
def test_the_parser_carries_the_default_and_an_opt_out() -> None:
|
||||
base = ["build", "src", "--bundle", "out", "--bundle-id", "x"]
|
||||
assert cli.parse_args(base).close_span_gaps is True
|
||||
assert cli.parse_args([*base, "--no-close-span-gaps"]).close_span_gaps is False
|
||||
|
||||
|
||||
def test_the_flag_reaches_the_proposer_from_the_build_command(tmp_path: Path) -> None:
|
||||
"""The plumbing, and it needs its own test rather than the smoke build.
|
||||
|
||||
Measured 2026-09-11: the operator's five-document folder has a coverage gap
|
||||
of ZERO under round 7's default already, so building it with the rule on
|
||||
and off gives `diff -rq` 0 differences. That is the rule behaving correctly
|
||||
on an input with nothing to carry -- and it means that build proves nothing
|
||||
about whether the flag arrives. This one uses a document that HAS a gap.
|
||||
"""
|
||||
inbox = tmp_path / "in"
|
||||
inbox.mkdir()
|
||||
(inbox / "d.md").write_text(CONTENTS_TAIL, encoding="utf-8")
|
||||
|
||||
def concepts(bundle: Path, **kwargs: object) -> str:
|
||||
report = cli.build(
|
||||
inbox,
|
||||
bundle=bundle,
|
||||
bundle_id="t",
|
||||
okf_version="0.2",
|
||||
ingested_at="1970-01-01T00:00:00Z",
|
||||
**kwargs, # type: ignore[arg-type]
|
||||
)
|
||||
assert report.codes == ()
|
||||
return "".join(path.read_text(encoding="utf-8") for path in sorted(bundle.rglob("*.md")))
|
||||
|
||||
on = concepts(tmp_path / "on")
|
||||
off = concepts(tmp_path / "off", close_span_gaps=False)
|
||||
assert "En avsluttende brodtekst som ingen kandidat dekker." in on
|
||||
assert "En avsluttende brodtekst som ingen kandidat dekker." not in off
|
||||
Loading…
Add table
Add a link
Reference in a new issue