Kjell Tore Guttormsen
6ff18fd703
Round 9: the four rests in STATE's NESTE that needed no operator decision.
CLAUSE 1 CLASSIFIED BY THE NUMBER, NOT THE TITLE. `_TRAILING_PAGE_NUMBER`
admitted a candidate into a contents run by asking whether the title ended in
an integer -- a question about the number. A drawing's dimension chain, a
schematic's labels, a door schedule, a coordinate column and a soil-layer
table all end in integers and name nothing. Measured over the 43-document
corpus: 68 candidates discarded over 11 of 39 readable documents, of which
19 over 5 documents are data rows.
That corrects round 8's own decomposition. Its "four misclassified numeric
tables and seven real contents listings" needs each document on one side, and
two of the eleven are both. Read across all 68 titles rather than the
three-title sample: 5 documents carry a data row, 8 carry a real entry.
`--contents-name` requires a NAME to survive stripping the page number. The
threshold is SWEPT, not chosen, and collapses at both ends: at an alphabetic
run of 1 a door schedule keeps a stray `V` and 13 of 19 are rescued; at 3 the
two-letter section name `VA` stops being a name, falls out of run membership,
and takes `RIB`, `MMI` and `Tittelfelt` below `CONTENTS_RUN` with it -- one
acronym costing four REAL entries. At 2: 16 of 19 rescued, 0 of 49 regressed.
The three not rescued carry a real word and are named rather than rounded off.
THE CONVERTER'S ANCHOR WAS IN THE CONCEPT ID. Pandoc writes a sheet as
`## <name> {#sheet-N}` and a titled slide as `## <title> {#slide-N}`. Because
a filename is reduced FROM the title, the anchor reached both. Operator
authorised the strip 2026-09-09 after the exposure was counted: 2 of 810
concepts on the previous default bundle, 2 of 1108 on Arm B, 1 of 26 on the
operator's folder. Two ids renamed, one of which `portfolio-optimiser` has
cited in writing; both are in the report so that message can be sent.
One rule in one function, read by BOTH title-forming sites -- a rule in only
one would leave the id and the title naming the same concept differently. The
known-negative is the point: `Mal for {kundenavn}` is a title an author wrote.
odt/rtf/pptx MEASURED END TO END FOR THE FIRST TIME, on hand-built documents,
because the corpus denominator is genuinely zero (86 files: 66 pdf, 10 docx,
4 xlsx, 2 zip, 2 smc, 2 doc). `_EVIDENCE` gains a third class rather than
stretching an existing one: `constructed` means the row has met a document,
but not one anyone wrote for their own purposes. odt 1 of 1 declared headings;
pptx 2 of 2 on a deck that declares slide titles and 0 of 2 on one that does
not -- round 7's reading of pptx was a fixture property, not the format; rtf
0 segments, because the container has no heading style and the author's title
is bold text. rtf is the one open finding.
ACCEPTANCE, all four. The 12-position reference is label-identical in BOTH
readings (pdf 7/8, docx 3/3, xlsx 0/1 or 1/1, sheet 10/12 or 11/12). One K2
bundle carrying both changes: 453 concepts / 865 md, hit@8 [1,1,1,1,1,None]
on it AND on Arm B, with the known-negative still reproducing on the new
bytes. `okf project` byte-equal to `okf build`, `diff -r` empty. Consumer
cost is a re-run: 436/832 -> 453/865, digest 21af4a1aa98315cf.
Three published numbers corrected: README's 596 tests (1515), README's "15
concepts out" for `okf project` (that was the O6 defect; it is 26), and O6's
print-mode method, which does not reproduce without --allowedTools.
Report: docs/2026-09-09-k3-runde9-restene.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
154 lines
7.3 KiB
Python
154 lines
7.3 KiB
Python
"""The bundle the DEFAULT build produces, pinned where a regression goes red.
|
|
|
|
`tests/test_okf_consume.py` pinned hit@8 against the Arm B bundle alone -- the
|
|
configuration `okf build` stopped emitting on 2026-09-08. A published number
|
|
measured on a bundle nobody produces is a number that cannot regress, so the
|
|
guarantee it looks like was never held by anything.
|
|
|
|
This file pins the CURRENT default: `--outline-run 3 --table-grid --unit-fold
|
|
--drop-wrapped-outline --outline-gate --first-span-from-zero
|
|
--sheet-section-rows --keep-table-heading --close-span-gaps`, plus the reading
|
|
side's `tie_shared_rank`. Round 6 moved the first five on 2026-09-09, round 7
|
|
moved four more on 2026-09-10 and round 8 moved the last on 2026-09-11, each
|
|
after measuring hit@8 on exactly the bundle its own default produces.
|
|
|
|
The gold set is LOCAL-ONLY and stays that way: no question and no
|
|
`gold_document` is reproduced here, and a row is named by its INDEX, the way
|
|
`docs/2026-09-07-okf-konsumskill-maaling.md` already names them.
|
|
|
|
The bundle itself is a build artefact, not a fixture: it is 832 files of a
|
|
consumer's corpus and this repository is public. Absent, these tests SKIP with
|
|
the command that rebuilds it -- "not measured", never zero.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
|
sys.path.insert(0, str(PROJECT_ROOT / "tools"))
|
|
|
|
import okf_consume # noqa: E402
|
|
import okf_consume_measure # noqa: E402
|
|
|
|
#: Built by:
|
|
#: okf build <corpus>/K2/trinn1 \
|
|
#: --bundle ~/corpora/okf-telling-20260829/K2-bundle-default-20260911 \
|
|
#: --bundle-id k2-trinn1-20260903 --okf-version 0.2
|
|
#: with no arm flag at all -- the package default, which is the point.
|
|
#:
|
|
#: Rebuilt 2026-09-09 for `--contents-name` (round 9). Digest, from inside the
|
|
#: bundle:
|
|
#: find . -type f -print0 | sort -z | xargs -0 shasum -a 256 | shasum -a 256
|
|
#: -> 21af4a1aa98315cf514c4cbc6b4a9b77ce63960224d6d7b31b34d55cc67fb2ad
|
|
#: (The previous default, `K2-bundle-default-20260911`, was
|
|
#: 8c93e5e3222577a2b3352ca83af980e403d3a571c3a467b83c3d8170b1df2b69 at 436
|
|
#: concepts and stays on disk.)
|
|
#: Two independent builds of it differ in NOTHING (`diff -rq`), including
|
|
#: `log.md`, which carries the corpus path and never the bundle's own.
|
|
#:
|
|
#: CONCEPT IDS MOVED IN THIS REBUILD, and not only because the count did.
|
|
#: Round 9 strips pandoc's `{#sheet-N}` / `{#slide-N}` anchor where a title is
|
|
#: formed, and a concept's filename is reduced FROM its title, so TWO ids on
|
|
#: this bundle are renamed:
|
|
#: del-ii-bilag-7-prisskjema/prissammenstilling-sheet-1 -> .../prissammenstilling
|
|
#: del-ii-bilag-0-dokumentliste-del-ii/ark1-sheet-1 -> .../ark1
|
|
#: The first is an id `portfolio-optimiser` has cited in writing. The rename
|
|
#: was authorised by the operator on 2026-09-09 after the exposure was counted:
|
|
#: 2 of 810 concepts on the previous default and 2 of 1108 on Arm B.
|
|
DEFAULT_BUNDLE = Path.home() / "corpora" / "okf-telling-20260829" / "K2-bundle-default-20260912"
|
|
GOLD_SET = PROJECT_ROOT / ".claude/projects/2026-09-07-okf-consume-prepass/hit-at-k-questions.json"
|
|
|
|
requires_default_bundle = pytest.mark.skipif(
|
|
not DEFAULT_BUNDLE.is_dir() or not GOLD_SET.is_file(),
|
|
reason=(
|
|
f"the default-configuration K2 bundle is not present at {DEFAULT_BUNDLE}. "
|
|
"NOT MEASURED, not zero: rebuild it with `okf build <corpus>/K2/trinn1 "
|
|
"--bundle <that path> --bundle-id k2-trinn1-20260903 --okf-version 0.2`"
|
|
),
|
|
)
|
|
|
|
#: Measured 2026-09-09 on the bundle above. The count moved 425 -> 436 with
|
|
#: `--sheet-section-rows --keep-table-heading`; `--first-span-from-zero` and
|
|
#: `--close-span-gaps` each moved it by NOTHING, which is the point of both --
|
|
#: they add no boundary, they only move a span's start or its end. Round 8's
|
|
#: rule closed 43 631 characters (2.51 % of the corpus) that were in no
|
|
#: segment, and the count was byte-for-byte the same 436.
|
|
#:
|
|
#: 436 -> 453 with round 9's `--contents-name`, which does add concepts: a run
|
|
#: of data rows is no longer read as a contents listing and discarded, so the
|
|
#: candidates it was taking with it survive. Corpus-wide, 429 -> 447 candidates
|
|
#: over 32 -> 33 documents with a plan, and characters in no segment stay 0.
|
|
EXPECTED_CONCEPTS = 453
|
|
EXPECTED_HITS = 5
|
|
#: Rank per question INDEX, `None` for the row that misses on every bundle and
|
|
#: every configuration measured so far. The identity is the index; the question
|
|
#: stays in the local-only gold set.
|
|
EXPECTED_RANKS = (1, 1, 1, 1, 1, None)
|
|
|
|
|
|
@requires_default_bundle
|
|
def test_the_default_bundle_holds_its_concept_count() -> None:
|
|
assert len(list(okf_consume.enumerate_concepts(DEFAULT_BUNDLE))) == EXPECTED_CONCEPTS
|
|
|
|
|
|
@requires_default_bundle
|
|
def test_hit_at_eight_holds_rank_one_on_every_row_it_held() -> None:
|
|
"""The acceptance criterion round 6's default move had to clear.
|
|
|
|
Not the hit COUNT alone: the count survived a configuration that lost a
|
|
row from rank 1 to rank 2, which is exactly how the previous round's
|
|
regression hid. The rank per row is the pin.
|
|
|
|
On THIS bundle that is not a hypothetical -- see the test below, which
|
|
reproduces the fall on these exact bytes by turning the reading-side
|
|
default off.
|
|
"""
|
|
questions = json.loads(GOLD_SET.read_text(encoding="utf-8"))["questions"]
|
|
assert len(questions) == len(EXPECTED_RANKS), "the gold set changed shape"
|
|
ranks = []
|
|
for entry in questions:
|
|
payload = okf_consume.build_payload(DEFAULT_BUNDLE, question=entry["question"])
|
|
excerpts = payload["excerpts"]
|
|
assert isinstance(excerpts, list)
|
|
ranks.append(okf_consume_measure.hit_rank(excerpts, entry["gold_document"]))
|
|
assert tuple(ranks) == EXPECTED_RANKS, f"hit@8 ranks moved: {ranks}"
|
|
assert sum(rank is not None for rank in ranks) == EXPECTED_HITS
|
|
|
|
|
|
@requires_default_bundle
|
|
def test_the_bundle_declares_the_identity_the_reader_needs() -> None:
|
|
"""Whatever else moves, the bundle stays one the reading direction opens."""
|
|
assert okf_consume.root_bundle_id_of(DEFAULT_BUNDLE) == "k2-trinn1-20260903"
|
|
|
|
|
|
@requires_default_bundle
|
|
def test_the_reading_default_is_what_holds_row_one_on_these_bytes() -> None:
|
|
"""The known-negative, on the shipped bundle rather than a fixture.
|
|
|
|
Round 7 moved `--sheet-section-rows --keep-table-heading` into the build
|
|
default, which splits row 1's gold document from 1 concept into 12. Round 6
|
|
measured that exact split costing row 1 its rank, and held the two rules
|
|
back for it. What removed the cost is `consume.DEFAULT_TIE_SHARED_RANK`,
|
|
and this test is the proof that it is still what removes it: turn it off
|
|
on these bytes and the fall comes back.
|
|
|
|
Without this, `EXPECTED_RANKS` above would be a green assertion with no
|
|
stated cause, and a later change to the fusion could take the cause away
|
|
while the pin stayed green on some other accident.
|
|
"""
|
|
questions = json.loads(GOLD_SET.read_text(encoding="utf-8"))["questions"]
|
|
ranks = []
|
|
for entry in questions:
|
|
payload = okf_consume.build_payload(
|
|
DEFAULT_BUNDLE, question=entry["question"], tie_shared_rank=False
|
|
)
|
|
excerpts = payload["excerpts"]
|
|
assert isinstance(excerpts, list)
|
|
ranks.append(okf_consume_measure.hit_rank(excerpts, entry["gold_document"]))
|
|
assert ranks[0] == 2, "the known-negative stopped being negative"
|
|
assert tuple(ranks[1:]) == EXPECTED_RANKS[1:]
|