feat(cli,consume): the first span, and the fusion that punished fine-graining
Round 6 left two things behind: a measured coverage hole with the repair built
but not the number to move it, and a retrieval regression that kept two
spreadsheet rules off the default. Both are measured; four defaults move.
THE HOLE IS BIGGER THAN THE RULE BUILT FOR IT. Measured over the 39-document
corpus, the pre-move default left 207 435 characters -- 11.92 % -- in no
segment: 163 804 above the first entry, 26 041 between entries, 17 590 after
the last. `--first-span-from-zero` closes the first part entirely, 79 % of the
whole, leaving 43 631 (2.51 %) over 8 of 32 documents. It adds no boundary and
the K2 concept count is identical with and without it (425 = 425); hit@8 holds
[1,1,1,1,1,-] under both tie-breaks and the 12-position reference does not move
one cell.
THE RANKER WAS PUNISHING FINE-GRAINING FOR BEING FINE-GRAINED. Round 6 held
`--sheet-section-rows --keep-table-heading` back because on a bundle built with
them row 1 fell rank 1 -> 2, and ruled out the document prior because its rank
over 39 DOCUMENTS was 1 on both bundles. RRF reads a rank over CONCEPTS:
splitting the gold document 1 -> 12 puts its own twelve concepts in that
signal's whole top tie group, so the one leading the body signal takes position
11, contributing 1/71 where the undivided concept contributed 1/61. A signal
that scored them all EQUALLY still emitted twelve ranks, ordered by concept_id,
and the fusion read alphabetical order as a measurement.
AND THE REPAIR WAS ALREADY IN THE TREE, OFF FOR A COST THAT HAD BEEN REMOVED.
`--tie-shared-rank` shipped off 2026-09-08 because hit@8 fell 5 of 6 to 4 of 6.
Swept over 2 prior exponents x 3 bundles x 6 rows: that fall exists only at
DOCUMENT_PRIOR_EXPONENT 1.0. Round 6 moved the exponent to 0.5 for an unrelated
reason and correctly reported it moved no hit@8 row -- nobody measured the
pair. A flag's "off by measurement" is a measurement of a CONFIGURATION, not a
property of the flag.
Shipped, each with an explicit opt-out and both directions measured rather than
asserted: --no-first-span-from-zero --no-sheet-section-rows
--no-keep-table-heading reproduces the previous bundle byte for byte (diff -rq,
0 differences), and the no-flag build equals the explicit-flag build.
--no-tie-shared-rank reproduces the previous excerpt order -- the one change
here that alters a payload with no bundle changing.
K2 default: 436 concepts / 832 files, digest 8dff8a8e6c15d2f7..., hit@8 5 of 6
ranks [1,1,1,1,1,-]. The pin now holds its own known-negative on those bytes:
read with --no-tie-shared-rank the same bundle gives [2,1,1,1,1,-], so the
green assertion names its cause. Consumer cost is a re-run: 425/810 -> 436/832
on the reference corpus, 15/30 -> 26/52 on a five-document folder.
pptx and md measured end to end for the first time, on two hand-built
documents: md recovers 3 of 4 declared headings, pptx segments per slide only
where the converter recognised a title placeholder. A converter attribute leaks
into titles ({#slide-N}, {#sheet-1}) on 2 of 810 K2 files and 1 of 30 on a
five-document folder -- not fixed, because a filename is reduced from its title
and the fix renames concept ids a consumer has cited.
Suite 1486 (1478 before), ruff and mypy --strict clean over 21 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
38104b7df5
commit
a364ef4c7d
11 changed files with 768 additions and 99 deletions
112
CLAUDE.md
112
CLAUDE.md
|
|
@ -27,6 +27,16 @@ one boundary rule:
|
||||||
fail-fast. The extra ships `pdfplumber` for `pdf` (chosen on ONE measured
|
fail-fast. The extra ships `pdfplumber` for `pdf` (chosen on ONE measured
|
||||||
property: it keeps a requirement table's label and value on the same line
|
property: it keeps a requirement table's label and value on the same line
|
||||||
where three alternatives do not); `docx`/`xlsx` still ship no parser.
|
where three alternatives do not); `docx`/`xlsx` still ship no parser.
|
||||||
|
`pptx` and `md` were measured end to end for the first time 2026-09-10
|
||||||
|
(`docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md` § 4) on two
|
||||||
|
hand-built documents, which is more than zero and is not a fasit: `md`
|
||||||
|
recovers 3 of 4 declared headings, and `pptx` segments per slide only where
|
||||||
|
the deck's slides carry title placeholders the converter recognises — a deck
|
||||||
|
whose slides do not lands as ONE concept. A converter attribute also leaks
|
||||||
|
into concept titles (`{#slide-N}`, `{#sheet-1}`), reaching 2 of 810 files on
|
||||||
|
the K2 default bundle and 1 of 30 on the operator's test folder; because a
|
||||||
|
filename is reduced from its title, fixing it RENAMES concept ids a consumer
|
||||||
|
has already cited, so it is an operator question and not a patch.
|
||||||
Structured table recovery is **out of scope** — two independent parsers
|
Structured table recovery is **out of scope** — two independent parsers
|
||||||
return the same wrong shape, so the breakage is document geometry, not a
|
return the same wrong shape, so the breakage is document geometry, not a
|
||||||
library choice. PDFs enter as prose, and drawn content (figures) does not
|
library choice. PDFs enter as prose, and drawn content (figures) does not
|
||||||
|
|
@ -266,12 +276,16 @@ and fixtures, never code.
|
||||||
collide); `--ingested-at` and `--proposed-at` default to one shared epoch
|
collide); `--ingested-at` and `--proposed-at` default to one shared epoch
|
||||||
constant rather than the clock, because a wall-clock default takes
|
constant rather than the clock, because a wall-clock default takes
|
||||||
rebuild-equals-incremental away from anyone who omits them.
|
rebuild-equals-incremental away from anyone who omits them.
|
||||||
**Eight segmentation rules are REACHABLE here, and FIVE of them are ON by
|
**Eight segmentation rules are REACHABLE here, and since 2026-09-10 ALL
|
||||||
default** -- `--outline-run 3`, `--table-grid` and `--unit-fold` since
|
EIGHT are ON by default** -- `--outline-run 3`, `--table-grid` and
|
||||||
2026-09-08, `--drop-wrapped-outline` and `--outline-gate` since 2026-09-09,
|
`--unit-fold` since 2026-09-08, `--drop-wrapped-outline` and
|
||||||
each with an explicit opt-out (`--outline-run 0`, `--no-table-grid`,
|
`--outline-gate` since 2026-09-09, `--sheet-section-rows`,
|
||||||
`--no-unit-fold`, `--keep-wrapped-outline`, `--no-outline-gate`) that
|
`--keep-table-heading` and `--first-span-from-zero` since 2026-09-10, each
|
||||||
together reproduce the pre-move bytes. **The 2026-09-09 pair is one
|
with an explicit opt-out (`--outline-run 0`, `--no-table-grid`,
|
||||||
|
`--no-unit-fold`, `--keep-wrapped-outline`, `--no-outline-gate`,
|
||||||
|
`--no-sheet-section-rows`, `--no-keep-table-heading`,
|
||||||
|
`--no-first-span-from-zero`) that together reproduce the pre-move bytes --
|
||||||
|
measured, `diff -rq` 0 differences, not asserted. **The 2026-09-09 pair is one
|
||||||
decision and cannot be split**: the gate takes `pdf` from 2 of 8 to 5 of 8
|
decision and cannot be split**: the gate takes `pdf` from 2 of 8 to 5 of 8
|
||||||
and the pair takes it to 7 of 8 (the sheet 5 of 12 -> 10 of 12, `docx`
|
and the pair takes it to 7 of 8 (the sheet 5 of 12 -> 10 of 12, `docx`
|
||||||
unchanged at 3 of 3). **The gate is G1+G2:** Arm D's RECOVERED headings are
|
unchanged at 3 of 3). **The gate is G1+G2:** Arm D's RECOVERED headings are
|
||||||
|
|
@ -284,9 +298,17 @@ and fixtures, never code.
|
||||||
text, which is why only one of them shipped. **The bar it had to clear is
|
text, which is why only one of them shipped. **The bar it had to clear is
|
||||||
now the bar**: reference cells up AND hit@8 holding rank 1 on every row on
|
now the bar**: reference cells up AND hit@8 holding rank 1 on every row on
|
||||||
every bundle. `--sheet-section-rows --keep-table-heading` reaches 11 of 12
|
every bundle. `--sheet-section-rows --keep-table-heading` reaches 11 of 12
|
||||||
and does NOT ship, because on a K2 bundle built with it row 1 falls rank
|
and SHIPPED 2026-09-10, after two rounds off. It was held back because on a
|
||||||
1 -> 2 (the gold document goes 1 concept -> 12) -- measured under BOTH prior
|
K2 bundle built with it row 1 fell rank 1 -> 2 (the gold document goes 1
|
||||||
exponents, so the sublinear prior does not buy it. Arm E joined a session after the other two, on a number measured
|
concept -> 12), under both prior exponents. **That was never these rules'
|
||||||
|
defect and it is not a segmentation question**: RRF emits a distinct rank
|
||||||
|
for every concept in a signal that scored them all EQUALLY, so the gold
|
||||||
|
document's own twelve concepts fill the document-prior tie group and the one
|
||||||
|
leading the body signal takes position 11 instead of 1. The repair is the
|
||||||
|
reading side's `consume.DEFAULT_TIE_SHARED_RANK`, and with it every
|
||||||
|
acceptance condition holds at once. **The fusion was punishing fine-graining
|
||||||
|
for being fine-grained**, which put the segmentation side and the retrieval
|
||||||
|
side in competition over one number for two rounds. Arm E joined a session after the other two, on a number measured
|
||||||
AFTER the first move: without it Arm F's table clause has no joined table to
|
AFTER the first move: without it Arm F's table clause has no joined table to
|
||||||
fold, and the shipped D+F default scored 2 of 12 with `docx` 0 of 3 against
|
fold, and the shipped D+F default scored 2 of 12 with `docx` 0 of 3 against
|
||||||
the 5 of 12 the fold was published with. **The proposer's own defaults did NOT move** (`propose.py`'s rules stay
|
the 5 of 12 the fold was published with. **The proposer's own defaults did NOT move** (`propose.py`'s rules stay
|
||||||
|
|
@ -295,16 +317,22 @@ and fixtures, never code.
|
||||||
`cli.DEFAULT_UNIT_FOLD` say where. The cost to a consumer is a re-run and it
|
`cli.DEFAULT_UNIT_FOLD` say where. The cost to a consumer is a re-run and it
|
||||||
is not small: the 43-document reference corpus goes 629 concepts / 1108 files
|
is not small: the 43-document reference corpus goes 629 concepts / 1108 files
|
||||||
(the delivered 2026-09-03 tree) to 492 / 944 after the 2026-09-08 move and to
|
(the delivered 2026-09-03 tree) to 492 / 944 after the 2026-09-08 move and to
|
||||||
**425 / 810** after the 2026-09-09 one (`bdf4977ca5a443c4...`, default flags,
|
425 / 810 after the 2026-09-09 one (`bdf4977ca5a443c4...`) and to
|
||||||
default epoch stamp, measured on `b01492b` + this round). Digests published
|
**436 / 832** after the 2026-09-10 one (`8dff8a8e6c15d2f7...`, default flags,
|
||||||
|
default epoch stamp, measured on `38104b7` + this round). On the operator's
|
||||||
|
own five-document folder the last move is 15 concepts / 30 files -> 26 / 52. Digests published
|
||||||
before 2026-09-09 were computed with a path-DEPENDENT command and are not
|
before 2026-09-09 were computed with a path-DEPENDENT command and are not
|
||||||
comparable to this one; the reproducible form is `find . -type f | sort |
|
comparable to this one; the reproducible form is `find . -type f | sort |
|
||||||
xargs shasum -a 256 | shasum -a 256` from inside the bundle, under which the
|
xargs shasum -a 256 | shasum -a 256` from inside the bundle, under which the
|
||||||
previous default is `862116da16e422f6...`. The pinned artifact lives at
|
previous default is `862116da16e422f6...`. The pinned artifact lives at
|
||||||
`~/corpora/okf-telling-20260829/K2-bundle-default-20260909` and
|
`~/corpora/okf-telling-20260829/K2-bundle-default-20260910` and
|
||||||
`tests/test_default_bundle_pin.py` holds its concept count AND its per-row
|
`tests/test_default_bundle_pin.py` holds its concept count AND its per-row
|
||||||
hit@8 ranks -- the count alone survived a configuration that lost a rank,
|
hit@8 ranks -- the count alone survived a configuration that lost a rank,
|
||||||
which is how the previous round's regression hid. **And the number the
|
which is how a previous round's regression hid. Since 2026-09-10 it also
|
||||||
|
holds the KNOWN-NEGATIVE on the same bytes: read with
|
||||||
|
`--no-tie-shared-rank`, the shipped default bundle reproduces the very fall
|
||||||
|
the rules were held back for, so the pin names its own cause instead of
|
||||||
|
being green for an unstated reason. **And the number the
|
||||||
decision cites belongs to another configuration:** Arm F's 5 of 12 was
|
decision cites belongs to another configuration:** Arm F's 5 of 12 was
|
||||||
measured with `--table-grid` ON; without it the same sample scores 2 of 12
|
measured with `--table-grid` ON; without it the same sample scores 2 of 12
|
||||||
and `docx` 0 of 3, because the fold's table clause has no joined table to
|
and `docx` 0 of 3, because the fold's table clause has no joined table to
|
||||||
|
|
@ -312,12 +340,18 @@ and fixtures, never code.
|
||||||
`--unit-fold` (Arm F), `--keep-table-heading` (D1), `--sheet-section-rows`
|
`--unit-fold` (Arm F), `--keep-table-heading` (D1), `--sheet-section-rows`
|
||||||
and `--drop-wrapped-outline` (both D3), `--outline-gate` (G1+G2) and
|
and `--drop-wrapped-outline` (both D3), `--outline-gate` (G1+G2) and
|
||||||
`--first-span-from-zero`, each passed to the proposer unchanged. That last
|
`--first-span-from-zero`, each passed to the proposer unchanged. That last
|
||||||
one is OFF and is not a segmentation rule at all -- it adds no boundary. It
|
one is ON since 2026-09-10 and is not a segmentation rule at all -- it adds
|
||||||
repairs a measured loss the round found while chasing one position's 940
|
no boundary, and the K2 concept count is identical with and without it
|
||||||
characters: **32 of the 32** documents that get a plan leave the text above
|
(425 = 425 on the 2026-09-09 default). It repairs a measured loss: **32 of
|
||||||
their first concept in NO segment, 159 704 characters, **9.18 %** of the
|
the 32** documents that get a plan left the text above their first concept
|
||||||
corpus, 45 841 from one document. It is off only because moving it changes
|
in NO segment. **The hole is bigger than that rule, and this is the number
|
||||||
every bundle's first span and no hit@8 number stands behind it yet. Until that day the build path called the proposer with no
|
to carry:** measured 2026-09-10, the pre-move default left **207 435
|
||||||
|
characters, 11.92 %** of the corpus in no segment -- 163 804 above the first
|
||||||
|
entry, 26 041 BETWEEN entries, 17 590 after the last. The rule closes the
|
||||||
|
first part entirely and 79 % of the whole; **43 631 characters, 2.51 %, over
|
||||||
|
8 of 32 documents remain**, and the between-part has a named mechanism (a
|
||||||
|
`rule:table-block` candidate displacing a DECLARED heading and opening below
|
||||||
|
it). Neither remainder is a ceiling; both are in STATE with their numbers. Until that day the build path called the proposer with no
|
||||||
arm flag at all, so a tender PDF that Arm D splits into nine concepts landed
|
arm flag at all, so a tender PDF that Arm D splits into nine concepts landed
|
||||||
as one -- a build path a full arm behind the proposer. Exposing them was not
|
as one -- a build path a full arm behind the proposer. Exposing them was not
|
||||||
the same decision as moving one, and the two were taken a session apart:
|
the same decision as moving one, and the two were taken a session apart:
|
||||||
|
|
@ -440,8 +474,11 @@ and fixtures, never code.
|
||||||
measurement behind it, including the control that FAILED, is
|
measurement behind it, including the control that FAILED, is
|
||||||
`docs/2026-09-07-okf-konsumskill-maaling.md`. **The ranking is this
|
`docs/2026-09-07-okf-konsumskill-maaling.md`. **The ranking is this
|
||||||
repository's own choice** — the contract binds a payload, not a retrieval
|
repository's own choice** — the contract binds a payload, not a retrieval
|
||||||
algorithm (§ 10) — and it has FOUR optional widenings, all **off by default**
|
algorithm (§ 10) — and it has FOUR optional widenings. Three are **off by
|
||||||
and all keeping the default payload byte-identical. `--cost-vocabulary`: a
|
default** and keep the default payload byte-identical; the fourth
|
||||||
|
(`--tie-shared-rank`) became the default 2026-09-10 and is the one change in
|
||||||
|
this repository that alters a payload with NO bundle changing, so a consumer
|
||||||
|
pinned to the old excerpt order needs `--no-tie-shared-rank`. `--cost-vocabulary`: a
|
||||||
declared cost/price/quantity vocabulary family that bridges a question and a
|
declared cost/price/quantity vocabulary family that bridges a question and a
|
||||||
document naming money with different words, gated on the QUESTION carrying
|
document naming money with different words, gated on the QUESTION carrying
|
||||||
such a term, so a question without one is byte-identical either way. It moves
|
such a term, so a question without one is byte-identical either way. It moves
|
||||||
|
|
@ -470,7 +507,9 @@ and fixtures, never code.
|
||||||
read as 135-of-446 common, and RRF consumes RANKS, so no weighting inside a
|
read as 135-of-446 common, and RRF consumes RANKS, so no weighting inside a
|
||||||
signal can move a gold that already leads it.
|
signal can move a gold that already leads it.
|
||||||
`docs/2026-09-08-sjeldenhetsvekt.md`.
|
`docs/2026-09-08-sjeldenhetsvekt.md`.
|
||||||
`--tie-shared-rank` is the fourth, and it is a correction to the TIE-BREAK
|
`--tie-shared-rank` is the fourth and **the only one that is now ON**
|
||||||
|
(2026-09-10, opt-out `--no-tie-shared-rank`). It is a correction to the
|
||||||
|
TIE-BREAK
|
||||||
rather than a weight: RRF ranks every concept in every signal, including a
|
rather than a weight: RRF ranks every concept in every signal, including a
|
||||||
signal that scored them all the same, and the declared `(-score, concept_id)`
|
signal that scored them all the same, and the declared `(-score, concept_id)`
|
||||||
tie-break then orders that group by id. Measured on N500, whose document
|
tie-break then orders that group by id. Measured on N500, whose document
|
||||||
|
|
@ -478,13 +517,28 @@ and fixtures, never code.
|
||||||
alphabetical UUID order and put a concept answering 7 of 7 question tokens at
|
alphabetical UUID order and put a concept answering 7 of 7 question tokens at
|
||||||
fused rank 14 — outside the cut — behind concepts sharing only `tunnel` and
|
fused rank 14 — outside the cut — behind concepts sharing only `tunnel` and
|
||||||
`vann`. Under shared ranks it is rank 3 and 2 of the 16 covering concepts are
|
`vann`. Under shared ranks it is rank 3 and 2 of the 16 covering concepts are
|
||||||
delivered. Off by default BY MEASUREMENT: the three requirement lookups hold
|
delivered. **It shipped OFF on a measurement that was CONDITIONAL and stopped
|
||||||
at rank 1 and the K2 digest holds, but hit@8 over the six published questions
|
being true in a commit reported as changing nothing.** The published cost —
|
||||||
falls 5 of 6 to 4 of 6, because K2's prior is coarse rather than degenerate
|
hit@8 falling 5 of 6 to 4 of 6 — is real only at `DOCUMENT_PRIOR_EXPONENT`
|
||||||
(6 values over 39 documents) and one gold sat early in its tie group — a
|
1.0. Round 6 moved that exponent to 0.5 for an unrelated reason and correctly
|
||||||
benefit that was never a measurement, but a published row all the same.
|
reported it moved no hit@8 row; nobody measured the PAIR. Swept 2026-09-10
|
||||||
`docs/2026-09-08-rangeringsbom-sammensatte-ord.md`.
|
over 2 exponents x 3 bundles x 6 rows: at 0.5 the rule holds
|
||||||
A FIFTH flag is not a ranking widening and is listed apart: `--withheld-titles`
|
`[1,1,1,1,1,–]` on all three bundles and FIXES the split bundle's row 1
|
||||||
|
(2 -> 1), which is what let `--sheet-section-rows --keep-table-heading`
|
||||||
|
become a build default. **A flag's "off by measurement" is a measurement of a
|
||||||
|
CONFIGURATION, not a property of the flag** — when a constant it interacts
|
||||||
|
with moves, its default is unmeasured again, and nothing in the tree says so
|
||||||
|
because the two decisions live in different files. The adverse case is
|
||||||
|
recorded rather than hidden: on a synthetic 30-concept fixture where one
|
||||||
|
signal separates and two do not, shared ranks move a gold from rank 18 to 30
|
||||||
|
(`tests/test_okf_consume.py`). Note also that
|
||||||
|
`docs/2026-09-08-sjeldenhetsvekt.md`'s figures were measured under the older
|
||||||
|
tie-break and are NOT re-measured — on one fixture the change takes the
|
||||||
|
weight's gold from fused rank 18 to 1.
|
||||||
|
`docs/2026-09-08-rangeringsbom-sammensatte-ord.md` and
|
||||||
|
`docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md`.
|
||||||
|
The other three stay off. A FIFTH flag is not a ranking widening and is
|
||||||
|
listed apart: `--withheld-titles`
|
||||||
gives each `withheld` entry the concept's `title`, so a reader can see WHAT
|
gives each `withheld` entry the concept's `title`, so a reader can see WHAT
|
||||||
was withheld without reading the bundle (§ 2.2 forbids going to look). The
|
was withheld without reading the bundle (§ 2.2 forbids going to look). The
|
||||||
code is 11 lines; the bytes are the reason it is off. Measured, it grows an
|
code is 11 lines; the bytes are the reason it is off. Measured, it grows an
|
||||||
|
|
|
||||||
59
README.md
59
README.md
|
|
@ -88,19 +88,25 @@ did not pass them.
|
||||||
|
|
||||||
### The segmentation flags
|
### The segmentation flags
|
||||||
|
|
||||||
Eight rules are reachable from `okf build`. **Five of them are ON by default**
|
Eight rules are reachable from `okf build`, and since 2026-09-10 **all eight
|
||||||
— `--outline-run 3`, `--table-grid` and `--unit-fold` since 2026-09-08, and
|
are ON by default** — `--outline-run 3`, `--table-grid` and `--unit-fold` since
|
||||||
`--drop-wrapped-outline` and `--outline-gate` since 2026-09-09 — each an
|
2026-09-08, `--drop-wrapped-outline` and `--outline-gate` since 2026-09-09, and
|
||||||
operator decision, and each with an explicit opt-out: `--outline-run 0`,
|
`--sheet-section-rows`, `--keep-table-heading` and `--first-span-from-zero`
|
||||||
`--no-table-grid`, `--no-unit-fold`, `--keep-wrapped-outline`,
|
since 2026-09-10 — each an operator decision, and each with an explicit
|
||||||
`--no-outline-gate`. Passing all five reproduces the pre-2026-09-08 bytes
|
opt-out: `--outline-run 0`, `--no-table-grid`, `--no-unit-fold`,
|
||||||
exactly. The other three are off. Each line below carries the number it was
|
`--keep-wrapped-outline`, `--no-outline-gate`, `--no-sheet-section-rows`,
|
||||||
measured at, and nothing beyond it.
|
`--no-keep-table-heading`, `--no-first-span-from-zero`. Passing all eight
|
||||||
|
reproduces the pre-2026-09-08 bytes exactly, and the last three reproduce the
|
||||||
|
pre-2026-09-10 bundle byte for byte — measured with `diff -rq`, 0 differences,
|
||||||
|
not asserted. Each line below carries the number it was measured at, and
|
||||||
|
nothing beyond it.
|
||||||
|
|
||||||
**A re-run is what this costs a consumer, and it is not a small one:** on the
|
**A re-run is what this costs a consumer, and it is not a small one:** on the
|
||||||
43-document reference corpus the default bundle goes from **629 concepts in
|
43-document reference corpus the default bundle goes from **629 concepts in
|
||||||
1108 files** (the 2026-09-03 tree) to **492 in 944** after the 2026-09-08 move
|
1108 files** (the 2026-09-03 tree) to **492 in 944** after the 2026-09-08 move,
|
||||||
and to **425 in 810** after the 2026-09-09 one. The proposer's own defaults
|
to **425 in 810** after the 2026-09-09 one and to **436 in 832** after the
|
||||||
|
2026-09-10 one (digest `8dff8a8e6c15d2f7…`). On a five-document folder the last
|
||||||
|
move is **15 concepts in 30 files → 26 in 52**. The proposer's own defaults
|
||||||
(`tools/okf_propose_segments.py`) did NOT move, so every published reproduction
|
(`tools/okf_propose_segments.py`) did NOT move, so every published reproduction
|
||||||
block still runs as written.
|
block still runs as written.
|
||||||
|
|
||||||
|
|
@ -117,11 +123,11 @@ rank was measured in the same session and did NOT ship; see
|
||||||
| `--outline-run N` (default **3**) | also propose a boundary where the document's own bare-integer numbering sustains an ascending run of at least `N`; `0` is this arm's opt-out | a tender PDF whose headings are bare integers: **no boundary** at `0`, **9 concepts** at `3`, against a reference of 9 |
|
| `--outline-run N` (default **3**) | also propose a boundary where the document's own bare-integer numbering sustains an ascending run of at least `N`; `0` is this arm's opt-out | a tender PDF whose headings are bare integers: **no boundary** at `0`, **9 concepts** at `3`, against a reference of 9 |
|
||||||
| `--table-grid` (**on** by default; opt out with `--no-table-grid`) | a pandoc grid-table rule line no longer closes an open table block, so one grid table is one concept | a `.docx` experience list: **21 → 6** concepts |
|
| `--table-grid` (**on** by default; opt out with `--no-table-grid`) | a pandoc grid-table rule line no longer closes an open table block, so one grid table is one concept | a `.docx` experience list: **21 → 6** concepts |
|
||||||
| `--unit-fold` (**on** by default; opt out with `--no-unit-fold`) | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match — but that figure was measured with `--table-grid` ON, and the shipped default does not include it. Measured without it the same sample scores **2 of 12**, `docx` **0 of 3**, because the fold's table clause has no joined table to fold |
|
| `--unit-fold` (**on** by default; opt out with `--no-unit-fold`) | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match — but that figure was measured with `--table-grid` ON, and the shipped default does not include it. Measured without it the same sample scores **2 of 12**, `docx` **0 of 3**, because the fold's table clause has no joined table to fold |
|
||||||
| `--keep-table-heading` | keep a heading whose body is empty only because a table opens under it, and absorb that table into its span | the two spreadsheets in that corpus, and **0 of 32 `pdf` and 0 of 5 `docx`**: the concept count does not move (1 → 1), its first byte does — the concept gains the heading line it was missing |
|
| `--keep-table-heading` (**on** by default since 2026-09-10; opt out with `--no-keep-table-heading`) | keep a heading whose body is empty only because a table opens under it, and absorb that table into its span | the two spreadsheets in that corpus, and **0 of 32 `pdf` and 0 of 5 `docx`**: the concept count does not move (1 → 1), its first byte does — the concept gains the heading line it was missing |
|
||||||
| `--sheet-section-rows` | cut an open table block at the rows that label its sections — a run of at least three rows whose first cell is a bare numeric label. The opposite direction from `--table-grid`, which decides how far a block extends | a tender price sheet whose whole body is one table block: **1 → 12 concepts**, against a reference of 11 cost groups plus the sheet's preamble. Whole corpus: **1 of 39** readable documents changes, **0 of 32 `pdf`, 0 of 5 `docx`, 1 of 2 `xlsx`** |
|
| `--sheet-section-rows` (**on** by default since 2026-09-10; opt out with `--no-sheet-section-rows`) | cut an open table block at the rows that label its sections — a run of at least three rows whose first cell is a bare numeric label. The opposite direction from `--table-grid`, which decides how far a block extends | a tender price sheet whose whole body is one table block: **1 → 12 concepts**, against a reference of 11 cost groups plus the sheet's preamble. Whole corpus: **1 of 39** readable documents changes, **0 of 32 `pdf`, 0 of 5 `docx`, 1 of 2 `xlsx`**. It reached 11 of 12 on the reference two rounds before it shipped, and was held back both times by a RETRIEVAL cost that turned out not to be its own: on a bundle built with it the gold document splits 1 → 12 concepts and row 1 of the hit@8 set fell rank 1 → 2. The repair is on the reading side (`--tie-shared-rank`, now the default), and with it in place the sheet reaches 11 of 12 with hit@8 holding **5 of 6 at ranks 1,1,1,1,1,–** |
|
||||||
| `--drop-wrapped-outline` (**on** by default since 2026-09-09; opt out with `--keep-wrapped-outline`) | do not admit an `--outline-run` candidate whose line continues onto the next one. Judges recovered candidates only, never a heading the document declares | quoted regulation text, whose numbered paragraphs match the outline grammar exactly: **4 → 1 concepts**, the reference. Whole corpus: **5 of 39**, all `pdf`; on the 12-document sample **8 of 34** outline candidates wrap, and none of the 26 the operator kept. On the reference it carries `pdf` from **5 of 8 to 6 of 8** together with the gate below, and neither reaches 7 of 8 without the other |
|
| `--drop-wrapped-outline` (**on** by default since 2026-09-09; opt out with `--keep-wrapped-outline`) | do not admit an `--outline-run` candidate whose line continues onto the next one. Judges recovered candidates only, never a heading the document declares | quoted regulation text, whose numbered paragraphs match the outline grammar exactly: **4 → 1 concepts**, the reference. Whole corpus: **5 of 39**, all `pdf`; on the 12-document sample **8 of 34** outline candidates wrap, and none of the 26 the operator kept. On the reference it carries `pdf` from **5 of 8 to 6 of 8** together with the gate below, and neither reaches 7 of 8 without the other |
|
||||||
| `--outline-gate` (**on** by default since 2026-09-09; opt out with `--no-outline-gate`) | admit `--outline-run`'s RECOVERED headings only where the document declares none of its own, plus any one recovered heading whose span covers `OUTLINE_SHARE` (0.20) of the text. Applied at admission, before spans are closed, so the text a removed mark opened is carried by the mark above it rather than lost | on the 12-document sample: `pdf` **2 of 8 → 5 of 8** alone and **7 of 8** with the rule above, `docx` unchanged at **3 of 3**. Whole corpus: it fires on **25 of 39** readable documents, changes the plan in **15 of 39**, and removes **64 of 485** proposed entries. No plan disappears (32 → 32) |
|
| `--outline-gate` (**on** by default since 2026-09-09; opt out with `--no-outline-gate`) | admit `--outline-run`'s RECOVERED headings only where the document declares none of its own, plus any one recovered heading whose span covers `OUTLINE_SHARE` (0.20) of the text. Applied at admission, before spans are closed, so the text a removed mark opened is carried by the mark above it rather than lost | on the 12-document sample: `pdf` **2 of 8 → 5 of 8** alone and **7 of 8** with the rule above, `docx` unchanged at **3 of 3**. Whole corpus: it fires on **25 of 39** readable documents, changes the plan in **15 of 39**, and removes **64 of 485** proposed entries. No plan disappears (32 → 32) |
|
||||||
| `--first-span-from-zero` | start the first concept at character 0, so the text above it belongs to a segment instead of to none. Adds no boundary and removes none | **off by default.** Measured over the 39-document corpus: **32 of the 32** documents that get a plan lose their opening today — **159 704 characters, 9.18 %** of the corpus, 45 841 from one document. On the 12-position reference it changes nothing (10 of 12 either way). It is off because it moves the first span of essentially every bundle and no hit@8 number has been measured behind it yet |
|
| `--first-span-from-zero` (**on** by default since 2026-09-10; opt out with `--no-first-span-from-zero`) | start the first concept at character 0, so the text above it belongs to a segment instead of to none. Adds no boundary and removes none | Measured over the 39-document corpus, the default before this rule left **207 435 characters — 11.92 %** — in no segment at all: **163 804 above the first entry** (in **32 of the 32** documents that get a plan), 26 041 *between* entries and 17 590 after the last. This rule closes the first part entirely, 79 % of the whole, leaving **43 631 characters (2.51 %) over 8 of 32 documents** with two named mechanisms of their own. It adds no boundary and the K2 concept count is identical with and without it (**425 = 425**); on the 12-position reference it changes **not one cell**, and hit@8 on a K2 bundle built with it holds **5 of 6 at ranks 1,1,1,1,1,–** under both tie-breaks |
|
||||||
|
|
||||||
They compose, and the order above is the order they apply in. Measured on a
|
They compose, and the order above is the order they apply in. Measured on a
|
||||||
five-document tender folder (2 `pdf`, 2 `docx`, 1 `xlsx`), concepts per
|
five-document tender folder (2 `pdf`, 2 `docx`, 1 `xlsx`), concepts per
|
||||||
|
|
@ -235,7 +241,32 @@ concept from withheld to delivered and a priced sheet from candidate rank 10 to
|
||||||
the four-character prefix matcher makes a unique identifier read as
|
the four-character prefix matcher makes a unique identifier read as
|
||||||
135-of-446 common on that bundle. Where it cannot help is decomposed rather
|
135-of-446 common on that bundle. Where it cannot help is decomposed rather
|
||||||
than guessed: RRF fuses RANKS, so a weight moves nothing on a signal the gold
|
than guessed: RRF fuses RANKS, so a weight moves nothing on a signal the gold
|
||||||
already leads. `docs/2026-09-08-sjeldenhetsvekt.md`.
|
already leads. `docs/2026-09-08-sjeldenhetsvekt.md`. Its published figures were
|
||||||
|
measured under the pre-2026-09-10 tie-break and are not re-measured.
|
||||||
|
|
||||||
|
`--tie-shared-rank` is **on** by default since 2026-09-10 (opt out with
|
||||||
|
`--no-tie-shared-rank`), and it is the one change in this library that alters a
|
||||||
|
payload with no bundle changing — a consumer pinned to the previous excerpt
|
||||||
|
order needs the opt-out. RRF emits a rank for every concept in every signal,
|
||||||
|
including a signal that scored them all the same, and the declared tie-break
|
||||||
|
then orders that group by `concept_id`; the fusion reads alphabetical order as
|
||||||
|
if it were a measurement. Shared ranks make a signal that separates nothing
|
||||||
|
contribute the same constant to each concept in the group. What it buys is
|
||||||
|
general rather than cosmetic: a document the segmenter splits from 1 concept
|
||||||
|
into 12 fills that signal's whole top tie group with its own concepts, so the
|
||||||
|
one that leads the body signal takes position 11 instead of 1 and the document
|
||||||
|
loses fused rank 1 to a single-concept competitor leading nothing — **the
|
||||||
|
fusion was punishing fine-graining for being fine-grained**, which put the
|
||||||
|
segmentation side and the retrieval side in competition over one number.
|
||||||
|
|
||||||
|
It shipped OFF on 2026-09-08 because hit@8 fell 5 of 6 to 4 of 6, and that
|
||||||
|
figure is real and **conditional**: swept over 2 document-prior exponents x 3
|
||||||
|
bundles x 6 rows, the lost row is lost only at exponent 1.0. The exponent moved
|
||||||
|
to 0.5 on 2026-09-09 for an unrelated reason, correctly reported as moving no
|
||||||
|
hit@8 row, and nobody measured the pair — so a rule sat behind a published
|
||||||
|
number that had stopped being true in the same commit. A flag's "off by
|
||||||
|
measurement" is a measurement of a *configuration*, not a property of the flag.
|
||||||
|
`docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md`.
|
||||||
|
|
||||||
It emits the § 8 shape — `contract`, `bundle` (`bundle_id` plus a
|
It emits the § 8 shape — `contract`, `bundle` (`bundle_id` plus a
|
||||||
`sha256-tree:` content identity), `budget` (unit, instrument, limit, spent and a
|
`sha256-tree:` content identity), `budget` (unit, instrument, limit, spent and a
|
||||||
|
|
|
||||||
304
docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md
Normal file
304
docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md
Normal file
|
|
@ -0,0 +1,304 @@
|
||||||
|
# K3 round 7: the first span, and the fusion that punished fine-graining
|
||||||
|
|
||||||
|
Round 6 shipped the outline gate and left two things behind: a measured
|
||||||
|
coverage hole it had built the repair for but not the number to move it, and a
|
||||||
|
retrieval regression that kept two spreadsheet rules off the default. This
|
||||||
|
round measured both. Three defaults moved and one file type turned out never to
|
||||||
|
have been measured at all.
|
||||||
|
|
||||||
|
Everything here is measured on this repository at `38104b7` plus this round's
|
||||||
|
changes, with `~/repos/llm-ingestion-okf/.venv/bin/python`, against the cached
|
||||||
|
rendering `/tmp/k3r3/text43` (39 readable documents) and the criterion in
|
||||||
|
`/tmp/k3r5/score5.py` imported unchanged. Bundle digests are
|
||||||
|
`find . -type f | sort | xargs shasum -a 256 | shasum -a 256` from inside the
|
||||||
|
bundle, so they are path-independent.
|
||||||
|
|
||||||
|
## 0. One divergence from the numbers this round was handed
|
||||||
|
|
||||||
|
The order states the coverage loss as **159 704 characters, 9.18 % of the
|
||||||
|
corpus**. Measured here with the shipped default over the same 39 documents,
|
||||||
|
the text above every plan's first entry is **163 804 characters, 9.42 %** — a
|
||||||
|
difference of 4 100 characters, 2.5 % of the figure. Both agree on the
|
||||||
|
denominator (39 documents, 32 with a plan) and on the shape of the finding.
|
||||||
|
The method used here is in § 1 and the 4 100 are not chased further; where the
|
||||||
|
two disagree the number in this report is the one measured today.
|
||||||
|
|
||||||
|
The same measurement also found more of the hole than the figure it was handed
|
||||||
|
covers, which is § 1.
|
||||||
|
|
||||||
|
## 1. Coverage: the hole is bigger than the first span, and the first span was 79 % of it
|
||||||
|
|
||||||
|
Characters lying in NO segment, over the 39-document corpus (1 739 511
|
||||||
|
characters, 32 documents with a plan):
|
||||||
|
|
||||||
|
| configuration | in no segment | share | documents | above first | between | after last |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| pre-round-7 default | 207 435 | **11.92 %** | 32 of 32 | 163 804 | 26 041 | 17 590 |
|
||||||
|
| + `--first-span-from-zero` | 43 631 | 2.51 % | 8 of 32 | **0** | 26 041 | 17 590 |
|
||||||
|
| round 7 default (all three) | 43 631 | **2.51 %** | 8 of 32 | 0 | 26 041 | 17 590 |
|
||||||
|
|
||||||
|
`--first-span-from-zero` closes **163 804 of 207 435**, 79 % of the hole, and
|
||||||
|
it closes the whole of the part that hits every document. What remains is
|
||||||
|
**43 631 characters, 2.51 % of the corpus, over 8 of 32 documents**, in two
|
||||||
|
named parts that are NOT the same defect and are not fixed here:
|
||||||
|
|
||||||
|
* **26 041 characters between one entry's end and the next one's start.** The
|
||||||
|
mechanism is § 4's: a `rule:table-block` candidate opens BELOW the heading
|
||||||
|
line that introduces it, so the heading's own line falls between two spans.
|
||||||
|
* **17 590 characters after the last entry ends.** Not investigated this round.
|
||||||
|
|
||||||
|
Both are written into STATE with their mechanism. Neither is a ceiling; each is
|
||||||
|
a measurement with a denominator and a named cause.
|
||||||
|
|
||||||
|
### 1.1 What it cost to move it
|
||||||
|
|
||||||
|
`--first-span-from-zero` adds no boundary, and the measurement says so twice
|
||||||
|
over. On the 12-position reference it changes **not one cell** — `pdf` 7 of 8,
|
||||||
|
`docx` 3 of 3, `xlsx` 0 of 1, sheet 10 of 12, identical to the default without
|
||||||
|
it, per position and per label. On a K2 bundle built with it: **425 concepts /
|
||||||
|
810 files**, the same count as the default without it, and hit@8 `[1, 1, 1, 1,
|
||||||
|
1, None]` = 5 of 6 — **under both tie-breaks**, so its acceptance does not
|
||||||
|
borrow from § 2's move.
|
||||||
|
|
||||||
|
## 2. The fusion punished fine-graining, and the repair was already in the tree
|
||||||
|
|
||||||
|
### 2.1 The decomposition
|
||||||
|
|
||||||
|
Round 6 measured that a K2 bundle built with `--sheet-section-rows
|
||||||
|
--keep-table-heading` takes row 1 of the hit@8 set from rank 1 to rank 2, and
|
||||||
|
held the two rules back for it. It also measured that the DOCUMENT-prior rank
|
||||||
|
is 1 on both bundles, which ruled the prior out — correctly, and that is why
|
||||||
|
the cause stayed unfound: the prior's rank over 39 DOCUMENTS is not the prior
|
||||||
|
signal's rank over 425 CONCEPTS, and RRF reads the second.
|
||||||
|
|
||||||
|
Per-signal, row 1, gold concept against the concept that takes fused rank 1:
|
||||||
|
|
||||||
|
| bundle | gold doc's units | gold: title / body / prior rank | competitor: title / body / prior | fused |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| default (425) | 1 | 6 / 4 / **1** | — (gold IS rank 1) | gold 1 |
|
||||||
|
| D1+D3 (436) | 12 | 16 / 4 / **11** | 5 / 9 / 13 | gold **2** |
|
||||||
|
|
||||||
|
The prior signal on the split bundle scores all twelve of the gold document's
|
||||||
|
concepts identically (2.8735) and they take positions **1 through 12**. The
|
||||||
|
one that leads the body signal lands at position 11 and contributes `1/71`
|
||||||
|
where the undivided concept contributed `1/61`. The competitor is a
|
||||||
|
one-concept document that leads nothing and is beaten in every signal it can
|
||||||
|
be compared on — it wins because its evidence is in one unit and the gold's is
|
||||||
|
spread over twelve, and because the gold's own siblings fill the tie group
|
||||||
|
ahead of it.
|
||||||
|
|
||||||
|
So the order's hypothesis is right in its conclusion and needed one correction
|
||||||
|
in its mechanism: it is not that twelve concepts are each individually weaker.
|
||||||
|
It is that a signal which scores them all EQUALLY still emits twelve distinct
|
||||||
|
ranks, ordered by `concept_id`, and the fusion reads that alphabetical order as
|
||||||
|
if it were a measurement.
|
||||||
|
|
||||||
|
### 2.2 That rule already existed and was off for a cost that had been removed
|
||||||
|
|
||||||
|
`--tie-shared-rank` — shipped 2026-09-08, off by default — is exactly this
|
||||||
|
repair. Its published reason for being off is that hit@8 fell 5 of 6 to 4 of 6.
|
||||||
|
**That figure is real and it is conditional.** Swept over the prior exponent,
|
||||||
|
3 bundles x 6 rows:
|
||||||
|
|
||||||
|
| exponent | bundle | `tie_shared_rank` off | on |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1.0 | Arm B (629) | `1,1,1,1,1,–` 5/6 | `1,–,1,1,1,–` **4/6** |
|
||||||
|
| 1.0 | default (425) | `1,1,1,1,1,–` 5/6 | `1,1,1,1,1,–` 5/6 |
|
||||||
|
| 1.0 | D1+D3 (436) | `2,1,1,1,1,–` 5/6 | `1,1,1,1,1,–` 5/6 |
|
||||||
|
| **0.5** | Arm B (629) | `1,1,1,1,1,–` 5/6 | `1,1,1,1,1,–` **5/6** |
|
||||||
|
| **0.5** | default (425) | `1,1,1,1,1,–` 5/6 | `1,1,1,1,1,–` 5/6 |
|
||||||
|
| **0.5** | D1+D3 (436) | `2,1,1,1,1,–` 5/6 | `1,1,1,1,1,–` 5/6 |
|
||||||
|
|
||||||
|
Round 6 moved `DOCUMENT_PRIOR_EXPONENT` from 1.0 to 0.5 for an unrelated
|
||||||
|
reason and reported, correctly, that it moved no hit@8 row end to end. What
|
||||||
|
nobody measured is the PAIR: the exponent move removed the only cost keeping
|
||||||
|
`--tie-shared-rank` off. A rule sat behind a published number that had stopped
|
||||||
|
being true in the same commit that was reported as changing nothing.
|
||||||
|
|
||||||
|
**The lesson is the durable one from this round**: a flag's "off by
|
||||||
|
measurement" is a measurement of a configuration, not a property of the flag.
|
||||||
|
When a constant that flag interacts with moves, its default is unmeasured
|
||||||
|
again — and nothing in the tree said so, because the two decisions live in
|
||||||
|
different files.
|
||||||
|
|
||||||
|
### 2.3 What shipped, and the control in the same measurement
|
||||||
|
|
||||||
|
`consume.DEFAULT_TIE_SHARED_RANK = True`, opt-out `--no-tie-shared-rank`. With
|
||||||
|
it, all three of the round's acceptance conditions hold at once, and the
|
||||||
|
build-side rules could move:
|
||||||
|
|
||||||
|
| bundle | flags over the pre-round-7 default | concepts | files | digest | hit@8 | ranks |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| pre-round-7 default | — | 425 | 810 | `bdf4977ca5a443c4…` | 5 of 6 | 1,1,1,1,1,– |
|
||||||
|
| + first span | `--first-span-from-zero` | 425 | 810 | — | 5 of 6 | 1,1,1,1,1,– |
|
||||||
|
| **round 7 default** | the above `--sheet-section-rows --keep-table-heading` | **436** | **832** | `8dff8a8e6c15d2f7…` | **5 of 6** | **1,1,1,1,1,–** |
|
||||||
|
| round 7 default, `--no-tie-shared-rank` | (same bytes, read differently) | 436 | 832 | same | 5 of 6 | **2**,1,1,1,1,– |
|
||||||
|
|
||||||
|
The last row is the known-negative and it is on the SHIPPED bytes rather than a
|
||||||
|
fixture: the regression round 6 refused to ship is reproduced on the bundle
|
||||||
|
that now ships, by turning off the one thing that removes it.
|
||||||
|
`tests/test_default_bundle_pin.py` holds both rows.
|
||||||
|
|
||||||
|
## 3. The reference sheet, and the two readings of `xlsx`
|
||||||
|
|
||||||
|
Reproduced before any code moved, criterion unchanged:
|
||||||
|
|
||||||
|
| configuration | `pdf` | `docx` | `xlsx` | sheet |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| pre-round-7 default | 7 of 8 | 3 of 3 | 0 of 1 | 10 of 12 |
|
||||||
|
| + `--first-span-from-zero` | 7 of 8 | 3 of 3 | 0 of 1 | 10 of 12 (not one cell moves) |
|
||||||
|
| + `--sheet-section-rows --keep-table-heading`, `|F|` = 11 | 7 of 8 | 3 of 3 | 0 of 1 | 10 of 12 |
|
||||||
|
| + the same, `|F|` = 12 | 7 of 8 | 3 of 3 | **1 of 1** | **11 of 12** |
|
||||||
|
|
||||||
|
Per position on the default, all at distance 0: pos 1 `treffer`/15, pos 2 /9,
|
||||||
|
pos 4 /1, pos 6 /6, pos 7 /30, pos 8 /8, pos 9 /4, pos 5, 10, 11 `docx`
|
||||||
|
`treffer`. Pos 0 is `for grovt` with U=1 and `|F|` undefined. Pos 3 is
|
||||||
|
`for grovt` with U=0, S=1.
|
||||||
|
|
||||||
|
**D-2 is unanswered and this report does not answer it.** `F[3]` was not
|
||||||
|
touched in the criterion. What the measurement adds is that the choice between
|
||||||
|
the two readings does not decide whether the rules ship: under `|F|` = 11 the
|
||||||
|
sheet rules leave the sheet at 10 of 12 and take position 3's distance from
|
||||||
|
**12 to 1** (U=12 against 11, `for fint`); under `|F|` = 12 they take it to
|
||||||
|
`treffer`. Neither reading is worse off with them on.
|
||||||
|
|
||||||
|
## 4. `pptx` and `md`: the mandate's weakest axis, now measured
|
||||||
|
|
||||||
|
`extract.py:76` marks `.pptx` `unmeasured` and the 12-position reference is 8
|
||||||
|
`pdf`, 3 `docx`, 1 `xlsx` — no `pptx`, no `md`. Two documents with a
|
||||||
|
hand-known structure were built for this (a markdown file with 1 + 3 declared
|
||||||
|
headings, and the same content written to `.pptx` through the converter). The
|
||||||
|
fasit is NOT invented: each document's structure is what its author wrote, and
|
||||||
|
the question asked is only whether the default recovers it.
|
||||||
|
|
||||||
|
**`pptx` extracts, and its segmentation depends on a property of the deck.**
|
||||||
|
Two decks, both extracted with rc 0 and no rejection code:
|
||||||
|
|
||||||
|
| deck | structure | concepts | matches |
|
||||||
|
|---|---|---|---|
|
||||||
|
| section slide + 3 content slides | 4 slides | **1** | no |
|
||||||
|
| 4 title slides | 4 slides | **4** | yes |
|
||||||
|
|
||||||
|
The mechanism: the converter emits one ATX heading per slide **that carries a
|
||||||
|
title placeholder it recognised**, and plain body lines for the rest. In deck 1
|
||||||
|
only slide 1 got a heading, so the document DECLARES one heading — and the
|
||||||
|
outline gate, correctly by its own rule, then admits none of Arm D's recovered
|
||||||
|
ones. The whole deck lands as one concept. This is round 6's principle working
|
||||||
|
exactly as designed on an input its measurement never contained.
|
||||||
|
|
||||||
|
**`md` extracts and loses one of four declared headings**: 3 concepts against
|
||||||
|
the 4 the document declares. Decomposed below.
|
||||||
|
|
||||||
|
**A converter attribute leaks into concept titles, and it is not a `pptx`
|
||||||
|
curiosity.** Every `pptx` concept measured carries `{#slide-N}` in its `title`,
|
||||||
|
and on real corpora the same leak appears as `{#sheet-1}`: **2 of 810** files
|
||||||
|
on the previous default K2 bundle, **2 of 1108** on Arm B, and **1 of 30** on
|
||||||
|
the operator's own test folder. Because a concept's FILENAME is reduced from
|
||||||
|
its title, the leak is in a concept id a consumer has already cited by name.
|
||||||
|
Not fixed here — see § 6.
|
||||||
|
|
||||||
|
## 5. § 4's `md` defect is § 1's remaining 26 041 characters
|
||||||
|
|
||||||
|
The markdown document declares `# Prosjektbeskrivelse` and `## 1`, `## 2`,
|
||||||
|
`## 3`. Candidates, decomposed:
|
||||||
|
|
||||||
|
* with `--no-unit-fold`: four candidates, and the fourth is
|
||||||
|
`rule:table-block` starting at 503 — **not** `rule:heading` at 489. The
|
||||||
|
heading line `## 3 Prising` is 14 characters that belong to no segment.
|
||||||
|
* with the fold on (the default): three candidates. The table block, having no
|
||||||
|
heading of its own, is folded into section 2, whose span now runs to the end
|
||||||
|
of the file. The boundary is gone.
|
||||||
|
|
||||||
|
The root cause is in the first step, not the fold: **a table-block candidate
|
||||||
|
displaces a heading the document DECLARED**, and opens below it. That is the
|
||||||
|
same principle round 6 established for Arm D against declared headings —
|
||||||
|
recovery yields to declaration — not yet applied between the table rule and the
|
||||||
|
heading rule.
|
||||||
|
|
||||||
|
Reach over the 39-document corpus, so this is a measurement and not an
|
||||||
|
anecdote: **5 of 39** documents carry a table-block candidate at all; the fold
|
||||||
|
changes the entry count on **17 of 39**, removing **198** entries; and with the
|
||||||
|
fold off, **18 527 characters over 15 of 39 documents** sit between one
|
||||||
|
candidate's end and the next one's start. Under the shipped default that
|
||||||
|
between-gap is **26 041 characters**.
|
||||||
|
|
||||||
|
Not fixed this round: it changes what the proposer emits on 5 of 39 documents,
|
||||||
|
so it needs its own K2 build and its own hit@8 read, which is one full arm's
|
||||||
|
worth of acceptance. It is in STATE with this mechanism and these numbers.
|
||||||
|
|
||||||
|
## 6. What did NOT move, and why
|
||||||
|
|
||||||
|
* **The `{#…}` leak.** The fix is small; the consequence is not. A concept's
|
||||||
|
filename is reduced from its title, so stripping the attribute RENAMES the
|
||||||
|
affected concepts — including one an external consumer has already cited by
|
||||||
|
its full id in writing. Renaming a concept id another repository names is not
|
||||||
|
this round's call to make; it is a question, with the exposure measured (2 of
|
||||||
|
810, 2 of 1108, 1 of 30).
|
||||||
|
* **`MIN_SHARED_PREFIX` and the prefix over-match.** Untouched, and
|
||||||
|
deliberately: § 2 changes the FUSION and the rarity weight changes a signal's
|
||||||
|
scores. Moving both would measure two changes with one number. The df figure
|
||||||
|
it rests on (159 of 270) is round 5's and is still not re-measured.
|
||||||
|
* **The remaining 17 590 characters after the last entry.** Named, not
|
||||||
|
investigated.
|
||||||
|
* **`--pdf-headings`, `--ocr`, `--rarity-weight`, `--cost-vocabulary`,
|
||||||
|
`--reserve-top-rank`, `--withheld-titles`, `--max-segment-chars`.** All
|
||||||
|
unchanged and all still off.
|
||||||
|
|
||||||
|
## 7. Cost to a consumer
|
||||||
|
|
||||||
|
A re-run, and it is not small. The 43-document reference corpus:
|
||||||
|
|
||||||
|
* **629 concepts / 1 108 files** (Arm B, delivered 2026-09-03)
|
||||||
|
* **492 / 944** after the 2026-09-08 move
|
||||||
|
* **425 / 810** after the 2026-09-09 move
|
||||||
|
* **436 / 832** after this one — digest `8dff8a8e6c15d2f7…`, default flags,
|
||||||
|
default epoch stamp.
|
||||||
|
|
||||||
|
On the operator's own five-document test folder the same move is **15 concepts
|
||||||
|
/ 30 files → 26 / 52**.
|
||||||
|
|
||||||
|
Every moved default has an explicit opt-out, and the opt-outs were measured
|
||||||
|
rather than asserted: `--no-first-span-from-zero --no-sheet-section-rows
|
||||||
|
--no-keep-table-heading` reproduces the pre-round-7 bundle **byte for byte**
|
||||||
|
(`diff -rq`, 0 differences), and `--no-tie-shared-rank` reproduces the previous
|
||||||
|
excerpt order.
|
||||||
|
|
||||||
|
The reading-side change is the one to flag hardest, because it has no bundle to
|
||||||
|
diff: `okf consume` now returns a different excerpt ORDER on any bundle with a
|
||||||
|
tied signal, without any bundle changing. `--no-tie-shared-rank` is the way
|
||||||
|
back.
|
||||||
|
|
||||||
|
## 8. Honesty limits
|
||||||
|
|
||||||
|
* **n = 12 positions on the reference sheet**, stratified 8 `pdf`, 3 `docx`,
|
||||||
|
1 `xlsx`. Every per-type fraction here is a fraction of 8, 3 or 1. A single
|
||||||
|
`xlsx` position carries the entire `xlsx` claim.
|
||||||
|
* **n = 6 questions on the retrieval side, one rater, one gold set.** Row 6
|
||||||
|
misses on every bundle and every configuration measured so far, so "5 of 6"
|
||||||
|
has always been "5 of 5 reachable rows plus one that no configuration has
|
||||||
|
reached".
|
||||||
|
* **Every bundle here is the SAME 43-document corpus read again.** "General"
|
||||||
|
in this report means "holds across configurations of one corpus", never
|
||||||
|
"holds across corpora". The `pptx`/`md` documents in § 4 are the only inputs
|
||||||
|
measured this round that are not from it, and they were written for it.
|
||||||
|
* **`pptx`, `md`, `odt` and `rtf` remain outside the reference sheet.** § 4
|
||||||
|
measures two hand-built documents, which is more than zero and is not a
|
||||||
|
fasit. `odt` and `rtf` are still `unmeasured` with no document behind them
|
||||||
|
at all.
|
||||||
|
* **`|F|` for position 0 is undefined and position 3's is unanswered.** Both
|
||||||
|
are operator decisions with the numbers already delivered; neither is
|
||||||
|
assumed here.
|
||||||
|
* **"Delivered" is not "answered".** hit@8 measures whether a concept from the
|
||||||
|
gold document reaches the payload, not whether the payload answers the
|
||||||
|
question. The last measurement of that, on 3 bundles, was rank 1 of 8 on 3 of
|
||||||
|
3 and a correct answer on 1 of 3.
|
||||||
|
* **The synthetic adverse case for § 2 is recorded, not hidden.** On a
|
||||||
|
30-concept fixture where one signal separates and two do not, shared ranks
|
||||||
|
move a gold from rank 18 to rank 30. It is in
|
||||||
|
`tests/test_okf_consume.py` with that number. Nothing like it appears in the
|
||||||
|
three real bundles, which is an absence over three corpora, not a proof.
|
||||||
|
* **`docs/2026-09-08-sjeldenhetsvekt.md`'s figures were measured under the
|
||||||
|
pre-2026-09-10 tie-break** and are not re-measured here. On one fixture the
|
||||||
|
change takes the weight's gold from fused rank 18 to rank 1, so at least one
|
||||||
|
of that report's numbers no longer describes the default.
|
||||||
|
* **The 4 100-character divergence in § 0 is unexplained.**
|
||||||
|
|
@ -150,16 +150,35 @@ DEFAULT_UNIT_FOLD = True
|
||||||
DEFAULT_OUTLINE_GATE = True
|
DEFAULT_OUTLINE_GATE = True
|
||||||
DEFAULT_DROP_WRAPPED_OUTLINE = True
|
DEFAULT_DROP_WRAPPED_OUTLINE = True
|
||||||
|
|
||||||
#: Round 6's third rule, and the only one whose default this round did NOT
|
#: Round 6 built it and left it off for want of one number; round 7 measured
|
||||||
#: decide. Measured over the 39-document corpus: every one of the 32 documents
|
#: that number and moved it (2026-09-10). Measured over the 39-document corpus:
|
||||||
#: that gets a plan loses the text above its first concept -- 159 704
|
#: every one of the 32 documents that gets a plan lost the text above its first
|
||||||
#: characters, 9.18 % of the corpus, 45 841 from a single document. The repair
|
#: concept -- 159 704 characters, 9.18 % of the corpus, 45 841 from a single
|
||||||
#: is one line and it adds no boundary. It is off because it moves the first
|
#: document. It adds NO boundary; it moves one span's start, so it is a repair
|
||||||
#: span of essentially every segmented bundle and this round's budget held one
|
#: to coverage rather than a segmentation rule, which is why it sits apart from
|
||||||
#: K2 build, which measured the outline gate; a default that changed every
|
#: the arms above.
|
||||||
#: bundle's first concept with no hit@8 number behind it would be the move this
|
#:
|
||||||
#: round refused to make elsewhere. The number it needs is named in STATE.
|
#: The acceptance it had to clear was the same one round 6 set: the 12-position
|
||||||
DEFAULT_FIRST_SPAN_FROM_ZERO = False
|
#: reference does not move (it changes not one cell, 7 of 8 `pdf`, 3 of 3
|
||||||
|
#: `docx`, 10 of 12) AND hit@8 holds rank 1 on every row it held, measured end
|
||||||
|
#: to end on a K2 bundle built with it. Opt-out `--no-first-span-from-zero`.
|
||||||
|
DEFAULT_FIRST_SPAN_FROM_ZERO = True
|
||||||
|
|
||||||
|
#: Round 3's two spreadsheet rules (D1 and D3), held back through rounds 5 and
|
||||||
|
#: 6 by a RETRIEVAL regression rather than by the reference: they take the
|
||||||
|
#: sheet to 11 of 12 and `xlsx` to its fasit, and on a K2 bundle built with
|
||||||
|
#: them row 1 of the hit@8 set fell from rank 1 to rank 2.
|
||||||
|
#:
|
||||||
|
#: Round 7 decomposed that fall and it was never these rules' fault. Splitting
|
||||||
|
#: the gold document from 1 concept into 12 makes its own 12 concepts occupy
|
||||||
|
#: the document-prior signal's entire top tie group, so the concept leading the
|
||||||
|
#: body signal takes prior position 11 instead of 1 -- RRF reading a tie-break
|
||||||
|
#: by `concept_id` as if it were a measurement. `consume.DEFAULT_TIE_SHARED_RANK`
|
||||||
|
#: is the repair, and with it in place all three of round 7's acceptance
|
||||||
|
#: conditions hold at once. Opt-outs `--no-sheet-section-rows`,
|
||||||
|
#: `--no-keep-table-heading`.
|
||||||
|
DEFAULT_SHEET_SECTION_ROWS = True
|
||||||
|
DEFAULT_KEEP_TABLE_HEADING = True
|
||||||
|
|
||||||
#: Round 4's two PDF READERS, both off. They are not arms and they are not in
|
#: Round 4's two PDF READERS, both off. They are not arms and they are not in
|
||||||
#: the paragraph above: an arm changes how the proposer cuts a text, and these
|
#: the paragraph above: an arm changes how the proposer cuts a text, and these
|
||||||
|
|
@ -530,27 +549,42 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||||
build_parser.add_argument(
|
build_parser.add_argument(
|
||||||
"--keep-table-heading",
|
"--keep-table-heading",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
|
default=DEFAULT_KEEP_TABLE_HEADING,
|
||||||
help=(
|
help=(
|
||||||
"D1, passed to the proposer unchanged: keep a heading whose body is "
|
"D1, passed to the proposer unchanged: keep a heading whose body is "
|
||||||
"empty only because a table opens under it, and absorb that table "
|
"empty only because a table opens under it, and absorb that table "
|
||||||
"into its span. Absent (the default) is OFF. Measured on a tender "
|
"into its span. ON since 2026-09-10. Measured on a tender price "
|
||||||
"price sheet: the concept count does not move (1 -> 1) and the "
|
"sheet: the concept count does not move (1 -> 1) and the concept "
|
||||||
"concept gains the heading line it was missing"
|
"gains the heading line it was missing"
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
build_parser.add_argument(
|
||||||
|
"--no-keep-table-heading",
|
||||||
|
action="store_false",
|
||||||
|
dest="keep_table_heading",
|
||||||
|
help="D1's explicit opt-out. A default a caller cannot turn off is not a default",
|
||||||
|
)
|
||||||
build_parser.add_argument(
|
build_parser.add_argument(
|
||||||
"--sheet-section-rows",
|
"--sheet-section-rows",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
|
default=DEFAULT_SHEET_SECTION_ROWS,
|
||||||
help=(
|
help=(
|
||||||
"D3, passed to the proposer unchanged: cut an open table block at "
|
"D3, passed to the proposer unchanged: cut an open table block at "
|
||||||
"the rows that label its sections -- a RUN of rows whose first cell "
|
"the rows that label its sections -- a RUN of rows whose first cell "
|
||||||
"is a bare numeric label. The opposite direction from Arm E, which "
|
"is a bare numeric label. The opposite direction from Arm E, which "
|
||||||
"decides how far a block extends; this decides where it is cut "
|
"decides how far a block extends; this decides where it is cut "
|
||||||
"inside. Absent (the default) is OFF. Measured on a tender price "
|
"inside. ON since 2026-09-10, once the fusion stopped punishing the "
|
||||||
"sheet whose whole body is one table block: 1 concept -> 12, "
|
"split it produces. Measured on a tender price sheet whose whole "
|
||||||
"against a reference of 11 cost groups plus the sheet's preamble"
|
"body is one table block: 1 concept -> 12, against a reference of "
|
||||||
|
"11 cost groups plus the sheet's preamble"
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
build_parser.add_argument(
|
||||||
|
"--no-sheet-section-rows",
|
||||||
|
action="store_false",
|
||||||
|
dest="sheet_section_rows",
|
||||||
|
help="D3's explicit opt-out. With --no-keep-table-heading it reproduces the pre-2026-09-10 bytes",
|
||||||
|
)
|
||||||
build_parser.add_argument(
|
build_parser.add_argument(
|
||||||
"--drop-wrapped-outline",
|
"--drop-wrapped-outline",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
|
|
@ -599,10 +633,20 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||||
default=DEFAULT_FIRST_SPAN_FROM_ZERO,
|
default=DEFAULT_FIRST_SPAN_FROM_ZERO,
|
||||||
help=(
|
help=(
|
||||||
"Start the first concept at character 0, so the text above it is "
|
"Start the first concept at character 0, so the text above it is "
|
||||||
"carried by a segment instead of by none. Absent (the default) is "
|
"carried by a segment instead of by none. Measured over the "
|
||||||
"OFF. Measured over the 39-document corpus: 32 of the 32 documents "
|
"39-document corpus: 32 of the 32 documents with a plan lost "
|
||||||
"with a plan lose their opening today -- 159 704 characters, 9.18 "
|
"their opening before this moved -- 159 704 characters, 9.18 "
|
||||||
"%% of the corpus. It adds no boundary and moves one span's start"
|
"%% of the corpus. It adds no boundary and moves one span's start. "
|
||||||
|
"ON since 2026-09-10"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
build_parser.add_argument(
|
||||||
|
"--no-first-span-from-zero",
|
||||||
|
action="store_false",
|
||||||
|
dest="first_span_from_zero",
|
||||||
|
help=(
|
||||||
|
"The repair's explicit opt-out, for a consumer who needs the "
|
||||||
|
"pre-2026-09-10 first span byte for byte"
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
build_parser.add_argument(
|
build_parser.add_argument(
|
||||||
|
|
|
||||||
|
|
@ -1063,6 +1063,29 @@ def document_scores(
|
||||||
#: fusion is invariant to any monotone transform of the individual signals.
|
#: fusion is invariant to any monotone transform of the individual signals.
|
||||||
RRF_K = 60
|
RRF_K = 60
|
||||||
|
|
||||||
|
#: Whether concepts a signal scores EQUALLY share that group's first rank
|
||||||
|
#: instead of being ordered inside it by `concept_id`. ON since 2026-09-10.
|
||||||
|
#:
|
||||||
|
#: WHY IT MOVED. It shipped OFF on 2026-09-08 on a measurement -- hit@8 over
|
||||||
|
#: the six published questions fell 5 of 6 to 4 of 6 on the 629-concept bundle.
|
||||||
|
#: Round 7 re-measured that fall and it is CONDITIONAL on the document prior
|
||||||
|
#: being a sum: swept over `DOCUMENT_PRIOR_EXPONENT` x 3 bundles x 6 rows, the
|
||||||
|
#: lost row is lost at exponent 1.0 and held at 0.5. Round 6 moved that
|
||||||
|
#: exponent to 0.5 for an unrelated reason and nobody re-measured the pair, so
|
||||||
|
#: a rule was left off by a cost that had already been removed.
|
||||||
|
#:
|
||||||
|
#: WHAT IT BUYS. It is the repair for the defect that kept `--sheet-section-rows
|
||||||
|
#: --keep-table-heading` off the build default: a document split from 1 concept
|
||||||
|
#: into 12 puts its own 12 concepts in the prior signal's whole top tie group,
|
||||||
|
#: so the concept leading the body signal takes prior position 11 rather than
|
||||||
|
#: 1 and the document loses fused rank 1 to a single-concept competitor that
|
||||||
|
#: leads nothing. Fine-graining was being punished for being fine-grained.
|
||||||
|
#:
|
||||||
|
#: Measured at exponent 0.5 on three bundles, ranks per row: `[1, 1, 1, 1, 1,
|
||||||
|
#: None]` on all three with it on, against `[2, 1, 1, 1, 1, None]` on the split
|
||||||
|
#: bundle with it off. Opt-out `--no-tie-shared-rank`.
|
||||||
|
DEFAULT_TIE_SHARED_RANK = True
|
||||||
|
|
||||||
|
|
||||||
def concept_scores(
|
def concept_scores(
|
||||||
concepts: Sequence[Concept],
|
concepts: Sequence[Concept],
|
||||||
|
|
@ -1072,7 +1095,7 @@ def concept_scores(
|
||||||
cost_vocabulary: bool = False,
|
cost_vocabulary: bool = False,
|
||||||
weights: Mapping[str, float] | None = None,
|
weights: Mapping[str, float] | None = None,
|
||||||
lookup: bool = True,
|
lookup: bool = True,
|
||||||
tie_shared_rank: bool = False,
|
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
|
||||||
) -> list[tuple[Concept, float, int]]:
|
) -> list[tuple[Concept, float, int]]:
|
||||||
"""Every concept, ordered best first, fused from three signals by RRF.
|
"""Every concept, ordered best first, fused from three signals by RRF.
|
||||||
|
|
||||||
|
|
@ -1464,7 +1487,7 @@ def build_payload(
|
||||||
cost_vocabulary: bool = False,
|
cost_vocabulary: bool = False,
|
||||||
reserve_top_rank: bool = False,
|
reserve_top_rank: bool = False,
|
||||||
rarity_weight: bool = False,
|
rarity_weight: bool = False,
|
||||||
tie_shared_rank: bool = False,
|
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
|
||||||
withheld_titles: bool = False,
|
withheld_titles: bool = False,
|
||||||
) -> dict[str, object]:
|
) -> dict[str, object]:
|
||||||
"""One bundle plus one question, cut to one contract-conformant payload.
|
"""One bundle plus one question, cut to one contract-conformant payload.
|
||||||
|
|
@ -1666,13 +1689,22 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--tie-shared-rank",
|
"--tie-shared-rank",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
|
default=DEFAULT_TIE_SHARED_RANK,
|
||||||
help=(
|
help=(
|
||||||
"let concepts a signal scores EQUALLY share that score group's "
|
"let concepts a signal scores EQUALLY share that score group's "
|
||||||
"first rank, so a signal that separates nothing contributes the "
|
"first rank, so a signal that separates nothing contributes the "
|
||||||
"same constant to each of them instead of ordering them by id. OFF "
|
"same constant to each of them instead of ordering them by id. ON "
|
||||||
"by default. See docs/2026-09-08-rangeringsbom-sammensatte-ord.md"
|
"since 2026-09-10. See docs/2026-09-08-rangeringsbom-sammensatte-ord.md "
|
||||||
|
"for the rule and docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md "
|
||||||
|
"for why its published cost no longer holds"
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--no-tie-shared-rank",
|
||||||
|
action="store_false",
|
||||||
|
dest="tie_shared_rank",
|
||||||
|
help="The rule's explicit opt-out, reproducing the pre-2026-09-10 order",
|
||||||
|
)
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--withheld-titles",
|
"--withheld-titles",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
|
|
|
||||||
|
|
@ -638,10 +638,13 @@ def test_sheet_section_rows_reaches_the_proposer_from_the_build_command(tmp_path
|
||||||
|
|
||||||
|
|
||||||
def test_the_build_default_leaves_a_sheet_as_one_concept(tmp_path: Path) -> None:
|
def test_the_build_default_leaves_a_sheet_as_one_concept(tmp_path: Path) -> None:
|
||||||
"""The control the test above rests on: same inbox, no flag, one concept.
|
"""The control the test above rests on: same inbox, D3 OFF, one concept.
|
||||||
|
|
||||||
Without it, a change that turned D3 on by default would leave the assertion
|
D3 became the default on 2026-09-10, so the control now names the opt-out
|
||||||
above green while cutting every table a consumer has ever ingested.
|
rather than saying nothing. It is still a control and still load-bearing:
|
||||||
|
it is what makes the assertion above a statement about D3 rather than a
|
||||||
|
statement about this fixture, and it is the test that goes red if
|
||||||
|
`--no-sheet-section-rows` ever stops reproducing the pre-2026-09-10 cut.
|
||||||
"""
|
"""
|
||||||
inbox = inbox_for_sheet_sections(tmp_path)
|
inbox = inbox_for_sheet_sections(tmp_path)
|
||||||
plans = tmp_path / "plans-sheet-plain"
|
plans = tmp_path / "plans-sheet-plain"
|
||||||
|
|
@ -653,6 +656,8 @@ def test_the_build_default_leaves_a_sheet_as_one_concept(tmp_path: Path) -> None
|
||||||
str(plans),
|
str(plans),
|
||||||
"--proposed-at",
|
"--proposed-at",
|
||||||
PROPOSED_AT,
|
PROPOSED_AT,
|
||||||
|
"--no-sheet-section-rows",
|
||||||
|
"--no-keep-table-heading",
|
||||||
)
|
)
|
||||||
== 0
|
== 0
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -6,14 +6,17 @@ measured on a bundle nobody produces is a number that cannot regress, so the
|
||||||
guarantee it looks like was never held by anything.
|
guarantee it looks like was never held by anything.
|
||||||
|
|
||||||
This file pins the CURRENT default: `--outline-run 3 --table-grid --unit-fold
|
This file pins the CURRENT default: `--outline-run 3 --table-grid --unit-fold
|
||||||
--drop-wrapped-outline --outline-gate`, the configuration round 6 moved the
|
--drop-wrapped-outline --outline-gate --first-span-from-zero
|
||||||
default to on 2026-09-09 after measuring hit@8 on exactly this bundle.
|
--sheet-section-rows --keep-table-heading`, plus the reading side's
|
||||||
|
`tie_shared_rank`. Round 6 moved the first five on 2026-09-09 and round 7 moved
|
||||||
|
the last four on 2026-09-10, each after measuring hit@8 on exactly the bundle
|
||||||
|
its own default produces.
|
||||||
|
|
||||||
The gold set is LOCAL-ONLY and stays that way: no question and no
|
The gold set is LOCAL-ONLY and stays that way: no question and no
|
||||||
`gold_document` is reproduced here, and a row is named by its INDEX, the way
|
`gold_document` is reproduced here, and a row is named by its INDEX, the way
|
||||||
`docs/2026-09-07-okf-konsumskill-maaling.md` already names them.
|
`docs/2026-09-07-okf-konsumskill-maaling.md` already names them.
|
||||||
|
|
||||||
The bundle itself is a build artefact, not a fixture: it is 810 files of a
|
The bundle itself is a build artefact, not a fixture: it is 832 files of a
|
||||||
consumer's corpus and this repository is public. Absent, these tests SKIP with
|
consumer's corpus and this repository is public. Absent, these tests SKIP with
|
||||||
the command that rebuilds it -- "not measured", never zero.
|
the command that rebuilds it -- "not measured", never zero.
|
||||||
"""
|
"""
|
||||||
|
|
@ -34,10 +37,10 @@ import okf_consume_measure # noqa: E402
|
||||||
|
|
||||||
#: Built by:
|
#: Built by:
|
||||||
#: okf build <corpus>/K2/trinn1 \
|
#: okf build <corpus>/K2/trinn1 \
|
||||||
#: --bundle ~/corpora/okf-telling-20260829/K2-bundle-default-20260909 \
|
#: --bundle ~/corpora/okf-telling-20260829/K2-bundle-default-20260910 \
|
||||||
#: --bundle-id k2-trinn1-20260903 --okf-version 0.2
|
#: --bundle-id k2-trinn1-20260903 --okf-version 0.2
|
||||||
#: with no arm flag at all -- the package default, which is the point.
|
#: with no arm flag at all -- the package default, which is the point.
|
||||||
DEFAULT_BUNDLE = Path.home() / "corpora" / "okf-telling-20260829" / "K2-bundle-default-20260909"
|
DEFAULT_BUNDLE = Path.home() / "corpora" / "okf-telling-20260829" / "K2-bundle-default-20260910"
|
||||||
GOLD_SET = PROJECT_ROOT / ".claude/projects/2026-09-07-okf-consume-prepass/hit-at-k-questions.json"
|
GOLD_SET = PROJECT_ROOT / ".claude/projects/2026-09-07-okf-consume-prepass/hit-at-k-questions.json"
|
||||||
|
|
||||||
requires_default_bundle = pytest.mark.skipif(
|
requires_default_bundle = pytest.mark.skipif(
|
||||||
|
|
@ -49,8 +52,11 @@ requires_default_bundle = pytest.mark.skipif(
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
#: Measured 2026-09-09 on the bundle above.
|
#: Measured 2026-09-10 on the bundle above. The count moved 425 -> 436 with
|
||||||
EXPECTED_CONCEPTS = 425
|
#: `--sheet-section-rows --keep-table-heading`; `--first-span-from-zero` moved
|
||||||
|
#: it by NOTHING, which is the point of that rule -- it adds no boundary and
|
||||||
|
#: only moves one span's start.
|
||||||
|
EXPECTED_CONCEPTS = 436
|
||||||
EXPECTED_HITS = 5
|
EXPECTED_HITS = 5
|
||||||
#: Rank per question INDEX, `None` for the row that misses on every bundle and
|
#: Rank per question INDEX, `None` for the row that misses on every bundle and
|
||||||
#: every configuration measured so far. The identity is the index; the question
|
#: every configuration measured so far. The identity is the index; the question
|
||||||
|
|
@ -70,6 +76,10 @@ def test_hit_at_eight_holds_rank_one_on_every_row_it_held() -> None:
|
||||||
Not the hit COUNT alone: the count survived a configuration that lost a
|
Not the hit COUNT alone: the count survived a configuration that lost a
|
||||||
row from rank 1 to rank 2, which is exactly how the previous round's
|
row from rank 1 to rank 2, which is exactly how the previous round's
|
||||||
regression hid. The rank per row is the pin.
|
regression hid. The rank per row is the pin.
|
||||||
|
|
||||||
|
On THIS bundle that is not a hypothetical -- see the test below, which
|
||||||
|
reproduces the fall on these exact bytes by turning the reading-side
|
||||||
|
default off.
|
||||||
"""
|
"""
|
||||||
questions = json.loads(GOLD_SET.read_text(encoding="utf-8"))["questions"]
|
questions = json.loads(GOLD_SET.read_text(encoding="utf-8"))["questions"]
|
||||||
assert len(questions) == len(EXPECTED_RANKS), "the gold set changed shape"
|
assert len(questions) == len(EXPECTED_RANKS), "the gold set changed shape"
|
||||||
|
|
@ -87,3 +97,31 @@ def test_hit_at_eight_holds_rank_one_on_every_row_it_held() -> None:
|
||||||
def test_the_bundle_declares_the_identity_the_reader_needs() -> None:
|
def test_the_bundle_declares_the_identity_the_reader_needs() -> None:
|
||||||
"""Whatever else moves, the bundle stays one the reading direction opens."""
|
"""Whatever else moves, the bundle stays one the reading direction opens."""
|
||||||
assert okf_consume.root_bundle_id_of(DEFAULT_BUNDLE) == "k2-trinn1-20260903"
|
assert okf_consume.root_bundle_id_of(DEFAULT_BUNDLE) == "k2-trinn1-20260903"
|
||||||
|
|
||||||
|
|
||||||
|
@requires_default_bundle
|
||||||
|
def test_the_reading_default_is_what_holds_row_one_on_these_bytes() -> None:
|
||||||
|
"""The known-negative, on the shipped bundle rather than a fixture.
|
||||||
|
|
||||||
|
Round 7 moved `--sheet-section-rows --keep-table-heading` into the build
|
||||||
|
default, which splits row 1's gold document from 1 concept into 12. Round 6
|
||||||
|
measured that exact split costing row 1 its rank, and held the two rules
|
||||||
|
back for it. What removed the cost is `consume.DEFAULT_TIE_SHARED_RANK`,
|
||||||
|
and this test is the proof that it is still what removes it: turn it off
|
||||||
|
on these bytes and the fall comes back.
|
||||||
|
|
||||||
|
Without this, `EXPECTED_RANKS` above would be a green assertion with no
|
||||||
|
stated cause, and a later change to the fusion could take the cause away
|
||||||
|
while the pin stayed green on some other accident.
|
||||||
|
"""
|
||||||
|
questions = json.loads(GOLD_SET.read_text(encoding="utf-8"))["questions"]
|
||||||
|
ranks = []
|
||||||
|
for entry in questions:
|
||||||
|
payload = okf_consume.build_payload(
|
||||||
|
DEFAULT_BUNDLE, question=entry["question"], tie_shared_rank=False
|
||||||
|
)
|
||||||
|
excerpts = payload["excerpts"]
|
||||||
|
assert isinstance(excerpts, list)
|
||||||
|
ranks.append(okf_consume_measure.hit_rank(excerpts, entry["gold_document"]))
|
||||||
|
assert ranks[0] == 2, "the known-negative stopped being negative"
|
||||||
|
assert tuple(ranks[1:]) == EXPECTED_RANKS[1:]
|
||||||
|
|
|
||||||
|
|
@ -68,5 +68,10 @@ def test_the_rule_adds_and_removes_no_boundary() -> None:
|
||||||
assert [c.start for c in plain][1:] == [c.start for c in moved][1:]
|
assert [c.start for c in plain][1:] == [c.start for c in moved][1:]
|
||||||
|
|
||||||
|
|
||||||
def test_it_is_off_by_default_in_the_build_command() -> None:
|
def test_it_is_on_by_default_in_the_build_command() -> None:
|
||||||
assert cli.DEFAULT_FIRST_SPAN_FROM_ZERO is False
|
"""Moved 2026-09-10, once the hit@8 number round 6 lacked was measured.
|
||||||
|
|
||||||
|
The opt-out is asserted in `tests/test_round7_defaults.py`; here the point
|
||||||
|
is only that the rule below is what a caller gets without saying anything.
|
||||||
|
"""
|
||||||
|
assert cli.DEFAULT_FIRST_SPAN_FROM_ZERO is True
|
||||||
|
|
|
||||||
|
|
@ -522,10 +522,19 @@ def test_reversing_the_input_order_does_not_change_the_output_order() -> None:
|
||||||
|
|
||||||
|
|
||||||
def test_a_concept_in_a_high_scoring_document_outranks_an_equally_lexical_one() -> None:
|
def test_a_concept_in_a_high_scoring_document_outranks_an_equally_lexical_one() -> None:
|
||||||
|
# `tie_shared_rank=False` for the same reason `lookup=False` appears
|
||||||
|
# elsewhere in this file: the claim is about the DOCUMENT PRIOR, and the
|
||||||
|
# default tie-break (shared since 2026-09-10) puts this fixture's two
|
||||||
|
# concepts in the same prior tie group, which makes the assertion true
|
||||||
|
# in both directions and so measures nothing. Isolate the stage under test.
|
||||||
concepts = _fixture_concepts()
|
concepts = _fixture_concepts()
|
||||||
question = "Hvordan skal prisene fylles ut?"
|
question = "Hvordan skal prisene fylles ut?"
|
||||||
lifted = okf_consume.concept_scores(concepts, question, {"krav": 10.0, "dyp": 0.0})
|
lifted = okf_consume.concept_scores(
|
||||||
dropped = okf_consume.concept_scores(concepts, question, {"krav": 0.0, "dyp": 10.0})
|
concepts, question, {"krav": 10.0, "dyp": 0.0}, tie_shared_rank=False
|
||||||
|
)
|
||||||
|
dropped = okf_consume.concept_scores(
|
||||||
|
concepts, question, {"krav": 0.0, "dyp": 10.0}, tie_shared_rank=False
|
||||||
|
)
|
||||||
krav_first = [c.concept_id for c, _, _ in lifted].index("krav/pristabell")
|
krav_first = [c.concept_id for c, _, _ in lifted].index("krav/pristabell")
|
||||||
krav_later = [c.concept_id for c, _, _ in dropped].index("krav/pristabell")
|
krav_later = [c.concept_id for c, _, _ in dropped].index("krav/pristabell")
|
||||||
assert krav_first < krav_later
|
assert krav_first < krav_later
|
||||||
|
|
@ -1771,7 +1780,9 @@ def test_counting_leaves_the_identifier_worth_no_more_than_the_common_word() ->
|
||||||
# The claim here is about the FUSION, so the fusion is what is read.
|
# The claim here is about the FUSION, so the fusion is what is read.
|
||||||
for identifier_in_body in (False, True):
|
for identifier_in_body in (False, True):
|
||||||
concepts = _rarity_corpus(identifier_in_body=identifier_in_body)
|
concepts = _rarity_corpus(identifier_in_body=identifier_in_body)
|
||||||
ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, lookup=False)
|
ranked = okf_consume.concept_scores(
|
||||||
|
concepts, RARITY_QUESTION, {}, lookup=False, tie_shared_rank=False
|
||||||
|
)
|
||||||
assert _rank_of(ranked, RARITY_GOLD) == 18
|
assert _rank_of(ranked, RARITY_GOLD) == 18
|
||||||
lexical = {concept.concept_id: value for concept, _, value in ranked}
|
lexical = {concept.concept_id: value for concept, _, value in ranked}
|
||||||
assert lexical["krav/c-00"] == 6
|
assert lexical["krav/c-00"] == 6
|
||||||
|
|
@ -1786,7 +1797,9 @@ def test_weighting_a_hit_by_its_rarity_lifts_the_identifier_into_the_cut() -> No
|
||||||
weights = okf_consume.rarity_weights(
|
weights = okf_consume.rarity_weights(
|
||||||
okf_consume.normalise(RARITY_QUESTION), okf_consume.searchable_text(concepts)
|
okf_consume.normalise(RARITY_QUESTION), okf_consume.searchable_text(concepts)
|
||||||
)
|
)
|
||||||
ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, weights=weights)
|
ranked = okf_consume.concept_scores(
|
||||||
|
concepts, RARITY_QUESTION, {}, weights=weights, tie_shared_rank=False
|
||||||
|
)
|
||||||
assert _rank_of(ranked, RARITY_GOLD) <= 8
|
assert _rank_of(ranked, RARITY_GOLD) <= 8
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -1817,9 +1830,28 @@ def test_a_rarity_weight_cannot_move_a_signal_the_gold_already_leads() -> None:
|
||||||
assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, None) == 1
|
assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, None) == 1
|
||||||
assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, weights) == 1
|
assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, weights) == 1
|
||||||
ranked = okf_consume.concept_scores(
|
ranked = okf_consume.concept_scores(
|
||||||
concepts, RARITY_QUESTION, {}, weights=weights, lookup=False
|
concepts, RARITY_QUESTION, {}, weights=weights, lookup=False, tie_shared_rank=False
|
||||||
)
|
)
|
||||||
assert _rank_of(ranked, RARITY_GOLD) == 18
|
assert _rank_of(ranked, RARITY_GOLD) == 18
|
||||||
|
# AND the falsification under the 2026-09-10 tie-break, measured rather
|
||||||
|
# than assumed -- with the COST it carries on this fixture written down.
|
||||||
|
# The claim survives and is stronger (30 -> 30 with and without the
|
||||||
|
# weight), but the gold's absolute rank is WORSE than the 18 above: the id
|
||||||
|
# order that had put it at 18 was luck inside a tie group, and shared ranks
|
||||||
|
# take that luck away in both directions. This fixture is the adverse
|
||||||
|
# shape for the rule -- one signal separates, two do not, and the one that
|
||||||
|
# separates is the one the gold loses. It is recorded here because it is
|
||||||
|
# the honest limit of the end-to-end measurement that moved the default:
|
||||||
|
# ranks held at 1 on every row of three real bundles, and a synthetic
|
||||||
|
# corpus of 30 near-identical concepts is not one of them.
|
||||||
|
under_default = okf_consume.concept_scores(
|
||||||
|
concepts, RARITY_QUESTION, {}, weights=weights, lookup=False
|
||||||
|
)
|
||||||
|
unweighted_under_default = okf_consume.concept_scores(
|
||||||
|
concepts, RARITY_QUESTION, {}, lookup=False
|
||||||
|
)
|
||||||
|
assert _rank_of(under_default, RARITY_GOLD) == 30
|
||||||
|
assert _rank_of(unweighted_under_default, RARITY_GOLD) == 30
|
||||||
|
|
||||||
|
|
||||||
def test_the_weight_leaves_the_lexical_count_a_count_so_the_cut_is_untouched() -> None:
|
def test_the_weight_leaves_the_lexical_count_a_count_so_the_cut_is_untouched() -> None:
|
||||||
|
|
@ -1836,7 +1868,9 @@ def test_the_weight_leaves_the_lexical_count_a_count_so_the_cut_is_untouched() -
|
||||||
}
|
}
|
||||||
weighted = {
|
weighted = {
|
||||||
c.concept_id: v
|
c.concept_id: v
|
||||||
for c, _, v in okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, weights=weights)
|
for c, _, v in okf_consume.concept_scores(
|
||||||
|
concepts, RARITY_QUESTION, {}, weights=weights, tie_shared_rank=False
|
||||||
|
)
|
||||||
}
|
}
|
||||||
assert plain == weighted
|
assert plain == weighted
|
||||||
|
|
||||||
|
|
@ -1989,14 +2023,29 @@ def test_the_weight_reaches_the_title_signal_and_reorders_it() -> None:
|
||||||
assert _signal_rank(concepts, question, "krav/t-29", weights) == 1
|
assert _signal_rank(concepts, question, "krav/t-29", weights) == 1
|
||||||
# `lookup=False`: the gold's title bears the identifier, so the lookup
|
# `lookup=False`: the gold's title bears the identifier, so the lookup
|
||||||
# partition answers this question at rank one. The fusion is the claim.
|
# partition answers this question at rank one. The fusion is the claim.
|
||||||
plain = okf_consume.concept_scores(concepts, question, {}, lookup=False)
|
plain = okf_consume.concept_scores(concepts, question, {}, lookup=False, tie_shared_rank=False)
|
||||||
weighted = okf_consume.concept_scores(concepts, question, {}, weights=weights, lookup=False)
|
weighted = okf_consume.concept_scores(
|
||||||
|
concepts, question, {}, weights=weights, lookup=False, tie_shared_rank=False
|
||||||
|
)
|
||||||
# Fused: 30 -> 18. The title signal is reordered from last to first and the
|
# Fused: 30 -> 18. The title signal is reordered from last to first and the
|
||||||
# fused rank moves by twelve, not to one -- the other two signals still
|
# fused rank moves by twelve, not to one -- the other two signals still
|
||||||
# cannot see the identifier. That gap IS the finding of this session, and
|
# cannot see the identifier. That gap IS the finding of this session, and
|
||||||
# the numbers are here so a change to either half shows up as a diff.
|
# the numbers are here so a change to either half shows up as a diff.
|
||||||
assert _rank_of(plain, "krav/t-29") == 30
|
assert _rank_of(plain, "krav/t-29") == 30
|
||||||
assert _rank_of(weighted, "krav/t-29") == 18
|
assert _rank_of(weighted, "krav/t-29") == 18
|
||||||
|
# AND WHAT THE 2026-09-10 TIE-BREAK DID TO THAT GAP, measured rather than
|
||||||
|
# left implicit. This fixture is the shape shared ranks were built for:
|
||||||
|
# every body is identical and the prior is empty, so both of the signals
|
||||||
|
# that "cannot see the identifier" separate NOTHING, and under shared ranks
|
||||||
|
# they stop ordering by id and contribute one constant each. The title
|
||||||
|
# signal is then alone in deciding and the gold goes to rank 1 -- the gap
|
||||||
|
# this test named does not survive the fusion change, and a reader of
|
||||||
|
# `docs/2026-09-08-sjeldenhetsvekt.md` needs to know the figure there was
|
||||||
|
# measured under the older tie-break.
|
||||||
|
under_default = okf_consume.concept_scores(
|
||||||
|
concepts, question, {}, weights=weights, lookup=False
|
||||||
|
)
|
||||||
|
assert _rank_of(under_default, "krav/t-29") == 1
|
||||||
|
|
||||||
|
|
||||||
# --- Step 14: exact identifier matching and the lookup signal (O2c) -----------
|
# --- Step 14: exact identifier matching and the lookup signal (O2c) -----------
|
||||||
|
|
|
||||||
84
tests/test_round7_defaults.py
Normal file
84
tests/test_round7_defaults.py
Normal file
|
|
@ -0,0 +1,84 @@
|
||||||
|
"""What round 7 moved to the default, and the opt-out that reproduces the old bytes.
|
||||||
|
|
||||||
|
Three moves, each with its own acceptance measurement in
|
||||||
|
`docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md`:
|
||||||
|
|
||||||
|
* `--first-span-from-zero`, because 32 of the 32 corpus documents with a plan
|
||||||
|
left their opening text in NO segment -- 9.18 % of the corpus.
|
||||||
|
* `--sheet-section-rows --keep-table-heading`, which were held back in round 6
|
||||||
|
by a retrieval regression the third move removes.
|
||||||
|
* `tie_shared_rank` on the reading side, which is what removed it.
|
||||||
|
|
||||||
|
Every move keeps an explicit opt-out: a default a caller cannot turn off is
|
||||||
|
not a default, and a consumer needing the previous bytes needs a way to say so.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from llm_ingestion_okf import cli, consume
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_build_defaults_carry_the_three_rules_round_seven_moved() -> None:
|
||||||
|
assert cli.DEFAULT_FIRST_SPAN_FROM_ZERO is True
|
||||||
|
assert cli.DEFAULT_SHEET_SECTION_ROWS is True
|
||||||
|
assert cli.DEFAULT_KEEP_TABLE_HEADING is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_reading_side_shares_a_rank_a_signal_did_not_earn_by_default() -> None:
|
||||||
|
"""`tie_shared_rank` is what let the two build rules above ship.
|
||||||
|
|
||||||
|
Measured on the D1+D3 bundle: the gold document splits from 1 concept into
|
||||||
|
12, its own 12 concepts occupy the document-prior signal's whole top tie
|
||||||
|
group, and the concept leading the body signal lands at prior position 11
|
||||||
|
instead of 1. Row 1 falls from fused rank 1 to 2. Shared ranks put it back.
|
||||||
|
"""
|
||||||
|
assert consume.DEFAULT_TIE_SHARED_RANK is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_every_moved_default_has_an_opt_out_that_the_parser_accepts() -> None:
|
||||||
|
args = cli.parse_args(
|
||||||
|
[
|
||||||
|
"build",
|
||||||
|
"src",
|
||||||
|
"--bundle",
|
||||||
|
"out",
|
||||||
|
"--bundle-id",
|
||||||
|
"x",
|
||||||
|
"--no-first-span-from-zero",
|
||||||
|
"--no-sheet-section-rows",
|
||||||
|
"--no-keep-table-heading",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
assert args.first_span_from_zero is False
|
||||||
|
assert args.sheet_section_rows is False
|
||||||
|
assert args.keep_table_heading is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_build_defaults_reach_the_parser_without_a_flag() -> None:
|
||||||
|
args = cli.parse_args(["build", "src", "--bundle", "out", "--bundle-id", "x"])
|
||||||
|
assert args.first_span_from_zero is True
|
||||||
|
assert args.sheet_section_rows is True
|
||||||
|
assert args.keep_table_heading is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_consume_parser_offers_an_opt_out_for_the_shared_rank() -> None:
|
||||||
|
assert consume.parse_args(["b", "--question", "q"]).tie_shared_rank is True
|
||||||
|
assert (
|
||||||
|
consume.parse_args(["b", "--question", "q", "--no-tie-shared-rank"]).tie_shared_rank
|
||||||
|
is False
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_payload_defaults_to_the_shared_rank(tmp_path: Path) -> None:
|
||||||
|
"""The default reaches the API, not only the CLI.
|
||||||
|
|
||||||
|
`okf skill` emits `okf consume`, but `tests/test_default_bundle_pin.py` and
|
||||||
|
every measurement harness call `build_payload` directly. A default living
|
||||||
|
in `argparse` alone would make the two disagree.
|
||||||
|
"""
|
||||||
|
import inspect
|
||||||
|
|
||||||
|
signature = inspect.signature(consume.build_payload)
|
||||||
|
assert signature.parameters["tie_shared_rank"].default is True
|
||||||
|
|
@ -137,31 +137,54 @@ def test_shared_rank_lifts_the_concept_the_measuring_signals_lead(tmp_path: Path
|
||||||
assert _gold_rank(root, tie_shared_rank=True) == 1
|
assert _gold_rank(root, tie_shared_rank=True) == 1
|
||||||
|
|
||||||
|
|
||||||
def test_the_payload_is_byte_identical_with_the_flag_off(tmp_path: Path) -> None:
|
def test_the_payload_is_byte_identical_with_the_flag_on(tmp_path: Path) -> None:
|
||||||
|
"""The rule became the default 2026-09-10; saying so explicitly changes nothing.
|
||||||
|
|
||||||
|
The assertion is unchanged in kind -- the implicit and the explicit value
|
||||||
|
must produce the same bytes -- only the value it names moved.
|
||||||
|
"""
|
||||||
root = _tie_bundle(tmp_path / "bundle")
|
root = _tie_bundle(tmp_path / "bundle")
|
||||||
without = okf_consume.serialise(okf_consume.build_payload(root, question=QUESTION))
|
without = okf_consume.serialise(okf_consume.build_payload(root, question=QUESTION))
|
||||||
explicit_off = okf_consume.serialise(
|
explicit_on = okf_consume.serialise(
|
||||||
okf_consume.build_payload(root, question=QUESTION, tie_shared_rank=False)
|
okf_consume.build_payload(root, question=QUESTION, tie_shared_rank=True)
|
||||||
)
|
)
|
||||||
assert without == explicit_off
|
assert without == explicit_on
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_opt_out_reproduces_the_order_the_default_used_to_give(tmp_path: Path) -> None:
|
||||||
|
"""The other half: a consumer needing the pre-2026-09-10 order can have it.
|
||||||
|
|
||||||
|
Load-bearing rather than symmetric. `--no-tie-shared-rank` is the only
|
||||||
|
thing standing between a consumer pinned to the old excerpt order and a
|
||||||
|
silent reordering, so the opt-out needs a test that goes red if it stops
|
||||||
|
being a real alternative -- which it would be if it produced the same
|
||||||
|
bytes as the default on the very fixture built to separate them.
|
||||||
|
"""
|
||||||
|
root = _tie_bundle(tmp_path / "bundle")
|
||||||
|
default = okf_consume.serialise(okf_consume.build_payload(root, question=QUESTION, k=3))
|
||||||
|
opted_out = okf_consume.serialise(
|
||||||
|
okf_consume.build_payload(root, question=QUESTION, k=3, tie_shared_rank=False)
|
||||||
|
)
|
||||||
|
assert default != opted_out
|
||||||
|
|
||||||
|
|
||||||
def test_the_flag_changes_the_payload_it_is_meant_to_change(tmp_path: Path) -> None:
|
def test_the_flag_changes_the_payload_it_is_meant_to_change(tmp_path: Path) -> None:
|
||||||
root = _tie_bundle(tmp_path / "bundle")
|
root = _tie_bundle(tmp_path / "bundle")
|
||||||
off = okf_consume.build_payload(root, question=QUESTION, k=3)
|
off = okf_consume.build_payload(root, question=QUESTION, k=3, tie_shared_rank=False)
|
||||||
on = okf_consume.build_payload(root, question=QUESTION, k=3, tie_shared_rank=True)
|
on = okf_consume.build_payload(root, question=QUESTION, k=3)
|
||||||
delivered_off = [excerpt["concept_id"] for excerpt in off["excerpts"]] # type: ignore[index]
|
delivered_off = [excerpt["concept_id"] for excerpt in off["excerpts"]] # type: ignore[index]
|
||||||
delivered_on = [excerpt["concept_id"] for excerpt in on["excerpts"]] # type: ignore[index]
|
delivered_on = [excerpt["concept_id"] for excerpt in on["excerpts"]] # type: ignore[index]
|
||||||
assert not any(str(cid).endswith("zz-gull") for cid in delivered_off)
|
assert not any(str(cid).endswith("zz-gull") for cid in delivered_off)
|
||||||
assert str(delivered_on[0]).endswith("zz-gull")
|
assert str(delivered_on[0]).endswith("zz-gull")
|
||||||
|
|
||||||
|
|
||||||
def test_the_cli_exposes_the_flag_and_defaults_it_off(tmp_path: Path) -> None:
|
def test_the_cli_exposes_the_flag_and_defaults_it_on(tmp_path: Path) -> None:
|
||||||
root = _tie_bundle(tmp_path / "bundle")
|
root = _tie_bundle(tmp_path / "bundle")
|
||||||
parsed = okf_consume.parse_args([str(root), "--question", QUESTION])
|
parsed = okf_consume.parse_args([str(root), "--question", QUESTION])
|
||||||
assert parsed.tie_shared_rank is False
|
assert parsed.tie_shared_rank is True
|
||||||
parsed_on = okf_consume.parse_args([str(root), "--question", QUESTION, "--tie-shared-rank"])
|
parsed_off = okf_consume.parse_args([str(root), "--question", QUESTION, "--no-tie-shared-rank"])
|
||||||
assert parsed_on.tie_shared_rank is True
|
assert parsed_off.tie_shared_rank is False
|
||||||
# The other flag this session added, asserted here so "both default off"
|
# `--withheld-titles` did NOT move with it, asserted here so the two are
|
||||||
# is one measurement rather than two files' worth of trust.
|
# one measurement rather than two files' worth of trust: it is off for a
|
||||||
|
# reason of BYTES, which nothing this round touched.
|
||||||
assert parsed.withheld_titles is False
|
assert parsed.withheld_titles is False
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue