fix(consume): a directory every concept id shares no longer ranks them

The first fusion signal read a concept's title together with every segment
of its id. On a one-document bundle every id starts with the same directory,
and since K3-19 an STS document names that directory after its own number, so
a question naming the document matched every concept except the one whose
title already named it. Measured on a 2 761-concept bundle, the known-positive
fell from rank 1 to not delivered at the default k (13 at k = 50).

`shared_id_prefix` returns the leading directory segments EVERY id shares,
never the leaf, and the signal reads the id below them. Where the ids share no
prefix the signal reads the same string as before.

Measured on a frozen export before this commit, four forms: the chosen one
gives KP rank 1 at both k with S1-S6 6/6, and K2 (12 payloads), N100/N200/N500
(15) and the five-document folder (5) byte-identical. Dropping each concept's
own document directory instead took a K2 hit@8 row from rank 5 to not
delivered, and is not shipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-11 04:48:51 +02:00
commit 9f8a1bca42
3 changed files with 88 additions and 12 deletions

View file

@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
- **A directory every concept id shares no longer ranks the concepts
(K3-20).** `okf consume`'s first fusion signal read a concept's title
together with every segment of its id. On a one-document bundle every id
starts with the same directory, and since K3-19 an STS document names that
directory after its own number, so a question naming the document matched
every concept -- except the one whose title already named it, which gained
nothing because the overlap counts a question token once. Measured on a
2 761-concept bundle, the known-positive fell from rank 1 to not delivered
at the default `k` (13 at `k` = 50). `consume.shared_id_prefix` now keeps
the leading directories EVERY id shares out of that signal: the
known-positive is rank 1 at both `k` and S1-S6 stay 6/6.
- **Consumer cost: payloads move only on a bundle whose ids all share a
leading directory**, which is what a one-document build produces. Where
they share none, the signal reads the same string as before, and the
measured multi-document bundles are byte-identical. The old order is
reproducible by no flag.
- **Measured and felled:** dropping each concept's own document directory
instead took a hit@8 row on the pinned 43-document bundle from rank 5 to
not delivered.
### Changed
- **A NISO-STS document's own identity names its directory and titles its