open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(consume): a directory every concept id shares no longer ranks them

Browse source 
The first fusion signal read a concept's title together with every segment
of its id. On a one-document bundle every id starts with the same directory,
and since K3-19 an STS document names that directory after its own number, so
a question naming the document matched every concept except the one whose
title already named it. Measured on a 2 761-concept bundle, the known-positive
fell from rank 1 to not delivered at the default k (13 at k = 50).

`shared_id_prefix` returns the leading directory segments EVERY id shares,
never the leaf, and the signal reads the id below them. Where the ids share no
prefix the signal reads the same string as before.

Measured on a frozen export before this commit, four forms: the chosen one
gives KP rank 1 at both k with S1-S6 6/6, and K2 (12 payloads), N100/N200/N500
(15) and the five-document folder (5) byte-identical. Dropping each concept's
own document directory instead took a K2 hit@8 row from rank 5 to not
delivered, and is not shipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-11 04:48:51 +02:00
commit 9f8a1bca42
3 changed files with 88 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

20
CLAUDE.md
View file

@ -137,13 +137,19 @@ one boundary rule:
sections on R761 carry a point, **1 807** are written (2 have no `<p>`, 217
carry `: ` and PyYAML refused exactly those frontmatters), none invented.
SS 4.1 sets no length, so the one-paragraph limit is ours. **The directory
name reaches the RANKING:** `consume`'s signal 1 reads the concept id's
segments, so on a one-document bundle every concept now carries the
document's own name, and a question naming the document matches all of
them -- measured, the known-positive went rank 1 -> not delivered at the
default `k` (13 at k=50) with S1-S6 unmoved, and renaming only the
directory back restores rank 1. Operator question, not a patch. Report:
`docs/2026-09-11-k3-runde19-dokumentidentitet-og-frontmatter.md`. The
name reached the RANKING, and K3-20 closed it in `consume`:** signal 1 read
every id segment, so on a one-document bundle every concept carried the
document's own name, and a question naming the document matched all of them
-- measured, the known-positive went rank 1 -> not delivered at the default
`k` (13 at k=50) with S1-S6 unmoved. `consume.shared_id_prefix` now keeps
the leading directories EVERY id shares out of that signal: KP rank 1 at
both `k`, S1-S6 6/6, and K2 (12 payloads), N100/N200/N500 (15) and the
five-document folder (5) byte-identical, because ids that share no prefix
read exactly as before. Dropping each concept's OWN document directory
instead was measured and felled -- a K2 hit@8 row went rank 5 -> not
delivered. Reports:
`docs/2026-09-11-k3-runde19-dokumentidentitet-og-frontmatter.md` and
`docs/2026-09-11-k3-runde20-delt-katalog-og-arvet-kontekst.md`. The
registries are COUPLED: a row in
`_CORE_EXTRACTORS` and not in `segmentation._STDLIB_EXTRACTOR_IDS` refuses
every proposal for the type, two layers away from the extractor.