open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(consume): the door's link line is out of the default reading

Browse source 
`link_in_signal` defaults to `False` on all three entry points --
`searchable_text`, `concept_scores` and `build_payload` -- through one named
constant, `DEFAULT_LINK_IN_SIGNAL`. This carries out K3-23's recommendation
(alternative c), and its numbers are the whole warrant: of the newcomers the
door's `Enclosing section:` line ever added a question token to, 39 of 39
gained it from the bundle-absolute PATH and 0 of 39 from the link's title, and
every token the path contributed is a segment of the document's own directory
-- the saturation `shared_id_prefix` takes out of the id signal, back in
through the body.

A CONSTANT rather than three naked literals, chosen against the module: the
other eight defaults here are constants (`DEFAULT_PROFILE`, `DEFAULT_LIMIT`,
`DEFAULT_TIE_SHARED_RANK`, `DEFAULT_STEM_PREFIX`, `DEFAULT_TITLE_COVERED`,
`DEFAULT_SOURCE_QUOTA`, `DEFAULT_FOLLOW_PARENT`, `DEFAULT_K`) and this was the
one default spelled out three times. It also lets a test fall on the rule from
the module that owns it.

NO CLI flag is added and none is changed. `grep -n '"--.*link'` over `src/`
gives 0 hits before and after, and `consume.main` never passed the parameter,
so the CLI inherits the function default. A flag was considered and NOT added,
because no caller could be named that needs the older reading: the three call
sites outside `consume.py` (`skill.py:360`, `tools/okf_consume_measure.py:155`
and `:247`) all inherit it deliberately.

Measured before landing: 0 of 5 shipped bundles carry the door's line
(denominators 446, 1 133, 270, 2 756, 2 761 concept files) and 5 of 5 payloads
are byte-identical across the move. The consumer repository was read only and
its tree is unchanged, listing identical including mtimes.

The docstrings move with the code rather than after it. `concept_scores` said
this was "an INSTRUMENT, not a flag and not a default" and `build_payload` that
it "carries no CLI flag"; the first is now false and the second still true, and
each says so. `cli.DEFAULT_SHELL_PARENT` keeps its value and loses its reason:
the cost it cited -- delivered sets moving through that path -- is what this
commit removes, so the comment now states why it stays off NOW, which is N = 1
document and a consumption half nobody has judged. README and CLAUDE.md carry
the same sentence where they already describe the line, measured rather than
written on reflex: neither said anything about the ranking before.

One test changed with the code. The characterisation of what the LINE does
called `concept_scores` with no parameter, because the reading it characterises
used to be the default; it now names that reading. Same fixture, same number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-12 18:56:53 +02:00
commit 38320cd509
6 changed files with 130 additions and 47 deletions
Download patch file Download diff file Expand all files Collapse all files

30
src/llm_ingestion_okf/cli.py
View file

@ -176,17 +176,25 @@ DEFAULT_CLOSE_SPAN_GAPS = True
#: naming the nearest ancestor that holds text (`propose._link_shells`) and,
#: since K3-21, one `Enclosing section:` link line in its body. NOT a
#: segmentation rule: it adds no boundary and copies no text, and the concept
#: count is unchanged. OFF, re-measured in K3-21 once `okf consume` read the
#: key -- the reason it was off before. On the one standard with shells hit@1,
#: hit@8 and hit@50 stay 6/6 at both k with the known-positive at rank 1, but
#: the link's bundle-absolute path repeats the document's directory in 675
#: bodies, and the delivered set moved on 2 of 8 questions at the default k
#: and 3 of 8 at k 50, newly delivered shells matching the question ONLY
#: through that path on 1 of 3, 4 of 7 and 6 of 23. That is the saturation
#: K3-20 took out of the id signal, re-entering through the body. It moves the
#: bytes of a bundle holding a heading-only section and of nothing else: 0 of
#: 453 K2 bodies are heading-only, and the five-document folder builds
#: byte-identical with it.
#: count is unchanged. It moves the bytes of a bundle holding a heading-only
#: section and of nothing else: 0 of 453 K2 bodies are heading-only, and the
#: five-document folder builds byte-identical with it.
#:
#: OFF -- and the reason it was off has been REMOVED rather than restated.
#: K3-21 measured hit@1/8/50 at 6/6 at both k with the known-positive at rank
#: 1, but the delivered SET moved, because the link's bundle-absolute path
#: repeats the document's own directory in 675 bodies -- the saturation K3-20
#: took out of the id signal, re-entering through the body. K3-25 made
#: `consume.DEFAULT_LINK_IN_SIGNAL` `False`, so that path is no longer scored,
#: and K3-23 measured what the acceptance then reads on the one standard with
#: shells: newcomers matching through the path 0, delivered sets moved 0 of 8
#: at both k, hit@k unchanged at 6/6.
#:
#: It stays OFF for what that measurement does NOT cover: **N = 1 document**.
#: Three of nothing is still one bundle from one publisher, the consumption
#: half is a judged reading nobody has run, and a build default is a promise to
#: every consumer and not to this one. Turning it on is its own decision with
#: its own denominator, and this line is not it.
DEFAULT_SHELL_PARENT = False
#: Round 9. Clause 1 asked whether a title ENDED in a number, which is a
#: question about the number rather than about the title: a drawing's dimension

83
src/llm_ingestion_okf/consume.py
View file

@ -945,15 +945,18 @@ _LINK_LINE = re.compile(rf"^{re.escape(ENCLOSING_SECTION)}: \[[^\]]*\]\(/[^)]*\)
def body_without_link_line(body: str) -> str:
"""`body` minus the one line the door appends to a heading-only body.
THE FILE KEEPS THE LINE AND SO DOES THE READER. This is what a caller uses
to score a body the way it read before `--shell-parent` existed, on the same
bytes, in the same process -- never a second build compared against a first.
THE FILE KEEPS THE LINE AND SO DOES THE READER. Since K3-25 this is the
DEFAULT path through the ranker (`DEFAULT_LINK_IN_SIGNAL`), on the same
bytes and in the same process -- never a second build compared against a
first. `link_in_signal=True` is how a caller asks for the older reading.
Three conditions, all the door's own, and each one is a known-negative that
would otherwise be stripped: the line is LAST, a blank line stands before
it, and it matches the door's form (`_LINK_LINE`). A human sentence opening
with the same two words has no link and survives; the door's exact form with
prose under it is not last and survives. What cannot be told apart is a
prose under it is not last and survives. Those two are now on the default
path rather than behind a parameter, which is what makes them load-bearing
rather than illustrative. What cannot be told apart is a
human line that is byte-for-byte the door's own, last in a body -- an
ambiguity of the form itself, stated rather than hidden.
"""
@ -963,7 +966,28 @@ def body_without_link_line(body: str) -> str:
return "\n".join(lines[:-2]) + "\n"
def searchable_text(concepts: Sequence["Concept"], *, link_in_signal: bool = True) -> list[str]:
#: Round 25 (K3-25). Whether the DOOR's `Enclosing section:` line is scored.
#: `False` since K3-25, on K3-23's decomposition: of the newcomers that line
#: ever added a question token to, **39 of 39** gained it from the
#: bundle-absolute PATH and **0 of 39** from the link's title, and every token
#: the path contributed is a segment of the document's own directory -- the
#: saturation `shared_id_prefix` takes OUT of the id signal, arriving back
#: through the body. Under this reading a `--shell-parent` bundle delivers
#: exactly what the unflagged build delivers: the same list, the same order and
#: the same `spent` on **16 of 16** rows, with hit@1/8/50 **6/6** and the
#: known-positive at rank 1 under every reading. It moves no byte of any bundle
#: shipped today -- **0 of 5** carry the line, so **5 of 5** payloads are
#: byte-identical -- and the day one does, it is the day the path would have
#: started costing rank instead. A constant rather than three naked literals
#: because the other eight defaults in this module are constants, and because a
#: test can then fall on the rule from the module that owns it.
#: `docs/2026-09-12-k3-runde23-stien-i-kroppssignalet.md` SS 4.
DEFAULT_LINK_IN_SIGNAL = False
def searchable_text(
concepts: Sequence["Concept"], *, link_in_signal: bool = DEFAULT_LINK_IN_SIGNAL
) -> list[str]:
"""The text a concept is scored against, one string per concept.
The same two fields the ranker's two lexical signals read -- title plus
@ -971,10 +995,11 @@ def searchable_text(concepts: Sequence["Concept"], *, link_in_signal: bool = Tru
a hit can be scored on. Counting rarity over one field and matching on
another would weight a token by how rare it is somewhere it is not read.
`link_in_signal=False` reads the bodies without the door's link line, for
the same reason: the stem vocabulary and the rarity `df` are counted over
what a hit is scored on, so an instrument that took the line out of the
signal and left it in the vocabulary would measure two different texts.
`link_in_signal` reads the bodies with or without the door's link line, and
the same reason binds it here: the stem vocabulary and the rarity `df` are
counted over what a hit is scored on, so a reading that took the line out of
the signal and left it in the vocabulary would measure two different texts.
Since K3-25 the default is `False` (`DEFAULT_LINK_IN_SIGNAL`).
"""
return [
f"{concept.title} {concept.concept_id.replace('/', ' ')} "
@ -1403,7 +1428,7 @@ def concept_scores(
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
title_covered: bool = DEFAULT_TITLE_COVERED,
stems: frozenset[str] | None = None,
link_in_signal: bool = True,
link_in_signal: bool = DEFAULT_LINK_IN_SIGNAL,
) -> list[tuple[Concept, float, int]]:
"""Every concept, ordered best first, fused from three signals by RRF.
@ -1448,16 +1473,18 @@ def concept_scores(
only `tunnel` and `vann` whose ids sorted earlier. With shared ranks it
fuses to rank 3.
**`link_in_signal=False` is an INSTRUMENT, not a flag and not a default.**
The body this reads is the file's, and `inbox._link_enclosing` puts one line
in it -- `Enclosing section: [<title>](/<bundle-absolute path>)` -- whose
path repeats the document's own directory in every linked body. SS 6.1's
argument for the absolute form is about the FILE; nothing decided it should
be SCORED. Passing `False` scores the body without that line, on the same
bytes and in the same process, so the two readings can be measured against
each other rather than two builds. The excerpt a reader gets is untouched
either way; no caller in the run path passes it and the CLI does not expose
it.
**`link_in_signal=False` is the DEFAULT reading since K3-25**
(`DEFAULT_LINK_IN_SIGNAL`), and it was K3-23's instrument before that. The
body this reads is the file's, and `inbox._link_enclosing` puts one line in
it -- `Enclosing section: [<title>](/<bundle-absolute path>)` -- whose path
repeats the document's own directory in every linked body. SS 6.1's argument
for the absolute form is about the FILE; nothing ever decided it should be
SCORED, and K3-23 measured what happens when it is: **39 of 39** newcomers
the line added a question token to gained it through the PATH and **0 of
39** through the link's title. The excerpt a reader gets is untouched either
way, so what moved here is ORDER and never bytes. `True` is still reachable
and still does the older thing; the CLI exposes neither value, because no
caller has been named that needs the older one.
The rule takes the FIRST position of a score group rather than its middle.
Both were measured on the same four cases; the middle put the same concept
@ -2024,7 +2051,7 @@ def build_payload(
stem_prefix: bool = DEFAULT_STEM_PREFIX,
source_quota: int | None = DEFAULT_SOURCE_QUOTA,
follow_parent: bool = DEFAULT_FOLLOW_PARENT,
link_in_signal: bool = True,
link_in_signal: bool = DEFAULT_LINK_IN_SIGNAL,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
@ -2049,12 +2076,14 @@ def build_payload(
Whether the naming is worth the bookkeeping is the caller's call, and the
flag is how it stays one.
**`link_in_signal` is an instrument and carries no CLI flag** -- see
:func:`concept_scores`. `False` keeps the door's `Enclosing section:` line
out of the ranking and out of the stem vocabulary; the excerpt still carries
it, so a payload built either way holds the same bytes per excerpt and only
the ORDER can differ. That is what separates a ranking movement from a
budget displacement on one bundle.
**`link_in_signal` defaults to `False` since K3-25 and carries no CLI
flag** -- see :func:`concept_scores`. It keeps the door's `Enclosing
section:` line out of the ranking and out of the stem vocabulary; the
excerpt still carries it, so a payload built either way holds the same bytes
per excerpt and only the ORDER can differ. That is what separates a ranking
movement from a budget displacement, and it is why the default moved without
a bundle changing: **0 of 5** bundles anyone ships carries the line, so **5
of 5** payloads are byte-identical across the move.
"""
case, expected, measured = known_positive()
if expected != measured: