open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): the door's link line is out of the default reading

Browse source 
`link_in_signal` defaults to `False` on all three entry points --
`searchable_text`, `concept_scores` and `build_payload` -- through one named
constant, `DEFAULT_LINK_IN_SIGNAL`. This carries out K3-23's recommendation
(alternative c), and its numbers are the whole warrant: of the newcomers the
door's `Enclosing section:` line ever added a question token to, 39 of 39
gained it from the bundle-absolute PATH and 0 of 39 from the link's title, and
every token the path contributed is a segment of the document's own directory
-- the saturation `shared_id_prefix` takes out of the id signal, back in
through the body.

A CONSTANT rather than three naked literals, chosen against the module: the
other eight defaults here are constants (`DEFAULT_PROFILE`, `DEFAULT_LIMIT`,
`DEFAULT_TIE_SHARED_RANK`, `DEFAULT_STEM_PREFIX`, `DEFAULT_TITLE_COVERED`,
`DEFAULT_SOURCE_QUOTA`, `DEFAULT_FOLLOW_PARENT`, `DEFAULT_K`) and this was the
one default spelled out three times. It also lets a test fall on the rule from
the module that owns it.

NO CLI flag is added and none is changed. `grep -n '"--.*link'` over `src/`
gives 0 hits before and after, and `consume.main` never passed the parameter,
so the CLI inherits the function default. A flag was considered and NOT added,
because no caller could be named that needs the older reading: the three call
sites outside `consume.py` (`skill.py:360`, `tools/okf_consume_measure.py:155`
and `:247`) all inherit it deliberately.

Measured before landing: 0 of 5 shipped bundles carry the door's line
(denominators 446, 1 133, 270, 2 756, 2 761 concept files) and 5 of 5 payloads
are byte-identical across the move. The consumer repository was read only and
its tree is unchanged, listing identical including mtimes.

The docstrings move with the code rather than after it. `concept_scores` said
this was "an INSTRUMENT, not a flag and not a default" and `build_payload` that
it "carries no CLI flag"; the first is now false and the second still true, and
each says so. `cli.DEFAULT_SHELL_PARENT` keeps its value and loses its reason:
the cost it cited -- delivered sets moving through that path -- is what this
commit removes, so the comment now states why it stays off NOW, which is N = 1
document and a consumption half nobody has judged. README and CLAUDE.md carry
the same sentence where they already describe the line, measured rather than
written on reflex: neither said anything about the ranking before.

One test changed with the code. The characterisation of what the LINE does
called `concept_scores` with no parameter, because the reading it characterises
used to be the default; it now names that reading. Same fixture, same number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-12 18:56:53 +02:00
commit 38320cd509
6 changed files with 130 additions and 47 deletions
Download patch file Download diff file Expand all files Collapse all files

27
CHANGELOG.md
View file

@ -28,9 +28,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
form -- so a human line opening with the same two words, and the door's exact
form anywhere but last, both keep every character they have.
`--shell-parent` therefore stays OFF at the current link form, and reading
the body without the line is recommended as `consume`'s default in a later
round: it is byte-identical on **5 of 5** bundles anyone ships today, none of
which carries the line.
the body without the line was recommended as `consume`'s default -- carried
out in the same unreleased block below: it is byte-identical on **5 of 5**
bundles anyone ships today, none of which carries the line.
### Changed
- **`consume` no longer scores the door's `Enclosing section:` line (K3-25).**
`link_in_signal` defaults to `False` (`consume.DEFAULT_LINK_IN_SIGNAL`) on
all three entry points -- `searchable_text`, `concept_scores` and
`build_payload` -- so the ranking and the stem vocabulary read a heading-only
body without the line the door appends under `--shell-parent`. The excerpt
still carries it and no CLI flag changed, so what moves is ORDER and never an
excerpt's bytes. Carrying out K3-23's recommendation with its numbers: of the
newcomers that line ever added a question token to, **39 of 39** gained it
from the bundle-absolute PATH and **0 of 39** from the link's title, every
such token being a segment of the document's own directory; under this
reading a flagged bundle delivers what the unflagged build delivers on **16
of 16** rows (list, order and `spent`), with hit@1/8/50 **6/6** at both `k`
and the known-positive at rank **1**. The cost to anyone shipping a bundle
today is zero bytes: **0 of 5** bundles carry the line, so **5 of 5** payloads
are byte-identical across the move. A bundle that does carry it -- only
`okf build --shell-parent` writes it, and that default is unchanged and still
OFF -- gets a different delivered order; `link_in_signal=True` is still
reachable for the older reading.
### Fixed

14
CLAUDE.md
View file

@ -638,6 +638,20 @@ and fixtures, never code.
absolute form), appended AFTER structure derivation -- read as body text it
became a second, unresolved `references` edge -- and screened on its own.
`okf check`'s seventeenth rule, `parent_unfollowable`, holds the form.
**Since K3-25 the RANKING does not score that line**
(`consume.DEFAULT_LINK_IN_SIGNAL = False` on `searchable_text`,
`concept_scores` and `build_payload`; no CLI flag, `True` still reachable).
The line reaches the file and the excerpt exactly as before, so what moved is
ORDER and never an excerpt's bytes. Measured: of the newcomers it ever added
a question token to, **39 of 39** gained it from the bundle-absolute PATH and
**0 of 39** from the link's TITLE, every such token being a segment of the
document's own directory -- the saturation `shared_id_prefix` took OUT of the
id signal, back in through the body. Under this reading a `--shell-parent`
bundle delivers what the unflagged build delivers on **16 of 16** rows (list,
order and `spent`), hit@1/8/50 **6/6** at both `k` with the known-positive at
rank 1. Exposure today is zero: **0 of 5** shipped bundles carry the line, so
**5 of 5** payloads are byte-identical across the move, and the day a bundle
carries it is the day the path would have started costing rank instead.
`okf consume --follow-parent` (K3-21 B) is the second form: `parent` also
carries the enclosing concept's `text` with that concept's own `sha256`,
placed AFTER the cut from the room it left, in rank order, so the delivered

11
README.md
View file

@ -189,7 +189,16 @@ heading-only body gains one line, `Enclosing section: [title](/path)`, in the
bundle-relative form SPEC § 6.1 recommends. `okf consume --follow-parent`
carries the enclosing section's text inside `parent` as well, with that
concept's own `sha256`, and only from the room the cut left — so it never
displaces an excerpt. Whether either default moves is measured separately. The index resolves the pointer too: a segment id is looked up among the
displaces an excerpt. Whether either default moves is measured separately.
**The ranking does not score that line (K3-25).** It reaches the excerpt and
the file exactly as before; `consume.DEFAULT_LINK_IN_SIGNAL` is `False`, so the
body signal and the stem vocabulary read a heading-only body without it. The
measurement: of the newcomers the line ever added a question token to, **39 of
39** gained it from the bundle-absolute PATH and **0 of 39** from the link's
title, every such token being a segment of the document's own directory — so a
bundle built with `--shell-parent` now delivers what the unflagged build
delivers, on **16 of 16** rows. It cost no shipped bundle a byte: **0 of 5**
carry the line, so **5 of 5** payloads are byte-identical across the move. The index resolves the pointer too: a segment id is looked up among the
concepts of its own document, so it renders `parent: p1` when the concept it
names is in the bundle and keeps the `?` only when nothing answers to it.

30
src/llm_ingestion_okf/cli.py
View file

@ -176,17 +176,25 @@ DEFAULT_CLOSE_SPAN_GAPS = True
#: naming the nearest ancestor that holds text (`propose._link_shells`) and,
#: since K3-21, one `Enclosing section:` link line in its body. NOT a
#: segmentation rule: it adds no boundary and copies no text, and the concept
#: count is unchanged. OFF, re-measured in K3-21 once `okf consume` read the
#: key -- the reason it was off before. On the one standard with shells hit@1,
#: hit@8 and hit@50 stay 6/6 at both k with the known-positive at rank 1, but
#: the link's bundle-absolute path repeats the document's directory in 675
#: bodies, and the delivered set moved on 2 of 8 questions at the default k
#: and 3 of 8 at k 50, newly delivered shells matching the question ONLY
#: through that path on 1 of 3, 4 of 7 and 6 of 23. That is the saturation
#: K3-20 took out of the id signal, re-entering through the body. It moves the
#: bytes of a bundle holding a heading-only section and of nothing else: 0 of
#: 453 K2 bodies are heading-only, and the five-document folder builds
#: byte-identical with it.
#: count is unchanged. It moves the bytes of a bundle holding a heading-only
#: section and of nothing else: 0 of 453 K2 bodies are heading-only, and the
#: five-document folder builds byte-identical with it.
#:
#: OFF -- and the reason it was off has been REMOVED rather than restated.
#: K3-21 measured hit@1/8/50 at 6/6 at both k with the known-positive at rank
#: 1, but the delivered SET moved, because the link's bundle-absolute path
#: repeats the document's own directory in 675 bodies -- the saturation K3-20
#: took out of the id signal, re-entering through the body. K3-25 made
#: `consume.DEFAULT_LINK_IN_SIGNAL` `False`, so that path is no longer scored,
#: and K3-23 measured what the acceptance then reads on the one standard with
#: shells: newcomers matching through the path 0, delivered sets moved 0 of 8
#: at both k, hit@k unchanged at 6/6.
#:
#: It stays OFF for what that measurement does NOT cover: **N = 1 document**.
#: Three of nothing is still one bundle from one publisher, the consumption
#: half is a judged reading nobody has run, and a build default is a promise to
#: every consumer and not to this one. Turning it on is its own decision with
#: its own denominator, and this line is not it.
DEFAULT_SHELL_PARENT = False
#: Round 9. Clause 1 asked whether a title ENDED in a number, which is a
#: question about the number rather than about the title: a drawing's dimension

83
src/llm_ingestion_okf/consume.py
View file

@ -945,15 +945,18 @@ _LINK_LINE = re.compile(rf"^{re.escape(ENCLOSING_SECTION)}: \[[^\]]*\]\(/[^)]*\)
def body_without_link_line(body: str) -> str:
"""`body` minus the one line the door appends to a heading-only body.
THE FILE KEEPS THE LINE AND SO DOES THE READER. This is what a caller uses
to score a body the way it read before `--shell-parent` existed, on the same
bytes, in the same process -- never a second build compared against a first.
THE FILE KEEPS THE LINE AND SO DOES THE READER. Since K3-25 this is the
DEFAULT path through the ranker (`DEFAULT_LINK_IN_SIGNAL`), on the same
bytes and in the same process -- never a second build compared against a
first. `link_in_signal=True` is how a caller asks for the older reading.
Three conditions, all the door's own, and each one is a known-negative that
would otherwise be stripped: the line is LAST, a blank line stands before
it, and it matches the door's form (`_LINK_LINE`). A human sentence opening
with the same two words has no link and survives; the door's exact form with
prose under it is not last and survives. What cannot be told apart is a
prose under it is not last and survives. Those two are now on the default
path rather than behind a parameter, which is what makes them load-bearing
rather than illustrative. What cannot be told apart is a
human line that is byte-for-byte the door's own, last in a body -- an
ambiguity of the form itself, stated rather than hidden.
"""
@ -963,7 +966,28 @@ def body_without_link_line(body: str) -> str:
return "\n".join(lines[:-2]) + "\n"
def searchable_text(concepts: Sequence["Concept"], *, link_in_signal: bool = True) -> list[str]:
#: Round 25 (K3-25). Whether the DOOR's `Enclosing section:` line is scored.
#: `False` since K3-25, on K3-23's decomposition: of the newcomers that line
#: ever added a question token to, **39 of 39** gained it from the
#: bundle-absolute PATH and **0 of 39** from the link's title, and every token
#: the path contributed is a segment of the document's own directory -- the
#: saturation `shared_id_prefix` takes OUT of the id signal, arriving back
#: through the body. Under this reading a `--shell-parent` bundle delivers
#: exactly what the unflagged build delivers: the same list, the same order and
#: the same `spent` on **16 of 16** rows, with hit@1/8/50 **6/6** and the
#: known-positive at rank 1 under every reading. It moves no byte of any bundle
#: shipped today -- **0 of 5** carry the line, so **5 of 5** payloads are
#: byte-identical -- and the day one does, it is the day the path would have
#: started costing rank instead. A constant rather than three naked literals
#: because the other eight defaults in this module are constants, and because a
#: test can then fall on the rule from the module that owns it.
#: `docs/2026-09-12-k3-runde23-stien-i-kroppssignalet.md` SS 4.
DEFAULT_LINK_IN_SIGNAL = False
def searchable_text(
concepts: Sequence["Concept"], *, link_in_signal: bool = DEFAULT_LINK_IN_SIGNAL
) -> list[str]:
"""The text a concept is scored against, one string per concept.
The same two fields the ranker's two lexical signals read -- title plus
@ -971,10 +995,11 @@ def searchable_text(concepts: Sequence["Concept"], *, link_in_signal: bool = Tru
a hit can be scored on. Counting rarity over one field and matching on
another would weight a token by how rare it is somewhere it is not read.
`link_in_signal=False` reads the bodies without the door's link line, for
the same reason: the stem vocabulary and the rarity `df` are counted over
what a hit is scored on, so an instrument that took the line out of the
signal and left it in the vocabulary would measure two different texts.
`link_in_signal` reads the bodies with or without the door's link line, and
the same reason binds it here: the stem vocabulary and the rarity `df` are
counted over what a hit is scored on, so a reading that took the line out of
the signal and left it in the vocabulary would measure two different texts.
Since K3-25 the default is `False` (`DEFAULT_LINK_IN_SIGNAL`).
"""
return [
f"{concept.title} {concept.concept_id.replace('/', ' ')} "
@ -1403,7 +1428,7 @@ def concept_scores(
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
title_covered: bool = DEFAULT_TITLE_COVERED,
stems: frozenset[str] | None = None,
link_in_signal: bool = True,
link_in_signal: bool = DEFAULT_LINK_IN_SIGNAL,
) -> list[tuple[Concept, float, int]]:
"""Every concept, ordered best first, fused from three signals by RRF.
@ -1448,16 +1473,18 @@ def concept_scores(
only `tunnel` and `vann` whose ids sorted earlier. With shared ranks it
fuses to rank 3.
**`link_in_signal=False` is an INSTRUMENT, not a flag and not a default.**
The body this reads is the file's, and `inbox._link_enclosing` puts one line
in it -- `Enclosing section: [<title>](/<bundle-absolute path>)` -- whose
path repeats the document's own directory in every linked body. SS 6.1's
argument for the absolute form is about the FILE; nothing decided it should
be SCORED. Passing `False` scores the body without that line, on the same
bytes and in the same process, so the two readings can be measured against
each other rather than two builds. The excerpt a reader gets is untouched
either way; no caller in the run path passes it and the CLI does not expose
it.
**`link_in_signal=False` is the DEFAULT reading since K3-25**
(`DEFAULT_LINK_IN_SIGNAL`), and it was K3-23's instrument before that. The
body this reads is the file's, and `inbox._link_enclosing` puts one line in
it -- `Enclosing section: [<title>](/<bundle-absolute path>)` -- whose path
repeats the document's own directory in every linked body. SS 6.1's argument
for the absolute form is about the FILE; nothing ever decided it should be
SCORED, and K3-23 measured what happens when it is: **39 of 39** newcomers
the line added a question token to gained it through the PATH and **0 of
39** through the link's title. The excerpt a reader gets is untouched either
way, so what moved here is ORDER and never bytes. `True` is still reachable
and still does the older thing; the CLI exposes neither value, because no
caller has been named that needs the older one.
The rule takes the FIRST position of a score group rather than its middle.
Both were measured on the same four cases; the middle put the same concept
@ -2024,7 +2051,7 @@ def build_payload(
stem_prefix: bool = DEFAULT_STEM_PREFIX,
source_quota: int | None = DEFAULT_SOURCE_QUOTA,
follow_parent: bool = DEFAULT_FOLLOW_PARENT,
link_in_signal: bool = True,
link_in_signal: bool = DEFAULT_LINK_IN_SIGNAL,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
@ -2049,12 +2076,14 @@ def build_payload(
Whether the naming is worth the bookkeeping is the caller's call, and the
flag is how it stays one.
**`link_in_signal` is an instrument and carries no CLI flag** -- see
:func:`concept_scores`. `False` keeps the door's `Enclosing section:` line
out of the ranking and out of the stem vocabulary; the excerpt still carries
it, so a payload built either way holds the same bytes per excerpt and only
the ORDER can differ. That is what separates a ranking movement from a
budget displacement on one bundle.
**`link_in_signal` defaults to `False` since K3-25 and carries no CLI
flag** -- see :func:`concept_scores`. It keeps the door's `Enclosing
section:` line out of the ranking and out of the stem vocabulary; the
excerpt still carries it, so a payload built either way holds the same bytes
per excerpt and only the ORDER can differ. That is what separates a ranking
movement from a budget displacement, and it is why the default moved without
a bundle changing: **0 of 5** bundles anyone ships carries the line, so **5
of 5** payloads are byte-identical across the move.
"""
case, expected, measured = known_positive()
if expected != measured:

12
tests/test_link_line_signal.py
View file

@ -107,12 +107,14 @@ def _lexical(root: Path, *, link_in_signal: bool) -> dict[str, int]:
# --- What the line does to the signal today -----------------------------------
def test_the_doors_line_is_the_shells_only_lexical_hit_today(tmp_path: Path) -> None:
"""CHARACTERISATION, green before the instrument exists.
def test_the_doors_line_is_the_shells_only_lexical_hit(tmp_path: Path) -> None:
"""CHARACTERISATION of what the LINE does, not of what the default is.
The shell's own text answers nothing in the question, and it still scores --
on the line the door wrote. It takes no new parameter to show, which is why
it is asked without one: the mechanism is HEAD's behaviour, not the fix's.
on the line the door wrote. Until K3-25 this was asked with no parameter,
because the reading it characterises was the default; it now names that
reading, because the mechanism is the line's and the default has moved away
from it. The same fixture, the same number, one word of spelling.
"""
root = _bundle(tmp_path / "bundle")
concepts = [
@ -122,7 +124,7 @@ def test_the_doors_line_is_the_shells_only_lexical_hit_today(tmp_path: Path) ->
for concept_id in okf_consume.enumerate_concepts(root)
]
ranked = okf_consume.concept_scores(
concepts, QUESTION, okf_consume.document_scores(root, QUESTION)
concepts, QUESTION, okf_consume.document_scores(root, QUESTION), link_in_signal=True
)
scored = {concept.concept_id: lexical for concept, _, lexical in ranked}
assert scored["r761/skall"] > 0