open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(index): one ordering helper, called by both doors

Browse source 
An index ordering a profile names must be honoured wherever this library
writes an index. Door B and Door C have separate index writers, so an
ordering built on Door B's `_index_sort_key` seam alone would have been a
profile field Door B obeys and Door C ignores -- silently, because
nothing raises and both files still parse. That is
`IndexPolicy.per_directory` again: a field that reads as global and acts
on one path.

`IndexPolicy` gains `sort_key`, `sort_order` and `sort_missing`. Both
order fields draw from CLOSED sets, and `sort_order` is deliberately not
a caller-supplied callable: a callable cannot be serialised into the
bundle, reproduced from it, or audited by a reader, which is the whole of
what a deterministic bundle claims. A `sort_key` the facet policy does
not name is refused too -- every entry would be missing the key and the
ordering would silently do nothing, which is this row's own defect class.

`IndexPolicy.sort_entries` is the one helper. Four stable passes, so each
is the tie-break of the next: concept path, then the named key, then the
missing group partitioned to whichever end the policy says, then
navigation last. Passes 2 and 3 are separate on purpose -- folding them
into one reversible key tuple would flip the missing group along with the
order, so `sort_missing="last"` would mean "first" under `descending`.

The tie-break is the CONCEPT PATH, not the link target, and that is
measured rather than assumed: `notes-beta.md` precedes `notes/alpha.md`
by concept path and follows it by generated filename, so ordering Door C
on the target would have re-ordered every existing Door C bundle.
`IndexEntry` carries the path for that reason; `parse_entry` leaves it
`None` and the ordering falls back to the target, which costs nothing
because no caller sorts entries it read back off disk.

Door B's two reprojection writers and Door C's index emission all route
through the helper. Door B's unfaceted path is not routed and does not
need to be: `sort_key` requires a facet policy, and a faceted profile
never reaches that writer. Door C's guarantee is bounded and stated in
the code -- `link_in_index` appends what is absent and leaves what is
present, so the order holds within a run and never re-orders entries an
earlier run wrote.

Default ordering, unchanged and now stated: with no `sort_key`, concepts
before navigation, each group ascending by concept path.

TDD, and the red was watched twice. First behaviourally with the fields
inert (both doors emitted the exact reverse of the named order), then
again with Door B routed and Door C not -- the broken world reproduced,
where a Door-B-only test would have passed.

882 tests (868 before). The five byte-pinned goldens are untouched and
green; no shipped profile moved.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-01 19:54:45 +02:00
commit d2a8c43d77
5 changed files with 391 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

34
src/llm_ingestion_okf/importer.py
View file

@ -42,7 +42,7 @@ from .materialize import (
validate_ingested_at,
write_bytes,
)
from .profiles import DEFAULT, BundleProfile, FacetPolicy
from .profiles import DEFAULT, BundleProfile, FacetPolicy, IndexEntry
# An OKF concept is a `.md` document by definition — the guard's path gate
# rejects anything else outright — so nothing else in the source tree is a
@ -542,8 +542,17 @@ def import_bundle(
index_path = bundle / profile.index.name
if not index_path.is_file():
write_bytes(bundle, profile.index.name, "")
# Projected first, in merge order, so the report of what could not
# be rendered reads in the order the concepts were merged. ORDERED
# second, through the POLICY's helper — the same one Door B calls,
# because an ordering honoured at one door and ignored at the other
# is a profile field that lies. What this door can offer is bounded
# and stated: `link_in_index` appends what is absent and leaves what
# is present where it is, so the order holds within a run and never
# re-orders entries an earlier run wrote.
lines: list[IndexEntry] = []
for entry in merged:
facets: dict[str, str] | None = None
facets: dict[str, str] = {}
if profile.index.facets is not None:
# Read from the WRITTEN file, like the pointer scan above,
# so the entry describes the concept that exists rather than
@ -555,12 +564,27 @@ def import_bundle(
UnrenderedFacet(concept_path=entry.concept_path, key=key, reason=reason)
for key, reason in dropped
)
lines.append(
IndexEntry(
label=_index_label(entry.concept_path, profile=profile),
target=entry.path.name,
facets=facets,
# The SENDER's path, never the generated filename: the
# two do not order alike, and this door has always
# ordered by the sender's.
concept_path=entry.concept_path,
)
)
for line in profile.index.sort_entries(lines):
link_in_index(
bundle,
entry.path.name,
_index_label(entry.concept_path, profile=profile),
line.target,
line.label,
profile=profile,
facets=facets,
# `None` and `{}` are different instructions to the writer —
# skip a present entry versus refresh it — and which one
# applies is the policy's, constant for the whole run.
facets=dict(line.facets) if profile.index.facets is not None else None,
)
return ImportResult(

40
src/llm_ingestion_okf/inbox.py
View file

@ -849,13 +849,6 @@ def _refresh_root_frontmatter(index_path: Path, head: str) -> None:
index_path.write_bytes((head + body).encode("utf-8"))
#: The basename a navigation entry points at. A nav line is a link to a child
#: directory's index, and recognising it by its TARGET -- rather than by a flag
#: carried beside it -- is what lets an entry read back off disk sort exactly
#: like one about to be written.
INDEX_NAV_SUFFIX = DEFAULT.index.name
def _owned_listing(bundle: Path, profile: BundleProfile) -> dict[str, DocumentStructure]:
"""Every inbox-owned concept in the bundle, keyed by bundle-relative path.
@ -872,19 +865,6 @@ def _owned_listing(bundle: Path, profile: BundleProfile) -> dict[str, DocumentSt
return listing
def _index_sort_key(entry: IndexEntry) -> tuple[bool, str]:
"""The ONE ordering seam for every index this door writes.
Concepts first, navigation last, each group by target. Routed through a
single named helper on purpose: a consumer-controlled ordering is then a
parameter passed to this function, not a refactor of every place that
happened to call `sorted`. Navigation is recognised by its target -- a link
to a child's index -- rather than by a flag carried alongside, so an entry
read back off disk sorts the same way as one about to be written.
"""
return (entry.target.rpartition("/")[2] == INDEX_NAV_SUFFIX, entry.target)
def _reproject_indexes(
bundle: Path,
profile: BundleProfile,
@ -939,6 +919,7 @@ def _reproject_indexes(
label=document.title or target,
target=target,
facets=facet_values(relative, resolved, profile.index.facets.keys),
concept_path=relative,
)
)
for child in children.get(directory, set()):
@ -952,7 +933,7 @@ def _reproject_indexes(
block = [
profile.index.render_link(entry.label, entry.target, facets=entry.facets or None) + "\n"
for entry in sorted(entries, key=_index_sort_key)
for entry in profile.index.sort_entries(entries)
]
_write_index(
bundle,
@ -1050,15 +1031,22 @@ def _reproject_index(bundle: Path, profile: BundleProfile) -> None:
documents[key] = structure_from_frontmatter(parse_frontmatter(path))
resolved = resolve_structure(documents)
block = [
profile.index.render_link(
documents[name].title or name,
name,
entries = [
IndexEntry(
label=documents[name].title or name,
target=name,
facets=facet_values(name, resolved, profile.index.facets.keys),
# Flat: the concept ID and the link target are the same string. It
# is still named, because the tie-break is the concept path and
# agreeing with the target here is this writer's fact, not a rule.
concept_path=name,
)
+ "\n"
for name in sorted(documents)
]
block = [
profile.index.render_link(entry.label, entry.target, facets=entry.facets) + "\n"
for entry in profile.index.sort_entries(entries)
]
index_path = bundle / profile.index.name
kept: list[str] = []

109
src/llm_ingestion_okf/profiles.py
View file

@ -21,7 +21,7 @@ are extension points, not v1 (settled with the operator at phase start).
from __future__ import annotations
import re
from collections.abc import Collection, Mapping, Sequence
from collections.abc import Collection, Iterable, Mapping, Sequence
from dataclasses import dataclass, field, replace
# The one layer no profile may admit (ingest-spec §3): the promotion gate is
@ -454,14 +454,45 @@ class FacetPolicy:
return parsed
#: The closed set `IndexPolicy.sort_order` draws from. A CALLER-SUPPLIED
#: CALLABLE is deliberately not an option here: it cannot be serialised into
#: the bundle, cannot be reproduced from it, and cannot be audited by anyone
#: reading it back — which is the whole of what a deterministic bundle claims.
SORT_ASCENDING = "ascending"
SORT_DESCENDING = "descending"
SORT_ORDERS = (SORT_ASCENDING, SORT_DESCENDING)
#: The closed set `IndexPolicy.sort_missing` draws from: where the entries that
#: do not carry the key at all are put. Closed for the same reason.
SORT_MISSING_FIRST = "first"
SORT_MISSING_LAST = "last"
SORT_MISSING = (SORT_MISSING_FIRST, SORT_MISSING_LAST)
@dataclass(frozen=True)
class IndexEntry:
"""One managed index line, read back into its parts."""
"""One managed index line, read back into its parts.
`concept_path` is the bundle-relative path of the concept the entry is
ABOUT, and it is the ordering tie-break. It is not part of the rendered
line and is therefore only ever populated on the write path — `parse_entry`
leaves it `None`, and the ordering then falls back to the link target.
That costs nothing today because no caller sorts entries it read back off
disk; every ordering happens where the entry is being built.
It exists because the two doors disagree about what the target IS. Door B's
target is the concept's own name, but Door C reduces a sender's concept
path to a generated filename, and the two do not order alike:
`notes-beta.md` precedes `notes/alpha.md` by concept path and follows it by
generated name. Ordering on the target would silently re-order every
existing Door C bundle.
"""
label: str
target: str
description: str | None = None
facets: Mapping[str, str] = field(default_factory=dict)
concept_path: str | None = None
@dataclass(frozen=True)
@ -533,6 +564,9 @@ class IndexPolicy:
root_frontmatter: tuple[str, ...] = ()
root_frontmatter_required: frozenset[str] = field(default_factory=frozenset)
facets: FacetPolicy | None = None
sort_key: str | None = None
sort_order: str = SORT_ASCENDING
sort_missing: str = SORT_MISSING_LAST
_faceted_pattern: re.Pattern[str] | None = field(
init=False, repr=False, compare=False, default=None
)
@ -564,6 +598,25 @@ class IndexPolicy:
f"(got {', '.join(repr(key) for key in stray)}) — a key demanded "
"but never named could not be judged for position"
)
if self.sort_order not in SORT_ORDERS:
raise ValueError(
f"sort_order must be one of {SORT_ORDERS}, got {self.sort_order!r} — "
"the set is closed so that an index order is reproducible from the "
"bundle rather than from the caller that happened to write it"
)
if self.sort_missing not in SORT_MISSING:
raise ValueError(
f"sort_missing must be one of {SORT_MISSING}, got {self.sort_missing!r}"
)
if self.sort_key is not None and (
self.facets is None or self.sort_key not in self.facets.keys
):
named = self.facets.keys if self.facets is not None else ()
raise ValueError(
f"sort_key {self.sort_key!r} is not named by this index policy's facets, "
f"which pin {named or '()'} — every entry would be missing the key and "
"the ordering would silently do nothing"
)
if self.requires_description and "description" not in self.link_pattern.groupindex:
raise ValueError(
"link_template names {description} but link_pattern has no "
@ -632,6 +685,58 @@ class IndexPolicy:
facets=self.facets.parse(tail) if (self.facets is not None and tail) else {},
)
def sort_entries(self, entries: Iterable[IndexEntry]) -> tuple[IndexEntry, ...]:
"""The ONE ordering every index this library writes goes through.
Both doors call this. Not because sharing is tidy, but because the
alternative was measured: with Door B and Door C writing their indexes
through separate code, an ordering wired into one of them is a profile
field the other ignores in silence — nothing raises, and both files
still parse. Two implementations of one ordering ARE the drift.
Four passes, each a stable sort, so an earlier pass is exactly the
tie-break of a later one:
1. by concept path, which is the final tie-break and makes the order
total — two entries sharing a key never fall back to chance;
2. by the named key, reversed for `descending`;
3. the entries missing the key partitioned to whichever end
`sort_missing` says. Separate from pass 2 on purpose: folding the two
into one reversible key tuple would flip the missing group along with
the order, so `sort_missing="last"` would mean "first" under
`descending`;
4. navigation last. An outer GROUPING rather than a competitor to the
key: a link down to a child index is about a directory, carries no
facets, and would lead the file under `sort_missing="first"`.
A value that is present but empty counts as missing, because
`FacetPolicy.render` already drops it — an entry that renders without
the facet must not sort as though it carried one.
With no `sort_key` the whole of passes 2 and 3 is skipped and the result
is concepts before navigation, each group ascending by concept path.
That is what every profile shipped today already emitted.
"""
ordered = sorted(entries, key=lambda entry: entry.concept_path or entry.target)
if self.sort_key is not None:
key = self.sort_key
ordered = sorted(
ordered,
key=lambda entry: entry.facets.get(key) or "",
reverse=self.sort_order == SORT_DESCENDING,
)
missing = [entry for entry in ordered if not entry.facets.get(key)]
if missing:
present = [entry for entry in ordered if entry.facets.get(key)]
ordered = (
[*missing, *present]
if self.sort_missing == SORT_MISSING_FIRST
else [*present, *missing]
)
return tuple(
sorted(ordered, key=lambda entry: entry.target.rpartition("/")[2] == self.name)
)
def required_indexes(self, directories: Sequence[str]) -> tuple[str, ...]:
"""The index paths this policy requires, given the caller's directories.

237
tests/test_index_sort.py Normal file
View file

@ -0,0 +1,237 @@
"""One ordering, both doors: an index order a profile names is honoured everywhere.
The defect this closes is a drift, not a bug in either door on its own. Door B
and Door C write their indexes through separate code today, so an ordering
built on Door B's `_index_sort_key` seam alone would be a field on the profile
that Door B obeys and Door C ignores — silently, because nothing raises and
both indexes still parse. That is `IndexPolicy.per_directory` again: a field
that reads as global and acts on one path.
So the ordering lives on `IndexPolicy`, is expressed in three fields whose
values come from CLOSED sets, and is applied by ONE helper both doors call.
`sort_order` is deliberately not a caller-supplied callable: a callable is not
serialisable, not reproducible from the bundle, and not auditable, which is
exactly what a deterministic bundle promises it is not.
The cross-door test below is written so it would have FAILED in the broken
world: it runs both doors under the same profile and asserts they agree with
each other AND with the order the profile names. A test covering Door B alone
would have passed while Door C ignored the field.
"""
from __future__ import annotations
from dataclasses import replace
from pathlib import Path
import pytest
from test_import_facets import run_with
from test_import_flow import StubImportGate, place
from llm_ingestion_okf.inbox import GateDecision, process_inbox
from llm_ingestion_okf.profiles import (
DEFAULT,
STRUCTURED_V1,
BundleProfile,
IndexEntry,
IndexPolicy,
)
INGESTED_AT = "2026-07-25T12:00:00Z"
# A profile that names an ordering. Built here rather than added to a shipped
# profile: STRUCTURED_V1 is a consumer's contract, and moving its index bytes
# from a test would decide another repo's ordering for it.
NUMBERED = replace(
STRUCTURED_V1,
index=replace(STRUCTURED_V1.index, sort_key="number"),
)
def numbers_in(bundle: Path, profile: BundleProfile) -> list[str]:
"""The `number` facet of every managed entry, in the order the file has them."""
text = (bundle / profile.index.name).read_text(encoding="utf-8")
entries = (profile.index.parse_entry(line) for line in text.splitlines())
return [entry.facets["number"] for entry in entries if entry is not None]
def run_door_b(tmp_path: Path, *, profile: BundleProfile) -> Path:
inbox = tmp_path / "round"
inbox.mkdir(parents=True, exist_ok=True)
# Names ascend while numbers descend, so an index left in arrival order is
# the exact reverse of the one the profile asks for.
for name, number in (("alpha.md", "N300"), ("beta.md", "N200"), ("gamma.md", "N100")):
(inbox / name).write_text(
f"# {number} {name[:-3]}\n\nBody.\n", encoding="utf-8", newline=""
)
bundle = tmp_path / "bundle"
process_inbox(
inbox,
bundle,
INGESTED_AT,
okf_type="reference",
gate=lambda text: GateDecision(sanitized_text=text, disposition="warn"),
profile=profile,
)
return bundle
def run_door_c(tmp_path: Path, *, profile: BundleProfile) -> Path:
source = tmp_path / "source"
for name, number in (("alpha.md", "N300"), ("beta.md", "N200"), ("gamma.md", "N100")):
place(source, name, f"---\ntype: dataset\nnumber: {number}\n---\n\nBody.\n")
_, bundle = run_with(tmp_path, StubImportGate(), profile=profile)
return bundle
# --- the cross-door requirement -------------------------------------------
def test_both_doors_order_by_the_key_the_profile_names(tmp_path: Path) -> None:
door_b = numbers_in(run_door_b(tmp_path / "b", profile=NUMBERED), NUMBERED)
door_c = numbers_in(run_door_c(tmp_path / "c", profile=NUMBERED), NUMBERED)
# Both agree with the profile...
assert door_b == ["N100", "N200", "N300"]
assert door_c == ["N100", "N200", "N300"]
# ...and therefore with each other. Stated separately on purpose: a door
# that ignored the field would still produce a parseable index, and the
# arrival order it would produce is the reverse of this one.
assert door_b == door_c
def test_door_c_keeps_its_order_when_the_profile_names_none(tmp_path: Path) -> None:
"""The additivity guard, on the one pair where the two candidate orders differ.
`notes-beta.md` precedes `notes/alpha.md` by concept path ('-' < '/') and
follows it by generated filename ('a' < 'b'). Door C has always ordered by
the sender's concept path, so the concept path — not the target — is the
final tie-break, and this bundle's bytes do not move.
"""
source = tmp_path / "source"
place(source, "notes/alpha.md", "---\ntype: dataset\n---\n\nOne.\n")
place(source, "notes-beta.md", "---\ntype: dataset\n---\n\nTwo.\n")
_, bundle = run_with(tmp_path, StubImportGate(), profile=DEFAULT)
assert (bundle / "index.md").read_text(encoding="utf-8") == (
"- [notes-beta](import-notes-beta.md)\n- [notes/alpha](import-notes-alpha.md)\n"
)
# --- the closed sets ------------------------------------------------------
def test_a_sort_order_outside_the_closed_set_is_refused() -> None:
with pytest.raises(ValueError, match="sort_order"):
replace(NUMBERED.index, sort_order="by-relevance")
def test_a_sort_missing_outside_the_closed_set_is_refused() -> None:
with pytest.raises(ValueError, match="sort_missing"):
replace(NUMBERED.index, sort_missing="somewhere")
def test_a_sort_key_the_facet_policy_does_not_name_is_refused() -> None:
# Every entry would be missing the key and the ordering would silently do
# nothing — the failure mode this whole row exists to prevent.
with pytest.raises(ValueError, match="sort_key"):
replace(STRUCTURED_V1.index, sort_key="relevance")
def test_a_sort_key_on_a_policy_carrying_no_facets_is_refused() -> None:
with pytest.raises(ValueError, match="sort_key"):
replace(DEFAULT.index, sort_key="number")
# --- the helper -----------------------------------------------------------
def entry(concept_path: str, target: str, **facets: str) -> IndexEntry:
return IndexEntry(label=target, target=target, facets=facets, concept_path=concept_path)
def test_the_concept_path_is_the_final_tie_break() -> None:
"""Two equal keys must not leave the order to chance — and the tie-break is
the concept path, which for Door C is not the generated filename."""
policy: IndexPolicy = NUMBERED.index
entries = [
entry("notes/alpha.md", "import-notes-alpha.md", number="N100"),
entry("notes-beta.md", "import-notes-beta.md", number="N100"),
]
for order in (entries, list(reversed(entries))):
assert [item.target for item in policy.sort_entries(order)] == [
"import-notes-beta.md",
"import-notes-alpha.md",
]
def test_descending_reverses_the_key_and_not_the_tie_break() -> None:
policy = replace(NUMBERED.index, sort_order="descending")
entries = [
entry("a.md", "import-a.md", number="N100"),
entry("c.md", "import-c.md", number="N300"),
entry("b.md", "import-b.md", number="N300"),
]
assert [item.target for item in policy.sort_entries(entries)] == [
"import-b.md",
"import-c.md",
"import-a.md",
]
@pytest.mark.parametrize("order", ["ascending", "descending"])
@pytest.mark.parametrize("missing", ["first", "last"])
def test_the_missing_group_lands_where_the_policy_says_in_both_directions(
order: str, missing: str
) -> None:
"""The subtle one: reversing a single key tuple would flip the missing group
with it, so `sort_missing` would mean the opposite thing under `descending`.
"""
policy = replace(NUMBERED.index, sort_order=order, sort_missing=missing)
entries = [
entry("a.md", "import-a.md", number="N100"),
entry("b.md", "import-b.md"),
entry("c.md", "import-c.md", number="N300"),
]
present = (
["import-a.md", "import-c.md"] if order == "ascending" else ["import-c.md", "import-a.md"]
)
expected = ["import-b.md", *present] if missing == "first" else [*present, "import-b.md"]
assert [item.target for item in policy.sort_entries(entries)] == expected
def test_an_empty_facet_value_counts_as_missing() -> None:
# `FacetPolicy.render` already drops a falsy value, so an entry rendered
# without the facet must not sort as though it carried one.
policy = replace(NUMBERED.index, sort_missing="last")
entries = [
entry("a.md", "import-a.md", number=""),
entry("b.md", "import-b.md", number="N300"),
]
assert [item.target for item in policy.sort_entries(entries)] == [
"import-b.md",
"import-a.md",
]
def test_navigation_entries_stay_last_under_a_sort_key() -> None:
# Navigation is an outer grouping, not a competitor to the key: a child
# index carries no facets, so under `sort_missing="first"` it would lead
# the file if the grouping were not applied over the ordering.
policy = replace(NUMBERED.index, sort_missing="first")
entries = [
entry("krav/n900.md", "n900.md", number="N900"),
IndexEntry(label="sub (underkapitler)", target=f"sub/{NUMBERED.index.name}"),
entry("krav/n100.md", "n100.md", number="N100"),
]
assert [item.target for item in policy.sort_entries(entries)] == [
"n100.md",
"n900.md",
f"sub/{NUMBERED.index.name}",
]

7
tests/test_segmented_index.py
View file

@ -13,9 +13,10 @@ Two traps this closes, both measured:
`](krav/3-1/a.md)` also contains `](3-1/a.md)`. The writer here recomputes
each index whole and recognises managed lines with the ANCHORED
`link_pattern`, never that substring.
- ordering is routed through ONE named helper, `_index_sort_key`. A future
consumer-controlled ordering is then a parameter, not a refactor of every
place that happened to sort.
- ordering is routed through ONE named helper, `IndexPolicy.sort_entries`,
which BOTH doors call. It lives on the policy rather than in this door
because a consumer-controlled ordering wired into one door alone is a
profile field the other ignores in silence -- see `test_index_sort.py`.
"""
from __future__ import annotations