Kjell Tore Guttormsen 770d8d4fbf docs(index): assess a consumer-controlled index ordering - accept, reshape

Order 20260827T123301Z-3790625395 asked for a judgement on a profile-named
`index.sort_key`, explicitly not for code. The finding is accepted and so is
the mechanism; the shape needs one correction.

Verified what the order asserts about us: Door C sorts its index on the
concept path and nothing else (`importer.py:421`, a plain `sorted(documents)`),
and no policy object anywhere in the chain can redirect it. `FacetPolicy`
governs what an entry CARRIES; nothing governs how entries are ARRANGED. A
consumer cannot reach this today, exactly as claimed.

The correction: naming a key is necessary and not sufficient, and the
insufficiency is where the requester's own measured gain lives. A sort needs a
key, a value and a COMPARATOR, and D5's formula seats only the first two.
Measured here: sorted() over N100.9/N100.10/N100.2/N100.20/N100.1/N99.1 puts
N100.10 before N100.2 and N99.1 after the whole N100 family. A `sort_key:
req_number` resolved through default ordering would therefore have produced a
different wrong order, not the 100.0% ascending over 24338 pairs that was
measured. The library already knows this one module over -- `_version_key`
(`structure.py:375`) exists for it and its comment says why.

So: three fields on `IndexPolicy`, not one on `FacetPolicy` -- `sort_key`,
`sort_order` from a closed set (`lexicographic` | `natural`, never a
caller-supplied callable, which would make emitted bytes depend on caller code
and break rebuild-equals-incremental), and `sort_missing`. Concept path stays
the final tie-break so the new ordering REFINES the existing total order rather
than replacing it; trading S8b determinism for retrieval quality would be the
worse failure.

Timing: build after the segmentation voyage lands, as a parameter on the
`_index_sort_key` seam Step 10 leaves behind, unifying both doors' index
writers in the same change. That seam is real and manifest-enforced, but it is
Door B's while this order is Door C's -- a profile naming an ordering that one
door honours and the other ignores would repeat the `IndexPolicy.per_directory`
trap.

Not claimed: that ordering explains the whole gap. T8-4 remains unexplained and
we attach no hypothesis to it either.

Order: 20260827T123301Z-3790625395-from-.claude

Co-Authored-By: Claude <claude-opus-5>

2026-08-31 23:35:25 +02:00

6.3 KiB

Raw Blame History

Index ordering as a consumer-controlled property

Status: assessment complete, nothing built. Requested by order 20260827T123301Z-3790625395 from .claude, which asked explicitly for a judgement and not for code.

Verdict: accept the finding, accept the mechanism, correct its shape. An index's order is a performance property, a consumer cannot control it today, and a profile-named ordering key is the right way to give them control. But naming a key is necessary and not sufficient, and the insufficiency is where the requester's own measured gain actually lives.

What was asked

vegnormal-okf measured four faceted arms against one corpus, 32 questions, locked scoring, shared denominator 17. Arm F4 was arm F3 with exactly one variable moved — the index's sort order — and scored 14/17 against F3's 12/17. The single-variable claim was verified against the files rather than assumed: line sets identical in 32 of 32 cells, byte lengths identical in 32 of 32, order different in 32 of 32.

The request: make the index's sort key something a profile can NAME, on the same footing as the facet keys, per this library's standing rule that a profile names a key and the caller owns its value (D5).

What is true today, measured

Door C sorts its index on the concept path, and nothing else: importer.py:421 iterates sorted(documents), a plain lexicographic sort over concept paths, with no policy field anywhere in the chain that could redirect it. FacetPolicy (profiles.py:388) governs what an entry CARRIES; no object governs how entries are ARRANGED. So the second half of the requester's premise holds exactly as stated: a consumer cannot control this today, at all.

For a UUID-named corpus that yields the 51.4% ascending-neighbour figure the order reports — which is to say, chance. The content was present in all three indexes; T8-1 failed because the answer was unfindable, not absent.

Why naming a key is not enough

A sort needs three things, and D5's formula seats only two of them. A profile can name the key; a caller can own the value. Neither of those is the comparator — how two values are placed relative to each other — and that is the part carrying the measured gain.

Measured, on this repo's own interpreter:

sorted(["N100.9", "N100.10", "N100.2", "N100.20", "N100.1", "N99.1"])
  -> ['N100.1', 'N100.10', 'N100.2', 'N100.20', 'N100.9', 'N99.1']

Two independent inversions: N100.10 sorts before N100.2, and N99.1 lands after the entire N100 family. A sort_key: req_number resolved through Python's default ordering would therefore have handed vegnormal-okf a different wrong order, not the 100.0% ascending they measured over 24 338 pairs. Their result came from comparing numerically. The key name alone does not carry that.

The library already knows this lesson one module over. structure._version_key (structure.py:375) exists for precisely this reason, and its comment says so: "Lexicographic order would put the 2026 edition of a document before its 9th revision." The gap is that the lesson is applied to versions and not to index order.

The shape we would build

Not FacetPolicy.sort_key. Ordering is not a property of what an entry carries, and putting it there would couple two questions that vary independently — a consumer may well want title faceted and req_number ordering. It belongs on IndexPolicy, beside name and facets.

Three fields, not one:

sort_key — the frontmatter key to order on, named by the profile.
sort_order — a member of a CLOSED set, lexicographic or natural, where natural is the digit-run comparison _version_key already implements. Closed, and never a caller-supplied callable: a callable makes the emitted bytes depend on the caller's code rather than on the profile, which would break the determinism contract that a from-scratch rebuild equals an incremental update. A profile has to be comparable across runs and across repositories, and a function is not.
sort_missing — where a concept lacking the key goes. It needs an answer, because a corpus is never uniformly numbered.

And one invariant: the new ordering must be a refinement of the existing one, never a replacement. The final tie-break stays the concept path, so any two entries the new key cannot separate keep the total order they have now. Without that, S8b's determinism requirement is traded away for retrieval quality, and a non-deterministic index is a worse failure than a badly ordered one.

Relation to the segmentation voyage

The plan at .claude/projects/2026-08-30-door-b-concept-granularity/plan.md Step 10 routes every index write through one named helper, _index_sort_key, currently (is_nav, target), and pins that name in the step's manifest as a must_contain check. That seam is real and machine-enforced, and it is what makes this order a later parameter rather than a refactor.

Two qualifications, because the seam does not do as much as it looks like it does. It is Door B's writer, and this order concerns Door C; the two doors have separate index writers today. And a seam is a place to put a comparator, not a comparator. When this is built, both doors should route through one shared helper so a profile field has a single implementation to reach — otherwise a profile would name an ordering that one door honours and the other silently ignores, which is the IndexPolicy.per_directory trap (profiles.py:863) repeating itself.

Recommendation

Build it, but not now, and not in this shape without the comparator. Door C's index writer and Door B's are both inside or adjacent to the surface the segmentation voyage owns for its next several sessions; changing index ordering underneath it would collide with a byte-stability pin over four profiles. The right sequence is: land the voyage, then add sort_key / sort_order / sort_missing to IndexPolicy as a parameter on the seam that voyage leaves behind, unifying the two doors' writers in the same change.

What we are NOT saying: that ordering explains the whole gap. The requester was explicit that one cell, T8-4, still separates F4 from F and is unexplained, and attached no hypothesis to it. Neither do we. The recommendation stands on the narrower claim the measurement actually supports — the ordering matters materially, and a consumer cannot reach it.

6.3 KiB Raw Blame History