Kjell Tore Guttormsen
770d8d4fbf
Order 20260827T123301Z-3790625395 asked for a judgement on a profile-named `index.sort_key`, explicitly not for code. The finding is accepted and so is the mechanism; the shape needs one correction. Verified what the order asserts about us: Door C sorts its index on the concept path and nothing else (`importer.py:421`, a plain `sorted(documents)`), and no policy object anywhere in the chain can redirect it. `FacetPolicy` governs what an entry CARRIES; nothing governs how entries are ARRANGED. A consumer cannot reach this today, exactly as claimed. The correction: naming a key is necessary and not sufficient, and the insufficiency is where the requester's own measured gain lives. A sort needs a key, a value and a COMPARATOR, and D5's formula seats only the first two. Measured here: sorted() over N100.9/N100.10/N100.2/N100.20/N100.1/N99.1 puts N100.10 before N100.2 and N99.1 after the whole N100 family. A `sort_key: req_number` resolved through default ordering would therefore have produced a different wrong order, not the 100.0% ascending over 24338 pairs that was measured. The library already knows this one module over -- `_version_key` (`structure.py:375`) exists for it and its comment says why. So: three fields on `IndexPolicy`, not one on `FacetPolicy` -- `sort_key`, `sort_order` from a closed set (`lexicographic` | `natural`, never a caller-supplied callable, which would make emitted bytes depend on caller code and break rebuild-equals-incremental), and `sort_missing`. Concept path stays the final tie-break so the new ordering REFINES the existing total order rather than replacing it; trading S8b determinism for retrieval quality would be the worse failure. Timing: build after the segmentation voyage lands, as a parameter on the `_index_sort_key` seam Step 10 leaves behind, unifying both doors' index writers in the same change. That seam is real and manifest-enforced, but it is Door B's while this order is Door C's -- a profile naming an ordering that one door honours and the other ignores would repeat the `IndexPolicy.per_directory` trap. Not claimed: that ordering explains the whole gap. T8-4 remains unexplained and we attach no hypothesis to it either. Order: 20260827T123301Z-3790625395-from-.claude Co-Authored-By: Claude <claude-opus-5>
124 lines
6.3 KiB
Markdown
124 lines
6.3 KiB
Markdown
# Index ordering as a consumer-controlled property
|
|
|
|
**Status:** assessment complete, nothing built. Requested by order
|
|
`20260827T123301Z-3790625395` from `.claude`, which asked explicitly for a
|
|
judgement and not for code.
|
|
|
|
**Verdict: accept the finding, accept the mechanism, correct its shape.** An
|
|
index's order is a performance property, a consumer cannot control it today,
|
|
and a profile-named ordering key is the right way to give them control. But
|
|
naming a key is *necessary and not sufficient*, and the insufficiency is where
|
|
the requester's own measured gain actually lives.
|
|
|
|
## What was asked
|
|
|
|
`vegnormal-okf` measured four faceted arms against one corpus, 32 questions,
|
|
locked scoring, shared denominator 17. Arm F4 was arm F3 with exactly one
|
|
variable moved — the index's sort order — and scored 14/17 against F3's 12/17.
|
|
The single-variable claim was verified against the files rather than assumed:
|
|
line sets identical in 32 of 32 cells, byte lengths identical in 32 of 32,
|
|
order different in 32 of 32.
|
|
|
|
The request: make the index's sort key something a profile can NAME, on the
|
|
same footing as the facet keys, per this library's standing rule that a profile
|
|
names a key and the caller owns its value (D5).
|
|
|
|
## What is true today, measured
|
|
|
|
Door C sorts its index on the concept path, and nothing else:
|
|
`importer.py:421` iterates `sorted(documents)`, a plain lexicographic sort over
|
|
concept paths, with no policy field anywhere in the chain that could redirect
|
|
it. `FacetPolicy` (`profiles.py:388`) governs what an entry CARRIES; no object
|
|
governs how entries are ARRANGED. So the second half of the requester's premise
|
|
holds exactly as stated: a consumer cannot control this today, at all.
|
|
|
|
For a UUID-named corpus that yields the 51.4% ascending-neighbour figure the
|
|
order reports — which is to say, chance. The content was present in all three
|
|
indexes; T8-1 failed because the answer was unfindable, not absent.
|
|
|
|
## Why naming a key is not enough
|
|
|
|
A sort needs three things, and D5's formula seats only two of them. A profile
|
|
can name the key; a caller can own the value. Neither of those is the
|
|
**comparator** — how two values are placed relative to each other — and that is
|
|
the part carrying the measured gain.
|
|
|
|
Measured, on this repo's own interpreter:
|
|
|
|
```
|
|
sorted(["N100.9", "N100.10", "N100.2", "N100.20", "N100.1", "N99.1"])
|
|
-> ['N100.1', 'N100.10', 'N100.2', 'N100.20', 'N100.9', 'N99.1']
|
|
```
|
|
|
|
Two independent inversions: `N100.10` sorts before `N100.2`, and `N99.1` lands
|
|
*after* the entire `N100` family. A `sort_key: req_number` resolved through
|
|
Python's default ordering would therefore have handed `vegnormal-okf` a
|
|
different wrong order, not the 100.0% ascending they measured over 24 338
|
|
pairs. Their result came from comparing numerically. The key name alone does
|
|
not carry that.
|
|
|
|
The library already knows this lesson one module over. `structure._version_key`
|
|
(`structure.py:375`) exists for precisely this reason, and its comment says so:
|
|
"Lexicographic order would put the 2026 edition of a document before its 9th
|
|
revision." The gap is that the lesson is applied to versions and not to index
|
|
order.
|
|
|
|
## The shape we would build
|
|
|
|
Not `FacetPolicy.sort_key`. Ordering is not a property of what an entry carries,
|
|
and putting it there would couple two questions that vary independently — a
|
|
consumer may well want `title` faceted and `req_number` ordering. It belongs on
|
|
`IndexPolicy`, beside `name` and `facets`.
|
|
|
|
Three fields, not one:
|
|
|
|
- **`sort_key`** — the frontmatter key to order on, named by the profile.
|
|
- **`sort_order`** — a member of a CLOSED set, `lexicographic` or `natural`,
|
|
where `natural` is the digit-run comparison `_version_key` already
|
|
implements. Closed, and never a caller-supplied callable: a callable makes
|
|
the emitted bytes depend on the caller's code rather than on the profile,
|
|
which would break the determinism contract that a from-scratch rebuild equals
|
|
an incremental update. A profile has to be comparable across runs and across
|
|
repositories, and a function is not.
|
|
- **`sort_missing`** — where a concept lacking the key goes. It needs an answer,
|
|
because a corpus is never uniformly numbered.
|
|
|
|
And one invariant: the new ordering must be a **refinement** of the existing
|
|
one, never a replacement. The final tie-break stays the concept path, so any
|
|
two entries the new key cannot separate keep the total order they have now.
|
|
Without that, S8b's determinism requirement is traded away for retrieval
|
|
quality, and a non-deterministic index is a worse failure than a badly ordered
|
|
one.
|
|
|
|
## Relation to the segmentation voyage
|
|
|
|
The plan at `.claude/projects/2026-08-30-door-b-concept-granularity/plan.md`
|
|
Step 10 routes every index write through one named helper, `_index_sort_key`,
|
|
currently `(is_nav, target)`, and pins that name in the step's manifest as a
|
|
`must_contain` check. That seam is real and machine-enforced, and it is what
|
|
makes this order a later **parameter** rather than a refactor.
|
|
|
|
Two qualifications, because the seam does not do as much as it looks like it
|
|
does. It is **Door B's** writer, and this order concerns **Door C**; the two
|
|
doors have separate index writers today. And a seam is a place to put a
|
|
comparator, not a comparator. When this is built, both doors should route
|
|
through one shared helper so a profile field has a single implementation to
|
|
reach — otherwise a profile would name an ordering that one door honours and
|
|
the other silently ignores, which is the `IndexPolicy.per_directory` trap
|
|
(`profiles.py:863`) repeating itself.
|
|
|
|
## Recommendation
|
|
|
|
Build it, but not now, and not in this shape without the comparator. Door C's
|
|
index writer and Door B's are both inside or adjacent to the surface the
|
|
segmentation voyage owns for its next several sessions; changing index ordering
|
|
underneath it would collide with a byte-stability pin over four profiles. The
|
|
right sequence is: land the voyage, then add `sort_key` / `sort_order` /
|
|
`sort_missing` to `IndexPolicy` as a parameter on the seam that voyage leaves
|
|
behind, unifying the two doors' writers in the same change.
|
|
|
|
What we are NOT saying: that ordering explains the whole gap. The requester was
|
|
explicit that one cell, T8-4, still separates F4 from F and is unexplained, and
|
|
attached no hypothesis to it. Neither do we. The recommendation stands on the
|
|
narrower claim the measurement actually supports — the ordering matters
|
|
materially, and a consumer cannot reach it.
|