docs(index): assess a consumer-controlled index ordering - accept, reshape

Order 20260827T123301Z-3790625395 asked for a judgement on a profile-named
`index.sort_key`, explicitly not for code. The finding is accepted and so is
the mechanism; the shape needs one correction.

Verified what the order asserts about us: Door C sorts its index on the
concept path and nothing else (`importer.py:421`, a plain `sorted(documents)`),
and no policy object anywhere in the chain can redirect it. `FacetPolicy`
governs what an entry CARRIES; nothing governs how entries are ARRANGED. A
consumer cannot reach this today, exactly as claimed.

The correction: naming a key is necessary and not sufficient, and the
insufficiency is where the requester's own measured gain lives. A sort needs a
key, a value and a COMPARATOR, and D5's formula seats only the first two.
Measured here: sorted() over N100.9/N100.10/N100.2/N100.20/N100.1/N99.1 puts
N100.10 before N100.2 and N99.1 after the whole N100 family. A `sort_key:
req_number` resolved through default ordering would therefore have produced a
different wrong order, not the 100.0% ascending over 24338 pairs that was
measured. The library already knows this one module over -- `_version_key`
(`structure.py:375`) exists for it and its comment says why.

So: three fields on `IndexPolicy`, not one on `FacetPolicy` -- `sort_key`,
`sort_order` from a closed set (`lexicographic` | `natural`, never a
caller-supplied callable, which would make emitted bytes depend on caller code
and break rebuild-equals-incremental), and `sort_missing`. Concept path stays
the final tie-break so the new ordering REFINES the existing total order rather
than replacing it; trading S8b determinism for retrieval quality would be the
worse failure.

Timing: build after the segmentation voyage lands, as a parameter on the
`_index_sort_key` seam Step 10 leaves behind, unifying both doors' index
writers in the same change. That seam is real and manifest-enforced, but it is
Door B's while this order is Door C's -- a profile naming an ordering that one
door honours and the other ignores would repeat the `IndexPolicy.per_directory`
trap.

Not claimed: that ordering explains the whole gap. T8-4 remains unexplained and
we attach no hypothesis to it either.

Order: 20260827T123301Z-3790625395-from-.claude

Co-Authored-By: Claude <claude-opus-5>

docs/plan/index-ordering.md

@@ -0,0 +1,124 @@
+# Index ordering as a consumer-controlled property
+
+**Status:** assessment complete, nothing built. Requested by order
+`20260827T123301Z-3790625395` from `.claude`, which asked explicitly for a
+judgement and not for code.
+
+**Verdict: accept the finding, accept the mechanism, correct its shape.** An
+index's order is a performance property, a consumer cannot control it today,
+and a profile-named ordering key is the right way to give them control. But
+naming a key is *necessary and not sufficient*, and the insufficiency is where
+the requester's own measured gain actually lives.
+
+## What was asked
+
+`vegnormal-okf` measured four faceted arms against one corpus, 32 questions,
+locked scoring, shared denominator 17. Arm F4 was arm F3 with exactly one
+variable moved — the index's sort order — and scored 14/17 against F3's 12/17.
+The single-variable claim was verified against the files rather than assumed:
+line sets identical in 32 of 32 cells, byte lengths identical in 32 of 32,
+order different in 32 of 32.
+
+The request: make the index's sort key something a profile can NAME, on the
+same footing as the facet keys, per this library's standing rule that a profile
+names a key and the caller owns its value (D5).
+
+## What is true today, measured
+
+Door C sorts its index on the concept path, and nothing else:
+`importer.py:421` iterates `sorted(documents)`, a plain lexicographic sort over
+concept paths, with no policy field anywhere in the chain that could redirect
+it. `FacetPolicy` (`profiles.py:388`) governs what an entry CARRIES; no object
+governs how entries are ARRANGED. So the second half of the requester's premise
+holds exactly as stated: a consumer cannot control this today, at all.
+
+For a UUID-named corpus that yields the 51.4% ascending-neighbour figure the
+order reports — which is to say, chance. The content was present in all three
+indexes; T8-1 failed because the answer was unfindable, not absent.
+
+## Why naming a key is not enough
+
+A sort needs three things, and D5's formula seats only two of them. A profile
+can name the key; a caller can own the value. Neither of those is the
+**comparator** — how two values are placed relative to each other — and that is
+the part carrying the measured gain.
+
+Measured, on this repo's own interpreter:
+
+```
+sorted(["N100.9", "N100.10", "N100.2", "N100.20", "N100.1", "N99.1"])
+  -> ['N100.1', 'N100.10', 'N100.2', 'N100.20', 'N100.9', 'N99.1']
+```
+
+Two independent inversions: `N100.10` sorts before `N100.2`, and `N99.1` lands
+*after* the entire `N100` family. A `sort_key: req_number` resolved through
+Python's default ordering would therefore have handed `vegnormal-okf` a
+different wrong order, not the 100.0% ascending they measured over 24 338
+pairs. Their result came from comparing numerically. The key name alone does
+not carry that.
+
+The library already knows this lesson one module over. `structure._version_key`
+(`structure.py:375`) exists for precisely this reason, and its comment says so:
+"Lexicographic order would put the 2026 edition of a document before its 9th
+revision." The gap is that the lesson is applied to versions and not to index
+order.
+
+## The shape we would build
+
+Not `FacetPolicy.sort_key`. Ordering is not a property of what an entry carries,
+and putting it there would couple two questions that vary independently — a
+consumer may well want `title` faceted and `req_number` ordering. It belongs on
+`IndexPolicy`, beside `name` and `facets`.
+
+Three fields, not one:
+
+- **`sort_key`** — the frontmatter key to order on, named by the profile.
+- **`sort_order`** — a member of a CLOSED set, `lexicographic` or `natural`,
+  where `natural` is the digit-run comparison `_version_key` already
+  implements. Closed, and never a caller-supplied callable: a callable makes
+  the emitted bytes depend on the caller's code rather than on the profile,
+  which would break the determinism contract that a from-scratch rebuild equals
+  an incremental update. A profile has to be comparable across runs and across
+  repositories, and a function is not.
+- **`sort_missing`** — where a concept lacking the key goes. It needs an answer,
+  because a corpus is never uniformly numbered.
+
+And one invariant: the new ordering must be a **refinement** of the existing
+one, never a replacement. The final tie-break stays the concept path, so any
+two entries the new key cannot separate keep the total order they have now.
+Without that, S8b's determinism requirement is traded away for retrieval
+quality, and a non-deterministic index is a worse failure than a badly ordered
+one.
+
+## Relation to the segmentation voyage
+
+The plan at `.claude/projects/2026-08-30-door-b-concept-granularity/plan.md`
+Step 10 routes every index write through one named helper, `_index_sort_key`,
+currently `(is_nav, target)`, and pins that name in the step's manifest as a
+`must_contain` check. That seam is real and machine-enforced, and it is what
+makes this order a later **parameter** rather than a refactor.
+
+Two qualifications, because the seam does not do as much as it looks like it
+does. It is **Door B's** writer, and this order concerns **Door C**; the two
+doors have separate index writers today. And a seam is a place to put a
+comparator, not a comparator. When this is built, both doors should route
+through one shared helper so a profile field has a single implementation to
+reach — otherwise a profile would name an ordering that one door honours and
+the other silently ignores, which is the `IndexPolicy.per_directory` trap
+(`profiles.py:863`) repeating itself.
+
+## Recommendation
+
+Build it, but not now, and not in this shape without the comparator. Door C's
+index writer and Door B's are both inside or adjacent to the surface the
+segmentation voyage owns for its next several sessions; changing index ordering
+underneath it would collide with a byte-stability pin over four profiles. The
+right sequence is: land the voyage, then add `sort_key` / `sort_order` /
+`sort_missing` to `IndexPolicy` as a parameter on the seam that voyage leaves
+behind, unifying the two doors' writers in the same change.
+
+What we are NOT saying: that ordering explains the whole gap. The requester was
+explicit that one cell, T8-4, still separates F4 from F and is unexplained, and
+attached no hypothesis to it. Neither do we. The recommendation stands on the
+narrower claim the measurement actually supports — the ordering matters
+materially, and a consumer cannot reach it.