Kjell Tore Guttormsen
9d1f4b14ed
Every fixture, test document, tool example and document now uses an invented kitchen-and-baking handbook series, written in this repository. The package's behaviour is unchanged; src/ changes are comments and help text only. - Generated fixtures are regenerated from their generators. Their structural counts are identical before and after: elements, images, rows, cells, headings, bookmarks and the witness inventory's per-document totals. The image-inbox and accounting documents are renamed kapittel-84-*. - tools/okf_accounting_gate.py: the two options that named one real corpus each are replaced by a generic, repeatable --corpus PATH with no default. Row 5 compares the PDF pair alone. Gate verdict unchanged: RED rows 2, 3, 6. - tools/okf_witness.py: the STS JSON reader for one publisher's delivery is removed, along with its three twins and five tests. The mutation harness loses W09. - docs/: 13 dated reports that documented runs on a retired reference corpus are removed, and 40 are neutralized. Dead links are removed, and no new dangling path is introduced. - The synthetic MCP-gate corpus and the residual probe words are neutral. Valgt: keep the `okf quality --fasit` bar value (the measured fraction, one corpus) and rewrite only its provenance, because the verdict stays unchanged and the number names nothing. Term check with the local list: 0 of 411 tracked files, 0 file names, 0 of 27 binary fixtures. Suite after git add: 2457 passed, 1 skipped. The base tree had 2460 passed and 2 skipped; five tests went with the JSON reader and four were added by the term check. ruff, ruff format and mypy --strict src/ are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
124 lines
6.3 KiB
Markdown
124 lines
6.3 KiB
Markdown
# Index ordering as a consumer-controlled property
|
|
|
|
**Status:** assessment complete, nothing built. Requested by order
|
|
`20260827T123301Z-3790625395` from `.claude`, which asked explicitly for a
|
|
judgement and not for code.
|
|
|
|
**Verdict: accept the finding, accept the mechanism, correct its shape.** An
|
|
index's order is a performance property, a consumer cannot control it today,
|
|
and a profile-named ordering key is the right way to give them control. But
|
|
naming a key is *necessary and not sufficient*, and the insufficiency is where
|
|
the requester's own measured gain actually lives.
|
|
|
|
## What was asked
|
|
|
|
A downstream consumer measured four faceted arms against one corpus, 32 questions,
|
|
locked scoring, shared denominator 17. Arm F4 was arm F3 with exactly one
|
|
variable moved — the index's sort order — and scored 14/17 against F3's 12/17.
|
|
The single-variable claim was verified against the files rather than assumed:
|
|
line sets identical in 32 of 32 cells, byte lengths identical in 32 of 32,
|
|
order different in 32 of 32.
|
|
|
|
The request: make the index's sort key something a profile can NAME, on the
|
|
same footing as the facet keys, per this library's standing rule that a profile
|
|
names a key and the caller owns its value (D5).
|
|
|
|
## What is true today, measured
|
|
|
|
Door C sorts its index on the concept path, and nothing else:
|
|
`importer.py:421` iterates `sorted(documents)`, a plain lexicographic sort over
|
|
concept paths, with no policy field anywhere in the chain that could redirect
|
|
it. `FacetPolicy` (`profiles.py:388`) governs what an entry CARRIES; no object
|
|
governs how entries are ARRANGED. So the second half of the requester's premise
|
|
holds exactly as stated: a consumer cannot control this today, at all.
|
|
|
|
For a UUID-named corpus that yields the 51.4% ascending-neighbour figure the
|
|
order reports — which is to say, chance. The content was present in all three
|
|
indexes; T8-1 failed because the answer was unfindable, not absent.
|
|
|
|
## Why naming a key is not enough
|
|
|
|
A sort needs three things, and D5's formula seats only two of them. A profile
|
|
can name the key; a caller can own the value. Neither of those is the
|
|
**comparator** — how two values are placed relative to each other — and that is
|
|
the part carrying the measured gain.
|
|
|
|
Measured, on this repo's own interpreter:
|
|
|
|
```
|
|
sorted(["Q100.9", "Q100.10", "Q100.2", "Q100.20", "Q100.1", "Q99.1"])
|
|
-> ['Q100.1', 'Q100.10', 'Q100.2', 'Q100.20', 'Q100.9', 'Q99.1']
|
|
```
|
|
|
|
Two independent inversions: `Q100.10` sorts before `Q100.2`, and `Q99.1` lands
|
|
*after* the entire `Q100` family. A `sort_key: req_number` resolved through
|
|
Python's default ordering would therefore have handed that consumer a
|
|
different wrong order, not the 100.0% ascending they measured over 24 338
|
|
pairs. Their result came from comparing numerically. The key name alone does
|
|
not carry that.
|
|
|
|
The library already knows this lesson one module over. `structure._version_key`
|
|
(`structure.py:375`) exists for precisely this reason, and its comment says so:
|
|
"Lexicographic order would put the 2026 edition of a document before its 9th
|
|
revision." The gap is that the lesson is applied to versions and not to index
|
|
order.
|
|
|
|
## The shape we would build
|
|
|
|
Not `FacetPolicy.sort_key`. Ordering is not a property of what an entry carries,
|
|
and putting it there would couple two questions that vary independently — a
|
|
consumer may well want `title` faceted and `req_number` ordering. It belongs on
|
|
`IndexPolicy`, beside `name` and `facets`.
|
|
|
|
Three fields, not one:
|
|
|
|
- **`sort_key`** — the frontmatter key to order on, named by the profile.
|
|
- **`sort_order`** — a member of a CLOSED set, `lexicographic` or `natural`,
|
|
where `natural` is the digit-run comparison `_version_key` already
|
|
implements. Closed, and never a caller-supplied callable: a callable makes
|
|
the emitted bytes depend on the caller's code rather than on the profile,
|
|
which would break the determinism contract that a from-scratch rebuild equals
|
|
an incremental update. A profile has to be comparable across runs and across
|
|
repositories, and a function is not.
|
|
- **`sort_missing`** — where a concept lacking the key goes. It needs an answer,
|
|
because a corpus is never uniformly numbered.
|
|
|
|
And one invariant: the new ordering must be a **refinement** of the existing
|
|
one, never a replacement. The final tie-break stays the concept path, so any
|
|
two entries the new key cannot separate keep the total order they have now.
|
|
Without that, S8b's determinism requirement is traded away for retrieval
|
|
quality, and a non-deterministic index is a worse failure than a badly ordered
|
|
one.
|
|
|
|
## Relation to the segmentation voyage
|
|
|
|
The plan at `.claude/projects/2026-08-30-door-b-concept-granularity/plan.md`
|
|
Step 10 routes every index write through one named helper, `_index_sort_key`,
|
|
currently `(is_nav, target)`, and pins that name in the step's manifest as a
|
|
`must_contain` check. That seam is real and machine-enforced, and it is what
|
|
makes this order a later **parameter** rather than a refactor.
|
|
|
|
Two qualifications, because the seam does not do as much as it looks like it
|
|
does. It is **Door B's** writer, and this order concerns **Door C**; the two
|
|
doors have separate index writers today. And a seam is a place to put a
|
|
comparator, not a comparator. When this is built, both doors should route
|
|
through one shared helper so a profile field has a single implementation to
|
|
reach — otherwise a profile would name an ordering that one door honours and
|
|
the other silently ignores, which is the `IndexPolicy.per_directory` trap
|
|
(`profiles.py:863`) repeating itself.
|
|
|
|
## Recommendation
|
|
|
|
Build it, but not now, and not in this shape without the comparator. Door C's
|
|
index writer and Door B's are both inside or adjacent to the surface the
|
|
segmentation voyage owns for its next several sessions; changing index ordering
|
|
underneath it would collide with a byte-stability pin over four profiles. The
|
|
right sequence is: land the voyage, then add `sort_key` / `sort_order` /
|
|
`sort_missing` to `IndexPolicy` as a parameter on the seam that voyage leaves
|
|
behind, unifying the two doors' writers in the same change.
|
|
|
|
What we are NOT saying: that ordering explains the whole gap. The requester was
|
|
explicit that one cell, T8-4, still separates F4 from F and is unexplained, and
|
|
attached no hypothesis to it. Neither do we. The recommendation stands on the
|
|
narrower claim the measurement actually supports — the ordering matters
|
|
materially, and a consumer cannot reach it.
|