open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

docs(plan): record the structure-derivation decisions, mutations and cost

Browse source 
Six decisions, each with the failure it prevents: confidence carried in the
format, certainty stated per field, supersession never inferred from one
document, the index as a projection rather than an append, facets as a new
profile, and Door B's ownership stamp left deliberately open.

Six findings reported as findings rather than as gaps, per the order: the cost
of closing the metadata gap is 3.3x-6.4x the flat index and whether that keeps
the OKF arm's cost advantage cannot be answered from this repo -- it needs
their bake-off re-run on their corpus; a reference to a parent-level number
does not resolve to a child, and the fix needs a real corpus before 'unique'
can be defended; OKF_V0_2's verified/status/stale_after are still expressible
and unwritten for the reason already recorded.

Every number carries its denominator, and the supersedes 0/55 carries a
known-positive control (50/55 on a corpus that has shared numbers) proving the
query can find what it reported missing.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-27 00:39:48 +02:00
commit cf65635648
3 changed files with 248 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

226
docs/plan/structure-derivation.md Normal file
View file

@ -0,0 +1,226 @@
# Structure derivation: numbering, hierarchy, cross-references, supersession
Answers order `20260826T124341Z-181848459` (from `.claude`, 2026-08-26). The
premise it waited on — the Door B / Door A capability gap — was closed in
`dc9ea59`: the answer was title derivation, not a door change and not a profile
on Door B alone.
The operator's requirement, quoted rather than paraphrased:
> "Og akkurat dette MÅ llm-ingestion-okf kunne lage automatisk og så ideelt som
> mulig når dokumenter legges inn i en okf bundle innboks (som kan skje en
> eller flere ganger, altså additivt)"
where "akkurat dette" is the form `.claude` described: documents that are
numbered, hierarchical, cross-referencing, and versioned such that one edition
supersedes another.
## What was measured, and what the measurement does and does not say
ms-ai-architect ran a pre-registered bake-off on 2026-08-26 over 55 documents
and 40 gold questions. Every arm retrieved 40/40, so the measurement could not
separate the arms on retrieval quality. The only axis that separated them was
trap exposure: 18/20 for the OKF-index arm against 8/20 for a frontmatter
head-scan over the same content. The reason was measured independently on both
sides — the flat `DEFAULT` index carries title/date/status/supersedes 0 times
while the documents in the same bundle carry them 55/55/55/5.
That is evidence against `DEFAULT`'s **index**, not against OKF. The metadata is
in the bundle; the index throws it away.
## Decisions
### D-S1. Every derived fact is marked, in the format
`derived` is a flat flow list naming exactly the fields this library inferred.
A field present and absent from that list was stated by the producer. The
format carries it, so this was not a case where "the format cannot express
confidence" — a flat list of strings round-trips through this library's
line-oriented parser and through the guard's T2 grammar alike.
Rationale: an unmarked heuristic is worse than no heuristic, because the
consumer cannot know when to doubt it. With the marker, a consumer that trusts
nothing derived can still use everything declared.
### D-S2. Certainty is stated per field, not per document
| Field | Source | Certainty |
|---|---|---|
| `title` | producer's `title` key | declared |
| | leading `# ` heading | **derived** |
| | filename stem | **derived** |
| `number` | producer's `number` key | declared |
| | leading number token of filename, then of title | **derived** |
| `parent` | arithmetic on `number` (drop last dotted part) | inherits `number`'s |
| `references` | producer's `references` key | declared |
| | bundle-local link targets + number mentions in the body | **derived** |
| `supersedes` | producer's `supersedes` key | declared |
| | same number + ordered `version` | **derived** |
| `version`, `status`, `date` | producer only | declared |
The number grammar is alpha-prefixed (`N500`, `V720`, `R610.4`) or dotted
numeric (`4.2.1`). A **bare integer is deliberately not a number**: admitting
`12-things.md` or `2026-notes.md` would stamp a document number on most of a
second brain that never had one.
A parent our own grammar could never admit is not emitted at all (`4.2` would
drop to `4`). An unresolved list that never clears is one a consumer learns to
ignore.
### D-S3. Supersession is never inferred from one document
Whether A supersedes B is a fact about a pair. `derive_document_structure`
records only what the producer declared; the bundle-level resolver may propose
a chain from a shared number plus ordered versions, and every edge it proposes
is marked derived. A group whose members do not all carry a version proposes
**nothing** — that is exactly the case where a guess would be indistinguishable
from a fact, and supersession is the relation a consumer is most likely to act
on.
### D-S4. The index is a projection, recomputed from the whole bundle
This is the answer to the additive requirement, and it is a design decision
rather than an algorithm: `resolve_structure` is a pure function of the whole
document set, and Door B rewrites the managed region of the index from it every
round. Nothing is diffed, so there is no diffing algorithm to prove correct.
Three of the order's four additive requirements fall out by construction:
- **rebuild == incremental** — both are the same function of the same files;
- **idempotence** — the concept name is the identity, so re-dropping a document
replaces its entry rather than doubling it;
- **a round-3 document can supersede a round-1 document** — the round-1 entry is
rewritten, which an append-only index could never do.
The fourth is explicit: an unresolved pointer is rendered with a `?` suffix
rather than omitted. During build-up, pointing at something not dropped yet is
the normal state; the dangerous version of it is the one that leaves no trace.
### D-S5. Facets arrive as a new profile (`STRUCTURED_V1`), never on `DEFAULT`
`DEFAULT` states commons' ingest-spec §6 index layer. Changing its rendered
bytes from here would be this repo editing a contract it does not own (O2), and
it would churn every golden fixture Door B has written. `entry_pattern` **is**
`link_pattern` when a policy carries no facets, so `DEFAULT` and `STRICT_V1`
match the same lines and emit the same bytes; a test asserts Door B's output is
byte-identical with and without the new parameter.
### D-S6. Door B keeps writing the literal `generated: true`
Not routed through `profile.ownership.stamp(...)`, which would move `DEFAULT`'s
bytes to the O2 mapping and orphan every bundle this door has already written.
Which stamp Door B should write is a separate question from this order's, and
answering it here would have answered it silently. **Open.**
## Verification
All commands are runnable and were run. Suite: 615 → 695.
```
.venv/bin/python -m pytest -q # 695 passed
.venv/bin/ruff check . && .venv/bin/ruff format --check .
.venv/bin/mypy --strict src/
```
### Mutation testing, with proof the mutation was applied
Each run asserts the anchor was present and that exactly one `# MUTANT` marker
landed in the file before the suite was read; a run whose anchor was absent
reports "NO RESULT READ" rather than a green suite. The first attempt reverted
mutations with `git checkout --`, which discarded uncommitted work; the harness
now snapshots the file text instead.
| # | Mutation | Result |
|---|---|---|
| M1 | index entry drops its facet tail | 6 failed |
| M2 | `unresolved` always returns empty | 2 failed |
| M3 | title derivation not marked derived | 3 failed |
| M4 | resolver iterates unsorted | **survived — equivalent** |
| M5 | self-reference guard removed | 1 failed |
| M6 | unnamed facet key accepted | 1 failed |
| M7 | index append-only (never removes a managed line) | 3 failed |
| M8 | reprojection claims any managed line | 1 failed |
| M9 | derived block emitted unsorted | **survived — equivalent** |
| M9b | *both* sorts removed | 1 failed (after adding the ordering test) |
| M10 | unresolved marker dropped | 2 failed |
| M11 | facet validation skipped | 1 failed |
| M12 | derived title not threaded to the entry | 1 failed |
M4 and M9 are equivalent mutants, not test gaps: `edges` is sorted before it is
returned, and `documents` is populated from `sorted(glob(...))`, so the loop
order cannot reach the output. M9 nonetheless showed the ordering property was
pinned only as a side effect of a byte comparison, so a test that states it
directly was added; M9b confirms it kills the mutation that removes both sorts.
### Cost, with the denominator stated
Measured on a **synthetic** 55-document corpus shaped like the one the bake-off
described — numbered, hierarchical, cross-referencing, versioned. This is a
measurement of **this library's emission**, not a re-run of anyone's bake-off,
and it is synthetic precisely because a private consumer's documents do not
come into this repo.
Facet coverage in the index, 55 documents, same counting query both sides:
| Facet | `DEFAULT` | `STRUCTURED_V1` |
|---|---|---|
| number / parent / status / date / version / references / derived | 0/55 | 55/55 |
| supersedes | 0/55 | 0/55 (corpus has no shared numbers) |
The `supersedes` zero is a property of that corpus, not of the code. Known
positive control, same query and same counter, on a corpus where documents do
share a number: **50/55** — 5 chains of 11 versions give 10 edges each, which
is the arithmetic the code should produce.
Index size for the same 55 documents, by facet key set — the dial a consumer
sets:
| Facet keys | Index chars | vs flat |
|---|---|---|
| none (`DEFAULT`) | 1 540 | 1.00x |
| status, supersedes, derived | 5 122 | 3.33x |
| number, status, supersedes, derived | 6 002 | 3.90x |
| number, parent, status, date, supersedes, derived | 7 817 | 5.08x |
| all eight (`STRUCTURED_V1`) | 9 797 | 6.36x |
## What was not achieved, and what would be needed
Reported as the order asked — what was tried, what happened, what would have to
change — rather than as "not possible".
1. **The cost of closing the gap is 3.3x–6.4x the flat index.** The bake-off's
OKF arm won on cost (6 031 characters against 21 879 for the head-scan it
lost to on traps). Whether a faceted index keeps that advantage **cannot be
answered from here**: it needs their bake-off re-run against
`STRUCTURED_V1`, on their corpus, with their questions. We own the emission
measurement; the arm comparison is theirs. The facet key set is the dial,
and a three-key set is the cheapest configuration that still carries status
and supersession.
2. **A reference to a parent-level number does not resolve to a child.** In the
synthetic corpus, `N130` in the body of one document does not resolve to a
document numbered `N130.2`; it is reported unresolved (`N130?`). This is
honest but incomplete — the fix is a prefix-resolution rule
(`subject` resolves to the unique document whose number starts with it), and
it was left out because "unique" has to be defined against a real corpus
before it can be defended. Deferred, not forgotten.
3. **`OKF_V0_2`'s `verified` / `status` / `stale_after` are still declared and
never written.** The order asked whether the alignment doc's "expressible"
reading is still right. It is, and for the reason already recorded: their
v0.2 values are block lists of mappings, which this library's value model
cannot round-trip until D1's structured reader lands. `STRUCTURED_V1`'s
`status` is a **scalar** facet and is a different key in a different profile;
it is not that field arriving early.
4. **Door B's ownership stamp is still the literal `true`** (D-S6). Open, and
deliberately not decided inside this order.
5. **A faceted index cannot be judged against a directory**
(`entries_match_directory`) in the same call, because `STRUCTURED_V1` does
not set it. Not attempted; no consumer has asked.
6. **No profile is applied to Door C.** Import writes concepts verbatim and
derives nothing, unchanged. Deriving structure for content another sender
wrote would put our inference into their bytes; if that is wanted it needs a
decision about whose claim the frontmatter makes.