1
0
Fork 0

docs(spec): ingest-spec — land ratified D1 stamp model

Trinn F ratified D1 (form B, commons owner + catalog co-sign). Writes the
ratified decisions into ingest-spec.md (decision record §3 items 1-5, 9):
prefix-not-closed-set + per-file conformance (§5), membership-from-disk/
ordering-from-caller axis (§5), title [/] restriction (§4/§6), door-C
complete-stamp rejection + residual-risk sentence (§3), two load-bearing
§11 seams. Framework-neutral throughout; no primitive named.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUCuNNvnCkayuUDy2WFgeU
This commit is contained in:
Kjell Tore Guttormsen 2026-07-21 09:16:09 +02:00
commit bfa5a9b51a

View file

@ -70,7 +70,19 @@ approved knowledge into. Two rules keep ingest and the learning loop apart:
the ingest stamp (`generated: true` plus an `ingest_manifest` reference, §7) and MUST NOT the ingest stamp (`generated: true` plus an `ingest_manifest` reference, §7) and MUST NOT
touch curated or promoted files. If a generated filename collides with an existing file that touch curated or promoted files. If a generated filename collides with an existing file that
does NOT carry the stamp, materialization MUST fail — never overwrite curated content. does NOT carry the stamp, materialization MUST fail — never overwrite curated content.
Index updating is idempotent and preserves curated links (§6). Index updating is idempotent and preserves curated links (§6). The stamp is unforgeable
against **accident**, not against **will**: an operator who hand-copies a generated file —
stamp and all — into curated content makes it indistinguishable from ingest-owned content,
and a later re-materialization may remove it. This residual risk is **documented, not
prevented**; the boundary's honesty rests on curated authoring not forging the stamp (next
bullet) and on operators not hand-copying generated files.
- **No other writer may forge the stamp.** The stamp is the sole mark distinguishing
ingest-owned files from curated ones, so any authoring primitive that materializes a concept
file from **caller-supplied** frontmatter MUST reject a frontmatter carrying the *complete*
ownership stamp — `generated: true` together with an `ingest_manifest` reference — while
permitting either field alone (curated content may legitimately carry a single provenance
field). The check is on the complete stamp, never on the individual field names, so a
legitimate verbatim round-trip is preserved; it is a **validation, never a repair**.
## 4. The ingest manifest (the contract) ## 4. The ingest manifest (the contract)
@ -110,7 +122,7 @@ environment. A manifest is therefore versionable and shareable without secrets.
| Field | Meaning | | Field | Meaning |
|---|---| |---|---|
| `id` | Unique within the manifest, matching `[a-z0-9][a-z0-9-]*`; names the generated file (§5). | | `id` | Unique within the manifest, matching `[a-z0-9][a-z0-9-]*`; names the generated file (§5). |
| `title` | Human-readable title; becomes the `title` frontmatter and the index link label. Single-line. | | `title` | Human-readable title; becomes the `title` frontmatter and the index link label. Single-line, and MUST NOT contain `[` or `]` (they would break index-link and navigation parsing, method spec §3 Step 1); `(`/`)` remain permitted. Validated fail-fast at manifest load — rendered verbatim thereafter (§5, §6). |
| `query` | The extraction query, polymorphic on the source type (below). | | `query` | The extraction query, polymorphic on the source type (below). |
| `okf_type` | The generated file's OKF `type`. MUST NOT be `verdict` (case-insensitive; §3). | | `okf_type` | The generated file's OKF `type`. MUST NOT be `verdict` (case-insensitive; §3). |
| `max_rows` | Required positive integer cap on the extraction size (§8). | | `max_rows` | Required positive integer cap on the extraction size (§8). |
@ -134,11 +146,19 @@ Materialization takes three EXPLICIT inputs: the manifest, the target bundle dir
grammar in §4 makes the name path-safe; the `ingest-` prefix keeps the namespace disjoint grammar in §4 makes the name path-safe; the `ingest-` prefix keeps the namespace disjoint
from `index.md` and `promoted-verdict-*`). Splitting one extraction into per-row files is from `index.md` and `promoted-verdict-*`). Splitting one extraction into per-row files is
an extension point, not version 1. an extension point, not version 1.
- **Frontmatter**exactly these keys, in exactly this order (OKF line-oriented - **Frontmatter**these seven keys, in this order, form a **mandatory ordered prefix** on a
`key: value`; §7 defines the provenance semantics): stamped file (OKF line-oriented `key: value`; §7 defines the provenance semantics):
`type`, `title`, `source_system`, `source_query`, `ingested_at`, `ingest_manifest`, `type`, `title`, `source_system`, `source_query`, `ingested_at`, `ingest_manifest`,
`generated`. All values MUST be single-line; the materializer MUST collapse whitespace runs `generated`. The version-1 materializer emits exactly this prefix and nothing after it, but
(including newlines) in `source_query` to single spaces. the CONTRACT is the prefix, not a closed set: additional frontmatter keys MAY follow it,
unconstrained in presence and order (OKF consumers preserve unknown fields, §7), so a
stamped file stays within the general OKF concept-frontmatter contract. Conformance to the
prefix is a **per-file** property — a non-conforming file is an individual failure and never
invalidates the rest of the bundle; the bundle carries no whole-bundle completeness
property. All values MUST be single-line; `title` is emitted **verbatim** (it is `[`/`]`-free
by §4, so verbatim rendering is safe — the invariant is met by validation, not repair); the
materializer MUST collapse whitespace runs (including newlines) in `source_query` to single
spaces.
- **Body rendering** — for `file` and `sql`: a markdown table, header row = column names, - **Body rendering** — for `file` and `sql`: a markdown table, header row = column names,
data rows in source order. Cell values: text verbatim with `\` escaped as `\\`, `|` escaped data rows in source order. Cell values: text verbatim with `\` escaped as `\\`, `|` escaped
as `\|`, and newlines replaced by single spaces; integers in plain decimal; non-integral as `\|`, and newlines replaced by single spaces; integers in plain decimal; non-integral
@ -152,6 +172,13 @@ Materialization takes three EXPLICIT inputs: the manifest, the target bundle dir
whose frontmatter carries the ingest stamp, then writes the new set, then updates the index whose frontmatter carries the ingest stamp, then writes the new set, then updates the index
(§6). Version 1 assumes ONE manifest per bundle; multiple manifests feeding one bundle is (§6). Version 1 assumes ONE manifest per bundle; multiple manifests feeding one bundle is
an extension point. an extension point.
- **Membership from disk, ordering from the caller (load-bearing axis).** The set of files a
re-materialization replaces is exactly those the bundle already carries the stamp for — read
from disk (§3), never a caller-supplied file list; a caller can therefore never name a file
for deletion. The ORDER of generated files and their index links (§6) is the manifest's
extraction order — supplied by the caller, never filesystem enumeration order. Splitting the
two keeps re-materialization from becoming a caller-controlled delete while keeping output
deterministic regardless of directory iteration order.
## 6. Index generation ## 6. Index generation
@ -164,9 +191,10 @@ Materialization takes three EXPLICIT inputs: the manifest, the target bundle dir
- On re-materialization, index links whose target is an ingest-owned file removed in this run - On re-materialization, index links whose target is an ingest-owned file removed in this run
(§5) MUST be removed; ALL other links — curated and promoted — are preserved verbatim. A (§5) MUST be removed; ALL other links — curated and promoted — are preserved verbatim. A
promoted verdict's index link therefore survives re-ingest (load-bearing, §11). promoted verdict's index link therefore survives re-ingest (load-bearing, §11).
- The link label is the extraction `title` — generated files are context-layer content, so a - The link label is the extraction `title`, rendered **verbatim** — safe because `title` is
descriptive label is correct here. (The fixed-neutral-label rule of method spec §6 protects `[`/`]`-free by §4. Generated files are context-layer content, so a descriptive label is
the VERDICT layer and is untouched by this spec.) correct here. (The fixed-neutral-label rule of method spec §6 protects the VERDICT layer and
is untouched by this spec.)
- *(reference limitation)* the index read-modify-write is not atomic — single-process use is - *(reference limitation)* the index read-modify-write is not atomic — single-process use is
assumed, as in the promotion gate. assumed, as in the promotion gate.
@ -244,8 +272,10 @@ spec §11 regime):
| Seam | The test MUST fail when… | | Seam | The test MUST fail when… |
|---|---| |---|---|
| Provenance stamping | a generated file no longer carries the §7 layer | | Provenance stamping | a generated file no longer carries the §7 layer |
| Stamp integrity (curated writers) | a caller-supplied frontmatter carrying the complete ownership stamp (`generated: true` with `ingest_manifest`) stops being rejected by the verbatim authoring path (§3) |
| Navigability | the generated bundle stops being consumable by the UNCHANGED bundle-navigation code, index links included | | Navigability | the generated bundle stops being consumable by the UNCHANGED bundle-navigation code, index links included |
| Verdict reservation | a manifest mapping to `type: verdict` (or the reserved filename namespace) stops being rejected | | Verdict reservation | a manifest mapping to `type: verdict` (or the reserved filename namespace) stops being rejected |
| Title link-safety | a `title` containing `[` or `]` stops being rejected fail-fast at manifest load (§4) |
| Re-ingest layer safety | re-materialization over a bundle with a promoted verdict deletes the verdict file or its index link | | Re-ingest layer safety | re-materialization over a bundle with a promoted verdict deletes the verdict file or its index link |
| Golden regression | any byte of a golden extraction's expected bundle diverges | | Golden regression | any byte of a golden extraction's expected bundle diverges |
| Network gate | an `http` source stops refusing fail-fast without the opt-in flag | | Network gate | an `http` source stops refusing fail-fast without the opt-in flag |