1
0
Fork 0

docs(spec): ingest-spec — land ratified D1 stamp model

Trinn F ratified D1 (form B, commons owner + catalog co-sign). Writes the
ratified decisions into ingest-spec.md (decision record §3 items 1-5, 9):
prefix-not-closed-set + per-file conformance (§5), membership-from-disk/
ordering-from-caller axis (§5), title [/] restriction (§4/§6), door-C
complete-stamp rejection + residual-risk sentence (§3), two load-bearing
§11 seams. Framework-neutral throughout; no primitive named.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUCuNNvnCkayuUDy2WFgeU
This commit is contained in:
Kjell Tore Guttormsen 2026-07-21 09:16:09 +02:00
commit bfa5a9b51a

View file

@ -70,7 +70,19 @@ approved knowledge into. Two rules keep ingest and the learning loop apart:
the ingest stamp (`generated: true` plus an `ingest_manifest` reference, §7) and MUST NOT
touch curated or promoted files. If a generated filename collides with an existing file that
does NOT carry the stamp, materialization MUST fail — never overwrite curated content.
Index updating is idempotent and preserves curated links (§6).
Index updating is idempotent and preserves curated links (§6). The stamp is unforgeable
against **accident**, not against **will**: an operator who hand-copies a generated file —
stamp and all — into curated content makes it indistinguishable from ingest-owned content,
and a later re-materialization may remove it. This residual risk is **documented, not
prevented**; the boundary's honesty rests on curated authoring not forging the stamp (next
bullet) and on operators not hand-copying generated files.
- **No other writer may forge the stamp.** The stamp is the sole mark distinguishing
ingest-owned files from curated ones, so any authoring primitive that materializes a concept
file from **caller-supplied** frontmatter MUST reject a frontmatter carrying the *complete*
ownership stamp — `generated: true` together with an `ingest_manifest` reference — while
permitting either field alone (curated content may legitimately carry a single provenance
field). The check is on the complete stamp, never on the individual field names, so a
legitimate verbatim round-trip is preserved; it is a **validation, never a repair**.
## 4. The ingest manifest (the contract)
@ -110,7 +122,7 @@ environment. A manifest is therefore versionable and shareable without secrets.
| Field | Meaning |
|---|---|
| `id` | Unique within the manifest, matching `[a-z0-9][a-z0-9-]*`; names the generated file (§5). |
| `title` | Human-readable title; becomes the `title` frontmatter and the index link label. Single-line. |
| `title` | Human-readable title; becomes the `title` frontmatter and the index link label. Single-line, and MUST NOT contain `[` or `]` (they would break index-link and navigation parsing, method spec §3 Step 1); `(`/`)` remain permitted. Validated fail-fast at manifest load — rendered verbatim thereafter (§5, §6). |
| `query` | The extraction query, polymorphic on the source type (below). |
| `okf_type` | The generated file's OKF `type`. MUST NOT be `verdict` (case-insensitive; §3). |
| `max_rows` | Required positive integer cap on the extraction size (§8). |
@ -134,11 +146,19 @@ Materialization takes three EXPLICIT inputs: the manifest, the target bundle dir
grammar in §4 makes the name path-safe; the `ingest-` prefix keeps the namespace disjoint
from `index.md` and `promoted-verdict-*`). Splitting one extraction into per-row files is
an extension point, not version 1.
- **Frontmatter**exactly these keys, in exactly this order (OKF line-oriented
`key: value`; §7 defines the provenance semantics):
- **Frontmatter**these seven keys, in this order, form a **mandatory ordered prefix** on a
stamped file (OKF line-oriented `key: value`; §7 defines the provenance semantics):
`type`, `title`, `source_system`, `source_query`, `ingested_at`, `ingest_manifest`,
`generated`. All values MUST be single-line; the materializer MUST collapse whitespace runs
(including newlines) in `source_query` to single spaces.
`generated`. The version-1 materializer emits exactly this prefix and nothing after it, but
the CONTRACT is the prefix, not a closed set: additional frontmatter keys MAY follow it,
unconstrained in presence and order (OKF consumers preserve unknown fields, §7), so a
stamped file stays within the general OKF concept-frontmatter contract. Conformance to the
prefix is a **per-file** property — a non-conforming file is an individual failure and never
invalidates the rest of the bundle; the bundle carries no whole-bundle completeness
property. All values MUST be single-line; `title` is emitted **verbatim** (it is `[`/`]`-free
by §4, so verbatim rendering is safe — the invariant is met by validation, not repair); the
materializer MUST collapse whitespace runs (including newlines) in `source_query` to single
spaces.
- **Body rendering** — for `file` and `sql`: a markdown table, header row = column names,
data rows in source order. Cell values: text verbatim with `\` escaped as `\\`, `|` escaped
as `\|`, and newlines replaced by single spaces; integers in plain decimal; non-integral
@ -152,6 +172,13 @@ Materialization takes three EXPLICIT inputs: the manifest, the target bundle dir
whose frontmatter carries the ingest stamp, then writes the new set, then updates the index
(§6). Version 1 assumes ONE manifest per bundle; multiple manifests feeding one bundle is
an extension point.
- **Membership from disk, ordering from the caller (load-bearing axis).** The set of files a
re-materialization replaces is exactly those the bundle already carries the stamp for — read
from disk (§3), never a caller-supplied file list; a caller can therefore never name a file
for deletion. The ORDER of generated files and their index links (§6) is the manifest's
extraction order — supplied by the caller, never filesystem enumeration order. Splitting the
two keeps re-materialization from becoming a caller-controlled delete while keeping output
deterministic regardless of directory iteration order.
## 6. Index generation
@ -164,9 +191,10 @@ Materialization takes three EXPLICIT inputs: the manifest, the target bundle dir
- On re-materialization, index links whose target is an ingest-owned file removed in this run
(§5) MUST be removed; ALL other links — curated and promoted — are preserved verbatim. A
promoted verdict's index link therefore survives re-ingest (load-bearing, §11).
- The link label is the extraction `title` — generated files are context-layer content, so a
descriptive label is correct here. (The fixed-neutral-label rule of method spec §6 protects
the VERDICT layer and is untouched by this spec.)
- The link label is the extraction `title`, rendered **verbatim** — safe because `title` is
`[`/`]`-free by §4. Generated files are context-layer content, so a descriptive label is
correct here. (The fixed-neutral-label rule of method spec §6 protects the VERDICT layer and
is untouched by this spec.)
- *(reference limitation)* the index read-modify-write is not atomic — single-process use is
assumed, as in the promotion gate.
@ -244,8 +272,10 @@ spec §11 regime):
| Seam | The test MUST fail when… |
|---|---|
| Provenance stamping | a generated file no longer carries the §7 layer |
| Stamp integrity (curated writers) | a caller-supplied frontmatter carrying the complete ownership stamp (`generated: true` with `ingest_manifest`) stops being rejected by the verbatim authoring path (§3) |
| Navigability | the generated bundle stops being consumable by the UNCHANGED bundle-navigation code, index links included |
| Verdict reservation | a manifest mapping to `type: verdict` (or the reserved filename namespace) stops being rejected |
| Title link-safety | a `title` containing `[` or `]` stops being rejected fail-fast at manifest load (§4) |
| Re-ingest layer safety | re-materialization over a bundle with a promoted verdict deletes the verdict file or its index link |
| Golden regression | any byte of a golden extraction's expected bundle diverges |
| Network gate | an `http` source stops refusing fail-fast without the opt-in flag |