docs(spec): ingest-spec — land ratified D1 stamp model
Trinn F ratified D1 (form B, commons owner + catalog co-sign). Writes the ratified decisions into ingest-spec.md (decision record §3 items 1-5, 9): prefix-not-closed-set + per-file conformance (§5), membership-from-disk/ ordering-from-caller axis (§5), title [/] restriction (§4/§6), door-C complete-stamp rejection + residual-risk sentence (§3), two load-bearing §11 seams. Framework-neutral throughout; no primitive named. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUCuNNvnCkayuUDy2WFgeU
This commit is contained in:
parent
5f046abcf4
commit
bfa5a9b51a
1 changed files with 39 additions and 9 deletions
|
|
@ -70,7 +70,19 @@ approved knowledge into. Two rules keep ingest and the learning loop apart:
|
|||
the ingest stamp (`generated: true` plus an `ingest_manifest` reference, §7) and MUST NOT
|
||||
touch curated or promoted files. If a generated filename collides with an existing file that
|
||||
does NOT carry the stamp, materialization MUST fail — never overwrite curated content.
|
||||
Index updating is idempotent and preserves curated links (§6).
|
||||
Index updating is idempotent and preserves curated links (§6). The stamp is unforgeable
|
||||
against **accident**, not against **will**: an operator who hand-copies a generated file —
|
||||
stamp and all — into curated content makes it indistinguishable from ingest-owned content,
|
||||
and a later re-materialization may remove it. This residual risk is **documented, not
|
||||
prevented**; the boundary's honesty rests on curated authoring not forging the stamp (next
|
||||
bullet) and on operators not hand-copying generated files.
|
||||
- **No other writer may forge the stamp.** The stamp is the sole mark distinguishing
|
||||
ingest-owned files from curated ones, so any authoring primitive that materializes a concept
|
||||
file from **caller-supplied** frontmatter MUST reject a frontmatter carrying the *complete*
|
||||
ownership stamp — `generated: true` together with an `ingest_manifest` reference — while
|
||||
permitting either field alone (curated content may legitimately carry a single provenance
|
||||
field). The check is on the complete stamp, never on the individual field names, so a
|
||||
legitimate verbatim round-trip is preserved; it is a **validation, never a repair**.
|
||||
|
||||
## 4. The ingest manifest (the contract)
|
||||
|
||||
|
|
@ -110,7 +122,7 @@ environment. A manifest is therefore versionable and shareable without secrets.
|
|||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `id` | Unique within the manifest, matching `[a-z0-9][a-z0-9-]*`; names the generated file (§5). |
|
||||
| `title` | Human-readable title; becomes the `title` frontmatter and the index link label. Single-line. |
|
||||
| `title` | Human-readable title; becomes the `title` frontmatter and the index link label. Single-line, and MUST NOT contain `[` or `]` (they would break index-link and navigation parsing, method spec §3 Step 1); `(`/`)` remain permitted. Validated fail-fast at manifest load — rendered verbatim thereafter (§5, §6). |
|
||||
| `query` | The extraction query, polymorphic on the source type (below). |
|
||||
| `okf_type` | The generated file's OKF `type`. MUST NOT be `verdict` (case-insensitive; §3). |
|
||||
| `max_rows` | Required positive integer cap on the extraction size (§8). |
|
||||
|
|
@ -134,11 +146,19 @@ Materialization takes three EXPLICIT inputs: the manifest, the target bundle dir
|
|||
grammar in §4 makes the name path-safe; the `ingest-` prefix keeps the namespace disjoint
|
||||
from `index.md` and `promoted-verdict-*`). Splitting one extraction into per-row files is
|
||||
an extension point, not version 1.
|
||||
- **Frontmatter** — exactly these keys, in exactly this order (OKF line-oriented
|
||||
`key: value`; §7 defines the provenance semantics):
|
||||
- **Frontmatter** — these seven keys, in this order, form a **mandatory ordered prefix** on a
|
||||
stamped file (OKF line-oriented `key: value`; §7 defines the provenance semantics):
|
||||
`type`, `title`, `source_system`, `source_query`, `ingested_at`, `ingest_manifest`,
|
||||
`generated`. All values MUST be single-line; the materializer MUST collapse whitespace runs
|
||||
(including newlines) in `source_query` to single spaces.
|
||||
`generated`. The version-1 materializer emits exactly this prefix and nothing after it, but
|
||||
the CONTRACT is the prefix, not a closed set: additional frontmatter keys MAY follow it,
|
||||
unconstrained in presence and order (OKF consumers preserve unknown fields, §7), so a
|
||||
stamped file stays within the general OKF concept-frontmatter contract. Conformance to the
|
||||
prefix is a **per-file** property — a non-conforming file is an individual failure and never
|
||||
invalidates the rest of the bundle; the bundle carries no whole-bundle completeness
|
||||
property. All values MUST be single-line; `title` is emitted **verbatim** (it is `[`/`]`-free
|
||||
by §4, so verbatim rendering is safe — the invariant is met by validation, not repair); the
|
||||
materializer MUST collapse whitespace runs (including newlines) in `source_query` to single
|
||||
spaces.
|
||||
- **Body rendering** — for `file` and `sql`: a markdown table, header row = column names,
|
||||
data rows in source order. Cell values: text verbatim with `\` escaped as `\\`, `|` escaped
|
||||
as `\|`, and newlines replaced by single spaces; integers in plain decimal; non-integral
|
||||
|
|
@ -152,6 +172,13 @@ Materialization takes three EXPLICIT inputs: the manifest, the target bundle dir
|
|||
whose frontmatter carries the ingest stamp, then writes the new set, then updates the index
|
||||
(§6). Version 1 assumes ONE manifest per bundle; multiple manifests feeding one bundle is
|
||||
an extension point.
|
||||
- **Membership from disk, ordering from the caller (load-bearing axis).** The set of files a
|
||||
re-materialization replaces is exactly those the bundle already carries the stamp for — read
|
||||
from disk (§3), never a caller-supplied file list; a caller can therefore never name a file
|
||||
for deletion. The ORDER of generated files and their index links (§6) is the manifest's
|
||||
extraction order — supplied by the caller, never filesystem enumeration order. Splitting the
|
||||
two keeps re-materialization from becoming a caller-controlled delete while keeping output
|
||||
deterministic regardless of directory iteration order.
|
||||
|
||||
## 6. Index generation
|
||||
|
||||
|
|
@ -164,9 +191,10 @@ Materialization takes three EXPLICIT inputs: the manifest, the target bundle dir
|
|||
- On re-materialization, index links whose target is an ingest-owned file removed in this run
|
||||
(§5) MUST be removed; ALL other links — curated and promoted — are preserved verbatim. A
|
||||
promoted verdict's index link therefore survives re-ingest (load-bearing, §11).
|
||||
- The link label is the extraction `title` — generated files are context-layer content, so a
|
||||
descriptive label is correct here. (The fixed-neutral-label rule of method spec §6 protects
|
||||
the VERDICT layer and is untouched by this spec.)
|
||||
- The link label is the extraction `title`, rendered **verbatim** — safe because `title` is
|
||||
`[`/`]`-free by §4. Generated files are context-layer content, so a descriptive label is
|
||||
correct here. (The fixed-neutral-label rule of method spec §6 protects the VERDICT layer and
|
||||
is untouched by this spec.)
|
||||
- *(reference limitation)* the index read-modify-write is not atomic — single-process use is
|
||||
assumed, as in the promotion gate.
|
||||
|
||||
|
|
@ -244,8 +272,10 @@ spec §11 regime):
|
|||
| Seam | The test MUST fail when… |
|
||||
|---|---|
|
||||
| Provenance stamping | a generated file no longer carries the §7 layer |
|
||||
| Stamp integrity (curated writers) | a caller-supplied frontmatter carrying the complete ownership stamp (`generated: true` with `ingest_manifest`) stops being rejected by the verbatim authoring path (§3) |
|
||||
| Navigability | the generated bundle stops being consumable by the UNCHANGED bundle-navigation code, index links included |
|
||||
| Verdict reservation | a manifest mapping to `type: verdict` (or the reserved filename namespace) stops being rejected |
|
||||
| Title link-safety | a `title` containing `[` or `]` stops being rejected fail-fast at manifest load (§4) |
|
||||
| Re-ingest layer safety | re-materialization over a bundle with a promoted verdict deletes the verdict file or its index link |
|
||||
| Golden regression | any byte of a golden extraction's expected bundle diverges |
|
||||
| Network gate | an `http` source stops refusing fail-fast without the opt-in flag |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue