Squashed 'shared/' changes from db86e15..7aa53fc
7aa53fc docs(spec): ingest-spec.md — normative ingest layer contract (I1) git-subtree-dir: shared git-subtree-split: 7aa53fcf6823bda8c492b53762e3fbfd39ea3058
This commit is contained in:
parent
7916e10c52
commit
ef1a2c57e0
2 changed files with 286 additions and 0 deletions
|
|
@ -31,6 +31,12 @@ so the only thing that differs is the agent framework itself.
|
|||
IR projection + golden suite as the only ground truth, and the budget/provenance requirements.
|
||||
The sibling implementation is built from this spec alone, without reverse-engineering the
|
||||
reference code.
|
||||
- [`ingest-spec.md`](ingest-spec.md) — the **normative ingest specification**, framework-neutral
|
||||
(same guard rule as the method spec): the deterministic ingest step that materializes real
|
||||
data sources as OKF bundles BEFORE the loop — the polymorphic manifest schema (file/CSV, SQL,
|
||||
HTTP as extension point), the credential-reference rule, the verdict-layer reservation, the
|
||||
ingest provenance frontmatter with an explicit timestamp, the index-generation requirement,
|
||||
and the golden-extraction format.
|
||||
|
||||
## Rules
|
||||
|
||||
|
|
|
|||
280
ingest-spec.md
Normal file
280
ingest-spec.md
Normal file
|
|
@ -0,0 +1,280 @@
|
|||
# Ingest specification — connectors and bundle materialization (framework-neutral)
|
||||
|
||||
> **Status:** normative. This document specifies the **ingest layer** both reference
|
||||
> implementations build: a deterministic step that connects the framework to real data sources
|
||||
> and materializes the extract as an OKF knowledge bundle, which the existing 8-step loop
|
||||
> ([method-spec.md](method-spec.md)) then consumes unchanged. It is written so the layer can be
|
||||
> implemented **from this spec alone** — without reverse-engineering any existing
|
||||
> implementation. The prose is framework-neutral by rule: it never names a concrete agent
|
||||
> toolkit or vendor stack, and a guard test keeps it that way.
|
||||
>
|
||||
> The key words MUST, MUST NOT, SHOULD, and MAY are to be interpreted as in RFC 2119.
|
||||
> Requirements are labelled normative; anything marked *(reference)* documents the reference
|
||||
> implementation's concrete choice and is informative, not binding — except where a golden
|
||||
> extraction (§11) freezes it.
|
||||
|
||||
## 1. Scope and conformance
|
||||
|
||||
Ingest is an **addition in front of** the loop: nothing in this document changes the method
|
||||
spec, the golden suite, or agent behaviour.
|
||||
|
||||
- The method spec (§3 Step 1) forbids query-time retrieval against the bundle. The consequence
|
||||
is architecture-defining: **data reaches the model ONLY via OKF bundles.** Connectors
|
||||
therefore live in a deterministic ingest step that runs BEFORE the optimiser and
|
||||
materializes the extract as a bundle — not RAG, not live lookups inside the agent loop, no
|
||||
retrieval tools in the run path.
|
||||
- Ingest makes **zero model calls** and is deterministic end to end (§11).
|
||||
- **Conformance:** a conforming implementation MUST implement the `file` and `sql` source
|
||||
types (§4) with the materialization contract of §5–§7 and the gates of §8–§9, and MUST
|
||||
reproduce the shared golden extractions (§11) byte for byte. The `http` source type is an
|
||||
OPTIONAL extension point: implementing it (e.g. against a local mock, or via an MCP-based
|
||||
connector) does not require any change to this spec, and NOT implementing it does not break
|
||||
conformance.
|
||||
- **Honesty rule (unwaivable, method spec §1):** a machine-generated bundle is labelled as
|
||||
such (`generated: true` plus a manifest reference, §7) everywhere it is presented.
|
||||
- **Boundary:** the deploying organisation owns processing purposes and impact assessments;
|
||||
ingest provides only the technical prerequisites (local-only default, provenance, no silent
|
||||
egress).
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
```
|
||||
manifest → connector → mapping → OKF bundle (incl. index generation) → [existing loop, unchanged]
|
||||
```
|
||||
|
||||
An expert's coupling to a source is one JSON **manifest** (§4). A **connector** executes the
|
||||
manifest's extractions against the source. **Materialization** (§5) maps each extraction to an
|
||||
OKF concept file with a provenance frontmatter layer (§7) and generates or updates the
|
||||
bundle's `index.md` (§6). The loop then reads the bundle exactly as it reads a hand-curated
|
||||
one — by navigation, never retrieval.
|
||||
|
||||
**Index generation is part of the contract.** A bundle without `index.md` is an error, and
|
||||
navigation follows ONLY index cross-links (method spec §3 Step 1) — a generated concept file
|
||||
without an index link is unreachable. The materializer therefore creates or updates `index.md`
|
||||
(§6); navigability is a load-bearing requirement (§11), not a detail.
|
||||
|
||||
## 3. Layer separation in the bundle (unwaivable)
|
||||
|
||||
The bundle is not only input — it is also the wiki the promotion gate (method spec §6) lifts
|
||||
approved knowledge into. Two rules keep ingest and the learning loop apart:
|
||||
|
||||
- **The verdict layer is RESERVED.** A manifest mapping MUST NOT produce `type: verdict`
|
||||
files, and generated filenames MUST NOT fall in the `promoted-verdict-*` namespace (nor be
|
||||
`index.md`, which is managed per §6). The promotion gate is the ONLY path into the verdict
|
||||
layer. Rationale: seeding (method spec §3 Step 1) turns every `type: verdict` file into a
|
||||
store entry with decision default `approved` — an ingest that could write verdict-typed
|
||||
files would inject machine-generated "approved" verdicts around the gate, exactly the
|
||||
self-contamination the gate exists to prevent. This MUST be enforced fail-fast at manifest
|
||||
validation (before any source call) and proven by a load-bearing test (§11).
|
||||
- **Ingest owns only its own files.** Re-materialization replaces EXACTLY the files carrying
|
||||
the ingest stamp (`generated: true` plus an `ingest_manifest` reference, §7) and MUST NOT
|
||||
touch curated or promoted files. If a generated filename collides with an existing file that
|
||||
does NOT carry the stamp, materialization MUST fail — never overwrite curated content.
|
||||
Index updating is idempotent and preserves curated links (§6).
|
||||
|
||||
## 4. The ingest manifest (the contract)
|
||||
|
||||
One JSON file per source coupling. The manifest MUST be schema-validated fail-fast BEFORE any
|
||||
source call — a malformed manifest never starts a run (the startup-contract discipline of
|
||||
method spec §10). Queries are **configuration, not code**: declarative strings the connector
|
||||
interprets; a connector MUST NOT evaluate manifest content as program code.
|
||||
|
||||
Top-level fields (all required):
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `manifest_version` | Integer schema version; this spec defines version `1`. |
|
||||
| `source` | The source description, polymorphic on `source.type` (below). |
|
||||
| `bundle_summary` | Prose used as the index body when the materializer creates a fresh `index.md` (§6). |
|
||||
| `extractions` | Non-empty list of extraction descriptions (below). |
|
||||
|
||||
**`source` — polymorphic on `type`** (common field: `id`, the source-system identifier
|
||||
matching `[a-z0-9][a-z0-9-]*`, stamped as `source_system` in §7):
|
||||
|
||||
- `type: "file"` — a local file catalogue. Field `root`: the directory the extraction paths
|
||||
resolve against. Path resolution MUST be boundary-checked against `root`, fail-closed (the
|
||||
OKF path rule) — an extraction can never read outside the catalogue.
|
||||
- `type: "sql"` — a SQL database. Field `connection_ref`: the NAME of a runtime-resolved
|
||||
reference (an environment variable) whose value is the connection string or database path.
|
||||
- `type: "http"` — a remote endpoint (OPTIONAL extension point, §1). Field `base_url`: the
|
||||
endpoint base; it MUST NOT embed credentials. Optional field `credential_ref`: the NAME of
|
||||
a runtime-resolved secret reference. An MCP-based connector is an extension of this family
|
||||
and MUST honour the same extraction, materialization, and gate contracts.
|
||||
|
||||
**Credentials never live in the manifest.** The manifest carries only references by name
|
||||
(`connection_ref`, `credential_ref`); the secret is resolved at run time from the
|
||||
environment. A manifest is therefore versionable and shareable without secrets.
|
||||
|
||||
**`extractions` — each entry** (all fields required unless marked optional):
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `id` | Unique within the manifest, matching `[a-z0-9][a-z0-9-]*`; names the generated file (§5). |
|
||||
| `title` | Human-readable title; becomes the `title` frontmatter and the index link label. Single-line. |
|
||||
| `query` | The extraction query, polymorphic on the source type (below). |
|
||||
| `okf_type` | The generated file's OKF `type`. MUST NOT be `verdict` (case-insensitive; §3). |
|
||||
| `max_rows` | Required positive integer cap on the extraction size (§8). |
|
||||
|
||||
`query` by source type: for `file`, a relative path (within `root`) to a CSV file whose first
|
||||
row is the header; for `sql`, a single read-only SELECT statement (one statement, no data
|
||||
modification — the connector SHOULD enforce read-only access, and the query SHOULD carry an
|
||||
explicit ORDER BY, since golden conformance requires a stable row order); for `http`, a path
|
||||
joined to `base_url`, fetched with GET.
|
||||
|
||||
## 5. Materialization (deterministic)
|
||||
|
||||
Materialization takes three EXPLICIT inputs: the manifest, the target bundle directory, and
|
||||
`ingested_at` — plus the network opt-in flag (§8) when the source type requires it.
|
||||
|
||||
- **`ingested_at` is an explicit required argument** — there is NO wall-clock default
|
||||
(mirroring the promotion gate's timestamp rule, method spec §6). Format: ISO-8601 UTC with
|
||||
a `Z` suffix (e.g. `2026-07-03T12:00:00Z`), stamped verbatim. This is what makes golden
|
||||
extractions bit-deterministic.
|
||||
- **One concept file per extraction**, named `ingest-{extraction id}.md` (the extraction-id
|
||||
grammar in §4 makes the name path-safe; the `ingest-` prefix keeps the namespace disjoint
|
||||
from `index.md` and `promoted-verdict-*`). Splitting one extraction into per-row files is
|
||||
an extension point, not version 1.
|
||||
- **Frontmatter** — exactly these keys, in exactly this order (OKF line-oriented
|
||||
`key: value`; §7 defines the provenance semantics):
|
||||
`type`, `title`, `source_system`, `source_query`, `ingested_at`, `ingest_manifest`,
|
||||
`generated`. All values MUST be single-line; the materializer MUST collapse whitespace runs
|
||||
(including newlines) in `source_query` to single spaces.
|
||||
- **Body rendering** — for `file` and `sql`: a markdown table, header row = column names,
|
||||
data rows in source order. Cell values: text verbatim with `\` escaped as `\\`, `|` escaped
|
||||
as `\|`, and newlines replaced by single spaces; integers in plain decimal; non-integral
|
||||
numbers in their shortest round-trip decimal form; SQL NULL as the empty string; any other
|
||||
value type MUST fail (never silent coercion). For `http`: the response body verbatim inside
|
||||
a fenced code block. Files are LF-only and end with exactly one trailing newline.
|
||||
- **`ingest_manifest`** is stamped as `{manifest filename stem}@{h}` where `{h}` is the first
|
||||
16 hex characters of the SHA-256 of the manifest file's raw bytes — so every generated file
|
||||
points at the exact manifest version that produced it.
|
||||
- **Replacement semantics (§3):** materialization first removes every file in the bundle
|
||||
whose frontmatter carries the ingest stamp, then writes the new set, then updates the index
|
||||
(§6). Version 1 assumes ONE manifest per bundle; multiple manifests feeding one bundle is
|
||||
an extension point.
|
||||
|
||||
## 6. Index generation
|
||||
|
||||
- If `index.md` is missing, the materializer creates it with `bundle_summary` as the body; if
|
||||
it exists, every line it does not itself manage is preserved byte for byte.
|
||||
- Each generated file gets one index cross-link, `- [{title}](ingest-{id}.md)`, appended in
|
||||
manifest extraction order. Linking is **idempotent by target**: a link whose target is
|
||||
already present in the index is never added twice *(reference: the reference
|
||||
implementation's existing index-linking primitive has exactly these semantics)*.
|
||||
- On re-materialization, index links whose target is an ingest-owned file removed in this run
|
||||
(§5) MUST be removed; ALL other links — curated and promoted — are preserved verbatim. A
|
||||
promoted verdict's index link therefore survives re-ingest (load-bearing, §11).
|
||||
- The link label is the extraction `title` — generated files are context-layer content, so a
|
||||
descriptive label is correct here. (The fixed-neutral-label rule of method spec §6 protects
|
||||
the VERDICT layer and is untouched by this spec.)
|
||||
- *(reference limitation)* the index read-modify-write is not atomic — single-process use is
|
||||
assumed, as in the promotion gate.
|
||||
|
||||
## 7. Provenance — a separate layer, an unbroken chain
|
||||
|
||||
Ingest provenance is its OWN frontmatter layer on generated concept files — NOT an extension
|
||||
of the method spec's §9 proposal provenance. §9 links a proposal to a bundle file and text
|
||||
span; ingest provenance links the bundle file to the source system. Two separate contracts
|
||||
that are never mixed — the same discipline as the two falsifiers.
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `source_system` | The `source.id` from the manifest. |
|
||||
| `source_query` | The query that fetched the content (whitespace-collapsed, §5). |
|
||||
| `ingested_at` | The explicit timestamp argument, verbatim (§5). |
|
||||
| `ingest_manifest` | The manifest reference `{stem}@{hash16}` (§5). |
|
||||
| `generated` | Literally `true` — the machine-generated marker (§1 honesty rule). |
|
||||
|
||||
- OKF consumers preserve unknown frontmatter fields, so this layer rides through navigation
|
||||
and context rendering unchanged.
|
||||
- **Chain integrity:** the chain expert → proposal → bundle file + span → source + query +
|
||||
timestamp holds GIVEN versioned bundles (the OKF premise: curated, version-controlled). A
|
||||
re-materialization is a new bundle version carrying the new stamp, so a run's citations can
|
||||
be tied to the concrete materialization. Bundle versioning is a documented deployer
|
||||
prerequisite — no claim of "unbroken" is made beyond it.
|
||||
|
||||
## 8. Security frame
|
||||
|
||||
- **Local-only default, no silent egress.** Network sources (`http`, and any other non-local
|
||||
transport) require an EXPLICIT per-run opt-in flag; without the flag the connector MUST
|
||||
refuse fail-fast. The flag is a run argument, never a manifest field — the manifest cannot
|
||||
grant itself network access.
|
||||
- **Source calls are logged** (which source, when, row count).
|
||||
- **Extraction size caps:** each extraction's `max_rows` (§4) is enforced fail-fast — an
|
||||
extraction exceeding its cap is an ERROR, never a silent truncation.
|
||||
- Impact assessments and processing purposes stay with the deployer (§1).
|
||||
|
||||
## 9. The HITL gate
|
||||
|
||||
Two layers, both required:
|
||||
|
||||
1. **Procedural:** ingest NEVER runs automatically — always an explicit operator/expert
|
||||
command with a manifest path. No scheduler, no implicit re-ingest.
|
||||
2. **Technical:** fail-fast schema validation before any source call (§4), plus the network
|
||||
opt-in flag (§8).
|
||||
|
||||
Richer approval mechanics (a signed `approved_by`, an approval registry) are an extension
|
||||
point — not version 1.
|
||||
|
||||
## 10. Re-ingest and versioning
|
||||
|
||||
Version 1 re-ingest is **full re-materialization of the ingest-owned files** (§5): given the
|
||||
same source content, manifest, and `ingested_at`, the result is byte-identical and repeated
|
||||
runs are idempotent. Curated and promoted content ALWAYS survives (§3, §6). Incremental
|
||||
re-ingest is an extension point.
|
||||
|
||||
## 11. Determinism, golden extractions, and load-bearing tests
|
||||
|
||||
Connectors are tested ONLY against fixtures and golden extractions — never against live
|
||||
sources. Test suites MUST run without credentials and without network access.
|
||||
|
||||
**Golden extraction format:** one directory per case, convention
|
||||
`examples/ingest-golden-{source type}/`, containing:
|
||||
|
||||
| Entry | Meaning |
|
||||
|---|---|
|
||||
| `manifest.json` | The manifest under test. |
|
||||
| `fixture/` | The source content (the file catalogue, the database file, or the mock payloads). |
|
||||
| `ingested-at.txt` | The fixed timestamp, one line, §5 format. |
|
||||
| `expected-bundle/` | The expected materialized bundle, compared file by file, byte for byte. |
|
||||
|
||||
**Load-bearing conformance tests** (a test MUST fail when its seam is detached — the method
|
||||
spec §11 regime):
|
||||
|
||||
| Seam | The test MUST fail when… |
|
||||
|---|---|
|
||||
| Provenance stamping | a generated file no longer carries the §7 layer |
|
||||
| Navigability | the generated bundle stops being consumable by the UNCHANGED bundle-navigation code, index links included |
|
||||
| Verdict reservation | a manifest mapping to `type: verdict` (or the reserved filename namespace) stops being rejected |
|
||||
| Re-ingest layer safety | re-materialization over a bundle with a promoted verdict deletes the verdict file or its index link |
|
||||
| Golden regression | any byte of a golden extraction's expected bundle diverges |
|
||||
| Network gate | an `http` source stops refusing fail-fast without the opt-in flag |
|
||||
| Spec integrity | this spec goes missing, names a concrete agent toolkit, or stops documenting a contract field |
|
||||
|
||||
## 12. Cross-check table
|
||||
|
||||
Every field of the machine-readable contracts, mapped to its normative section (completeness
|
||||
is enforced by the spec-integrity test):
|
||||
|
||||
| Field | Contract | Section |
|
||||
|---|---|---|
|
||||
| `manifest_version` | manifest (top level) | §4 |
|
||||
| `source` | manifest (top level) | §4 |
|
||||
| `bundle_summary` | manifest (top level) | §4, §6 |
|
||||
| `extractions` | manifest (top level) | §4 |
|
||||
| `type` | manifest source discriminator / OKF frontmatter | §4, §5 |
|
||||
| `id` | manifest source and extraction identifiers | §4 |
|
||||
| `root` | manifest source (`file`) | §4 |
|
||||
| `connection_ref` | manifest source (`sql`) | §4 |
|
||||
| `base_url` | manifest source (`http`) | §4 |
|
||||
| `credential_ref` | manifest source (`http`) | §4 |
|
||||
| `title` | manifest extraction / frontmatter / index label | §4, §5, §6 |
|
||||
| `query` | manifest extraction | §4 |
|
||||
| `okf_type` | manifest extraction | §4, §3 |
|
||||
| `max_rows` | manifest extraction | §4, §8 |
|
||||
| `source_system` | provenance frontmatter | §7 |
|
||||
| `source_query` | provenance frontmatter | §5, §7 |
|
||||
| `ingested_at` | provenance frontmatter / materialization argument | §5, §7 |
|
||||
| `ingest_manifest` | provenance frontmatter | §5, §7 |
|
||||
| `generated` | provenance frontmatter | §3, §7 |
|
||||
| `manifest.json`, `fixture/`, `ingested-at.txt`, `expected-bundle/` | golden extraction case | §11 |
|
||||
Loading…
Add table
Add a link
Reference in a new issue