diff --git a/shared/README.md b/shared/README.md index 66da623..001bfb9 100644 --- a/shared/README.md +++ b/shared/README.md @@ -31,6 +31,12 @@ so the only thing that differs is the agent framework itself. IR projection + golden suite as the only ground truth, and the budget/provenance requirements. The sibling implementation is built from this spec alone, without reverse-engineering the reference code. +- [`ingest-spec.md`](ingest-spec.md) — the **normative ingest specification**, framework-neutral + (same guard rule as the method spec): the deterministic ingest step that materializes real + data sources as OKF bundles BEFORE the loop — the polymorphic manifest schema (file/CSV, SQL, + HTTP as extension point), the credential-reference rule, the verdict-layer reservation, the + ingest provenance frontmatter with an explicit timestamp, the index-generation requirement, + and the golden-extraction format. ## Rules diff --git a/shared/ingest-spec.md b/shared/ingest-spec.md new file mode 100644 index 0000000..af206ba --- /dev/null +++ b/shared/ingest-spec.md @@ -0,0 +1,280 @@ +# Ingest specification — connectors and bundle materialization (framework-neutral) + +> **Status:** normative. This document specifies the **ingest layer** both reference +> implementations build: a deterministic step that connects the framework to real data sources +> and materializes the extract as an OKF knowledge bundle, which the existing 8-step loop +> ([method-spec.md](method-spec.md)) then consumes unchanged. It is written so the layer can be +> implemented **from this spec alone** — without reverse-engineering any existing +> implementation. The prose is framework-neutral by rule: it never names a concrete agent +> toolkit or vendor stack, and a guard test keeps it that way. +> +> The key words MUST, MUST NOT, SHOULD, and MAY are to be interpreted as in RFC 2119. +> Requirements are labelled normative; anything marked *(reference)* documents the reference +> implementation's concrete choice and is informative, not binding — except where a golden +> extraction (§11) freezes it. + +## 1. Scope and conformance + +Ingest is an **addition in front of** the loop: nothing in this document changes the method +spec, the golden suite, or agent behaviour. + +- The method spec (§3 Step 1) forbids query-time retrieval against the bundle. The consequence + is architecture-defining: **data reaches the model ONLY via OKF bundles.** Connectors + therefore live in a deterministic ingest step that runs BEFORE the optimiser and + materializes the extract as a bundle — not RAG, not live lookups inside the agent loop, no + retrieval tools in the run path. +- Ingest makes **zero model calls** and is deterministic end to end (§11). +- **Conformance:** a conforming implementation MUST implement the `file` and `sql` source + types (§4) with the materialization contract of §5–§7 and the gates of §8–§9, and MUST + reproduce the shared golden extractions (§11) byte for byte. The `http` source type is an + OPTIONAL extension point: implementing it (e.g. against a local mock, or via an MCP-based + connector) does not require any change to this spec, and NOT implementing it does not break + conformance. +- **Honesty rule (unwaivable, method spec §1):** a machine-generated bundle is labelled as + such (`generated: true` plus a manifest reference, §7) everywhere it is presented. +- **Boundary:** the deploying organisation owns processing purposes and impact assessments; + ingest provides only the technical prerequisites (local-only default, provenance, no silent + egress). + +## 2. Architecture + +``` +manifest → connector → mapping → OKF bundle (incl. index generation) → [existing loop, unchanged] +``` + +An expert's coupling to a source is one JSON **manifest** (§4). A **connector** executes the +manifest's extractions against the source. **Materialization** (§5) maps each extraction to an +OKF concept file with a provenance frontmatter layer (§7) and generates or updates the +bundle's `index.md` (§6). The loop then reads the bundle exactly as it reads a hand-curated +one — by navigation, never retrieval. + +**Index generation is part of the contract.** A bundle without `index.md` is an error, and +navigation follows ONLY index cross-links (method spec §3 Step 1) — a generated concept file +without an index link is unreachable. The materializer therefore creates or updates `index.md` +(§6); navigability is a load-bearing requirement (§11), not a detail. + +## 3. Layer separation in the bundle (unwaivable) + +The bundle is not only input — it is also the wiki the promotion gate (method spec §6) lifts +approved knowledge into. Two rules keep ingest and the learning loop apart: + +- **The verdict layer is RESERVED.** A manifest mapping MUST NOT produce `type: verdict` + files, and generated filenames MUST NOT fall in the `promoted-verdict-*` namespace (nor be + `index.md`, which is managed per §6). The promotion gate is the ONLY path into the verdict + layer. Rationale: seeding (method spec §3 Step 1) turns every `type: verdict` file into a + store entry with decision default `approved` — an ingest that could write verdict-typed + files would inject machine-generated "approved" verdicts around the gate, exactly the + self-contamination the gate exists to prevent. This MUST be enforced fail-fast at manifest + validation (before any source call) and proven by a load-bearing test (§11). +- **Ingest owns only its own files.** Re-materialization replaces EXACTLY the files carrying + the ingest stamp (`generated: true` plus an `ingest_manifest` reference, §7) and MUST NOT + touch curated or promoted files. If a generated filename collides with an existing file that + does NOT carry the stamp, materialization MUST fail — never overwrite curated content. + Index updating is idempotent and preserves curated links (§6). + +## 4. The ingest manifest (the contract) + +One JSON file per source coupling. The manifest MUST be schema-validated fail-fast BEFORE any +source call — a malformed manifest never starts a run (the startup-contract discipline of +method spec §10). Queries are **configuration, not code**: declarative strings the connector +interprets; a connector MUST NOT evaluate manifest content as program code. + +Top-level fields (all required): + +| Field | Meaning | +|---|---| +| `manifest_version` | Integer schema version; this spec defines version `1`. | +| `source` | The source description, polymorphic on `source.type` (below). | +| `bundle_summary` | Prose used as the index body when the materializer creates a fresh `index.md` (§6). | +| `extractions` | Non-empty list of extraction descriptions (below). | + +**`source` — polymorphic on `type`** (common field: `id`, the source-system identifier +matching `[a-z0-9][a-z0-9-]*`, stamped as `source_system` in §7): + +- `type: "file"` — a local file catalogue. Field `root`: the directory the extraction paths + resolve against. Path resolution MUST be boundary-checked against `root`, fail-closed (the + OKF path rule) — an extraction can never read outside the catalogue. +- `type: "sql"` — a SQL database. Field `connection_ref`: the NAME of a runtime-resolved + reference (an environment variable) whose value is the connection string or database path. +- `type: "http"` — a remote endpoint (OPTIONAL extension point, §1). Field `base_url`: the + endpoint base; it MUST NOT embed credentials. Optional field `credential_ref`: the NAME of + a runtime-resolved secret reference. An MCP-based connector is an extension of this family + and MUST honour the same extraction, materialization, and gate contracts. + +**Credentials never live in the manifest.** The manifest carries only references by name +(`connection_ref`, `credential_ref`); the secret is resolved at run time from the +environment. A manifest is therefore versionable and shareable without secrets. + +**`extractions` — each entry** (all fields required unless marked optional): + +| Field | Meaning | +|---|---| +| `id` | Unique within the manifest, matching `[a-z0-9][a-z0-9-]*`; names the generated file (§5). | +| `title` | Human-readable title; becomes the `title` frontmatter and the index link label. Single-line. | +| `query` | The extraction query, polymorphic on the source type (below). | +| `okf_type` | The generated file's OKF `type`. MUST NOT be `verdict` (case-insensitive; §3). | +| `max_rows` | Required positive integer cap on the extraction size (§8). | + +`query` by source type: for `file`, a relative path (within `root`) to a CSV file whose first +row is the header; for `sql`, a single read-only SELECT statement (one statement, no data +modification — the connector SHOULD enforce read-only access, and the query SHOULD carry an +explicit ORDER BY, since golden conformance requires a stable row order); for `http`, a path +joined to `base_url`, fetched with GET. + +## 5. Materialization (deterministic) + +Materialization takes three EXPLICIT inputs: the manifest, the target bundle directory, and +`ingested_at` — plus the network opt-in flag (§8) when the source type requires it. + +- **`ingested_at` is an explicit required argument** — there is NO wall-clock default + (mirroring the promotion gate's timestamp rule, method spec §6). Format: ISO-8601 UTC with + a `Z` suffix (e.g. `2026-07-03T12:00:00Z`), stamped verbatim. This is what makes golden + extractions bit-deterministic. +- **One concept file per extraction**, named `ingest-{extraction id}.md` (the extraction-id + grammar in §4 makes the name path-safe; the `ingest-` prefix keeps the namespace disjoint + from `index.md` and `promoted-verdict-*`). Splitting one extraction into per-row files is + an extension point, not version 1. +- **Frontmatter** — exactly these keys, in exactly this order (OKF line-oriented + `key: value`; §7 defines the provenance semantics): + `type`, `title`, `source_system`, `source_query`, `ingested_at`, `ingest_manifest`, + `generated`. All values MUST be single-line; the materializer MUST collapse whitespace runs + (including newlines) in `source_query` to single spaces. +- **Body rendering** — for `file` and `sql`: a markdown table, header row = column names, + data rows in source order. Cell values: text verbatim with `\` escaped as `\\`, `|` escaped + as `\|`, and newlines replaced by single spaces; integers in plain decimal; non-integral + numbers in their shortest round-trip decimal form; SQL NULL as the empty string; any other + value type MUST fail (never silent coercion). For `http`: the response body verbatim inside + a fenced code block. Files are LF-only and end with exactly one trailing newline. +- **`ingest_manifest`** is stamped as `{manifest filename stem}@{h}` where `{h}` is the first + 16 hex characters of the SHA-256 of the manifest file's raw bytes — so every generated file + points at the exact manifest version that produced it. +- **Replacement semantics (§3):** materialization first removes every file in the bundle + whose frontmatter carries the ingest stamp, then writes the new set, then updates the index + (§6). Version 1 assumes ONE manifest per bundle; multiple manifests feeding one bundle is + an extension point. + +## 6. Index generation + +- If `index.md` is missing, the materializer creates it with `bundle_summary` as the body; if + it exists, every line it does not itself manage is preserved byte for byte. +- Each generated file gets one index cross-link, `- [{title}](ingest-{id}.md)`, appended in + manifest extraction order. Linking is **idempotent by target**: a link whose target is + already present in the index is never added twice *(reference: the reference + implementation's existing index-linking primitive has exactly these semantics)*. +- On re-materialization, index links whose target is an ingest-owned file removed in this run + (§5) MUST be removed; ALL other links — curated and promoted — are preserved verbatim. A + promoted verdict's index link therefore survives re-ingest (load-bearing, §11). +- The link label is the extraction `title` — generated files are context-layer content, so a + descriptive label is correct here. (The fixed-neutral-label rule of method spec §6 protects + the VERDICT layer and is untouched by this spec.) +- *(reference limitation)* the index read-modify-write is not atomic — single-process use is + assumed, as in the promotion gate. + +## 7. Provenance — a separate layer, an unbroken chain + +Ingest provenance is its OWN frontmatter layer on generated concept files — NOT an extension +of the method spec's §9 proposal provenance. §9 links a proposal to a bundle file and text +span; ingest provenance links the bundle file to the source system. Two separate contracts +that are never mixed — the same discipline as the two falsifiers. + +| Field | Meaning | +|---|---| +| `source_system` | The `source.id` from the manifest. | +| `source_query` | The query that fetched the content (whitespace-collapsed, §5). | +| `ingested_at` | The explicit timestamp argument, verbatim (§5). | +| `ingest_manifest` | The manifest reference `{stem}@{hash16}` (§5). | +| `generated` | Literally `true` — the machine-generated marker (§1 honesty rule). | + +- OKF consumers preserve unknown frontmatter fields, so this layer rides through navigation + and context rendering unchanged. +- **Chain integrity:** the chain expert → proposal → bundle file + span → source + query + + timestamp holds GIVEN versioned bundles (the OKF premise: curated, version-controlled). A + re-materialization is a new bundle version carrying the new stamp, so a run's citations can + be tied to the concrete materialization. Bundle versioning is a documented deployer + prerequisite — no claim of "unbroken" is made beyond it. + +## 8. Security frame + +- **Local-only default, no silent egress.** Network sources (`http`, and any other non-local + transport) require an EXPLICIT per-run opt-in flag; without the flag the connector MUST + refuse fail-fast. The flag is a run argument, never a manifest field — the manifest cannot + grant itself network access. +- **Source calls are logged** (which source, when, row count). +- **Extraction size caps:** each extraction's `max_rows` (§4) is enforced fail-fast — an + extraction exceeding its cap is an ERROR, never a silent truncation. +- Impact assessments and processing purposes stay with the deployer (§1). + +## 9. The HITL gate + +Two layers, both required: + +1. **Procedural:** ingest NEVER runs automatically — always an explicit operator/expert + command with a manifest path. No scheduler, no implicit re-ingest. +2. **Technical:** fail-fast schema validation before any source call (§4), plus the network + opt-in flag (§8). + +Richer approval mechanics (a signed `approved_by`, an approval registry) are an extension +point — not version 1. + +## 10. Re-ingest and versioning + +Version 1 re-ingest is **full re-materialization of the ingest-owned files** (§5): given the +same source content, manifest, and `ingested_at`, the result is byte-identical and repeated +runs are idempotent. Curated and promoted content ALWAYS survives (§3, §6). Incremental +re-ingest is an extension point. + +## 11. Determinism, golden extractions, and load-bearing tests + +Connectors are tested ONLY against fixtures and golden extractions — never against live +sources. Test suites MUST run without credentials and without network access. + +**Golden extraction format:** one directory per case, convention +`examples/ingest-golden-{source type}/`, containing: + +| Entry | Meaning | +|---|---| +| `manifest.json` | The manifest under test. | +| `fixture/` | The source content (the file catalogue, the database file, or the mock payloads). | +| `ingested-at.txt` | The fixed timestamp, one line, §5 format. | +| `expected-bundle/` | The expected materialized bundle, compared file by file, byte for byte. | + +**Load-bearing conformance tests** (a test MUST fail when its seam is detached — the method +spec §11 regime): + +| Seam | The test MUST fail when… | +|---|---| +| Provenance stamping | a generated file no longer carries the §7 layer | +| Navigability | the generated bundle stops being consumable by the UNCHANGED bundle-navigation code, index links included | +| Verdict reservation | a manifest mapping to `type: verdict` (or the reserved filename namespace) stops being rejected | +| Re-ingest layer safety | re-materialization over a bundle with a promoted verdict deletes the verdict file or its index link | +| Golden regression | any byte of a golden extraction's expected bundle diverges | +| Network gate | an `http` source stops refusing fail-fast without the opt-in flag | +| Spec integrity | this spec goes missing, names a concrete agent toolkit, or stops documenting a contract field | + +## 12. Cross-check table + +Every field of the machine-readable contracts, mapped to its normative section (completeness +is enforced by the spec-integrity test): + +| Field | Contract | Section | +|---|---|---| +| `manifest_version` | manifest (top level) | §4 | +| `source` | manifest (top level) | §4 | +| `bundle_summary` | manifest (top level) | §4, §6 | +| `extractions` | manifest (top level) | §4 | +| `type` | manifest source discriminator / OKF frontmatter | §4, §5 | +| `id` | manifest source and extraction identifiers | §4 | +| `root` | manifest source (`file`) | §4 | +| `connection_ref` | manifest source (`sql`) | §4 | +| `base_url` | manifest source (`http`) | §4 | +| `credential_ref` | manifest source (`http`) | §4 | +| `title` | manifest extraction / frontmatter / index label | §4, §5, §6 | +| `query` | manifest extraction | §4 | +| `okf_type` | manifest extraction | §4, §3 | +| `max_rows` | manifest extraction | §4, §8 | +| `source_system` | provenance frontmatter | §7 | +| `source_query` | provenance frontmatter | §5, §7 | +| `ingested_at` | provenance frontmatter / materialization argument | §5, §7 | +| `ingest_manifest` | provenance frontmatter | §5, §7 | +| `generated` | provenance frontmatter | §3, §7 | +| `manifest.json`, `fixture/`, `ingested-at.txt`, `expected-bundle/` | golden extraction case | §11 |