Merge commit '6c73d1f7c1'

shared/README.md

@@ -31,6 +31,12 @@ so the only thing that differs is the agent framework itself.
   IR projection + golden suite as the only ground truth, and the budget/provenance requirements.
   The sibling implementation is built from this spec alone, without reverse-engineering the
   reference code.
+- [`ingest-spec.md`](ingest-spec.md) — the **normative ingest specification**, framework-neutral
+  (same guard rule as the method spec): the deterministic ingest step that materializes real
+  data sources as OKF bundles BEFORE the loop — the polymorphic manifest schema (file/CSV, SQL,
+  HTTP as extension point), the credential-reference rule, the verdict-layer reservation, the
+  ingest provenance frontmatter with an explicit timestamp, the index-generation requirement,
+  and the golden-extraction format.
 
 ## Rules
 

shared/ingest-spec.md

@@ -0,0 +1,280 @@
+# Ingest specification — connectors and bundle materialization (framework-neutral)
+
+> **Status:** normative. This document specifies the **ingest layer** both reference
+> implementations build: a deterministic step that connects the framework to real data sources
+> and materializes the extract as an OKF knowledge bundle, which the existing 8-step loop
+> ([method-spec.md](method-spec.md)) then consumes unchanged. It is written so the layer can be
+> implemented **from this spec alone** — without reverse-engineering any existing
+> implementation. The prose is framework-neutral by rule: it never names a concrete agent
+> toolkit or vendor stack, and a guard test keeps it that way.
+>
+> The key words MUST, MUST NOT, SHOULD, and MAY are to be interpreted as in RFC 2119.
+> Requirements are labelled normative; anything marked *(reference)* documents the reference
+> implementation's concrete choice and is informative, not binding — except where a golden
+> extraction (§11) freezes it.
+
+## 1. Scope and conformance
+
+Ingest is an **addition in front of** the loop: nothing in this document changes the method
+spec, the golden suite, or agent behaviour.
+
+- The method spec (§3 Step 1) forbids query-time retrieval against the bundle. The consequence
+  is architecture-defining: **data reaches the model ONLY via OKF bundles.** Connectors
+  therefore live in a deterministic ingest step that runs BEFORE the optimiser and
+  materializes the extract as a bundle — not RAG, not live lookups inside the agent loop, no
+  retrieval tools in the run path.
+- Ingest makes **zero model calls** and is deterministic end to end (§11).
+- **Conformance:** a conforming implementation MUST implement the `file` and `sql` source
+  types (§4) with the materialization contract of §5–§7 and the gates of §8–§9, and MUST
+  reproduce the shared golden extractions (§11) byte for byte. The `http` source type is an
+  OPTIONAL extension point: implementing it (e.g. against a local mock, or via an MCP-based
+  connector) does not require any change to this spec, and NOT implementing it does not break
+  conformance.
+- **Honesty rule (unwaivable, method spec §1):** a machine-generated bundle is labelled as
+  such (`generated: true` plus a manifest reference, §7) everywhere it is presented.
+- **Boundary:** the deploying organisation owns processing purposes and impact assessments;
+  ingest provides only the technical prerequisites (local-only default, provenance, no silent
+  egress).
+
+## 2. Architecture
+
+```
+manifest → connector → mapping → OKF bundle (incl. index generation) → [existing loop, unchanged]
+```
+
+An expert's coupling to a source is one JSON **manifest** (§4). A **connector** executes the
+manifest's extractions against the source. **Materialization** (§5) maps each extraction to an
+OKF concept file with a provenance frontmatter layer (§7) and generates or updates the
+bundle's `index.md` (§6). The loop then reads the bundle exactly as it reads a hand-curated
+one — by navigation, never retrieval.
+
+**Index generation is part of the contract.** A bundle without `index.md` is an error, and
+navigation follows ONLY index cross-links (method spec §3 Step 1) — a generated concept file
+without an index link is unreachable. The materializer therefore creates or updates `index.md`
+(§6); navigability is a load-bearing requirement (§11), not a detail.
+
+## 3. Layer separation in the bundle (unwaivable)
+
+The bundle is not only input — it is also the wiki the promotion gate (method spec §6) lifts
+approved knowledge into. Two rules keep ingest and the learning loop apart:
+
+- **The verdict layer is RESERVED.** A manifest mapping MUST NOT produce `type: verdict`
+  files, and generated filenames MUST NOT fall in the `promoted-verdict-*` namespace (nor be
+  `index.md`, which is managed per §6). The promotion gate is the ONLY path into the verdict
+  layer. Rationale: seeding (method spec §3 Step 1) turns every `type: verdict` file into a
+  store entry with decision default `approved` — an ingest that could write verdict-typed
+  files would inject machine-generated "approved" verdicts around the gate, exactly the
+  self-contamination the gate exists to prevent. This MUST be enforced fail-fast at manifest
+  validation (before any source call) and proven by a load-bearing test (§11).
+- **Ingest owns only its own files.** Re-materialization replaces EXACTLY the files carrying
+  the ingest stamp (`generated: true` plus an `ingest_manifest` reference, §7) and MUST NOT
+  touch curated or promoted files. If a generated filename collides with an existing file that
+  does NOT carry the stamp, materialization MUST fail — never overwrite curated content.
+  Index updating is idempotent and preserves curated links (§6).
+
+## 4. The ingest manifest (the contract)
+
+One JSON file per source coupling. The manifest MUST be schema-validated fail-fast BEFORE any
+source call — a malformed manifest never starts a run (the startup-contract discipline of
+method spec §10). Queries are **configuration, not code**: declarative strings the connector
+interprets; a connector MUST NOT evaluate manifest content as program code.
+
+Top-level fields (all required):
+
+| Field | Meaning |
+|---|---|
+| `manifest_version` | Integer schema version; this spec defines version `1`. |
+| `source` | The source description, polymorphic on `source.type` (below). |
+| `bundle_summary` | Prose used as the index body when the materializer creates a fresh `index.md` (§6). |
+| `extractions` | Non-empty list of extraction descriptions (below). |
+
+**`source` — polymorphic on `type`** (common field: `id`, the source-system identifier
+matching `[a-z0-9][a-z0-9-]*`, stamped as `source_system` in §7):
+
+- `type: "file"` — a local file catalogue. Field `root`: the directory the extraction paths
+  resolve against. Path resolution MUST be boundary-checked against `root`, fail-closed (the
+  OKF path rule) — an extraction can never read outside the catalogue.
+- `type: "sql"` — a SQL database. Field `connection_ref`: the NAME of a runtime-resolved
+  reference (an environment variable) whose value is the connection string or database path.
+- `type: "http"` — a remote endpoint (OPTIONAL extension point, §1). Field `base_url`: the
+  endpoint base; it MUST NOT embed credentials. Optional field `credential_ref`: the NAME of
+  a runtime-resolved secret reference. An MCP-based connector is an extension of this family
+  and MUST honour the same extraction, materialization, and gate contracts.
+
+**Credentials never live in the manifest.** The manifest carries only references by name
+(`connection_ref`, `credential_ref`); the secret is resolved at run time from the
+environment. A manifest is therefore versionable and shareable without secrets.
+
+**`extractions` — each entry** (all fields required unless marked optional):
+
+| Field | Meaning |
+|---|---|
+| `id` | Unique within the manifest, matching `[a-z0-9][a-z0-9-]*`; names the generated file (§5). |
+| `title` | Human-readable title; becomes the `title` frontmatter and the index link label. Single-line. |
+| `query` | The extraction query, polymorphic on the source type (below). |
+| `okf_type` | The generated file's OKF `type`. MUST NOT be `verdict` (case-insensitive; §3). |
+| `max_rows` | Required positive integer cap on the extraction size (§8). |
+
+`query` by source type: for `file`, a relative path (within `root`) to a CSV file whose first
+row is the header; for `sql`, a single read-only SELECT statement (one statement, no data
+modification — the connector SHOULD enforce read-only access, and the query SHOULD carry an
+explicit ORDER BY, since golden conformance requires a stable row order); for `http`, a path
+joined to `base_url`, fetched with GET.
+
+## 5. Materialization (deterministic)
+
+Materialization takes three EXPLICIT inputs: the manifest, the target bundle directory, and
+`ingested_at` — plus the network opt-in flag (§8) when the source type requires it.
+
+- **`ingested_at` is an explicit required argument** — there is NO wall-clock default
+  (mirroring the promotion gate's timestamp rule, method spec §6). Format: ISO-8601 UTC with
+  a `Z` suffix (e.g. `2026-07-03T12:00:00Z`), stamped verbatim. This is what makes golden
+  extractions bit-deterministic.
+- **One concept file per extraction**, named `ingest-{extraction id}.md` (the extraction-id
+  grammar in §4 makes the name path-safe; the `ingest-` prefix keeps the namespace disjoint
+  from `index.md` and `promoted-verdict-*`). Splitting one extraction into per-row files is
+  an extension point, not version 1.
+- **Frontmatter** — exactly these keys, in exactly this order (OKF line-oriented
+  `key: value`; §7 defines the provenance semantics):
+  `type`, `title`, `source_system`, `source_query`, `ingested_at`, `ingest_manifest`,
+  `generated`. All values MUST be single-line; the materializer MUST collapse whitespace runs
+  (including newlines) in `source_query` to single spaces.
+- **Body rendering** — for `file` and `sql`: a markdown table, header row = column names,
+  data rows in source order. Cell values: text verbatim with `\` escaped as `\\`, `|` escaped
+  as `\|`, and newlines replaced by single spaces; integers in plain decimal; non-integral
+  numbers in their shortest round-trip decimal form; SQL NULL as the empty string; any other
+  value type MUST fail (never silent coercion). For `http`: the response body verbatim inside
+  a fenced code block. Files are LF-only and end with exactly one trailing newline.
+- **`ingest_manifest`** is stamped as `{manifest filename stem}@{h}` where `{h}` is the first
+  16 hex characters of the SHA-256 of the manifest file's raw bytes — so every generated file
+  points at the exact manifest version that produced it.
+- **Replacement semantics (§3):** materialization first removes every file in the bundle
+  whose frontmatter carries the ingest stamp, then writes the new set, then updates the index
+  (§6). Version 1 assumes ONE manifest per bundle; multiple manifests feeding one bundle is
+  an extension point.
+
+## 6. Index generation
+
+- If `index.md` is missing, the materializer creates it with `bundle_summary` as the body; if
+  it exists, every line it does not itself manage is preserved byte for byte.
+- Each generated file gets one index cross-link, `- [{title}](ingest-{id}.md)`, appended in
+  manifest extraction order. Linking is **idempotent by target**: a link whose target is
+  already present in the index is never added twice *(reference: the reference
+  implementation's existing index-linking primitive has exactly these semantics)*.
+- On re-materialization, index links whose target is an ingest-owned file removed in this run
+  (§5) MUST be removed; ALL other links — curated and promoted — are preserved verbatim. A
+  promoted verdict's index link therefore survives re-ingest (load-bearing, §11).
+- The link label is the extraction `title` — generated files are context-layer content, so a
+  descriptive label is correct here. (The fixed-neutral-label rule of method spec §6 protects
+  the VERDICT layer and is untouched by this spec.)
+- *(reference limitation)* the index read-modify-write is not atomic — single-process use is
+  assumed, as in the promotion gate.
+
+## 7. Provenance — a separate layer, an unbroken chain
+
+Ingest provenance is its OWN frontmatter layer on generated concept files — NOT an extension
+of the method spec's §9 proposal provenance. §9 links a proposal to a bundle file and text
+span; ingest provenance links the bundle file to the source system. Two separate contracts
+that are never mixed — the same discipline as the two falsifiers.
+
+| Field | Meaning |
+|---|---|
+| `source_system` | The `source.id` from the manifest. |
+| `source_query` | The query that fetched the content (whitespace-collapsed, §5). |
+| `ingested_at` | The explicit timestamp argument, verbatim (§5). |
+| `ingest_manifest` | The manifest reference `{stem}@{hash16}` (§5). |
+| `generated` | Literally `true` — the machine-generated marker (§1 honesty rule). |
+
+- OKF consumers preserve unknown frontmatter fields, so this layer rides through navigation
+  and context rendering unchanged.
+- **Chain integrity:** the chain expert → proposal → bundle file + span → source + query +
+  timestamp holds GIVEN versioned bundles (the OKF premise: curated, version-controlled). A
+  re-materialization is a new bundle version carrying the new stamp, so a run's citations can
+  be tied to the concrete materialization. Bundle versioning is a documented deployer
+  prerequisite — no claim of "unbroken" is made beyond it.
+
+## 8. Security frame
+
+- **Local-only default, no silent egress.** Network sources (`http`, and any other non-local
+  transport) require an EXPLICIT per-run opt-in flag; without the flag the connector MUST
+  refuse fail-fast. The flag is a run argument, never a manifest field — the manifest cannot
+  grant itself network access.
+- **Source calls are logged** (which source, when, row count).
+- **Extraction size caps:** each extraction's `max_rows` (§4) is enforced fail-fast — an
+  extraction exceeding its cap is an ERROR, never a silent truncation.
+- Impact assessments and processing purposes stay with the deployer (§1).
+
+## 9. The HITL gate
+
+Two layers, both required:
+
+1. **Procedural:** ingest NEVER runs automatically — always an explicit operator/expert
+   command with a manifest path. No scheduler, no implicit re-ingest.
+2. **Technical:** fail-fast schema validation before any source call (§4), plus the network
+   opt-in flag (§8).
+
+Richer approval mechanics (a signed `approved_by`, an approval registry) are an extension
+point — not version 1.
+
+## 10. Re-ingest and versioning
+
+Version 1 re-ingest is **full re-materialization of the ingest-owned files** (§5): given the
+same source content, manifest, and `ingested_at`, the result is byte-identical and repeated
+runs are idempotent. Curated and promoted content ALWAYS survives (§3, §6). Incremental
+re-ingest is an extension point.
+
+## 11. Determinism, golden extractions, and load-bearing tests
+
+Connectors are tested ONLY against fixtures and golden extractions — never against live
+sources. Test suites MUST run without credentials and without network access.
+
+**Golden extraction format:** one directory per case, convention
+`examples/ingest-golden-{source type}/`, containing:
+
+| Entry | Meaning |
+|---|---|
+| `manifest.json` | The manifest under test. |
+| `fixture/` | The source content (the file catalogue, the database file, or the mock payloads). |
+| `ingested-at.txt` | The fixed timestamp, one line, §5 format. |
+| `expected-bundle/` | The expected materialized bundle, compared file by file, byte for byte. |
+
+**Load-bearing conformance tests** (a test MUST fail when its seam is detached — the method
+spec §11 regime):
+
+| Seam | The test MUST fail when… |
+|---|---|
+| Provenance stamping | a generated file no longer carries the §7 layer |
+| Navigability | the generated bundle stops being consumable by the UNCHANGED bundle-navigation code, index links included |
+| Verdict reservation | a manifest mapping to `type: verdict` (or the reserved filename namespace) stops being rejected |
+| Re-ingest layer safety | re-materialization over a bundle with a promoted verdict deletes the verdict file or its index link |
+| Golden regression | any byte of a golden extraction's expected bundle diverges |
+| Network gate | an `http` source stops refusing fail-fast without the opt-in flag |
+| Spec integrity | this spec goes missing, names a concrete agent toolkit, or stops documenting a contract field |
+
+## 12. Cross-check table
+
+Every field of the machine-readable contracts, mapped to its normative section (completeness
+is enforced by the spec-integrity test):
+
+| Field | Contract | Section |
+|---|---|---|
+| `manifest_version` | manifest (top level) | §4 |
+| `source` | manifest (top level) | §4 |
+| `bundle_summary` | manifest (top level) | §4, §6 |
+| `extractions` | manifest (top level) | §4 |
+| `type` | manifest source discriminator / OKF frontmatter | §4, §5 |
+| `id` | manifest source and extraction identifiers | §4 |
+| `root` | manifest source (`file`) | §4 |
+| `connection_ref` | manifest source (`sql`) | §4 |
+| `base_url` | manifest source (`http`) | §4 |
+| `credential_ref` | manifest source (`http`) | §4 |
+| `title` | manifest extraction / frontmatter / index label | §4, §5, §6 |
+| `query` | manifest extraction | §4 |
+| `okf_type` | manifest extraction | §4, §3 |
+| `max_rows` | manifest extraction | §4, §8 |
+| `source_system` | provenance frontmatter | §7 |
+| `source_query` | provenance frontmatter | §5, §7 |
+| `ingested_at` | provenance frontmatter / materialization argument | §5, §7 |
+| `ingest_manifest` | provenance frontmatter | §5, §7 |
+| `generated` | provenance frontmatter | §3, §7 |
+| `manifest.json`, `fixture/`, `ingested-at.txt`, `expected-bundle/` | golden extraction case | §11 |