portfolio-optimiser/shared/ingest-spec.md

# Ingest specification — connectors and bundle materialization (framework-neutral)

> **Status:** normative. This document specifies the **ingest layer** both reference
> implementations build: a deterministic step that connects the framework to real data sources
> and materializes the extract as an OKF knowledge bundle, which the existing 8-step loop
> ([method-spec.md](method-spec.md)) then consumes unchanged. It is written so the layer can be
> implemented **from this spec alone** — without reverse-engineering any existing
> implementation. The prose is framework-neutral by rule: it never names a concrete agent
> toolkit or vendor stack, and a guard test keeps it that way.
>
> The key words MUST, MUST NOT, SHOULD, and MAY are to be interpreted as in RFC 2119.
> Requirements are labelled normative; anything marked *(reference)* documents the reference
> implementation's concrete choice and is informative, not binding — except where a golden
> extraction (§11) freezes it.

## 1. Scope and conformance

Ingest is an **addition in front of** the loop: nothing in this document changes the method
spec, the golden suite, or agent behaviour.

- The method spec (§3 Step 1) forbids query-time retrieval against the bundle. The consequence
  is architecture-defining: **data reaches the model ONLY via OKF bundles.** Connectors
  therefore live in a deterministic ingest step that runs BEFORE the optimiser and
  materializes the extract as a bundle — not RAG, not live lookups inside the agent loop, no
  retrieval tools in the run path.
- Ingest makes **zero model calls** and is deterministic end to end (§11).
- **Conformance:** a conforming implementation MUST implement the `file` and `sql` source
  types (§4) with the materialization contract of §5–§7 and the gates of §8–§9, and MUST
  reproduce the shared golden extractions (§11) byte for byte. The `http` source type is an
  OPTIONAL extension point: implementing it (e.g. against a local mock, or via an MCP-based
  connector) does not require any change to this spec, and NOT implementing it does not break
  conformance.
- **Honesty rule (unwaivable, method spec §1):** a machine-generated bundle is labelled as
  such (`generated: true` plus a manifest reference, §7) everywhere it is presented.
- **Boundary:** the deploying organisation owns processing purposes and impact assessments;
  ingest provides only the technical prerequisites (local-only default, provenance, no silent
  egress).

## 2. Architecture

```
manifest → connector → mapping → OKF bundle (incl. index generation) → [existing loop, unchanged]
```

An expert's coupling to a source is one JSON **manifest** (§4). A **connector** executes the
manifest's extractions against the source. **Materialization** (§5) maps each extraction to an
OKF concept file with a provenance frontmatter layer (§7) and generates or updates the
bundle's `index.md` (§6). The loop then reads the bundle exactly as it reads a hand-curated
one — by navigation, never retrieval.

**Index generation is part of the contract.** A bundle without `index.md` is an error, and
navigation follows ONLY index cross-links (method spec §3 Step 1) — a generated concept file
without an index link is unreachable. The materializer therefore creates or updates `index.md`
(§6); navigability is a load-bearing requirement (§11), not a detail.

## 3. Layer separation in the bundle (unwaivable)

The bundle is not only input — it is also the wiki the promotion gate (method spec §6) lifts
approved knowledge into. Two rules keep ingest and the learning loop apart:

- **The verdict layer is RESERVED.** A manifest mapping MUST NOT produce `type: verdict`
  files, and generated filenames MUST NOT fall in the `promoted-verdict-*` namespace (nor be
  `index.md`, which is managed per §6). The promotion gate is the ONLY path into the verdict
  layer. Rationale: seeding (method spec §3 Step 1) turns every `type: verdict` file into a
  store entry with decision default `approved` — an ingest that could write verdict-typed
  files would inject machine-generated "approved" verdicts around the gate, exactly the
  self-contamination the gate exists to prevent. This MUST be enforced fail-fast at manifest
  validation (before any source call) and proven by a load-bearing test (§11).
- **Ingest owns only its own files.** Re-materialization replaces EXACTLY the files carrying
  the ingest stamp (`generated: true` plus an `ingest_manifest` reference, §7) and MUST NOT
  touch curated or promoted files. If a generated filename collides with an existing file that
  does NOT carry the stamp, materialization MUST fail — never overwrite curated content.
  Index updating is idempotent and preserves curated links (§6).

## 4. The ingest manifest (the contract)

One JSON file per source coupling. The manifest MUST be schema-validated fail-fast BEFORE any
source call — a malformed manifest never starts a run (the startup-contract discipline of
method spec §10). Queries are **configuration, not code**: declarative strings the connector
interprets; a connector MUST NOT evaluate manifest content as program code.

Top-level fields (all required):

| Field | Meaning |
|---|---|
| `manifest_version` | Integer schema version; this spec defines version `1`. |
| `source` | The source description, polymorphic on `source.type` (below). |
| `bundle_summary` | Prose used as the index body when the materializer creates a fresh `index.md` (§6). |
| `extractions` | Non-empty list of extraction descriptions (below). |

**`source` — polymorphic on `type`** (common field: `id`, the source-system identifier
matching `[a-z0-9][a-z0-9-]*`, stamped as `source_system` in §7):

- `type: "file"` — a local file catalogue. Field `root`: the directory the extraction paths
  resolve against. Path resolution MUST be boundary-checked against `root`, fail-closed (the
  OKF path rule) — an extraction can never read outside the catalogue.
- `type: "sql"` — a SQL database. Field `connection_ref`: the NAME of a runtime-resolved
  reference (an environment variable) whose value is the connection string or database path.
- `type: "http"` — a remote endpoint (OPTIONAL extension point, §1). Field `base_url`: the
  endpoint base; it MUST NOT embed credentials. Optional field `credential_ref`: the NAME of
  a runtime-resolved secret reference. An MCP-based connector is an extension of this family
  and MUST honour the same extraction, materialization, and gate contracts.

**Credentials never live in the manifest.** The manifest carries only references by name
(`connection_ref`, `credential_ref`); the secret is resolved at run time from the
environment. A manifest is therefore versionable and shareable without secrets.

**`extractions` — each entry** (all fields required unless marked optional):

| Field | Meaning |
|---|---|
| `id` | Unique within the manifest, matching `[a-z0-9][a-z0-9-]*`; names the generated file (§5). |
| `title` | Human-readable title; becomes the `title` frontmatter and the index link label. Single-line. |
| `query` | The extraction query, polymorphic on the source type (below). |
| `okf_type` | The generated file's OKF `type`. MUST NOT be `verdict` (case-insensitive; §3). |
| `max_rows` | Required positive integer cap on the extraction size (§8). |

`query` by source type: for `file`, a relative path (within `root`) to a CSV file whose first
row is the header; for `sql`, a single read-only SELECT statement (one statement, no data
modification — the connector SHOULD enforce read-only access, and the query SHOULD carry an
explicit ORDER BY, since golden conformance requires a stable row order); for `http`, a path
joined to `base_url`, fetched with GET.

## 5. Materialization (deterministic)

Materialization takes three EXPLICIT inputs: the manifest, the target bundle directory, and
`ingested_at` — plus the network opt-in flag (§8) when the source type requires it.

- **`ingested_at` is an explicit required argument** — there is NO wall-clock default
  (mirroring the promotion gate's timestamp rule, method spec §6). Format: ISO-8601 UTC with
  a `Z` suffix (e.g. `2026-07-03T12:00:00Z`), stamped verbatim. This is what makes golden
  extractions bit-deterministic.
- **One concept file per extraction**, named `ingest-{extraction id}.md` (the extraction-id
  grammar in §4 makes the name path-safe; the `ingest-` prefix keeps the namespace disjoint
  from `index.md` and `promoted-verdict-*`). Splitting one extraction into per-row files is
  an extension point, not version 1.
- **Frontmatter** — exactly these keys, in exactly this order (OKF line-oriented
  `key: value`; §7 defines the provenance semantics):
  `type`, `title`, `source_system`, `source_query`, `ingested_at`, `ingest_manifest`,
  `generated`. All values MUST be single-line; the materializer MUST collapse whitespace runs
  (including newlines) in `source_query` to single spaces.
- **Body rendering** — for `file` and `sql`: a markdown table, header row = column names,
  data rows in source order. Cell values: text verbatim with `\` escaped as `\\`, `|` escaped
  as `\|`, and newlines replaced by single spaces; integers in plain decimal; non-integral
  numbers in their shortest round-trip decimal form; SQL NULL as the empty string; any other
  value type MUST fail (never silent coercion). For `http`: the response body verbatim inside
  a fenced code block. Files are LF-only and end with exactly one trailing newline.
- **`ingest_manifest`** is stamped as `{manifest filename stem}@{h}` where `{h}` is the first
  16 hex characters of the SHA-256 of the manifest file's raw bytes — so every generated file
  points at the exact manifest version that produced it.
- **Replacement semantics (§3):** materialization first removes every file in the bundle
  whose frontmatter carries the ingest stamp, then writes the new set, then updates the index
  (§6). Version 1 assumes ONE manifest per bundle; multiple manifests feeding one bundle is
  an extension point.

## 6. Index generation

- If `index.md` is missing, the materializer creates it with `bundle_summary` as the body; if
  it exists, every line it does not itself manage is preserved byte for byte.
- Each generated file gets one index cross-link, `- [{title}](ingest-{id}.md)`, appended in
  manifest extraction order. Linking is **idempotent by target**: a link whose target is
  already present in the index is never added twice *(reference: the reference
  implementation's existing index-linking primitive has exactly these semantics)*.
- On re-materialization, index links whose target is an ingest-owned file removed in this run
  (§5) MUST be removed; ALL other links — curated and promoted — are preserved verbatim. A
  promoted verdict's index link therefore survives re-ingest (load-bearing, §11).
- The link label is the extraction `title` — generated files are context-layer content, so a
  descriptive label is correct here. (The fixed-neutral-label rule of method spec §6 protects
  the VERDICT layer and is untouched by this spec.)
- *(reference limitation)* the index read-modify-write is not atomic — single-process use is
  assumed, as in the promotion gate.

## 7. Provenance — a separate layer, an unbroken chain

Ingest provenance is its OWN frontmatter layer on generated concept files — NOT an extension
of the method spec's §9 proposal provenance. §9 links a proposal to a bundle file and text
span; ingest provenance links the bundle file to the source system. Two separate contracts
that are never mixed — the same discipline as the two falsifiers.

| Field | Meaning |
|---|---|
| `source_system` | The `source.id` from the manifest. |
| `source_query` | The query that fetched the content (whitespace-collapsed, §5). |
| `ingested_at` | The explicit timestamp argument, verbatim (§5). |
| `ingest_manifest` | The manifest reference `{stem}@{hash16}` (§5). |
| `generated` | Literally `true` — the machine-generated marker (§1 honesty rule). |

- OKF consumers preserve unknown frontmatter fields, so this layer rides through navigation
  and context rendering unchanged.
- **Chain integrity:** the chain expert → proposal → bundle file + span → source + query +
  timestamp holds GIVEN versioned bundles (the OKF premise: curated, version-controlled). A
  re-materialization is a new bundle version carrying the new stamp, so a run's citations can
  be tied to the concrete materialization. Bundle versioning is a documented deployer
  prerequisite — no claim of "unbroken" is made beyond it.

## 8. Security frame

- **Local-only default, no silent egress.** Network sources (`http`, and any other non-local
  transport) require an EXPLICIT per-run opt-in flag; without the flag the connector MUST
  refuse fail-fast. The flag is a run argument, never a manifest field — the manifest cannot
  grant itself network access.
- **Source calls are logged** (which source, when, row count).
- **Extraction size caps:** each extraction's `max_rows` (§4) is enforced fail-fast — an
  extraction exceeding its cap is an ERROR, never a silent truncation.
- Impact assessments and processing purposes stay with the deployer (§1).

## 9. The HITL gate

Two layers, both required:

1. **Procedural:** ingest NEVER runs automatically — always an explicit operator/expert
   command with a manifest path. No scheduler, no implicit re-ingest.
2. **Technical:** fail-fast schema validation before any source call (§4), plus the network
   opt-in flag (§8).

Richer approval mechanics (a signed `approved_by`, an approval registry) are an extension
point — not version 1.

## 10. Re-ingest and versioning

Version 1 re-ingest is **full re-materialization of the ingest-owned files** (§5): given the
same source content, manifest, and `ingested_at`, the result is byte-identical and repeated
runs are idempotent. Curated and promoted content ALWAYS survives (§3, §6). Incremental
re-ingest is an extension point.

## 11. Determinism, golden extractions, and load-bearing tests

Connectors are tested ONLY against fixtures and golden extractions — never against live
sources. Test suites MUST run without credentials and without network access.

**Golden extraction format:** one directory per case, convention
`examples/ingest-golden-{source type}/`, containing:

| Entry | Meaning |
|---|---|
| `manifest.json` | The manifest under test. |
| `fixture/` | The source content (the file catalogue, the database file, or the mock payloads). |
| `ingested-at.txt` | The fixed timestamp, one line, §5 format. |
| `expected-bundle/` | The expected materialized bundle, compared file by file, byte for byte. |

**Load-bearing conformance tests** (a test MUST fail when its seam is detached — the method
spec §11 regime):

| Seam | The test MUST fail when… |
|---|---|
| Provenance stamping | a generated file no longer carries the §7 layer |
| Navigability | the generated bundle stops being consumable by the UNCHANGED bundle-navigation code, index links included |
| Verdict reservation | a manifest mapping to `type: verdict` (or the reserved filename namespace) stops being rejected |
| Re-ingest layer safety | re-materialization over a bundle with a promoted verdict deletes the verdict file or its index link |
| Golden regression | any byte of a golden extraction's expected bundle diverges |
| Network gate | an `http` source stops refusing fail-fast without the opt-in flag |
| Spec integrity | this spec goes missing, names a concrete agent toolkit, or stops documenting a contract field |

## 12. Cross-check table

Every field of the machine-readable contracts, mapped to its normative section (completeness
is enforced by the spec-integrity test):

| Field | Contract | Section |
|---|---|---|
| `manifest_version` | manifest (top level) | §4 |
| `source` | manifest (top level) | §4 |
| `bundle_summary` | manifest (top level) | §4, §6 |
| `extractions` | manifest (top level) | §4 |
| `type` | manifest source discriminator / OKF frontmatter | §4, §5 |
| `id` | manifest source and extraction identifiers | §4 |
| `root` | manifest source (`file`) | §4 |
| `connection_ref` | manifest source (`sql`) | §4 |
| `base_url` | manifest source (`http`) | §4 |
| `credential_ref` | manifest source (`http`) | §4 |
| `title` | manifest extraction / frontmatter / index label | §4, §5, §6 |
| `query` | manifest extraction | §4 |
| `okf_type` | manifest extraction | §4, §3 |
| `max_rows` | manifest extraction | §4, §8 |
| `source_system` | provenance frontmatter | §7 |
| `source_query` | provenance frontmatter | §5, §7 |
| `ingested_at` | provenance frontmatter / materialization argument | §5, §7 |
| `ingest_manifest` | provenance frontmatter | §5, §7 |
| `generated` | provenance frontmatter | §3, §7 |
| `manifest.json`, `fixture/`, `ingested-at.txt`, `expected-bundle/` | golden extraction case | §11 |