portfolio-optimiser/shared/ingest-spec.md

Ingest specification — connectors and bundle materialization (framework-neutral)

Status: normative. This document specifies the ingest layer both reference implementations build: a deterministic step that connects the framework to real data sources and materializes the extract as an OKF knowledge bundle, which the existing 8-step loop (method-spec.md) then consumes unchanged. It is written so the layer can be implemented from this spec alone — without reverse-engineering any existing implementation. The prose is framework-neutral by rule: it never names a concrete agent toolkit or vendor stack, and a guard test keeps it that way.

The key words MUST, MUST NOT, SHOULD, and MAY are to be interpreted as in RFC 2119. Requirements are labelled normative; anything marked (reference) documents the reference implementation's concrete choice and is informative, not binding — except where a golden extraction (§11) freezes it.

1. Scope and conformance

Ingest is an addition in front of the loop: nothing in this document changes the method spec, the golden suite, or agent behaviour.

• The method spec (§3 Step 1) forbids query-time retrieval against the bundle. The consequence is architecture-defining: data reaches the model ONLY via OKF bundles. Connectors therefore live in a deterministic ingest step that runs BEFORE the optimiser and materializes the extract as a bundle — not RAG, not live lookups inside the agent loop, no retrieval tools in the run path.
• Ingest makes zero model calls and is deterministic end to end (§11).
• Conformance: a conforming implementation MUST implement the file and sql source types (§4) with the materialization contract of §5–§7 and the gates of §8–§9, and MUST reproduce the shared golden extractions (§11) byte for byte. The http source type is an OPTIONAL extension point: implementing it (e.g. against a local mock, or via an MCP-based connector) does not require any change to this spec, and NOT implementing it does not break conformance.
• Honesty rule (unwaivable, method spec §1): a machine-generated bundle is labelled as such (generated: true plus a manifest reference, §7) everywhere it is presented.
• Boundary: the deploying organisation owns processing purposes and impact assessments; ingest provides only the technical prerequisites (local-only default, provenance, no silent egress).

2. Architecture

manifest → connector → mapping → OKF bundle (incl. index generation) → [existing loop, unchanged]

An expert's coupling to a source is one JSON manifest (§4). A connector executes the manifest's extractions against the source. Materialization (§5) maps each extraction to an OKF concept file with a provenance frontmatter layer (§7) and generates or updates the bundle's index.md (§6). The loop then reads the bundle exactly as it reads a hand-curated one — by navigation, never retrieval.

Index generation is part of the contract. A bundle without index.md is an error, and navigation follows ONLY index cross-links (method spec §3 Step 1) — a generated concept file without an index link is unreachable. The materializer therefore creates or updates index.md (§6); navigability is a load-bearing requirement (§11), not a detail.

3. Layer separation in the bundle (unwaivable)

The bundle is not only input — it is also the wiki the promotion gate (method spec §6) lifts approved knowledge into. Two rules keep ingest and the learning loop apart:

• The verdict layer is RESERVED. A manifest mapping MUST NOT produce type: verdict files, and generated filenames MUST NOT fall in the promoted-verdict-* namespace (nor be index.md, which is managed per §6). The promotion gate is the ONLY path into the verdict layer. Rationale: seeding (method spec §3 Step 1) turns every type: verdict file into a store entry with decision default approved — an ingest that could write verdict-typed files would inject machine-generated "approved" verdicts around the gate, exactly the self-contamination the gate exists to prevent. This MUST be enforced fail-fast at manifest validation (before any source call) and proven by a load-bearing test (§11).
• Ingest owns only its own files. Re-materialization replaces EXACTLY the files carrying the ingest stamp (generated: true plus an ingest_manifest reference, §7) and MUST NOT touch curated or promoted files. If a generated filename collides with an existing file that does NOT carry the stamp, materialization MUST fail — never overwrite curated content. Index updating is idempotent and preserves curated links (§6).

4. The ingest manifest (the contract)

One JSON file per source coupling. The manifest MUST be schema-validated fail-fast BEFORE any source call — a malformed manifest never starts a run (the startup-contract discipline of method spec §10). Queries are configuration, not code: declarative strings the connector interprets; a connector MUST NOT evaluate manifest content as program code.

Top-level fields (all required):

Field	Meaning
manifest_version	Integer schema version; this spec defines version 1.
source	The source description, polymorphic on source.type (below).
bundle_summary	Prose used as the index body when the materializer creates a fresh index.md (§6).
extractions	Non-empty list of extraction descriptions (below).

source — polymorphic on type (common field: id, the source-system identifier matching [a-z0-9][a-z0-9-]*, stamped as source_system in §7):

• type: "file" — a local file catalogue. Field root: the directory the extraction paths resolve against. Path resolution MUST be boundary-checked against root, fail-closed (the OKF path rule) — an extraction can never read outside the catalogue.
• type: "sql" — a SQL database. Field connection_ref: the NAME of a runtime-resolved reference (an environment variable) whose value is the connection string or database path.
• type: "http" — a remote endpoint (OPTIONAL extension point, §1). Field base_url: the endpoint base; it MUST NOT embed credentials. Optional field credential_ref: the NAME of a runtime-resolved secret reference. An MCP-based connector is an extension of this family and MUST honour the same extraction, materialization, and gate contracts.

Credentials never live in the manifest. The manifest carries only references by name (connection_ref, credential_ref); the secret is resolved at run time from the environment. A manifest is therefore versionable and shareable without secrets.

extractions — each entry (all fields required unless marked optional):

Field	Meaning
id	Unique within the manifest, matching [a-z0-9][a-z0-9-]*; names the generated file (§5).
title	Human-readable title; becomes the title frontmatter and the index link label. Single-line.
query	The extraction query, polymorphic on the source type (below).
okf_type	The generated file's OKF type. MUST NOT be verdict (case-insensitive; §3).
max_rows	Required positive integer cap on the extraction size (§8).

query by source type: for file, a relative path (within root) to a CSV file whose first row is the header; for sql, a single read-only SELECT statement (one statement, no data modification — the connector SHOULD enforce read-only access, and the query SHOULD carry an explicit ORDER BY, since golden conformance requires a stable row order); for http, a path joined to base_url, fetched with GET.

5. Materialization (deterministic)

Materialization takes three EXPLICIT inputs: the manifest, the target bundle directory, and ingested_at — plus the network opt-in flag (§8) when the source type requires it.

• ingested_at is an explicit required argument — there is NO wall-clock default (mirroring the promotion gate's timestamp rule, method spec §6). Format: ISO-8601 UTC with a Z suffix (e.g. 2026-07-03T12:00:00Z), stamped verbatim. This is what makes golden extractions bit-deterministic.
• One concept file per extraction, named ingest-{extraction id}.md (the extraction-id grammar in §4 makes the name path-safe; the ingest- prefix keeps the namespace disjoint from index.md and promoted-verdict-*). Splitting one extraction into per-row files is an extension point, not version 1.
• Frontmatter — exactly these keys, in exactly this order (OKF line-oriented key: value; §7 defines the provenance semantics): type, title, source_system, source_query, ingested_at, ingest_manifest, generated. All values MUST be single-line; the materializer MUST collapse whitespace runs (including newlines) in source_query to single spaces.
• Body rendering — for file and sql: a markdown table, header row = column names, data rows in source order. Cell values: text verbatim with \ escaped as \\, | escaped as \|, and newlines replaced by single spaces; integers in plain decimal; non-integral numbers in their shortest round-trip decimal form; SQL NULL as the empty string; any other value type MUST fail (never silent coercion). For http: the response body verbatim inside a fenced code block. Files are LF-only and end with exactly one trailing newline.
• ingest_manifest is stamped as {manifest filename stem}@{h} where {h} is the first 16 hex characters of the SHA-256 of the manifest file's raw bytes — so every generated file points at the exact manifest version that produced it.
• Replacement semantics (§3): materialization first removes every file in the bundle whose frontmatter carries the ingest stamp, then writes the new set, then updates the index (§6). Version 1 assumes ONE manifest per bundle; multiple manifests feeding one bundle is an extension point.

6. Index generation

• If index.md is missing, the materializer creates it with bundle_summary as the body; if it exists, every line it does not itself manage is preserved byte for byte.
• Each generated file gets one index cross-link, - [{title}](ingest-{id}.md), appended in manifest extraction order. Linking is idempotent by target: a link whose target is already present in the index is never added twice (reference: the reference implementation's existing index-linking primitive has exactly these semantics).
• On re-materialization, index links whose target is an ingest-owned file removed in this run (§5) MUST be removed; ALL other links — curated and promoted — are preserved verbatim. A promoted verdict's index link therefore survives re-ingest (load-bearing, §11).
• The link label is the extraction title — generated files are context-layer content, so a descriptive label is correct here. (The fixed-neutral-label rule of method spec §6 protects the VERDICT layer and is untouched by this spec.)
• (reference limitation) the index read-modify-write is not atomic — single-process use is assumed, as in the promotion gate.

7. Provenance — a separate layer, an unbroken chain

Ingest provenance is its OWN frontmatter layer on generated concept files — NOT an extension of the method spec's §9 proposal provenance. §9 links a proposal to a bundle file and text span; ingest provenance links the bundle file to the source system. Two separate contracts that are never mixed — the same discipline as the two falsifiers.

Field	Meaning
source_system	The source.id from the manifest.
source_query	The query that fetched the content (whitespace-collapsed, §5).
ingested_at	The explicit timestamp argument, verbatim (§5).
ingest_manifest	The manifest reference {stem}@{hash16} (§5).
generated	Literally true — the machine-generated marker (§1 honesty rule).

• OKF consumers preserve unknown frontmatter fields, so this layer rides through navigation and context rendering unchanged.
• Chain integrity: the chain expert → proposal → bundle file + span → source + query + timestamp holds GIVEN versioned bundles (the OKF premise: curated, version-controlled). A re-materialization is a new bundle version carrying the new stamp, so a run's citations can be tied to the concrete materialization. Bundle versioning is a documented deployer prerequisite — no claim of "unbroken" is made beyond it.

8. Security frame

• Local-only default, no silent egress. Network sources (http, and any other non-local transport) require an EXPLICIT per-run opt-in flag; without the flag the connector MUST refuse fail-fast. The flag is a run argument, never a manifest field — the manifest cannot grant itself network access.
• Source calls are logged (which source, when, row count).
• Extraction size caps: each extraction's max_rows (§4) is enforced fail-fast — an extraction exceeding its cap is an ERROR, never a silent truncation.
• Impact assessments and processing purposes stay with the deployer (§1).

9. The HITL gate

Two layers, both required:

1. Procedural: ingest NEVER runs automatically — always an explicit operator/expert command with a manifest path. No scheduler, no implicit re-ingest.
2. Technical: fail-fast schema validation before any source call (§4), plus the network opt-in flag (§8).

Richer approval mechanics (a signed approved_by, an approval registry) are an extension point — not version 1.

10. Re-ingest and versioning

Version 1 re-ingest is full re-materialization of the ingest-owned files (§5): given the same source content, manifest, and ingested_at, the result is byte-identical and repeated runs are idempotent. Curated and promoted content ALWAYS survives (§3, §6). Incremental re-ingest is an extension point.

11. Determinism, golden extractions, and load-bearing tests

Connectors are tested ONLY against fixtures and golden extractions — never against live sources. Test suites MUST run without credentials and without network access.

Golden extraction format: one directory per case, convention examples/ingest-golden-{source type}/, containing:

Entry	Meaning
manifest.json	The manifest under test.
fixture/	The source content (the file catalogue, the database file, or the mock payloads).
ingested-at.txt	The fixed timestamp, one line, §5 format.
expected-bundle/	The expected materialized bundle, compared file by file, byte for byte.

Load-bearing conformance tests (a test MUST fail when its seam is detached — the method spec §11 regime):

Seam	The test MUST fail when…
Provenance stamping	a generated file no longer carries the §7 layer
Navigability	the generated bundle stops being consumable by the UNCHANGED bundle-navigation code, index links included
Verdict reservation	a manifest mapping to type: verdict (or the reserved filename namespace) stops being rejected
Re-ingest layer safety	re-materialization over a bundle with a promoted verdict deletes the verdict file or its index link
Golden regression	any byte of a golden extraction's expected bundle diverges
Network gate	an http source stops refusing fail-fast without the opt-in flag
Spec integrity	this spec goes missing, names a concrete agent toolkit, or stops documenting a contract field

12. Cross-check table

Every field of the machine-readable contracts, mapped to its normative section (completeness is enforced by the spec-integrity test):

Field	Contract	Section
manifest_version	manifest (top level)	§4
source	manifest (top level)	§4
bundle_summary	manifest (top level)	§4, §6
extractions	manifest (top level)	§4
type	manifest source discriminator / OKF frontmatter	§4, §5
id	manifest source and extraction identifiers	§4
root	manifest source (file)	§4
connection_ref	manifest source (sql)	§4
base_url	manifest source (http)	§4
credential_ref	manifest source (http)	§4
title	manifest extraction / frontmatter / index label	§4, §5, §6
query	manifest extraction	§4
okf_type	manifest extraction	§4, §3
max_rows	manifest extraction	§4, §8
source_system	provenance frontmatter	§7
source_query	provenance frontmatter	§5, §7
ingested_at	provenance frontmatter / materialization argument	§5, §7
ingest_manifest	provenance frontmatter	§5, §7
generated	provenance frontmatter	§3, §7
manifest.json, fixture/, ingested-at.txt, expected-bundle/	golden extraction case	§11