portfolio-optimiser-claude/docs/extending.md

Extending the ingest layer

This document explains the ingest layer as implemented in this repo (D7) and how to extend it. It is scoped to what D7 actually ships: the file (CSV) and sql (SQLite) source types. The http/MCP source type is an optional extension point that D7 does not implement — see HTTP/MCP — an extension point D7 does not build.

The layer is built from the framework-neutral ingest-spec.md alone (section references below, §n, point into it). Everything here is deterministic and runs offline: ingest makes zero model calls, touches no network, and ingest.py imports nothing from the Agent SDK — it is pure standard library.

Ingest is the automated path to a bundle: it covers the sources that are already tabular. Everything a domain expert knows that no table holds is built by hand, and that process is a separate deliverable — see the knowledge-base recipe.

Where ingest sits

manifest → connector → materialization → OKF bundle (+ index) → [existing 8-step loop, unchanged]

The method spec forbids query-time retrieval against the bundle (method-spec §3 Step 1), so data reaches the model only via OKF bundles. Ingest is the deterministic step in front of the loop that makes that true: a connector reads a real source, and the extract is materialized as an OKF bundle the loop then reads exactly as it reads a hand-curated one — by navigation, never retrieval (§1–§2). This is not RAG and not a live lookup inside the loop.

The manifest — the whole coupling to a source

One JSON file couples an expert to a source (§4). It is schema-validated fail-fast before any source call (ManifestContract); a malformed manifest never starts a run. Queries are configuration, not code — declarative strings the connector interprets, never evaluated as program code.

A file manifest (from examples/ingest-golden-file/manifest.json):

{
  "manifest_version": 1,
  "source": {"type": "file", "id": "prosjekt-arkiv", "root": "fixture"},
  "bundle_summary": "Two CSV extracts from a local project archive (file source type).",
  "extractions": [
    {"id": "costs", "title": "Project costs", "query": "costs.csv",
     "okf_type": "dataset", "max_rows": 10}
  ]
}

A sql manifest differs only in source; the query becomes a single read-only SELECT (from examples/ingest-golden-sql/manifest.json):

{
  "source": {"type": "sql", "id": "portefolje-db", "connection_ref": "PORTEFOLJE_SQL_DSN"},
  "extractions": [
    {"id": "costs", "title": "Project costs",
     "query": "SELECT id, item, amount, note FROM costs ORDER BY id",
     "okf_type": "dataset", "max_rows": 10}
  ]
}

source is polymorphic on type — a Pydantic discriminated union (FileSource | SqlSource). A manifest whose source.type is anything other than file or sql (including http, or an unknown tag) fails validation fail-fast; it is never silently accepted.

The two source types D7 implements

file — a local CSV catalogue

source.root is the directory extraction paths resolve against; query is a relative path to a CSV whose first row is the header. Path resolution is boundary-checked fail-closed against root (_resolve_within) — an extraction can never read outside the catalogue.

sql — a local SQLite database

source.connection_ref is the name of an environment variable whose value is the database path — the location/credential never lives in the manifest (§4, §8), so a manifest is versionable and shareable without secrets. An unset reference fails fast. The connection is opened read-only (sqlite3 URI with ?mode=ro), and SQLite executes exactly one statement per execute, so the single-statement rule is enforced by the driver. Give the query an explicit ORDER BY: golden conformance requires a stable row order (§4).

Both connectors enforce max_rows fail-fast — an extraction over its cap is an error, never a silent truncation (§8).

Materialization — what a connector's rows become

Each extraction becomes one concept file ingest-{id}.md (§5) with:

• a provenance frontmatter layer (§7) — type, title, source_system, source_query, ingested_at, ingest_manifest, generated: true, in exactly that order. This is a separate contract from the method spec's §9 proposal provenance — the two are never mixed. generated: true plus the manifest reference is the honesty marker: a machine-made bundle is always labelled as such (§1).
• a markdown table body — header row = column names, data rows in source order, with §5 cell escaping. Cell typing is the connector's contract: file cells are all str; sql cells are typed (int → decimal, float → shortest round-trip form, NULL → empty string, any other type → fail, never a silent coercion). See _render_sql_cell.

ingested_at is an explicit required argument to materialize — there is no wall-clock default. That is what makes golden extractions bit-deterministic. Re-materialization removes exactly the ingest-stamped files, writes the new set, and updates index.md — curated and promoted files (and their index links) always survive (§3, §5, §6).

How to add a new source type

Adding sql on top of file touched four places; a new local source type follows the same shape (all offline, no spec change if the type already exists in §4):

1. A source model. Add a Pydantic variant (like SqlSource) with type: Literal["yourtype"], its own fields, and the shared id grammar validator. Add it to the discriminated union in ManifestContract.source.
2. A connector. Add a _read_yourtype(...) -> list[list[str]] returning the header row followed by data rows in source order, and enforcing max_rows fail-fast. Route to it from _read_extraction on source.type. If cells can be non-str, add a _render_*_cell that types them explicitly and fails on any unsupported type.
3. A golden case. Add examples/ingest-golden-{type}/ with manifest.json, fixture/, ingested-at.txt, and the byte-exact expected-bundle/ (§11). This is the only fasit.
4. Load-bearing tests. Mirror the existing seam tests so each goes red when its seam is detached (§11): golden regression, provenance stamping, navigability through the unchanged okf.py, verdict-layer reservation, and re-ingest safety. Green-but-dead is the failure mode the rule exists for.

Keep it deterministic and offline: connectors are tested only against fixtures and golden extractions, and the suite must run without credentials or network (§11).

HTTP/MCP — an extension point D7 does not build

The spec names http as an optional source type (§1, §4): a remote endpoint with a base_url and an optional named credential_ref, and "an MCP-based connector is an extension of this family." Implementing it does not require a spec change, and not implementing it does not break conformance (§1).

D7 does not implement it. There is no HTTP connector and no MCP connector in this repo. A manifest naming type: "http" is rejected fail-fast at validation (proven by test_malformed_manifest_is_rejected, which parametrizes both http and an unknown discriminator). This repo ships no network egress path and no live-source integration.

If you were to build it, three honest pointers — none of which exist in D7 today:

• The spec's http schema (§4, §5, §8). base_url must not embed credentials; credential_ref names a runtime-resolved secret; the network opt-in flag (§8) is a run argument, never a manifest field — without it the connector MUST refuse fail-fast; the response body materializes verbatim inside a fenced code block (§5).
• The MAF sibling's demonstration (I6). The sibling implementation (open/portfolio-optimiser) demonstrates the HTTP/MCP extension point against a local mock, behind the opt-in network flag — no live source. It is referenced here as a pointer only; D7 is built from the shared spec, not from the sibling's code.
• The SDK vehicle for an MCP connector. An MCP-based connector would be built on the Agent SDK's create_sdk_mcp_server / @tool primitives. These are not used anywhere in this repo — ingest.py is pure standard library. They are named here only as the mechanism you would reach for, not as something D7 wires up.

Whatever the transport, an http/MCP connector MUST honour the same manifest, materialization, gate, and golden contracts as the local connectors (§4) — and the network gate itself is a load-bearing seam (§11): the connector must refuse fail-fast without the opt-in flag.