portfolio-optimiser-claude/docs/extending.md

# Extending the ingest layer

This document explains the ingest layer as **implemented in this repo (D7)** and how to
extend it. It is scoped to what D7 actually ships: the `file` (CSV) and `sql` (SQLite)
source types. The `http`/MCP source type is an optional extension point that D7 does **not**
implement — see [HTTP/MCP — an extension point D7 does not build](#httpmcp--an-extension-point-d7-does-not-build).

The layer is built from the framework-neutral [`ingest-spec.md`](../shared/ingest-spec.md)
alone (section references below, `§n`, point into it). Everything here is deterministic and
runs offline: ingest makes **zero model calls**, touches no network, and
[`ingest.py`](../src/portfolio_optimiser_claude/ingest.py) imports nothing from the Agent SDK
— it is pure standard library.

## Where ingest sits

```
manifest → connector → materialization → OKF bundle (+ index) → [existing 8-step loop, unchanged]
```

The method spec forbids query-time retrieval against the bundle (method-spec §3 Step 1), so
**data reaches the model only via OKF bundles.** Ingest is the deterministic step in front of
the loop that makes that true: a connector reads a real source, and the extract is
materialized as an OKF bundle the loop then reads exactly as it reads a hand-curated one —
by navigation, never retrieval (§1–§2). This is not RAG and not a live lookup inside the
loop.

## The manifest — the whole coupling to a source

One JSON file couples an expert to a source (§4). It is schema-validated **fail-fast before
any source call** ([`ManifestContract`](../src/portfolio_optimiser_claude/ingest.py)); a
malformed manifest never starts a run. Queries are **configuration, not code** — declarative
strings the connector interprets, never evaluated as program code.

A `file` manifest (from [`examples/ingest-golden-file/manifest.json`](../examples/ingest-golden-file/manifest.json)):

```json
{
  "manifest_version": 1,
  "source": {"type": "file", "id": "prosjekt-arkiv", "root": "fixture"},
  "bundle_summary": "Two CSV extracts from a local project archive (file source type).",
  "extractions": [
    {"id": "costs", "title": "Project costs", "query": "costs.csv",
     "okf_type": "dataset", "max_rows": 10}
  ]
}
```

A `sql` manifest differs only in `source`; the `query` becomes a single read-only SELECT
(from [`examples/ingest-golden-sql/manifest.json`](../examples/ingest-golden-sql/manifest.json)):

```json
{
  "source": {"type": "sql", "id": "portefolje-db", "connection_ref": "PORTEFOLJE_SQL_DSN"},
  "extractions": [
    {"id": "costs", "title": "Project costs",
     "query": "SELECT id, item, amount, note FROM costs ORDER BY id",
     "okf_type": "dataset", "max_rows": 10}
  ]
}
```

`source` is **polymorphic on `type`** — a Pydantic discriminated union
(`FileSource | SqlSource`). A manifest whose `source.type` is anything other than `file` or
`sql` (including `http`, or an unknown tag) fails validation fail-fast; it is never silently
accepted.

## The two source types D7 implements

### `file` — a local CSV catalogue

`source.root` is the directory extraction paths resolve against; `query` is a relative path
to a CSV whose first row is the header. Path resolution is **boundary-checked fail-closed**
against `root` (`_resolve_within`) — an extraction can never read outside the catalogue.

### `sql` — a local SQLite database

`source.connection_ref` is the **name** of an environment variable whose value is the
database path — the location/credential never lives in the manifest (§4, §8), so a manifest
is versionable and shareable without secrets. An unset reference fails fast. The connection
is opened **read-only** (`sqlite3` URI with `?mode=ro`), and SQLite executes exactly one
statement per `execute`, so the single-statement rule is enforced by the driver. Give the
`query` an explicit `ORDER BY`: golden conformance requires a stable row order (§4).

Both connectors enforce `max_rows` fail-fast — an extraction over its cap is an error, never
a silent truncation (§8).

## Materialization — what a connector's rows become

Each extraction becomes one concept file `ingest-{id}.md` (§5) with:

- a **provenance frontmatter layer** (§7) — `type`, `title`, `source_system`,
  `source_query`, `ingested_at`, `ingest_manifest`, `generated: true`, in exactly that
  order. This is a *separate* contract from the method spec's §9 proposal provenance — the
  two are never mixed. `generated: true` plus the manifest reference is the honesty marker:
  a machine-made bundle is always labelled as such (§1).
- a **markdown table** body — header row = column names, data rows in source order, with §5
  cell escaping. Cell typing is the connector's contract: `file` cells are all `str`; `sql`
  cells are typed (`int` → decimal, `float` → shortest round-trip form, `NULL` → empty
  string, **any other type → fail**, never a silent coercion). See `_render_sql_cell`.

`ingested_at` is an **explicit required argument** to `materialize` — there is no wall-clock
default. That is what makes golden extractions bit-deterministic. Re-materialization removes
exactly the ingest-stamped files, writes the new set, and updates `index.md` — curated and
promoted files (and their index links) always survive (§3, §5, §6).

## How to add a new source type

Adding `sql` on top of `file` touched four places; a new local source type follows the same
shape (all offline, no spec change if the type already exists in §4):

1. **A source model.** Add a Pydantic variant (like `SqlSource`) with `type:
   Literal["yourtype"]`, its own fields, and the shared `id` grammar validator. Add it to
   the discriminated union in `ManifestContract.source`.
2. **A connector.** Add a `_read_yourtype(...) -> list[list[str]]` returning the header row
   followed by data rows in source order, and enforcing `max_rows` fail-fast. Route to it
   from `_read_extraction` on `source.type`. If cells can be non-`str`, add a
   `_render_*_cell` that types them explicitly and **fails on any unsupported type**.
3. **A golden case.** Add `examples/ingest-golden-{type}/` with `manifest.json`, `fixture/`,
   `ingested-at.txt`, and the byte-exact `expected-bundle/` (§11). This is the only fasit.
4. **Load-bearing tests.** Mirror the existing seam tests so each goes **red when its seam is
   detached** (§11): golden regression, provenance stamping, navigability through the
   unchanged `okf.py`, verdict-layer reservation, and re-ingest safety. Green-but-dead is the
   failure mode the rule exists for.

Keep it deterministic and offline: connectors are tested only against fixtures and golden
extractions, and the suite must run without credentials or network (§11).

## HTTP/MCP — an extension point D7 does not build

The spec names `http` as an **optional** source type (§1, §4): a remote endpoint with a
`base_url` and an optional named `credential_ref`, and "an MCP-based connector is an
extension of this family." Implementing it does not require a spec change, and **not**
implementing it does not break conformance (§1).

**D7 does not implement it.** There is no HTTP connector and no MCP connector in this repo. A
manifest naming `type: "http"` is rejected fail-fast at validation (proven by
`test_malformed_manifest_is_rejected`, which parametrizes both `http` and an unknown
discriminator). This repo ships **no** network egress path and **no** live-source
integration.

If you were to build it, three honest pointers — none of which exist in D7 today:

- **The spec's `http` schema (§4, §5, §8).** `base_url` must not embed credentials;
  `credential_ref` names a runtime-resolved secret; the network opt-in flag (§8) is a **run
  argument, never a manifest field** — without it the connector MUST refuse fail-fast; the
  response body materializes verbatim inside a fenced code block (§5).
- **The MAF sibling's demonstration (I6).** The sibling implementation
  ([`open/portfolio-optimiser`](https://git.fromaitochitta.com/open/portfolio-optimiser))
  demonstrates the HTTP/MCP extension point against a **local mock**, behind the opt-in
  network flag — no live source. It is referenced here as a pointer only; D7 is built from
  the shared spec, not from the sibling's code.
- **The SDK vehicle for an MCP connector.** An MCP-based connector would be built on the
  Agent SDK's `create_sdk_mcp_server` / `@tool` primitives. These are **not used anywhere in
  this repo** — `ingest.py` is pure standard library. They are named here only as the
  mechanism you would reach for, not as something D7 wires up.

Whatever the transport, an `http`/MCP connector MUST honour the same manifest,
materialization, gate, and golden contracts as the local connectors (§4) — and the network
gate itself is a load-bearing seam (§11): the connector must refuse fail-fast without the
opt-in flag.