docs(ingest): I7 — programavslutning, D7-docs avgrenset til CSV + SQL
Docs-only program-avslutning for ingest-laget (D7-stacken). Bygget fra commons-spec + eget repo alene — MAF-koden kun eksistens-bekreftet (I6-gate), aldri lest/reverse-engineert. - README: ingest-seksjon (file/CSV + sql/sqlite RO) som post-S10-tillegg; testtall 187 -> 265; HTTP/MCP kun peker (D7 har ingen HTTP-konnektor). - docs/extending.md (ny): hvordan ingest-laget virker + hvordan legge til en kildetype; HTTP/MCP som ærlig extension-point-peker (spec §4, MAF I6-demo, create_sdk_mcp_server som ubygd vehikkel — brukt ingen steder i src/). - docs/2026-07-04-I7-brief.md + -statusrapport.md: I7-brief + D7-lokal statusrapport m/ verifiseringslogg. Kryss-stack-kriteriet peker til MAF-programrapporten. Ærlighetsregelen (method-spec §1): ingen artefakt påstår HTTP/MCP-støtte i D7 eller live-kilde-integrasjon. Grep-sjekk mot overpåstand ren. Verifisering: 265 passed uten nøkkel/nettverk · ruff/mypy rene. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R7nuUMUyWB9viiBjz8Bo7R
This commit is contained in:
parent
32640deffc
commit
0c42cebf90
4 changed files with 301 additions and 6 deletions
160
docs/extending.md
Normal file
160
docs/extending.md
Normal file
|
|
@ -0,0 +1,160 @@
|
|||
# Extending the ingest layer
|
||||
|
||||
This document explains the ingest layer as **implemented in this repo (D7)** and how to
|
||||
extend it. It is scoped to what D7 actually ships: the `file` (CSV) and `sql` (SQLite)
|
||||
source types. The `http`/MCP source type is an optional extension point that D7 does **not**
|
||||
implement — see [HTTP/MCP — an extension point D7 does not build](#httpmcp--an-extension-point-d7-does-not-build).
|
||||
|
||||
The layer is built from the framework-neutral [`ingest-spec.md`](../shared/ingest-spec.md)
|
||||
alone (section references below, `§n`, point into it). Everything here is deterministic and
|
||||
runs offline: ingest makes **zero model calls**, touches no network, and
|
||||
[`ingest.py`](../src/portfolio_optimiser_claude/ingest.py) imports nothing from the Agent SDK
|
||||
— it is pure standard library.
|
||||
|
||||
## Where ingest sits
|
||||
|
||||
```
|
||||
manifest → connector → materialization → OKF bundle (+ index) → [existing 8-step loop, unchanged]
|
||||
```
|
||||
|
||||
The method spec forbids query-time retrieval against the bundle (method-spec §3 Step 1), so
|
||||
**data reaches the model only via OKF bundles.** Ingest is the deterministic step in front of
|
||||
the loop that makes that true: a connector reads a real source, and the extract is
|
||||
materialized as an OKF bundle the loop then reads exactly as it reads a hand-curated one —
|
||||
by navigation, never retrieval (§1–§2). This is not RAG and not a live lookup inside the
|
||||
loop.
|
||||
|
||||
## The manifest — the whole coupling to a source
|
||||
|
||||
One JSON file couples an expert to a source (§4). It is schema-validated **fail-fast before
|
||||
any source call** ([`ManifestContract`](../src/portfolio_optimiser_claude/ingest.py)); a
|
||||
malformed manifest never starts a run. Queries are **configuration, not code** — declarative
|
||||
strings the connector interprets, never evaluated as program code.
|
||||
|
||||
A `file` manifest (from [`examples/ingest-golden-file/manifest.json`](../examples/ingest-golden-file/manifest.json)):
|
||||
|
||||
```json
|
||||
{
|
||||
"manifest_version": 1,
|
||||
"source": {"type": "file", "id": "prosjekt-arkiv", "root": "fixture"},
|
||||
"bundle_summary": "Two CSV extracts from a local project archive (file source type).",
|
||||
"extractions": [
|
||||
{"id": "costs", "title": "Project costs", "query": "costs.csv",
|
||||
"okf_type": "dataset", "max_rows": 10}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
A `sql` manifest differs only in `source`; the `query` becomes a single read-only SELECT
|
||||
(from [`examples/ingest-golden-sql/manifest.json`](../examples/ingest-golden-sql/manifest.json)):
|
||||
|
||||
```json
|
||||
{
|
||||
"source": {"type": "sql", "id": "portefolje-db", "connection_ref": "PORTEFOLJE_SQL_DSN"},
|
||||
"extractions": [
|
||||
{"id": "costs", "title": "Project costs",
|
||||
"query": "SELECT id, item, amount, note FROM costs ORDER BY id",
|
||||
"okf_type": "dataset", "max_rows": 10}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`source` is **polymorphic on `type`** — a Pydantic discriminated union
|
||||
(`FileSource | SqlSource`). A manifest whose `source.type` is anything other than `file` or
|
||||
`sql` (including `http`, or an unknown tag) fails validation fail-fast; it is never silently
|
||||
accepted.
|
||||
|
||||
## The two source types D7 implements
|
||||
|
||||
### `file` — a local CSV catalogue
|
||||
|
||||
`source.root` is the directory extraction paths resolve against; `query` is a relative path
|
||||
to a CSV whose first row is the header. Path resolution is **boundary-checked fail-closed**
|
||||
against `root` (`_resolve_within`) — an extraction can never read outside the catalogue.
|
||||
|
||||
### `sql` — a local SQLite database
|
||||
|
||||
`source.connection_ref` is the **name** of an environment variable whose value is the
|
||||
database path — the location/credential never lives in the manifest (§4, §8), so a manifest
|
||||
is versionable and shareable without secrets. An unset reference fails fast. The connection
|
||||
is opened **read-only** (`sqlite3` URI with `?mode=ro`), and SQLite executes exactly one
|
||||
statement per `execute`, so the single-statement rule is enforced by the driver. Give the
|
||||
`query` an explicit `ORDER BY`: golden conformance requires a stable row order (§4).
|
||||
|
||||
Both connectors enforce `max_rows` fail-fast — an extraction over its cap is an error, never
|
||||
a silent truncation (§8).
|
||||
|
||||
## Materialization — what a connector's rows become
|
||||
|
||||
Each extraction becomes one concept file `ingest-{id}.md` (§5) with:
|
||||
|
||||
- a **provenance frontmatter layer** (§7) — `type`, `title`, `source_system`,
|
||||
`source_query`, `ingested_at`, `ingest_manifest`, `generated: true`, in exactly that
|
||||
order. This is a *separate* contract from the method spec's §9 proposal provenance — the
|
||||
two are never mixed. `generated: true` plus the manifest reference is the honesty marker:
|
||||
a machine-made bundle is always labelled as such (§1).
|
||||
- a **markdown table** body — header row = column names, data rows in source order, with §5
|
||||
cell escaping. Cell typing is the connector's contract: `file` cells are all `str`; `sql`
|
||||
cells are typed (`int` → decimal, `float` → shortest round-trip form, `NULL` → empty
|
||||
string, **any other type → fail**, never a silent coercion). See `_render_sql_cell`.
|
||||
|
||||
`ingested_at` is an **explicit required argument** to `materialize` — there is no wall-clock
|
||||
default. That is what makes golden extractions bit-deterministic. Re-materialization removes
|
||||
exactly the ingest-stamped files, writes the new set, and updates `index.md` — curated and
|
||||
promoted files (and their index links) always survive (§3, §5, §6).
|
||||
|
||||
## How to add a new source type
|
||||
|
||||
Adding `sql` on top of `file` touched four places; a new local source type follows the same
|
||||
shape (all offline, no spec change if the type already exists in §4):
|
||||
|
||||
1. **A source model.** Add a Pydantic variant (like `SqlSource`) with `type:
|
||||
Literal["yourtype"]`, its own fields, and the shared `id` grammar validator. Add it to
|
||||
the discriminated union in `ManifestContract.source`.
|
||||
2. **A connector.** Add a `_read_yourtype(...) -> list[list[str]]` returning the header row
|
||||
followed by data rows in source order, and enforcing `max_rows` fail-fast. Route to it
|
||||
from `_read_extraction` on `source.type`. If cells can be non-`str`, add a
|
||||
`_render_*_cell` that types them explicitly and **fails on any unsupported type**.
|
||||
3. **A golden case.** Add `examples/ingest-golden-{type}/` with `manifest.json`, `fixture/`,
|
||||
`ingested-at.txt`, and the byte-exact `expected-bundle/` (§11). This is the only fasit.
|
||||
4. **Load-bearing tests.** Mirror the existing seam tests so each goes **red when its seam is
|
||||
detached** (§11): golden regression, provenance stamping, navigability through the
|
||||
unchanged `okf.py`, verdict-layer reservation, and re-ingest safety. Green-but-dead is the
|
||||
failure mode the rule exists for.
|
||||
|
||||
Keep it deterministic and offline: connectors are tested only against fixtures and golden
|
||||
extractions, and the suite must run without credentials or network (§11).
|
||||
|
||||
## HTTP/MCP — an extension point D7 does not build
|
||||
|
||||
The spec names `http` as an **optional** source type (§1, §4): a remote endpoint with a
|
||||
`base_url` and an optional named `credential_ref`, and "an MCP-based connector is an
|
||||
extension of this family." Implementing it does not require a spec change, and **not**
|
||||
implementing it does not break conformance (§1).
|
||||
|
||||
**D7 does not implement it.** There is no HTTP connector and no MCP connector in this repo. A
|
||||
manifest naming `type: "http"` is rejected fail-fast at validation (proven by
|
||||
`test_malformed_manifest_is_rejected`, which parametrizes both `http` and an unknown
|
||||
discriminator). This repo ships **no** network egress path and **no** live-source
|
||||
integration.
|
||||
|
||||
If you were to build it, three honest pointers — none of which exist in D7 today:
|
||||
|
||||
- **The spec's `http` schema (§4, §5, §8).** `base_url` must not embed credentials;
|
||||
`credential_ref` names a runtime-resolved secret; the network opt-in flag (§8) is a **run
|
||||
argument, never a manifest field** — without it the connector MUST refuse fail-fast; the
|
||||
response body materializes verbatim inside a fenced code block (§5).
|
||||
- **The MAF sibling's demonstration (I6).** The sibling implementation
|
||||
([`open/portfolio-optimiser`](https://git.fromaitochitta.com/open/portfolio-optimiser))
|
||||
demonstrates the HTTP/MCP extension point against a **local mock**, behind the opt-in
|
||||
network flag — no live source. It is referenced here as a pointer only; D7 is built from
|
||||
the shared spec, not from the sibling's code.
|
||||
- **The SDK vehicle for an MCP connector.** An MCP-based connector would be built on the
|
||||
Agent SDK's `create_sdk_mcp_server` / `@tool` primitives. These are **not used anywhere in
|
||||
this repo** — `ingest.py` is pure standard library. They are named here only as the
|
||||
mechanism you would reach for, not as something D7 wires up.
|
||||
|
||||
Whatever the transport, an `http`/MCP connector MUST honour the same manifest,
|
||||
materialization, gate, and golden contracts as the local connectors (§4) — and the network
|
||||
gate itself is a load-bearing seam (§11): the connector must refuse fail-fast without the
|
||||
opt-in flag.
|
||||
Loading…
Add table
Add a link
Reference in a new issue