diff --git a/README.md b/README.md index 55d3e61..7cdf9c2 100644 --- a/README.md +++ b/README.md @@ -6,11 +6,12 @@ project in a portfolio of independent projects: agents generate candidate measur mandatory deterministic validator gates the numbers, domain experts judge via human-in-the-loop, and the system learns from the verdicts. -> **Status:** the D7 build (S5–S10) is complete. The deterministic backbone, the agentic -> loop, and the learning loop are wired seam by seam, each proven by load-bearing tests -> (187 tests, all running offline without an API key). The programme's single budgeted -> **live model run has been executed and validated** — its artifacts are committed under -> [`runs/s10/`](runs/s10/) (see below). +> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer** +> (CSV and SQL source types) has since been added in front of the loop. The deterministic +> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by +> seam, each proven by load-bearing tests (265 tests, all running offline without an API +> key). The programme's single budgeted **live model run has been executed and validated** — +> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below). > **Disclaimer — technical framework only.** The deployer owns DPIA, risk assessment, and > the legal basis for any processing. The framework ships only the technical @@ -87,6 +88,36 @@ empty-inbox control), `test_step8_promotion_loadbearing.py` (the gate refuses no verdicts; the promoted signal stays out of the read-context), and `test_sdk_isolation.py` (local config cannot capture the checker). +## The ingest layer — CSV and SQL, in front of the loop + +The method spec forbids query-time retrieval against the bundle (§3 Step 1), so **data +reaches the model only via OKF bundles.** The ingest layer is the deterministic step that +satisfies that: a connector reads a real source, and the extract is materialized as an OKF +bundle the existing 8-step loop then consumes **unchanged**. It makes **zero model calls**, +touches no network, and `ingest.py` imports nothing from the SDK — it is pure standard +library. Built from the shared [`ingest-spec.md`](shared/ingest-spec.md) alone. + +D7 implements the two conformance-required source types: + +- **`file`** — a local CSV catalogue; extraction paths are boundary-checked fail-closed + against the source `root` (the OKF path rule). +- **`sql`** — a local SQLite database, opened **read-only** (`mode=ro`), one SELECT per + extraction; the connection location is resolved at run time from a named environment + variable (`connection_ref`), never stored in the manifest. + +Both are frozen by byte-identical golden extractions +([`examples/ingest-golden-file/`](examples/ingest-golden-file/), +[`examples/ingest-golden-sql/`](examples/ingest-golden-sql/)) and by the load-bearing seam +tests (provenance stamping, navigability through the unchanged `okf.py`, the reserved +verdict layer, and re-ingest safety over a promoted verdict). + +**Honesty rule (§1):** the `http` source type is an **optional extension point, not built in +D7** — a manifest naming it is rejected fail-fast at validation +(`test_malformed_manifest_is_rejected`), never silently accepted. The HTTP/MCP extension +point is demonstrated only in the MAF sibling (against a local mock, behind an opt-in +network flag); this repo ships no network connector and no live-source integration. How the +layer works and how one would extend it is documented in [`docs/extending.md`](docs/extending.md). + ## The live run — S10, executed and validated Where the MAF sibling proves its loop end-to-end with a scripted offline simulation, this @@ -115,7 +146,7 @@ Python ≥3.10 · [`claude-agent-sdk`](https://pypi.org/project/claude-agent-sdk ```bash uv sync # install dependencies -uv run pytest # 187 tests — run without any API key and without network +uv run pytest # 265 tests — run without any API key and without network uv run ruff check . && uv run ruff format --check . uv run mypy src # strict ``` diff --git a/docs/2026-07-04-I7-brief.md b/docs/2026-07-04-I7-brief.md new file mode 100644 index 0000000..da938bb --- /dev/null +++ b/docs/2026-07-04-I7-brief.md @@ -0,0 +1,47 @@ +# I7-brief (D7) — program-avslutning: dokumentasjon + statusrapport + +> **Type:** docs-avslutning, **ingen ny kode**. Komprimert pipeline (ingen +> research/plan/execute-runder — deterministisk docs-oppgave). Forankret i sesjonsplan +> [§I7](/Users/ktg/repos/portfolio-optimiser/docs/plan/2026-07-03-sesjonsplan-ingest.md) og +> STATE-ens «når I7 åpnes»-oppskrift. + +## Kontekst / gate + +I7 var GATET på at MAF lander **I6** (HTTP/MCP demonstrert MAF-only mot lokal mock, bak +opt-in-nettverksflagget). Gaten er **oppfylt**: MAF I6 landet (commits `58dda46`→`696f19a`, +review `72cb15f`, WARN akseptert). Bekreftet ved `git -C …/portfolio-optimiser log` — +**kun eksistens**, ingen lesing/reverse-engineering av MAF-koden. + +## Scope (denne økta rører KUN dette repoet) + +1. **README.md** — flett inn ingest-laget (CSV + SQL) som post-S10-tillegg; oppdater + testtall (187 → 265); HTTP/MCP kun som peker (D7 har ingen HTTP-konnektor). +2. **`docs/extending.md`** (ny) — avgrenset til det D7 faktisk har: `file` (CSV) + `sql` + (sqlite RO). Hvordan legge til en kildetype. HTTP/MCP KUN som peker: spec §4-skjema + + MAF-søskenets I6-demonstrasjon + SDK-primitivet `create_sdk_mcp_server` som *ubygd* + vehikkel (D7-`src/` bruker det ingen steder; `ingest.py` er ren stdlib). +3. **Grep-sjekk mot overpåstand** før commit. +4. **Norsk statusrapport** med verifiseringslogg (hver påstand → kommando/kilde). + +## Ærlighetsregelen (ufravikelig, method-spec §1 / ingest-spec §1) + +Ingen artefakt påstår: +- HTTP/MCP-**støtte i D7** (D7 avviser `http`-manifest fail-fast — testbevist + `test_malformed_manifest_is_rejected`). +- **Live-kilde-integrasjon** (ingest gjør null modellkall, null nettverk; testene kjører + uten nøkkel/nettverk). +- Mer live-atferd enn den ENE dokumenterte S10-kjøringen (uendret påstand). + +## Verifisering (øktas suksesskriterier) + +- `uv run pytest -q` → **265 passed** uten nøkkel/nettverk. +- Grep: ingen artefakt påstår HTTP/MCP-støtte i D7 eller live-kilde-integrasjon. +- `uv run ruff check .` + `uv run mypy src` grønne (docs-only, men suite + lint holdes grønn). +- Statusrapport med verifiseringslogg levert. + +## Ikke i scope (gated på operatør) + +- Åpent-speil-synk (`open/…`) av I3+I5+I7-koden — egen outward-facing operatør-handling. +- LICENSE / S12-release. +- Kryss-stack-sluttkriteriet (begge stacker identiske golden) eies av MAF-programrapporten; + denne D7-rapporten peker dit i stedet for å duplisere. diff --git a/docs/2026-07-04-I7-statusrapport.md b/docs/2026-07-04-I7-statusrapport.md new file mode 100644 index 0000000..422c4e6 --- /dev/null +++ b/docs/2026-07-04-I7-statusrapport.md @@ -0,0 +1,57 @@ +# I7 — statusrapport (D7): ingest-programmet, D7-stacken + +> **Scope:** denne rapporten dekker **D7-stacken alene** (dette repoet). Kryss-stack- +> sluttkriteriet (begge stacker materialiserer identiske golden-uttrekk) eies av +> MAF-programrapporten — se [Kryss-stack](#kryss-stack-eies-av-maf-programrapporten). Dato: +> 2026-07-04. Ingen ny kode i I7 (docs-avslutning). + +## Hva D7 har bevist + +| # | Påstand | Bevis | +|---|---------|-------| +| 1 | D7 implementerer `file` (CSV) + `sql` (sqlite RO) mot delt spec | `src/…/ingest.py` (`FileSource`, `SqlSource`, diskriminert union); `test_ingest_golden.py`, `test_ingest_sql_golden.py` | +| 2 | Golden-uttrekk er byte-identisk (CSV + SQL) | `examples/ingest-golden-file/`, `examples/ingest-golden-sql/` mot `expected-bundle/` — golden-regresjonstester grønne | +| 3 | Alle sømmer load-bearing (rød ved detach) | `test_ingest_loadbearing.py`, `test_ingest_sql_loadbearing.py`, `test_ingest_spec_loadbearing.py` (provenance, navigerbarhet via UENDRET `okf.py`, verdict-reservasjon, re-ingest-sikkerhet) | +| 4 | `http`-manifest avvises fail-fast (ingen stille aksept) | `test_ingest.py::test_malformed_manifest_is_rejected` (parametriserer `http` OG ukjent diskriminator → `ValidationError`) | +| 5 | Ingest gjør null modellkall / null nettverk | `ingest.py` er ren stdlib (ingen SDK/`requests`-import); hele suiten kjører uten nøkkel/nettverk | +| 6 | Full suite grønn uten nøkkel/nettverk | `uv run pytest -q` → **265 passed** | +| 7 | Lint + type rene | `uv run ruff check .` → All checks passed · `uv run mypy src` → no issues (16 filer) | +| 8 | Dokumentasjonen er sannferdig avgrenset (CSV+SQL; HTTP/MCP kun peker) | `README.md` (ingest-seksjon + honesty-note), `docs/extending.md`; grep-sjekk mot overpåstand ren | + +## Hva D7 IKKE har (ærlighetsregelen, method-spec §1 / ingest-spec §1) + +- **Ingen HTTP-konnektor, ingen MCP-konnektor.** `http` er et OPTIONAL extension point (spec + §1) som D7 bevisst ikke bygger; et `http`-manifest avvises fail-fast (bevis #4). +- **Ingen live-kilde-integrasjon.** Ingest er deterministisk, offline, null egress. +- **`create_sdk_mcp_server`/`@tool` brukes ingen steder i `src/`** — omtales i `docs/extending.md` + kun som det ubygde vehikkelet man *ville* grepet til for en MCP-konnektor. +- **Ingen ny live-atferd utover S10.** Programmets ENE ekte API-kjøring forblir S10 (D6); + ingest la ingenting til på den fronten. + +## Verifiseringslogg (hver påstand → kommando/kilde) + +| Påstand | Kommando / kilde | Resultat | +|---------|------------------|----------| +| Suite grønn uten nøkkel/nettverk | `uv run pytest -q` | `265 passed in 1.18s` | +| Lint rent | `uv run ruff check .` | `All checks passed!` | +| Type rent | `uv run mypy src` | `Success: no issues found in 16 source files` | +| `http` avvist fail-fast | `tests/test_ingest.py:71-72` (parametrisert `http`+`unknown` → `ValidationError`) | testbevist | +| Ingest rører ikke SDK/nett | `grep -n "sdk\|claude_agent\|create_sdk" src/…/ingest.py` | ingen treff (ren stdlib) | +| `create_sdk_mcp_server` ubrukt i src | `grep -rn "create_sdk_mcp_server\|@tool" src/` | ingen treff | +| Ingen overpåstand i docs | grep `http\|mcp\|live\|egress` over `README.md` + `docs/extending.md` + briefer | hvert treff = negasjon eller peker; ingen påstand om D7-HTTP/MCP-støtte eller live-kilde | +| MAF I6 landet (I7-gaten) | `git -C /Users/ktg/repos/portfolio-optimiser log --oneline` | `58dda46`→`696f19a`, review `72cb15f` — kun eksistens bekreftet | +| Golden byte-identisk | `test_ingest_golden.py`, `test_ingest_sql_golden.py` | grønne | + +## Kryss-stack (eies av MAF-programrapporten) + +Program-sluttkriteriet «begge stacker materialiserer identiske golden-uttrekk fra delt spec +(CSV + SQL)» er en **kryss-stack**-påstand. Den eies av MAF-programrapporten (hovedrepoet), +som ser begge stackene; denne D7-rapporten dupliserer den ikke. D7-siden av kontrakten er +oppfylt: D7 konsumerer den delte spec-en uendret og reproduserer golden-bytene (bevis #1, #2). +HTTP/MCP-extension-pointet er demonstrert **MAF-only** (I6, lokal mock, bak opt-in-flagg); +D7 har det bevisst ikke. + +## Utestående (gated på operatør — ikke I7-scope) + +- Åpent-speil-synk (`open/…`) av I3+I5+I7-koden — outward-facing operatør-handling. +- LICENSE-valg / S12-release. diff --git a/docs/extending.md b/docs/extending.md new file mode 100644 index 0000000..581a53f --- /dev/null +++ b/docs/extending.md @@ -0,0 +1,160 @@ +# Extending the ingest layer + +This document explains the ingest layer as **implemented in this repo (D7)** and how to +extend it. It is scoped to what D7 actually ships: the `file` (CSV) and `sql` (SQLite) +source types. The `http`/MCP source type is an optional extension point that D7 does **not** +implement — see [HTTP/MCP — an extension point D7 does not build](#httpmcp--an-extension-point-d7-does-not-build). + +The layer is built from the framework-neutral [`ingest-spec.md`](../shared/ingest-spec.md) +alone (section references below, `§n`, point into it). Everything here is deterministic and +runs offline: ingest makes **zero model calls**, touches no network, and +[`ingest.py`](../src/portfolio_optimiser_claude/ingest.py) imports nothing from the Agent SDK +— it is pure standard library. + +## Where ingest sits + +``` +manifest → connector → materialization → OKF bundle (+ index) → [existing 8-step loop, unchanged] +``` + +The method spec forbids query-time retrieval against the bundle (method-spec §3 Step 1), so +**data reaches the model only via OKF bundles.** Ingest is the deterministic step in front of +the loop that makes that true: a connector reads a real source, and the extract is +materialized as an OKF bundle the loop then reads exactly as it reads a hand-curated one — +by navigation, never retrieval (§1–§2). This is not RAG and not a live lookup inside the +loop. + +## The manifest — the whole coupling to a source + +One JSON file couples an expert to a source (§4). It is schema-validated **fail-fast before +any source call** ([`ManifestContract`](../src/portfolio_optimiser_claude/ingest.py)); a +malformed manifest never starts a run. Queries are **configuration, not code** — declarative +strings the connector interprets, never evaluated as program code. + +A `file` manifest (from [`examples/ingest-golden-file/manifest.json`](../examples/ingest-golden-file/manifest.json)): + +```json +{ + "manifest_version": 1, + "source": {"type": "file", "id": "prosjekt-arkiv", "root": "fixture"}, + "bundle_summary": "Two CSV extracts from a local project archive (file source type).", + "extractions": [ + {"id": "costs", "title": "Project costs", "query": "costs.csv", + "okf_type": "dataset", "max_rows": 10} + ] +} +``` + +A `sql` manifest differs only in `source`; the `query` becomes a single read-only SELECT +(from [`examples/ingest-golden-sql/manifest.json`](../examples/ingest-golden-sql/manifest.json)): + +```json +{ + "source": {"type": "sql", "id": "portefolje-db", "connection_ref": "PORTEFOLJE_SQL_DSN"}, + "extractions": [ + {"id": "costs", "title": "Project costs", + "query": "SELECT id, item, amount, note FROM costs ORDER BY id", + "okf_type": "dataset", "max_rows": 10} + ] +} +``` + +`source` is **polymorphic on `type`** — a Pydantic discriminated union +(`FileSource | SqlSource`). A manifest whose `source.type` is anything other than `file` or +`sql` (including `http`, or an unknown tag) fails validation fail-fast; it is never silently +accepted. + +## The two source types D7 implements + +### `file` — a local CSV catalogue + +`source.root` is the directory extraction paths resolve against; `query` is a relative path +to a CSV whose first row is the header. Path resolution is **boundary-checked fail-closed** +against `root` (`_resolve_within`) — an extraction can never read outside the catalogue. + +### `sql` — a local SQLite database + +`source.connection_ref` is the **name** of an environment variable whose value is the +database path — the location/credential never lives in the manifest (§4, §8), so a manifest +is versionable and shareable without secrets. An unset reference fails fast. The connection +is opened **read-only** (`sqlite3` URI with `?mode=ro`), and SQLite executes exactly one +statement per `execute`, so the single-statement rule is enforced by the driver. Give the +`query` an explicit `ORDER BY`: golden conformance requires a stable row order (§4). + +Both connectors enforce `max_rows` fail-fast — an extraction over its cap is an error, never +a silent truncation (§8). + +## Materialization — what a connector's rows become + +Each extraction becomes one concept file `ingest-{id}.md` (§5) with: + +- a **provenance frontmatter layer** (§7) — `type`, `title`, `source_system`, + `source_query`, `ingested_at`, `ingest_manifest`, `generated: true`, in exactly that + order. This is a *separate* contract from the method spec's §9 proposal provenance — the + two are never mixed. `generated: true` plus the manifest reference is the honesty marker: + a machine-made bundle is always labelled as such (§1). +- a **markdown table** body — header row = column names, data rows in source order, with §5 + cell escaping. Cell typing is the connector's contract: `file` cells are all `str`; `sql` + cells are typed (`int` → decimal, `float` → shortest round-trip form, `NULL` → empty + string, **any other type → fail**, never a silent coercion). See `_render_sql_cell`. + +`ingested_at` is an **explicit required argument** to `materialize` — there is no wall-clock +default. That is what makes golden extractions bit-deterministic. Re-materialization removes +exactly the ingest-stamped files, writes the new set, and updates `index.md` — curated and +promoted files (and their index links) always survive (§3, §5, §6). + +## How to add a new source type + +Adding `sql` on top of `file` touched four places; a new local source type follows the same +shape (all offline, no spec change if the type already exists in §4): + +1. **A source model.** Add a Pydantic variant (like `SqlSource`) with `type: + Literal["yourtype"]`, its own fields, and the shared `id` grammar validator. Add it to + the discriminated union in `ManifestContract.source`. +2. **A connector.** Add a `_read_yourtype(...) -> list[list[str]]` returning the header row + followed by data rows in source order, and enforcing `max_rows` fail-fast. Route to it + from `_read_extraction` on `source.type`. If cells can be non-`str`, add a + `_render_*_cell` that types them explicitly and **fails on any unsupported type**. +3. **A golden case.** Add `examples/ingest-golden-{type}/` with `manifest.json`, `fixture/`, + `ingested-at.txt`, and the byte-exact `expected-bundle/` (§11). This is the only fasit. +4. **Load-bearing tests.** Mirror the existing seam tests so each goes **red when its seam is + detached** (§11): golden regression, provenance stamping, navigability through the + unchanged `okf.py`, verdict-layer reservation, and re-ingest safety. Green-but-dead is the + failure mode the rule exists for. + +Keep it deterministic and offline: connectors are tested only against fixtures and golden +extractions, and the suite must run without credentials or network (§11). + +## HTTP/MCP — an extension point D7 does not build + +The spec names `http` as an **optional** source type (§1, §4): a remote endpoint with a +`base_url` and an optional named `credential_ref`, and "an MCP-based connector is an +extension of this family." Implementing it does not require a spec change, and **not** +implementing it does not break conformance (§1). + +**D7 does not implement it.** There is no HTTP connector and no MCP connector in this repo. A +manifest naming `type: "http"` is rejected fail-fast at validation (proven by +`test_malformed_manifest_is_rejected`, which parametrizes both `http` and an unknown +discriminator). This repo ships **no** network egress path and **no** live-source +integration. + +If you were to build it, three honest pointers — none of which exist in D7 today: + +- **The spec's `http` schema (§4, §5, §8).** `base_url` must not embed credentials; + `credential_ref` names a runtime-resolved secret; the network opt-in flag (§8) is a **run + argument, never a manifest field** — without it the connector MUST refuse fail-fast; the + response body materializes verbatim inside a fenced code block (§5). +- **The MAF sibling's demonstration (I6).** The sibling implementation + ([`open/portfolio-optimiser`](https://git.fromaitochitta.com/open/portfolio-optimiser)) + demonstrates the HTTP/MCP extension point against a **local mock**, behind the opt-in + network flag — no live source. It is referenced here as a pointer only; D7 is built from + the shared spec, not from the sibling's code. +- **The SDK vehicle for an MCP connector.** An MCP-based connector would be built on the + Agent SDK's `create_sdk_mcp_server` / `@tool` primitives. These are **not used anywhere in + this repo** — `ingest.py` is pure standard library. They are named here only as the + mechanism you would reach for, not as something D7 wires up. + +Whatever the transport, an `http`/MCP connector MUST honour the same manifest, +materialization, gate, and golden contracts as the local connectors (§4) — and the network +gate itself is a load-bearing seam (§11): the connector must refuse fail-fast without the +opt-in flag.