Kjell Tore Guttormsen
da93a68ce7
The last ungated build session: the operator now drives the whole build from the command line, and the documents claim exactly what the code does (§1). run.py becomes the collecting entrance. Exactly one of --bundle (one project) or --portfolio (N projects from a schema-validated reference config, with --verdict-dir as the portfolio-level expert inbox) is required; both and neither are refused. --goals loads a goal contract and checks it against --ledger's realized sum BEFORE the first model call: the §8 caps bound spend, the goal bounds achievement, so a hard target the book already meets stops the run at exit 4 without constructing a client. A soft target reached is a flag and the run continues; an absent ledger is an empty book, so the goal is still evaluated, never skipped. The one declared goal also drives --value-report's goal progress — one contract, never two figures that can disagree. The portfolio path persists nothing (K3 returns typed results; the outbox names pairs by run_id, which a portfolio pass has none of). Rather than accept --out/--outbox/--run-id/--value-report/--inbox/--live-dry-run there and silently ignore them, the entrance refuses them and says why. run_portfolio is imported lazily — portfolio.py imports this module, so a module-level import is circular. Three seams, each detach-proven RED: - unwire the goal check → the run proceeds and spends → red - unwire the portfolio branch → the configured projects never run → red - document a flag no CLI offers → the README honesty grep goes red That last one is the doc-sync made load-bearing: the test reads README.md, collects every --flag it documents (excluding third-party dev-tooling lines) and asserts each exists in the --help of a CLI the README names. The drift it exists to close was real — README claimed 562 tests, CHANGELOG claimed 265, actual 597. Docs synced to the code: README gains an operator-CLI section and honest goal/ portfolio descriptions, CHANGELOG is rewritten to what actually shipped, and docs/oppskrift-kunnskapsbase.md delivers D-H point 1 — the documented team process for building a knowledge base, with the honest 1–2 week expectation and every factory-dependent step (verdict translation, demo path) marked NOT BUILT. 597 passed · ruff clean · mypy strict clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQu2xxwedckjU56byu1aUG
164 lines
9 KiB
Markdown
164 lines
9 KiB
Markdown
# Extending the ingest layer
|
||
|
||
This document explains the ingest layer as **implemented in this repo (D7)** and how to
|
||
extend it. It is scoped to what D7 actually ships: the `file` (CSV) and `sql` (SQLite)
|
||
source types. The `http`/MCP source type is an optional extension point that D7 does **not**
|
||
implement — see [HTTP/MCP — an extension point D7 does not build](#httpmcp--an-extension-point-d7-does-not-build).
|
||
|
||
The layer is built from the framework-neutral [`ingest-spec.md`](../shared/ingest-spec.md)
|
||
alone (section references below, `§n`, point into it). Everything here is deterministic and
|
||
runs offline: ingest makes **zero model calls**, touches no network, and
|
||
[`ingest.py`](../src/portfolio_optimiser_claude/ingest.py) imports nothing from the Agent SDK
|
||
— it is pure standard library.
|
||
|
||
Ingest is the **automated** path to a bundle: it covers the sources that are already tabular.
|
||
Everything a domain expert knows that no table holds is built by hand, and that process is a
|
||
separate deliverable — see [the knowledge-base recipe](oppskrift-kunnskapsbase.md).
|
||
|
||
## Where ingest sits
|
||
|
||
```
|
||
manifest → connector → materialization → OKF bundle (+ index) → [existing 8-step loop, unchanged]
|
||
```
|
||
|
||
The method spec forbids query-time retrieval against the bundle (method-spec §3 Step 1), so
|
||
**data reaches the model only via OKF bundles.** Ingest is the deterministic step in front of
|
||
the loop that makes that true: a connector reads a real source, and the extract is
|
||
materialized as an OKF bundle the loop then reads exactly as it reads a hand-curated one —
|
||
by navigation, never retrieval (§1–§2). This is not RAG and not a live lookup inside the
|
||
loop.
|
||
|
||
## The manifest — the whole coupling to a source
|
||
|
||
One JSON file couples an expert to a source (§4). It is schema-validated **fail-fast before
|
||
any source call** ([`ManifestContract`](../src/portfolio_optimiser_claude/ingest.py)); a
|
||
malformed manifest never starts a run. Queries are **configuration, not code** — declarative
|
||
strings the connector interprets, never evaluated as program code.
|
||
|
||
A `file` manifest (from [`examples/ingest-golden-file/manifest.json`](../examples/ingest-golden-file/manifest.json)):
|
||
|
||
```json
|
||
{
|
||
"manifest_version": 1,
|
||
"source": {"type": "file", "id": "prosjekt-arkiv", "root": "fixture"},
|
||
"bundle_summary": "Two CSV extracts from a local project archive (file source type).",
|
||
"extractions": [
|
||
{"id": "costs", "title": "Project costs", "query": "costs.csv",
|
||
"okf_type": "dataset", "max_rows": 10}
|
||
]
|
||
}
|
||
```
|
||
|
||
A `sql` manifest differs only in `source`; the `query` becomes a single read-only SELECT
|
||
(from [`examples/ingest-golden-sql/manifest.json`](../examples/ingest-golden-sql/manifest.json)):
|
||
|
||
```json
|
||
{
|
||
"source": {"type": "sql", "id": "portefolje-db", "connection_ref": "PORTEFOLJE_SQL_DSN"},
|
||
"extractions": [
|
||
{"id": "costs", "title": "Project costs",
|
||
"query": "SELECT id, item, amount, note FROM costs ORDER BY id",
|
||
"okf_type": "dataset", "max_rows": 10}
|
||
]
|
||
}
|
||
```
|
||
|
||
`source` is **polymorphic on `type`** — a Pydantic discriminated union
|
||
(`FileSource | SqlSource`). A manifest whose `source.type` is anything other than `file` or
|
||
`sql` (including `http`, or an unknown tag) fails validation fail-fast; it is never silently
|
||
accepted.
|
||
|
||
## The two source types D7 implements
|
||
|
||
### `file` — a local CSV catalogue
|
||
|
||
`source.root` is the directory extraction paths resolve against; `query` is a relative path
|
||
to a CSV whose first row is the header. Path resolution is **boundary-checked fail-closed**
|
||
against `root` (`_resolve_within`) — an extraction can never read outside the catalogue.
|
||
|
||
### `sql` — a local SQLite database
|
||
|
||
`source.connection_ref` is the **name** of an environment variable whose value is the
|
||
database path — the location/credential never lives in the manifest (§4, §8), so a manifest
|
||
is versionable and shareable without secrets. An unset reference fails fast. The connection
|
||
is opened **read-only** (`sqlite3` URI with `?mode=ro`), and SQLite executes exactly one
|
||
statement per `execute`, so the single-statement rule is enforced by the driver. Give the
|
||
`query` an explicit `ORDER BY`: golden conformance requires a stable row order (§4).
|
||
|
||
Both connectors enforce `max_rows` fail-fast — an extraction over its cap is an error, never
|
||
a silent truncation (§8).
|
||
|
||
## Materialization — what a connector's rows become
|
||
|
||
Each extraction becomes one concept file `ingest-{id}.md` (§5) with:
|
||
|
||
- a **provenance frontmatter layer** (§7) — `type`, `title`, `source_system`,
|
||
`source_query`, `ingested_at`, `ingest_manifest`, `generated: true`, in exactly that
|
||
order. This is a *separate* contract from the method spec's §9 proposal provenance — the
|
||
two are never mixed. `generated: true` plus the manifest reference is the honesty marker:
|
||
a machine-made bundle is always labelled as such (§1).
|
||
- a **markdown table** body — header row = column names, data rows in source order, with §5
|
||
cell escaping. Cell typing is the connector's contract: `file` cells are all `str`; `sql`
|
||
cells are typed (`int` → decimal, `float` → shortest round-trip form, `NULL` → empty
|
||
string, **any other type → fail**, never a silent coercion). See `_render_sql_cell`.
|
||
|
||
`ingested_at` is an **explicit required argument** to `materialize` — there is no wall-clock
|
||
default. That is what makes golden extractions bit-deterministic. Re-materialization removes
|
||
exactly the ingest-stamped files, writes the new set, and updates `index.md` — curated and
|
||
promoted files (and their index links) always survive (§3, §5, §6).
|
||
|
||
## How to add a new source type
|
||
|
||
Adding `sql` on top of `file` touched four places; a new local source type follows the same
|
||
shape (all offline, no spec change if the type already exists in §4):
|
||
|
||
1. **A source model.** Add a Pydantic variant (like `SqlSource`) with `type:
|
||
Literal["yourtype"]`, its own fields, and the shared `id` grammar validator. Add it to
|
||
the discriminated union in `ManifestContract.source`.
|
||
2. **A connector.** Add a `_read_yourtype(...) -> list[list[str]]` returning the header row
|
||
followed by data rows in source order, and enforcing `max_rows` fail-fast. Route to it
|
||
from `_read_extraction` on `source.type`. If cells can be non-`str`, add a
|
||
`_render_*_cell` that types them explicitly and **fails on any unsupported type**.
|
||
3. **A golden case.** Add `examples/ingest-golden-{type}/` with `manifest.json`, `fixture/`,
|
||
`ingested-at.txt`, and the byte-exact `expected-bundle/` (§11). This is the only fasit.
|
||
4. **Load-bearing tests.** Mirror the existing seam tests so each goes **red when its seam is
|
||
detached** (§11): golden regression, provenance stamping, navigability through the
|
||
unchanged `okf.py`, verdict-layer reservation, and re-ingest safety. Green-but-dead is the
|
||
failure mode the rule exists for.
|
||
|
||
Keep it deterministic and offline: connectors are tested only against fixtures and golden
|
||
extractions, and the suite must run without credentials or network (§11).
|
||
|
||
## HTTP/MCP — an extension point D7 does not build
|
||
|
||
The spec names `http` as an **optional** source type (§1, §4): a remote endpoint with a
|
||
`base_url` and an optional named `credential_ref`, and "an MCP-based connector is an
|
||
extension of this family." Implementing it does not require a spec change, and **not**
|
||
implementing it does not break conformance (§1).
|
||
|
||
**D7 does not implement it.** There is no HTTP connector and no MCP connector in this repo. A
|
||
manifest naming `type: "http"` is rejected fail-fast at validation (proven by
|
||
`test_malformed_manifest_is_rejected`, which parametrizes both `http` and an unknown
|
||
discriminator). This repo ships **no** network egress path and **no** live-source
|
||
integration.
|
||
|
||
If you were to build it, three honest pointers — none of which exist in D7 today:
|
||
|
||
- **The spec's `http` schema (§4, §5, §8).** `base_url` must not embed credentials;
|
||
`credential_ref` names a runtime-resolved secret; the network opt-in flag (§8) is a **run
|
||
argument, never a manifest field** — without it the connector MUST refuse fail-fast; the
|
||
response body materializes verbatim inside a fenced code block (§5).
|
||
- **The MAF sibling's demonstration (I6).** The sibling implementation
|
||
([`open/portfolio-optimiser`](https://git.fromaitochitta.com/open/portfolio-optimiser))
|
||
demonstrates the HTTP/MCP extension point against a **local mock**, behind the opt-in
|
||
network flag — no live source. It is referenced here as a pointer only; D7 is built from
|
||
the shared spec, not from the sibling's code.
|
||
- **The SDK vehicle for an MCP connector.** An MCP-based connector would be built on the
|
||
Agent SDK's `create_sdk_mcp_server` / `@tool` primitives. These are **not used anywhere in
|
||
this repo** — `ingest.py` is pure standard library. They are named here only as the
|
||
mechanism you would reach for, not as something D7 wires up.
|
||
|
||
Whatever the transport, an `http`/MCP connector MUST honour the same manifest,
|
||
materialization, gate, and golden contracts as the local connectors (§4) — and the network
|
||
gate itself is a load-bearing seam (§11): the connector must refuse fail-fast without the
|
||
opt-in flag.
|