portfolio-optimiser/docs/extending.md

# Extending the framework (extension points)

portfolio-optimiser is a generic core with explicit **config seams** (D4/D5, 90 %-prinsippet):
you onboard a new project, a new data source, or a new model-map **without editing the core
`src/portfolio_optimiser/*.py`**. The three guides below name the exact seam for each.

> **Honesty note (rent teknisk rammeverk).** The bundled reference domain
> (`data/reference_projects.json` + `data/docs/<id>/`) is a set of **SYNTHETIC, AI-authored
> fixtures** — fictional construction-cost projects, dummy estimates, and placeholder
> `verdict_input` decisions. They are flagged in each file's `_note`. A production deployer
> **replaces the data source** with their own and supplies **Layer-2 verdicts via real HITL**
> (fageksperter), not static config. The static `verdict_input` field is a test-fixture
> convenience that stands in for the durable HITL verdict in the offline synthetic framework.

## Legg til eget prosjekt

A new project is **config + docs only** — no code change (this is exercised by the SC1 test
`test_e_new_project_flows_through_via_config_only` + the `test_f_no_hardcoded_project_ids_in_src`
guard, which fails if any project id leaks into `src/`).

1. Append an entry to `src/portfolio_optimiser/data/reference_projects.json` with the full key
   set: `id`, `name`, `description`, `currency`, `cost_items`, **`docs_dir`**, and
   **`verdict_input`** (`{"decision", "rationale"}`).
   - `docs_dir` is a path **relative to the package `data/` root** (e.g. `"docs/MY-PROJECT"`);
     the loader (`reference_domain.load_reference_projects`) resolves it to an absolute path.
   - `verdict_input` carries the (synthetic) Layer-2 decision/rationale; flag it in the file's
     `_note` as synthetic if it is not a real expert verdict.
2. Create the bundled docs folder `src/portfolio_optimiser/data/docs/<id>/` with at least one
   text file whose content names the cost-saving measure/terms (so `retrieve_chunks` returns at
   least one citable chunk).
3. Run the project: `run_portfolio(["MY-PROJECT"], "local", client_factory=...)` (or include it
   in the default fan-out by passing no `project_ids`).

## Legg til egen datakilde

The retriever (`retrieval.py` / `datasource.py`) reads a **local docs folder** per project,
selected by the project's `docs_dir` in `reference_projects.json`. To point a project at your own
data, change its `docs_dir` to your folder and drop your cost documentation there — the citation
seam (`{file, locator, snippet, score}`) is identical on the in-process tool and MCP paths. The
folder is boundary-checked (fail-closed) against path traversal, so keep documents inside the
configured `docs_dir`. A real deployer swaps the bundled synthetic docs for their own source.

## Legg til egen modell-map

Model choice is **config, not code** (B12): `src/portfolio_optimiser/data/model_map.json` maps
`profile -> role -> model/deployment` (`resolve_model(profile, role)`). To use your own models:

- Edit the `local` block to your local model ids (Ollama/LM Studio), and/or
- Edit the `azure` block to your Foundry deployment names (the placeholders
  `REPLACE-WITH-FOUNDRY-DEPLOYMENT` are tenant-specific — replace them or supply via env).

The role keys (`proposer`, `checker`, `default`) let you assign a distinct model per debate role;
`default` is the fallback when a role is unmapped.

## Legg til en ingest-kilde (http-familien som worked example)

The **ingest layer** (`ingest.py`, spec `shared/ingest-spec.md`) is a second, distinct source
seam from the retriever above: one JSON **manifest** per source coupling declares a `source.type`
and a list of extractions; `materialize(...)` runs the connector for that type and writes an OKF
bundle. The spec ships three source families — `file`/CSV (I2), `sql` (I4), and **`http`** (I6) —
and `http` is the framework's **worked extension-point example**: it shows exactly how a third,
network-transport family plugs into the same connector / materialization / gate contracts.

The `http` manifest contract (spec §4):

- `source.base_url` — the endpoint root; it **must not embed credentials** (userinfo is rejected
  at schema validation).
- `source.credential_ref` — the **name of an environment variable** holding the secret, resolved
  at run time (sent as `Authorization: Bearer <secret>`); the secret is never read from the
  manifest, never logged, never stamped in the generated frontmatter. `null` → no auth.
- each extraction's `query` is joined onto `base_url` and its response body is rendered
  **verbatim inside a fenced code block** (not a markdown table — a raw `|` or `\` survives
  un-escaped); `max_rows` caps the response line count, fail-fast (never silent truncation).

**Network is a per-run grant, never a manifest field (spec §8).** `materialize` refuses an `http`
source unless the caller passes `allow_network=True`:

```python
# refused fail-fast — the manifest cannot grant itself network access:
materialize(manifest, bundle_dir, ingested_at="2026-07-04T12:00:00Z")
# opted in explicitly by the operator for this run:
materialize(manifest, bundle_dir, ingested_at="2026-07-04T12:00:00Z", allow_network=True)
```

This is the **local-only default, no silent egress** principle made mechanical: configuration is
data that cannot escalate its own authority; only the runtime `allow_network` grant can. The
transport itself is an **injectable seam** — `materialize(..., http_get=<callable>)` swaps the GET
implementation (the default `_urllib_get` is the only socket path). The golden case
(`examples/ingest-golden-http/`) and every test inject a canned `get` over committed fixture
payloads, so the suite runs offline against a **local mock** — no live source, no credentials.

### D7 sibling hook — MCP as an extension of this family

The spec (§4) documents an **MCP-based connector as an extension of the `http` family**, not a new
client wired into the optimiser run path — that stays a **Non-Goal** here: the in-process
`FunctionTool` seam is the default in the run path, and MCP is demonstrated (via `build_mcp_server`
in `datasource.py`), not wired in. On the Claude Agent SDK sibling (D7), the in-process server
hook is `create_sdk_mcp_server(name, version="1.0.0", tools=...) -> McpSdkServerConfig` (package
`claude-agent-sdk`) — verified 2026-07-04 against the official Claude Agent SDK Python docs. A
deployer who wants a network- or MCP-mediated source extends this family behind the same explicit,
per-run network grant; nothing here contacts a live endpoint.

## Bevisst ikke bygget (90 %-kuttlista)

Per the design philosophy (a ~90 % generic core with clear extension points — we do not chase
the last 10 %), the following are **deliberate cuts**, not roadmap debt. Each is extension
territory for a deployer, with the seam named:

- **B10 — full verdict-conflict taxonomy.** Chosen minimal semantics (documented in
  `verdicts.py` + README): the in-memory store is first-write-wins per verdict id; the disk
  layers (`write_verdict`, `promote_verdict`) are last-write-wins per file. The full taxonomy
  (rejection categories + a rule for conflicting expert verdicts) is deferred until real experts
  produce conflicting verdicts.
- **B11 — expert notification.** `run_project(notify=...)` is a stub seam (`run.py`): pass any
  callable; no delivery mechanism (e-mail/Teams/webhook) ships with the core.
- **U12 — checkpointing / crash-survival of a run.** A run either completes or is re-run; the
  async verdict inbox (step 7) is the resumable boundary, not intra-run state.
- **U14 — OpenTelemetry / observability.** Provenance stamping is the audit trail the core
  ships; OTEL wiring is a deployer concern.
- **Concurrent fan-out.** `run_portfolio` iterates projects sequentially by design (fresh
  per-project execution state; one threaded `VerdictStore`); parallel orchestration is left to
  the deployer and would need budget-cap coordination.