diff --git a/docs/okf-ingestion/plan.md b/docs/okf-ingestion/plan.md new file mode 100644 index 0000000..696055d --- /dev/null +++ b/docs/okf-ingestion/plan.md @@ -0,0 +1,191 @@ +# OKF-ingestion plan — mapping this repo against `llm-ingestion-okf` (phase 4) + +> **Status: `planned` (2026-07-20).** Survey + requirements only — nothing wired, nothing built. The +> library's Node half does not exist yet (its phase 4), so there is nothing to adopt today. This document +> is the repo's durable answer to *what door A or door B must support before this plugin can adopt*, and +> *what must never move into the shared library*. +> +> **Library:** `llm-ingestion-okf` v0.3.1 — Python 3.10+, zero runtime deps, **door A only** +> (manifest → connector → deterministic materialization of `ingest-{id}.md` → index generation, no model +> calls). Doors B (inbox) and C (external bundle import) are phase 2; the Node half is phase 4. +> **Spec:** `ingest-spec.md`, owned by `portfolio-optimiser-commons` — changes go via commons, never locally. +> **Security boundary:** `llm-ingestion-guard` (repo `llm-ingestion-pipeline-security`) owns all security. +> Door A is **ungated** — it calls no guard function before writing to disk (verified: no guard import or +> call site anywhere in the library's `src/`; `dependencies = []`). "Security is delegated" does **not** +> mean "safe by default"; gating is the caller's responsibility. +> +> **Separate, do not merge:** `docs/ingestion-guard/plan.md` (`planned`) is this repo's *security* wiring +> plan. It shares persist points with this document by necessity, but is a distinct decision with a +> distinct dependency. §6 states the boundary. + +## 1. Executive answer + +**Nothing here is a clean door-A fit today, and the reason is not the one the library expects.** + +The library's open finding F1 is *"door A has no free-text connector"*. That is real for us, but it is not +our blocker. Our blocker is one level deeper: + +> **Door A reaches sources with deterministic code. This plugin does not fetch anything with code.** + +Every external input in this repo arrives through a *model turn* — the `trend-spotter` agent's +WebSearch/WebFetch/MCP calls, `/linkedin:react`'s WebFetch of an operator-supplied URL. There is no raw +HTTP fetch anywhere in plugin code (verified across `scripts/`). Door A's three connectors (`file`, `sql`, +`http`) all assume code can go get the bytes. For our largest and least-trusted surface, code cannot. + +So the requirement we contribute to F1 is not *"add a free-text connector"* — it is **"add a door that +accepts an already-fetched, model-produced payload and materializes it deterministically."** That door +would immediately fit `trends capture`, which is already exactly this shape (stdin JSON envelope → +validate → deterministic store write). Details in §4. + +Adoption verdict: **`planned`, conditional.** Adopting door A as it stands would mean converting typed, +incrementally-accreting stores into rewrite-on-run markdown documents — a downgrade. We adopt when §4's +four requirements land, not before. + +## 2. Our sources — tabular vs free-text + +| Source | On disk | Shape | Trust boundary | Door-A fit today | +|---|---|---|---|---| +| **Analytics CSV export** → batches | `exports/*.csv` → `posts/-.json` | **Tabular** (header row + rows) | First-party (operator's own LinkedIn export), but an unvalidated parse surface | **Closest fit.** Satisfies `read_csv`'s header requirement. But see §3 — the output shape is wrong. | +| **Trend capture** | `trends/trends.json` (schema v4) | JSON envelope; `title`/`url`/`summary` are **verbatim free text** | **External**, model-mediated (web search/fetch via agent) | **No connector exists.** Not `http` — the fetch already happened, in a model turn. | +| **Morning brief** | `trends/morning-brief/.md` | Structured md, carries external strings forward | Machine-rendered from store; re-injected by SessionStart hook | Destination, not a source. | +| **Brain inbox → published** | `ingest/inbox/*.md` → `ingest/published/.md` | **Free text**, byte-exact body | Operator-dropped today; drop-zone has no origin control | **Excluded by design** — see §5. | +| **Brain profile** | `brain/profile.md` | Line grammar (6 constrained tokens + one-line value) | Machine-folded, operator-gated (`--apply --confirm`) | Destination. Already an OKF-conformant bundle (Stage 1, 2026-06-26). | +| **Specifics-bank** | `specifics-bank/specifics-bank.json` | JSON + verbatim free-text `content` | Operator-written by invariant (never AI-generated lived experience) | Destination. | +| **Contract-gate** | — writes nothing | Reads local operator files only | Local | Not applicable. | + +**Summary:** one genuinely tabular *source* (analytics CSV). One free-text external source that is not +reachable by any connector shape the library has (trend capture). One free-text local source that is +excluded by a fixed decision (published posts). Everything else is a destination store, not an ingestion +input. + +## 3. Where door A's contract collides with ours + +Four structural mismatches, each verified against the library's code. These are not preferences — each one +would break an invariant we hold today. + +**(1) Fixed 7-key frontmatter, no extension keys.** `materialize.py` emits exactly `type`, `title`, +`source_system`, `source_query`, `ingested_at`, `ingest_manifest`, `generated` — in fixed order, all values +collapsed to single lines. Our records carry domain fields that must survive: `provenance`, `status`, +`first_seen`, `last_seen`, `evidence_count` (brain), `score`, `pillar`, `topics`, `surfaced` (trends). +Note the tension with OKF the *format*, which explicitly requires consumers to preserve unknown keys — the +library's materializer is stricter than the spec it implements. Our Stage-1 OKF conformance work +(`docs/okf-convergence-brief.md`) depends on rich fields riding along as extension keys; door A has no +mechanism for that. + +**(2) Rewrite-on-run ownership vs incremental accretion.** Door A owns files via `generated: true` + +`ingest_manifest`, then **deletes every stamped file and rewrites the set** each run. Our stores accrete: +dedupe by content id, topic-union on re-capture, last-wins score, collision-suffix on body divergence, +compare-then-skip on scaffold. A door-A run against our data would be destructive by design. + +**(3) Operator-chosen slug ids vs content-addressed ids.** Library id = the manifest's `extraction.id` +(a hand-written slug, `[a-z0-9][a-z0-9-]*`), with provenance carried by a manifest stamp +(`{stem}@{sha256(manifest bytes)[:16]}`). Ours = `sha256(content)[:12]` — the id *is* the dedupe key, which +is why re-capturing the same trend or re-ingesting the same post is idempotent for free. A slug-keyed +model cannot express "same content, seen twice." + +**(4) Table rendering destroys free text.** `render_table` escapes `\` then `|`, then collapses CRLF/CR/LF +**to a single space**. Any multi-line body loses its line structure. This is fatal for post bodies (§5) and +lossy for trend summaries. + +## 4. What door A / door B must support before we can adopt — the concrete asks + +Ordered by how much they unblock. (1) is the one that decides F1 for us. + +**R1 — A model-mediated door ("already-fetched payload").** Accept a caller-supplied, schema-validated +payload (stdin JSON envelope or in-process record array) instead of a connector-fetched one, and +materialize it with the same determinism, same stamping, same idempotence as door A. The manifest would +declare shape and destination but not a fetch. This is the door that fits how an agentic plugin actually +ingests, and we have a working precedent to donate: `trends capture` is this contract already +(`normalizeItem` → "the one schema downstream never branches on" → deterministic store write). +**Why this over a free-text connector:** a `read_text`/`read_markdown` connector would serve repos whose +free text sits in a folder. Ours sits in a model's tool result. Both are needed; they are not the same ask, +and solving only the folder case leaves us exactly where we are. +*Security note:* this door is precisely where untrusted, model-touched bytes cross into a persisted store, +so it is where a guard call belongs. That wiring is `docs/ingestion-guard/plan.md`'s decision, not this +document's — but the door must at minimum expose a seam for it rather than writing straight through. + +**R2 — Extension keys preserved through materialization.** Let a record carry arbitrary additional +frontmatter keys, emitted after the 7 reserved ones, order-stable, never coerced. Without this, every +domain field we have is lost on the way through the library, and OKF's own "preserve unknown keys" rule is +violated by the tool that writes OKF. + +**R3 — Incremental/upsert materialization mode.** A mode where a run merges into an existing bundle keyed +by record id — add new, update changed, leave untouched what this run did not see — instead of +delete-all-stamped-then-rewrite. Door A's index maintenance already does exactly this kind of careful +merge for `index.md` (managed lines refreshed, curated lines preserved byte-verbatim); the ask is to extend +that discipline from the index to the records. + +**R4 — Content-addressed record ids as a first-class option.** Allow `id = hash(content)` rather than a +manifest slug, so dedupe and idempotence come from the data. Our three stores (brain, trends, +specifics-bank) all independently converged on `sha256(...)[:12]`; the pattern is not LinkedIn-specific. + +**R5 (phase 4, Node half) — contract parity details.** Stable string error codes on the error object +(the Python half's `exc.code` discipline, mirrored so Node consumers assert on `err.code`, never on message +text); zero runtime dependencies; ESM with `node:` prefixes; LF-only output with exactly one trailing +newline; and the golden-fixture set shared across both halves so byte-identity is the test, not the +promise. Our packages already meet the zero-dep/ESM/LF conventions (sole exception: `scripts/analytics` +depends on `csv-parse`), so parity is cheap on our side. + +## 5. What must never move here + +**`ingest/published/` provenance grammar stays plugin-local — fixed decision, and it has a technical +floor.** This is not merely a scope preference: + +- The store holds a **byte-exact round-trip invariant** (`parse ∘ serialize = identity`, SC2) on the + verbatim post body. Door A's table renderer collapses newlines to spaces (§3.4); its frontmatter + collapses whitespace runs. Either would break the invariant on contact. +- The record id is `mintContentId(verbatim body)` — deliberately **un-normalized**, so two + structurally-different posts never collide and the write path never silently drops a differing body. +- The grammar is deliberately YAML-free (a fixed 5-line header + `\n---\n` sentinel), which is also why our + own Stage-1 OKF conformance work scoped the concept-bundle to `brain/` and **excluded the `ingest/` + tributary** — verified 2026-06-26, `okf-check.mjs` exit 0 on `brain/`. +- `provenance=published` carries **model-collapse-guard semantics**: the voice/profile learning surface + learns from human-published content only, never from `ai-draft`. The *form* is generic; the *guarantee* + is domain policy and belongs where the policy is enforced. + +We are happy to describe the interface (this section is that description). We are not planning to hand it +over. + +**Also staying — LinkedIn domain logic:** trend scoring weights and mode SSOT +(`references/trend-scoring-modes.md`); brief ranking, pillar logic and Norwegian rendering; the +analytics↔post join heuristic (title-prefix matching with the 110-char hook rule behind `PREFIX_FLOOR`); +CSV column fuzzing against LinkedIn's export column names, the `engagementRate` formula and `saves` +semantics; the specifics taxonomy and its verification rules; contract-gate's writing-contract rules. + +**Candidates for sharing (bundle mechanics, if the Node half wants them):** content-addressed id minting; +dedupe + tag-union upsert; migrate-on-load `schemaVersion` stamping; collision-safe idempotent writes +(compare-then-skip, collision-suffix); path-traversal-safe filename resolution (our `storage.ts` twin of +the library's fail-closed `safe_resolve`); managed-line index maintenance. Several of these modules already +declare themselves generic by architecture. Note we carry the data-root resolver in **five** inlined copies +(four `scripts/*` packages + a zero-dep hooks twin, synchrony guarded by test) — deliberate, since the +*path convention* is plugin-local even where the *idiom* is not; a shared library should take the idiom and +leave the path. + +## 6. Boundary against the guard plan + +`docs/ingestion-guard/plan.md` identifies four persist gates: trends `capture`, brain +`writePublished`/`scanInbox`, newsletter→`ekstern` bindings, analytics `saveBatch`. Two of those (trends +capture, analytics import) reappear here as adoption candidates — unavoidably, since they are the same +boundary viewed from two sides. + +The split: **the guard plan decides whether bytes are safe to persist; this plan decides what writes them +and in what shape.** They share a dependency in one place only — R1's seam is where a guard call would +land — and that is noted as a seam requirement, not a security decision. Neither plan is a prerequisite for +the other's approval. The guard plan's own interop blocker (Python↔Node) is tracked there. + +## 7. Open questions for the library owners + +1. Does R1 (model-mediated payload door) belong to door B, or is it a fourth door? It is neither an inbox + scan nor an external bundle import. +2. Is R2 (extension keys) a spec change via commons, or a materializer relaxation within the current spec? + OKF v0.1 already mandates preserving unknown keys, which suggests the latter. +3. For phase 4: is the Node half expected to reach parity with door A only, or with whatever doors exist + when it starts? Our adoption depends on R1, which is not door A. + +## 8. References + +- Library: `https://git.fromaitochitta.com/open/llm-ingestion-okf` +- Guard: `https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security` +- Spec owner: `portfolio-optimiser-commons/ingest-spec.md` +- This repo: `docs/okf-convergence-brief.md` (Stage-1 OKF conformance, `brain/` bundle), + `docs/ingestion-guard/plan.md` (separate, `planned`), `docs/second-brain/architecture.md`