docs(linkedin-studio): OKF-ingestion kartlegging + kravgrunnlag mot llm-ingestion-okf fase 4

Kartlegger repoets inntaksflater mot dor A i llm-ingestion-okf v0.3.1 og
leverer kravgrunnlaget som avgjor F1 for dette repoet.

Kjernefunn: F1 (manglende fritekst-connector) er ikke var blokker. All
ekstern henting i pluginen skjer i en modell-tur (trend-spotter WebSearch/
WebFetch, /linkedin:react WebFetch) - det finnes ingen rah HTTP-fetch i
plugin-kode. Dor A forutsetter at kode kan hente bytene; for var storste
og minst betrodde flate kan den ikke det. Kravet vart er derfor R1: en
dor som tar imot allerede-hentet, modellprodusert payload og
materialiserer den deterministisk. trends capture er den kontrakten
allerede, og donerbar som presedens.

Fire verifiserte kollisjoner mot dor A: fast 7-nokkels frontmatter uten
extension keys; rewrite-on-run-eierskap mot inkrementell akkresjon;
slug-ids mot content-adresserte ids; tabell-render som kollapser
linjeskift.

ingest/published/ forblir plugin-lokal ved design - na med teknisk gulv,
ikke bare beslutning: byte-eksakt round-trip (SC2), un-normalisert
mintContentId(body), YAML-fri grammatikk (samme grunn som Stage-1-bundlen
ekskluderte ingest/), og model-collapse-guard-semantikk.

Markorlinje satt i STATE.md (local-only): planned.
Separat fra docs/ingestion-guard/plan.md - grensen er beskrevet i par. 6.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019uzhvLjCm39mNQXVFwG7w2

docs/okf-ingestion/plan.md

@@ -0,0 +1,191 @@
+# OKF-ingestion plan — mapping this repo against `llm-ingestion-okf` (phase 4)
+
+> **Status: `planned` (2026-07-20).** Survey + requirements only — nothing wired, nothing built. The
+> library's Node half does not exist yet (its phase 4), so there is nothing to adopt today. This document
+> is the repo's durable answer to *what door A or door B must support before this plugin can adopt*, and
+> *what must never move into the shared library*.
+>
+> **Library:** `llm-ingestion-okf` v0.3.1 — Python 3.10+, zero runtime deps, **door A only**
+> (manifest → connector → deterministic materialization of `ingest-{id}.md` → index generation, no model
+> calls). Doors B (inbox) and C (external bundle import) are phase 2; the Node half is phase 4.
+> **Spec:** `ingest-spec.md`, owned by `portfolio-optimiser-commons` — changes go via commons, never locally.
+> **Security boundary:** `llm-ingestion-guard` (repo `llm-ingestion-pipeline-security`) owns all security.
+> Door A is **ungated** — it calls no guard function before writing to disk (verified: no guard import or
+> call site anywhere in the library's `src/`; `dependencies = []`). "Security is delegated" does **not**
+> mean "safe by default"; gating is the caller's responsibility.
+>
+> **Separate, do not merge:** `docs/ingestion-guard/plan.md` (`planned`) is this repo's *security* wiring
+> plan. It shares persist points with this document by necessity, but is a distinct decision with a
+> distinct dependency. §6 states the boundary.
+
+## 1. Executive answer
+
+**Nothing here is a clean door-A fit today, and the reason is not the one the library expects.**
+
+The library's open finding F1 is *"door A has no free-text connector"*. That is real for us, but it is not
+our blocker. Our blocker is one level deeper:
+
+> **Door A reaches sources with deterministic code. This plugin does not fetch anything with code.**
+
+Every external input in this repo arrives through a *model turn* — the `trend-spotter` agent's
+WebSearch/WebFetch/MCP calls, `/linkedin:react`'s WebFetch of an operator-supplied URL. There is no raw
+HTTP fetch anywhere in plugin code (verified across `scripts/`). Door A's three connectors (`file`, `sql`,
+`http`) all assume code can go get the bytes. For our largest and least-trusted surface, code cannot.
+
+So the requirement we contribute to F1 is not *"add a free-text connector"* — it is **"add a door that
+accepts an already-fetched, model-produced payload and materializes it deterministically."** That door
+would immediately fit `trends capture`, which is already exactly this shape (stdin JSON envelope →
+validate → deterministic store write). Details in §4.
+
+Adoption verdict: **`planned`, conditional.** Adopting door A as it stands would mean converting typed,
+incrementally-accreting stores into rewrite-on-run markdown documents — a downgrade. We adopt when §4's
+four requirements land, not before.
+
+## 2. Our sources — tabular vs free-text
+
+| Source | On disk | Shape | Trust boundary | Door-A fit today |
+|---|---|---|---|---|
+| **Analytics CSV export** → batches | `exports/*.csv` → `posts/<date>-<id>.json` | **Tabular** (header row + rows) | First-party (operator's own LinkedIn export), but an unvalidated parse surface | **Closest fit.** Satisfies `read_csv`'s header requirement. But see §3 — the output shape is wrong. |
+| **Trend capture** | `trends/trends.json` (schema v4) | JSON envelope; `title`/`url`/`summary` are **verbatim free text** | **External**, model-mediated (web search/fetch via agent) | **No connector exists.** Not `http` — the fetch already happened, in a model turn. |
+| **Morning brief** | `trends/morning-brief/<date>.md` | Structured md, carries external strings forward | Machine-rendered from store; re-injected by SessionStart hook | Destination, not a source. |
+| **Brain inbox → published** | `ingest/inbox/*.md` → `ingest/published/<id>.md` | **Free text**, byte-exact body | Operator-dropped today; drop-zone has no origin control | **Excluded by design** — see §5. |
+| **Brain profile** | `brain/profile.md` | Line grammar (6 constrained tokens + one-line value) | Machine-folded, operator-gated (`--apply --confirm`) | Destination. Already an OKF-conformant bundle (Stage 1, 2026-06-26). |
+| **Specifics-bank** | `specifics-bank/specifics-bank.json` | JSON + verbatim free-text `content` | Operator-written by invariant (never AI-generated lived experience) | Destination. |
+| **Contract-gate** | — writes nothing | Reads local operator files only | Local | Not applicable. |
+
+**Summary:** one genuinely tabular *source* (analytics CSV). One free-text external source that is not
+reachable by any connector shape the library has (trend capture). One free-text local source that is
+excluded by a fixed decision (published posts). Everything else is a destination store, not an ingestion
+input.
+
+## 3. Where door A's contract collides with ours
+
+Four structural mismatches, each verified against the library's code. These are not preferences — each one
+would break an invariant we hold today.
+
+**(1) Fixed 7-key frontmatter, no extension keys.** `materialize.py` emits exactly `type`, `title`,
+`source_system`, `source_query`, `ingested_at`, `ingest_manifest`, `generated` — in fixed order, all values
+collapsed to single lines. Our records carry domain fields that must survive: `provenance`, `status`,
+`first_seen`, `last_seen`, `evidence_count` (brain), `score`, `pillar`, `topics`, `surfaced` (trends).
+Note the tension with OKF the *format*, which explicitly requires consumers to preserve unknown keys — the
+library's materializer is stricter than the spec it implements. Our Stage-1 OKF conformance work
+(`docs/okf-convergence-brief.md`) depends on rich fields riding along as extension keys; door A has no
+mechanism for that.
+
+**(2) Rewrite-on-run ownership vs incremental accretion.** Door A owns files via `generated: true` +
+`ingest_manifest`, then **deletes every stamped file and rewrites the set** each run. Our stores accrete:
+dedupe by content id, topic-union on re-capture, last-wins score, collision-suffix on body divergence,
+compare-then-skip on scaffold. A door-A run against our data would be destructive by design.
+
+**(3) Operator-chosen slug ids vs content-addressed ids.** Library id = the manifest's `extraction.id`
+(a hand-written slug, `[a-z0-9][a-z0-9-]*`), with provenance carried by a manifest stamp
+(`{stem}@{sha256(manifest bytes)[:16]}`). Ours = `sha256(content)[:12]` — the id *is* the dedupe key, which
+is why re-capturing the same trend or re-ingesting the same post is idempotent for free. A slug-keyed
+model cannot express "same content, seen twice."
+
+**(4) Table rendering destroys free text.** `render_table` escapes `\` then `|`, then collapses CRLF/CR/LF
+**to a single space**. Any multi-line body loses its line structure. This is fatal for post bodies (§5) and
+lossy for trend summaries.
+
+## 4. What door A / door B must support before we can adopt — the concrete asks
+
+Ordered by how much they unblock. (1) is the one that decides F1 for us.
+
+**R1 — A model-mediated door ("already-fetched payload").** Accept a caller-supplied, schema-validated
+payload (stdin JSON envelope or in-process record array) instead of a connector-fetched one, and
+materialize it with the same determinism, same stamping, same idempotence as door A. The manifest would
+declare shape and destination but not a fetch. This is the door that fits how an agentic plugin actually
+ingests, and we have a working precedent to donate: `trends capture` is this contract already
+(`normalizeItem` → "the one schema downstream never branches on" → deterministic store write).
+**Why this over a free-text connector:** a `read_text`/`read_markdown` connector would serve repos whose
+free text sits in a folder. Ours sits in a model's tool result. Both are needed; they are not the same ask,
+and solving only the folder case leaves us exactly where we are.
+*Security note:* this door is precisely where untrusted, model-touched bytes cross into a persisted store,
+so it is where a guard call belongs. That wiring is `docs/ingestion-guard/plan.md`'s decision, not this
+document's — but the door must at minimum expose a seam for it rather than writing straight through.
+
+**R2 — Extension keys preserved through materialization.** Let a record carry arbitrary additional
+frontmatter keys, emitted after the 7 reserved ones, order-stable, never coerced. Without this, every
+domain field we have is lost on the way through the library, and OKF's own "preserve unknown keys" rule is
+violated by the tool that writes OKF.
+
+**R3 — Incremental/upsert materialization mode.** A mode where a run merges into an existing bundle keyed
+by record id — add new, update changed, leave untouched what this run did not see — instead of
+delete-all-stamped-then-rewrite. Door A's index maintenance already does exactly this kind of careful
+merge for `index.md` (managed lines refreshed, curated lines preserved byte-verbatim); the ask is to extend
+that discipline from the index to the records.
+
+**R4 — Content-addressed record ids as a first-class option.** Allow `id = hash(content)` rather than a
+manifest slug, so dedupe and idempotence come from the data. Our three stores (brain, trends,
+specifics-bank) all independently converged on `sha256(...)[:12]`; the pattern is not LinkedIn-specific.
+
+**R5 (phase 4, Node half) — contract parity details.** Stable string error codes on the error object
+(the Python half's `exc.code` discipline, mirrored so Node consumers assert on `err.code`, never on message
+text); zero runtime dependencies; ESM with `node:` prefixes; LF-only output with exactly one trailing
+newline; and the golden-fixture set shared across both halves so byte-identity is the test, not the
+promise. Our packages already meet the zero-dep/ESM/LF conventions (sole exception: `scripts/analytics`
+depends on `csv-parse`), so parity is cheap on our side.
+
+## 5. What must never move here
+
+**`ingest/published/` provenance grammar stays plugin-local — fixed decision, and it has a technical
+floor.** This is not merely a scope preference:
+
+- The store holds a **byte-exact round-trip invariant** (`parse ∘ serialize = identity`, SC2) on the
+  verbatim post body. Door A's table renderer collapses newlines to spaces (§3.4); its frontmatter
+  collapses whitespace runs. Either would break the invariant on contact.
+- The record id is `mintContentId(verbatim body)` — deliberately **un-normalized**, so two
+  structurally-different posts never collide and the write path never silently drops a differing body.
+- The grammar is deliberately YAML-free (a fixed 5-line header + `\n---\n` sentinel), which is also why our
+  own Stage-1 OKF conformance work scoped the concept-bundle to `brain/` and **excluded the `ingest/`
+  tributary** — verified 2026-06-26, `okf-check.mjs` exit 0 on `brain/`.
+- `provenance=published` carries **model-collapse-guard semantics**: the voice/profile learning surface
+  learns from human-published content only, never from `ai-draft`. The *form* is generic; the *guarantee*
+  is domain policy and belongs where the policy is enforced.
+
+We are happy to describe the interface (this section is that description). We are not planning to hand it
+over.
+
+**Also staying — LinkedIn domain logic:** trend scoring weights and mode SSOT
+(`references/trend-scoring-modes.md`); brief ranking, pillar logic and Norwegian rendering; the
+analytics↔post join heuristic (title-prefix matching with the 110-char hook rule behind `PREFIX_FLOOR`);
+CSV column fuzzing against LinkedIn's export column names, the `engagementRate` formula and `saves`
+semantics; the specifics taxonomy and its verification rules; contract-gate's writing-contract rules.
+
+**Candidates for sharing (bundle mechanics, if the Node half wants them):** content-addressed id minting;
+dedupe + tag-union upsert; migrate-on-load `schemaVersion` stamping; collision-safe idempotent writes
+(compare-then-skip, collision-suffix); path-traversal-safe filename resolution (our `storage.ts` twin of
+the library's fail-closed `safe_resolve`); managed-line index maintenance. Several of these modules already
+declare themselves generic by architecture. Note we carry the data-root resolver in **five** inlined copies
+(four `scripts/*` packages + a zero-dep hooks twin, synchrony guarded by test) — deliberate, since the
+*path convention* is plugin-local even where the *idiom* is not; a shared library should take the idiom and
+leave the path.
+
+## 6. Boundary against the guard plan
+
+`docs/ingestion-guard/plan.md` identifies four persist gates: trends `capture`, brain
+`writePublished`/`scanInbox`, newsletter→`ekstern` bindings, analytics `saveBatch`. Two of those (trends
+capture, analytics import) reappear here as adoption candidates — unavoidably, since they are the same
+boundary viewed from two sides.
+
+The split: **the guard plan decides whether bytes are safe to persist; this plan decides what writes them
+and in what shape.** They share a dependency in one place only — R1's seam is where a guard call would
+land — and that is noted as a seam requirement, not a security decision. Neither plan is a prerequisite for
+the other's approval. The guard plan's own interop blocker (Python↔Node) is tracked there.
+
+## 7. Open questions for the library owners
+
+1. Does R1 (model-mediated payload door) belong to door B, or is it a fourth door? It is neither an inbox
+   scan nor an external bundle import.
+2. Is R2 (extension keys) a spec change via commons, or a materializer relaxation within the current spec?
+   OKF v0.1 already mandates preserving unknown keys, which suggests the latter.
+3. For phase 4: is the Node half expected to reach parity with door A only, or with whatever doors exist
+   when it starts? Our adoption depends on R1, which is not door A.
+
+## 8. References
+
+- Library: `https://git.fromaitochitta.com/open/llm-ingestion-okf`
+- Guard: `https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security`
+- Spec owner: `portfolio-optimiser-commons/ingest-spec.md`
+- This repo: `docs/okf-convergence-brief.md` (Stage-1 OKF conformance, `brain/` bundle),
+  `docs/ingestion-guard/plan.md` (separate, `planned`), `docs/second-brain/architecture.md`