linkedin-studio/docs/okf-ingestion/plan.md

# OKF-ingestion plan — mapping this repo against `llm-ingestion-okf` (phase 4)

> **Status: `planned` (2026-07-20).** Survey + requirements only — nothing wired, nothing built. The
> library's Node half does not exist yet (its phase 4), so there is nothing to adopt today. This document
> is the repo's durable answer to *what door A or door B must support before this plugin can adopt*, and
> *what must never move into the shared library*.
>
> **Library:** `llm-ingestion-okf` v0.3.1 — Python 3.10+, zero runtime deps, **door A only**
> (manifest → connector → deterministic materialization of `ingest-{id}.md` → index generation, no model
> calls). Doors B (inbox) and C (external bundle import) are phase 2; the Node half is phase 4.
> **Spec:** `ingest-spec.md`, owned by `portfolio-optimiser-commons` — changes go via commons, never locally.
> **Security boundary:** `llm-ingestion-guard` (repo `llm-ingestion-pipeline-security`) owns all security.
> Door A is **ungated** — it calls no guard function before writing to disk (verified: no guard import or
> call site anywhere in the library's `src/`; `dependencies = []`). "Security is delegated" does **not**
> mean "safe by default"; gating is the caller's responsibility.
>
> **Separate, do not merge:** `docs/ingestion-guard/plan.md` (`planned`) is this repo's *security* wiring
> plan. It shares persist points with this document by necessity, but is a distinct decision with a
> distinct dependency. §7 states the boundary.

## 1. Executive answer

**Nothing here is a clean door-A fit today, and the reason is not the one the library expects.**

The library's open finding F1 is *"door A has no free-text connector"*. That is real for us, but it is not
our blocker. Our blocker is one level deeper:

> **Door A reaches sources with deterministic code. This plugin does not fetch anything with code.**

Every external input in this repo arrives through a *model turn* — the `trend-spotter` agent's
WebSearch/WebFetch/MCP calls, `/linkedin:react`'s WebFetch of an operator-supplied URL. There is no raw
HTTP fetch anywhere in plugin code (verified across `scripts/`). Door A's three connectors (`file`, `sql`,
`http`) all assume code can go get the bytes. For our largest and least-trusted surface, code cannot.

So the requirement we contribute to F1 is not *"add a free-text connector"* — it is **"add a door that
accepts an already-fetched, model-produced payload and materializes it deterministically."** That door
would immediately fit `trends capture`, which is already exactly this shape (stdin JSON envelope →
validate → deterministic store write). Details in §4.

Adoption verdict: **`planned`, conditional.** Adopting door A as it stands would mean converting typed,
incrementally-accreting stores into rewrite-on-run markdown documents — a downgrade. We adopt when §4's
four requirements land, not before.

## 2. Our sources — tabular vs free-text

| Source | On disk | Shape | Trust boundary | Door-A fit today |
|---|---|---|---|---|
| **Analytics CSV export** → batches | `exports/*.csv` → `posts/<date>-<id>.json` | **Tabular** (header row + rows) | First-party (operator's own LinkedIn export), but an unvalidated parse surface | **Closest fit.** Satisfies `read_csv`'s header requirement. But see §3 — the output shape is wrong. |
| **Trend capture** | `trends/trends.json` (schema v4) | JSON envelope; `title`/`url`/`summary` are **verbatim free text** | **External**, model-mediated (web search/fetch via agent) | **No connector exists.** Not `http` — the fetch already happened, in a model turn. |
| **Morning brief** | `trends/morning-brief/<date>.md` | Structured md, carries external strings forward | Machine-rendered from store; re-injected by SessionStart hook | Destination, not a source. |
| **Brain inbox → published** | `ingest/inbox/*.md` → `ingest/published/<id>.md` | **Free text**, byte-exact body | Operator-dropped today; drop-zone has no origin control | **Excluded by design** — see §5. |
| **Brain profile** | `brain/profile.md` | Line grammar (6 constrained tokens + one-line value) | Machine-folded, operator-gated (`--apply --confirm`) | Destination. Already an OKF-conformant bundle (Stage 1, 2026-06-26). |
| **Specifics-bank** | `specifics-bank/specifics-bank.json` | JSON + verbatim free-text `content` | Operator-written by invariant (never AI-generated lived experience) | Destination. |
| **Contract-gate** | — writes nothing | Reads local operator files only | Local | Not applicable. |

**Summary:** one genuinely tabular *source* (analytics CSV). One free-text external source that is not
reachable by any connector shape the library has (trend capture). One free-text local source that is
excluded by a fixed decision (published posts). Everything else is a destination store, not an ingestion
input.

## 3. Where door A's contract collides with ours

Four structural mismatches, each verified against the library's code. These are not preferences — each one
would break an invariant we hold today.

**(1) Fixed 7-key frontmatter, no extension keys.** `materialize.py` emits exactly `type`, `title`,
`source_system`, `source_query`, `ingested_at`, `ingest_manifest`, `generated` — in fixed order, all values
collapsed to single lines. Our records carry domain fields that must survive: `provenance`, `status`,
`first_seen`, `last_seen`, `evidence_count` (brain), `score`, `pillar`, `topics`, `surfaced` (trends).
Note the tension with OKF the *format*, which explicitly requires consumers to preserve unknown keys — the
library's materializer is stricter than the spec it implements. Our Stage-1 OKF conformance work
(`docs/okf-convergence-brief.md`) depends on rich fields riding along as extension keys; door A has no
mechanism for that.

**(2) Rewrite-on-run ownership vs incremental accretion.** Door A owns files via `generated: true` +
`ingest_manifest`, then **deletes every stamped file and rewrites the set** each run. Our stores accrete:
dedupe by content id, topic-union on re-capture, last-wins score, collision-suffix on body divergence,
compare-then-skip on scaffold. A door-A run against our data would be destructive by design.

**(3) Operator-chosen slug ids vs content-addressed ids.** Library id = the manifest's `extraction.id`
(a hand-written slug, `[a-z0-9][a-z0-9-]*`), with provenance carried by a manifest stamp
(`{stem}@{sha256(manifest bytes)[:16]}`). Ours = `sha256(content)[:12]` — the id *is* the dedupe key, which
is why re-capturing the same trend or re-ingesting the same post is idempotent for free. A slug-keyed
model cannot express "same content, seen twice."

**(4) Table rendering destroys free text.** `render_table` escapes `\` then `|`, then collapses CRLF/CR/LF
**to a single space**. Any multi-line body loses its line structure. This is fatal for post bodies (§5) and
lossy for trend summaries.

## 4. What door A / door B must support before we can adopt — the concrete asks

Ordered by how much they unblock. (1) is the one that decides F1 for us.

**R1 — A model-mediated door ("already-fetched payload").** Accept a caller-supplied, schema-validated
payload (stdin JSON envelope or in-process record array) instead of a connector-fetched one, and
materialize it with the same determinism, same stamping, same idempotence as door A. The manifest would
declare shape and destination but not a fetch. This is the door that fits how an agentic plugin actually
ingests, and we have a working precedent to donate: `trends capture` is this contract already
(`normalizeItem` → "the one schema downstream never branches on" → deterministic store write).
**Why this over a free-text connector:** a `read_text`/`read_markdown` connector would serve repos whose
free text sits in a folder. Ours sits in a model's tool result. Both are needed; they are not the same ask,
and solving only the folder case leaves us exactly where we are.
*Security note:* this door is precisely where untrusted, model-touched bytes cross into a persisted store,
so it is where a guard call belongs. That wiring is `docs/ingestion-guard/plan.md`'s decision, not this
document's — but the door must at minimum expose a seam for it rather than writing straight through.

**R2 — Extension keys preserved through materialization.** Let a record carry arbitrary additional
frontmatter keys, emitted after the 7 reserved ones, order-stable, never coerced. Without this, every
domain field we have is lost on the way through the library, and OKF's own "preserve unknown keys" rule is
violated by the tool that writes OKF.

**R3 — Incremental/upsert materialization mode.** A mode where a run merges into an existing bundle keyed
by record id — add new, update changed, leave untouched what this run did not see — instead of
delete-all-stamped-then-rewrite. Door A's index maintenance already does exactly this kind of careful
merge for `index.md` (managed lines refreshed, curated lines preserved byte-verbatim); the ask is to extend
that discipline from the index to the records.

**R4 — Content-addressed record ids as a first-class option.** Allow `id = hash(content)` rather than a
manifest slug, so dedupe and idempotence come from the data. Our three stores (brain, trends,
specifics-bank) all independently converged on `sha256(...)[:12]`; the pattern is not LinkedIn-specific.

**R5 (phase 4, Node half) — contract parity details.** Stable string error codes on the error object
(the Python half's `exc.code` discipline, mirrored so Node consumers assert on `err.code`, never on message
text); zero runtime dependencies; ESM with `node:` prefixes; and the golden-fixture set shared across both
halves so byte-identity is the test, not the promise.

> **Correction, 2026-07-20 (trinn D-review).** An earlier revision of this section claimed we "already meet
> the zero-dep/ESM/LF conventions". Verified against the code, two parts of that were false:
>
> - **We have no stable error codes.** None of the four CLIs set a machine-readable code; every failure is a
>   free-text `Error` plus a numeric `process.exit` (`scripts/brain/src/ingest.ts:99,:113`,
>   `scripts/trends/src/cli.ts:125,:255`). `err.code` is an ask *of* the library, not a discipline we mirror.
> - **"Exactly one trailing newline" is not asserted here either.** Nothing tests it, and
>   `serializePublishedRecord` deliberately omits it (`scripts/brain/src/ingest.ts:71`) because the record
>   must end on the verbatim body or the byte-exact round-trip (SC2, §5) breaks.
>
> **Second correction, same day, narrowing the one above.** The trinn D-review first read this as a
> conflict with `claude-playlist-corpus`, who asked the library to own frame normalization (LF-only, one
> blank line, trailing newline). It is not a conflict, and the check should have come first: our
> newline-free serializer governs `ingest/published/`, which is **excluded from the OKF bundle by design**
> (§5, `scripts/brain/tests/okf-conform.test.ts:22-27`). The files `write_concept` would actually write for
> us — the `brain/` bundle — get their trailing newline from our own serializer
> (`scripts/brain/src/profile.ts:65`). On the concept-writing path we agree with them. What survives is
> narrower: `write_concept` is also being asked to serve as a *general* verbatim writer callable outside a
> bundle (`ms-ai-architect`, `po-claude`), so frame normalization should be a documented parameter rather
> than an unconditional guarantee. The fixture ask applies to that general writer, not to the concept corpus.
>
> Zero-dep/ESM does hold (sole exception: `scripts/analytics` depends on `csv-parse`).

## 5. What must never move here

**`ingest/published/` provenance grammar stays plugin-local — fixed decision, and it has a technical
floor.** This is not merely a scope preference:

- The store holds a **byte-exact round-trip invariant** (`parse ∘ serialize = identity`, SC2) on the
  verbatim post body. Door A's table renderer collapses newlines to spaces (§3.4); its frontmatter
  collapses whitespace runs. Either would break the invariant on contact.
- The record id is `mintContentId(verbatim body)` — deliberately **un-normalized**, so two
  structurally-different posts never collide and the write path never silently drops a differing body.
- The grammar is deliberately YAML-free (a fixed 5-line header + `\n---\n` sentinel), which is also why our
  own Stage-1 OKF conformance work scoped the concept-bundle to `brain/` and **excluded the `ingest/`
  tributary** — verified 2026-06-26, `okf-check.mjs` exit 0 on `brain/`.
- `provenance=published` carries **model-collapse-guard semantics**: the voice/profile learning surface
  learns from human-published content only, never from `ai-draft`. The *form* is generic; the *guarantee*
  is domain policy and belongs where the policy is enforced.

We are happy to describe the interface (this section is that description). We are not planning to hand it
over.

**Also staying — LinkedIn domain logic:** trend scoring weights and mode SSOT
(`references/trend-scoring-modes.md`); brief ranking, pillar logic and Norwegian rendering; the
analytics↔post join heuristic (title-prefix matching with the 110-char hook rule behind `PREFIX_FLOOR`);
CSV column fuzzing against LinkedIn's export column names, the `engagementRate` formula and `saves`
semantics; the specifics taxonomy and its verification rules; contract-gate's writing-contract rules.

**Candidates for sharing (bundle mechanics, if the Node half wants them):** content-addressed id minting;
dedupe + tag-union upsert; migrate-on-load `schemaVersion` stamping; collision-safe idempotent writes
(compare-then-skip, collision-suffix); path-traversal-safe filename resolution (our `storage.ts` twin of
the library's fail-closed `safe_resolve`); managed-line index maintenance. Several of these modules already
declare themselves generic by architecture. Note we carry the data-root resolver in **five** inlined copies
(four `scripts/*` packages + a zero-dep hooks twin, synchrony guarded by test) — deliberate, since the
*path convention* is plugin-local even where the *idiom* is not; a shared library should take the idiom and
leave the path.

## 6. Bundle inventory and placement

Operator decision, 2026-07-20: **user-owned OKF bundles live outside the plugin/repo tree and are only
referenced from it.** Plugin-generated bundles that are part of the plugin's own delivery stay.

**We already comply — the migration happened in M0 (v0.6.0), before the decision existed.** Every
user-owned artifact resolves through `${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/`
(`references/data-path-convention.md`), with an idempotent session-start migration. Nothing user-owned has
lived in the repo tree since.

| Artifact | Path | OKF bundle? | Class |
|---|---|---|---|
| **Brain** | `<data-dir>/brain/` | **Yes** — the repo's only OKF bundle (`type` + per-level `index.md` + root `okf_version: 0.1`; `okf-check.mjs` exit 0, 2026-06-26) | **User-owned** — see boundary case below |
| Published-post tributary | `<data-dir>/ingest/{inbox,published}/` | No — deliberately excluded (§5) | User-owned |
| Trend store | `<data-dir>/trends/trends.json` | No — a single JSON file, not a markdown bundle | User-owned |
| Morning briefs | `<data-dir>/trends/morning-brief/<date>.md` | No — dated briefs with frontmatter, no `index.md`, not concept files | User-owned |
| Specifics-bank | `<data-dir>/specifics-bank/specifics-bank.json` | No | User-owned |
| Analytics batches | `<analytics-root>/posts/*.json` | No | User-owned |
| Reference docs (28) | `references/*.md` **in repo** | No — no `type`, no `index.md` | **Plugin-owned**, ships with the plugin, correctly in-tree |

**Nothing breaks on migration, because there is nothing left to migrate.** The one caveat is historical:
pre-M0 installs kept data in-tree, and the session-start migration already handles that path. A second
caveat worth naming: `assets/drafts/queue.json` and `assets/analytics/` remain in-tree as gitignored
scratch — they are not bundles, but they are the last in-tree paths that hold user bytes, and they should
be revisited if the decision is ever tightened to "no user bytes in-tree at all."

### The boundary case — and it says the taxonomy's axis is wrong

The decision's table discriminates on **who writes**: plugin writes → plugin-owned → lives in the repo.
Our brain is **written by the plugin** (`consolidate`/fold, operator-gated at `--apply --confirm`) and
never hand-authored. By that table it is plugin-owned and belongs in the repo. **That conclusion is
exactly wrong** — it is the single most personal artifact we hold, it must survive plugin upgrade and
uninstall, and the repo is publicly distributed.

The discriminator that gives the right answer everywhere in our inventory is **whose lifecycle the content
follows**, not whose hand writes the bytes. Our brain is machine-written and user-owned, and that
combination is not a rare corner — it is what any learning system produces. We suggest the axis be
lifecycle/ownership, with authorship as a non-determinative attribute. Otherwise every plugin that
*derives* user knowledge lands on the wrong side of a table it read correctly.

## 7. Boundary against the guard plan

`docs/ingestion-guard/plan.md` identifies four persist gates: trends `capture`, brain
`writePublished`/`scanInbox`, newsletter→`ekstern` bindings, analytics `saveBatch`. Two of those (trends
capture, analytics import) reappear here as adoption candidates — unavoidably, since they are the same
boundary viewed from two sides.

The split: **the guard plan decides whether bytes are safe to persist; this plan decides what writes them
and in what shape.** They share a dependency in one place only — R1's seam is where a guard call would
land — and that is noted as a seam requirement, not a security decision. Neither plan is a prerequisite for
the other's approval. The guard plan's own interop blocker (Python↔Node) is tracked there.

## 8. Open questions for the library owners

1. Does R1 (model-mediated payload door) belong to door B, or is it a fourth door? It is neither an inbox
   scan nor an external bundle import.
2. Is R2 (extension keys) a spec change via commons, or a materializer relaxation within the current spec?
   OKF v0.1 already mandates preserving unknown keys, which suggests the latter.
3. For phase 4: is the Node half expected to reach parity with door A only, or with whatever doors exist
   when it starts? Our adoption depends on R1, which is not door A.

## 9. References

- Library: `https://git.fromaitochitta.com/open/llm-ingestion-okf`
- Guard: `https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security`
- Spec owner: `portfolio-optimiser-commons/ingest-spec.md`
- This repo: `docs/okf-convergence-brief.md` (Stage-1 OKF conformance, `brain/` bundle),
  `docs/ingestion-guard/plan.md` (separate, `planned`), `docs/second-brain/architecture.md`