linkedin-studio/docs/okf-ingestion/plan.md

OKF-ingestion plan — mapping this repo against llm-ingestion-okf (phase 4)

Status: planned (2026-07-20). Survey + requirements only — nothing wired, nothing built. The library's Node half does not exist yet (its phase 4), so there is nothing to adopt today. This document is the repo's durable answer to what door A or door B must support before this plugin can adopt, and what must never move into the shared library.

Library: llm-ingestion-okf v0.3.1 — Python 3.10+, zero runtime deps, door A only (manifest → connector → deterministic materialization of ingest-{id}.md → index generation, no model calls). Doors B (inbox) and C (external bundle import) are phase 2; the Node half is phase 4. Spec: ingest-spec.md, owned by portfolio-optimiser-commons — changes go via commons, never locally. Security boundary: llm-ingestion-guard (repo llm-ingestion-pipeline-security) owns all security. Door A is ungated — it calls no guard function before writing to disk (verified: no guard import or call site anywhere in the library's src/; dependencies = []). "Security is delegated" does not mean "safe by default"; gating is the caller's responsibility.

Separate, do not merge: docs/ingestion-guard/plan.md (planned) is this repo's security wiring plan. It shares persist points with this document by necessity, but is a distinct decision with a distinct dependency. §6 states the boundary.

1. Executive answer

Nothing here is a clean door-A fit today, and the reason is not the one the library expects.

The library's open finding F1 is "door A has no free-text connector". That is real for us, but it is not our blocker. Our blocker is one level deeper:

Door A reaches sources with deterministic code. This plugin does not fetch anything with code.

Every external input in this repo arrives through a model turn — the trend-spotter agent's WebSearch/WebFetch/MCP calls, /linkedin:react's WebFetch of an operator-supplied URL. There is no raw HTTP fetch anywhere in plugin code (verified across scripts/). Door A's three connectors (file, sql, http) all assume code can go get the bytes. For our largest and least-trusted surface, code cannot.

So the requirement we contribute to F1 is not "add a free-text connector" — it is "add a door that accepts an already-fetched, model-produced payload and materializes it deterministically." That door would immediately fit trends capture, which is already exactly this shape (stdin JSON envelope → validate → deterministic store write). Details in §4.

Adoption verdict: planned, conditional. Adopting door A as it stands would mean converting typed, incrementally-accreting stores into rewrite-on-run markdown documents — a downgrade. We adopt when §4's four requirements land, not before.

2. Our sources — tabular vs free-text

Source	On disk	Shape	Trust boundary	Door-A fit today
Analytics CSV export → batches	exports/*.csv → posts/<date>-<id>.json	Tabular (header row + rows)	First-party (operator's own LinkedIn export), but an unvalidated parse surface	Closest fit. Satisfies read_csv's header requirement. But see §3 — the output shape is wrong.
Trend capture	trends/trends.json (schema v4)	JSON envelope; title/url/summary are verbatim free text	External, model-mediated (web search/fetch via agent)	No connector exists. Not http — the fetch already happened, in a model turn.
Morning brief	trends/morning-brief/<date>.md	Structured md, carries external strings forward	Machine-rendered from store; re-injected by SessionStart hook	Destination, not a source.
Brain inbox → published	ingest/inbox/*.md → ingest/published/<id>.md	Free text, byte-exact body	Operator-dropped today; drop-zone has no origin control	Excluded by design — see §5.
Brain profile	brain/profile.md	Line grammar (6 constrained tokens + one-line value)	Machine-folded, operator-gated (--apply --confirm)	Destination. Already an OKF-conformant bundle (Stage 1, 2026-06-26).
Specifics-bank	specifics-bank/specifics-bank.json	JSON + verbatim free-text content	Operator-written by invariant (never AI-generated lived experience)	Destination.
Contract-gate	— writes nothing	Reads local operator files only	Local	Not applicable.

Summary: one genuinely tabular source (analytics CSV). One free-text external source that is not reachable by any connector shape the library has (trend capture). One free-text local source that is excluded by a fixed decision (published posts). Everything else is a destination store, not an ingestion input.

3. Where door A's contract collides with ours

Four structural mismatches, each verified against the library's code. These are not preferences — each one would break an invariant we hold today.

(1) Fixed 7-key frontmatter, no extension keys. materialize.py emits exactly type, title, source_system, source_query, ingested_at, ingest_manifest, generated — in fixed order, all values collapsed to single lines. Our records carry domain fields that must survive: provenance, status, first_seen, last_seen, evidence_count (brain), score, pillar, topics, surfaced (trends). Note the tension with OKF the format, which explicitly requires consumers to preserve unknown keys — the library's materializer is stricter than the spec it implements. Our Stage-1 OKF conformance work (docs/okf-convergence-brief.md) depends on rich fields riding along as extension keys; door A has no mechanism for that.

(2) Rewrite-on-run ownership vs incremental accretion. Door A owns files via generated: true + ingest_manifest, then deletes every stamped file and rewrites the set each run. Our stores accrete: dedupe by content id, topic-union on re-capture, last-wins score, collision-suffix on body divergence, compare-then-skip on scaffold. A door-A run against our data would be destructive by design.

(3) Operator-chosen slug ids vs content-addressed ids. Library id = the manifest's extraction.id (a hand-written slug, [a-z0-9][a-z0-9-]*), with provenance carried by a manifest stamp ({stem}@{sha256(manifest bytes)[:16]}). Ours = sha256(content)[:12] — the id is the dedupe key, which is why re-capturing the same trend or re-ingesting the same post is idempotent for free. A slug-keyed model cannot express "same content, seen twice."

(4) Table rendering destroys free text. render_table escapes \ then |, then collapses CRLF/CR/LF to a single space. Any multi-line body loses its line structure. This is fatal for post bodies (§5) and lossy for trend summaries.

4. What door A / door B must support before we can adopt — the concrete asks

Ordered by how much they unblock. (1) is the one that decides F1 for us.

R1 — A model-mediated door ("already-fetched payload"). Accept a caller-supplied, schema-validated payload (stdin JSON envelope or in-process record array) instead of a connector-fetched one, and materialize it with the same determinism, same stamping, same idempotence as door A. The manifest would declare shape and destination but not a fetch. This is the door that fits how an agentic plugin actually ingests, and we have a working precedent to donate: trends capture is this contract already (normalizeItem → "the one schema downstream never branches on" → deterministic store write). Why this over a free-text connector: a read_text/read_markdown connector would serve repos whose free text sits in a folder. Ours sits in a model's tool result. Both are needed; they are not the same ask, and solving only the folder case leaves us exactly where we are. Security note: this door is precisely where untrusted, model-touched bytes cross into a persisted store, so it is where a guard call belongs. That wiring is docs/ingestion-guard/plan.md's decision, not this document's — but the door must at minimum expose a seam for it rather than writing straight through.

R2 — Extension keys preserved through materialization. Let a record carry arbitrary additional frontmatter keys, emitted after the 7 reserved ones, order-stable, never coerced. Without this, every domain field we have is lost on the way through the library, and OKF's own "preserve unknown keys" rule is violated by the tool that writes OKF.

R3 — Incremental/upsert materialization mode. A mode where a run merges into an existing bundle keyed by record id — add new, update changed, leave untouched what this run did not see — instead of delete-all-stamped-then-rewrite. Door A's index maintenance already does exactly this kind of careful merge for index.md (managed lines refreshed, curated lines preserved byte-verbatim); the ask is to extend that discipline from the index to the records.

R4 — Content-addressed record ids as a first-class option. Allow id = hash(content) rather than a manifest slug, so dedupe and idempotence come from the data. Our three stores (brain, trends, specifics-bank) all independently converged on sha256(...)[:12]; the pattern is not LinkedIn-specific.

R5 (phase 4, Node half) — contract parity details. Stable string error codes on the error object (the Python half's exc.code discipline, mirrored so Node consumers assert on err.code, never on message text); zero runtime dependencies; ESM with node: prefixes; LF-only output with exactly one trailing newline; and the golden-fixture set shared across both halves so byte-identity is the test, not the promise. Our packages already meet the zero-dep/ESM/LF conventions (sole exception: scripts/analytics depends on csv-parse), so parity is cheap on our side.

5. What must never move here

ingest/published/ provenance grammar stays plugin-local — fixed decision, and it has a technical floor. This is not merely a scope preference:

• The store holds a byte-exact round-trip invariant (parse ∘ serialize = identity, SC2) on the verbatim post body. Door A's table renderer collapses newlines to spaces (§3.4); its frontmatter collapses whitespace runs. Either would break the invariant on contact.
• The record id is mintContentId(verbatim body) — deliberately un-normalized, so two structurally-different posts never collide and the write path never silently drops a differing body.
• The grammar is deliberately YAML-free (a fixed 5-line header + \n---\n sentinel), which is also why our own Stage-1 OKF conformance work scoped the concept-bundle to brain/ and excluded the ingest/ tributary — verified 2026-06-26, okf-check.mjs exit 0 on brain/.
• provenance=published carries model-collapse-guard semantics: the voice/profile learning surface learns from human-published content only, never from ai-draft. The form is generic; the guarantee is domain policy and belongs where the policy is enforced.

We are happy to describe the interface (this section is that description). We are not planning to hand it over.

Also staying — LinkedIn domain logic: trend scoring weights and mode SSOT (references/trend-scoring-modes.md); brief ranking, pillar logic and Norwegian rendering; the analytics↔post join heuristic (title-prefix matching with the 110-char hook rule behind PREFIX_FLOOR); CSV column fuzzing against LinkedIn's export column names, the engagementRate formula and saves semantics; the specifics taxonomy and its verification rules; contract-gate's writing-contract rules.

Candidates for sharing (bundle mechanics, if the Node half wants them): content-addressed id minting; dedupe + tag-union upsert; migrate-on-load schemaVersion stamping; collision-safe idempotent writes (compare-then-skip, collision-suffix); path-traversal-safe filename resolution (our storage.ts twin of the library's fail-closed safe_resolve); managed-line index maintenance. Several of these modules already declare themselves generic by architecture. Note we carry the data-root resolver in five inlined copies (four scripts/* packages + a zero-dep hooks twin, synchrony guarded by test) — deliberate, since the path convention is plugin-local even where the idiom is not; a shared library should take the idiom and leave the path.

6. Boundary against the guard plan

docs/ingestion-guard/plan.md identifies four persist gates: trends capture, brain writePublished/scanInbox, newsletter→ekstern bindings, analytics saveBatch. Two of those (trends capture, analytics import) reappear here as adoption candidates — unavoidably, since they are the same boundary viewed from two sides.

The split: the guard plan decides whether bytes are safe to persist; this plan decides what writes them and in what shape. They share a dependency in one place only — R1's seam is where a guard call would land — and that is noted as a seam requirement, not a security decision. Neither plan is a prerequisite for the other's approval. The guard plan's own interop blocker (Python↔Node) is tracked there.

7. Open questions for the library owners

1. Does R1 (model-mediated payload door) belong to door B, or is it a fourth door? It is neither an inbox scan nor an external bundle import.
2. Is R2 (extension keys) a spec change via commons, or a materializer relaxation within the current spec? OKF v0.1 already mandates preserving unknown keys, which suggests the latter.
3. For phase 4: is the Node half expected to reach parity with door A only, or with whatever doors exist when it starts? Our adoption depends on R1, which is not door A.

8. References

• Library: https://git.fromaitochitta.com/open/llm-ingestion-okf
• Guard: https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security
• Spec owner: portfolio-optimiser-commons/ingest-spec.md
• This repo: docs/okf-convergence-brief.md (Stage-1 OKF conformance, brain/ bundle), docs/ingestion-guard/plan.md (separate, planned), docs/second-brain/architecture.md