linkedin-studio/docs/okf-ingestion/plan.md

# OKF-ingestion plan — mapping this repo against `llm-ingestion-okf` (phase 4)

> **Status: `planned` (2026-07-20).** Survey + requirements only — nothing wired, nothing built. The
> library's Node half does not exist yet (its phase 4), so there is nothing to adopt today. This document
> is the repo's durable answer to *what door A or door B must support before this plugin can adopt*, and
> *what must never move into the shared library*.
>
> **Library:** `llm-ingestion-okf` v0.3.1 — Python 3.10+, zero runtime deps, **door A only**
> (manifest → connector → deterministic materialization of `ingest-{id}.md` → index generation, no model
> calls). Doors B (inbox) and C (external bundle import) are phase 2; the Node half is phase 4.
> **Spec:** `ingest-spec.md`, owned by `portfolio-optimiser-commons` — changes go via commons, never locally.
> **Security boundary:** `llm-ingestion-guard` (repo `llm-ingestion-pipeline-security`) owns all security.
> Door A is **ungated** — it calls no guard function before writing to disk (verified: no guard import or
> call site anywhere in the library's `src/`; `dependencies = []`). "Security is delegated" does **not**
> mean "safe by default"; gating is the caller's responsibility.
>
> **Separate, do not merge:** `docs/ingestion-guard/plan.md` (`planned`) is this repo's *security* wiring
> plan. It shares persist points with this document by necessity, but is a distinct decision with a
> distinct dependency. §7 states the boundary.

## 1. Executive answer

**Nothing here is a clean door-A fit today, and the reason is not the one the library expects.**

The library's open finding F1 is *"door A has no free-text connector"*. That is real for us, but it is not
our blocker. Our blocker is one level deeper:

> **Door A reaches sources with deterministic code. This plugin does not fetch anything with code.**

Every external input in this repo arrives through a *model turn* — the `trend-spotter` agent's
WebSearch/WebFetch/MCP calls, `/linkedin:react`'s WebFetch of an operator-supplied URL. There is no raw
HTTP fetch anywhere in plugin code (verified across `scripts/`). Door A's three connectors (`file`, `sql`,
`http`) all assume code can go get the bytes. For our largest and least-trusted surface, code cannot.

So the requirement we contribute to F1 is not *"add a free-text connector"* — it is **"add a door that
accepts an already-fetched, model-produced payload and materializes it deterministically."** That door
would immediately fit `trends capture`, which is already exactly this shape (stdin JSON envelope →
validate → deterministic store write). Details in §4.

Adoption verdict: **`planned`, conditional.** Adopting door A as it stands would mean converting typed,
incrementally-accreting stores into rewrite-on-run markdown documents — a downgrade. We adopt when §4's
four requirements land, not before.

## 2. Our sources — tabular vs free-text

| Source | On disk | Shape | Trust boundary | Door-A fit today |
|---|---|---|---|---|
| **Analytics CSV export** → batches | `exports/*.csv` → `posts/<date>-<id>.json` | **Tabular** (header row + rows) | First-party (operator's own LinkedIn export), but an unvalidated parse surface | **Closest fit.** Satisfies `read_csv`'s header requirement. But see §3 — the output shape is wrong. |
| **Trend capture** | `trends/trends.json` (schema v4) | JSON envelope; `title`/`url`/`summary` are **verbatim free text** | **External**, model-mediated (web search/fetch via agent) | **No connector exists.** Not `http` — the fetch already happened, in a model turn. |
| **Morning brief** | `trends/morning-brief/<date>.md` | Structured md, carries external strings forward | Machine-rendered from store; re-injected by SessionStart hook | Destination, not a source. |
| **Brain inbox → published** | `ingest/inbox/*.md` → `ingest/published/<id>.md` | **Free text**, byte-exact body | Operator-dropped today; drop-zone has no origin control | **Excluded by design** — see §5. |
| **Brain profile** | `brain/profile.md` | Line grammar (6 constrained tokens + one-line value) | Machine-folded, operator-gated (`--apply --confirm`) | Destination. Already an OKF-conformant bundle (Stage 1, 2026-06-26). |
| **Specifics-bank** | `specifics-bank/specifics-bank.json` | JSON + verbatim free-text `content` | Operator-written by invariant (never AI-generated lived experience) | Destination. |
| **Contract-gate** | — writes nothing | Reads local operator files only | Local | Not applicable. |

**Summary:** one genuinely tabular *source* (analytics CSV). One free-text external source that is not
reachable by any connector shape the library has (trend capture). One free-text local source that is
excluded by a fixed decision (published posts). Everything else is a destination store, not an ingestion
input.

## 3. Where door A's contract collides with ours

Four structural mismatches, each verified against the library's code. These are not preferences — each one
would break an invariant we hold today.

**(1) Fixed 7-key frontmatter, no extension keys.** `materialize.py` emits exactly `type`, `title`,
`source_system`, `source_query`, `ingested_at`, `ingest_manifest`, `generated` — in fixed order, all values
collapsed to single lines. Our records carry domain fields that must survive: `provenance`, `status`,
`first_seen`, `last_seen`, `evidence_count` (brain), `score`, `pillar`, `topics`, `surfaced` (trends).
Note the tension with OKF the *format*, which explicitly requires consumers to preserve unknown keys — the
library's materializer is stricter than the spec it implements. Our Stage-1 OKF conformance work
(`docs/okf-convergence-brief.md`) depends on rich fields riding along as extension keys; door A has no
mechanism for that.

> **Addition, 2026-07-20 (trinn E).** Two runs of `okr/scripts/okf-check.mjs` against a fresh
> `brain init` scaffold sharpen this into a concrete argument, and correct one of our own claims:
>
> - The reference checker's own constant is named `RECOMMENDED`, not `REQUIRED` (`okf-check.mjs:21`).
>   A bundle missing `resource`/`timestamp` gets **warnings and exit 0**. The de-facto conformance
>   tool already treats every §5 key beyond `type` as advisory — so a mandatory ordered prefix would
>   turn a green bundle red without a byte changing. Argument fed to commons' D1.
> - **Correction:** trinn C claimed our `type` vocabulary is five values. It is three —
>   `Profile` (`scripts/brain/src/profile.ts:48`), `Operations` (`scaffold.ts:56`),
>   `PendingDiff` (`cli.ts:252`). `JournalEntry`/`TributarySummary` exist only as examples in
>   `docs/okf-convergence-brief.md:101`, never in code.
> - **Correction:** we assented to a closed `type` vocabulary without checking its contents.
>   `okf-check.mjs --strict-ingest` **exits 1** on our conformant bundle — `okr/lib/okf-vocab.mjs:11-21`
>   is nine okr-domain values, and the safe default (`Dokument`) collapses all three of our types into
>   one, erasing the distinction `okf-conform.test.ts` rests on. Our assent now carries a condition:
>   an absent vocabulary must mean *do not snap*, and the vocabulary is per-bundle, never spec-global.

**(2) Rewrite-on-run ownership vs incremental accretion.** Door A owns files via `generated: true` +
`ingest_manifest`, then **deletes every stamped file and rewrites the set** each run. Our stores accrete:
dedupe by content id, topic-union on re-capture, last-wins score, collision-suffix on body divergence,
compare-then-skip on scaffold. A door-A run against our data would be destructive by design.

**(3) Operator-chosen slug ids vs content-addressed ids.** Library id = the manifest's `extraction.id`
(a hand-written slug, `[a-z0-9][a-z0-9-]*`), with provenance carried by a manifest stamp
(`{stem}@{sha256(manifest bytes)[:16]}`). Ours = `sha256(content)[:12]` — the id *is* the dedupe key, which
is why re-capturing the same trend or re-ingesting the same post is idempotent for free. A slug-keyed
model cannot express "same content, seen twice."

**(4) Table rendering destroys free text.** `render_table` escapes `\` then `|`, then collapses CRLF/CR/LF
**to a single space**. Any multi-line body loses its line structure. This is fatal for post bodies (§5) and
lossy for trend summaries.

## 4. What door A / door B must support before we can adopt — the concrete asks

Ordered by how much they unblock. (1) is the one that decides F1 for us.

**R1 — A model-mediated door ("already-fetched payload").** Accept a caller-supplied, schema-validated
payload (stdin JSON envelope or in-process record array) instead of a connector-fetched one, and
materialize it with the same determinism, same stamping, same idempotence as door A. The manifest would
declare shape and destination but not a fetch. This is the door that fits how an agentic plugin actually
ingests, and we have a working precedent to donate: `trends capture` is this contract already
(`normalizeItem` → "the one schema downstream never branches on" → deterministic store write).
**Why this over a free-text connector:** a `read_text`/`read_markdown` connector would serve repos whose
free text sits in a folder. Ours sits in a model's tool result. Both are needed; they are not the same ask,
and solving only the folder case leaves us exactly where we are.
*Security note:* this door is precisely where untrusted, model-touched bytes cross into a persisted store,
so it is where a guard call belongs. That wiring is `docs/ingestion-guard/plan.md`'s decision, not this
document's — but the door must at minimum expose a seam for it rather than writing straight through.

**R2 — Extension keys preserved through materialization.** Let a record carry arbitrary additional
frontmatter keys, emitted after the 7 reserved ones, order-stable, never coerced. Without this, every
domain field we have is lost on the way through the library, and OKF's own "preserve unknown keys" rule is
violated by the tool that writes OKF.

**R3 — Incremental/upsert materialization mode.** A mode where a run merges into an existing bundle keyed
by record id — add new, update changed, leave untouched what this run did not see — instead of
delete-all-stamped-then-rewrite. Door A's index maintenance already does exactly this kind of careful
merge for `index.md` (managed lines refreshed, curated lines preserved byte-verbatim); the ask is to extend
that discipline from the index to the records.

**R4 — Content-addressed record ids as a first-class option.** Allow `id = hash(content)` rather than a
manifest slug, so dedupe and idempotence come from the data. Our three stores (brain, trends,
specifics-bank) all independently converged on `sha256(...)[:12]`; the pattern is not LinkedIn-specific.

**R5 (phase 4, Node half) — contract parity details.** Stable string error codes on the error object
(the Python half's `exc.code` discipline, mirrored so Node consumers assert on `err.code`, never on message
text); zero runtime dependencies; ESM with `node:` prefixes; and the golden-fixture set shared across both
halves so byte-identity is the test, not the promise.

> **Correction, 2026-07-20 (trinn D-review).** An earlier revision of this section claimed we "already meet
> the zero-dep/ESM/LF conventions". Verified against the code, two parts of that were false:
>
> - **We have no stable error codes.** None of the four CLIs set a machine-readable code; every failure is a
>   free-text `Error` plus a numeric `process.exit` (`scripts/brain/src/ingest.ts:99,:113`,
>   `scripts/trends/src/cli.ts:125,:255`). `err.code` is an ask *of* the library, not a discipline we mirror.
> - **"Exactly one trailing newline" is not asserted here either.** Nothing tests it, and
>   `serializePublishedRecord` deliberately omits it (`scripts/brain/src/ingest.ts:71`) because the record
>   must end on the verbatim body or the byte-exact round-trip (SC2, §5) breaks.
>
> **Second correction, same day, narrowing the one above.** The trinn D-review first read this as a
> conflict with `claude-playlist-corpus`, who asked the library to own frame normalization (LF-only, one
> blank line, trailing newline). It is not a conflict, and the check should have come first: our
> newline-free serializer governs `ingest/published/`, which is **excluded from the OKF bundle by design**
> (§5, `scripts/brain/tests/okf-conform.test.ts:22-27`). The files `write_concept` would actually write for
> us — the `brain/` bundle — get their trailing newline from our own serializer
> (`scripts/brain/src/profile.ts:65`). On the concept-writing path we agree with them. What survives is
> narrower: `write_concept` is also being asked to serve as a *general* verbatim writer callable outside a
> bundle (`ms-ai-architect`, `po-claude`), so frame normalization should be a documented parameter rather
> than an unconditional guarantee. The fixture ask applies to that general writer, not to the concept corpus.
>
> Zero-dep/ESM does hold (sole exception: `scripts/analytics` depends on `csv-parse`).

## 5. What must never move here

**`ingest/published/` provenance grammar stays plugin-local — fixed decision, and it has a technical
floor.** This is not merely a scope preference:

- The store holds a **byte-exact round-trip invariant** (`parse ∘ serialize = identity`, SC2) on the
  verbatim post body. Door A's table renderer collapses newlines to spaces (§3.4); its frontmatter
  collapses whitespace runs. Either would break the invariant on contact.
- The record id is `mintContentId(verbatim body)` — deliberately **un-normalized**, so two
  structurally-different posts never collide and the write path never silently drops a differing body.
- The grammar is deliberately YAML-free (a fixed 5-line header + `\n---\n` sentinel), which is also why our
  own Stage-1 OKF conformance work scoped the concept-bundle to `brain/` and **excluded the `ingest/`
  tributary** — verified 2026-06-26, `okf-check.mjs` exit 0 on `brain/`.
- `provenance=published` carries **model-collapse-guard semantics**: the voice/profile learning surface
  learns from human-published content only, never from `ai-draft`. The *form* is generic; the *guarantee*
  is domain policy and belongs where the policy is enforced.

We are happy to describe the interface (this section is that description). We are not planning to hand it
over.

**Also staying — LinkedIn domain logic:** trend scoring weights and mode SSOT
(`references/trend-scoring-modes.md`); brief ranking, pillar logic and Norwegian rendering; the
analytics↔post join heuristic (title-prefix matching with the 110-char hook rule behind `PREFIX_FLOOR`);
CSV column fuzzing against LinkedIn's export column names, the `engagementRate` formula and `saves`
semantics; the specifics taxonomy and its verification rules; contract-gate's writing-contract rules.

**Candidates for sharing (bundle mechanics, if the Node half wants them):** content-addressed id minting;
dedupe + tag-union upsert; migrate-on-load `schemaVersion` stamping; collision-safe idempotent writes
(compare-then-skip, collision-suffix); path-traversal-safe filename resolution (our `storage.ts` twin of
the library's fail-closed `safe_resolve`); managed-line index maintenance. Several of these modules already
declare themselves generic by architecture. Note we carry the data-root resolver in **five** inlined copies
(four `scripts/*` packages + a zero-dep hooks twin, synchrony guarded by test) — deliberate, since the
*path convention* is plugin-local even where the *idiom* is not; a shared library should take the idiom and
leave the path.

## 6. Bundle inventory and placement

Operator decision, 2026-07-20: **user-owned OKF bundles live outside the plugin/repo tree and are only
referenced from it.** Plugin-generated bundles that are part of the plugin's own delivery stay.

**We already comply — the migration happened in M0 (v0.6.0), before the decision existed.** Every
user-owned artifact resolves through `${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/`
(`references/data-path-convention.md`), with an idempotent session-start migration. Nothing user-owned has
lived in the repo tree since.

| Artifact | Path | OKF bundle? | Class |
|---|---|---|---|
| **Brain** | `<data-dir>/brain/` | **Yes** — the repo's only OKF bundle (`type` + per-level `index.md` + root `okf_version: 0.1`; `okf-check.mjs` exit 0, 2026-06-26) | **User-owned** — see boundary case below |
| Published-post tributary | `<data-dir>/ingest/{inbox,published}/` | No — deliberately excluded (§5) | User-owned |
| Trend store | `<data-dir>/trends/trends.json` | No — a single JSON file, not a markdown bundle | User-owned |
| Morning briefs | `<data-dir>/trends/morning-brief/<date>.md` | No — dated briefs with frontmatter, no `index.md`, not concept files | User-owned |
| Specifics-bank | `<data-dir>/specifics-bank/specifics-bank.json` | No | User-owned |
| Analytics batches | `<analytics-root>/posts/*.json` | No | User-owned |
| Reference docs (28) | `references/*.md` **in repo** | No — no `type`, no `index.md` | **Plugin-owned**, ships with the plugin, correctly in-tree |

**Nothing breaks on migration, because there is nothing left to migrate.** The one caveat is historical:
pre-M0 installs kept data in-tree, and the session-start migration already handles that path. A second
caveat worth naming: `assets/drafts/queue.json` and `assets/analytics/` remain in-tree as gitignored
scratch — they are not bundles, but they are the last in-tree paths that hold user bytes, and they should
be revisited if the decision is ever tightened to "no user bytes in-tree at all."

### The boundary case — and it says the taxonomy's axis is wrong

The decision's table discriminates on **who writes**: plugin writes → plugin-owned → lives in the repo.
Our brain is **written by the plugin** (`consolidate`/fold, operator-gated at `--apply --confirm`) and
never hand-authored. By that table it is plugin-owned and belongs in the repo. **That conclusion is
exactly wrong** — it is the single most personal artifact we hold, it must survive plugin upgrade and
uninstall, and the repo is publicly distributed.

The discriminator that gives the right answer everywhere in our inventory is **whose lifecycle the content
follows**, not whose hand writes the bytes. Our brain is machine-written and user-owned, and that
combination is not a rare corner — it is what any learning system produces. We suggest the axis be
lifecycle/ownership, with authorship as a non-determinative attribute. Otherwise every plugin that
*derives* user knowledge lands on the wrong side of a table it read correctly.

## 7. Boundary against the guard plan

`docs/ingestion-guard/plan.md` identifies four persist gates: trends `capture`, brain
`writePublished`/`scanInbox`, newsletter→`ekstern` bindings, analytics `saveBatch`. Two of those (trends
capture, analytics import) reappear here as adoption candidates — unavoidably, since they are the same
boundary viewed from two sides.

The split: **the guard plan decides whether bytes are safe to persist; this plan decides what writes them
and in what shape.** They share a dependency in one place only — R1's seam is where a guard call would
land — and that is noted as a seam requirement, not a security decision. Neither plan is a prerequisite for
the other's approval. The guard plan's own interop blocker (Python↔Node) is tracked there.

## 8. Open questions for the library owners

1. Does R1 (model-mediated payload door) belong to door B, or is it a fourth door? It is neither an inbox
   scan nor an external bundle import.
2. Is R2 (extension keys) a spec change via commons, or a materializer relaxation within the current spec?
   OKF v0.1 already mandates preserving unknown keys, which suggests the latter.
3. ~~For phase 4: is the Node half expected to reach parity with door A only, or with whatever doors exist
   when it starts?~~ **Answered 2026-07-20 (trinn E, felles §6): phase 4 delivers the layer-1 primitives,
   not `materialize_bundle` parity** — no Node repo in the set runs door A (verified here: the only Python
   in this repo is `hooks/scripts/compile-hooks.py`, a build-time script never on a data path). Our register
   status therefore stays **`planned`**, not `blocked on phase 4 scope`. One follow-up outstanding: layer 1
   grew after the question was asked (`read_concept`, `navigate_bundle`, `routeLevel`), so we asked whether
   phase 4 tracks the set as it stands or as it evolves — the reader is decision-relevant for us, since our
   dependency is `serialize ∘ parse = identity`, not the writer alone.

## 9. References

- Library: `https://git.fromaitochitta.com/open/llm-ingestion-okf`
- Guard: `https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security`
- Spec owner: `portfolio-optimiser-commons/ingest-spec.md`
- This repo: `docs/okf-convergence-brief.md` (Stage-1 OKF conformance, `brain/` bundle),
  `docs/ingestion-guard/plan.md` (separate, `planned`), `docs/second-brain/architecture.md`

## 10. Trinn E — consolidated positions (preserved before postbox teardown, 2026-07-21)

The OKF adoption round closed at trinn F (consensus). Our trinn E answer lived in the interim
postbox (`~/repos/_okf-interim/svar/linkedin-studio.md`), which is now torn down. Its durable
substance is preserved here; everything below was verified against code on 2026-07-20. The
technical requirements (R1–R5) already live in §4; this section captures the round-positional and
follow-up material that existed only in the postbox.

**Our §7 register row (status + position).**
- Status stays **`planned`**, not `blocked on phase 4 scope` — verified: the only Python in this
  repo is `hooks/scripts/compile-hooks.py`, a build-time script never on a data path; we do not
  run door A and would gain nothing from parity.
- We do **not** need door-A stamping (`generated` / `ingest_manifest`). The negative requirement
  holds: inject nothing, normalize nothing.
- Unforgeability must live **outside the file** — our content-address `mintContentId` (byte-exact
  sha256, `scripts/brain/src/id.ts:53-55`) is unforgeable without the writer forming an opinion
  about the frontmatter. If the writer inspects the string to enforce a stamp, passthrough (R2) is
  gone in the same move.
- §5 must tolerate extension keys (R2, unchanged); the mandatory reserved-key prefix is limited to
  **stamped, extracted** concepts, not concepts as such.

**D1 (frontmatter/stamp) — we are affected, not owner.** The ratified D1 **matches our ask**:
enforcement moved to `check_bundle` (per-file outcome); the door-C writer rejects the *complete*
ownership stamp (`generated:true` **and** `ingest_manifest` together), not the five names, so
round-trip for legitimate door C is preserved; the mandatory prefix binds stamped (door-A) files
alone. We accept commons' point that passthrough makes §5's key contract unenforceable at the
writer by construction — that is the right outcome, since the writer cannot make the
stamped-vs-curated distinction without reading what it promised not to read.

**D3 (`partial`) — accepted with one caveat, now resolved.** We accepted the definition and flagged
that "comparable path" was undefined (it decided our value outright: `brain/` writing through the
library while trend-store / specifics-bank stay standalone JSON — `scripts/trends/src/store.ts:126-141`).
The ratified definition (trinn F) is: comparable path = would have gone through the same door
(A/B/C). Under that, our JSON/CSV/domain-store paths are not comparable and do not force us
`partial`. **Our status today is `planned`, not `partial`** — zero production paths go through the
library; we do not emit `partial` (now a transitory value carrying owner + next step) until it is
ratified.

**Durable technical additions (would otherwise be lost with the postbox).**
- **Empty vocabulary must mean "don't snap", not "snap to default".** Verified:
  `node okr/scripts/okf-check.mjs <brain> --strict-ingest` collapses our three real types
  (`Profile` / `Operations` / `PendingDiff`) to `DEFAULT_TYPE='Dokument'`
  (`okr/lib/okf-vocab.mjs:26`), erasing the distinction our conformance test rests on
  (`scripts/brain/tests/okf-conform.test.ts:15-20`). Vocabulary is per-bundle, never spec-global.
  Ask: absence of an injected vocab list is a legal state (= "don't snap"), not "snap all to default".
- **Supporting evidence, and it is not ours:** `okf-check.mjs:21` names the constant `RECOMMENDED`,
  not `REQUIRED` — the de-facto checker all nine repos read already treats §5 keys beyond `type` as
  warnings, not requirements. Making the prefix mandatory would turn a green bundle red with zero
  byte change, contradicting the tool that defined conformance in practice during the round.
- **`type` vocabulary is three values, not five** (corrects trinn C §5): only `Profile`
  (`scripts/brain/src/profile.ts:48-50`), `Operations` (`scaffold.ts:56-58`), `PendingDiff`
  (`cli.ts:252`) are emitted by code; `JournalEntry` / `TributarySummary` exist only as "e.g." in a
  design note (`docs/okf-convergence-brief.md:101`), never in code.
- **Index `preserve-unowned` boundary.** We have a fourth index mode — create-if-absent, no managed
  lines, whole file hand-editable (`scripts/brain/src/scaffold.ts:33-36,126-141`). Concrete
  requirement for eventual adoption: `write_index(mode="preserve-unowned")` against an existing file
  with no managed lines must leave it **byte-untouched**, not normalize the frame "while it is there
  anyway" — else we lose a file the operator hand-writes.
- **Layer 1 = the evolving set, not a frozen snapshot** (also §8 item 3). Since §5 added
  `read_concept`, `navigate_bundle`, and `routeLevel` after Q3 was asked, "the layer-1 primitives"
  must read as the set as it stands, not a snapshot: our invariant is `serialize ∘ parse = identity`
  (`scripts/brain/tests/ingest.test.ts:102`), and a Node half with a writer but no reader gives us
  half the invariant. Derived (not a new ask): the shared golden fixtures for `resolve_link` must
  cover all four orthogonal axes across runtime — our R5 stated the fixture requirement for the
  writer's frame alone, which is now too narrow.