llm-ingestion-okf/docs/plan/phase-4-node-half.md

# Phase 4 plan — the Node half (`node/`)

Status: approved roadmap phase (see `CLAUDE.md`); details settled here before code.
Depends on: Phase 3 (the profile object must be able to express the second-brain
contract; that expressiveness is checked during Phase 3's split-table step).
Hard precondition: the coordination steps below — this phase does not start with
code, it starts with agreements.

## Goal

A zero-dependency Node/ESM package under `node/` for the second-brain world:
bundle check, index generation, inbox split/frontmatter/write, and document
conversion (docx/pdf/eml/html → md). Importable as a library AND invokable as a
CLI (`node <path> <args>`, exit code + stdout — the established plugin
invocation pattern), consumed by vendoring per plugin, not npm publishing.
The Python and Node halves share the OKF contract and the fixture suite, never
code.

## Coordination first (ordered, each with an owner sign-off)

1. **okr agreement.** The reference implementations (`okf-check.mjs`,
   `okf-index.mjs`, the inbox libraries and vocabulary/link modules) are lifted
   only in agreement with okr; their test suite is ported alongside the code as
   the seed corpus. Known deviations between okr's current behavior and the
   catalog spec are resolved WITH the convention owner before the lift — the
   library implements the agreed contract, not one repo's drift.
2. **Catalog stays convention owner.** The second-brain spec remains in the
   marketplace catalog; this repo implements it. Two premises here were wrong
   and are corrected by the convention owner (2026-07-25):
   - *There is no re-pin to make.* The catalog's version mechanism operates
     only over entries in its marketplace manifest, and this library is not a
     plugin. Adding an entry to make the mechanism fit would misrepresent a
     library as an installable plugin. The integration is instead a registry
     entry in the catalog's parity gate (`check-okf-parity.mjs`), whose
     implementation registry is already n-way and names this repo's future
     checker explicitly. We register when an implementation legitimately
     exists — never a placeholder.
   - *Parity is not byte identity, and the vendored copy is not an oracle.*
     The catalog's vendored `okf-check.mjs` diverges from okr's by ~141 lines
     of CODE (strictIngest mode, file scoping, walk filtering, shared-lib
     dependencies, BOM/CRLF handling) — an intentional decision to keep
     layer 1 at the bare spec §3 floor, not drift to be repaired. Parity is
     BEHAVIOURAL over that floor in default read-mode, at per-file
     granularity, measured by the catalog's parity gate over an adversarial
     corpus that carries a proof it can go red. No artifact self-certifies
     parity.
3. **linkedin-studio stays plugin-local.** Its `ingest/published/`
   provenance-record grammar has a different lifecycle by design and is NOT
   normalized into this library. Explicit non-goal, agreed up front, and
   confirmed by them on 2026-07-25 as their own fixed decision rather than
   something conceded to us.

   Two requirements they named on 2026-07-25, before the scope freeze, which
   it must therefore answer explicitly rather than pass over. Both are about
   not breaking what they have, not about building toward them:

   - **`write_index` must have a do-nothing mode.** They run a fourth index
     mode the spec does not describe: create-if-absent, no managed lines,
     the whole file hand-editable. Their scaffold never opens an existing
     index, so it cannot normalize one. The requirement is that this
     library's writer match that — against an existing index carrying no
     managed lines, write nothing, and do not normalize the frame "while it
     is open anyway". This is the writer-side sibling of the Phase 3
     decision that nothing enumerates a directory unless the profile says
     the index is derived. `write_index` is deliverable 3, not deliverable
     4's `write`; that was ambiguous in this plan and is now stated.
   - **An absent vocabulary means "do not snap", never "snap to default".**
     The reference checker collapses any non-matching type to a
     `DEFAULT_TYPE`, which would erase their Profile / Operations /
     PendingDiff distinction with zero byte change. Vocabulary is
     per-bundle, never spec-global. This is already the Python half's shape
     — `TypePolicy(allowed=None)` is the default and rewrites nothing — so
     the Node half inherits a design rather than adopting a constraint, and
     the collapse is a known deviation to resolve with the convention owner
     under step 1 rather than lift.

   They have offered a minimal tolerance fixture pair (a `brain/` hub whose
   `index.md` carries `okf_version` as markdown text, and an
   `ingest/published/` record: YAML-free five-line header terminated by
   `\n---\n`, no trailing newline). Take the bytes rather than the
   description — it is held as a tolerance case, never as a format this
   library emits.
4. **Persist gate = guard-as-contract.** There is no Node guard today. The Node
   half documents and enforces the guard's *contract* at its persist seam
   (fail-closed hook point, no content-safety logic of its own) so a future Node
   guard from the security repo can drop in. No security reimplementation in
   either runtime.
5. **ms-ai-architect as greenfield consumer.** Its designed-but-unbuilt needs
   (writer, index generator, checker, retrieval support) are the acceptance
   sketch for the package's API surface.

## Deliverables

1. **`node/` package skeleton:** ESM, `package.json` with NO `dependencies`,
   `node:` builtins only, `node:test` for tests, no build step (vendorable as
   plain files).
2. **Bundle check** (`okf check`): the catalog spec's rules — `type:` required,
   per-level `index.md` without frontmatter, root `okf_version`, canonical
   `resource` field, preserve-unknown / tolerate-broken posture.
3. **Index generation** (`okf index`): per-level index maintenance matching the
   reference behavior.
4. **Inbox primitives:** split, frontmatter emit/parse, relations, write — the
   lifted library modules under this repo's test suite.
5. **Document conversion** (`okf convert`): docx/pdf/eml/html → md. Conversion
   parsers cannot be zero-dep; this mirrors the Python `[extract]` pattern —
   the core package stays dependency-free, the conversion module lazy-loads
   optional parsers and rejects those types fail-fast when they are absent.
   Which parsers, and whether they are vendored or peer-installed, is settled in
   coordination step 1 (okr had concrete candidates).
6. **CLI:** one entry (`node/bin/okf.mjs check|index|inbox|convert …`),
   deterministic stdout, meaningful exit codes, no interactive prompts.
7. **Shared fixture suite:** cross-runtime fixtures for the second-brain
   contract, consumed by BOTH the Node tests and (via the Phase 3 profile) the
   Python tests, so the halves cannot silently diverge.

## Key assumptions (each with its test)

| # | Assumption | Test |
|---|---|---|
| D1 | okr's reference code + tests transfer cleanly to this repo's layout | Ported suite green before any behavior change; behavioural parity over the spec §3 floor via the catalog's parity gate, not a byte comparison |
| D2 | The catalog spec is the single contract both halves implement | Phase 3 profile expresses it; cross-runtime fixture suite passes in both runtimes |
| D3 | Zero-dep core is compatible with doc conversion | Lazy-load pattern proven by a test run WITHOUT optional parsers installed: core commands work, `convert` fails fast with a clear message |
| D4 | Vendoring workflow works for consumers | Dry-run: copy `node/` into a scratch plugin layout, run the CLI from there, exit codes intact |

## Non-goals

- npm publishing (vendoring only), bundling/transpiling, TypeScript build chain.
- Normalizing linkedin-studio's published-record grammar.
- A Node guard, or ANY content-safety logic — guard-as-contract until the
  security repo ships a Node implementation.
- Python↔Node code sharing of any kind (contract and fixtures only).

## Verification

1. `node --test node/` green; ported reference tests included.
2. Zero-dep proof: `package.json` has no `dependencies`/`optionalDependencies`;
   grep for non-`node:` imports in core modules returns nothing (conversion
   module exempted, lazy-load only).
3. Parity: this repo's check gate is registered in the catalog's parity gate
   and agrees with the other implementations over the spec §3 floor in default
   read-mode, per file, on the adversarial corpus that gate owns. Behavioural
   agreement, never byte comparison against a vendored copy.
4. Cross-runtime agreement: Python (Phase 3 profile) and Node checkers give the
   same accept/reject verdict per shared fixture (scripted matrix run).
5. CLI contract: each subcommand exercised via `node node/bin/okf.mjs …` in
   tests; exit codes asserted (0 clean, non-zero on violation), stdout
   deterministic (two runs diff-empty).
6. D3 test: suite run in an environment without optional parsers — core green,
   `convert` on a docx fails fast naming the missing parser.
7. Coordination artifacts: each of the five steps above has a recorded sign-off
   before its dependent code lands (tracked in the session state, summarized in
   the commit history).