llm-ingestion-okf/docs/plan/phase-4-node-half.md

Phase 4 plan — the Node half (node/)

Status: approved roadmap phase (see CLAUDE.md); details settled here before code. Depends on: Phase 3 (the profile object must be able to express the second-brain contract; that expressiveness is checked during Phase 3's split-table step). Hard precondition: the coordination steps below — this phase does not start with code, it starts with agreements.

Goal

A zero-dependency Node/ESM package under node/ for the second-brain world: bundle check, index generation, inbox split/frontmatter/write, and document conversion (docx/pdf/eml/html → md). Importable as a library AND invokable as a CLI (node <path> <args>, exit code + stdout — the established plugin invocation pattern), consumed by vendoring per plugin, not npm publishing. The Python and Node halves share the OKF contract and the fixture suite, never code.

Coordination first (ordered, each with an owner sign-off)

1. okr agreement. The reference implementations (okf-check.mjs, okf-index.mjs, the inbox libraries and vocabulary/link modules) are lifted only in agreement with okr; their test suite is ported alongside the code as the seed corpus. Known deviations between okr's current behavior and the catalog spec are resolved WITH the convention owner before the lift — the library implements the agreed contract, not one repo's drift.

2. Catalog stays convention owner. The second-brain spec remains in the marketplace catalog; this repo implements it. Two premises here were wrong and are corrected by the convention owner (2026-07-25):

   • There is no re-pin to make. The catalog's version mechanism operates only over entries in its marketplace manifest, and this library is not a plugin. Adding an entry to make the mechanism fit would misrepresent a library as an installable plugin. The integration is instead a registry entry in the catalog's parity gate (check-okf-parity.mjs), whose implementation registry is already n-way and names this repo's future checker explicitly. We register when an implementation legitimately exists — never a placeholder.
   • Parity is not byte identity, and the vendored copy is not an oracle. The catalog's vendored okf-check.mjs diverges from okr's by ~141 lines of CODE (strictIngest mode, file scoping, walk filtering, shared-lib dependencies, BOM/CRLF handling) — an intentional decision to keep layer 1 at the bare spec §3 floor, not drift to be repaired. Parity is BEHAVIOURAL over that floor in default read-mode, at per-file granularity, measured by the catalog's parity gate over an adversarial corpus that carries a proof it can go red. No artifact self-certifies parity.

3. linkedin-studio stays plugin-local. Its ingest/published/ provenance-record grammar has a different lifecycle by design and is NOT normalized into this library. Explicit non-goal, agreed up front, and confirmed by them on 2026-07-25 as their own fixed decision rather than something conceded to us.

   Two requirements they named on 2026-07-25, before the scope freeze, which it must therefore answer explicitly rather than pass over. Both are about not breaking what they have, not about building toward them:

   • write_index must have a do-nothing mode. They run a fourth index mode the spec does not describe: create-if-absent, no managed lines, the whole file hand-editable. Their scaffold never opens an existing index, so it cannot normalize one. The requirement is that this library's writer match that — against an existing index carrying no managed lines, write nothing, and do not normalize the frame "while it is open anyway". This is the writer-side sibling of the Phase 3 decision that nothing enumerates a directory unless the profile says the index is derived. write_index is deliverable 3, not deliverable 4's write; that was ambiguous in this plan and is now stated.
   • An absent vocabulary means "do not snap", never "snap to default". The reference checker collapses any non-matching type to a DEFAULT_TYPE, which would erase their Profile / Operations / PendingDiff distinction with zero byte change. Vocabulary is per-bundle, never spec-global. This is already the Python half's shape — TypePolicy(allowed=None) is the default and rewrites nothing — so the Node half inherits a design rather than adopting a constraint, and the collapse is a known deviation to resolve with the convention owner under step 1 rather than lift.

   They have offered a minimal tolerance fixture pair (a brain/ hub whose index.md carries okf_version as markdown text, and an ingest/published/ record: YAML-free five-line header terminated by \n---\n, no trailing newline). Take the bytes rather than the description — it is held as a tolerance case, never as a format this library emits.

4. Persist gate = guard-as-contract. There is no Node guard today. The Node half documents and enforces the guard's contract at its persist seam (fail-closed hook point, no content-safety logic of its own) so a future Node guard from the security repo can drop in. No security reimplementation in either runtime.

5. ms-ai-architect as greenfield consumer. Its designed-but-unbuilt needs (writer, index generator, checker, retrieval support) are the acceptance sketch for the package's API surface.

Deliverables

1. node/ package skeleton: ESM, package.json with NO dependencies, node: builtins only, node:test for tests, no build step (vendorable as plain files).
2. Bundle check (okf check): the catalog spec's rules — type: required, per-level index.md without frontmatter, root okf_version, canonical resource field, preserve-unknown / tolerate-broken posture.
3. Index generation (okf index): per-level index maintenance matching the reference behavior.
4. Inbox primitives: split, frontmatter emit/parse, relations, write — the lifted library modules under this repo's test suite.
5. Document conversion (okf convert): docx/pdf/eml/html → md. Conversion parsers cannot be zero-dep; this mirrors the Python [extract] pattern — the core package stays dependency-free, the conversion module lazy-loads optional parsers and rejects those types fail-fast when they are absent. Which parsers, and whether they are vendored or peer-installed, is settled in coordination step 1 (okr had concrete candidates).
6. CLI: one entry (node/bin/okf.mjs check|index|inbox|convert …), deterministic stdout, meaningful exit codes, no interactive prompts.
7. Shared fixture suite: cross-runtime fixtures for the second-brain contract, consumed by BOTH the Node tests and (via the Phase 3 profile) the Python tests, so the halves cannot silently diverge.

Key assumptions (each with its test)

#	Assumption	Test
D1	okr's reference code + tests transfer cleanly to this repo's layout	Ported suite green before any behavior change; behavioural parity over the spec §3 floor via the catalog's parity gate, not a byte comparison
D2	The catalog spec is the single contract both halves implement	Phase 3 profile expresses it; cross-runtime fixture suite passes in both runtimes
D3	Zero-dep core is compatible with doc conversion	Lazy-load pattern proven by a test run WITHOUT optional parsers installed: core commands work, convert fails fast with a clear message
D4	Vendoring workflow works for consumers	Dry-run: copy node/ into a scratch plugin layout, run the CLI from there, exit codes intact

Non-goals

• npm publishing (vendoring only), bundling/transpiling, TypeScript build chain.
• Normalizing linkedin-studio's published-record grammar.
• A Node guard, or ANY content-safety logic — guard-as-contract until the security repo ships a Node implementation.
• Python↔Node code sharing of any kind (contract and fixtures only).

Verification

1. node --test node/ green; ported reference tests included.
2. Zero-dep proof: package.json has no dependencies/optionalDependencies; grep for non-node: imports in core modules returns nothing (conversion module exempted, lazy-load only).
3. Parity: this repo's check gate is registered in the catalog's parity gate and agrees with the other implementations over the spec §3 floor in default read-mode, per file, on the adversarial corpus that gate owns. Behavioural agreement, never byte comparison against a vendored copy.
4. Cross-runtime agreement: Python (Phase 3 profile) and Node checkers give the same accept/reject verdict per shared fixture (scripted matrix run).
5. CLI contract: each subcommand exercised via node node/bin/okf.mjs … in tests; exit codes asserted (0 clean, non-zero on violation), stdout deterministic (two runs diff-empty).
6. D3 test: suite run in an environment without optional parsers — core green, convert on a docx fails fast naming the missing parser.
7. Coordination artifacts: each of the five steps above has a recorded sign-off before its dependent code lands (tracked in the session state, summarized in the commit history).