linkedin-studio/docs/second-brain/plan-sb-s1.md

Implementation Plan — SB-S1 (Ingest + gold signal)

Status: review-hardened (plan-critic REVISE 63→folded; scope-guardian GAP→folded). Ready for operator approval / "go". Brief: brief-sb-s1.md (review-hardened). Arc: architecture.md. Predecessor: SB-S0 (scripts/brain/). Review delta (2026-06-23): B1 separator pinned to one exact string + edge battery; B2 id → verbatim-body hash (no normalizeContent) + collision-safe write; floors pinned EMPIRICALLY (not hand-arithmetic); SC5 grep → exact grep -F literals + AI-generated negative probe; CLI = net-new flag infra + init regression; per-step On-failure/Checkpoint + 2 circuit-breakers; D3 auto-append FENCE restored; voice-trainer anchors fixed (:136/:144); malformed-file try/catch + scanInbox *.md-only filter. Scope: SB-S1 only — the published-record data layer + brain ingest CLI, and wiring voice-trainer to the published-only invariant (gate-enforced). No consolidation loop, no automatic profile mutation, no session-start wiring, no cross-silo threading, no connector (brief §7).

---

1. What SB-S1 delivers (recap)

A new TS module scripts/brain/src/ingest.ts (matching the SB-S0/specifics-bank idiom) defining a PublishedRecord shape with a pinned file-per-post constrained-header grammar (no YAML) that round-trips, a content-hash id over the verbatim body (sha256(body)[:12] — byte-identity dedup, no normalization), and idempotent, collision-safe IO (writePublished, scanInbox, listPublished) under the brain package's dataRoot. The brain CLI grows ingest + published list subcommands. agents/voice-trainer.md is wired to add ingest/published/ as the primary gold source and forbid learning from provenance=ai-draft, and that wiring is gate-enforced by a new structure-lint (Section 16c). All unit-tested, TDD (failing test first). Gate stays green; BRAIN + assertion floors rise.

2. Package additions (scripts/brain/) — no new package, extend the existing one

scripts/brain/
  src/
    ingest.ts          # NEW — PublishedRecord type + parse/serialize (pure) + writePublished/scanInbox/listPublished (IO)
    id.ts              # EDIT — add normalizeContent() + mintContentId() (the specifics-bank content-hash idiom)
    cli.ts             # EDIT — add `ingest` + `published` dispatch (parseFlags idiom from specifics-bank/src/cli.ts)
    (types.ts, profile.ts, scaffold.ts, dataRoot.ts unchanged)
  tests/
    ingest.test.ts     # NEW — record round-trip + id determinism/dedup (pure)
    publish.test.ts    # NEW — temp-dir IO: write/idempotency/scan-inbox/create-on-demand
  fixtures/
    sample-inbox-post.md  # NEW — a synthetic published post for scan-inbox + round-trip tests

No package.json/tsconfig.json change (devDeps already tsx+typescript; no new dep — the grammar is regex, no YAML). The test command (node --import tsx --test tests/*.test.ts) already globs tests/*.test.ts, so the two new suites are picked up automatically.

3. Module designs

3.1 id.ts addition — mintContentId on the VERBATIM body (no normalization — fixes plan-critic B2)

Verify-first (step 1): mirror the specificId hash idiom at scripts/specifics-bank/src/bank.ts:57 (the function is named specificId, NOT mintContentId — that name is net-new here; normalizeContent lives at bank.ts:52). Decision: do NOT reuse normalizeContent for a post body.

// mintContentId: sha256 of the VERBATIM body, first 12 hex. The id IS byte-identity.
export function mintContentId(text: string): string {
  return createHash("sha256").update(text).digest("hex").slice(0, 12);
}

• Why verbatim, not normalizeContent: normalizeContent (bank.ts:52) does trim().toLowerCase().replace(/\s+/g," ") — correct for short specifics dedupe, but for a full post body it collapses ALL whitespace + case, so two structurally different published posts (same words, different line breaks/paragraphing/case) would mint the same id and the second would be silently skipped on write (data loss of the exact gold signal SB-S1 exists to capture, plan-critic B2). Hashing the verbatim body makes the id true byte-identity: only an identical re-ingest collides; any real difference mints a distinct id.
• mintContentId is content-keyed — distinct from mintEntityId({kind,key}) which slugifies a label (a body must never be slugified). Pure, deterministic. SC3.
• Defence-in-depth (write-side, §3.3): even with byte-identity hashing, writePublished compares bodies on any id-collision and never silently drops a differing body — so no hash strategy can cause data loss.

3.2 ingest.ts — PublishedRecord + constrained-header grammar (no YAML)

export interface PublishedRecord {
  id: string;            // mintContentId(body) — 12 hex, the dedupe key + filename stem
  provenance: 'published';  // S1 only ever writes `published`; the type pins it
  published_date: string;   // YYYY-MM-DD (operator-supplied or defaults to captured_at)
  captured_at: string;      // YYYY-MM-DD (ingest run date; supplied by caller — pure fn)
  source: string;           // 'manual' (default) | future connector token
  body: string;             // the VERBATIM post text (may contain ] | quotes newlines ---)
}

File grammar (ingest/published/<id>.md) — the EXACT byte layout, used identically by serialize and parse (fixes plan-critic B1):

id: <12hex>
provenance: published
published_date: YYYY-MM-DD
captured_at: YYYY-MM-DD
source: manual
---
<verbatim body>

• Pinned separator (one string, both directions): the five header lines, then a line that is exactly ---, then the verbatim body. Concretely: serialize = headerLines.join("\n") + "\n---\n" + body. There is NO blank-line padding around the --- and NO # Published Record title (both removed — they were the B1 contradiction). The header block is fixed at exactly 5 lines.
• parsePublishedRecord(text) → split on the FIRST occurrence of \n---\n. Left = header (parse the 5 scalars by the in-tree scalar-regex idiom: ^id:\s*([0-9a-f]{12})$/m, ^published_date:\s*(\d{4}-\d{2}-\d{2})$/m, etc.). Right = body, verbatim, byte-exact (a body that itself contains a \n---\n is preserved — we split on the FIRST sentinel only, via indexOf, never a global split). provenance is parsed and asserted to equal published (a published/ record with any other provenance is a corruption signal → throw/skip, never silently accepted — fixes plan-critic minor 11; the type pins 'published').
• parse(serialize(rec)) === rec over all six fields incl. the body (SC2). Edge cases the round-trip test MUST cover (plan-critic B1): empty body (""), body = "\n", body with trailing newlines, body starting with ---, body containing \n---\n mid-text, and a body whose first line is header-shaped (e.g. id: 0123456789ab). Because the header is a fixed 5-line block terminated by the first \n---\n, a header-shaped body line cannot leak into the header (it sits after the sentinel). The only constraint on the body: none needed — it is captured byte-exact after the first sentinel.

3.3 ingest.ts — IO (impure; caller supplies the run date, like scaffold.ts)

• writePublished(rec): {written: boolean; path: string; collision?: boolean} — dataRoot('ingest/published'), mkdir-recursive (create-on-demand — no hard dependency on brain init, SC1), path = <id>.md. Collision-safe compare-then-skip (fixes plan-critic B2 defence-in-depth): if <id>.md exists, READ it, parse its body: (a) body byte-identical to rec.body → true duplicate → {written:false} (idempotent, no clobber — the initBrain discipline); (b) body DIFFERS (astronomically rare 48-bit collision, or a truncation artifact) → write to a disambiguated path <id>-2.md (next free suffix) and return {written:true, collision:true} — never silently drop a differing body.
• ingestText({body, source?, published_date?, captured_at}) — mintContentId(body), build the record (provenance:'published'; published_date defaults to captured_at when absent), writePublished. Returns the record + written flag.
• scanInbox({captured_at}) — read top-level *.md files only, skipping dotfiles (.DS_Store etc. — macOS), no recursion (fixes plan-critic minor 9), in dataRoot('ingest/inbox') (empty/absent dir → clean no-op, SC4); for each, ingestText({body: fileContent, source:'manual', captured_at}). Non-destructive in S1: inbox files are LEFT in place (idempotent via the published dedup — a re-scan re-skips); auto-move/delete of processed inbox files is deferred (noted §8). Returns {processed[], skipped[]}.
• listPublished() — read ingest/published/*.md; parse each inside a try/catch — a malformed/hand-dropped file is skipped+counted, never crashes the command (parsePublishedRecord can throw via the provenance assertion / normalizeProvenance, id.ts:45; fixes plan-critic minor 8). Return {records: [{id, provenance, published_date, firstLine}], skipped: number} sorted by date. Surfaces provenance so the operator can eyeball that nothing ai-draft leaked (brief §5.1 D2).

3.4 cli.ts — add NET-NEW flag-routing infra (not just "extend dispatch" — fixes plan-critic major 6)

The brain cli.ts currently has no parseFlags, no --json, no today() — it dispatches purely on positional argv[2] (just init). So this step ADDS that infrastructure (copy parseFlags from specifics-bank/src/cli.ts:41; a local today() — ingest.ts itself stays pure with a caller-supplied date, so the CLI's today() is the only clock and does not shadow scaffold.ts:28's private one). The existing init positional branch is preserved verbatim, coexisting with the new flag-driven branches:

• brain ingest --file <path> [--source <s>] [--date <YYYY-MM-DD>] — read the file, ingestText, print Wrote ingest/published/<id>.md / Duplicate — already published: <id> / Collision → wrote <id>-2.md.
• brain ingest --scan-inbox [--source <s>] — scanInbox, print the processed/skipped split.
• brain published list [--json] — listPublished; human form prints id · provenance · date · first-line; --json emits {records:[{id,provenance,published_date,firstLine}], skipped} (the listPublished return verbatim — pinned shape, fixes plan-critic minor 12).
• Usage text updated; exit 2 on usage error (existing idiom). Regression guard: a test (or step-5 smoke-check) asserts brain init still works after the refactor (a shipped subcommand must not break).

3.5 agents/voice-trainer.md — wire the published-only invariant (brief §5.1 D3, all four bullets)

Surgical edits (existing file → Edit, pathguard-irrelevant; model stays Sonnet). Anchors corrected (scope-guardian DEP-1): the Gather bullet is at :136, the "Sample Quality Priorities" heading at :142 with its list at :144 (the earlier :134/:152 were a header and the next section).

1. Gather step (:136) — prepend the gold source + the negative rule (ADD + FORBID + KEEP):

   Gather (published-only gold signal first): Read the user's published posts from ${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/ingest/published/ (provenance=published — the highest-trust source). Then read the existing voice-samples/ corpus (human-curated; kept as a tributary, not reshaped) and the profile/template. Never learn voice patterns from any content marked provenance=ai-draft — a content engine that learns its own voice from its own drafts collapses toward its priors. (Generalises voice-scrubber's "gold standard = approved editions, never the draft corpus".)

2. Sample Quality Priorities (:144) — promote the published gold source to #1 and add the hard exclusion line:

   0. ingest/published/ records (provenance=published) — the gold signal; rank above all else. … (existing 1–5 retained) … Exclude unconditionally: anything marked provenance=ai-draft.

3. FENCE the auto-append trap (brief §5.1 D3 bullet 4 — restored, scope-guardian SCOPE-GAP-1). Add a short note in the Gather step (and mirror it in the §3.6 contract doc): the voice-samples/ template hints at a not-yet-built Stop-hook that auto-appends posts after content sessions (assets/voice-samples/authentic-voice-samples.template.md:61). State explicitly:

   Fence: any future auto-capture into a learning corpus (the template's Stop-hook hint) MUST tag provenance and admit provenance=published only — never blind-append generated drafts.

Exact literals (fixes plan-critic major 5 — non-vacuity): the edits MUST write the literal strings ingest/published and provenance=ai-draft. The SC5 lint greps for these EXACT literals (grep -F), NOT a loose ai.?draft/ai-generated pattern — voice-trainer.md:175 already contains "AI-generated" (a drift-cause description), so a loose grep would pass vacuously without the real exclusion being added.

3.6 docs/second-brain/ingest-manual-import.md — the manual-import contract

A short design doc: the PublishedRecord file grammar (§3.2), the CLI usage (brain ingest --file / --scan-inbox, brain published list), the inbox→published drop-zone model, the published-only learning rule (with the honest runtime limit: an agent instruction can be ignored at runtime; the wiring is gate-enforced, the behaviour is not provable), and the auto-append fence (§3.5 bullet 3 — any future auto-capture must tag provenance and admit published only). Includes the exact literal provenance=ai-draft (so SC5's doc-grep is non-vacuous). Lives in docs/ (not references/) → no reference-count guard.

3.7 scripts/test-runner.sh — new SC5 lint (Section 16c) + floor bumps

• New Section 16c "Brain Published-Only Invariant (SB-S1)" placed after Section 16b (Brain Foundation), mirroring Section 17's structure: a non-vacuity self-test (POSITIVE/NEGATIVE heredocs) + the file checks. Greps use the exact literals ingest/published and provenance=ai-draft via grep -F / grep -qiE with the provenance= qualifier — never a loose ai.?draft/ai-generated pattern (plan-critic major 5). Three pass() lines (none in a loop):
  1. self-test — POSITIVE probe contains BOTH literals → must pass; NEGATIVE probes: one missing ingest/published, one missing provenance=ai-draft, and one containing AI-generated but NOT provenance=ai-draft (proves the check doesn't pass on voice-trainer's pre-existing drift prose) → each must fail. Non-vacuous like Sections 8/13/17.
  2. voice-trainer.md contains ingest/published AND provenance=ai-draft.
  3. docs/second-brain/ingest-manual-import.md contains provenance=ai-draft (the rule is documented).
• BRAIN_TESTS_FLOOR=34 (:714) → the actual post-implementation count. Pinned at step 8 by reading the brain suite's real tests N AFTER the new suites land (TDD determines it; do NOT guess). Step 8 must first assert the brain suite actually RAN (BR_TESTS non-empty — i.e. tsx installed) before pinning, else a warn-skip would let the gate go green while SB-S1's own tests never ran (plan-critic major 4).
• ASSERT_BASELINE_FLOOR=75 (:798) → the actual printed TOTAL_CHECKS. The floor counts runtime PASS+FAIL increments, not static pass() call-sites (the file has 72 static sites but a 75 floor because some checks loop). The three new Section 16c checks are non-looping, so the delta is +3 → 78 — but pin it empirically: run the gate at step 8, read the printed TOTAL_CHECKS, set the floor to that exact number rather than hand-arithmetic (plan-critic major 3). Update the inline history comment at :796.

4. Files created / edited

Created: scripts/brain/src/ingest.ts · scripts/brain/tests/{ingest,publish}.test.ts · scripts/brain/fixtures/sample-inbox-post.md · docs/second-brain/ingest-manual-import.md.

Edited: scripts/brain/src/id.ts (+normalizeContent/+mintContentId) · scripts/brain/src/cli.ts (+ingest/+published) · agents/voice-trainer.md (gather + priorities) · scripts/test-runner.sh (Section 16c + two floor bumps) · docs/second-brain/architecture.md (flip SB-S1 row → landed).

Conditional (only if Q4 = bump to 0.5.1): CLAUDE.md header + README badge + CHANGELOG + the marketplace catalog ref (via catalog/scripts/release-plugin.mjs — the polyrepo release path; tag + catalog bump atomic). Decided at "go".

5. TDD test plan (failing-first) → success criteria

Test (tests/*.test.ts)	Asserts	SC
ingest.test.ts	parsePublishedRecord(serializePublishedRecord(rec)) === rec over the B1 edge-case battery: empty body, body="\n", trailing newlines, body starting with ---, body containing \n---\n mid-text, header-shaped first body line (id: 0123456789ab), and a body with ]/`	/quotes; mintContentIddeterministic (same body→same id; whitespace/case difference→DIFFERENT id, proving no normalize-collision); apublished/ record with non-published` provenance throws (corruption assertion)
publish.test.ts	LINKEDIN_STUDIO_DATA=tmp, NO prior init → writePublished creates ingest/published/<id>.md with provenance=published + verbatim body (create-on-demand); re-ingest identical body = {written:false}, no dup; id-collision with a DIFFERING body → disambiguated <id>-2.md, no data loss; scanInbox processes top-level *.md only (skips a .DS_Store fixture), → published; empty/absent inbox = no-op; re-scan re-skips; listPublished skips a malformed dropped file without throwing; brain init still works after the CLI refactor (regression)	SC1, SC3, SC4
(gate) scripts/test-runner.sh Section 16c	self-test non-vacuous (incl. the AI-generated-but-not-ai-draft negative probe); voice-trainer.md + contract doc carry the exact literals	SC5
(gate) scripts/test-runner.sh	green; trends ≥24, specifics ≥28, contract ≥33, brain ≥ new floor (brain suite confirmed RAN, not skipped); hooks pass; assertion floor = printed TOTAL_CHECKS	SC6
(n/a)	no new seam fn; data-root.test.mjs twin-sync untouched	SC6

Each test written before its module (iron law): red → implement → green.

6. Step sequence

Discipline (plan-critic major 7): each TDD step is a Checkpoint — commit after its cycle goes green (small, revertible commits). Each step carries an On-failure clause; two hard circuit-breakers: if step 3 (round-trip) cannot be made green after pinning the separator, HALT (the grammar is wrong — do not proceed); if step 8 (full gate) is not green, HALT (do not commit a red gate).

1. Verify-first: read scripts/specifics-bank/src/bank.ts:52,57 → confirm the specificId hash idiom (mirror it as mintContentId on the VERBATIM body; do NOT reuse normalizeContent). Confirm scripts/brain/node_modules/.bin/tsx exists (else npm install in scripts/brain — the gate warn-skips a depless suite; "green" must not mean green-because-skipped). On-failure: if tsx won't install, STOP and report — the slice cannot be gate-verified.
2. TDD id.ts mintContentId (ingest.test.ts id cases: determinism + whitespace/case → different id). → SC3. Checkpoint.
3. TDD ingest.ts grammar parse/serialize over the B1 edge-case battery (ingest.test.ts round-trip). → SC2. On-failure: CIRCUIT-BREAKER — if identity can't hold, the separator is wrong; halt and re-pin §3.2 before any further step. Checkpoint.
4. Write fixtures/sample-inbox-post.md (+ a .DS_Store + a malformed file in test temp dirs); TDD ingest.ts IO — writePublished (incl. collision→disambiguate), ingestText, scanInbox (filter), listPublished (try/catch) (publish.test.ts). → SC1, SC4. Checkpoint.
5. Add the CLI flag-routing infra + ingest/published branches (preserve init); regression-test brain init; smoke-run against a temp LINKEDIN_STUDIO_DATA (ingest a file, scan an inbox, list). On-failure: if brain init regresses, revert the CLI refactor and isolate. Checkpoint.
6. Edit agents/voice-trainer.md (:136 gather + :144 priorities + the provenance=ai-draft exclusion + the auto-append fence — exact literals). Checkpoint.
7. Write docs/second-brain/ingest-manual-import.md (incl. the provenance=ai-draft literal). Checkpoint.
8. Add scripts/test-runner.sh Section 16c (self-test + voice-trainer + doc checks, exact-literal greps); run the gate, confirm the brain suite RAN (BR_TESTS non-empty), then pin BRAIN_TESTS_FLOOR to the real tests N and ASSERT_BASELINE_FLOOR to the printed TOTAL_CHECKS; re-run the FULL gate green. → SC5, SC6. On-failure: CIRCUIT-BREAKER — a red gate is never committed. Checkpoint.
9. Flip architecture.md SB-S1 row → landed. Exact edit (scope-guardian/plan-critic minor 10): the row at architecture.md:78 begins | **SB-S1 — Ingest + gold signal** | — change its trailing status cell to the "landed" marker matching SB-S0's row style (verify the exact current cell text at edit time). Persist brief + plan. Commit; confirm before push (PUBLIC open/ Forgejo), only inside the window; STATE.md stays gitignored. Conditional version bump per Q4.

7. Scope fence (echo — SB-S1 does NOT)

No consolidation loop · no write to brain/profile.md (ingest writes only under ingest/) · no session-start wiring · no id threading into tributaries · no triple-post reconciliation · no content-history.md retirement · no tributary-schema edits (voice-samples store untouched; only voice-trainer.md instructions edited) · no LinkedIn API/connector · no new seam function · no state-file/two-roots reconciliation · no GUI.

8. Known limits / deferred (honest)

• Invariant is wired, not behaviourally proven. SC5 gate-enforces that voice-trainer carries the published-only wiring + exclusion; it cannot prove the agent obeys it at runtime (an instruction can be ignored). Stated in the contract doc. The strongest feasible enforcement short of running the agent.
• scan-inbox is non-destructive in S1 — processed inbox files are left in place (idempotent via published dedup). Auto-move/delete of processed inbox files (the full drop-zone lifecycle) is deferred; the operator clears inbox manually for now.
• published_date defaults to captured_at when the operator doesn't pass --date (and for scan-inbox, which has no per-file date). A real published date is operator-supplied; SB-S2's loop can refine from analytics.
• voice-samples/ stays un-provenanced. S1 keeps it as a human-curated tributary and does not retro-tag it (that would be a tributary-schema change, §7). The published-only invariant bites on the NEW gold source + the ai-draft exclusion; voice-samples is trusted as human by its existing contract.
• No cross-silo id. A published post ingested here mints a content-hash id but is NOT yet linked to its analytics/state/specifics records (SB-S3).

9. Risks

• Grammar identity edge (B1): a body containing \n---\n could confuse the header/body split — mitigated by splitting on the first sentinel only (indexOf, never global) + the fixed 5-line header; the B1 edge-case battery (§5) is the proof. Circuit-breaker at step 3.
• Silent data loss (B2) — CLOSED: id is now sha256(verbatim body)[:12] (no normalizeContent), so structurally-different posts never collide; and writePublished compares bodies on any id-collision and disambiguates rather than skipping — so no hash strategy can drop a differing gold record. Tested in publish.test.ts.
• Gate "green" honest only with deps installed — npm install scripts/brain (step 1) AND step 8 asserts BR_TESTS non-empty before pinning the floor; otherwise a warn-skip lets the gate go green while SB-S1's own tests never ran.
• Floor-bump arithmetic (corrected): ASSERT_BASELINE_FLOOR counts runtime PASS+FAIL, not static pass() sites — so do NOT hand-compute 75+3; read the printed TOTAL_CHECKS at step 8 and set the floor to it. BRAIN_TESTS_FLOOR = the actual brain tests N after the new suites land. Both pinned empirically at step 8, never guessed.
• Scope creep into S2: the temptation is to consume published/ into profile.md. Fenced by §7 + the test plan (no test asserts profile mutation; writePublished targets only ingest/).