linkedin-studio/docs/second-brain/plan-sb-s1.md

# Implementation Plan — SB-S1 (Ingest + gold signal)

> **Status:** review-hardened (`plan-critic` REVISE 63→folded; `scope-guardian` GAP→folded). Ready for operator approval / "go".
> **Brief:** `brief-sb-s1.md` (review-hardened). **Arc:** `architecture.md`. **Predecessor:** SB-S0 (`scripts/brain/`).
> **Review delta (2026-06-23):** B1 separator pinned to one exact string + edge battery; B2 id → verbatim-body hash (no `normalizeContent`) + collision-safe write; floors pinned EMPIRICALLY (not hand-arithmetic); SC5 grep → exact `grep -F` literals + `AI-generated` negative probe; CLI = net-new flag infra + `init` regression; per-step On-failure/Checkpoint + 2 circuit-breakers; D3 auto-append FENCE restored; voice-trainer anchors fixed (:136/:144); malformed-file try/catch + scanInbox `*.md`-only filter.
> **Scope:** SB-S1 only — the published-record data layer + `brain ingest` CLI, and wiring `voice-trainer` to the published-only invariant (gate-enforced). **No consolidation loop, no automatic profile mutation, no session-start wiring, no cross-silo threading, no connector** (brief §7).

---

## 1. What SB-S1 delivers (recap)

A new TS module `scripts/brain/src/ingest.ts` (matching the SB-S0/specifics-bank idiom) defining a `PublishedRecord` shape with a pinned file-per-post **constrained-header grammar** (no YAML) that round-trips, a content-hash id over the **verbatim body** (`sha256(body)[:12]` — byte-identity dedup, no normalization), and idempotent, collision-safe IO (`writePublished`, `scanInbox`, `listPublished`) under the brain package's `dataRoot`. The `brain` CLI grows `ingest` + `published list` subcommands. `agents/voice-trainer.md` is wired to add `ingest/published/` as the primary gold source and forbid learning from `provenance=ai-draft`, and that wiring is **gate-enforced** by a new structure-lint (Section 16c). All unit-tested, TDD (failing test first). Gate stays green; BRAIN + assertion floors rise.

## 2. Package additions (`scripts/brain/`) — no new package, extend the existing one

```
scripts/brain/
  src/
    ingest.ts          # NEW — PublishedRecord type + parse/serialize (pure) + writePublished/scanInbox/listPublished (IO)
    id.ts              # EDIT — add normalizeContent() + mintContentId() (the specifics-bank content-hash idiom)
    cli.ts             # EDIT — add `ingest` + `published` dispatch (parseFlags idiom from specifics-bank/src/cli.ts)
    (types.ts, profile.ts, scaffold.ts, dataRoot.ts unchanged)
  tests/
    ingest.test.ts     # NEW — record round-trip + id determinism/dedup (pure)
    publish.test.ts    # NEW — temp-dir IO: write/idempotency/scan-inbox/create-on-demand
  fixtures/
    sample-inbox-post.md  # NEW — a synthetic published post for scan-inbox + round-trip tests
```

No `package.json`/`tsconfig.json` change (devDeps already tsx+typescript; no new dep — the grammar is regex, no YAML). The test command (`node --import tsx --test tests/*.test.ts`) already globs `tests/*.test.ts`, so the two new suites are picked up automatically.

## 3. Module designs

### 3.1 `id.ts` addition — `mintContentId` on the VERBATIM body (no normalization — fixes plan-critic B2)
**Verify-first (step 1):** mirror the `specificId` hash idiom at `scripts/specifics-bank/src/bank.ts:57` (the function is named `specificId`, NOT `mintContentId` — that name is net-new here; `normalizeContent` lives at `bank.ts:52`). **Decision: do NOT reuse `normalizeContent` for a post body.**
```ts
// mintContentId: sha256 of the VERBATIM body, first 12 hex. The id IS byte-identity.
export function mintContentId(text: string): string {
  return createHash("sha256").update(text).digest("hex").slice(0, 12);
}
```
- **Why verbatim, not `normalizeContent`:** `normalizeContent` (bank.ts:52) does `trim().toLowerCase().replace(/\s+/g," ")` — correct for *short* specifics dedupe, but for a full post body it collapses ALL whitespace + case, so two **structurally different** published posts (same words, different line breaks/paragraphing/case) would mint the **same id** and the second would be silently skipped on write (data loss of the exact gold signal SB-S1 exists to capture, plan-critic B2). Hashing the verbatim body makes the id true byte-identity: only an identical re-ingest collides; any real difference mints a distinct id.
- `mintContentId` is content-keyed — distinct from `mintEntityId({kind,key})` which slugifies a *label* (a body must never be slugified). Pure, deterministic. SC3.
- **Defence-in-depth (write-side, §3.3):** even with byte-identity hashing, `writePublished` compares bodies on any id-collision and never silently drops a differing body — so no hash strategy can cause data loss.

### 3.2 `ingest.ts` — `PublishedRecord` + constrained-header grammar (no YAML)
```ts
export interface PublishedRecord {
  id: string;            // mintContentId(body) — 12 hex, the dedupe key + filename stem
  provenance: 'published';  // S1 only ever writes `published`; the type pins it
  published_date: string;   // YYYY-MM-DD (operator-supplied or defaults to captured_at)
  captured_at: string;      // YYYY-MM-DD (ingest run date; supplied by caller — pure fn)
  source: string;           // 'manual' (default) | future connector token
  body: string;             // the VERBATIM post text (may contain ] | quotes newlines ---)
}
```
**File grammar** (`ingest/published/<id>.md`) — the EXACT byte layout, used identically by serialize and parse (fixes plan-critic B1):
```
id: <12hex>
provenance: published
published_date: YYYY-MM-DD
captured_at: YYYY-MM-DD
source: manual
---
<verbatim body>
```
- **Pinned separator (one string, both directions):** the five header lines, then a line that is exactly `---`, then the verbatim body. Concretely: `serialize` = `headerLines.join("\n") + "\n---\n" + body`. There is **NO** blank-line padding around the `---` and **NO** `# Published Record` title (both removed — they were the B1 contradiction). The header block is fixed at exactly 5 lines.
- **`parsePublishedRecord(text)`** → split on the FIRST occurrence of `\n---\n`. Left = header (parse the 5 scalars by the in-tree scalar-regex idiom: `^id:\s*([0-9a-f]{12})$/m`, `^published_date:\s*(\d{4}-\d{2}-\d{2})$/m`, etc.). Right = `body`, **verbatim, byte-exact** (a body that itself contains a `\n---\n` is preserved — we split on the FIRST sentinel only, via `indexOf`, never a global split). **`provenance`** is parsed and asserted to equal `published` (a `published/` record with any other provenance is a corruption signal → throw/skip, never silently accepted — fixes plan-critic minor 11; the type pins `'published'`).
- **`parse(serialize(rec)) === rec`** over all six fields incl. the body (SC2). **Edge cases the round-trip test MUST cover** (plan-critic B1): empty body (`""`), body = `"\n"`, body with trailing newlines, body starting with `---`, body containing `\n---\n` mid-text, and a body whose first line is header-shaped (e.g. `id: 0123456789ab`). Because the header is a fixed 5-line block terminated by the first `\n---\n`, a header-shaped body line cannot leak into the header (it sits after the sentinel). The only constraint on the body: none needed — it is captured byte-exact after the first sentinel.

### 3.3 `ingest.ts` — IO (impure; caller supplies the run date, like `scaffold.ts`)
- **`writePublished(rec): {written: boolean; path: string; collision?: boolean}`** — `dataRoot('ingest/published')`, mkdir-recursive (**create-on-demand — no hard dependency on `brain init`**, SC1), path = `<id>.md`. **Collision-safe compare-then-skip (fixes plan-critic B2 defence-in-depth):** if `<id>.md` exists, READ it, parse its body: (a) body byte-identical to `rec.body` → true duplicate → `{written:false}` (idempotent, no clobber — the `initBrain` discipline); (b) body DIFFERS (astronomically rare 48-bit collision, or a truncation artifact) → write to a disambiguated path `<id>-2.md` (next free suffix) and return `{written:true, collision:true}` — **never silently drop a differing body.**
- **`ingestText({body, source?, published_date?, captured_at})`** — `mintContentId(body)`, build the record (`provenance:'published'`; `published_date` defaults to `captured_at` when absent), `writePublished`. Returns the record + written flag.
- **`scanInbox({captured_at})`** — read **top-level `*.md` files only**, skipping dotfiles (`.DS_Store` etc. — macOS), no recursion (fixes plan-critic minor 9), in `dataRoot('ingest/inbox')` (empty/absent dir → clean no-op, SC4); for each, `ingestText({body: fileContent, source:'manual', captured_at})`. **Non-destructive in S1:** inbox files are LEFT in place (idempotent via the published dedup — a re-scan re-skips); auto-move/delete of processed inbox files is deferred (noted §8). Returns `{processed[], skipped[]}`.
- **`listPublished()`** — read `ingest/published/*.md`; **parse each inside a try/catch — a malformed/hand-dropped file is skipped+counted, never crashes the command** (`parsePublishedRecord` can throw via the provenance assertion / `normalizeProvenance`, id.ts:45; fixes plan-critic minor 8). Return `{records: [{id, provenance, published_date, firstLine}], skipped: number}` sorted by date. Surfaces `provenance` so the operator can eyeball that nothing `ai-draft` leaked (brief §5.1 D2).

### 3.4 `cli.ts` — add NET-NEW flag-routing infra (not just "extend dispatch" — fixes plan-critic major 6)
The brain `cli.ts` currently has **no** `parseFlags`, **no** `--json`, **no** `today()` — it dispatches purely on positional `argv[2]` (just `init`). So this step ADDS that infrastructure (copy `parseFlags` from `specifics-bank/src/cli.ts:41`; a local `today()` — `ingest.ts` itself stays pure with a caller-supplied date, so the CLI's `today()` is the only clock and does not shadow `scaffold.ts:28`'s private one). The **existing `init` positional branch is preserved verbatim**, coexisting with the new flag-driven branches:
- **`brain ingest --file <path> [--source <s>] [--date <YYYY-MM-DD>]`** — read the file, `ingestText`, print `Wrote ingest/published/<id>.md` / `Duplicate — already published: <id>` / `Collision → wrote <id>-2.md`.
- **`brain ingest --scan-inbox [--source <s>]`** — `scanInbox`, print the processed/skipped split.
- **`brain published list [--json]`** — `listPublished`; human form prints `id · provenance · date · first-line`; `--json` emits `{records:[{id,provenance,published_date,firstLine}], skipped}` (the `listPublished` return verbatim — pinned shape, fixes plan-critic minor 12).
- Usage text updated; exit 2 on usage error (existing idiom). **Regression guard:** a test (or step-5 smoke-check) asserts `brain init` still works after the refactor (a shipped subcommand must not break).

### 3.5 `agents/voice-trainer.md` — wire the published-only invariant (brief §5.1 D3, all four bullets)
Surgical edits (existing file → **Edit**, pathguard-irrelevant; model stays Sonnet). **Anchors corrected (scope-guardian DEP-1):** the Gather bullet is at **`:136`**, the "Sample Quality Priorities" heading at **`:142`** with its list at **`:144`** (the earlier `:134`/`:152` were a header and the next section).
1. **Gather step** (`:136`) — prepend the gold source + the negative rule (ADD + FORBID + KEEP):
   > **Gather (published-only gold signal first):** Read the user's published posts from `${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/ingest/published/` (`provenance=published` — the highest-trust source). Then read the existing `voice-samples/` corpus (human-curated; kept as a tributary, not reshaped) and the profile/template. **Never learn voice patterns from any content marked `provenance=ai-draft`** — a content engine that learns its own voice from its own drafts collapses toward its priors. (Generalises voice-scrubber's "gold standard = approved editions, never the draft corpus".)
2. **Sample Quality Priorities** (`:144`) — promote the published gold source to #1 and add the hard exclusion line:
   > 0. **`ingest/published/` records (`provenance=published`) — the gold signal; rank above all else.**
   > … (existing 1–5 retained) …
   > **Exclude unconditionally: anything marked `provenance=ai-draft`.**
3. **FENCE the auto-append trap** (brief §5.1 D3 bullet 4 — restored, scope-guardian SCOPE-GAP-1). Add a short note in the Gather step (and mirror it in the §3.6 contract doc): the `voice-samples/` template hints at a not-yet-built Stop-hook that auto-appends posts after content sessions (`assets/voice-samples/authentic-voice-samples.template.md:61`). State explicitly:
   > **Fence:** any future auto-capture into a learning corpus (the template's Stop-hook hint) MUST tag provenance and admit `provenance=published` only — never blind-append generated drafts.

**Exact literals (fixes plan-critic major 5 — non-vacuity):** the edits MUST write the literal strings `ingest/published` and `provenance=ai-draft`. The SC5 lint greps for these EXACT literals (`grep -F`), NOT a loose `ai.?draft`/`ai-generated` pattern — `voice-trainer.md:175` already contains "AI-generated" (a drift-cause description), so a loose grep would pass vacuously without the real exclusion being added.

### 3.6 `docs/second-brain/ingest-manual-import.md` — the manual-import contract
A short design doc: the `PublishedRecord` file grammar (§3.2), the CLI usage (`brain ingest --file` / `--scan-inbox`, `brain published list`), the inbox→published drop-zone model, the **published-only learning rule** (with the honest runtime limit: an agent instruction can be ignored at runtime; the *wiring* is gate-enforced, the *behaviour* is not provable), and the **auto-append fence** (§3.5 bullet 3 — any future auto-capture must tag provenance and admit `published` only). Includes the exact literal `provenance=ai-draft` (so SC5's doc-grep is non-vacuous). Lives in `docs/` (not `references/`) → no reference-count guard.

### 3.7 `scripts/test-runner.sh` — new SC5 lint (Section 16c) + floor bumps
- **New Section 16c "Brain Published-Only Invariant (SB-S1)"** placed after Section 16b (Brain Foundation), mirroring Section 17's structure: a non-vacuity self-test (POSITIVE/NEGATIVE heredocs) + the file checks. Greps use the **exact literals** `ingest/published` and `provenance=ai-draft` via `grep -F` / `grep -qiE` with the `provenance=` qualifier — never a loose `ai.?draft`/`ai-generated` pattern (plan-critic major 5). Three `pass()` lines (none in a loop):
  1. **self-test** — POSITIVE probe contains BOTH literals → must pass; NEGATIVE probes: one missing `ingest/published`, one missing `provenance=ai-draft`, **and one containing `AI-generated` but NOT `provenance=ai-draft`** (proves the check doesn't pass on voice-trainer's pre-existing drift prose) → each must fail. Non-vacuous like Sections 8/13/17.
  2. **`voice-trainer.md`** contains `ingest/published` AND `provenance=ai-draft`.
  3. **`docs/second-brain/ingest-manual-import.md`** contains `provenance=ai-draft` (the rule is documented).
- **`BRAIN_TESTS_FLOOR=34` (`:714`) → the actual post-implementation count.** Pinned at step 8 by reading the brain suite's real `tests N` AFTER the new suites land (TDD determines it; do NOT guess). **Step 8 must first assert the brain suite actually RAN (`BR_TESTS` non-empty — i.e. tsx installed) before pinning**, else a warn-skip would let the gate go green while SB-S1's own tests never ran (plan-critic major 4).
- **`ASSERT_BASELINE_FLOOR=75` (`:798`) → the actual printed `TOTAL_CHECKS`.** The floor counts **runtime** `PASS+FAIL` increments, not static `pass()` call-sites (the file has 72 static sites but a 75 floor because some checks loop). The three new Section 16c checks are non-looping, so the delta is +3 → 78 — **but pin it empirically: run the gate at step 8, read the printed `TOTAL_CHECKS`, set the floor to that exact number** rather than hand-arithmetic (plan-critic major 3). Update the inline history comment at `:796`.

## 4. Files created / edited

**Created:** `scripts/brain/src/ingest.ts` · `scripts/brain/tests/{ingest,publish}.test.ts` · `scripts/brain/fixtures/sample-inbox-post.md` · `docs/second-brain/ingest-manual-import.md`.

**Edited:** `scripts/brain/src/id.ts` (+normalizeContent/+mintContentId) · `scripts/brain/src/cli.ts` (+ingest/+published) · `agents/voice-trainer.md` (gather + priorities) · `scripts/test-runner.sh` (Section 16c + two floor bumps) · `docs/second-brain/architecture.md` (flip SB-S1 row → landed).

**Conditional (only if Q4 = bump to 0.5.1):** `CLAUDE.md` header + README badge + CHANGELOG + the marketplace catalog `ref` (via `catalog/scripts/release-plugin.mjs` — the polyrepo release path; tag + catalog bump atomic). Decided at "go".

## 5. TDD test plan (failing-first) → success criteria

| Test (`tests/*.test.ts`) | Asserts | SC |
|---|---|---|
| `ingest.test.ts` | `parsePublishedRecord(serializePublishedRecord(rec)) === rec` over the **B1 edge-case battery**: empty body, body=`"\n"`, trailing newlines, body starting with `---`, body containing `\n---\n` mid-text, header-shaped first body line (`id: 0123456789ab`), and a body with `]`/`|`/quotes; `mintContentId` deterministic (same body→same id; whitespace/case difference→DIFFERENT id, proving no normalize-collision); a `published/` record with non-`published` provenance throws (corruption assertion) | SC2, SC3 |
| `publish.test.ts` | `LINKEDIN_STUDIO_DATA`=tmp, NO prior init → `writePublished` creates `ingest/published/<id>.md` with `provenance=published` + verbatim body (create-on-demand); re-ingest identical body = `{written:false}`, no dup; **id-collision with a DIFFERING body → disambiguated `<id>-2.md`, no data loss**; `scanInbox` processes top-level `*.md` only (skips a `.DS_Store` fixture), → published; empty/absent inbox = no-op; re-scan re-skips; `listPublished` skips a malformed dropped file without throwing; **`brain init` still works after the CLI refactor** (regression) | SC1, SC3, SC4 |
| (gate) `scripts/test-runner.sh` Section 16c | self-test non-vacuous (incl. the `AI-generated`-but-not-`ai-draft` negative probe); `voice-trainer.md` + contract doc carry the exact literals | SC5 |
| (gate) `scripts/test-runner.sh` | green; trends ≥24, specifics ≥28, contract ≥33, **brain ≥ new floor (brain suite confirmed RAN, not skipped)**; hooks pass; assertion floor = printed `TOTAL_CHECKS` | SC6 |
| (n/a) | no new seam fn; `data-root.test.mjs` twin-sync untouched | SC6 |

Each test written **before** its module (iron law): red → implement → green.

## 6. Step sequence

**Discipline (plan-critic major 7):** each TDD step is a **Checkpoint** — commit after its cycle goes green (small, revertible commits). Each step carries an **On-failure** clause; two hard **circuit-breakers**: if step 3 (round-trip) cannot be made green after pinning the separator, HALT (the grammar is wrong — do not proceed); if step 8 (full gate) is not green, HALT (do not commit a red gate).

1. **Verify-first:** read `scripts/specifics-bank/src/bank.ts:52,57` → confirm the `specificId` hash idiom (mirror it as `mintContentId` on the VERBATIM body; do NOT reuse `normalizeContent`). Confirm `scripts/brain/node_modules/.bin/tsx` exists (else `npm install` in `scripts/brain` — the gate warn-skips a depless suite; "green" must not mean green-because-skipped). *On-failure: if tsx won't install, STOP and report — the slice cannot be gate-verified.*
2. TDD `id.ts` `mintContentId` (`ingest.test.ts` id cases: determinism + whitespace/case → different id). → SC3. *Checkpoint.*
3. TDD `ingest.ts` grammar parse/serialize over the B1 edge-case battery (`ingest.test.ts` round-trip). → SC2. *On-failure: CIRCUIT-BREAKER — if identity can't hold, the separator is wrong; halt and re-pin §3.2 before any further step.* *Checkpoint.*
4. Write `fixtures/sample-inbox-post.md` (+ a `.DS_Store` + a malformed file in test temp dirs); TDD `ingest.ts` IO — `writePublished` (incl. collision→disambiguate), `ingestText`, `scanInbox` (filter), `listPublished` (try/catch) (`publish.test.ts`). → SC1, SC4. *Checkpoint.*
5. Add the CLI flag-routing infra + `ingest`/`published` branches (preserve `init`); regression-test `brain init`; smoke-run against a temp `LINKEDIN_STUDIO_DATA` (ingest a file, scan an inbox, list). *On-failure: if `brain init` regresses, revert the CLI refactor and isolate.* *Checkpoint.*
6. Edit `agents/voice-trainer.md` (`:136` gather + `:144` priorities + the `provenance=ai-draft` exclusion + the auto-append fence — exact literals). *Checkpoint.*
7. Write `docs/second-brain/ingest-manual-import.md` (incl. the `provenance=ai-draft` literal). *Checkpoint.*
8. Add `scripts/test-runner.sh` Section 16c (self-test + voice-trainer + doc checks, exact-literal greps); run the gate, **confirm the brain suite RAN (`BR_TESTS` non-empty)**, then pin `BRAIN_TESTS_FLOOR` to the real `tests N` and `ASSERT_BASELINE_FLOOR` to the printed `TOTAL_CHECKS`; re-run the FULL gate green. → SC5, SC6. *On-failure: CIRCUIT-BREAKER — a red gate is never committed.* *Checkpoint.*
9. Flip `architecture.md` SB-S1 row → landed. **Exact edit (scope-guardian/plan-critic minor 10):** the row at `architecture.md:78` begins `| **SB-S1 — Ingest + gold signal** |` — change its trailing status cell to the "landed" marker matching SB-S0's row style (verify the exact current cell text at edit time). Persist brief + plan. **Commit; confirm before push (PUBLIC `open/` Forgejo), only inside the window; STATE.md stays gitignored.** Conditional version bump per Q4.

## 7. Scope fence (echo — SB-S1 does NOT)

No consolidation loop · **no write to `brain/profile.md`** (ingest writes only under `ingest/`) · no session-start wiring · no id threading into tributaries · no triple-post reconciliation · no `content-history.md` retirement · no tributary-schema edits (voice-samples store untouched; only `voice-trainer.md` instructions edited) · no LinkedIn API/connector · no new seam function · no state-file/two-roots reconciliation · no GUI.

## 8. Known limits / deferred (honest)

- **Invariant is wired, not behaviourally proven.** SC5 gate-enforces that `voice-trainer` *carries* the published-only wiring + exclusion; it cannot prove the agent *obeys* it at runtime (an instruction can be ignored). Stated in the contract doc. The strongest feasible enforcement short of running the agent.
- **scan-inbox is non-destructive in S1** — processed inbox files are left in place (idempotent via published dedup). Auto-move/delete of processed inbox files (the full drop-zone lifecycle) is deferred; the operator clears inbox manually for now.
- **`published_date` defaults to `captured_at`** when the operator doesn't pass `--date` (and for scan-inbox, which has no per-file date). A real published date is operator-supplied; SB-S2's loop can refine from analytics.
- **`voice-samples/` stays un-provenanced.** S1 keeps it as a human-curated tributary and does not retro-tag it (that would be a tributary-schema change, §7). The published-only invariant bites on the NEW gold source + the ai-draft exclusion; voice-samples is trusted as human by its existing contract.
- **No cross-silo id.** A published post ingested here mints a content-hash id but is NOT yet linked to its analytics/state/specifics records (SB-S3).

## 9. Risks

- **Grammar identity edge (B1):** a body containing `\n---\n` could confuse the header/body split — mitigated by splitting on the **first** sentinel only (`indexOf`, never global) + the fixed 5-line header; the B1 edge-case battery (§5) is the proof. Circuit-breaker at step 3.
- **Silent data loss (B2) — CLOSED:** id is now `sha256(verbatim body)[:12]` (no `normalizeContent`), so structurally-different posts never collide; and `writePublished` compares bodies on any id-collision and disambiguates rather than skipping — so no hash strategy can drop a differing gold record. Tested in `publish.test.ts`.
- **Gate "green" honest only with deps installed** — `npm install scripts/brain` (step 1) AND step 8 asserts `BR_TESTS` non-empty before pinning the floor; otherwise a warn-skip lets the gate go green while SB-S1's own tests never ran.
- **Floor-bump arithmetic (corrected):** `ASSERT_BASELINE_FLOOR` counts **runtime** `PASS+FAIL`, not static `pass()` sites — so do NOT hand-compute 75+3; read the printed `TOTAL_CHECKS` at step 8 and set the floor to it. `BRAIN_TESTS_FLOOR` = the actual brain `tests N` after the new suites land. Both pinned empirically at step 8, never guessed.
- **Scope creep into S2:** the temptation is to consume `published/` into `profile.md`. Fenced by §7 + the test plan (no test asserts profile mutation; `writePublished` targets only `ingest/`).