linkedin-studio/docs/second-brain/brief-sb-s1.md

Task Brief — SB-S1 (Ingest + gold signal)

Status: DRAFT for adversarial brief-review (light-Voyage step 2). Not yet operator-approved. Arc: architecture.md (approved 2026-06-23). Predecessor: SB-S0 landed (8c92719) — scripts/brain/ (types/id/profile/scaffold/cli, 34 tests, gate-wired). This brief: the second slice in detail; S2–S4 stay one-line in the arc. Inputs: architecture.md §"evolution loop" steps 1–2, §"invariants" (provenance-weighted learning); SB-S0 code under scripts/brain/; the silo inventory (docs/second-brain/research/silo-inventory.md). Review delta (2026-06-23): folded the brief-review (PROCEED_WITH_RISKS). Resolver standardised on dataRoot (M-resolver); D3 voice-samples↔ingest relationship pinned (M-D3); SC5 hardened to a gate-checkable lint (M-SC5); test-runner floor edits named as deliverables (m-floor); input path + D4 net-new note + Q3 sharpened (minors).

---

1. Objective

Capture the gold signal — the user's actual published posts — into ingest/published/ tagged provenance=published, and wire the voice/profile-learning surface to learn from that published signal only, never from AI-drafted content. This is the one invariant the whole second-brain stands or falls on (the model-collapse guard): a content engine that learns its own voice from what it itself drafted collapses toward its own priors. SB-S1 lays the gold signal before the consolidation loop (SB-S2) that will consume it.

2. Background (why this slice, why now)

• The invariant already has a single-agent precedent. voice-scrubber enforces "gold standard = approved Norwegian editions, never the English post corpus." SB-S1 generalises that one agent's rule into a system-wide, data-backed invariant: learning consumes provenance=published only.
• The signal is genuinely new, not a re-count of existing silos. A published post lands today in three non-referencing places — state ## Recent Posts (hook + topic only), the dead analytics/content-history.md, and analytics/posts/*.json (metrics, from CSV). None holds the full post text. Voice learning needs the verbatim text; ingest/published/ holding it is additive, not redundant. (Cross-silo threading of a shared id is SB-S3, explicitly out of scope here.)
• The provenance shape is already in place. SB-S0 pinned Provenance = human | published | ai-draft and put a provenance field on every ProfileFact. SB-S1 is the first slice to actually produce published records and gate a learning consumer on them — it spends the spine S0 built.
• Manual-first is the contract. Auto-ingest (EU/EEA DMA portability) is a later additive tributary (SB-S4); SB-S1's ingest is the manual path: the user supplies the published post text, the CLI lands it as a gold record.

3. Confirmed decisions (inherited from the approved arc — not for re-litigation)

1. Boundary: engine → plugin · user data → data dir · cockpit → Maskinrommet.
2. Plain-text/JSON + git, no vector/graph DB. Claude is the retrieval engine.
3. Provenance-weighted learning (published only, never ai-draft) is an invariant, not a toggle.
4. Tributaries keep their own schema/lifecycle — SB-S1 does NOT reshape voice-samples, specifics, trends, or analytics stores.
5. No new seam function — ingest paths resolve through the brain package's own inlined dataRoot(sub) resolver (scripts/brain/src/dataRoot.ts, the repo idiom — NOT the hooks-side getDataRoot() seam, a different function in a different package). So the twin-sync surface (data-root.mjs ⇄ storage.ts) stays untouched.

Review SHOULD challenge: the S1/S2 boundary (§5/§7), the store format (§5.1), feasibility, hidden coupling, testability, scope leak, missing risks. Review SHOULD NOT re-open the five decisions above unless internally contradictory or technically impossible.

4. Scope — where SB-S1 sits in the arc

Slice	One-line scope	This brief
SB-S0 — Foundation	scaffold + profile fold + id/provenance spine	✅ landed
SB-S1 — Ingest + gold signal	manual import → ingest/published/ provenance=published; wire the voice/profile-learning surface to published-only	← this
SB-S2 — Evolution loop	sleep-time consolidation: journal+published+tributary deltas → profile diff (evidence/provenance/timestamps); threshold promotion; contradiction reconciliation; freshness nudge. Owns session-start wiring of init.	later
SB-S3 — Cross-silo graph + ops centre	thread the id through tributaries (post↔specific↔trend↔analytics); flesh out operations.md; retire dead content-history.md	later
SB-S4 (optional)	EU/EEA DMA portability as auto-tributary into ingest/inbox/	later

5. SB-S1 in detail (the build target)

5.1 Deliverables

1. A published-record data layer (TS, scripts/brain/). A PublishedRecord shape and pure parse/serialize functions for a published post file. Reuse the SB-S0 idioms: mintEntityId for the canonical id, the constrained-header line-grammar idea (NO YAML dep), normalizeProvenance for the provenance token. Fields (proposed, open to the plan): id (sha256[:12] of normalized post text) · provenance (always published) · published_date (YYYY-MM-DD) · captured_at (YYYY-MM-DD) · source (manual | future connector) · the verbatim post body. Pure functions, deterministic, unit-tested.

2. A brain ingest CLI extending scripts/brain/src/cli.ts (same dispatch idiom). Proposed subcommands (final shape open to the plan):

   • brain ingest <file> — read a file containing one published post, mint id, write ingest/published/<id>.md with provenance=published. Idempotent: same text → same id → no duplicate (compare-then-skip, matching initBrain).
   • brain ingest --scan-inbox — process every file in ingest/inbox/ into ingest/published/, then mark/clear the inbox source (drop-zone → processed, per the architecture's inbox→published model).
   • brain published list — inspect the gold corpus (id · provenance · date · first line). Surfacing provenance per record makes the invariant observable — the operator can eyeball that nothing ai-draft leaked into published/.

3. Wire the voice/profile-learning surface to published-only (the invariant made enforceable). Edit agents/voice-trainer.md's "Gather" step. The pinned relationship to the existing corpus (resolves brief-review M-D3):

   • ADD ingest/published/ (provenance=published) as the primary gold source, ranked above the legacy voice-samples/ corpus.
   • KEEP voice-samples/ as a human-curated tributary — not reshaped, not redirected, not abandoned (§7 honoured). By its existing contract its files are the user's own real writing (human-supplied via /setup//onboarding); they are not ai-draft. voice-trainer keeps reading it but treats ingest/published/ as the higher-trust signal.
   • FORBID learning voice/profile patterns from any content marked provenance=ai-draft — the explicit negative rule that generalises voice-scrubber's "gold standard = approved editions, never the draft corpus" precedent to the system invariant.
   • FENCE the auto-append trap: the voice-samples/ template hints at a (not-yet-built) Stop-hook that auto-appends posts after content sessions. S1 states explicitly that any such future auto-capture MUST tag provenance and admit published only — never blind-append generated drafts into a learning corpus. (Forward-looking guard, no S1 code; names the exact model-collapse trap so a later slice cannot reintroduce it.)
   • voice-trainer stays Sonnet (no model/frontmatter change → no structure-lint model-consistency break).

4. Manual-import contract doc — docs/second-brain/ingest-manual-import.md: the published-record file format, the CLI usage, and the published-only learning rule. (Design doc → docs/, NOT references/, so the reference-doc count guard does not fire.) Net-new (verified): SB-S0 shipped the ingest/{inbox,published} dirs only; no contract doc exists yet — the architecture's "SB-S0 … manual-import contract" wording referred to the dir shape, not a doc.

5. Tests (TDD, failing-first) — published-record parse/serialize round-trip (incl. a body with ]/|/quotes/newlines), id determinism + dedup, CLI idempotency on re-ingest and --scan-inbox, source-absent / empty-inbox no-op. Plus the SC5 published-only structure-lint (a new gate check; see SC5). Gate stays green.

6. Gate floor edits (named — Edit, not Write, on existing files → pathguard-safe): bump BRAIN_TESTS_FLOOR in scripts/test-runner.sh to the new brain-suite count, and bump the assertion-count floor (ASSERT_BASELINE_FLOOR / the TOTAL_CHECKS anti-erosion floor) by the number of new pass() lines added (the BRAIN suite line is unchanged-count; the new SC5 structure-lint check adds one). Exact line numbers confirmed in the plan against the live scripts/test-runner.sh.

7. Docs — flip the architecture.md SB-S1 row to "landed"; this brief + the SB-S1 plan persisted under docs/second-brain/.

5.2 The S1/S2 boundary (the decision the review must pin)

• SB-S1 ships: the gold-signal data layer (records + CLI), and wires the existing on-demand learning surface (voice-trainer, invoked when the user runs a voice command) to the published-only rule. This is a data module + an agent-doc edit — no scheduler, no automatic profile mutation.
• SB-S2 owns: the automated sleep-time consolidation loop (session-start cadence) that reads published + journal + tributary deltas and proposes profile.md diffs with evidence/provenance/timestamps + threshold promotion. No profile.md is auto-mutated in S1.

This boundary keeps S1 a clean, testable data+invariant slice and leaves all automatic profile evolution to S2. It is the #1 thing for the operator to confirm.

6. Success criteria (SB-S1 — testable)

• SC1 (ingest, runtime): with LINKEDIN_STUDIO_DATA set to a temp dir, brain ingest <file> creates ingest/published/<id>.md with provenance=published and the verbatim body, at the runtime data-path via the brain package's dataRoot. ingest creates ingest/published/ on demand (mkdir-recursive) — it does NOT hard-depend on a prior brain init, so ingest-before-init works. Verified by a test asserting file existence + content with no prior init.
• SC2 (record round-trip): a PublishedRecord serializes and parses back to an identical record, including a body containing ], |, quotes, and embedded newlines. Verified by a parse/serialize identity unit test.
• SC3 (id determinism + dedup): the same post text mints the same id (and re-ingesting it is a no-op — no duplicate file); two different posts mint different ids. Verified by id + idempotency unit tests.
• SC4 (scan-inbox): files dropped in ingest/inbox/ are processed into ingest/published/ by --scan-inbox; an empty inbox is a clean no-op; re-running does not re-create already-published records. Verified by a temp-dir test.
• SC5 (published-only invariant, gate-checkable): a new structure-lint check in scripts/test-runner.sh (matching the Section 8–17 honesty-lint idiom: positive + negative assertion + a non-vacuity self-test) asserts that agents/voice-trainer.md (a) names ingest/published/ in its gather/source instructions AND (b) carries the explicit negative string forbidding learning from provenance=ai-draft. The self-test proves the check is non-vacuous (it would FAIL on a fixture missing either string). The honest runtime limit (an agent instruction can still be ignored at runtime) is stated in the doc — but the presence of the wiring is gate-enforced, not checklist-trusted. (Resolves brief-review M-SC5: no "OR checklist" escape hatch.)
• SC6 (no regression / no new seam): scripts/test-runner.sh green; floors hold (trends ≥24, specifics ≥28, contract ≥33, brain ≥ its new floor); hook tests pass; SB-S1 adds no new seam function so data-root.test.mjs twin-sync is untouched.

7. Non-goals / scope fence (SB-S1 does NOT)

• No consolidation loop / no automatic profile mutation (SB-S2). brain ingest writes only under ingest/, never to brain/profile.md.
• No session-start wiring of init or ingest (SB-S2 owns scaffold wiring; ingest stays a manual CLI in S1).
• No cross-silo id threading — ingest mints its own record id but does NOT weave it into trends/specifics/analytics/state (SB-S3).
• No reconciliation of the triple-recorded post and no retirement of content-history.md (SB-S3).
• No connector / no LinkedIn API (SB-S4). source allows a future connector value but no connector code ships.
• No tributary-schema edits — voice-samples, specifics, trends, analytics stores keep their current shape. (voice-trainer.md is edited; the voice-samples store format is not.)
• No new seam function, no state-file / two-roots reconciliation, no GUI/cockpit.

8. Constraints / risks

• TDD iron law: no production code without a failing test first.
• Code placement: all new code is TS under scripts/brain/ — no new .mjs under hooks/scripts/, so the pre-write-pathguard.mjs Category-6 guard does not fire (it matches …/hooks/scripts/*.mjs on Write only; SB-S1 writes none). agents/voice-trainer.md is an existing file → Edit, not Write → unaffected regardless.
• Public repo (open/ Forgejo): confirm before push; push only inside the window; STATE.md stays gitignored.
• Structure lint: SB-S1 adds no agents/, commands/, references/, or skills/ file (ingest is a CLI subcommand in an existing package; voice-trainer is edited; the contract doc lives in docs/). So no count guard fires absent a version bump. Version bump is an open question (§9): the agent behavioural change may warrant 0.5.0 → 0.5.1.
• Node v25 test quirk: node --test <dir> fails — use node --import tsx --test tests/*.test.ts; run TS tools from their own dir; npm install in scripts/brain before the gate or the suite warn-skips.
• Risk — store-format ambiguity: the published-record file format is the one data-shaping decision in S1 (mirrors SB-S0's fold risk). Mitigate by pinning the grammar + a round-trip test (SC2) before the CLI wires it.
• Risk — invariant enforceability: "wire voice-trainer to published-only" is an agent-doc instruction, not executable code; an instruction can be ignored at runtime. Mitigate by making SC5 a gate-checkable lint where feasible (grep for the rule's presence), and by stating the limit honestly in the doc.
• Risk — S1/S2 scope creep: the temptation is to start consuming published/ into profile.md here. Fence held by §7 + SC (no test asserts profile mutation in S1).

9. Open questions for the operator (settle at "go")

1. S1/S2 boundary (§5.2): confirm S1 = data layer + voice-trainer invariant wiring, S2 = the automatic profile-diff loop. (Recommended.)
2. Profile vs voice in S1: confirm that profile learning is invariant-only in S1 (provenance already on facts; no profile mutation) and that the only active consumer wired is voice-trainer. (Recommended.)
3. Store format (~decided, confirm): published records as file-per-post constrained-header markdown (the brain/profile.md line-grammar idiom, no YAML) — matching the architecture's inbox→published per-file drop-zone. This deliberately does NOT reuse the specifics-bank single-JSON-index idiom (that store is a topic-tagged bank, not a per-file drop-zone). Confirm.
4. Version bump: take 0.5.0 → 0.5.1, or no bump? SB-S0 set a no-bump precedent for count-neutral slices, but SB-S1 differs: it changes a shipped agent's learning behaviour (voice-trainer), a user-visible capability change — a stronger case for 0.5.1 than SB-S0 had. (Lean 0.5.1.)

10. Definition of done (SB-S1)

SC1–SC6 pass; deliverables 1–7 exist; non-goals untouched; the published-only invariant is wired into voice-trainer + documented + gate-enforced (SC5 structure-lint); the BRAIN + assertion floors raised; committed (push per window, confirm first); architecture.md SB-S1 row → "landed"; version/CLAUDE.md/README touched only if a bump is taken.