linkedin-studio/docs/second-brain/architecture.md

Second Brain — Architecture Design

Status: architecture approved by operator 2026-06-23. SB-S0 (Foundation) + SB-S1 (Ingest + gold signal) + SB-S2 (Evolution loop) landed 2026-06-23 (scripts/brain/, 82 tests, gate-wired; ingest CLI + published-only invariant + operator-gated consolidation loop + session-start nudge); S3–S4 remain design-phase. Boundary (confirmed 2026-06-23): the engine (store schema · evolution loop · ingest seam) → the plugin (domain-general, shareable); the user's data (posts · articles · newsletters · plans · ideas) → the per-user data dir (${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/, survives reinstall); the personal cockpit (the operator's day-to-day operations centre) → Maskinrommet (a thin layer that reads/writes through the plugin's store, never a fork of the engine). Research inputs (three parallel threads, 2026-06-23): research/connector-egress.md · research/secondbrain-sota.md · research/silo-inventory.md.

The problem

A "second brain" stores everything about one creator — posts, articles, newsletters, data, plans, ideas — and compounds it into an ever-improving, user-aligned profile. It is memory AND an operations centre.

The three research threads reframed the task in a decisive way:

1. There is nothing to migrate. M0 already routed all 12 existing per-user silos through one tested seam (hooks/scripts/data-root.mjs getDataRoot() + its TS twin scripts/analytics/src/utils/storage.ts), default ${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/. They all survive reinstall today. The task is unify, not move.
2. Not a vector/graph DB. For our case the portable, auditable, drift-resistant 2026 standard is plain-text + JSON + git — because Claude is the semantic-retrieval engine (it reads files in-context). A heavyweight store adds infra, breaks portability, and invites lock-in for no gain.
3. Auto-ingest exists — for an EU/EEA member. The DMA Member Data Portability API is self-serve in EU/EEA + Switzerland and gives automated pull of content (post text + articles), but not received-analytics. Analytics stays the existing manual CSV path. Scraping is a ToS breach with active enforcement → never baked in. So ingest is manual-first as the contract; any connector is an additive tributary.

The real gap (from the silo inventory): the 12 silos are per-user but siloed and heterogeneous — three storage idioms, two roots, no cross-references, provenance reinvented under different field names in each. A published post lands in three non-referencing places with no shared id. The question "which raw material actually performs?" (specific → post → measured analytics) is unanswerable today — and it is exactly what a second brain should answer.

The shape: a thin Markdown hub over typed tributaries

${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/
  brain/
    index.md       # MOC / hub — one-screen pointer to every tributary + freshness flags
                   #   (the "memory AND ops centre" front page)
    profile.md     # SEMANTIC, two-layer: §static (stable) + §dynamic (evolving).
                   #   Each fact: value · first_seen · last_seen · evidence_count ·
                   #   provenance(human|published|ai-draft) · status(active|superseded)
    operations.md  # PLANS / IDEAS + the "who I am now" anchor (frozen-past-self guard)
    journal/       # EPISODIC, append-only: YYYY-MM-session.md — raw, never edited;
                   #   the source the consolidation loop distils FROM
  ingest/
    inbox/         # NEW drop-zone: manual import OR (EU/EEA) connector output
    published/     # processed, provenance=published — the GOLD signal
  voice-samples/   # tributary (style)            — stays its own store
  specifics-bank/  # tributary (raw material)     — stays its own store
  trends/          # tributary (external signal)  — stays its own store
  analytics/       # tributary (performance)      — stays its own store

Tributaries vs hub — the explicit call

• Stay separate (tributaries), feed the hub via distilled pointer-summaries: voice-samples, specifics-bank, trends, analytics, ingest. Each has a different dedupe key + lifecycle (trends: title+url, fast decay · specifics-bank: human-only, near-permanent · analytics: numeric, immutable). Forcing one schema would destroy those distinctions — the silo inventory warns of this explicitly. The hub holds a distilled summary of each + a pointer.
• Fold INTO the hub: the flat profile/user-profile.md → two-layer brain/profile.md. Scattered audience-insights/ / examples/ become dynamic-layer sections or tributary-summaries referenced from index.md. The operations/plans centre is genuinely new.

Alternatives considered (and rejected)

• Vector/graph DB (Pinecone/Neo4j/Zep-style): rejected — adds infra, breaks plain-text portability + git audit/rollback, invites lock-in, and duplicates retrieval that Claude already does in-context. The four SOTA ideas that port cleanly to files (episodic/semantic split, provenance-weighting, evidence-threshold promotion, temporal-validity frontmatter) give most of the value; the vector/graph machinery does not port and is not needed.
• One unified schema across all silos: rejected — destroys each silo's distinct dedupe key + lifecycle. Keep tributaries; add a thin hub + a cross-silo id.

Invariants (the spine)

• Provenance-weighted learning (the critical guard): profile/voice learn from provenance=published (human-edited) only, never ai-draft. This is the model-collapse guard for our exact risk — a content engine that learns voice from what it itself drafted collapses toward its own priors. Generalises voice-scrubber's existing rule ("gold standard = approved editions, not the English post corpus") to a system-wide invariant.
• One canonical entity id + one provenance shape, threaded through the silos → makes the post ↔ specific ↔ trend ↔ analytics graph assemblable. Fixes the inventory's hardest problem.
• Episodic / semantic physically separated (raw events in journal/, distilled facts in profile.md).
• Evidence-threshold promotion (anti-overfit): the dynamic layer decays; a fact enters the static layer only on explicit confirmation OR ≥N independent observations. Single weird sessions never reshape identity.
• Temporal-validity + freshness nudge: every fact carries last_seen; retrieval weights recent over old; the session-start nudge flags facts not refreshed in T days (extend the existing trend-staleness nudge brain-wide).
• Anti-sycophancy as a built-in default, not a toggle: consolidation + content agents treat the profile as evidence to test, not flatter. (Personalization context measurably increases sycophancy, so counter-pressure it every time the profile is used.)
• Frozen-past-self guard: operations.md holds a periodic user-authored "where I'm headed now" anchor that deprecates older inferences.
• Zero required curation (graveyard guard): the loop compounds value with ~zero mandatory upkeep; curation is available but never required. git = free rollback.

The evolution loop (the compounding mechanism)

Run on a cadence (reuse the existing session-start staleness-nudge surface) as a sleep-time consolidation pass, not inline cost:

1. Capture (episodic, auto): real-signal sessions append to brain/journal/. The ingest seam (ingest/inbox/) takes the user's actual published posts — manual import first; published output tagged provenance=published.
2. Consolidate (reflection): an Opus agent reads new journal + published + tributary deltas, proposes profile updates as a diff, never a silent overwrite. Each candidate carries evidence_count, provenance, timestamps.
3. Promote with thresholds: static-layer entry needs confirmation or ≥N observations; one-offs stay in the decaying dynamic layer.
4. Reconcile contradictions: classify temporal-update (supersede, keep old status: superseded + date) · contradictory (keep both, timestamped) · condition-dependent (scope) · distractor (drop). Bias toward "keep both + surface the conflict" over silent overwrite.
5. Decay + freshness: weight recent over old; flag stale facts at session-start.
6. Curate (optional, never required): the diff is presentable; confirm/edit/delete/roll-back via git.

Build sequence (full ambition, incremental — one slice per session)

Slice	Content	Why this order
SB-S0 — Foundation ✅ landed 2026-06-23	brain/ scaffold + two-layer profile.md (fold in user-profile.md) + index.md MOC + operations.md + journal/; entity-id + provenance shape as a small typed, tested module; ingest/ dirs + manual-import contract. No loop yet. Shipped as scripts/brain/ (TS, 34 tests, gate-wired BRAIN floor); fold = P1 labeled-scalars + P2 expertise (checkbox-prefs deferred, §8 of plan-sb-s0.md).	Smallest thing that stands up and is testable; locks the id/provenance spine everything hangs on
SB-S1 — Ingest + gold signal ✅ landed 2026-06-23	Manual import → ingest/published/ with provenance=published (CLI brain ingest); voice-trainer wired to learn from published-only, never ai-draft, gate-enforced. Shipped as scripts/brain/src/ingest.ts + docs/second-brain/ingest-manual-import.md (v0.5.1). No profile.md mutation (SB-S2).	The gold signal before the loop that consumes it
SB-S2 — Evolution loop ✅ landed 2026-06-23	Operator-invoked, operator-gated consolidation: brain consolidate (--gather/--propose/--apply --confirm) → profile diff w/ evidence_count/provenance/timestamps; threshold-promotion (N=3); contradiction → keep-both with distinct ids (no supersede); decay-flag (90d); consolidation-state.json sidecar; zero-dep session-start consolidation-due nudge + scaffold-ensure. Shipped as scripts/brain/src/consolidate.ts + docs/second-brain/consolidation-loop.md (v0.5.2). Operator decisions: journal deferred · no new agent (session extracts) · motor-only (no reader until S3).	The compounding mechanism
SB-S3 — Cross-silo graph + ops centre	Thread the id through tributaries (post↔specific↔trend↔analytics assemblable); flesh out operations.md; retire the dead content-history.md + triple-post reconciliation. S3a ✅ first reader · S3b ✅ supersede · S3c ✅ cross-silo id-threading (hub-side: the published record carries the specifics/trends ids it was built from + a pure analytics resolver — scripts/brain/src/assemble.ts, brain assemble; tributaries untouched). S3d ✅ ops centre (operations.md is a read tributary; strategy-advisor honours the dated "who I am now" anchor that deprecates older inferences — advisory/reader-side). S3e ✅ hygiene + reconciliation (dead content-history.md retired across its 8 plumbing surfaces; brain reconcile reconciles silo 1 ## Recent Posts ↔ the silo 2↔3 graph read-side, surfacing the coverage gap — created posts never brain ingest-ed; auto-capture of specifics/trends = a flagged follow-up). The second-brain arc is complete (S4 EØS DMA-connector optional).	Finally answers "which raw material performs?"
(later / optional) SB-S4	EU/EEA DMA portability API as an auto-tributary into ingest/inbox/	Additive; never a dependency

What's genuinely hard (honest flags)

1. Voice fidelity to a private individual is limited even at SOTA — the real win is grounded content (specifics-bank) over mimicked style. Manage expectations.
2. Detecting that a high-relevance fact went stale is UNSOLVED field-wide (STALE benchmark: all tested models/frameworks fail). We timestamp + nudge; we cannot auto-detect "changed jobs." Keep a human in that loop.
3. Contradiction classification (real change vs context-scoped vs noise) is emerging + error-prone → bias to keep-both-timestamped.
4. The connector — no clean LinkedIn self-serve content API outside EU/EEA. Build the manual ingest seam as the contract; any connector is a tributary, never a dependency. Newsletter-edition coverage by the portability ARTICLES domain is not fully verified — check against a real export.
5. Sycophancy is structural — app-level mitigations reduce, not eliminate.
6. Avoiding the graveyard is a product problem — the loop must compound value with ~zero required curation or it dies in 6 months like 82% of second brains.

Verification (SB-S0, when we build it)

• Set LINKEDIN_STUDIO_DATA to a temp dir, run init → assert brain/{index,profile,operations}.md + journal/ + ingest/{inbox,published} exist via the getDataRoot seam.
• profile.md parses two-layer (§static / §dynamic); user-profile.md fields fold in without loss (diff check).
• entity-id module: deterministic id mint + provenance shape, unit-tested.
• No regression: gate scripts/test-runner.sh 89/0/0 green; trends 24/24; specifics 28/28; contract 33/33.
• Key assumption to test early: "Claude-as-retrieval-engine over plain files is sufficient (no vector DB)" — testable with a retrieval scenario once the brain holds content. Marked as assumption until proven.

Bottom line

A thin two-layer Markdown hub (brain/: semantic profile.md + episodic journal/ + ops operations.md + index.md MOC) over the existing typed tributaries, fed by a provenance-tagged ingest seam, maintained by a sleep-time consolidation loop with evidence-threshold promotion, temporal-validity reconciliation, and a built-in anti-sycophancy / anti-collapse stance. Every silo stays a tributary except the flat user-profile.md, which folds in.