linkedin-studio/docs/second-brain/architecture.md

# Second Brain — Architecture Design

> **Status:** architecture **approved by operator 2026-06-23**. Design phase — no code yet.
> **Boundary (confirmed 2026-06-23):** the **engine** (store schema · evolution loop · ingest seam) → **the plugin** (domain-general, shareable); the **user's data** (posts · articles · newsletters · plans · ideas) → the **per-user data dir** (`${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/`, survives reinstall); the **personal cockpit** (the operator's day-to-day operations centre) → **Maskinrommet** (a thin layer that reads/writes *through* the plugin's store, never a fork of the engine).
> **Research inputs (three parallel threads, 2026-06-23):** `research/connector-egress.md` · `research/secondbrain-sota.md` · `research/silo-inventory.md`.

## The problem

A "second brain" stores everything about one creator — posts, articles, newsletters, data, plans, ideas — and compounds it into an ever-improving, user-aligned profile. It is **memory AND an operations centre**.

The three research threads reframed the task in a decisive way:

1. **There is nothing to migrate.** M0 already routed all **12 existing per-user silos** through one tested seam (`hooks/scripts/data-root.mjs` `getDataRoot()` + its TS twin `scripts/analytics/src/utils/storage.ts`), default `${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/`. They all survive reinstall today. The task is **unify**, not move.
2. **Not a vector/graph DB.** For *our* case the portable, auditable, drift-resistant 2026 standard is **plain-text + JSON + git** — because **Claude *is* the semantic-retrieval engine** (it reads files in-context). A heavyweight store adds infra, breaks portability, and invites lock-in for no gain.
3. **Auto-ingest exists — for an EU/EEA member.** The DMA Member Data Portability API is self-serve in EU/EEA + Switzerland and gives automated pull of **content** (post text + articles), but **not** received-analytics. Analytics stays the existing manual CSV path. Scraping is a ToS breach with active enforcement → never baked in. So ingest is **manual-first as the contract**; any connector is an additive tributary.

The real gap (from the silo inventory): the 12 silos are **per-user but siloed and heterogeneous** — three storage idioms, two roots, no cross-references, provenance reinvented under different field names in each. A published post lands in three non-referencing places with no shared id. The question *"which raw material actually performs?"* (specific → post → measured analytics) is **unanswerable today** — and it is exactly what a second brain should answer.

## The shape: a thin Markdown hub over typed tributaries

```
${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/
  brain/
    index.md       # MOC / hub — one-screen pointer to every tributary + freshness flags
                   #   (the "memory AND ops centre" front page)
    profile.md     # SEMANTIC, two-layer: §static (stable) + §dynamic (evolving).
                   #   Each fact: value · first_seen · last_seen · evidence_count ·
                   #   provenance(human|published|ai-draft) · status(active|superseded)
    operations.md  # PLANS / IDEAS + the "who I am now" anchor (frozen-past-self guard)
    journal/       # EPISODIC, append-only: YYYY-MM-session.md — raw, never edited;
                   #   the source the consolidation loop distils FROM
  ingest/
    inbox/         # NEW drop-zone: manual import OR (EU/EEA) connector output
    published/     # processed, provenance=published — the GOLD signal
  voice-samples/   # tributary (style)            — stays its own store
  specifics-bank/  # tributary (raw material)     — stays its own store
  trends/          # tributary (external signal)  — stays its own store
  analytics/       # tributary (performance)      — stays its own store
```

### Tributaries vs hub — the explicit call

- **Stay separate (tributaries), feed the hub via distilled pointer-summaries:** voice-samples, specifics-bank, trends, analytics, ingest. Each has a *different dedupe key + lifecycle* (trends: title+url, fast decay · specifics-bank: human-only, near-permanent · analytics: numeric, immutable). Forcing one schema would destroy those distinctions — the silo inventory warns of this explicitly. The hub holds a distilled summary of each + a pointer.
- **Fold INTO the hub:** the flat `profile/user-profile.md` → two-layer `brain/profile.md`. Scattered `audience-insights/` / `examples/` become dynamic-layer sections or tributary-summaries referenced from `index.md`. The **operations/plans centre is genuinely new**.

### Alternatives considered (and rejected)

- **Vector/graph DB (Pinecone/Neo4j/Zep-style):** rejected — adds infra, breaks plain-text portability + git audit/rollback, invites lock-in, and duplicates retrieval that Claude already does in-context. The four SOTA ideas that port cleanly to files (episodic/semantic split, provenance-weighting, evidence-threshold promotion, temporal-validity frontmatter) give most of the value; the vector/graph machinery does not port and is not needed.
- **One unified schema across all silos:** rejected — destroys each silo's distinct dedupe key + lifecycle. Keep tributaries; add a thin hub + a cross-silo id.

## Invariants (the spine)

- **Provenance-weighted learning (the critical guard):** profile/voice learn from `provenance=published` (human-edited) **only**, never `ai-draft`. This is the model-collapse guard for our exact risk — a content engine that learns voice from what it itself drafted collapses toward its own priors. Generalises `voice-scrubber`'s existing rule ("gold standard = approved editions, not the English post corpus") to a system-wide invariant.
- **One canonical entity id + one provenance shape**, threaded through the silos → makes the post ↔ specific ↔ trend ↔ analytics graph assemblable. Fixes the inventory's hardest problem.
- **Episodic / semantic physically separated** (raw events in `journal/`, distilled facts in `profile.md`).
- **Evidence-threshold promotion (anti-overfit):** the dynamic layer decays; a fact enters the **static** layer only on explicit confirmation OR ≥N independent observations. Single weird sessions never reshape identity.
- **Temporal-validity + freshness nudge:** every fact carries `last_seen`; retrieval weights recent over old; the session-start nudge flags facts not refreshed in T days (extend the existing trend-staleness nudge brain-wide).
- **Anti-sycophancy as a built-in default, not a toggle:** consolidation + content agents treat the profile as evidence to *test*, not flatter. (Personalization context measurably *increases* sycophancy, so counter-pressure it every time the profile is used.)
- **Frozen-past-self guard:** `operations.md` holds a periodic user-authored "where I'm headed now" anchor that deprecates older inferences.
- **Zero required curation (graveyard guard):** the loop compounds value with ~zero mandatory upkeep; curation is available but never required. git = free rollback.

## The evolution loop (the compounding mechanism)

Run on a cadence (reuse the existing session-start staleness-nudge surface) as a **sleep-time consolidation pass**, not inline cost:

1. **Capture (episodic, auto):** real-signal sessions append to `brain/journal/`. The ingest seam (`ingest/inbox/`) takes the user's actual published posts — manual import first; published output tagged `provenance=published`.
2. **Consolidate (reflection):** an Opus agent reads new journal + published + tributary deltas, proposes profile updates as a **diff**, never a silent overwrite. Each candidate carries `evidence_count`, `provenance`, timestamps.
3. **Promote with thresholds:** static-layer entry needs confirmation or ≥N observations; one-offs stay in the decaying dynamic layer.
4. **Reconcile contradictions:** classify temporal-update (supersede, keep old `status: superseded` + date) · contradictory (keep both, timestamped) · condition-dependent (scope) · distractor (drop). Bias toward "keep both + surface the conflict" over silent overwrite.
5. **Decay + freshness:** weight recent over old; flag stale facts at session-start.
6. **Curate (optional, never required):** the diff is presentable; confirm/edit/delete/roll-back via git.

## Build sequence (full ambition, incremental — one slice per session)

| Slice | Content | Why this order |
|-------|---------|----------------|
| **SB-S0 — Foundation** | `brain/` scaffold + two-layer `profile.md` (fold in `user-profile.md`) + `index.md` MOC + `operations.md` + `journal/`; entity-id + provenance shape as a small typed, tested module; `ingest/` dirs + manual-import contract. **No loop yet.** | Smallest thing that stands up and is testable; locks the id/provenance spine everything hangs on |
| **SB-S1 — Ingest + gold signal** | Manual import → `ingest/published/` with `provenance=published`; wire profile/voice to learn from published-only | The gold signal before the loop that consumes it |
| **SB-S2 — Evolution loop** | Sleep-time consolidation (reuse session-start surface): journal+published+tributary deltas → profile **diff** w/ evidence_count/provenance/timestamps; threshold promotion; contradiction reconciliation; brain-wide freshness nudge | The compounding mechanism |
| **SB-S3 — Cross-silo graph + ops centre** | Thread the id through tributaries (post↔specific↔trend↔analytics assemblable); flesh out `operations.md`; retire the dead `content-history.md` + triple-post reconciliation | Finally answers "which raw material performs?" |
| *(later / optional)* **SB-S4** | EU/EEA DMA portability API as an auto-tributary into `ingest/inbox/` | Additive; never a dependency |

## What's genuinely hard (honest flags)

1. **Voice fidelity to a private individual is limited even at SOTA** — the real win is grounded *content* (specifics-bank) over mimicked *style*. Manage expectations.
2. **Detecting that a high-relevance fact went stale is UNSOLVED field-wide** (STALE benchmark: all tested models/frameworks fail). We timestamp + nudge; we cannot auto-detect "changed jobs." Keep a human in that loop.
3. **Contradiction classification** (real change vs context-scoped vs noise) is emerging + error-prone → bias to keep-both-timestamped.
4. **The connector** — no clean LinkedIn self-serve content API outside EU/EEA. Build the manual ingest seam as the contract; any connector is a tributary, never a dependency. Newsletter-edition coverage by the portability `ARTICLES` domain is **not fully verified** — check against a real export.
5. **Sycophancy is structural** — app-level mitigations reduce, not eliminate.
6. **Avoiding the graveyard is a product problem** — the loop must compound value with ~zero required curation or it dies in 6 months like 82% of second brains.

## Verification (SB-S0, when we build it)

- Set `LINKEDIN_STUDIO_DATA` to a temp dir, run init → assert `brain/{index,profile,operations}.md` + `journal/` + `ingest/{inbox,published}` exist via the `getDataRoot` seam.
- `profile.md` parses two-layer (§static / §dynamic); `user-profile.md` fields fold in without loss (diff check).
- entity-id module: deterministic id mint + provenance shape, unit-tested.
- **No regression:** gate `scripts/test-runner.sh` 89/0/0 green; trends 24/24; specifics 28/28; contract 33/33.
- **Key assumption to test early:** "Claude-as-retrieval-engine over plain files is sufficient (no vector DB)" — testable with a retrieval scenario once the brain holds content. Marked as assumption until proven.

## Bottom line

A thin two-layer Markdown hub (`brain/`: semantic `profile.md` + episodic `journal/` + ops `operations.md` + `index.md` MOC) over the existing typed tributaries, fed by a provenance-tagged ingest seam, maintained by a sleep-time consolidation loop with evidence-threshold promotion, temporal-validity reconciliation, and a built-in anti-sycophancy / anti-collapse stance. Every silo stays a tributary except the flat `user-profile.md`, which folds in.