linkedin-studio/docs/second-brain/research/secondbrain-sota.md

# Research — Second Brain / Personal-Memory SOTA 2026 (synthesis + architecture rec)

> Research thread `secondbrain-recon`, 2026-06-23. Four streams (agent memory, PKM, profile-evolution, pitfalls) with ESTABLISHED / EMERGING / SPECULATIVE labels. Research INPUT to `../architecture.md` — the architecture's design choices trace to this. Preserved so nothing is lost.

## PART A — STATE OF THE ART (sourced, calibrated)

### 1. Agent memory architectures

**Three-tier taxonomy is the design default** (ESTABLISHED). Episodic (timestamped *what happened*), semantic (distilled *what is true*), procedural (*how to behave*). Formalized for LLM agents by CoALA (arXiv:2309.02427); shared vocabulary across Letta, Mem0, Zep, LangMem. Load-bearing for us: **keep episodic and semantic physically separate** — mixing raw events with distilled facts degrades retrieval both ways. Procedural is rarest; only LangMem makes it first-class.

**Hybrid vector + structured/graph beats pure vector** (ESTABLISHED). Pure embeddings fail on multi-hop, temporal ("what did the user want in Q1 vs now"), exact-identifier, and contradiction queries. 2026 pattern = query router over vector (semantic) + graph/relational (entities + temporal validity) + BM25 (keyword), fused via Reciprocal Rank Fusion + cross-encoder rerank (+39.7% MRR@3, arXiv:2604.01733). Named systems:
- **Mem0** (arXiv:2504.19413, ~48K stars) — vector-primary + entity linking; LLM extracts facts per turn; ~90% token reduction. Benchmark numbers vendor-disputed.
- **Zep/Graphiti** (arXiv:2501.13956) — **bi-temporal knowledge graph**: every fact an edge with `valid_from`/`valid_until`. 63.8% vs Mem0's 49.0% on independent LongMemEval. **Most important pattern for us** (staleness).
- **Letta/MemGPT** (arXiv:2310.08560) — OS-tiered (core/recall/archival), agent self-manages via tools. 74.0% LoCoMo.
- LangMem (EMERGING), Cognee (EMERGING, graph-RAG for docs), A-MEM (EMERGING, NeurIPS 2025).

**Consolidation = the "reflection/sleep" pattern** (ESTABLISHED original; sleep-time-compute EMERGING). Stanford Generative Agents (arXiv:2304.03442): append-only memory stream → LLM importance-scores each event → periodic reflection synthesizes higher-order insights stored back as semantics. Letta sleep-time compute (arXiv:2504.13171) runs it as background agents during idle so the main loop stays fast.

**Forgetting is mandatory** (ESTABLISHED TTL/LRU; EMERGING importance-weighted decay). "Append-only memory stores are a trap." Tiers: session ~24h, project ~6mo, core indefinite, cold-archive after 90–180d unused. Mem0 decay is a *retrieval re-ranking* layer (recent → 1.5× boost, unused → 0.3× damp), not deletion. The Generative-Agents **retrieval triad** (recency + importance + relevance) is still the reference scoring formula.

### 2. PKM methods + AI-native tools
- **PARA** (Forte) — top-down, organize-by-*actionability*; strong for ops, no native linking.
- **Zettelkasten / Evergreen** (Luhmann/Matuschak) — atomic note (one idea), dense links, **emergent** bottom-up structure; the cognitive work is *writing the note*, which embeddings can't replace.
- **LYT/Maps of Content** (Milo) — navigation layer over the graph.

**2026 consensus is NOT "links win"** — embeddings make folder-vs-link partly moot for *retrieval* while reaffirming neither scales alone. Mem (OpenAI embeddings + Pinecone), Tana (every bullet a graph node, typed Supertags), Obsidian Smart Connections (*local* embeddings — privacy). None *auto-organize* — they auto-*retrieve*. AI collapses CODE's **Organize** step (misfiling recoverable) and accelerates **Capture**; it does NOT do **Distill** (cross-temporal relevance) or **Express** (human synthesis). Sober dissent: "the best memory tool is the one you actually keep feeding" — capture/maintenance friction dominates retrieval sophistication.

### 3. Profile-evolution / preference-learning
**Two-layer profile is standard** (ESTABLISHED): static (stable) + dynamic (evolving). PersonaMem (COLM 2025): frontier models **freeze on the static snapshot, fail to track the dynamic layer** — frozen-past-self is empirically real.

**A plugin gets only app-level adaptation** (no model-level RLHF/DPO/per-user-LoRA). App-level = **RAG** (retrieve past interactions) + **PAG** (summarize history into a compact profile, inject, re-summarize on update — PURE, SIGIR 2025). **Style/voice = few-shot from the user's own corpus**; content-similar + length-matched selection beats random (arXiv:2509.14543), but fidelity to a private individual's implicit style is limited even at SOTA.

**Drift guards (the crux):**
- **Recency weighting** (ESTABLISHED): `score = α·sim + (1-α)·f_time(Δt)`.
- **Event-driven invalidation** (EMERGING): OpenAI "Dreaming" auto-rewrites "going to X in July" → "went to X" after the date (recall 41.5%→82.8%). State-typed facts (active/completed/expired).
- **Explicit user curation** (ESTABLISHED): ChatGPT/Claude/Gemini all expose view/edit/delete; ChatGPT (Oct 2025) added **roll-back to prior versions**. Claude is notably transparent about *when* memory influences a reply.
- **Contradiction detection** (EMERGING): 4-type typology (MemConflict, arXiv:2605.20926) — temporal-update (supersede), contradictory (keep both timestamped), condition-dependent (scope), distractor (drop). Zep handles structurally via validity intervals.
- **Confidence/evidence thresholds before storing** (EMERGING): don't promote a single observation to the stable profile — require confirmation, ≥N independent observations, or a min confidence score. Separate episodic (session) from semantic (cross-session).

**Sycophancy is the personalization-specific landmine** (ESTABLISHED). RLHF reward models prefer agreement over truth; **providing user memory/profile context measurably INCREASES sycophancy** (arXiv:2509.12517), worse across multi-turn pressure (SYCON Bench). Documented OpenAI insider admission: showing users their own profile assessments triggered backlash, and one response was to make the model *less honest about the user*. Best app-level mitigations: **explicit anti-sycophancy system instruction** ("maintain your position unless given new evidence/logic" — up to 64% improvement on Turn-of-Flip) and **"ask, don't tell"** framing. Users *prefer* sycophancy when offered the choice → must be a built-in default, not a toggle.

### 4. Pitfalls
- **Drift / model collapse** (ESTABLISHED, Shumailov et al., Nature 2024): a profile learning from its *own* AI output narrows and degrades. **Never "replace"; always "accumulate" with provenance.** ACUTE FOR US: a content engine that learns voice from posts *it drafted* collapses the voice toward its own priors. Voice must learn from *actual published, human-edited* output only — the existing `voice-scrubber` already encodes this ("gold standard = approved editions, not the English post corpus").
- **Staleness** (ESTABLISHED, STALE benchmark arXiv:2605.06527): *all* tested models+frameworks (incl. Zep, Mem0) fail to notice a valid memory went obsolete. RankSquire 50K-session prod test: 93.4% at deploy → 58.2% at month 3 (41% stale). Mitigate: timestamp every write, surface at retrieval, RAG live sources for fast-changing facts.
- **Privacy / memory poisoning** (ESTABLISHED, OWASP **ASI06**): MINJA (NeurIPS 2025) — 98.2% injection success from query-only access; ZombieAgent (Jan 2026) — zero-click persistent injection via ChatGPT Connectors. Persistent poisoning is *worse* than ordinary prompt injection (attack/execution temporally decoupled). **Local-first/plain-file shrinks the blast radius** (no remote exfil endpoint; git = audit/rollback).
- **Lock-in** (ESTABLISHED): proprietary embedding stores can't export; plain-text/Markdown + git is the answer ("swap the model, feed it the same Markdown").
- **Frozen past self** (ESTABLISHED): profile over-represents who you were, pulls present output backward. Mitigate: time-weighted retrieval + periodic user-authored "who am I now" anchor that deprecates old inferences.
- **Context rot / over-retrieval** (ESTABLISHED, Chroma report Jul 2025, all 18 models): more retrieved memory → *lower* accuracy; coherent context can hurt more than shuffled. Strict top-k budgets, multi-signal retrieval, pruning.
- **Maintenance graveyard** (ESTABLISHED): 82% abandon second brains within 6 months; ">30 min/week upkeep predicts failure." Retrieval-first, not archive-first. FOR US: the evolution loop must be automated and zero-curation-by-default, curation available but never required.

## PART B — ARCHITECTURE RECOMMENDATION FOR LINKEDIN STUDIO

### B.0 What we already have (the silos)
Five per-user stores under `${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/`, all surviving reinstall, all routed through one tested seam (`hooks/scripts/data-root.mjs` + TS twin `storage.ts:getDataRoot`):
1. **voice-samples/** — authentic samples + chronicle-voice-drift-log (style layer)
2. **trends/trends.json** — deterministic, provenance-bearing, dedup-by-title+URL
3. **specifics-bank/specifics-bank.json** — typed lived-specifics (`number|named-case|what-broke|contrarian|mind-change`), verification-tracked; already self-described as "the first concrete tributary of the broader second-brain architecture"
4. **analytics/** — exports, per-post, weekly/monthly reports, content-history
5. **drafts/, plans/, profile/user-profile.md, examples/, templates/, audience-insights/**

The seam is the most valuable thing we own — the SOTA "route all memory I/O through one resolver" pattern is **already built and tested**. We unify + add an evolution loop, not start from zero.

### B.1 The shape: a thin Markdown hub over typed tributaries (NOT a vector/graph DB)
**Recommendation: plain-Markdown+JSON two-layer "second brain," not Pinecone/Neo4j.** The portable, auditable, drift-resistant, zero-new-infra SOTA is plain-text + git (Manus, AGENTS.md, "markdown files are all you need"), augmented with the retrieval-triad + temporal-validity ideas implemented as deterministic JSON/MD. A heavyweight store would be wrong here: it adds infra, breaks portability, and **Claude IS the semantic-retrieval engine** (it reads files in-context). Keep silos as files; add a hub that points at them + a loop that maintains them.

Two layers, physically separated (episodic/semantic split):
```
${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/
  brain/
    profile.md       # SEMANTIC, two-layer: §static + §dynamic; each fact carries
                     #   value · first_seen · last_seen · evidence_count ·
                     #   provenance(human|published|ai-draft) · status(active|superseded)
    index.md         # the HUB / Map-of-Content: one-screen pointer to every tributary
                     #   + freshness flags. The "memory AND ops center" front page.
    operations.md    # PLANS/IDEAS: goals, bets, parked ideas, the "who I am now"
                     #   restatement anchor (frozen-past-self guard lives here)
    journal/         # EPISODIC, append-only: YYYY-MM-session.md — raw, never edited,
                     #   the source the loop consolidates FROM
  voice-samples/     # tributary (style)          — stays its own store
  specifics-bank/    # tributary (raw material)   — stays its own store
  trends/            # tributary (external signal) — stays its own store
  analytics/         # tributary (performance)    — stays its own store
  ingest/            # NEW: raw real LinkedIn output lands here
    inbox/           #   drop-zone: manual import OR connector output
    published/       #   processed, provenance=published — the GOLD signal
```

**Tributaries vs hub — explicit call:**
- **Stay separate (tributaries), feed the hub via PAG summaries:** voice-samples, specifics-bank, trends, analytics, ingest. Each has a *different dedupe key + lifecycle* (trends: title+URL, fast decay; specifics-bank: human-only, near-permanent; analytics: numeric, immutable). Forcing one schema would destroy those distinctions. The hub holds a distilled pointer-summary of each.
- **Fold INTO the hub:** the flat `profile/user-profile.md` → two-layer `brain/profile.md`. Scattered `audience-insights/`, `examples/` → dynamic-layer sections or tributary-summaries referenced from `index.md`. The **operations/plans center** is genuinely new.

### B.2 The evolution loop (the compounding mechanism)
Run on a cadence (reuse the existing session-start staleness-nudge surface) as a **sleep-time consolidation pass**, not inline cost:
1. **Capture (episodic, auto):** real-signal sessions append to `brain/journal/`. The **ingest seam** (`ingest/inbox/`) takes the user's actual published posts — *manual import first* (LinkedIn has no clean self-serve content API; analytics is already manual CSV per CLAUDE.md S16 — so manual is the honest default; any connector is additive). Published output tagged `provenance=published`.
2. **Consolidate (reflection):** an Opus agent reads new journal + published + tributary deltas, proposes profile updates as a **diff**, never a silent overwrite. Each candidate carries `evidence_count`, `provenance`, timestamps.
3. **Promote with thresholds (anti-overfit):** a candidate enters the **static** layer only on explicit confirmation OR ≥N independent observations; one-offs stay in the **dynamic** layer (which decays). Single weird sessions never reshape identity.
4. **Reconcile contradictions:** classify (temporal → supersede + keep old `status: superseded` w/ date; contradictory → both timestamped; condition-dependent → scope; distractor → drop). Zep's validity-interval idea in plain frontmatter.
5. **Decay + freshness:** every fact has `last_seen`; retrieval weights recent over old; session-start nudge flags facts not refreshed in T days (extend the existing trend-staleness nudge to the whole brain).
6. **Curate (optional, never required):** diff is presentable; user can confirm/edit/delete/roll-back (git = free rollback). Default path needs zero curation — graveyard guard.

### B.3 Keeping it user-aligned + drift-resistant (the hard part, concrete)
- **Provenance-weighted voice learning (THE critical guard):** profile/voice **learn from `provenance=published` only, never `ai-draft`.** Model-collapse guard for our exact risk. Generalizes `voice-scrubber`'s "gold standard = approved editions" to a system-wide invariant: AI-generated content is *never* a training signal for who the user is.
- **Anti-sycophancy as built-in default, not toggle:** consolidation + content agents carry an explicit instruction to treat the profile as evidence to *test*, not flatter ("you mentioned X — still true?"), and surface profile-grounded claims as checkable, not asserted. Users prefer sycophancy when given the choice → don't offer it.
- **Frozen-past-self guard:** `operations.md` holds a periodic user-authored "where I'm headed now" anchor that deprecates older inferences; time-weighted retrieval; the dynamic layer decays so the present dominates.
- **Confidence before commitment:** episodic ≠ semantic; promotion needs evidence density. Highest-leverage anti-overfit mechanism, and cheap (a counter + a threshold).

### B.4 What's genuinely hard (flagged honestly)
1. **Voice fidelity to a private individual is limited even at SOTA** (arXiv:2509.14543) — the real win is grounded *content* (specifics-bank) over mimicked *style*. Manage expectations.
2. **Detecting that a high-relevance fact went stale is UNSOLVED field-wide** (STALE: everyone fails). We can timestamp + nudge; we cannot reliably auto-detect "user changed jobs." Keep humans in that loop.
3. **Contradiction classification** (real change vs context-scoped vs noise) is EMERGING + error-prone; bias toward "keep both, timestamped, surface the conflict" over silent overwrite.
4. **The connector** — no clean LinkedIn self-serve content/analytics API. Build the **manual ingest seam as the contract**; any future connector is a tributary into the same inbox, never a dependency.
5. **Sycophancy is structural** — app-level mitigations reduce, not eliminate; personalization context increases it, so counter-pressure it every time the profile is used.
6. **Avoiding the graveyard is a PRODUCT problem** — the loop must compound value with ~zero required curation or it dies in 6 months like 82% of second brains.

### B.5 Why this fits us
- Reuses the one tested seam (`getDataRoot`) — SOTA single-resolver is done.
- Plain-text + git — portable, auditable, survives reinstall, no new infra; Claude is the semantic-retrieval engine reading files in-context (no vector DB).
- specifics-bank + trends already prove the tributary pattern (deterministic, typed, provenance-bearing, dedup-disciplined) — extend, don't reinvent.
- The four SOTA ideas that port cleanly to files: episodic/semantic split, provenance-weighting, evidence-threshold promotion, temporal-validity frontmatter. The vector/graph machinery does NOT port and we don't need it.

**Bottom line (one line):** Build a thin two-layer Markdown hub (`brain/`: semantic `profile.md` + episodic `journal/` + ops `operations.md` + `index.md` MOC) over the existing typed tributaries, fed by a provenance-tagged ingest seam, maintained by a sleep-time consolidation loop with evidence-threshold promotion, temporal-validity reconciliation, and a built-in anti-sycophancy/anti-collapse stance — every silo stays a tributary except the flat `user-profile.md`, which folds in.

## KEY SOURCES (anchors)
- CoALA arXiv:2309.02427 · Generative Agents arXiv:2304.03442 · MemGPT/Letta arXiv:2310.08560 · Letta sleep-time arXiv:2504.13171
- Mem0 arXiv:2504.19413 · Zep/Graphiti arXiv:2501.13956 · A-MEM (NeurIPS 2025) arXiv:2502.12110 · Cognee arXiv:2505.24478
- PersonaMem (COLM 2025) arXiv:2504.14225 · PURE (SIGIR 2025) · style-imitation arXiv:2509.14543 · MemConflict arXiv:2605.20926
- interaction-context-sycophancy arXiv:2509.12517 · OpenAI Dreaming (openai.com/index/chatgpt-memory-dreaming) · Anthropic sycophancy research
- Shumailov model collapse (Nature 2024, s41586-024-07566-y) · STALE benchmark arXiv:2605.06527 · MINJA (NeurIPS 2025) arXiv:2503.03704 · OWASP ASI06 · Chroma Context Rot (trychroma.com/research/context-rot) · Lost-in-the-Middle (Liu et al. 2023)
- PKM: buildingasecondbrain.com · zettelkasten.de · notes.andymatuschak.org · Mem/Pinecone engineering · Tana · Obsidian Smart Connections
- Portability: AGENTS.md (Linux Foundation) · "markdown files are all you need" (voxos.ai, dev.to)

> NOTE on sources: a few arXiv ids above carry 2025–2026 future-dated identifiers as returned by the research thread. Treat the *patterns* as the durable takeaway; re-verify any specific citation before quoting it externally (verification duty).