Kjell Tore Guttormsen
8c927198f5
SB-S0 (Foundation) of the second-brain arc: a new TS package scripts/brain/
(structural copy of specifics-bank) establishing the spine later slices hang on.
TDD throughout (failing test first); fold scope = P1+P2 only per operator decision.
- types.ts: one provenance vocab (human|published|ai-draft) + the six-field
ProfileFact record + two-layer ProfileDoc.
- id.ts: slugify + mintEntityId (sha256[:12], kind-namespaced, slug-keyed so the
id is stable across value edits) + normalizeProvenance (throws on unknown). [SC4]
- profile.ts: no-YAML line-grammar parse/serialize (parse∘serialize = identity over
the whole doc; values may contain ]/|/quotes) [SC2] + foldUserProfile: lossless,
idempotent, source-absent-aware fold of config/user-profile.template.md. Pinned
extraction — P1 labeled scalars (group-headers skipped, [placeholder]→empty) + P2
expertise group; stops at "### Research Tooling" so deferred explainer prose can't
leak in as fields. Checkbox-prefs (Goals/Tone/MCPs/Assets) deferred (§8). [SC3]
- dataRoot.ts: inline per-package resolver (repo idiom; no new seam → SC6).
- scaffold.ts/cli.ts: idempotent `brain init` — brain/{index,profile,operations}.md
+ journal/ + ingest/{inbox,published} under the data-dir; compare-then-skip, never
clobbers a user edit. No session-start wiring (SB-S2 owns it). [SC1]
Gate: new BRAIN floor in test-runner.sh (34 tests; trends/specifics/contract floors
unchanged) + anti-erosion floor 74→75. Full gate 90/0/0, tsc --noEmit clean. [SC5/SC6]
No new command/agent/reference → no version bump, no structure-count change.
Refs docs/second-brain/{brief,plan-sb-s0,architecture}.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RigJBiRFNtFZKCz21qNbQ4
102 lines
12 KiB
Markdown
102 lines
12 KiB
Markdown
# Second Brain — Architecture Design
|
||
|
||
> **Status:** architecture **approved by operator 2026-06-23**. **SB-S0 (Foundation) landed 2026-06-23** (`scripts/brain/`, 34 tests, gate-wired); S1–S4 remain design-phase.
|
||
> **Boundary (confirmed 2026-06-23):** the **engine** (store schema · evolution loop · ingest seam) → **the plugin** (domain-general, shareable); the **user's data** (posts · articles · newsletters · plans · ideas) → the **per-user data dir** (`${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/`, survives reinstall); the **personal cockpit** (the operator's day-to-day operations centre) → **Maskinrommet** (a thin layer that reads/writes *through* the plugin's store, never a fork of the engine).
|
||
> **Research inputs (three parallel threads, 2026-06-23):** `research/connector-egress.md` · `research/secondbrain-sota.md` · `research/silo-inventory.md`.
|
||
|
||
## The problem
|
||
|
||
A "second brain" stores everything about one creator — posts, articles, newsletters, data, plans, ideas — and compounds it into an ever-improving, user-aligned profile. It is **memory AND an operations centre**.
|
||
|
||
The three research threads reframed the task in a decisive way:
|
||
|
||
1. **There is nothing to migrate.** M0 already routed all **12 existing per-user silos** through one tested seam (`hooks/scripts/data-root.mjs` `getDataRoot()` + its TS twin `scripts/analytics/src/utils/storage.ts`), default `${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/`. They all survive reinstall today. The task is **unify**, not move.
|
||
2. **Not a vector/graph DB.** For *our* case the portable, auditable, drift-resistant 2026 standard is **plain-text + JSON + git** — because **Claude *is* the semantic-retrieval engine** (it reads files in-context). A heavyweight store adds infra, breaks portability, and invites lock-in for no gain.
|
||
3. **Auto-ingest exists — for an EU/EEA member.** The DMA Member Data Portability API is self-serve in EU/EEA + Switzerland and gives automated pull of **content** (post text + articles), but **not** received-analytics. Analytics stays the existing manual CSV path. Scraping is a ToS breach with active enforcement → never baked in. So ingest is **manual-first as the contract**; any connector is an additive tributary.
|
||
|
||
The real gap (from the silo inventory): the 12 silos are **per-user but siloed and heterogeneous** — three storage idioms, two roots, no cross-references, provenance reinvented under different field names in each. A published post lands in three non-referencing places with no shared id. The question *"which raw material actually performs?"* (specific → post → measured analytics) is **unanswerable today** — and it is exactly what a second brain should answer.
|
||
|
||
## The shape: a thin Markdown hub over typed tributaries
|
||
|
||
```
|
||
${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/
|
||
brain/
|
||
index.md # MOC / hub — one-screen pointer to every tributary + freshness flags
|
||
# (the "memory AND ops centre" front page)
|
||
profile.md # SEMANTIC, two-layer: §static (stable) + §dynamic (evolving).
|
||
# Each fact: value · first_seen · last_seen · evidence_count ·
|
||
# provenance(human|published|ai-draft) · status(active|superseded)
|
||
operations.md # PLANS / IDEAS + the "who I am now" anchor (frozen-past-self guard)
|
||
journal/ # EPISODIC, append-only: YYYY-MM-session.md — raw, never edited;
|
||
# the source the consolidation loop distils FROM
|
||
ingest/
|
||
inbox/ # NEW drop-zone: manual import OR (EU/EEA) connector output
|
||
published/ # processed, provenance=published — the GOLD signal
|
||
voice-samples/ # tributary (style) — stays its own store
|
||
specifics-bank/ # tributary (raw material) — stays its own store
|
||
trends/ # tributary (external signal) — stays its own store
|
||
analytics/ # tributary (performance) — stays its own store
|
||
```
|
||
|
||
### Tributaries vs hub — the explicit call
|
||
|
||
- **Stay separate (tributaries), feed the hub via distilled pointer-summaries:** voice-samples, specifics-bank, trends, analytics, ingest. Each has a *different dedupe key + lifecycle* (trends: title+url, fast decay · specifics-bank: human-only, near-permanent · analytics: numeric, immutable). Forcing one schema would destroy those distinctions — the silo inventory warns of this explicitly. The hub holds a distilled summary of each + a pointer.
|
||
- **Fold INTO the hub:** the flat `profile/user-profile.md` → two-layer `brain/profile.md`. Scattered `audience-insights/` / `examples/` become dynamic-layer sections or tributary-summaries referenced from `index.md`. The **operations/plans centre is genuinely new**.
|
||
|
||
### Alternatives considered (and rejected)
|
||
|
||
- **Vector/graph DB (Pinecone/Neo4j/Zep-style):** rejected — adds infra, breaks plain-text portability + git audit/rollback, invites lock-in, and duplicates retrieval that Claude already does in-context. The four SOTA ideas that port cleanly to files (episodic/semantic split, provenance-weighting, evidence-threshold promotion, temporal-validity frontmatter) give most of the value; the vector/graph machinery does not port and is not needed.
|
||
- **One unified schema across all silos:** rejected — destroys each silo's distinct dedupe key + lifecycle. Keep tributaries; add a thin hub + a cross-silo id.
|
||
|
||
## Invariants (the spine)
|
||
|
||
- **Provenance-weighted learning (the critical guard):** profile/voice learn from `provenance=published` (human-edited) **only**, never `ai-draft`. This is the model-collapse guard for our exact risk — a content engine that learns voice from what it itself drafted collapses toward its own priors. Generalises `voice-scrubber`'s existing rule ("gold standard = approved editions, not the English post corpus") to a system-wide invariant.
|
||
- **One canonical entity id + one provenance shape**, threaded through the silos → makes the post ↔ specific ↔ trend ↔ analytics graph assemblable. Fixes the inventory's hardest problem.
|
||
- **Episodic / semantic physically separated** (raw events in `journal/`, distilled facts in `profile.md`).
|
||
- **Evidence-threshold promotion (anti-overfit):** the dynamic layer decays; a fact enters the **static** layer only on explicit confirmation OR ≥N independent observations. Single weird sessions never reshape identity.
|
||
- **Temporal-validity + freshness nudge:** every fact carries `last_seen`; retrieval weights recent over old; the session-start nudge flags facts not refreshed in T days (extend the existing trend-staleness nudge brain-wide).
|
||
- **Anti-sycophancy as a built-in default, not a toggle:** consolidation + content agents treat the profile as evidence to *test*, not flatter. (Personalization context measurably *increases* sycophancy, so counter-pressure it every time the profile is used.)
|
||
- **Frozen-past-self guard:** `operations.md` holds a periodic user-authored "where I'm headed now" anchor that deprecates older inferences.
|
||
- **Zero required curation (graveyard guard):** the loop compounds value with ~zero mandatory upkeep; curation is available but never required. git = free rollback.
|
||
|
||
## The evolution loop (the compounding mechanism)
|
||
|
||
Run on a cadence (reuse the existing session-start staleness-nudge surface) as a **sleep-time consolidation pass**, not inline cost:
|
||
|
||
1. **Capture (episodic, auto):** real-signal sessions append to `brain/journal/`. The ingest seam (`ingest/inbox/`) takes the user's actual published posts — manual import first; published output tagged `provenance=published`.
|
||
2. **Consolidate (reflection):** an Opus agent reads new journal + published + tributary deltas, proposes profile updates as a **diff**, never a silent overwrite. Each candidate carries `evidence_count`, `provenance`, timestamps.
|
||
3. **Promote with thresholds:** static-layer entry needs confirmation or ≥N observations; one-offs stay in the decaying dynamic layer.
|
||
4. **Reconcile contradictions:** classify temporal-update (supersede, keep old `status: superseded` + date) · contradictory (keep both, timestamped) · condition-dependent (scope) · distractor (drop). Bias toward "keep both + surface the conflict" over silent overwrite.
|
||
5. **Decay + freshness:** weight recent over old; flag stale facts at session-start.
|
||
6. **Curate (optional, never required):** the diff is presentable; confirm/edit/delete/roll-back via git.
|
||
|
||
## Build sequence (full ambition, incremental — one slice per session)
|
||
|
||
| Slice | Content | Why this order |
|
||
|-------|---------|----------------|
|
||
| **SB-S0 — Foundation** ✅ *landed 2026-06-23* | `brain/` scaffold + two-layer `profile.md` (fold in `user-profile.md`) + `index.md` MOC + `operations.md` + `journal/`; entity-id + provenance shape as a small typed, tested module; `ingest/` dirs + manual-import contract. **No loop yet.** Shipped as `scripts/brain/` (TS, 34 tests, gate-wired BRAIN floor); fold = P1 labeled-scalars + P2 expertise (checkbox-prefs deferred, §8 of `plan-sb-s0.md`). | Smallest thing that stands up and is testable; locks the id/provenance spine everything hangs on |
|
||
| **SB-S1 — Ingest + gold signal** | Manual import → `ingest/published/` with `provenance=published`; wire profile/voice to learn from published-only | The gold signal before the loop that consumes it |
|
||
| **SB-S2 — Evolution loop** | Sleep-time consolidation (reuse session-start surface): journal+published+tributary deltas → profile **diff** w/ evidence_count/provenance/timestamps; threshold promotion; contradiction reconciliation; brain-wide freshness nudge | The compounding mechanism |
|
||
| **SB-S3 — Cross-silo graph + ops centre** | Thread the id through tributaries (post↔specific↔trend↔analytics assemblable); flesh out `operations.md`; retire the dead `content-history.md` + triple-post reconciliation | Finally answers "which raw material performs?" |
|
||
| *(later / optional)* **SB-S4** | EU/EEA DMA portability API as an auto-tributary into `ingest/inbox/` | Additive; never a dependency |
|
||
|
||
## What's genuinely hard (honest flags)
|
||
|
||
1. **Voice fidelity to a private individual is limited even at SOTA** — the real win is grounded *content* (specifics-bank) over mimicked *style*. Manage expectations.
|
||
2. **Detecting that a high-relevance fact went stale is UNSOLVED field-wide** (STALE benchmark: all tested models/frameworks fail). We timestamp + nudge; we cannot auto-detect "changed jobs." Keep a human in that loop.
|
||
3. **Contradiction classification** (real change vs context-scoped vs noise) is emerging + error-prone → bias to keep-both-timestamped.
|
||
4. **The connector** — no clean LinkedIn self-serve content API outside EU/EEA. Build the manual ingest seam as the contract; any connector is a tributary, never a dependency. Newsletter-edition coverage by the portability `ARTICLES` domain is **not fully verified** — check against a real export.
|
||
5. **Sycophancy is structural** — app-level mitigations reduce, not eliminate.
|
||
6. **Avoiding the graveyard is a product problem** — the loop must compound value with ~zero required curation or it dies in 6 months like 82% of second brains.
|
||
|
||
## Verification (SB-S0, when we build it)
|
||
|
||
- Set `LINKEDIN_STUDIO_DATA` to a temp dir, run init → assert `brain/{index,profile,operations}.md` + `journal/` + `ingest/{inbox,published}` exist via the `getDataRoot` seam.
|
||
- `profile.md` parses two-layer (§static / §dynamic); `user-profile.md` fields fold in without loss (diff check).
|
||
- entity-id module: deterministic id mint + provenance shape, unit-tested.
|
||
- **No regression:** gate `scripts/test-runner.sh` 89/0/0 green; trends 24/24; specifics 28/28; contract 33/33.
|
||
- **Key assumption to test early:** "Claude-as-retrieval-engine over plain files is sufficient (no vector DB)" — testable with a retrieval scenario once the brain holds content. Marked as assumption until proven.
|
||
|
||
## Bottom line
|
||
|
||
A thin two-layer Markdown hub (`brain/`: semantic `profile.md` + episodic `journal/` + ops `operations.md` + `index.md` MOC) over the existing typed tributaries, fed by a provenance-tagged ingest seam, maintained by a sleep-time consolidation loop with evidence-threshold promotion, temporal-validity reconciliation, and a built-in anti-sycophancy / anti-collapse stance. Every silo stays a tributary except the flat `user-profile.md`, which folds in.
|