From edd3e15ef7152a7f8f90e0a70ad8064c04670560 Mon Sep 17 00:00:00 2001 From: Kjell Tore Guttormsen Date: Tue, 23 Jun 2026 20:47:34 +0200 Subject: [PATCH] =?UTF-8?q?feat(linkedin-studio):=20SB-S3c=20=E2=80=94=20c?= =?UTF-8?q?ross-silo=20id-threading=20+=20post=E2=86=92analytics=20assembl?= =?UTF-8?q?er=20[skip-docs]?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hub-side design: the published record now carries the specifics/trends ids it was built from (additive, omit-empty → byte-backward-compatible), and a new pure assembler (scripts/brain/src/assemble.ts + `brain assemble`) joins post↔analytics by normalized title-prefix + date with honest confidence tiers (high/low/none). Answers the arc's north-star query: which raw material actually performs? (specific → post → measured analytics). All four tributaries untouched (analytics READ-only via inlined raw-JSON, no package import); profile.md grammar untouched (the fact→post link stays OUT — C-1). The repeatable --specific/--trend ingest flags collect via a new collectRepeated helper, leaving parseFlags untouched. TDD: 19 new brain tests (ingest 4 + publish 3 + assemble 8 + cli 4), all SC1–SC12. brain 113/113, gate 95/0/0, BRAIN_TESTS_FLOOR 94→113, ASSERT_BASELINE_FLOOR unchanged at 80. Light-Voyage hardened (brief-review 5 FIX · plan-critic 1 BLOCK+4 MAJOR+4 MINOR · scope-guardian ALIGNED). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01RigJBiRFNtFZKCz21qNbQ4 --- docs/second-brain/architecture.md | 2 +- docs/second-brain/brief-sb-s3c.md | 2 +- docs/second-brain/plan-sb-s3c.md | 2 +- scripts/brain/src/assemble.ts | 156 +++++++++++++++++++++++++++ scripts/brain/src/cli.ts | 80 +++++++++++++- scripts/brain/src/id.ts | 6 +- scripts/brain/src/ingest.ts | 56 +++++++++- scripts/brain/tests/assemble.test.ts | 95 ++++++++++++++++ scripts/brain/tests/cli.test.ts | 56 +++++++++- scripts/brain/tests/ingest.test.ts | 40 +++++++ scripts/brain/tests/publish.test.ts | 32 +++++- scripts/test-runner.sh | 2 +- 12 files changed, 515 insertions(+), 14 deletions(-) create mode 100644 scripts/brain/src/assemble.ts create mode 100644 scripts/brain/tests/assemble.test.ts diff --git a/docs/second-brain/architecture.md b/docs/second-brain/architecture.md index 8946ac3..1f3b2a0 100644 --- a/docs/second-brain/architecture.md +++ b/docs/second-brain/architecture.md @@ -77,7 +77,7 @@ Run on a cadence (reuse the existing session-start staleness-nudge surface) as a | **SB-S0 — Foundation** ✅ *landed 2026-06-23* | `brain/` scaffold + two-layer `profile.md` (fold in `user-profile.md`) + `index.md` MOC + `operations.md` + `journal/`; entity-id + provenance shape as a small typed, tested module; `ingest/` dirs + manual-import contract. **No loop yet.** Shipped as `scripts/brain/` (TS, 34 tests, gate-wired BRAIN floor); fold = P1 labeled-scalars + P2 expertise (checkbox-prefs deferred, §8 of `plan-sb-s0.md`). | Smallest thing that stands up and is testable; locks the id/provenance spine everything hangs on | | **SB-S1 — Ingest + gold signal** ✅ *landed 2026-06-23* | Manual import → `ingest/published/` with `provenance=published` (CLI `brain ingest`); voice-trainer wired to learn from published-only, never `ai-draft`, gate-enforced. Shipped as `scripts/brain/src/ingest.ts` + `docs/second-brain/ingest-manual-import.md` (v0.5.1). **No profile.md mutation** (SB-S2). | The gold signal before the loop that consumes it | | **SB-S2 — Evolution loop** ✅ *landed 2026-06-23* | Operator-invoked, operator-gated consolidation: `brain consolidate` (`--gather`/`--propose`/`--apply --confirm`) → profile **diff** w/ evidence_count/provenance/timestamps; threshold-promotion (N=3); contradiction → keep-both with distinct ids (no supersede); decay-flag (90d); `consolidation-state.json` sidecar; zero-dep session-start consolidation-due nudge + scaffold-ensure. Shipped as `scripts/brain/src/consolidate.ts` + `docs/second-brain/consolidation-loop.md` (v0.5.2). **Operator decisions:** journal deferred · no new agent (session extracts) · motor-only (no reader until S3). | The compounding mechanism | -| **SB-S3 — Cross-silo graph + ops centre** | Thread the id through tributaries (post↔specific↔trend↔analytics assemblable); flesh out `operations.md`; retire the dead `content-history.md` + triple-post reconciliation | Finally answers "which raw material performs?" | +| **SB-S3 — Cross-silo graph + ops centre** | Thread the id through tributaries (post↔specific↔trend↔analytics assemblable); flesh out `operations.md`; retire the dead `content-history.md` + triple-post reconciliation. **S3a ✅ first reader · S3b ✅ supersede · S3c ✅ cross-silo id-threading** (hub-side: the published record carries the `specifics`/`trends` ids it was built from + a pure analytics resolver — `scripts/brain/src/assemble.ts`, `brain assemble`; tributaries untouched). **S3d remains** (ops centre / `content-history.md` retirement / triple-post reconciliation). | Finally answers "which raw material performs?" | | *(later / optional)* **SB-S4** | EU/EEA DMA portability API as an auto-tributary into `ingest/inbox/` | Additive; never a dependency | ## What's genuinely hard (honest flags) diff --git a/docs/second-brain/brief-sb-s3c.md b/docs/second-brain/brief-sb-s3c.md index 71ffba9..4552d32 100644 --- a/docs/second-brain/brief-sb-s3c.md +++ b/docs/second-brain/brief-sb-s3c.md @@ -1,6 +1,6 @@ # Brief — SB-S3c: cross-silo id-threading (the "which raw material performs?" graph) -> **Slice:** SB-S3c (third sub-slice of SB-S3). **Status:** DRAFT — awaiting operator "go" before any code. Light-Voyage **complete**: brief-review **APPROVE-WITH-FIXES** (5 FIX folded §9), plan-critic **REVISE → 1 BLOCK + 4 MAJOR + 4 MINOR folded** (§10), scope-guardian **ALIGNED** (0 creep/0 gap). +> **Slice:** SB-S3c (third sub-slice of SB-S3). **Status:** **LANDED** — operator go 2026-06-23; TDD complete, brain suite 113/113, gate 95/0/0 (`BRAIN_TESTS_FLOOR` 94→113, `ASSERT_BASELINE_FLOOR` unchanged at 80). Light-Voyage: brief-review **APPROVE-WITH-FIXES** (5 FIX folded §9), plan-critic **REVISE → 1 BLOCK + 4 MAJOR + 4 MINOR folded** (§10), scope-guardian **ALIGNED** (0 creep/0 gap). > **Predecessors:** SB-S0 (id/provenance substrate — `mintEntityId`/`mintContentId`, `id.ts:6-7` reserved "SB-S3 will thread this id through the tributaries") · SB-S1 (published-gold ingest — the post anchor `ingest/published/.md`) · SB-S2 (consolidation motor) · SB-S3a (first reader `strategy-advisor` — LANDED `4fa411f`) · **SB-S3b (supersede arm — LANDED `585f972`, gate 95/0/0, BRAIN floor 94, ASSERT floor 80).** > **Grounded in:** a read-only code map of all four tributaries + `scripts/brain/` (file:line cited throughout) — Explore sweep + firsthand reads of `id.ts`, `consolidate.ts`, `types.ts`, `profile.ts`, `ingest.ts`. diff --git a/docs/second-brain/plan-sb-s3c.md b/docs/second-brain/plan-sb-s3c.md index 7877016..f30bd1e 100644 --- a/docs/second-brain/plan-sb-s3c.md +++ b/docs/second-brain/plan-sb-s3c.md @@ -1,7 +1,7 @@ # Plan — SB-S3c: cross-silo id-threading > Brief: `docs/second-brain/brief-sb-s3c.md`. Slice: SB-S3c. -> **Design (recommended, pending operator go at the gate):** hub-side threading — `specifics`/`trends` ids onto the brain's published record + a pure analytics resolver. Tributaries untouched. (Fork-1; operator may redirect to mutate-tributaries.) +> **Status:** **LANDED** (operator go 2026-06-23) — brain 113/113, gate 95/0/0, `BRAIN_TESTS_FLOOR` 94→113. **Design:** hub-side threading — `specifics`/`trends` ids onto the brain's published record + a pure analytics resolver. Tributaries untouched. > **Light-Voyage:** brief-review APPROVE-WITH-FIXES (5 FIX folded into the brief; 3 RISK carried here). plan-critic **REVISE → 1 BLOCK + 4 MAJOR + 4 MINOR folded** (Step 1 compile-fixup · Step 2.2 non-throwing parse reader · Step 2.3 producer guard · Step 3.3 floor rationale · Step 3.4 deterministic row-sort · Step 3.5 analytics-root caveat · Step 4.2 `runIngest` signature+call-site · Step 4.3 full-record load + desc sort · R-G sub-count). scope-guardian **ALIGNED** (0 creep / 0 gap). > Order is TDD: failing brain-suite tests land BEFORE the code (iron law); each rule pinned before the CLI is wired. diff --git a/scripts/brain/src/assemble.ts b/scripts/brain/src/assemble.ts new file mode 100644 index 0000000..05cd781 --- /dev/null +++ b/scripts/brain/src/assemble.ts @@ -0,0 +1,156 @@ +/** + * SB-S3c — the cross-silo assembler (the payoff). + * + * Answers the arc's north-star query (`architecture.md:17`): *which raw material + * actually performs?* — `specific → post → measured analytics`. The post record + * (`ingest/published/.md`) already carries the `specifics`/`trends` ids + * it was built from (SB-S3c hub-side threading); this module joins each post to + * its measured analytics row and surfaces the whole graph. + * + * PURE core: `assemblePostGraph({records, analytics})` takes already-loaded inputs + * and returns the graph — no FS/clock/network. The analytics↔post join is an honest + * HEURISTIC, never a guaranteed key: analytics carries no body and no URN (only a + * title-prefix + date), so `matchRow` joins by normalized title-prefix + date with + * explicit confidence tiers (`high`/`low`/`none`) — a real-CSV `none` is a + * normalization-tightening signal, not a proof of no match. + * + * DECOUPLED: this module treats tributary ids as opaque 12-hex strings and takes a + * minimal `AnalyticsRowInput` shape — it never imports the analytics/trends/ + * specifics packages. The thin read-only `loadAnalyticsRows` IO inlines a raw-JSON + * read of the shared data-root (NOT the analytics package's `loadAllPosts`). + */ + +import { existsSync, readFileSync, readdirSync } from "node:fs"; +import { join } from "node:path"; + +import { dataRoot } from "./dataRoot.js"; +import type { PublishedRecord } from "./ingest.js"; + +/** + * The minimal analytics-row shape the resolver needs, extracted from the raw + * `AnalyticsBatch.posts[]` JSON (`analytics/src/models/types.ts`). Note the field + * is `publishedDate` (analytics) vs `published_date` (the brain record). + */ +export interface AnalyticsRowInput { + title: string; + publishedDate: string; // YYYY-MM-DD + metrics?: { engagementRate?: number } & Record; +} + +export type MatchConfidence = "high" | "low" | "none"; + +/** A post's matched analytics: the WHOLE row reference (FIX 4), or none. */ +export interface PostMatch { + confidence: MatchConfidence; + row?: AnalyticsRowInput; +} + +export interface PostGraphNode { + contentId: string; + published_date: string; + specifics: string[]; + trends: string[]; + match: PostMatch; +} + +/** + * Minimum normalized-title length to attempt a prefix match. The hook quality-rule + * floor is 110 chars; 24 normalized chars (~3–5 words) is the shortest opener + * specific enough that a prefix-match is not coincidental, while staying well under + * any real hook. Below floor → `none` (an operator can still eyeball). + */ +const PREFIX_FLOOR = 24; + +/** Brain-local copy of the specifics-bank `normalizeContent` idiom (NOT imported). */ +export function normalize(s: string): string { + return s.trim().toLowerCase().replace(/\s+/g, " "); +} + +/** Strip a trailing LinkedIn truncation marker (`…`/`...`) so a `…`-suffixed export title still prefix-matches. */ +function stripTrailingEllipsis(s: string): string { + return s.replace(/(?:…|\.{3})\s*$/, "").trimEnd(); +} + +/** + * Match one analytics row to one published record. Returns the tiered match, or + * `null` when the row does not qualify (no prefix / below floor) — STUB until S3c + * Step 3. + */ +export function matchRow(record: PublishedRecord, row: AnalyticsRowInput): PostMatch | null { + const nt = stripTrailingEllipsis(normalize(row.title)); + if (nt.length < PREFIX_FLOOR) return null; // too short to discriminate → none + if (!normalize(record.body).startsWith(nt)) return null; // no prefix → none + const confidence: MatchConfidence = record.published_date === row.publishedDate ? "high" : "low"; + return { confidence, row }; +} + +/** + * Assemble the post → raw-material → performance graph. Pure (no FS/clock/network). + * For each record, the BEST qualifying analytics row: `high` (same date) beats `low` + * (different date); within a tier, the longest matched title wins. The analytics rows + * are sorted once (publishedDate desc, title asc) so an exact-length tie is stable — + * never `readdirSync`-order-dependent. + */ +export function assemblePostGraph(args: { + records: PublishedRecord[]; + analytics: AnalyticsRowInput[]; +}): PostGraphNode[] { + const analytics = [...args.analytics].sort( + (a, b) => b.publishedDate.localeCompare(a.publishedDate) || a.title.localeCompare(b.title), + ); + return args.records.map((record) => { + let best: PostMatch | null = null; + let bestLen = -1; + for (const row of analytics) { + const m = matchRow(record, row); + if (!m) continue; + const len = stripTrailingEllipsis(normalize(row.title)).length; + const better = + best === null || + (m.confidence === "high" && best.confidence === "low") || + (m.confidence === best.confidence && len > bestLen); + if (better) { + best = m; + bestLen = len; + } + } + return { + contentId: record.id, + published_date: record.published_date, + specifics: record.specifics, + trends: record.trends, + match: best ?? { confidence: "none" }, + }; + }); +} + +/** + * Read-only loader: inline a raw-JSON read of the analytics batches under the shared + * data-root and extract the minimal row shape. STUB until S3c Step 3. + * + * NOTE (root-skew caveat): resolves via the brain `dataRoot` (`${LINKEDIN_STUDIO_DATA}/ + * analytics/posts`); the analytics package additionally honours the deprecated + * `ANALYTICS_ROOT` override, which this path does NOT — if set to a non-default + * path, the join degrades to every-post-`none` (accepted cost of the no-import + * decoupling; the M0 default leaves `ANALYTICS_ROOT` unset). + */ +export function loadAnalyticsRows(): AnalyticsRowInput[] { + const dir = dataRoot(join("analytics", "posts")); + if (!existsSync(dir)) return []; // fresh-clone / no imports yet → no rows + const rows: AnalyticsRowInput[] = []; + for (const name of readdirSync(dir)) { + if (!name.endsWith(".json") || name.startsWith(".")) continue; + try { + const batch = JSON.parse(readFileSync(join(dir, name), "utf8")) as { posts?: unknown[] }; + for (const p of batch?.posts ?? []) { + const row = p as Partial; + if (typeof row?.title === "string" && typeof row?.publishedDate === "string") { + rows.push({ title: row.title, publishedDate: row.publishedDate, metrics: row.metrics }); + } + } + } catch { + // skip a malformed/unreadable batch file — never crash (mirrors listPublished) + } + } + return rows; +} diff --git a/scripts/brain/src/cli.ts b/scripts/brain/src/cli.ts index e9b998c..9901331 100644 --- a/scripts/brain/src/cli.ts +++ b/scripts/brain/src/cli.ts @@ -25,6 +25,7 @@ import { type Candidate, type ProfileDiff, } from "./consolidate.js"; +import { assemblePostGraph, loadAnalyticsRows } from "./assemble.js"; import { dataRoot } from "./dataRoot.js"; import { ingestText, listPublished, parsePublishedRecord, scanInbox } from "./ingest.js"; import { parseProfile, serializeProfile } from "./profile.js"; @@ -59,6 +60,25 @@ function parseFlags(args: string[]): Record { return out; } +/** + * SB-S3c: collect EVERY value of a repeatable `--key ` flag into an array + * (the single-value `parseFlags` keeps only the last). Scans the raw args directly, + * leaving `parseFlags` untouched — so single-value flag behaviour is unchanged. + */ +function collectRepeated(args: string[], key: string): string[] { + const out: string[] = []; + for (let i = 0; i < args.length; i++) { + if (args[i] === `--${key}`) { + const next = args[i + 1]; + if (next !== undefined && !next.startsWith("--")) { + out.push(next); + i++; + } + } + } + return out; +} + function today(): string { return new Date().toISOString().slice(0, 10); } @@ -68,9 +88,10 @@ function usage(msg: string): never { console.error( "usage:\n" + " init\n" + - " ingest --file [--source ] [--date ]\n" + + " ingest --file [--source ] [--date ] [--specific ]… [--trend ]…\n" + " ingest --scan-inbox [--source ]\n" + - " published list [--json]", + " published list [--json]\n" + + " assemble", ); process.exit(2); } @@ -89,7 +110,7 @@ function runInit(): void { if (created.length === 0) console.log("Already initialised — nothing to do."); } -function runIngest(flags: Record): void { +function runIngest(rest: string[], flags: Record): void { if (flags["scan-inbox"] === "true") { const res = scanInbox({ captured_at: today(), source: flags.source }); console.log( @@ -100,11 +121,14 @@ function runIngest(flags: Record): void { const file = flags.file; if (!file || file === "true") usage("ingest needs --file or --scan-inbox"); const body = readFileSync(file, "utf8"); + // SB-S3c: --specific / --trend are repeatable — read ONLY via collectRepeated. const res = ingestText({ body, captured_at: today(), source: flags.source, published_date: flags.date === "true" ? undefined : flags.date, + specifics: collectRepeated(rest, "specific"), + trends: collectRepeated(rest, "trend"), }); if (res.written && res.collision) { console.log(`Collision (different body, same id) → wrote ${res.path}`); @@ -129,6 +153,53 @@ function runPublished(rest: string[], flags: Record): void { } } +/** + * SB-S3c: read-only cross-silo assembler view — post → raw-material → performance. + * Loads published records + analytics rows (inlined raw-JSON, no analytics import), + * prints the join newest-first. Writes NOTHING. + */ +function runAssemble(_flags: Record): void { + const pubDir = dataRoot(join("ingest", "published")); + const records = existsSync(pubDir) + ? readdirSync(pubDir) + .filter((f) => f.endsWith(".md") && !f.startsWith(".")) + .map((f) => { + try { + return parsePublishedRecord(readFileSync(join(pubDir, f), "utf8")); + } catch { + return null; + } + }) + .filter((r): r is NonNullable => r !== null) + : []; + const analytics = loadAnalyticsRows(); + const bodyById = new Map(records.map((r) => [r.id, r.body])); + const graph = assemblePostGraph({ records, analytics }).sort((a, b) => + b.published_date.localeCompare(a.published_date), + ); + if (graph.length === 0) { + console.log( + "No published records to assemble. Ingest posts with `brain ingest --file

[--specific ] [--trend ]`.", + ); + return; + } + console.log(`Post graph — ${graph.length} record(s); ${analytics.length} analytics row(s):`); + for (const node of graph) { + const firstLine = (bodyById.get(node.contentId) ?? "").split("\n", 1)[0]; + console.log(`\n· ${node.contentId} · ${node.published_date} · ${firstLine}`); + console.log(` specifics: ${node.specifics.length ? node.specifics.join(", ") : "—"}`); + console.log(` trends: ${node.trends.length ? node.trends.join(", ") : "—"}`); + if (node.match.confidence === "none") { + console.log(" analytics: none (no title-prefix+date match)"); + } else { + const eng = node.match.row?.metrics?.engagementRate; + console.log( + ` analytics: ${node.match.confidence}${eng !== undefined ? ` [eng ${eng}%]` : ""}`, + ); + } + } +} + function renderDiffMd(diff: ProfileDiff): string { const lines = ["# Pending profile diff", "", "> Operator-gated. Review, then `brain consolidate --apply --diff brain/pending-diff.json --confirm`.", ""]; const section = (title: string, items: string[]) => { @@ -235,9 +306,10 @@ function main(): void { const flags = parseFlags(rest); if (command === "init") return runInit(); - if (command === "ingest") return runIngest(flags); + if (command === "ingest") return runIngest(rest, flags); if (command === "published") return runPublished(rest, flags); if (command === "consolidate") return runConsolidate(flags); + if (command === "assemble") return runAssemble(flags); usage(command ? `unknown command: ${command}` : "no command given"); } diff --git a/scripts/brain/src/id.ts b/scripts/brain/src/id.ts index c575cd9..3cbb4ce 100644 --- a/scripts/brain/src/id.ts +++ b/scripts/brain/src/id.ts @@ -3,8 +3,10 @@ * * Pure + deterministic — no filesystem, no clock, no network. The id is keyed on * a STABLE SLUG of the label, not the raw label, so editing a fact's *value* (or - * the label's case/whitespace) never re-mints the id. SB-S3 will thread this id - * through the tributaries; SB-S0 only establishes mint + shape. + * the label's case/whitespace) never re-mints the id. SB-S3c threads this id + * through the tributaries (the published record carries the `specifics`/`trends` + * ids it was built from; analytics is joined by resolver — see `assemble.ts`); + * SB-S0 established mint + shape. */ import { createHash } from "node:crypto"; diff --git a/scripts/brain/src/ingest.ts b/scripts/brain/src/ingest.ts index a59fcce..2b31757 100644 --- a/scripts/brain/src/ingest.ts +++ b/scripts/brain/src/ingest.ts @@ -35,6 +35,14 @@ export interface PublishedRecord { captured_at: string; /** `manual` (default) | a future connector token. */ source: string; + /** + * SB-S3c — the raw material this post was built from: specifics-bank + trends + * ids (12-hex), threaded hub-side so the post↔specific↔trend↔analytics graph is + * assemblable. Default `[]`; serialized only when non-empty (omit-empty keeps a + * pre-S3c record byte-identical). Validated to 12-hex on parse + at the producer. + */ + specifics: string[]; + trends: string[]; /** The verbatim post body — byte-exact, may contain anything (`]`/`|`/newlines/`---`). */ body: string; } @@ -54,8 +62,36 @@ export function serializePublishedRecord(rec: PublishedRecord): string { `published_date: ${rec.published_date}`, `captured_at: ${rec.captured_at}`, `source: ${rec.source}`, - ].join("\n"); - return header + SENTINEL + rec.body; + ]; + // SB-S3c: the raw-material id lines are appended AFTER `source:` and emitted ONLY + // when non-empty — so a pre-S3c record (empty arrays) serializes to the unchanged + // 5-line header, byte-identical (SC2). Body-only `mintContentId` is unaffected. + if (rec.specifics.length > 0) header.push(`specifics: ${rec.specifics.join(",")}`); + if (rec.trends.length > 0) header.push(`trends: ${rec.trends.join(",")}`); + return header.join("\n") + SENTINEL + rec.body; +} + +const ID_RE = /^[0-9a-f]{12}$/; + +/** + * SB-S3c: read an optional comma-separated 12-hex id list from the header slice. + * Absent key → `[]` (a NON-throwing reader — NOT `headerScalar`, which throws on a + * missing key and would break every pre-S3c record). A present-but-malformed id + * throws (never silently dropped). + */ +function headerIdList(header: string, key: string): string[] { + const m = header.match(new RegExp(`^${key}:\\s*(.*?)\\s*$`, "m")); + if (!m) return []; + return m[1] + .split(",") + .map((s) => s.trim()) + .filter((s) => s !== "") + .map((id) => { + if (!ID_RE.test(id)) { + throw new Error(`malformed published record: bad ${key} id ${JSON.stringify(id)}`); + } + return id; + }); } function headerScalar(header: string, key: string): string { @@ -94,6 +130,8 @@ export function parsePublishedRecord(text: string): PublishedRecord { published_date: headerScalar(header, "published_date"), captured_at: headerScalar(header, "captured_at"), source: headerScalar(header, "source"), + specifics: headerIdList(header, "specifics"), + trends: headerIdList(header, "trends"), body, }; } @@ -167,13 +205,27 @@ export function ingestText(opts: { captured_at: string; source?: string; published_date?: string; + specifics?: string[]; + trends?: string[]; }): IngestResult { + // SB-S3c producer guard: a malformed raw-material id fails fast HERE (not only on + // a later re-parse). Empty/absent arrays skip validation, so every pre-S3c caller + // (incl. scanInbox) is unaffected. + const validateIds = (ids: string[] | undefined, kind: string): string[] => { + const list = ids ?? []; + for (const id of list) { + if (!ID_RE.test(id)) throw new Error(`ingest: bad ${kind} id ${JSON.stringify(id)}`); + } + return list; + }; const record: PublishedRecord = { id: mintContentId(opts.body), provenance: "published", published_date: opts.published_date ?? opts.captured_at, captured_at: opts.captured_at, source: opts.source ?? "manual", + specifics: validateIds(opts.specifics, "specifics"), + trends: validateIds(opts.trends, "trends"), body: opts.body, }; return { record, ...writePublished(record) }; diff --git a/scripts/brain/tests/assemble.test.ts b/scripts/brain/tests/assemble.test.ts new file mode 100644 index 0000000..a8f7439 --- /dev/null +++ b/scripts/brain/tests/assemble.test.ts @@ -0,0 +1,95 @@ +import { describe, test } from "node:test"; +import assert from "node:assert/strict"; + +import { mintContentId } from "../src/id.js"; +import type { PublishedRecord } from "../src/ingest.js"; +import { assemblePostGraph, type AnalyticsRowInput } from "../src/assemble.js"; + +const DATE = "2026-05-26"; + +function rec(body: string, over: Partial = {}): PublishedRecord { + return { + id: mintContentId(body), + provenance: "published", + published_date: DATE, + captured_at: "2026-06-23", + source: "manual", + specifics: [], + trends: [], + body, + ...over, + }; +} + +function row(title: string, over: Partial = {}): AnalyticsRowInput { + return { title, publishedDate: DATE, metrics: { engagementRate: 4.2 }, ...over }; +} + +// A body whose normalized opener is ≥ PREFIX_FLOOR (24) chars. +const BODY = "Jeg lærte noe viktig om dømmekraft i dag.\n\nDel 1 av serien."; +const FULL_TITLE = "Jeg lærte noe viktig om dømmekraft i dag."; + +describe("SB-S3c assembler — assemblePostGraph (SC6/SC7/SC8)", () => { + // SC6 — high confidence: body begins with the row title (≥floor) + same date. + test("SC6: prefix-match + same date → high, whole row attached, specifics/trends surfaced", () => { + const r = rec(BODY, { specifics: ["aaaaaaaaaaaa"], trends: ["bbbbbbbbbbbb"] }); + const graph = assemblePostGraph({ records: [r], analytics: [row(FULL_TITLE)] }); + assert.equal(graph.length, 1); + assert.equal(graph[0].contentId, r.id); + assert.equal(graph[0].match.confidence, "high"); + assert.deepEqual(graph[0].match.row, row(FULL_TITLE)); // the whole row reference (FIX 4) + assert.deepEqual(graph[0].specifics, ["aaaaaaaaaaaa"]); + assert.deepEqual(graph[0].trends, ["bbbbbbbbbbbb"]); + }); + + // SC7 — low / none / below-floor / ellipsis near-miss. + test("SC7a: no prefix match → none", () => { + const graph = assemblePostGraph({ + records: [rec(BODY)], + analytics: [row("Completely unrelated opening sentence here")], + }); + assert.equal(graph[0].match.confidence, "none"); + assert.equal(graph[0].match.row, undefined); + }); + + test("SC7b: a too-short (< floor) title that is a literal prefix → none (floor guards false high)", () => { + const graph = assemblePostGraph({ records: [rec(BODY)], analytics: [row("Jeg lærte")] }); + assert.equal(graph[0].match.confidence, "none"); + }); + + test("SC7c: a LinkedIn-truncated '…' title + same date → high (ellipsis stripped)", () => { + const graph = assemblePostGraph({ + records: [rec(BODY)], + analytics: [row("Jeg lærte noe viktig om dømmekraft i…")], + }); + assert.equal(graph[0].match.confidence, "high"); + }); + + test("SC7d: prefix match but a different date → low (surfaced, not hidden)", () => { + const graph = assemblePostGraph({ + records: [rec(BODY)], + analytics: [row(FULL_TITLE, { publishedDate: "2026-05-20" })], + }); + assert.equal(graph[0].match.confidence, "low"); + assert.ok(graph[0].match.row, "row still attached on a low match"); + }); + + // SC8 — pure + total. + test("SC8: empty records → empty graph", () => { + assert.deepEqual(assemblePostGraph({ records: [], analytics: [row(FULL_TITLE)] }), []); + }); + + test("SC8: empty analytics → every post present with match none, no throw", () => { + const graph = assemblePostGraph({ records: [rec(BODY), rec("another post body here")], analytics: [] }); + assert.equal(graph.length, 2); + assert.ok(graph.every((g) => g.match.confidence === "none")); + }); + + test("SC8: best-of prefers high over low and is deterministic", () => { + const r = rec(BODY); + const high = row(FULL_TITLE); // same date + const low = row(FULL_TITLE, { publishedDate: "2026-01-01" }); // different date + const graph = assemblePostGraph({ records: [r], analytics: [low, high] }); + assert.equal(graph[0].match.confidence, "high", "high beats low regardless of input order"); + }); +}); diff --git a/scripts/brain/tests/cli.test.ts b/scripts/brain/tests/cli.test.ts index c66d9ba..1c7060e 100644 --- a/scripts/brain/tests/cli.test.ts +++ b/scripts/brain/tests/cli.test.ts @@ -1,7 +1,7 @@ import { describe, test, beforeEach, afterEach } from "node:test"; import assert from "node:assert/strict"; import { execFileSync } from "node:child_process"; -import { mkdtempSync, rmSync, existsSync, writeFileSync } from "node:fs"; +import { mkdtempSync, rmSync, existsSync, writeFileSync, readFileSync, readdirSync, mkdirSync } from "node:fs"; import { join, dirname } from "node:path"; import { fileURLToPath } from "node:url"; import { tmpdir } from "node:os"; @@ -75,4 +75,58 @@ describe("brain CLI dispatch (SB-S1)", () => { const { code } = runCli(root, ["bogus"]); assert.equal(code, 2); }); + + // SC5 — repeatable producer flags collect into arrays. + test("SC5: `ingest --specific a --specific c --trend b` tags the record [a,c]/[b]", () => { + const f = join(root, "post.md"); + writeFileSync(f, "A tagged published post body.", "utf8"); + const { stdout, code } = runCli(root, [ + "ingest", "--file", f, + "--specific", "aaaaaaaaaaaa", "--specific", "cccccccccccc", "--trend", "bbbbbbbbbbbb", + ]); + assert.equal(code, 0); + const idMatch = stdout.match(/published\/([0-9a-f]{12})\.md/); + assert.ok(idMatch, "wrote a record"); + const recText = readFileSync(join(root, "ingest", "published", `${idMatch![1]}.md`), "utf8"); + assert.match(recText, /specifics: aaaaaaaaaaaa,cccccccccccc/); + assert.match(recText, /trends: bbbbbbbbbbbb/); + }); + + // SC12 — single-value flags unregressed by the repeatable-flag change. + test("SC12: single-value `--source` + boolean `--scan-inbox` still parse as today", () => { + const f = join(root, "p.md"); + writeFileSync(f, "single-flag body", "utf8"); + runCli(root, ["ingest", "--file", f, "--source", "connector-x"]); + const recDir = join(root, "ingest", "published"); + const recFile = readFileSync(join(recDir, readdirSync(recDir)[0]), "utf8"); + assert.match(recFile, /source: connector-x/); + const { code } = runCli(root, ["ingest", "--scan-inbox"]); + assert.equal(code, 0); + }); + + // SC9 — read-only `assemble` prints the join and writes nothing. + test("SC9: `assemble` joins post↔analytics and writes nothing", () => { + const f = join(root, "post.md"); + const body = "Jeg lærte noe viktig om dømmekraft i dag. Del 1."; + writeFileSync(f, body, "utf8"); + runCli(root, ["ingest", "--file", f, "--date", "2026-05-26", "--specific", "aaaaaaaaaaaa"]); + // Seed an analytics batch JSON (raw shape; assemble inlines the read). + const postsDir = join(root, "analytics", "posts"); + mkdirSync(postsDir, { recursive: true }); + writeFileSync( + join(postsDir, "2026-05-26-batch.json"), + JSON.stringify({ posts: [{ title: "Jeg lærte noe viktig om dømmekraft i dag.", publishedDate: "2026-05-26", metrics: { engagementRate: 5.1 } }] }), + "utf8", + ); + const { stdout, code } = runCli(root, ["assemble"]); + assert.equal(code, 0); + assert.match(stdout, /aaaaaaaaaaaa/, "surfaces the specific id"); + assert.match(stdout, /high/i, "shows the high-confidence analytics match"); + assert.ok(!existsSync(join(root, "brain", "profile.md")), "assemble wrote no profile.md"); + }); + + test("SC9: `assemble` on an empty root degrades cleanly (no crash)", () => { + const { code } = runCli(root, ["assemble"]); + assert.equal(code, 0); + }); }); diff --git a/scripts/brain/tests/ingest.test.ts b/scripts/brain/tests/ingest.test.ts index 0149067..2a00026 100644 --- a/scripts/brain/tests/ingest.test.ts +++ b/scripts/brain/tests/ingest.test.ts @@ -14,6 +14,8 @@ const baseRec = (body: string): PublishedRecord => ({ published_date: "2026-05-26", captured_at: "2026-06-23", source: "manual", + specifics: [], + trends: [], body, }); @@ -82,3 +84,41 @@ describe("PublishedRecord grammar — parse∘serialize identity (SC2, B1 edge b assert.throws(() => parsePublishedRecord(corrupt), /provenance/i); }); }); + +describe("SB-S3c — raw-material ids on the published record", () => { + const ids2 = ["aaaaaaaaaaaa", "bbbbbbbbbbbb"]; + + // SC1 — round-trip with non-empty specifics/trends, order preserved. + test("SC1: parse∘serialize round-trips a record carrying specifics/trends (order preserved)", () => { + const rec: PublishedRecord = { ...baseRec("a tagged post"), specifics: ids2, trends: ["cccccccccccc"] }; + assert.deepEqual(roundTrip(rec), rec); + }); + + // SC2 — empty arrays → unchanged 5-line header; a pre-S3c fixture round-trips byte-identically. + test("SC2: empty specifics/trends serialize byte-identically to a pre-S3c record", () => { + const oldText = + "id: 0123456789ab\nprovenance: published\npublished_date: 2026-05-26\n" + + "captured_at: 2026-06-23\nsource: manual\n---\nthe body text"; + assert.equal(serializePublishedRecord(parsePublishedRecord(oldText)), oldText); + // and the live empty-array record still emits exactly 5 header lines. + const text = serializePublishedRecord(baseRec("hi")); + assert.equal(text.split("\n---\n")[0].split("\n").length, 5); + }); + + // SC2b — when present, the new lines are appended AFTER source: and before the sentinel. + test("SC2b: non-empty specifics/trends are appended after source:, before the --- sentinel", () => { + const text = serializePublishedRecord({ ...baseRec("x"), specifics: [ids2[0]], trends: [ids2[1]] }); + const header = text.split("\n---\n")[0]; + assert.match(header, /source: manual\nspecifics: aaaaaaaaaaaa\ntrends: bbbbbbbbbbbb$/); + }); + + // SC3 (parse side) — a non-12-hex id in the header throws, never silently dropped. + test("SC3: a malformed specifics/trends id throws on parse", () => { + const bad = + serializePublishedRecord(baseRec("x")).replace( + "source: manual", + "source: manual\nspecifics: NOTAHEXID", + ); + assert.throws(() => parsePublishedRecord(bad), /bad specifics id/i); + }); +}); diff --git a/scripts/brain/tests/publish.test.ts b/scripts/brain/tests/publish.test.ts index bc77c8f..c62d2f8 100644 --- a/scripts/brain/tests/publish.test.ts +++ b/scripts/brain/tests/publish.test.ts @@ -77,7 +77,7 @@ describe("ingest IO — writePublished / ingestText / scanInbox / listPublished mkdirSync(publishedDir(root), { recursive: true }); const squatter: PublishedRecord = { id, provenance: "published", published_date: CAPTURED, captured_at: CAPTURED, - source: "manual", body: "a DIFFERENT body that happens to share the id", + source: "manual", specifics: [], trends: [], body: "a DIFFERENT body that happens to share the id", }; writeFileSync(join(publishedDir(root), `${id}.md`), serializePublishedRecord(squatter), "utf8"); @@ -125,4 +125,34 @@ describe("ingest IO — writePublished / ingestText / scanInbox / listPublished assert.equal(res.records[0].provenance, "published"); assert.equal(res.skipped, 1, "the malformed file counted as skipped, not a crash"); }); + + // SC4 — ingestText threads specifics/trends onto the written record; absent → []. + test("SC4: ingestText threads specifics/trends; round-trips off disk", () => { + const spec = "aaaaaaaaaaaa"; + const trend = "bbbbbbbbbbbb"; + const res = ingestText({ body: "tagged post body", captured_at: CAPTURED, specifics: [spec], trends: [trend] }); + assert.deepEqual(res.record.specifics, [spec]); + assert.deepEqual(res.record.trends, [trend]); + const onDisk = parsePublishedRecord(readFileSync(res.path, "utf8")); + assert.deepEqual(onDisk.specifics, [spec]); + assert.deepEqual(onDisk.trends, [trend]); + }); + + test("SC4: ingestText with no specifics/trends keeps them empty (regression)", () => { + const res = ingestText({ body: "plain post", captured_at: CAPTURED }); + assert.deepEqual(res.record.specifics, []); + assert.deepEqual(res.record.trends, []); + }); + + // SC3 (producer side) — a malformed id thrown fast at the producer boundary. + test("SC3: ingestText with a non-12-hex specifics id throws at the producer", () => { + assert.throws( + () => ingestText({ body: "x", captured_at: CAPTURED, specifics: ["NOTAHEX"] }), + /bad specifics id/i, + ); + assert.throws( + () => ingestText({ body: "y", captured_at: CAPTURED, trends: ["zzz"] }), + /bad trends id/i, + ); + }); }); diff --git a/scripts/test-runner.sh b/scripts/test-runner.sh index 090e97f..435b029 100755 --- a/scripts/test-runner.sh +++ b/scripts/test-runner.sh @@ -713,7 +713,7 @@ if [ -x "$BR_DIR/node_modules/.bin/tsx" ]; then BR_OUT=$( set +e; (cd "$BR_DIR" && npm test) 2>&1; echo "BR_EXIT:$?" ) BR_EXIT=$(echo "$BR_OUT" | grep -oE 'BR_EXIT:[0-9]+' | grep -oE '[0-9]+' | head -1) BR_TESTS=$(echo "$BR_OUT" | grep -oE 'tests [0-9]+' | grep -oE '[0-9]+' | tail -1) - BRAIN_TESTS_FLOOR=94 # SB-S0 34 [id(11)+profile(6)+fold(12)+scaffold(5)] + SB-S1 29 [ingest(14)+publish(9)+cli(6)] + SB-S2 19 [consolidate(12)+consolidate-cli(7)] + SB-S3b 12 [consolidate(10)+consolidate-cli(2)] + BRAIN_TESTS_FLOOR=113 # SB-S0 34 [id(11)+profile(6)+fold(12)+scaffold(5)] + SB-S1 29 [ingest(14)+publish(9)+cli(6)] + SB-S2 19 [consolidate(12)+consolidate-cli(7)] + SB-S3b 12 [consolidate(10)+consolidate-cli(2)] + SB-S3c 19 [ingest(4)+publish(3)+assemble(8)+cli(4)] if [ "$BR_EXIT" = "0" ] && [ -n "$BR_TESTS" ] && [ "$BR_TESTS" -ge "$BRAIN_TESTS_FLOOR" ]; then pass "brain suite green: $BR_TESTS tests pass (floor $BRAIN_TESTS_FLOOR)" else