feat(linkedin-studio): SB-S3c — cross-silo id-threading + post→analytics assembler [skip-docs]
Hub-side design: the published record now carries the specifics/trends ids it was built from (additive, omit-empty → byte-backward-compatible), and a new pure assembler (scripts/brain/src/assemble.ts + `brain assemble`) joins post↔analytics by normalized title-prefix + date with honest confidence tiers (high/low/none). Answers the arc's north-star query: which raw material actually performs? (specific → post → measured analytics). All four tributaries untouched (analytics READ-only via inlined raw-JSON, no package import); profile.md grammar untouched (the fact→post link stays OUT — C-1). The repeatable --specific/--trend ingest flags collect via a new collectRepeated helper, leaving parseFlags untouched. TDD: 19 new brain tests (ingest 4 + publish 3 + assemble 8 + cli 4), all SC1–SC12. brain 113/113, gate 95/0/0, BRAIN_TESTS_FLOOR 94→113, ASSERT_BASELINE_FLOOR unchanged at 80. Light-Voyage hardened (brief-review 5 FIX · plan-critic 1 BLOCK+4 MAJOR+4 MINOR · scope-guardian ALIGNED). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RigJBiRFNtFZKCz21qNbQ4
This commit is contained in:
parent
016d823f3b
commit
edd3e15ef7
12 changed files with 515 additions and 14 deletions
|
|
@ -77,7 +77,7 @@ Run on a cadence (reuse the existing session-start staleness-nudge surface) as a
|
|||
| **SB-S0 — Foundation** ✅ *landed 2026-06-23* | `brain/` scaffold + two-layer `profile.md` (fold in `user-profile.md`) + `index.md` MOC + `operations.md` + `journal/`; entity-id + provenance shape as a small typed, tested module; `ingest/` dirs + manual-import contract. **No loop yet.** Shipped as `scripts/brain/` (TS, 34 tests, gate-wired BRAIN floor); fold = P1 labeled-scalars + P2 expertise (checkbox-prefs deferred, §8 of `plan-sb-s0.md`). | Smallest thing that stands up and is testable; locks the id/provenance spine everything hangs on |
|
||||
| **SB-S1 — Ingest + gold signal** ✅ *landed 2026-06-23* | Manual import → `ingest/published/` with `provenance=published` (CLI `brain ingest`); voice-trainer wired to learn from published-only, never `ai-draft`, gate-enforced. Shipped as `scripts/brain/src/ingest.ts` + `docs/second-brain/ingest-manual-import.md` (v0.5.1). **No profile.md mutation** (SB-S2). | The gold signal before the loop that consumes it |
|
||||
| **SB-S2 — Evolution loop** ✅ *landed 2026-06-23* | Operator-invoked, operator-gated consolidation: `brain consolidate` (`--gather`/`--propose`/`--apply --confirm`) → profile **diff** w/ evidence_count/provenance/timestamps; threshold-promotion (N=3); contradiction → keep-both with distinct ids (no supersede); decay-flag (90d); `consolidation-state.json` sidecar; zero-dep session-start consolidation-due nudge + scaffold-ensure. Shipped as `scripts/brain/src/consolidate.ts` + `docs/second-brain/consolidation-loop.md` (v0.5.2). **Operator decisions:** journal deferred · no new agent (session extracts) · motor-only (no reader until S3). | The compounding mechanism |
|
||||
| **SB-S3 — Cross-silo graph + ops centre** | Thread the id through tributaries (post↔specific↔trend↔analytics assemblable); flesh out `operations.md`; retire the dead `content-history.md` + triple-post reconciliation | Finally answers "which raw material performs?" |
|
||||
| **SB-S3 — Cross-silo graph + ops centre** | Thread the id through tributaries (post↔specific↔trend↔analytics assemblable); flesh out `operations.md`; retire the dead `content-history.md` + triple-post reconciliation. **S3a ✅ first reader · S3b ✅ supersede · S3c ✅ cross-silo id-threading** (hub-side: the published record carries the `specifics`/`trends` ids it was built from + a pure analytics resolver — `scripts/brain/src/assemble.ts`, `brain assemble`; tributaries untouched). **S3d remains** (ops centre / `content-history.md` retirement / triple-post reconciliation). | Finally answers "which raw material performs?" |
|
||||
| *(later / optional)* **SB-S4** | EU/EEA DMA portability API as an auto-tributary into `ingest/inbox/` | Additive; never a dependency |
|
||||
|
||||
## What's genuinely hard (honest flags)
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Brief — SB-S3c: cross-silo id-threading (the "which raw material performs?" graph)
|
||||
|
||||
> **Slice:** SB-S3c (third sub-slice of SB-S3). **Status:** DRAFT — awaiting operator "go" before any code. Light-Voyage **complete**: brief-review **APPROVE-WITH-FIXES** (5 FIX folded §9), plan-critic **REVISE → 1 BLOCK + 4 MAJOR + 4 MINOR folded** (§10), scope-guardian **ALIGNED** (0 creep/0 gap).
|
||||
> **Slice:** SB-S3c (third sub-slice of SB-S3). **Status:** **LANDED** — operator go 2026-06-23; TDD complete, brain suite 113/113, gate 95/0/0 (`BRAIN_TESTS_FLOOR` 94→113, `ASSERT_BASELINE_FLOOR` unchanged at 80). Light-Voyage: brief-review **APPROVE-WITH-FIXES** (5 FIX folded §9), plan-critic **REVISE → 1 BLOCK + 4 MAJOR + 4 MINOR folded** (§10), scope-guardian **ALIGNED** (0 creep/0 gap).
|
||||
> **Predecessors:** SB-S0 (id/provenance substrate — `mintEntityId`/`mintContentId`, `id.ts:6-7` reserved "SB-S3 will thread this id through the tributaries") · SB-S1 (published-gold ingest — the post anchor `ingest/published/<contentId>.md`) · SB-S2 (consolidation motor) · SB-S3a (first reader `strategy-advisor` — LANDED `4fa411f`) · **SB-S3b (supersede arm — LANDED `585f972`, gate 95/0/0, BRAIN floor 94, ASSERT floor 80).**
|
||||
> **Grounded in:** a read-only code map of all four tributaries + `scripts/brain/` (file:line cited throughout) — Explore sweep + firsthand reads of `id.ts`, `consolidate.ts`, `types.ts`, `profile.ts`, `ingest.ts`.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# Plan — SB-S3c: cross-silo id-threading
|
||||
|
||||
> Brief: `docs/second-brain/brief-sb-s3c.md`. Slice: SB-S3c.
|
||||
> **Design (recommended, pending operator go at the gate):** hub-side threading — `specifics`/`trends` ids onto the brain's published record + a pure analytics resolver. Tributaries untouched. (Fork-1; operator may redirect to mutate-tributaries.)
|
||||
> **Status:** **LANDED** (operator go 2026-06-23) — brain 113/113, gate 95/0/0, `BRAIN_TESTS_FLOOR` 94→113. **Design:** hub-side threading — `specifics`/`trends` ids onto the brain's published record + a pure analytics resolver. Tributaries untouched.
|
||||
> **Light-Voyage:** brief-review APPROVE-WITH-FIXES (5 FIX folded into the brief; 3 RISK carried here). plan-critic **REVISE → 1 BLOCK + 4 MAJOR + 4 MINOR folded** (Step 1 compile-fixup · Step 2.2 non-throwing parse reader · Step 2.3 producer guard · Step 3.3 floor rationale · Step 3.4 deterministic row-sort · Step 3.5 analytics-root caveat · Step 4.2 `runIngest` signature+call-site · Step 4.3 full-record load + desc sort · R-G sub-count). scope-guardian **ALIGNED** (0 creep / 0 gap).
|
||||
> Order is TDD: failing brain-suite tests land BEFORE the code (iron law); each rule pinned before the CLI is wired.
|
||||
|
||||
|
|
|
|||
156
scripts/brain/src/assemble.ts
Normal file
156
scripts/brain/src/assemble.ts
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
/**
|
||||
* SB-S3c — the cross-silo assembler (the payoff).
|
||||
*
|
||||
* Answers the arc's north-star query (`architecture.md:17`): *which raw material
|
||||
* actually performs?* — `specific → post → measured analytics`. The post record
|
||||
* (`ingest/published/<contentId>.md`) already carries the `specifics`/`trends` ids
|
||||
* it was built from (SB-S3c hub-side threading); this module joins each post to
|
||||
* its measured analytics row and surfaces the whole graph.
|
||||
*
|
||||
* PURE core: `assemblePostGraph({records, analytics})` takes already-loaded inputs
|
||||
* and returns the graph — no FS/clock/network. The analytics↔post join is an honest
|
||||
* HEURISTIC, never a guaranteed key: analytics carries no body and no URN (only a
|
||||
* title-prefix + date), so `matchRow` joins by normalized title-prefix + date with
|
||||
* explicit confidence tiers (`high`/`low`/`none`) — a real-CSV `none` is a
|
||||
* normalization-tightening signal, not a proof of no match.
|
||||
*
|
||||
* DECOUPLED: this module treats tributary ids as opaque 12-hex strings and takes a
|
||||
* minimal `AnalyticsRowInput` shape — it never imports the analytics/trends/
|
||||
* specifics packages. The thin read-only `loadAnalyticsRows` IO inlines a raw-JSON
|
||||
* read of the shared data-root (NOT the analytics package's `loadAllPosts`).
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, readdirSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { dataRoot } from "./dataRoot.js";
|
||||
import type { PublishedRecord } from "./ingest.js";
|
||||
|
||||
/**
|
||||
* The minimal analytics-row shape the resolver needs, extracted from the raw
|
||||
* `AnalyticsBatch.posts[]` JSON (`analytics/src/models/types.ts`). Note the field
|
||||
* is `publishedDate` (analytics) vs `published_date` (the brain record).
|
||||
*/
|
||||
export interface AnalyticsRowInput {
|
||||
title: string;
|
||||
publishedDate: string; // YYYY-MM-DD
|
||||
metrics?: { engagementRate?: number } & Record<string, unknown>;
|
||||
}
|
||||
|
||||
export type MatchConfidence = "high" | "low" | "none";
|
||||
|
||||
/** A post's matched analytics: the WHOLE row reference (FIX 4), or none. */
|
||||
export interface PostMatch {
|
||||
confidence: MatchConfidence;
|
||||
row?: AnalyticsRowInput;
|
||||
}
|
||||
|
||||
export interface PostGraphNode {
|
||||
contentId: string;
|
||||
published_date: string;
|
||||
specifics: string[];
|
||||
trends: string[];
|
||||
match: PostMatch;
|
||||
}
|
||||
|
||||
/**
|
||||
* Minimum normalized-title length to attempt a prefix match. The hook quality-rule
|
||||
* floor is 110 chars; 24 normalized chars (~3–5 words) is the shortest opener
|
||||
* specific enough that a prefix-match is not coincidental, while staying well under
|
||||
* any real hook. Below floor → `none` (an operator can still eyeball).
|
||||
*/
|
||||
const PREFIX_FLOOR = 24;
|
||||
|
||||
/** Brain-local copy of the specifics-bank `normalizeContent` idiom (NOT imported). */
|
||||
export function normalize(s: string): string {
|
||||
return s.trim().toLowerCase().replace(/\s+/g, " ");
|
||||
}
|
||||
|
||||
/** Strip a trailing LinkedIn truncation marker (`…`/`...`) so a `…`-suffixed export title still prefix-matches. */
|
||||
function stripTrailingEllipsis(s: string): string {
|
||||
return s.replace(/(?:…|\.{3})\s*$/, "").trimEnd();
|
||||
}
|
||||
|
||||
/**
|
||||
* Match one analytics row to one published record. Returns the tiered match, or
|
||||
* `null` when the row does not qualify (no prefix / below floor) — STUB until S3c
|
||||
* Step 3.
|
||||
*/
|
||||
export function matchRow(record: PublishedRecord, row: AnalyticsRowInput): PostMatch | null {
|
||||
const nt = stripTrailingEllipsis(normalize(row.title));
|
||||
if (nt.length < PREFIX_FLOOR) return null; // too short to discriminate → none
|
||||
if (!normalize(record.body).startsWith(nt)) return null; // no prefix → none
|
||||
const confidence: MatchConfidence = record.published_date === row.publishedDate ? "high" : "low";
|
||||
return { confidence, row };
|
||||
}
|
||||
|
||||
/**
|
||||
* Assemble the post → raw-material → performance graph. Pure (no FS/clock/network).
|
||||
* For each record, the BEST qualifying analytics row: `high` (same date) beats `low`
|
||||
* (different date); within a tier, the longest matched title wins. The analytics rows
|
||||
* are sorted once (publishedDate desc, title asc) so an exact-length tie is stable —
|
||||
* never `readdirSync`-order-dependent.
|
||||
*/
|
||||
export function assemblePostGraph(args: {
|
||||
records: PublishedRecord[];
|
||||
analytics: AnalyticsRowInput[];
|
||||
}): PostGraphNode[] {
|
||||
const analytics = [...args.analytics].sort(
|
||||
(a, b) => b.publishedDate.localeCompare(a.publishedDate) || a.title.localeCompare(b.title),
|
||||
);
|
||||
return args.records.map((record) => {
|
||||
let best: PostMatch | null = null;
|
||||
let bestLen = -1;
|
||||
for (const row of analytics) {
|
||||
const m = matchRow(record, row);
|
||||
if (!m) continue;
|
||||
const len = stripTrailingEllipsis(normalize(row.title)).length;
|
||||
const better =
|
||||
best === null ||
|
||||
(m.confidence === "high" && best.confidence === "low") ||
|
||||
(m.confidence === best.confidence && len > bestLen);
|
||||
if (better) {
|
||||
best = m;
|
||||
bestLen = len;
|
||||
}
|
||||
}
|
||||
return {
|
||||
contentId: record.id,
|
||||
published_date: record.published_date,
|
||||
specifics: record.specifics,
|
||||
trends: record.trends,
|
||||
match: best ?? { confidence: "none" },
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Read-only loader: inline a raw-JSON read of the analytics batches under the shared
|
||||
* data-root and extract the minimal row shape. STUB until S3c Step 3.
|
||||
*
|
||||
* NOTE (root-skew caveat): resolves via the brain `dataRoot` (`${LINKEDIN_STUDIO_DATA}/
|
||||
* analytics/posts`); the analytics package additionally honours the deprecated
|
||||
* `ANALYTICS_ROOT` override, which this path does NOT — if set to a non-default
|
||||
* path, the join degrades to every-post-`none` (accepted cost of the no-import
|
||||
* decoupling; the M0 default leaves `ANALYTICS_ROOT` unset).
|
||||
*/
|
||||
export function loadAnalyticsRows(): AnalyticsRowInput[] {
|
||||
const dir = dataRoot(join("analytics", "posts"));
|
||||
if (!existsSync(dir)) return []; // fresh-clone / no imports yet → no rows
|
||||
const rows: AnalyticsRowInput[] = [];
|
||||
for (const name of readdirSync(dir)) {
|
||||
if (!name.endsWith(".json") || name.startsWith(".")) continue;
|
||||
try {
|
||||
const batch = JSON.parse(readFileSync(join(dir, name), "utf8")) as { posts?: unknown[] };
|
||||
for (const p of batch?.posts ?? []) {
|
||||
const row = p as Partial<AnalyticsRowInput>;
|
||||
if (typeof row?.title === "string" && typeof row?.publishedDate === "string") {
|
||||
rows.push({ title: row.title, publishedDate: row.publishedDate, metrics: row.metrics });
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// skip a malformed/unreadable batch file — never crash (mirrors listPublished)
|
||||
}
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
|
@ -25,6 +25,7 @@ import {
|
|||
type Candidate,
|
||||
type ProfileDiff,
|
||||
} from "./consolidate.js";
|
||||
import { assemblePostGraph, loadAnalyticsRows } from "./assemble.js";
|
||||
import { dataRoot } from "./dataRoot.js";
|
||||
import { ingestText, listPublished, parsePublishedRecord, scanInbox } from "./ingest.js";
|
||||
import { parseProfile, serializeProfile } from "./profile.js";
|
||||
|
|
@ -59,6 +60,25 @@ function parseFlags(args: string[]): Record<string, string> {
|
|||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* SB-S3c: collect EVERY value of a repeatable `--key <value>` flag into an array
|
||||
* (the single-value `parseFlags` keeps only the last). Scans the raw args directly,
|
||||
* leaving `parseFlags` untouched — so single-value flag behaviour is unchanged.
|
||||
*/
|
||||
function collectRepeated(args: string[], key: string): string[] {
|
||||
const out: string[] = [];
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
if (args[i] === `--${key}`) {
|
||||
const next = args[i + 1];
|
||||
if (next !== undefined && !next.startsWith("--")) {
|
||||
out.push(next);
|
||||
i++;
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function today(): string {
|
||||
return new Date().toISOString().slice(0, 10);
|
||||
}
|
||||
|
|
@ -68,9 +88,10 @@ function usage(msg: string): never {
|
|||
console.error(
|
||||
"usage:\n" +
|
||||
" init\n" +
|
||||
" ingest --file <path> [--source <s>] [--date <YYYY-MM-DD>]\n" +
|
||||
" ingest --file <path> [--source <s>] [--date <YYYY-MM-DD>] [--specific <id>]… [--trend <id>]…\n" +
|
||||
" ingest --scan-inbox [--source <s>]\n" +
|
||||
" published list [--json]",
|
||||
" published list [--json]\n" +
|
||||
" assemble",
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
|
|
@ -89,7 +110,7 @@ function runInit(): void {
|
|||
if (created.length === 0) console.log("Already initialised — nothing to do.");
|
||||
}
|
||||
|
||||
function runIngest(flags: Record<string, string>): void {
|
||||
function runIngest(rest: string[], flags: Record<string, string>): void {
|
||||
if (flags["scan-inbox"] === "true") {
|
||||
const res = scanInbox({ captured_at: today(), source: flags.source });
|
||||
console.log(
|
||||
|
|
@ -100,11 +121,14 @@ function runIngest(flags: Record<string, string>): void {
|
|||
const file = flags.file;
|
||||
if (!file || file === "true") usage("ingest needs --file <path> or --scan-inbox");
|
||||
const body = readFileSync(file, "utf8");
|
||||
// SB-S3c: --specific / --trend are repeatable — read ONLY via collectRepeated.
|
||||
const res = ingestText({
|
||||
body,
|
||||
captured_at: today(),
|
||||
source: flags.source,
|
||||
published_date: flags.date === "true" ? undefined : flags.date,
|
||||
specifics: collectRepeated(rest, "specific"),
|
||||
trends: collectRepeated(rest, "trend"),
|
||||
});
|
||||
if (res.written && res.collision) {
|
||||
console.log(`Collision (different body, same id) → wrote ${res.path}`);
|
||||
|
|
@ -129,6 +153,53 @@ function runPublished(rest: string[], flags: Record<string, string>): void {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* SB-S3c: read-only cross-silo assembler view — post → raw-material → performance.
|
||||
* Loads published records + analytics rows (inlined raw-JSON, no analytics import),
|
||||
* prints the join newest-first. Writes NOTHING.
|
||||
*/
|
||||
function runAssemble(_flags: Record<string, string>): void {
|
||||
const pubDir = dataRoot(join("ingest", "published"));
|
||||
const records = existsSync(pubDir)
|
||||
? readdirSync(pubDir)
|
||||
.filter((f) => f.endsWith(".md") && !f.startsWith("."))
|
||||
.map((f) => {
|
||||
try {
|
||||
return parsePublishedRecord(readFileSync(join(pubDir, f), "utf8"));
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
})
|
||||
.filter((r): r is NonNullable<typeof r> => r !== null)
|
||||
: [];
|
||||
const analytics = loadAnalyticsRows();
|
||||
const bodyById = new Map(records.map((r) => [r.id, r.body]));
|
||||
const graph = assemblePostGraph({ records, analytics }).sort((a, b) =>
|
||||
b.published_date.localeCompare(a.published_date),
|
||||
);
|
||||
if (graph.length === 0) {
|
||||
console.log(
|
||||
"No published records to assemble. Ingest posts with `brain ingest --file <p> [--specific <id>] [--trend <id>]`.",
|
||||
);
|
||||
return;
|
||||
}
|
||||
console.log(`Post graph — ${graph.length} record(s); ${analytics.length} analytics row(s):`);
|
||||
for (const node of graph) {
|
||||
const firstLine = (bodyById.get(node.contentId) ?? "").split("\n", 1)[0];
|
||||
console.log(`\n· ${node.contentId} · ${node.published_date} · ${firstLine}`);
|
||||
console.log(` specifics: ${node.specifics.length ? node.specifics.join(", ") : "—"}`);
|
||||
console.log(` trends: ${node.trends.length ? node.trends.join(", ") : "—"}`);
|
||||
if (node.match.confidence === "none") {
|
||||
console.log(" analytics: none (no title-prefix+date match)");
|
||||
} else {
|
||||
const eng = node.match.row?.metrics?.engagementRate;
|
||||
console.log(
|
||||
` analytics: ${node.match.confidence}${eng !== undefined ? ` [eng ${eng}%]` : ""}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function renderDiffMd(diff: ProfileDiff): string {
|
||||
const lines = ["# Pending profile diff", "", "> Operator-gated. Review, then `brain consolidate --apply --diff brain/pending-diff.json --confirm`.", ""];
|
||||
const section = (title: string, items: string[]) => {
|
||||
|
|
@ -235,9 +306,10 @@ function main(): void {
|
|||
const flags = parseFlags(rest);
|
||||
|
||||
if (command === "init") return runInit();
|
||||
if (command === "ingest") return runIngest(flags);
|
||||
if (command === "ingest") return runIngest(rest, flags);
|
||||
if (command === "published") return runPublished(rest, flags);
|
||||
if (command === "consolidate") return runConsolidate(flags);
|
||||
if (command === "assemble") return runAssemble(flags);
|
||||
|
||||
usage(command ? `unknown command: ${command}` : "no command given");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -3,8 +3,10 @@
|
|||
*
|
||||
* Pure + deterministic — no filesystem, no clock, no network. The id is keyed on
|
||||
* a STABLE SLUG of the label, not the raw label, so editing a fact's *value* (or
|
||||
* the label's case/whitespace) never re-mints the id. SB-S3 will thread this id
|
||||
* through the tributaries; SB-S0 only establishes mint + shape.
|
||||
* the label's case/whitespace) never re-mints the id. SB-S3c threads this id
|
||||
* through the tributaries (the published record carries the `specifics`/`trends`
|
||||
* ids it was built from; analytics is joined by resolver — see `assemble.ts`);
|
||||
* SB-S0 established mint + shape.
|
||||
*/
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
|
|
|
|||
|
|
@ -35,6 +35,14 @@ export interface PublishedRecord {
|
|||
captured_at: string;
|
||||
/** `manual` (default) | a future connector token. */
|
||||
source: string;
|
||||
/**
|
||||
* SB-S3c — the raw material this post was built from: specifics-bank + trends
|
||||
* ids (12-hex), threaded hub-side so the post↔specific↔trend↔analytics graph is
|
||||
* assemblable. Default `[]`; serialized only when non-empty (omit-empty keeps a
|
||||
* pre-S3c record byte-identical). Validated to 12-hex on parse + at the producer.
|
||||
*/
|
||||
specifics: string[];
|
||||
trends: string[];
|
||||
/** The verbatim post body — byte-exact, may contain anything (`]`/`|`/newlines/`---`). */
|
||||
body: string;
|
||||
}
|
||||
|
|
@ -54,8 +62,36 @@ export function serializePublishedRecord(rec: PublishedRecord): string {
|
|||
`published_date: ${rec.published_date}`,
|
||||
`captured_at: ${rec.captured_at}`,
|
||||
`source: ${rec.source}`,
|
||||
].join("\n");
|
||||
return header + SENTINEL + rec.body;
|
||||
];
|
||||
// SB-S3c: the raw-material id lines are appended AFTER `source:` and emitted ONLY
|
||||
// when non-empty — so a pre-S3c record (empty arrays) serializes to the unchanged
|
||||
// 5-line header, byte-identical (SC2). Body-only `mintContentId` is unaffected.
|
||||
if (rec.specifics.length > 0) header.push(`specifics: ${rec.specifics.join(",")}`);
|
||||
if (rec.trends.length > 0) header.push(`trends: ${rec.trends.join(",")}`);
|
||||
return header.join("\n") + SENTINEL + rec.body;
|
||||
}
|
||||
|
||||
const ID_RE = /^[0-9a-f]{12}$/;
|
||||
|
||||
/**
|
||||
* SB-S3c: read an optional comma-separated 12-hex id list from the header slice.
|
||||
* Absent key → `[]` (a NON-throwing reader — NOT `headerScalar`, which throws on a
|
||||
* missing key and would break every pre-S3c record). A present-but-malformed id
|
||||
* throws (never silently dropped).
|
||||
*/
|
||||
function headerIdList(header: string, key: string): string[] {
|
||||
const m = header.match(new RegExp(`^${key}:\\s*(.*?)\\s*$`, "m"));
|
||||
if (!m) return [];
|
||||
return m[1]
|
||||
.split(",")
|
||||
.map((s) => s.trim())
|
||||
.filter((s) => s !== "")
|
||||
.map((id) => {
|
||||
if (!ID_RE.test(id)) {
|
||||
throw new Error(`malformed published record: bad ${key} id ${JSON.stringify(id)}`);
|
||||
}
|
||||
return id;
|
||||
});
|
||||
}
|
||||
|
||||
function headerScalar(header: string, key: string): string {
|
||||
|
|
@ -94,6 +130,8 @@ export function parsePublishedRecord(text: string): PublishedRecord {
|
|||
published_date: headerScalar(header, "published_date"),
|
||||
captured_at: headerScalar(header, "captured_at"),
|
||||
source: headerScalar(header, "source"),
|
||||
specifics: headerIdList(header, "specifics"),
|
||||
trends: headerIdList(header, "trends"),
|
||||
body,
|
||||
};
|
||||
}
|
||||
|
|
@ -167,13 +205,27 @@ export function ingestText(opts: {
|
|||
captured_at: string;
|
||||
source?: string;
|
||||
published_date?: string;
|
||||
specifics?: string[];
|
||||
trends?: string[];
|
||||
}): IngestResult {
|
||||
// SB-S3c producer guard: a malformed raw-material id fails fast HERE (not only on
|
||||
// a later re-parse). Empty/absent arrays skip validation, so every pre-S3c caller
|
||||
// (incl. scanInbox) is unaffected.
|
||||
const validateIds = (ids: string[] | undefined, kind: string): string[] => {
|
||||
const list = ids ?? [];
|
||||
for (const id of list) {
|
||||
if (!ID_RE.test(id)) throw new Error(`ingest: bad ${kind} id ${JSON.stringify(id)}`);
|
||||
}
|
||||
return list;
|
||||
};
|
||||
const record: PublishedRecord = {
|
||||
id: mintContentId(opts.body),
|
||||
provenance: "published",
|
||||
published_date: opts.published_date ?? opts.captured_at,
|
||||
captured_at: opts.captured_at,
|
||||
source: opts.source ?? "manual",
|
||||
specifics: validateIds(opts.specifics, "specifics"),
|
||||
trends: validateIds(opts.trends, "trends"),
|
||||
body: opts.body,
|
||||
};
|
||||
return { record, ...writePublished(record) };
|
||||
|
|
|
|||
95
scripts/brain/tests/assemble.test.ts
Normal file
95
scripts/brain/tests/assemble.test.ts
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
import { describe, test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { mintContentId } from "../src/id.js";
|
||||
import type { PublishedRecord } from "../src/ingest.js";
|
||||
import { assemblePostGraph, type AnalyticsRowInput } from "../src/assemble.js";
|
||||
|
||||
const DATE = "2026-05-26";
|
||||
|
||||
function rec(body: string, over: Partial<PublishedRecord> = {}): PublishedRecord {
|
||||
return {
|
||||
id: mintContentId(body),
|
||||
provenance: "published",
|
||||
published_date: DATE,
|
||||
captured_at: "2026-06-23",
|
||||
source: "manual",
|
||||
specifics: [],
|
||||
trends: [],
|
||||
body,
|
||||
...over,
|
||||
};
|
||||
}
|
||||
|
||||
function row(title: string, over: Partial<AnalyticsRowInput> = {}): AnalyticsRowInput {
|
||||
return { title, publishedDate: DATE, metrics: { engagementRate: 4.2 }, ...over };
|
||||
}
|
||||
|
||||
// A body whose normalized opener is ≥ PREFIX_FLOOR (24) chars.
|
||||
const BODY = "Jeg lærte noe viktig om dømmekraft i dag.\n\nDel 1 av serien.";
|
||||
const FULL_TITLE = "Jeg lærte noe viktig om dømmekraft i dag.";
|
||||
|
||||
describe("SB-S3c assembler — assemblePostGraph (SC6/SC7/SC8)", () => {
|
||||
// SC6 — high confidence: body begins with the row title (≥floor) + same date.
|
||||
test("SC6: prefix-match + same date → high, whole row attached, specifics/trends surfaced", () => {
|
||||
const r = rec(BODY, { specifics: ["aaaaaaaaaaaa"], trends: ["bbbbbbbbbbbb"] });
|
||||
const graph = assemblePostGraph({ records: [r], analytics: [row(FULL_TITLE)] });
|
||||
assert.equal(graph.length, 1);
|
||||
assert.equal(graph[0].contentId, r.id);
|
||||
assert.equal(graph[0].match.confidence, "high");
|
||||
assert.deepEqual(graph[0].match.row, row(FULL_TITLE)); // the whole row reference (FIX 4)
|
||||
assert.deepEqual(graph[0].specifics, ["aaaaaaaaaaaa"]);
|
||||
assert.deepEqual(graph[0].trends, ["bbbbbbbbbbbb"]);
|
||||
});
|
||||
|
||||
// SC7 — low / none / below-floor / ellipsis near-miss.
|
||||
test("SC7a: no prefix match → none", () => {
|
||||
const graph = assemblePostGraph({
|
||||
records: [rec(BODY)],
|
||||
analytics: [row("Completely unrelated opening sentence here")],
|
||||
});
|
||||
assert.equal(graph[0].match.confidence, "none");
|
||||
assert.equal(graph[0].match.row, undefined);
|
||||
});
|
||||
|
||||
test("SC7b: a too-short (< floor) title that is a literal prefix → none (floor guards false high)", () => {
|
||||
const graph = assemblePostGraph({ records: [rec(BODY)], analytics: [row("Jeg lærte")] });
|
||||
assert.equal(graph[0].match.confidence, "none");
|
||||
});
|
||||
|
||||
test("SC7c: a LinkedIn-truncated '…' title + same date → high (ellipsis stripped)", () => {
|
||||
const graph = assemblePostGraph({
|
||||
records: [rec(BODY)],
|
||||
analytics: [row("Jeg lærte noe viktig om dømmekraft i…")],
|
||||
});
|
||||
assert.equal(graph[0].match.confidence, "high");
|
||||
});
|
||||
|
||||
test("SC7d: prefix match but a different date → low (surfaced, not hidden)", () => {
|
||||
const graph = assemblePostGraph({
|
||||
records: [rec(BODY)],
|
||||
analytics: [row(FULL_TITLE, { publishedDate: "2026-05-20" })],
|
||||
});
|
||||
assert.equal(graph[0].match.confidence, "low");
|
||||
assert.ok(graph[0].match.row, "row still attached on a low match");
|
||||
});
|
||||
|
||||
// SC8 — pure + total.
|
||||
test("SC8: empty records → empty graph", () => {
|
||||
assert.deepEqual(assemblePostGraph({ records: [], analytics: [row(FULL_TITLE)] }), []);
|
||||
});
|
||||
|
||||
test("SC8: empty analytics → every post present with match none, no throw", () => {
|
||||
const graph = assemblePostGraph({ records: [rec(BODY), rec("another post body here")], analytics: [] });
|
||||
assert.equal(graph.length, 2);
|
||||
assert.ok(graph.every((g) => g.match.confidence === "none"));
|
||||
});
|
||||
|
||||
test("SC8: best-of prefers high over low and is deterministic", () => {
|
||||
const r = rec(BODY);
|
||||
const high = row(FULL_TITLE); // same date
|
||||
const low = row(FULL_TITLE, { publishedDate: "2026-01-01" }); // different date
|
||||
const graph = assemblePostGraph({ records: [r], analytics: [low, high] });
|
||||
assert.equal(graph[0].match.confidence, "high", "high beats low regardless of input order");
|
||||
});
|
||||
});
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
import { describe, test, beforeEach, afterEach } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdtempSync, rmSync, existsSync, writeFileSync } from "node:fs";
|
||||
import { mkdtempSync, rmSync, existsSync, writeFileSync, readFileSync, readdirSync, mkdirSync } from "node:fs";
|
||||
import { join, dirname } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { tmpdir } from "node:os";
|
||||
|
|
@ -75,4 +75,58 @@ describe("brain CLI dispatch (SB-S1)", () => {
|
|||
const { code } = runCli(root, ["bogus"]);
|
||||
assert.equal(code, 2);
|
||||
});
|
||||
|
||||
// SC5 — repeatable producer flags collect into arrays.
|
||||
test("SC5: `ingest --specific a --specific c --trend b` tags the record [a,c]/[b]", () => {
|
||||
const f = join(root, "post.md");
|
||||
writeFileSync(f, "A tagged published post body.", "utf8");
|
||||
const { stdout, code } = runCli(root, [
|
||||
"ingest", "--file", f,
|
||||
"--specific", "aaaaaaaaaaaa", "--specific", "cccccccccccc", "--trend", "bbbbbbbbbbbb",
|
||||
]);
|
||||
assert.equal(code, 0);
|
||||
const idMatch = stdout.match(/published\/([0-9a-f]{12})\.md/);
|
||||
assert.ok(idMatch, "wrote a record");
|
||||
const recText = readFileSync(join(root, "ingest", "published", `${idMatch![1]}.md`), "utf8");
|
||||
assert.match(recText, /specifics: aaaaaaaaaaaa,cccccccccccc/);
|
||||
assert.match(recText, /trends: bbbbbbbbbbbb/);
|
||||
});
|
||||
|
||||
// SC12 — single-value flags unregressed by the repeatable-flag change.
|
||||
test("SC12: single-value `--source` + boolean `--scan-inbox` still parse as today", () => {
|
||||
const f = join(root, "p.md");
|
||||
writeFileSync(f, "single-flag body", "utf8");
|
||||
runCli(root, ["ingest", "--file", f, "--source", "connector-x"]);
|
||||
const recDir = join(root, "ingest", "published");
|
||||
const recFile = readFileSync(join(recDir, readdirSync(recDir)[0]), "utf8");
|
||||
assert.match(recFile, /source: connector-x/);
|
||||
const { code } = runCli(root, ["ingest", "--scan-inbox"]);
|
||||
assert.equal(code, 0);
|
||||
});
|
||||
|
||||
// SC9 — read-only `assemble` prints the join and writes nothing.
|
||||
test("SC9: `assemble` joins post↔analytics and writes nothing", () => {
|
||||
const f = join(root, "post.md");
|
||||
const body = "Jeg lærte noe viktig om dømmekraft i dag. Del 1.";
|
||||
writeFileSync(f, body, "utf8");
|
||||
runCli(root, ["ingest", "--file", f, "--date", "2026-05-26", "--specific", "aaaaaaaaaaaa"]);
|
||||
// Seed an analytics batch JSON (raw shape; assemble inlines the read).
|
||||
const postsDir = join(root, "analytics", "posts");
|
||||
mkdirSync(postsDir, { recursive: true });
|
||||
writeFileSync(
|
||||
join(postsDir, "2026-05-26-batch.json"),
|
||||
JSON.stringify({ posts: [{ title: "Jeg lærte noe viktig om dømmekraft i dag.", publishedDate: "2026-05-26", metrics: { engagementRate: 5.1 } }] }),
|
||||
"utf8",
|
||||
);
|
||||
const { stdout, code } = runCli(root, ["assemble"]);
|
||||
assert.equal(code, 0);
|
||||
assert.match(stdout, /aaaaaaaaaaaa/, "surfaces the specific id");
|
||||
assert.match(stdout, /high/i, "shows the high-confidence analytics match");
|
||||
assert.ok(!existsSync(join(root, "brain", "profile.md")), "assemble wrote no profile.md");
|
||||
});
|
||||
|
||||
test("SC9: `assemble` on an empty root degrades cleanly (no crash)", () => {
|
||||
const { code } = runCli(root, ["assemble"]);
|
||||
assert.equal(code, 0);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -14,6 +14,8 @@ const baseRec = (body: string): PublishedRecord => ({
|
|||
published_date: "2026-05-26",
|
||||
captured_at: "2026-06-23",
|
||||
source: "manual",
|
||||
specifics: [],
|
||||
trends: [],
|
||||
body,
|
||||
});
|
||||
|
||||
|
|
@ -82,3 +84,41 @@ describe("PublishedRecord grammar — parse∘serialize identity (SC2, B1 edge b
|
|||
assert.throws(() => parsePublishedRecord(corrupt), /provenance/i);
|
||||
});
|
||||
});
|
||||
|
||||
describe("SB-S3c — raw-material ids on the published record", () => {
|
||||
const ids2 = ["aaaaaaaaaaaa", "bbbbbbbbbbbb"];
|
||||
|
||||
// SC1 — round-trip with non-empty specifics/trends, order preserved.
|
||||
test("SC1: parse∘serialize round-trips a record carrying specifics/trends (order preserved)", () => {
|
||||
const rec: PublishedRecord = { ...baseRec("a tagged post"), specifics: ids2, trends: ["cccccccccccc"] };
|
||||
assert.deepEqual(roundTrip(rec), rec);
|
||||
});
|
||||
|
||||
// SC2 — empty arrays → unchanged 5-line header; a pre-S3c fixture round-trips byte-identically.
|
||||
test("SC2: empty specifics/trends serialize byte-identically to a pre-S3c record", () => {
|
||||
const oldText =
|
||||
"id: 0123456789ab\nprovenance: published\npublished_date: 2026-05-26\n" +
|
||||
"captured_at: 2026-06-23\nsource: manual\n---\nthe body text";
|
||||
assert.equal(serializePublishedRecord(parsePublishedRecord(oldText)), oldText);
|
||||
// and the live empty-array record still emits exactly 5 header lines.
|
||||
const text = serializePublishedRecord(baseRec("hi"));
|
||||
assert.equal(text.split("\n---\n")[0].split("\n").length, 5);
|
||||
});
|
||||
|
||||
// SC2b — when present, the new lines are appended AFTER source: and before the sentinel.
|
||||
test("SC2b: non-empty specifics/trends are appended after source:, before the --- sentinel", () => {
|
||||
const text = serializePublishedRecord({ ...baseRec("x"), specifics: [ids2[0]], trends: [ids2[1]] });
|
||||
const header = text.split("\n---\n")[0];
|
||||
assert.match(header, /source: manual\nspecifics: aaaaaaaaaaaa\ntrends: bbbbbbbbbbbb$/);
|
||||
});
|
||||
|
||||
// SC3 (parse side) — a non-12-hex id in the header throws, never silently dropped.
|
||||
test("SC3: a malformed specifics/trends id throws on parse", () => {
|
||||
const bad =
|
||||
serializePublishedRecord(baseRec("x")).replace(
|
||||
"source: manual",
|
||||
"source: manual\nspecifics: NOTAHEXID",
|
||||
);
|
||||
assert.throws(() => parsePublishedRecord(bad), /bad specifics id/i);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -77,7 +77,7 @@ describe("ingest IO — writePublished / ingestText / scanInbox / listPublished
|
|||
mkdirSync(publishedDir(root), { recursive: true });
|
||||
const squatter: PublishedRecord = {
|
||||
id, provenance: "published", published_date: CAPTURED, captured_at: CAPTURED,
|
||||
source: "manual", body: "a DIFFERENT body that happens to share the id",
|
||||
source: "manual", specifics: [], trends: [], body: "a DIFFERENT body that happens to share the id",
|
||||
};
|
||||
writeFileSync(join(publishedDir(root), `${id}.md`), serializePublishedRecord(squatter), "utf8");
|
||||
|
||||
|
|
@ -125,4 +125,34 @@ describe("ingest IO — writePublished / ingestText / scanInbox / listPublished
|
|||
assert.equal(res.records[0].provenance, "published");
|
||||
assert.equal(res.skipped, 1, "the malformed file counted as skipped, not a crash");
|
||||
});
|
||||
|
||||
// SC4 — ingestText threads specifics/trends onto the written record; absent → [].
|
||||
test("SC4: ingestText threads specifics/trends; round-trips off disk", () => {
|
||||
const spec = "aaaaaaaaaaaa";
|
||||
const trend = "bbbbbbbbbbbb";
|
||||
const res = ingestText({ body: "tagged post body", captured_at: CAPTURED, specifics: [spec], trends: [trend] });
|
||||
assert.deepEqual(res.record.specifics, [spec]);
|
||||
assert.deepEqual(res.record.trends, [trend]);
|
||||
const onDisk = parsePublishedRecord(readFileSync(res.path, "utf8"));
|
||||
assert.deepEqual(onDisk.specifics, [spec]);
|
||||
assert.deepEqual(onDisk.trends, [trend]);
|
||||
});
|
||||
|
||||
test("SC4: ingestText with no specifics/trends keeps them empty (regression)", () => {
|
||||
const res = ingestText({ body: "plain post", captured_at: CAPTURED });
|
||||
assert.deepEqual(res.record.specifics, []);
|
||||
assert.deepEqual(res.record.trends, []);
|
||||
});
|
||||
|
||||
// SC3 (producer side) — a malformed id thrown fast at the producer boundary.
|
||||
test("SC3: ingestText with a non-12-hex specifics id throws at the producer", () => {
|
||||
assert.throws(
|
||||
() => ingestText({ body: "x", captured_at: CAPTURED, specifics: ["NOTAHEX"] }),
|
||||
/bad specifics id/i,
|
||||
);
|
||||
assert.throws(
|
||||
() => ingestText({ body: "y", captured_at: CAPTURED, trends: ["zzz"] }),
|
||||
/bad trends id/i,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -713,7 +713,7 @@ if [ -x "$BR_DIR/node_modules/.bin/tsx" ]; then
|
|||
BR_OUT=$( set +e; (cd "$BR_DIR" && npm test) 2>&1; echo "BR_EXIT:$?" )
|
||||
BR_EXIT=$(echo "$BR_OUT" | grep -oE 'BR_EXIT:[0-9]+' | grep -oE '[0-9]+' | head -1)
|
||||
BR_TESTS=$(echo "$BR_OUT" | grep -oE 'tests [0-9]+' | grep -oE '[0-9]+' | tail -1)
|
||||
BRAIN_TESTS_FLOOR=94 # SB-S0 34 [id(11)+profile(6)+fold(12)+scaffold(5)] + SB-S1 29 [ingest(14)+publish(9)+cli(6)] + SB-S2 19 [consolidate(12)+consolidate-cli(7)] + SB-S3b 12 [consolidate(10)+consolidate-cli(2)]
|
||||
BRAIN_TESTS_FLOOR=113 # SB-S0 34 [id(11)+profile(6)+fold(12)+scaffold(5)] + SB-S1 29 [ingest(14)+publish(9)+cli(6)] + SB-S2 19 [consolidate(12)+consolidate-cli(7)] + SB-S3b 12 [consolidate(10)+consolidate-cli(2)] + SB-S3c 19 [ingest(4)+publish(3)+assemble(8)+cli(4)]
|
||||
if [ "$BR_EXIT" = "0" ] && [ -n "$BR_TESTS" ] && [ "$BR_TESTS" -ge "$BRAIN_TESTS_FLOOR" ]; then
|
||||
pass "brain suite green: $BR_TESTS tests pass (floor $BRAIN_TESTS_FLOOR)"
|
||||
else
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue