linkedin-studio/docs/research-engine/plan-re-r2a.md

Plan — RE-R2a: capture bridge (item→store) + publishedAt persistence

Brief: docs/research-engine/brief-re-r2a.md. Slice: RE-R2a (research-engine rung-2, slice 2 — R2 data layer). TDD-order: RED (migration + bridge + capture tests as logic-RED) → GREEN (types.ts/store.ts migration + persist) → GREEN (item.ts bridge) → GREEN (cli.ts capture + tests) → wire trend-spotter.md → gate floors → behavioural → land. Counts recounted live at land, never pinned/guessed. Light-Voyage hardened: scope-guardian / brief-reviewer / plan-critic findings folded (see §Plan-critic — folded).

Goal

Build the item→store bridge RE-R1 deferred: a pure itemToInput mapping the validated TrendItem envelope to a store input (injecting capturedAt, carrying publishedAt), a publishedAt schema bump (v1→v2) with a lossless migrate-on-load, and a capture CLI that closes the poll→normalize→store loop. No scoring change; no hook touch; the dated brief + surfacing are R2b.

Files touched (exhaustive — for scope-guardian)

File	Change	SC
scripts/trends/src/types.ts	EDIT — SCHEMA_VERSION 1→2; publishedAt?: string on TrendRecord (after capturedAt, with the capturedAt-distinction comment)	SC3
scripts/trends/src/store.ts	EDIT — TrendInput +publishedAt?; addTrend persists it (conditional-spread, first-sight kept on merge); loadStore forward migrate-on-load (max(onDisk, SCHEMA_VERSION))	SC2, SC3
scripts/trends/src/item.ts	EDIT — pure itemToInput(item, capturedAt): TrendInput (injects capturedAt; no id; no re-validate)	SC1
scripts/trends/src/cli.ts	EDIT — capture subcommand only (stdin→normalize→bridge→addTrend→saveStore→summary; exit 2 bad invocation; --json). (add --published-at deferred at light-Voyage — see brief §4.)	SC4
scripts/trends/tests/store.test.ts	EDIT — migration cases (v1→v2 lossless, idempotent, round-trip) + publishedAt persist/first-sight-merge cases	SC2, SC3
scripts/trends/tests/item.test.ts	EDIT — itemToInput mapping cases (capturedAt inject, carry-through, no id, absent publishedAt omitted)	SC1
scripts/trends/tests/cli.test.ts	EDIT — capture happy path (stdin→store), duplicate/merge, content-invalid in errors[], exit-2 bad invocation, --json summary	SC4
scripts/trends/README.md	EDIT — add publishedAt? to the record-shape block + a capture example (honest schema/CLI doc)	—
agents/trend-spotter.md	EDIT (Open Q#1, default) — Step 4.5 add→capture; carries literal src/cli.ts capture; domain-general	SC6
scripts/test-runner.sh	EDIT — TRENDS_TESTS_FLOOR 62→recount (stays inside deps guard); NEW unconditional Section 16h (before Section 18); ASSERT_BASELINE_FLOOR 87→recount; anti-erosion header enumeration	SC5, SC6
docs/research-engine/{brief,plan}-re-r2a.md	NEW — slice docs (TRACKED, like docs/second-brain/*)	—
STATE.md	EDIT at land — Telling-block reconcile (trends floor, ASSERT floor, gate total). Land bookkeeping, not slice scope; LOCAL-ONLY.	—

Not touched (scope fence): scripts/trends/src/score.ts (no scoring change) · references/trend-scoring-modes.md + references/* (SSOT unchanged, no new ref) · hooks/** (no surfacing — R2b) · config/trends-sources.template.md · no new .ts source file (bridge in item.ts) · agents/* count (19) · commands/* (29) · .gitignore (trends lines present).

Step 1 — (RED) failing tests for migration + bridge + capture

Extend the three existing test files against the not-yet-changed code. Logic-RED (not import-RED), stub-by-assertion-type. Critical RED-vs-GREEN-guard split (plan-critic blocker): against old code (SCHEMA_VERSION=1, loadStore returns parsed.schemaVersion ?? SCHEMA_VERSION), only some assertions actually fail:

• Genuinely RED (old code fails): v1-fixture load → schemaVersion===2 (old returns 1); v1 round-trip loadStore→saveStore writes schemaVersion:2 (old writes 1); addTrend({…, publishedAt}) persists it (old drops it); itemToInput carries/injects correctly (add a thin throwing stub for the not-yet-existing export so the case fails on assertion, not on undefined is not a function).
• GREEN-only regression guards (pass against old code — NOT labelled RED): a v2 fixture load → schemaVersion===2 (old already returns 2 via ?? ); missing schemaVersion → 2 (old ?? already yields current); non-numeric schemaVersion → 2; empty/absent store → {schemaVersion:2,trends:[]}. These are written in Step 1 but documented as regression guards, so the RED proof is not falsely claimed for them.

store.test.ts: the RED migration cases above + the regression-guard cases; addTrend without publishedAt omits the key; re-capture (same title+url) leaves existing publishedAt unchanged and only unions topics; absent→present re-capture (first sighting lacked publishedAt, re-capture carries one) does NOT add it (no back-fill, Open Q#2), merged reflects topic change alone. item.test.ts: itemToInput(item,"2026-06-24") → capturedAt injected (==="2026-06-24"), all fields carried verbatim, no id, item without publishedAt → input without the key; field-confusion guard: an item whose publishedAt differs from the injected date → result.capturedAt !== result.publishedAt. cli.test.ts: capture happy (one valid item piped → store gains it, added:1); a batch with one invalid item → errors[] carries it, valid one added, exit 0; the summary tally sums to the input size; the captured record's capturedAt matches /^\d{4}-\d{2}-\d{2}$/ and !== the item's publishedAt; empty/unparseable stdin → exit 2; --json emits the summary object.

RED proof (record in commit): (cd scripts/trends && npm test) → the genuinely-RED cases fail on assertion (logic-RED), not module-not-found; the regression-guard cases may pass pre-change (documented, not claimed RED).

Step 2 — (GREEN) schema migration: types.ts + store.ts

types.ts: SCHEMA_VERSION = 2; add publishedAt?: string to TrendRecord after capturedAt with the distinction comment (source publish-date; forward-compat for B4; distinct from capturedAt). store.ts: TrendInput gains publishedAt?; addTrend adds ...(input.publishedAt !== undefined ? { publishedAt: input.publishedAt } : {}) to the new-record literal (after capturedAt); the merge branch is unchanged (topics union only — first-sight publishedAt kept, no back-fill). loadStore returns schemaVersion: Math.max(onDisk, SCHEMA_VERSION) where onDisk = typeof parsed.schemaVersion === "number" ? parsed.schemaVersion : SCHEMA_VERSION (forward-only stamp handling string/NaN/absent → current; never crashes); the existing Array.isArray(parsed.trends) ? … : [] coercion (store.ts:79) is preserved verbatim — a corrupt trends field stays out of the losslessness claim. Make Step 1's RED migration + persist cases (and the regression guards) green.

Step 3 — (GREEN) bridge: itemToInput in item.ts

Add import type { TrendInput } from "./store.js"; (item.ts already imports the normalizeField value from there; this adds the type — dependency direction item.ts → store.ts stays acyclic). Then replace the Step-1 throwing stub with export function itemToInput(item: TrendItem, capturedAt: string): TrendInput returning { source, title, url, topics: [...item.topics], capturedAt, ...(item.publishedAt !== undefined ? { publishedAt: item.publishedAt } : {}), ...(item.summary !== undefined ? { summary: item.summary } : {}) }. No id; no re-validation (the envelope is already validated). Confirm no throwing stub survives into GREEN. Make Step 1's bridge cases green.

Step 4 — (GREEN) CLI capture + cli.test.ts

Add capture to cli.ts's main dispatch: read stdin via the existing readStdinJson() (its empty/unparseable path already does usage()→exit 2); Array.isArray(payload) ? normalizeItems(payload) : normalizeItem(payload); for each valid item, itemToInput(item, today()) → addTrend(store, res.store…); saveStore once. Tally derived from AddResult {added, merged} (no duplicates field, store.ts:35-41): added += res.added ? 1 : 0; merged += (!res.added && res.merged) ? 1 : 0; duplicates += (!res.added && !res.merged) ? 1 : 0. Human summary by default; --json prints {added, duplicates, merged, errors}. add --published-at is NOT added (deferred). Write the cli.test.ts cases (subprocess: node --import tsx src/cli.ts capture with piped stdin + a --store temp path), including an explicit added + merged + duplicates + errors.length === payload.length assertion and a capturedAt shape (/^\d{4}-\d{2}-\d{2}$/) + !== publishedAt check. capturedAt exact-value assertions live in item.test.ts (injected fixed date), never in cli.test.ts (which reads the wall clock — would be flaky).

Step 5 — wire trend-spotter.md (Open Q#1, default) + README

Replace Step 4.5's N× add block (trend-spotter.md:282-301) with: build a raw-item JSON batch (the same items already scored), pipe it to scripts/trends/src/cli.ts capture in one call — it normalizes + folds + persists publishedAt. Replacement prose must contain the literal src/cli.ts capture (Section 16h grep -qF). Keep the "skip silently if no deps" escape hatch + domain-general phrasing (no vendor/sector tokens — Section 17). Update scripts/trends/README.md: add publishedAt? to the record-shape block, add a capture CLI example, and correct the add framing — README.md:47-64 currently calls add "the capture path"; after the re-point add is the manual single-trend path and capture is the normalizing batch path. Fix the framing, don't just append (else the README contradicts the new agent wiring).

Step 6 — gate: floors + new unconditional Section 16h

In scripts/test-runner.sh:

• Bump TRENDS_TESTS_FLOOR 62 → live recount after Steps 1–4. Stays inside the if [ -x …/tsx ] deps guard (conditional — do not hoist). Also update the inline breakdown comment at test-runner.sh:695 (# store 24 + RE-R1: item 18 + score 16 + cli 4) — extend it with the R2a additions (e.g. + RE-R2a: store +N + item +M + cli +K), so the comment doesn't drift from the number (the exact erosion the anti-erosion section guards against).
• Add Section 16h ("Trends Capture Wiring"), mirroring Section 16g's three-check shape, after Section 17 / before Section 18. Three unconditional, deps-absent-safe checks (pure grep/self-test, no tsx): (1) a non-vacuity self-test (accept a probe carrying src/cli.ts capture, reject one without it); (2) grep -qF 'command === "capture"' scripts/trends/src/cli.ts — deps-absent proof the handler exists (grep -qF exact, mirroring 16g lines 1058/1064 — not grep -E); (3) grep -qF "src/cli.ts capture" agents/trend-spotter.md.
• Bump ASSERT_BASELINE_FLOOR 87 → live recount (= 87 + the new unconditional emitters in 16h; expected ~90, recounted at land, not pinned). Update the Section-18 anti-erosion header enumeration comment.

Step 7 — behavioural verification

(cd scripts/trends && npm install) if needed, then: echo '[{"source":"tavily","title":"X","url":"https://e/x","topics":["a"],"publishedAt":"2026-06-20"},{"title":"bad"}]' | node --import tsx src/cli.ts capture --store /tmp/r2a-trends.json --json → confirm added:1, the invalid entry in errors[], and node … list --store /tmp/r2a-trends.json --json shows the persisted publishedAt. Author a v1 fixture ({"schemaVersion":1,"trends":[{…no publishedAt}]}), run status --json → confirm count intact (migration lossless). Run full bash scripts/test-runner.sh → FAIL=0.

Step 8 — land

Recount all touched floors live; reconcile STATE.md "Telling" block (trends N/N, ASSERT floor, gate total). Commit order (house style): (1) docs commit docs/research-engine/{brief,plan}-re-r2a.md (no suffix, tracked); (2) code commit scripts/trends/* + agents/trend-spotter.md + scripts/trends/README.md + scripts/test-runner.sh with [skip-docs]. Push freely (window restriction lifted 2026-06-24; gitleaks runs at commit; origin = PUBLIC open/ — STATE/*.local.* never pushed). No version bump (additive; v0.5.2 dev).

Verification (testable)

SC	Check	Command	Expected
—	RED proof	(cd scripts/trends && npm test) after Step 1	new cases fail on assertion (logic-RED), not module-not-found
SC1	bridge	npm test (item.test.ts)	capturedAt injected (==="2026-06-24"); fields carried; no id; absent publishedAt omitted; capturedAt !== publishedAt (field-confusion guard)
SC2	persist	npm test (store.test.ts)	publishedAt persisted when present / omitted when absent; first-sight kept on merge; absent→present re-capture does NOT back-fill
SC3	migrate	npm test (store.test.ts)	v1→v2 lossless (records intact, no publishedAt invented); missing/non-numeric schemaVersion → 2; empty store → v2/[]; idempotent; round-trip writes v2
SC4	capture	npm test (cli.test.ts) + manual	normalize+fold; errors[] for invalid; tally sums to input size; capturedAt today-shaped & ≠ publishedAt; exit 2 bad stdin / 0 well-formed; --json summary
SC5	gate	bash scripts/test-runner.sh	FAIL=0; trends ≥ new floor; Section 16h green; ASSERT floor = live recount
SC6	wiring + de-niche	Section 16h greps + Section 17	src/cli.ts capture in trend-spotter.md; no vendor/sector tokens; counts 19/29/27

Risks

• R1 — migration eats or rewrites records. Mitigated: migrate-on-load is a version stamp only (max(onDisk, current)); records pass through untouched; SC3 pins lossless + idempotent + round-trip on a real v1 fixture.
• R2 — publishedAt first-sight vs back-fill ambiguity. Mitigated: brief Open Q#2 settles it at first-sight (no back-fill); SC2 pins "existing publishedAt unchanged on re-capture"; back-fill explicitly deferred.
• R3 — capture exit-code semantics drift (content-invalid item leaking into a non-zero exit). Mitigated: SC4 pins exit 2 = malformed invocation only; content-invalid → errors[] at exit 0; mirrors normalize/score.
• R4 — editing trend-spotter.md trips the de-niche guard. Mitigated: Section 17 runs in the gate; replacement prose is pillar/source-driven and vendor/sector-free; only the store-fold mechanism changes (add→capture).
• R5 — new gate checks must survive a deps-absent fresh clone. Mitigated: Section 16h is pure grep/self-test on tracked source (no tsx) → unconditional; TRENDS_TESTS_FLOOR stays inside the deps guard.
• R6 — future schema downgrade (on-disk > current) silently drops unknown fields on save. Mitigated/deferred: cannot happen pre-v3; max() already refuses to downgrade the stamp; field-preservation-on-save is an R-future concern, noted not handled (no impossible-scenario code).

Plan-critic — folded

plan-critic returned REVISE (1 blocker, 5 majors, 5 minors); brief-reviewer PROCEED_WITH_RISKS; scope-guardian ALIGNED. Resolution, each verified against live code:

• [BLOCKER] "v2 load idempotent" is not RED (store.ts:78 ?? already returns 2 for a v2 fixture under old SCHEMA_VERSION=1). ✅ Step 1 now splits genuinely-RED (v1→2 load + round-trip + addTrend-persist + itemToInput) from GREEN-only regression guards (v2-idempotence, missing/non-numeric/empty); the RED proof is claimed only for the former.
• [MAJOR] capture tally mis-maps onto AddResult (no duplicates field, store.ts:35-41). ✅ Step 4 pins the exact derivation (added/merged/duplicates from res.added+res.merged) + a sum === payload.length test.
• [MAJOR] TRENDS_TESTS_FLOOR breakdown comment (:695) left stale. ✅ Step 6 now extends the inline breakdown comment alongside the number.
• [MAJOR] import isValidIso impossible (item.ts:51 private). ✅ add --published-at deferred entirely (brief §4) — no export-vs-inline decision, item.ts edit stays itemToInput-only.
• [MAJOR] README add-as-capture framing would contradict the re-point. ✅ Step 5 corrects the framing (add = manual single-trend; capture = normalizing batch), not just appends.
• [MAJOR] empty "folded" placeholders. ✅ this section + brief §9 filled.
• [MINOR] stub cleanup unstated ✅ Step 3 states the throwing stub is replaced, none survives into GREEN. [MINOR] today() capturedAt untestable in cli.test ✅ Step 4 routes exact-value assertions to item.test.ts, cli.test asserts shape + ≠ publishedAt. [MINOR] 16h grep flag ✅ Step 6 specifies grep -qF. [MINOR] non-numeric schemaVersion untested ✅ added to Step 1 regression guards + SC3. [MINOR] README under [skip-docs] ✅ kept in the code commit (it documents the shipped code, like R1's trend-spotter.md); noted.
• [brief-reviewer MAJOR] capturedAt injection unverified at the new ingress ✅ SC1 + SC4 field-confusion guards. [brief-reviewer MAJOR] "lossless" over-claim ✅ §5/§3 scope it to well-formed stores; malformed-trends coercion unchanged + out of scope. [brief-reviewer MINOR] absent→present back-fill ✅ pinned by an SC2 test.
• [plan-critic headless-readiness 60] N/A — R2a is executed in-session, operator-driven (driftsmodell), not as a headless autonomous run, so per-step revert/halt clauses aren't needed (R1's plan had none either).

scope-guardian — ALIGNED: every SC1–SC6 traces to a step; zero creep (README is in-change documentation); every §4 non-goal held (no hook touch, no brief artifact, no relevance/saturation/status, no back-fill, no scoring change).