linkedin-studio/docs/research-engine/brief-re-r2a.md

Brief — RE-R2a: capture bridge (item→store) + publishedAt persistence

Slice: RE-R2a (research-engine rung-2, slice 2 — the data layer of R2). The first of the two halves R2 ("the visible topic-stream") was split into at the 2026-06-24 go-gate: R2a = the pure scripts/trends/ data layer (this brief); R2b = the dated morning-brief artifact + session-start surfacing (separate brief, after R2a lands). The split was chosen because (b)+(c) are code-independent of (a), R2a mirrors R1's pure-TDD shape exactly (lowest risk), and it closes the capture loop the store was built for. Predecessor: RE-R1 (brief-re-r1.md) delivered the validated ingress envelope (item.ts: normalizeItem/normalizeItems) and the triage scorer (score.ts) behind a CLI seam, and explicitly deferred the item→store bridge ("injecting the store's capturedAt and persisting publishedAt — is R2 orchestration", brief-re-r1.md §3 B1). R2a builds exactly that deferred bridge. Substrate: docs/research-engine-concepts.local.md §1 (hull 3 "store-schema mangler felt" / the item→store gap), §3 B1 (one schema downstream never branches on). TDD-order: RED tests land before code.

1. Operator decision context (2026-06-24)

The research engine is Tier-1 (operator, 2026-06-23): "hele min arbeidsflyt hviler på at jeg får en jevn strøm av gode forslag til tema å skrive om." At the R2 go-gate (2026-06-24) the operator chose to build R2 as two sequenced slices, foundation-first: R2a (this — the bridge + schema, pure trends/) before R2b (the visible brief + surfacing). R2a delivers no visible change; it closes the capture loop so the store accumulates publish-dated history that R2b's brief and R3's freshness window will rank on.

2. The gap — grounded in code

• normalize (R1) and the store are disconnected. item.ts validates a raw source item into the canonical TrendItem envelope {source,title,url,publishedAt?,topics[],summary?} (item.ts:21-38), but nothing folds that envelope into the store. The only ingress to the store is cli.ts add (cli.ts:111-135), which builds a TrendInput from raw flags — it never calls normalizeItem, so a capture path that polls → normalizes → stores does not exist. The bridge is the missing link R1 named and deferred.
• publishedAt is carried but dropped. The envelope carries the source's own publish date (item.ts:28-33, "NOT persisted in R1"), but TrendRecord (types.ts:26-41) has no publishedAt field and addTrend (store.ts:111-130) never persists it. So every capture loses the publish date — the field B4 freshness (R3) and R2b's "fresh today" sort will both need. R2a persists it now, before history accumulates undated, so the loop is publish-dated from this slice forward.
• SCHEMA_VERSION is pinned at 1 with no migration path exercised. loadStore (store.ts:74-81) reads parsed.schemaVersion ?? SCHEMA_VERSION but the store has never had a second version, so the forward-migration path is untested. Adding publishedAt is the first additive-optional schema bump (v1→v2) and the right moment to prove a lossless migrate-on-load.

3. Scope — what is IN (RE-R2a)

B-bridge — itemToInput (pure mapping, scripts/trends/src/item.ts, EDIT)

A pure itemToInput(item: TrendItem, capturedAt: string): TrendInput that maps the validated envelope to a store input by injecting capturedAt (the store's "when WE saw it", supplied by the caller — never derived inside the pure function) and carrying publishedAt/topics/summary/title/url/source through verbatim. It does not re-validate (the item is already validated by normalizeItem) and does not derive an id (the store owns id via addTrend→trendId). Lives in item.ts (it is the item's mapping to the store; item.ts already imports from store.ts) — not a new single-function bridge.ts (single-use abstraction).

Schema migration — publishedAt persisted (types.ts + store.ts, EDIT)

• types.ts: SCHEMA_VERSION 1 → 2; add publishedAt?: string to TrendRecord (after capturedAt), with a comment distinguishing it from capturedAt (source publish-date; forward-compat for B4 freshness).
• store.ts: TrendInput gains publishedAt?: string; addTrend persists it when present (same conditional-spread idiom as summary, store.ts:126). On re-capture/merge it is NOT overwritten — it joins the store's documented "FIRST sighting's source + capturedAt are kept (provenance of first sight)" rule (store.ts:108-110); no back-fill (a deferred enhancement, §4).
• loadStore: forward migrate-on-load — stamp the in-memory store to Math.max(onDisk, SCHEMA_VERSION), where onDisk = typeof parsed.schemaVersion === "number" ? parsed.schemaVersion : SCHEMA_VERSION (a string / NaN / absent version falls back to current, never crashes). v1→v2 is purely additive-optional (old records are already valid v2 records that simply lack the optional field), so the migration is the stamp alone — no record rewrite; lossless + idempotent for any well-formed v1/v2 store. A subsequent saveStore persists the v2 stamp. The existing non-array trends coercion (store.ts:79, Array.isArray(parsed.trends) ? … : []) is UNCHANGED and out of R2a's migration scope — losslessness is claimed only for well-formed stores, not for a corrupt trends field. (No downgrade branch: a hypothetical future on-disk > current can't happen yet — Karpathy #2, no handling for impossible scenarios; noted as a deferred risk.)

CLI capture subcommand (scripts/trends/src/cli.ts, EDIT)

echo '<raw item | batch>' | node --import tsx src/cli.ts capture [--store <path>] [--json] — reads the raw JSON payload from stdin (same stdin contract as normalize/score, so --json stays an output toggle), runs it through normalizeItem/normalizeItems, maps each valid item via itemToInput(item, today()), folds each into the store via addTrend, and saveStores once. Emits a summary — {added, duplicates, merged, errors}

• store path/count — human-readable by default, JSON with --json. Content-invalid items are reported in the summary (errors[]), never via the exit code; exit 2 only on a malformed invocation (unparseable/empty stdin), matching normalize/score. Note the contract difference from add: capture items must carry source (the normalizer requires it — no "manual" default), because a capture is from a real source.

Wiring + gate (see Open Question #1 — included by default, trimmable to minimal)

• agents/trend-spotter.md (EDIT): re-point Step 4.5 (trend-spotter.md:282-301) from N× flag-based add to a single batch capture — the agent builds a raw-item JSON batch (it already builds JSON for score) and pipes it to capture in one call. Strictly better than add (it normalizes + carries publishedAt; one call, not N). The replacement prose carries the literal src/cli.ts capture. Kept domain-general (no vendor/sector tokens).
• scripts/test-runner.sh (EDIT): bump TRENDS_TESTS_FLOOR (62 → live recount, stays inside the deps guard). Add Section 16h ("Trends Capture Wiring"), mirroring Section 16g's three-check shape, placed after Section 17 / before Section 18 (anti-erosion last): (1) a non-vacuity self-test; (2) grep that cli.ts has a capture handler (deps-absent proof the path exists); (3) grep -qF "src/cli.ts capture" agents/trend-spotter.md. These are unconditional → bump ASSERT_BASELINE_FLOOR 87 → live recount (expected ~90).

4. Non-goals — what is OUT (deferred)

• The dated morning-brief artifact (B3) — R2b. R2a closes the capture loop; R2b makes the stream visible.
• Session-start surfacing of the brief (hull 4) — R2b. R2a does not touch hooks/**.
• publishedAt back-fill on re-capture (fill an absent existing publishedAt from a later sighting) — deferred; first-sight provenance is kept, matching the existing merge rule. Revisit if undated-first-sight proves common.
• add --published-at flag — deferred (was a proposed "minor"; folded out at light-Voyage). The flag-based add is the manual single-trend path; it stays publish-date-free for now. The store layer still GAINS publishedAt (on TrendInput + addTrend) — that is what capture needs — but exposing it on the add CLI (and the isValidIso-export-vs-inline decision it would force) is out of R2a. Trivial to add any later slice.
• relevance / saturation / status / lifecycle fields (hull 5) — R3. R2a adds only publishedAt.
• Freshness window / dedup-vs-seen-log / autonomous trigger (B4) — R3.
• Research-deepening (A1–A4), adapter sub-agents, MCP fetch fan-out — R2b/R3.
• A store-reading brief / ranking on accumulated relevance — R2b/R3 (needs persisted scores).

5. Boundaries / invariants (must hold)

• TDD iron law: the failing migration / itemToInput / capture tests land BEFORE the implementation. RED proofs recorded (logic-RED, not import-RED). NB: only the v1→v2 load + round-trip are genuinely RED (old loadStore returns 1); v2-idempotence + missing/non-numeric-schemaVersion pass against old code, so they are GREEN-only regression guards, not RED cases (see plan Step 1).
• Lossless migration (well-formed stores): every existing well-formed v1 store loads as v2 with records intact (no publishedAt invented, no topic/summary/capturedAt/url/title change); idempotent (v2 → v2). A corrupt trends field is coerced by the existing (unchanged) Array.isArray guard — explicitly out of R2a's losslessness claim, not a regression introduced here.
• First-sight provenance preserved: re-capturing an existing trend never overwrites its publishedAt, capturedAt, or source — only topics union (unchanged from R1).
• No scoring change: score.ts and references/trend-scoring-modes.md (the SSOT) are untouched — R2a is the data layer, not the scorer.
• Domain-general: de-niche guard (Section 17) stays green; no vendor/sector tokens enter the edited trend-spotter.md prose.
• House conventions: ESM + node:test + tsx; data-seam stays inline; no new .mjs under hooks/scripts/ (R2a touches no hook); .gitignore already covers scripts/trends/{node_modules,build}.
• No new references/*.md, no new agent/command (counts stay 19/29/27); no new .ts source file (bridge lives in item.ts). Brief+plan live under docs/ (uncounted), TRACKED like docs/second-brain/*.
• Counts recounted live at land, never pinned/guessed.

6. Success criteria (testable)

• SC1 (bridge) — itemToInput(item, "2026-06-24") returns a TrendInput with capturedAt injected (=== "2026-06-24"), publishedAt/topics/summary/title/url/source carried verbatim, and no id. Absent publishedAt on the item → absent on the input (key omitted, not undefined-valued). Field-confusion guard: for an item whose publishedAt differs from the injected capturedAt, the result's capturedAt !== result.publishedAt (proves the bridge never confuses the two dates — the whole point of the slice).
• SC2 (persist) — addTrend with a publishedAt persists it on the new record; a record without it omits the key. On a re-capture (same title+url) the existing publishedAt is unchanged (first-sight kept), and only topics union — merged reflects topic change alone. No back-fill: a re-capture carrying a publishedAt onto a record that lacked one does not add it (Open Q#2) — the absent→present case is the one where "first-sight kept" is counterintuitive, so it is pinned by a test.
• SC3 (migrate) — loadStore on a v1 store fixture ({schemaVersion:1, trends:[…without publishedAt]}) returns schemaVersion === 2 with every record intact and no publishedAt invented. Also pinned: a store with missing schemaVersion → stamped 2 (records intact); an empty/absent store → {schemaVersion:2, trends:[]}; a non-numeric schemaVersion ("weird"/NaN) → falls back to 2 (records intact); idempotent on a v2 store; a round-trip loadStore→saveStore writes schemaVersion: 2.
• SC4 (capture CLI) — echo '<batch>' | … capture normalizes + folds: a well-formed item is added (or reported duplicate/merged), a content-invalid item appears in the summary errors[], the store file is written once, and the summary counts are correct — added + merged + duplicates + errors.length === payload.length (the tally is derived from AddResult {added, merged}, which has no duplicates field: added=res.added, merged=!res.added && res.merged, duplicates=!res.added && !res.merged). The captured record's capturedAt is a today-shaped ISO date distinct from the item's publishedAt (field-confusion guard at the ingress; the exact-value assertion lives in item.test.ts with an injected date, since capture reads the wall clock). Exit 2 on empty/unparseable stdin; 0 on a well-formed call even with content-invalid items. --json emits the summary object.
• SC5 (gate) — (cd scripts/trends && npm test) green at the bumped TRENDS_TESTS_FLOOR; new Section 16h green; ASSERT_BASELINE_FLOOR bumped to the live recount; overall gate FAIL=0.
• SC6 (wiring + de-niche) — trend-spotter.md references src/cli.ts capture (Section 16h grep green); de-niche guard (Section 17) green; structure counts unchanged (19/29/27). (If Open Q#1 → minimal: SC6 drops the wiring/16h clauses; de-niche + counts still asserted.)

7. Verification

Deterministic (gate): bash scripts/test-runner.sh → FAIL=0; trends suite ≥ new floor; new Section 16h self-test + greps pass; Section 17 de-niche green; ref/agent/command counts unchanged. Behavioural (manual): echo '<2-item batch incl. one publishedAt + one invalid>' | node --import tsx src/cli.ts capture --store /tmp/r2a-trends.json --json; confirm the valid item lands with publishedAt persisted, the invalid one is in errors[], and node … list --store /tmp/r2a-trends.json --json shows the persisted publishedAt. Then load an authored v1 fixture and confirm status --json reports it migrated (count intact).

8. Open questions for the go-gate

1. Wire trend-spotter.md + add Section 16h, or keep R2a minimal? — RESOLVED at go-gate (operator, 2026-06-24): WIRE. Build R2a with the agent re-point (Step 4.5 add→capture) + Section 16h grep/self-test + ASSERT-floor bump. The minimal alternative (bridge code + tests only, no agent edit) was declined — wiring mirrors R1's "the lift is real and grep-able" discipline and makes the bridge actually used (one normalizing capture vs N× add).
2. publishedAt merge policy. Proposed keep first-sight (no back-fill), matching the existing source/capturedAt provenance rule (now pinned by the absent→present SC2 test). Confirm, or request back-fill-if-absent (adds a branch
   • a merged semantic question).
3. add --published-at flag — RESOLVED at light-Voyage: deferred (see §4). The manual add stays publish-date-free; capture is the path that carries publishedAt. Noted here only so the resolution is traceable; no go-gate action.

9. Light-Voyage review — folded

Three Opus reviewers ran on the drafts, each verifying claims against live code. scope-guardian: ALIGNED (every SC1–SC6 traces to a step; zero creep; all §4 non-goals held; 2 minors). brief-reviewer: PROCEED_WITH_RISKS (2 majors, 4 minors). plan-critic: REVISE (1 blocker, 5 majors, 5 minors). All findings folded; see plan-re-r2a.md §Plan-critic — folded for the per-step resolution:

• [BLOCKER, folded] "v2 load idempotent" cannot be a RED case — old loadStore (store.ts:78) returns parsed.schemaVersion ?? SCHEMA_VERSION, so a v2 fixture already loads as 2 against unchanged code. → §6 SC3 + §5 now split the migration cases: only v1→2 + round-trip are RED; v2-idempotence + missing/non-numeric are GREEN-only regression guards.
• [MAJOR, folded] "lossless / byte-for-byte intact" over-claimed vs loadStore's non-array trends coercion (store.ts:79). → §5 + §3 scope losslessness to well-formed v1/v2 stores; the coercion is unchanged + out of scope (not a regression).
• [MAJOR, folded] capturedAt injection unverified at the new capture ingress (the field-confusion the slice exists to prevent). → SC1 + SC4 now pin capturedAt !== publishedAt (item.test exact-value; cli.test shape + distinct).
• [MAJOR, folded] capture tally {added, duplicates, merged} mis-maps onto AddResult {added, merged} (no duplicates field). → SC4 pins the exact derivation + sum === payload.length.
• [MAJOR, folded] add --published-at validation referenced the non-exported isValidIso (item.ts:51). → add --published-at deferred entirely (§4), removing the export-vs-inline decision and keeping item.ts's edit to itemToInput only.
• [MAJOR, folded] README "Capture … add" framing would contradict the agent's add→capture re-point. → plan Step 5 now corrects the README framing (add = manual single-trend; capture = normalizing batch), not just appends.
• [MAJOR, folded] empty "folded" placeholders shipped in the doc bodies. → this section + plan §Plan-critic now filled.
• [MINOR, folded in plan] missing-/non-numeric-schemaVersion test, absent→present back-fill test, stub-replacement note, import type { TrendInput } line, grep -qF for the 16h literal, TRENDS_TESTS_FLOOR breakdown-comment update, README in the code commit, headless per-step clauses N/A (in-session execution).