docs(linkedin-studio): RE-R2a brief + plan — item→store capture bridge + publishedAt persistence (light-Voyage hardened, go-gate: wire)
R2 ("the visible topic-stream") split into two sequenced slices at the
2026-06-24 go-gate, foundation-first: R2a (this — the pure scripts/trends/
data layer) before R2b (dated morning-brief + session-start surfacing).
R2a builds the item→store bridge RE-R1 deferred: a pure itemToInput mapping
the validated TrendItem envelope to a store input (injecting capturedAt,
carrying publishedAt), a publishedAt schema bump (SCHEMA_VERSION 1→2) with a
lossless forward migrate-on-load, and a `capture` CLI closing the
poll→normalize→store loop. No scoring change, no hook touch.
Go-gate: WIRE (operator) — re-point trend-spotter Step 4.5 add→capture +
Section 16h grep/self-test. Q2 publishedAt-merge = first-sight, no back-fill.
Light-Voyage hardened: scope-guardian ALIGNED, brief-reviewer
PROCEED_WITH_RISKS, plan-critic REVISE — all folded (BLOCKER: v2-idempotence
is GREEN-only not RED; capture tally mapped exactly onto AddResult; "lossless"
scoped to well-formed stores; add --published-at deferred; README framing fix).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VmHCQjJHUyWwxGAVVjNLgp
This commit is contained in:
parent
24775f4493
commit
b4e500fad4
2 changed files with 400 additions and 0 deletions
208
docs/research-engine/brief-re-r2a.md
Normal file
208
docs/research-engine/brief-re-r2a.md
Normal file
|
|
@ -0,0 +1,208 @@
|
|||
# Brief — RE-R2a: capture bridge (item→store) + publishedAt persistence
|
||||
|
||||
> **Slice:** RE-R2a (research-engine rung-2, slice 2 — the data layer of R2). The first of the two
|
||||
> halves R2 ("the visible topic-stream") was split into at the 2026-06-24 go-gate: **R2a** = the pure
|
||||
> `scripts/trends/` data layer (this brief); **R2b** = the dated morning-brief artifact + session-start
|
||||
> surfacing (separate brief, after R2a lands). The split was chosen because (b)+(c) are code-independent
|
||||
> of (a), R2a mirrors R1's pure-TDD shape exactly (lowest risk), and it closes the capture loop the store
|
||||
> was built for.
|
||||
> **Predecessor:** RE-R1 (`brief-re-r1.md`) delivered the validated ingress envelope (`item.ts`:
|
||||
> `normalizeItem`/`normalizeItems`) and the triage scorer (`score.ts`) behind a CLI seam, and **explicitly
|
||||
> deferred the item→store bridge** ("injecting the store's `capturedAt` and persisting `publishedAt` — is
|
||||
> R2 orchestration", `brief-re-r1.md` §3 B1). R2a builds exactly that deferred bridge.
|
||||
> **Substrate:** `docs/research-engine-concepts.local.md` §1 (hull 3 "store-schema mangler felt" / the
|
||||
> item→store gap), §3 B1 (one schema downstream never branches on). **TDD-order:** RED tests land before code.
|
||||
|
||||
## 1. Operator decision context (2026-06-24)
|
||||
|
||||
The research engine is **Tier-1** (operator, 2026-06-23): *"hele min arbeidsflyt hviler på at jeg får en
|
||||
jevn strøm av gode forslag til tema å skrive om."* At the R2 go-gate (2026-06-24) the operator chose to
|
||||
build R2 as **two sequenced slices, foundation-first**: R2a (this — the bridge + schema, pure trends/) before
|
||||
R2b (the visible brief + surfacing). R2a delivers no *visible* change; it closes the capture loop so the
|
||||
store accumulates publish-dated history that R2b's brief and R3's freshness window will rank on.
|
||||
|
||||
## 2. The gap — grounded in code
|
||||
|
||||
- **`normalize` (R1) and the store are disconnected.** `item.ts` validates a raw source item into the
|
||||
canonical `TrendItem` envelope `{source,title,url,publishedAt?,topics[],summary?}` (`item.ts:21-38`), but
|
||||
**nothing folds that envelope into the store.** The only ingress to the store is `cli.ts add` (`cli.ts:111-135`),
|
||||
which builds a `TrendInput` from raw flags — it never calls `normalizeItem`, so a capture path that polls →
|
||||
normalizes → stores **does not exist**. The bridge is the missing link R1 named and deferred.
|
||||
- **`publishedAt` is carried but dropped.** The envelope carries the source's own publish date
|
||||
(`item.ts:28-33`, "NOT persisted in R1"), but `TrendRecord` (`types.ts:26-41`) has no `publishedAt` field and
|
||||
`addTrend` (`store.ts:111-130`) never persists it. So every capture **loses** the publish date — the field
|
||||
B4 freshness (R3) and R2b's "fresh today" sort will both need. R2a persists it now, before history accumulates
|
||||
undated, so the loop is publish-dated from this slice forward.
|
||||
- **`SCHEMA_VERSION` is pinned at 1 with no migration path exercised.** `loadStore` (`store.ts:74-81`) reads
|
||||
`parsed.schemaVersion ?? SCHEMA_VERSION` but the store has never had a second version, so the forward-migration
|
||||
path is untested. Adding `publishedAt` is the first additive-optional schema bump (v1→v2) and the right moment
|
||||
to prove a lossless migrate-on-load.
|
||||
|
||||
## 3. Scope — what is IN (RE-R2a)
|
||||
|
||||
### B-bridge — `itemToInput` (pure mapping, `scripts/trends/src/item.ts`, EDIT)
|
||||
A pure `itemToInput(item: TrendItem, capturedAt: string): TrendInput` that maps the validated envelope to a
|
||||
store input by **injecting `capturedAt`** (the store's "when WE saw it", supplied by the caller — never derived
|
||||
inside the pure function) and **carrying `publishedAt`/`topics`/`summary`/`title`/`url`/`source` through verbatim**.
|
||||
It does **not** re-validate (the item is already validated by `normalizeItem`) and does **not** derive an `id`
|
||||
(the store owns id via `addTrend`→`trendId`). Lives in `item.ts` (it is the item's mapping to the store; `item.ts`
|
||||
already imports from `store.ts`) — **not** a new single-function `bridge.ts` (single-use abstraction).
|
||||
|
||||
### Schema migration — `publishedAt` persisted (`types.ts` + `store.ts`, EDIT)
|
||||
- `types.ts`: `SCHEMA_VERSION 1 → 2`; add `publishedAt?: string` to `TrendRecord` (after `capturedAt`), with a
|
||||
comment distinguishing it from `capturedAt` (source publish-date; forward-compat for B4 freshness).
|
||||
- `store.ts`: `TrendInput` gains `publishedAt?: string`; `addTrend` persists it when present (same
|
||||
conditional-spread idiom as `summary`, `store.ts:126`). **On re-capture/merge it is NOT overwritten** —
|
||||
it joins the store's documented "FIRST sighting's source + capturedAt are kept (provenance of first sight)"
|
||||
rule (`store.ts:108-110`); no back-fill (a deferred enhancement, §4).
|
||||
- `loadStore`: **forward migrate-on-load** — stamp the in-memory store to `Math.max(onDisk, SCHEMA_VERSION)`,
|
||||
where `onDisk = typeof parsed.schemaVersion === "number" ? parsed.schemaVersion : SCHEMA_VERSION` (a string /
|
||||
`NaN` / absent version falls back to current, never crashes). v1→v2 is purely additive-optional (old records are
|
||||
already valid v2 records that simply lack the optional field), so the migration is the stamp alone — **no record
|
||||
rewrite; lossless + idempotent for any well-formed v1/v2 store**. A subsequent `saveStore` persists the v2 stamp.
|
||||
**The existing non-array `trends` coercion (`store.ts:79`, `Array.isArray(parsed.trends) ? … : []`) is UNCHANGED
|
||||
and out of R2a's migration scope** — losslessness is claimed only for well-formed stores, not for a corrupt
|
||||
`trends` field. (No downgrade branch: a hypothetical future on-disk > current can't happen yet — Karpathy #2, no
|
||||
handling for impossible scenarios; noted as a deferred risk.)
|
||||
|
||||
### CLI `capture` subcommand (`scripts/trends/src/cli.ts`, EDIT)
|
||||
`echo '<raw item | batch>' | node --import tsx src/cli.ts capture [--store <path>] [--json]` — reads the raw
|
||||
JSON payload **from stdin** (same stdin contract as `normalize`/`score`, so `--json` stays an *output* toggle),
|
||||
runs it through `normalizeItem`/`normalizeItems`, maps each valid item via `itemToInput(item, today())`, folds
|
||||
each into the store via `addTrend`, and `saveStore`s once. Emits a summary — `{added, duplicates, merged, errors}`
|
||||
+ store path/count — human-readable by default, JSON with `--json`. Content-invalid items are reported in the
|
||||
summary (`errors[]`), **never via the exit code**; exit **2** only on a malformed invocation (unparseable/empty
|
||||
stdin), matching `normalize`/`score`. Note the contract difference from `add`: `capture` items must carry `source`
|
||||
(the normalizer requires it — no "manual" default), because a capture is from a real source.
|
||||
|
||||
### Wiring + gate (see Open Question #1 — included by default, trimmable to minimal)
|
||||
- `agents/trend-spotter.md` (EDIT): re-point **Step 4.5** (`trend-spotter.md:282-301`) from N× flag-based `add` to
|
||||
a single batch `capture` — the agent builds a raw-item JSON batch (it already builds JSON for `score`) and pipes
|
||||
it to `capture` in one call. Strictly better than `add` (it normalizes + carries `publishedAt`; one call, not N).
|
||||
The replacement prose carries the literal `src/cli.ts capture`. Kept domain-general (no vendor/sector tokens).
|
||||
- `scripts/test-runner.sh` (EDIT): bump `TRENDS_TESTS_FLOOR` (62 → **live recount**, stays inside the deps guard).
|
||||
Add **Section 16h** ("Trends Capture Wiring"), mirroring Section 16g's three-check shape, placed **after Section
|
||||
17 / before Section 18** (anti-erosion last): (1) a non-vacuity self-test; (2) `grep` that `cli.ts` has a `capture`
|
||||
handler (deps-absent proof the path exists); (3) `grep -qF "src/cli.ts capture" agents/trend-spotter.md`. These are
|
||||
**unconditional** → bump `ASSERT_BASELINE_FLOOR` 87 → **live recount** (expected ~90).
|
||||
|
||||
## 4. Non-goals — what is OUT (deferred)
|
||||
|
||||
- **The dated morning-brief artifact (B3)** — R2b. *R2a closes the capture loop; R2b makes the stream visible.*
|
||||
- **Session-start surfacing** of the brief (hull 4) — R2b. R2a does **not** touch `hooks/**`.
|
||||
- **`publishedAt` back-fill on re-capture** (fill an absent existing `publishedAt` from a later sighting) — deferred;
|
||||
first-sight provenance is kept, matching the existing merge rule. Revisit if undated-first-sight proves common.
|
||||
- **`add --published-at` flag** — deferred (was a proposed "minor"; folded out at light-Voyage). The flag-based
|
||||
`add` is the *manual* single-trend path; it stays publish-date-free for now. The store layer still GAINS
|
||||
`publishedAt` (on `TrendInput` + `addTrend`) — that is what `capture` needs — but exposing it on the `add` CLI
|
||||
(and the `isValidIso`-export-vs-inline decision it would force) is out of R2a. Trivial to add any later slice.
|
||||
- **`relevance` / `saturation` / `status` / lifecycle fields** (hull 5) — R3. R2a adds **only** `publishedAt`.
|
||||
- **Freshness window / dedup-vs-seen-log / autonomous trigger (B4)** — R3.
|
||||
- **Research-deepening (A1–A4), adapter sub-agents, MCP fetch fan-out** — R2b/R3.
|
||||
- **A store-reading brief / ranking on accumulated relevance** — R2b/R3 (needs persisted scores).
|
||||
|
||||
## 5. Boundaries / invariants (must hold)
|
||||
|
||||
- **TDD iron law:** the failing migration / `itemToInput` / `capture` tests land BEFORE the implementation.
|
||||
RED proofs recorded (logic-RED, not import-RED). **NB:** only the **v1→v2** load + round-trip are genuinely RED
|
||||
(old `loadStore` returns 1); v2-idempotence + missing/non-numeric-`schemaVersion` pass against old code, so they
|
||||
are **GREEN-only regression guards**, not RED cases (see plan Step 1).
|
||||
- **Lossless migration (well-formed stores):** every existing **well-formed** v1 store loads as v2 with **records
|
||||
intact** (no `publishedAt` invented, no topic/summary/capturedAt/url/title change); idempotent (v2 → v2). A
|
||||
corrupt `trends` field is coerced by the existing (unchanged) `Array.isArray` guard — explicitly **out of R2a's
|
||||
losslessness claim**, not a regression introduced here.
|
||||
- **First-sight provenance preserved:** re-capturing an existing trend never overwrites its `publishedAt`,
|
||||
`capturedAt`, or `source` — only topics union (unchanged from R1).
|
||||
- **No scoring change:** `score.ts` and `references/trend-scoring-modes.md` (the SSOT) are untouched — R2a is the
|
||||
data layer, not the scorer.
|
||||
- **Domain-general:** de-niche guard (Section 17) stays green; no vendor/sector tokens enter the edited
|
||||
`trend-spotter.md` prose.
|
||||
- **House conventions:** ESM + `node:test` + `tsx`; data-seam stays inline; **no new `.mjs` under `hooks/scripts/`**
|
||||
(R2a touches no hook); `.gitignore` already covers `scripts/trends/{node_modules,build}`.
|
||||
- **No new `references/*.md`**, **no new agent/command** (counts stay 19/29/27); no new `.ts` source file
|
||||
(bridge lives in `item.ts`). Brief+plan live under `docs/` (uncounted), TRACKED like `docs/second-brain/*`.
|
||||
- **Counts recounted live at land**, never pinned/guessed.
|
||||
|
||||
## 6. Success criteria (testable)
|
||||
|
||||
- **SC1 (bridge)** — `itemToInput(item, "2026-06-24")` returns a `TrendInput` with `capturedAt` injected (`=== "2026-06-24"`),
|
||||
`publishedAt`/`topics`/`summary`/`title`/`url`/`source` carried verbatim, and **no `id`**. Absent `publishedAt`
|
||||
on the item → absent on the input (key omitted, not `undefined`-valued). **Field-confusion guard:** for an item
|
||||
whose `publishedAt` differs from the injected `capturedAt`, the result's `capturedAt !== result.publishedAt`
|
||||
(proves the bridge never confuses the two dates — the whole point of the slice).
|
||||
- **SC2 (persist)** — `addTrend` with a `publishedAt` persists it on the new record; a record without it omits the
|
||||
key. On a re-capture (same title+url) the existing `publishedAt` is **unchanged** (first-sight kept), and only
|
||||
topics union — `merged` reflects topic change alone. **No back-fill:** a re-capture carrying a `publishedAt` onto
|
||||
a record that **lacked** one does **not** add it (Open Q#2) — the absent→present case is the one where "first-sight
|
||||
kept" is counterintuitive, so it is pinned by a test.
|
||||
- **SC3 (migrate)** — `loadStore` on a v1 store fixture (`{schemaVersion:1, trends:[…without publishedAt]}`) returns
|
||||
`schemaVersion === 2` with every record intact and **no `publishedAt` invented**. Also pinned: a store with
|
||||
**missing** `schemaVersion` → stamped 2 (records intact); an **empty/absent** store → `{schemaVersion:2, trends:[]}`;
|
||||
a **non-numeric** `schemaVersion` (`"weird"`/`NaN`) → falls back to 2 (records intact); idempotent on a v2 store;
|
||||
a round-trip `loadStore`→`saveStore` writes `schemaVersion: 2`.
|
||||
- **SC4 (capture CLI)** — `echo '<batch>' | … capture` normalizes + folds: a well-formed item is added (or reported
|
||||
duplicate/merged), a content-invalid item appears in the summary `errors[]`, the store file is written once, and
|
||||
the summary counts are correct — **`added + merged + duplicates + errors.length === payload.length`** (the tally
|
||||
is derived from `AddResult {added, merged}`, which has no `duplicates` field: `added`=`res.added`,
|
||||
`merged`=`!res.added && res.merged`, `duplicates`=`!res.added && !res.merged`). The captured record's `capturedAt`
|
||||
is a today-shaped ISO date **distinct from the item's `publishedAt`** (field-confusion guard at the ingress; the
|
||||
exact-value assertion lives in `item.test.ts` with an injected date, since `capture` reads the wall clock). Exit
|
||||
**2** on empty/unparseable stdin; **0** on a well-formed call even with content-invalid items. `--json` emits the
|
||||
summary object.
|
||||
- **SC5 (gate)** — `(cd scripts/trends && npm test)` green at the bumped `TRENDS_TESTS_FLOOR`; new Section 16h green;
|
||||
`ASSERT_BASELINE_FLOOR` bumped to the live recount; overall gate `FAIL=0`.
|
||||
- **SC6 (wiring + de-niche)** — `trend-spotter.md` references `src/cli.ts capture` (Section 16h grep green); de-niche
|
||||
guard (Section 17) green; structure counts unchanged (19/29/27). *(If Open Q#1 → minimal: SC6 drops the wiring/16h
|
||||
clauses; de-niche + counts still asserted.)*
|
||||
|
||||
## 7. Verification
|
||||
|
||||
**Deterministic (gate):** `bash scripts/test-runner.sh` → `FAIL=0`; trends suite ≥ new floor; new Section 16h
|
||||
self-test + greps pass; Section 17 de-niche green; ref/agent/command counts unchanged.
|
||||
**Behavioural (manual):** `echo '<2-item batch incl. one publishedAt + one invalid>' | node --import tsx src/cli.ts
|
||||
capture --store /tmp/r2a-trends.json --json`; confirm the valid item lands with `publishedAt` persisted, the invalid
|
||||
one is in `errors[]`, and `node … list --store /tmp/r2a-trends.json --json` shows the persisted `publishedAt`.
|
||||
Then load an authored v1 fixture and confirm `status --json` reports it migrated (count intact).
|
||||
|
||||
## 8. Open questions for the go-gate
|
||||
|
||||
1. **Wire `trend-spotter.md` + add Section 16h, or keep R2a minimal? — RESOLVED at go-gate (operator, 2026-06-24): WIRE.**
|
||||
Build R2a with the agent re-point (Step 4.5 `add`→`capture`) + Section 16h grep/self-test + ASSERT-floor bump. The
|
||||
minimal alternative (bridge code + tests only, no agent edit) was declined — wiring mirrors R1's "the lift is real
|
||||
and grep-able" discipline and makes the bridge actually used (one normalizing `capture` vs N× `add`).
|
||||
2. **`publishedAt` merge policy.** Proposed **keep first-sight** (no back-fill), matching the existing source/capturedAt
|
||||
provenance rule (now pinned by the absent→present SC2 test). Confirm, or request back-fill-if-absent (adds a branch
|
||||
+ a `merged` semantic question).
|
||||
3. **`add --published-at` flag — RESOLVED at light-Voyage: deferred** (see §4). The manual `add` stays publish-date-free;
|
||||
`capture` is the path that carries `publishedAt`. Noted here only so the resolution is traceable; no go-gate action.
|
||||
|
||||
## 9. Light-Voyage review — folded
|
||||
|
||||
Three Opus reviewers ran on the drafts, each verifying claims against live code. **scope-guardian: ALIGNED**
|
||||
(every SC1–SC6 traces to a step; zero creep; all §4 non-goals held; 2 minors). **brief-reviewer:
|
||||
PROCEED_WITH_RISKS** (2 majors, 4 minors). **plan-critic: REVISE** (1 blocker, 5 majors, 5 minors). All findings
|
||||
folded; see `plan-re-r2a.md` §Plan-critic — folded for the per-step resolution:
|
||||
|
||||
- **[BLOCKER, folded]** "v2 load idempotent" cannot be a RED case — old `loadStore` (`store.ts:78`) returns
|
||||
`parsed.schemaVersion ?? SCHEMA_VERSION`, so a v2 fixture already loads as 2 against unchanged code. → §6 SC3 +
|
||||
§5 now split the migration cases: **only v1→2 + round-trip are RED**; v2-idempotence + missing/non-numeric are
|
||||
**GREEN-only regression guards**.
|
||||
- **[MAJOR, folded]** "lossless / byte-for-byte intact" over-claimed vs `loadStore`'s non-array `trends` coercion
|
||||
(`store.ts:79`). → §5 + §3 scope losslessness to **well-formed** v1/v2 stores; the coercion is unchanged + out of
|
||||
scope (not a regression).
|
||||
- **[MAJOR, folded]** `capturedAt` injection unverified at the new `capture` ingress (the field-confusion the slice
|
||||
exists to prevent). → SC1 + SC4 now pin `capturedAt !== publishedAt` (item.test exact-value; cli.test shape +
|
||||
distinct).
|
||||
- **[MAJOR, folded]** `capture` tally `{added, duplicates, merged}` mis-maps onto `AddResult {added, merged}` (no
|
||||
`duplicates` field). → SC4 pins the exact derivation + `sum === payload.length`.
|
||||
- **[MAJOR, folded]** `add --published-at` validation referenced the non-exported `isValidIso` (`item.ts:51`). →
|
||||
**`add --published-at` deferred entirely** (§4), removing the export-vs-inline decision and keeping `item.ts`'s
|
||||
edit to `itemToInput` only.
|
||||
- **[MAJOR, folded]** README "Capture … `add`" framing would contradict the agent's `add`→`capture` re-point. →
|
||||
plan Step 5 now corrects the README framing (`add` = manual single-trend; `capture` = normalizing batch), not just
|
||||
appends.
|
||||
- **[MAJOR, folded]** empty "folded" placeholders shipped in the doc bodies. → this section + plan §Plan-critic now
|
||||
filled.
|
||||
- **[MINOR, folded in plan]** missing-/non-numeric-`schemaVersion` test, absent→present back-fill test, stub-replacement
|
||||
note, `import type { TrendInput }` line, `grep -qF` for the 16h literal, `TRENDS_TESTS_FLOOR` breakdown-comment update,
|
||||
README in the code commit, headless per-step clauses N/A (in-session execution).
|
||||
192
docs/research-engine/plan-re-r2a.md
Normal file
192
docs/research-engine/plan-re-r2a.md
Normal file
|
|
@ -0,0 +1,192 @@
|
|||
# Plan — RE-R2a: capture bridge (item→store) + publishedAt persistence
|
||||
|
||||
> **Brief:** `docs/research-engine/brief-re-r2a.md`. **Slice:** RE-R2a (research-engine rung-2, slice 2 — R2 data layer).
|
||||
> **TDD-order:** RED (migration + bridge + capture tests as logic-RED) → GREEN (`types.ts`/`store.ts` migration + persist) →
|
||||
> GREEN (`item.ts` bridge) → GREEN (`cli.ts capture` + tests) → wire `trend-spotter.md` → gate floors → behavioural → land.
|
||||
> **Counts recounted live at land, never pinned/guessed.**
|
||||
> **Light-Voyage hardened:** scope-guardian / brief-reviewer / plan-critic findings folded (see §Plan-critic — folded).
|
||||
|
||||
## Goal
|
||||
|
||||
Build the item→store bridge RE-R1 deferred: a pure `itemToInput` mapping the validated `TrendItem` envelope to a
|
||||
store input (injecting `capturedAt`, carrying `publishedAt`), a `publishedAt` schema bump (v1→v2) with a lossless
|
||||
migrate-on-load, and a `capture` CLI that closes the poll→normalize→store loop. No scoring change; no hook touch; the
|
||||
dated brief + surfacing are R2b.
|
||||
|
||||
## Files touched (exhaustive — for scope-guardian)
|
||||
|
||||
| File | Change | SC |
|
||||
|---|---|---|
|
||||
| `scripts/trends/src/types.ts` | **EDIT** — `SCHEMA_VERSION 1→2`; `publishedAt?: string` on `TrendRecord` (after `capturedAt`, with the capturedAt-distinction comment) | SC3 |
|
||||
| `scripts/trends/src/store.ts` | **EDIT** — `TrendInput` +`publishedAt?`; `addTrend` persists it (conditional-spread, first-sight kept on merge); `loadStore` forward migrate-on-load (`max(onDisk, SCHEMA_VERSION)`) | SC2, SC3 |
|
||||
| `scripts/trends/src/item.ts` | **EDIT** — pure `itemToInput(item, capturedAt): TrendInput` (injects capturedAt; no id; no re-validate) | SC1 |
|
||||
| `scripts/trends/src/cli.ts` | **EDIT** — `capture` subcommand only (stdin→normalize→bridge→addTrend→saveStore→summary; exit 2 bad invocation; `--json`). *(`add --published-at` deferred at light-Voyage — see brief §4.)* | SC4 |
|
||||
| `scripts/trends/tests/store.test.ts` | **EDIT** — migration cases (v1→v2 lossless, idempotent, round-trip) + `publishedAt` persist/first-sight-merge cases | SC2, SC3 |
|
||||
| `scripts/trends/tests/item.test.ts` | **EDIT** — `itemToInput` mapping cases (capturedAt inject, carry-through, no id, absent publishedAt omitted) | SC1 |
|
||||
| `scripts/trends/tests/cli.test.ts` | **EDIT** — `capture` happy path (stdin→store), duplicate/merge, content-invalid in errors[], exit-2 bad invocation, `--json` summary | SC4 |
|
||||
| `scripts/trends/README.md` | **EDIT** — add `publishedAt?` to the record-shape block + a `capture` example (honest schema/CLI doc) | — |
|
||||
| `agents/trend-spotter.md` | **EDIT (Open Q#1, default)** — Step 4.5 `add`→`capture`; carries literal `src/cli.ts capture`; domain-general | SC6 |
|
||||
| `scripts/test-runner.sh` | **EDIT** — `TRENDS_TESTS_FLOOR` 62→recount (stays inside deps guard); NEW unconditional **Section 16h** (before Section 18); `ASSERT_BASELINE_FLOOR` 87→recount; anti-erosion header enumeration | SC5, SC6 |
|
||||
| `docs/research-engine/{brief,plan}-re-r2a.md` | **NEW** — slice docs (TRACKED, like `docs/second-brain/*`) | — |
|
||||
| `STATE.md` | **EDIT at land** — Telling-block reconcile (trends floor, ASSERT floor, gate total). *Land bookkeeping, not slice scope; LOCAL-ONLY.* | — |
|
||||
|
||||
**Not touched (scope fence):** `scripts/trends/src/score.ts` (no scoring change) · `references/trend-scoring-modes.md` + `references/*` (SSOT unchanged, no new ref) · `hooks/**` (no surfacing — R2b) · `config/trends-sources.template.md` · no new `.ts` source file (bridge in `item.ts`) · `agents/*` count (19) · `commands/*` (29) · `.gitignore` (trends lines present).
|
||||
|
||||
## Step 1 — (RED) failing tests for migration + bridge + capture
|
||||
|
||||
Extend the three existing test files against the not-yet-changed code. Logic-RED (not import-RED), stub-by-assertion-type.
|
||||
**Critical RED-vs-GREEN-guard split** (plan-critic blocker): against old code (`SCHEMA_VERSION=1`, `loadStore` returns
|
||||
`parsed.schemaVersion ?? SCHEMA_VERSION`), only some assertions actually fail:
|
||||
- **Genuinely RED** (old code fails): v1-fixture load → `schemaVersion===2` (old returns 1); v1 round-trip
|
||||
`loadStore`→`saveStore` writes `schemaVersion:2` (old writes 1); `addTrend({…, publishedAt})` persists it (old
|
||||
drops it); `itemToInput` carries/injects correctly (add a thin **throwing** stub for the not-yet-existing export so
|
||||
the case fails on assertion, not on `undefined is not a function`).
|
||||
- **GREEN-only regression guards** (pass against old code — NOT labelled RED): a v2 fixture load → `schemaVersion===2`
|
||||
(old already returns 2 via `?? `); **missing** `schemaVersion` → 2 (old `??` already yields current); **non-numeric**
|
||||
`schemaVersion` → 2; empty/absent store → `{schemaVersion:2,trends:[]}`. These are written in Step 1 but documented
|
||||
as regression guards, so the RED proof is not falsely claimed for them.
|
||||
|
||||
`store.test.ts`: the RED migration cases above + the regression-guard cases; `addTrend` without `publishedAt` omits the
|
||||
key; re-capture (same title+url) leaves existing `publishedAt` unchanged and only unions topics; **absent→present**
|
||||
re-capture (first sighting lacked `publishedAt`, re-capture carries one) does **NOT** add it (no back-fill, Open Q#2),
|
||||
`merged` reflects topic change alone.
|
||||
`item.test.ts`: `itemToInput(item,"2026-06-24")` → `capturedAt` injected (`==="2026-06-24"`), all fields carried verbatim,
|
||||
**no `id`**, item without `publishedAt` → input without the key; **field-confusion guard:** an item whose `publishedAt`
|
||||
differs from the injected date → `result.capturedAt !== result.publishedAt`.
|
||||
`cli.test.ts`: `capture` happy (one valid item piped → store gains it, `added:1`); a batch with one invalid item →
|
||||
`errors[]` carries it, valid one added, exit 0; the summary tally **sums to the input size**; the captured record's
|
||||
`capturedAt` matches `/^\d{4}-\d{2}-\d{2}$/` and `!==` the item's `publishedAt`; empty/unparseable stdin → exit 2;
|
||||
`--json` emits the summary object.
|
||||
|
||||
**RED proof (record in commit):** `(cd scripts/trends && npm test)` → the **genuinely-RED** cases fail on assertion
|
||||
(logic-RED), not module-not-found; the regression-guard cases may pass pre-change (documented, not claimed RED).
|
||||
|
||||
## Step 2 — (GREEN) schema migration: `types.ts` + `store.ts`
|
||||
|
||||
`types.ts`: `SCHEMA_VERSION = 2`; add `publishedAt?: string` to `TrendRecord` after `capturedAt` with the
|
||||
distinction comment (source publish-date; forward-compat for B4; distinct from capturedAt).
|
||||
`store.ts`: `TrendInput` gains `publishedAt?`; `addTrend` adds `...(input.publishedAt !== undefined ? { publishedAt: input.publishedAt } : {})` to the new-record literal (after `capturedAt`); the merge branch is **unchanged** (topics union only — first-sight `publishedAt` kept, no back-fill). `loadStore` returns `schemaVersion: Math.max(onDisk, SCHEMA_VERSION)` where `onDisk = typeof parsed.schemaVersion === "number" ? parsed.schemaVersion : SCHEMA_VERSION` (forward-only stamp handling string/`NaN`/absent → current; never crashes); **the existing `Array.isArray(parsed.trends) ? … : []` coercion (`store.ts:79`) is preserved verbatim — a corrupt `trends` field stays out of the losslessness claim**. Make Step 1's RED migration + persist cases (and the regression guards) green.
|
||||
|
||||
## Step 3 — (GREEN) bridge: `itemToInput` in `item.ts`
|
||||
|
||||
Add `import type { TrendInput } from "./store.js";` (item.ts already imports the `normalizeField` *value* from there;
|
||||
this adds the *type* — dependency direction `item.ts → store.ts` stays acyclic). Then **replace the Step-1 throwing
|
||||
stub** with `export function itemToInput(item: TrendItem, capturedAt: string): TrendInput` returning
|
||||
`{ source, title, url, topics: [...item.topics], capturedAt, ...(item.publishedAt !== undefined ? { publishedAt: item.publishedAt } : {}), ...(item.summary !== undefined ? { summary: item.summary } : {}) }`. No `id`; no re-validation (the envelope is already validated). Confirm no throwing stub survives into GREEN. Make Step 1's bridge cases green.
|
||||
|
||||
## Step 4 — (GREEN) CLI `capture` + `cli.test.ts`
|
||||
|
||||
Add `capture` to `cli.ts`'s `main` dispatch: read stdin via the existing `readStdinJson()` (its empty/unparseable path
|
||||
already does `usage()`→exit 2); `Array.isArray(payload) ? normalizeItems(payload) : normalizeItem(payload)`; for each
|
||||
valid item, `itemToInput(item, today())` → `addTrend(store, res.store…)`; `saveStore` once. **Tally derived from
|
||||
`AddResult {added, merged}` (no `duplicates` field, `store.ts:35-41`):** `added += res.added ? 1 : 0`;
|
||||
`merged += (!res.added && res.merged) ? 1 : 0`; `duplicates += (!res.added && !res.merged) ? 1 : 0`. Human summary by
|
||||
default; `--json` prints `{added, duplicates, merged, errors}`. **`add --published-at` is NOT added (deferred).** Write
|
||||
the `cli.test.ts` cases (subprocess: `node --import tsx src/cli.ts capture` with piped stdin + a `--store` temp path),
|
||||
including an explicit `added + merged + duplicates + errors.length === payload.length` assertion and a
|
||||
`capturedAt` shape (`/^\d{4}-\d{2}-\d{2}$/`) + `!== publishedAt` check. **`capturedAt` *exact-value* assertions live in
|
||||
`item.test.ts` (injected fixed date), never in `cli.test.ts` (which reads the wall clock — would be flaky).**
|
||||
|
||||
## Step 5 — wire `trend-spotter.md` (Open Q#1, default) + README
|
||||
|
||||
Replace Step 4.5's N× `add` block (`trend-spotter.md:282-301`) with: build a raw-item JSON batch (the same items
|
||||
already scored), pipe it to **`scripts/trends/src/cli.ts capture`** in one call — it normalizes + folds + persists
|
||||
`publishedAt`. Replacement prose **must contain the literal `src/cli.ts capture`** (Section 16h `grep -qF`). Keep the
|
||||
"skip silently if no deps" escape hatch + domain-general phrasing (no vendor/sector tokens — Section 17). Update
|
||||
`scripts/trends/README.md`: add `publishedAt?` to the record-shape block, add a `capture` CLI example, **and correct
|
||||
the `add` framing** — `README.md:47-64` currently calls `add` "the capture path"; after the re-point `add` is the
|
||||
**manual single-trend** path and `capture` is the **normalizing batch** path. Fix the framing, don't just append (else
|
||||
the README contradicts the new agent wiring).
|
||||
|
||||
## Step 6 — gate: floors + new unconditional Section 16h
|
||||
|
||||
In `scripts/test-runner.sh`:
|
||||
- Bump `TRENDS_TESTS_FLOOR` 62 → **live recount** after Steps 1–4. Stays **inside** the `if [ -x …/tsx ]` deps guard
|
||||
(conditional — do not hoist). **Also update the inline breakdown comment** at `test-runner.sh:695`
|
||||
(`# store 24 + RE-R1: item 18 + score 16 + cli 4`) — extend it with the R2a additions (e.g.
|
||||
`+ RE-R2a: store +N + item +M + cli +K`), so the comment doesn't drift from the number (the exact erosion the
|
||||
anti-erosion section guards against).
|
||||
- Add **Section 16h** ("Trends Capture Wiring"), mirroring Section 16g's three-check shape, **after Section 17 /
|
||||
before Section 18**. Three **unconditional**, deps-absent-safe checks (pure `grep`/self-test, no `tsx`):
|
||||
(1) a non-vacuity self-test (accept a probe carrying `src/cli.ts capture`, reject one without it);
|
||||
(2) `grep -qF 'command === "capture"' scripts/trends/src/cli.ts` — deps-absent proof the handler exists (`grep -qF`
|
||||
exact, mirroring 16g lines 1058/1064 — not `grep -E`);
|
||||
(3) `grep -qF "src/cli.ts capture" agents/trend-spotter.md`.
|
||||
- Bump `ASSERT_BASELINE_FLOOR` 87 → **live recount** (= 87 + the new unconditional emitters in 16h; expected ~90,
|
||||
**recounted at land, not pinned**). Update the Section-18 anti-erosion header enumeration comment.
|
||||
|
||||
## Step 7 — behavioural verification
|
||||
|
||||
`(cd scripts/trends && npm install)` if needed, then:
|
||||
`echo '[{"source":"tavily","title":"X","url":"https://e/x","topics":["a"],"publishedAt":"2026-06-20"},{"title":"bad"}]' | node --import tsx src/cli.ts capture --store /tmp/r2a-trends.json --json` →
|
||||
confirm `added:1`, the invalid entry in `errors[]`, and `node … list --store /tmp/r2a-trends.json --json` shows the
|
||||
persisted `publishedAt`. Author a v1 fixture (`{"schemaVersion":1,"trends":[{…no publishedAt}]}`), run `status --json`
|
||||
→ confirm count intact (migration lossless). Run full `bash scripts/test-runner.sh` → `FAIL=0`.
|
||||
|
||||
## Step 8 — land
|
||||
|
||||
Recount all touched floors live; reconcile STATE.md "Telling" block (trends N/N, ASSERT floor, gate total). Commit
|
||||
order (house style): **(1)** docs commit `docs/research-engine/{brief,plan}-re-r2a.md` (no suffix, tracked); **(2)**
|
||||
code commit `scripts/trends/*` + `agents/trend-spotter.md` + `scripts/trends/README.md` + `scripts/test-runner.sh`
|
||||
with `[skip-docs]`. Push freely (window restriction lifted 2026-06-24; gitleaks runs at commit; `origin` = PUBLIC
|
||||
`open/` — STATE/`*.local.*` never pushed). No version bump (additive; `v0.5.2` dev).
|
||||
|
||||
## Verification (testable)
|
||||
|
||||
| SC | Check | Command | Expected |
|
||||
|---|---|---|---|
|
||||
| — | RED proof | `(cd scripts/trends && npm test)` after Step 1 | new cases fail on assertion (logic-RED), not module-not-found |
|
||||
| SC1 | bridge | `npm test` (item.test.ts) | capturedAt injected (`==="2026-06-24"`); fields carried; no `id`; absent publishedAt omitted; `capturedAt !== publishedAt` (field-confusion guard) |
|
||||
| SC2 | persist | `npm test` (store.test.ts) | publishedAt persisted when present / omitted when absent; first-sight kept on merge; absent→present re-capture does NOT back-fill |
|
||||
| SC3 | migrate | `npm test` (store.test.ts) | v1→v2 lossless (records intact, no publishedAt invented); missing/non-numeric schemaVersion → 2; empty store → v2/[]; idempotent; round-trip writes v2 |
|
||||
| SC4 | capture | `npm test` (cli.test.ts) + manual | normalize+fold; errors[] for invalid; tally sums to input size; capturedAt today-shaped & ≠ publishedAt; exit 2 bad stdin / 0 well-formed; `--json` summary |
|
||||
| SC5 | gate | `bash scripts/test-runner.sh` | `FAIL=0`; trends ≥ new floor; Section 16h green; ASSERT floor = live recount |
|
||||
| SC6 | wiring + de-niche | Section 16h greps + Section 17 | `src/cli.ts capture` in trend-spotter.md; no vendor/sector tokens; counts 19/29/27 |
|
||||
|
||||
## Risks
|
||||
|
||||
- **R1 — migration eats or rewrites records.** *Mitigated:* migrate-on-load is a version *stamp* only (`max(onDisk,
|
||||
current)`); records pass through untouched; SC3 pins lossless + idempotent + round-trip on a real v1 fixture.
|
||||
- **R2 — `publishedAt` first-sight vs back-fill ambiguity.** *Mitigated:* brief Open Q#2 settles it at first-sight
|
||||
(no back-fill); SC2 pins "existing publishedAt unchanged on re-capture"; back-fill explicitly deferred.
|
||||
- **R3 — `capture` exit-code semantics drift** (content-invalid item leaking into a non-zero exit). *Mitigated:* SC4
|
||||
pins exit 2 = malformed invocation only; content-invalid → `errors[]` at exit 0; mirrors `normalize`/`score`.
|
||||
- **R4 — editing `trend-spotter.md` trips the de-niche guard.** *Mitigated:* Section 17 runs in the gate; replacement
|
||||
prose is pillar/source-driven and vendor/sector-free; only the store-fold mechanism changes (`add`→`capture`).
|
||||
- **R5 — new gate checks must survive a deps-absent fresh clone.** *Mitigated:* Section 16h is pure `grep`/self-test on
|
||||
tracked source (no `tsx`) → unconditional; `TRENDS_TESTS_FLOOR` stays inside the deps guard.
|
||||
- **R6 — future schema downgrade (on-disk > current) silently drops unknown fields on save.** *Mitigated/deferred:*
|
||||
cannot happen pre-v3; `max()` already refuses to downgrade the stamp; field-preservation-on-save is an R-future
|
||||
concern, noted not handled (no impossible-scenario code).
|
||||
|
||||
## Plan-critic — folded
|
||||
|
||||
plan-critic returned **REVISE** (1 blocker, 5 majors, 5 minors); brief-reviewer **PROCEED_WITH_RISKS**;
|
||||
scope-guardian **ALIGNED**. Resolution, each verified against live code:
|
||||
|
||||
- **[BLOCKER] "v2 load idempotent" is not RED** (`store.ts:78` `?? ` already returns 2 for a v2 fixture under old
|
||||
`SCHEMA_VERSION=1`). ✅ Step 1 now splits **genuinely-RED** (v1→2 load + round-trip + addTrend-persist + itemToInput)
|
||||
from **GREEN-only regression guards** (v2-idempotence, missing/non-numeric/empty); the RED proof is claimed only for
|
||||
the former.
|
||||
- **[MAJOR] capture tally mis-maps onto `AddResult`** (no `duplicates` field, `store.ts:35-41`). ✅ Step 4 pins the
|
||||
exact derivation (`added`/`merged`/`duplicates` from `res.added`+`res.merged`) + a `sum === payload.length` test.
|
||||
- **[MAJOR] `TRENDS_TESTS_FLOOR` breakdown comment (`:695`) left stale.** ✅ Step 6 now extends the inline breakdown
|
||||
comment alongside the number.
|
||||
- **[MAJOR] `import isValidIso` impossible** (`item.ts:51` private). ✅ `add --published-at` **deferred entirely**
|
||||
(brief §4) — no export-vs-inline decision, `item.ts` edit stays `itemToInput`-only.
|
||||
- **[MAJOR] README `add`-as-capture framing would contradict the re-point.** ✅ Step 5 corrects the framing
|
||||
(`add` = manual single-trend; `capture` = normalizing batch), not just appends.
|
||||
- **[MAJOR] empty "folded" placeholders.** ✅ this section + brief §9 filled.
|
||||
- **[MINOR] stub cleanup unstated** ✅ Step 3 states the throwing stub is **replaced**, none survives into GREEN.
|
||||
**[MINOR] `today()` capturedAt untestable in cli.test** ✅ Step 4 routes exact-value assertions to `item.test.ts`,
|
||||
cli.test asserts shape + `≠ publishedAt`. **[MINOR] 16h grep flag** ✅ Step 6 specifies `grep -qF`.
|
||||
**[MINOR] non-numeric schemaVersion untested** ✅ added to Step 1 regression guards + SC3. **[MINOR] README under
|
||||
`[skip-docs]`** ✅ kept in the code commit (it documents the shipped code, like R1's `trend-spotter.md`); noted.
|
||||
- **[brief-reviewer MAJOR] capturedAt injection unverified at the new ingress** ✅ SC1 + SC4 field-confusion guards.
|
||||
**[brief-reviewer MAJOR] "lossless" over-claim** ✅ §5/§3 scope it to well-formed stores; malformed-`trends`
|
||||
coercion unchanged + out of scope. **[brief-reviewer MINOR] absent→present back-fill** ✅ pinned by an SC2 test.
|
||||
- **[plan-critic headless-readiness 60]** N/A — R2a is executed **in-session, operator-driven** (driftsmodell), not
|
||||
as a headless autonomous run, so per-step revert/halt clauses aren't needed (R1's plan had none either).
|
||||
|
||||
**scope-guardian — ALIGNED:** every SC1–SC6 traces to a step; zero creep (README is in-change documentation); every
|
||||
§4 non-goal held (no hook touch, no brief artifact, no relevance/saturation/status, no back-fill, no scoring change).
|
||||
Loading…
Add table
Add a link
Reference in a new issue