linkedin-studio/docs/research-engine/brief-re-r1.md

# Brief — RE-R1: lift the research engine's item-schema + triage-scorer from agent prose to tested code

> **Slice:** RE-R1 (research-engine rung-2, slice 1) — the first build slice of the daily research motor.
> **Status:** drafted, awaiting go-gate. **Predecessors:** "slice 2a" = the scoring SSOT `references/trend-scoring-modes.md`; "slice 2b" = the trend store `scripts/trends/` (24/24). This slice adds the **scoring/normalization code those two anticipate** (`trend-scoring-modes.md:96-100` names "any future research-engine pass that scores candidates before writing them to the trend store" as a consumer).
> **Substrate:** `docs/research-engine-concepts.local.md` §1, §3 (B1/B2), §5 ("anbefalt minste neste slice"). **TDD-order:** RED tests + RED gate self-test land before code.

## 1. Operator decision context (2026-06-23)

The operator lifted the **research engine to Tier-1**. Rationale, verbatim: *"hele min arbeidsflyt hviler på at jeg får en jevn strøm av gode forslag til tema å skrive om."* A/moat (lived-specifics · positioning · shareability · analytics→loop) remains the long-term nordstjerne but is built **interactively in Maskinrommet from lived use**, not as a TDD slice now.

This slice is exactly the one `research-engine-concepts.local.md` §5 named as the minimal next move *if research is lifted*: **B1 (uniform item-schema, tested) + B2 (triage-scorer as code with threshold config)**. It moves "the only untestable layer" from agent prose into gated code, and is the foundation every later rung (brief artifact, surfacing, freshness, deepening) hangs on.

## 2. The gap — grounded in code

- **Storage exists; scoring does not.** `scripts/trends/src/store.ts` (183 LOC) stores/dedupes/queries `TrendRecord {id,title,url,source,capturedAt,topics[],summary?}` (`types.ts:26-41`). Its header states scoring "lives in the agent/command layer; this module only stores, dedupes, and serves." There is **no scoring/routing/saturation/digest code anywhere** in the repo.
- **The arithmetic is prose the LLM runs in its head.** `references/trend-scoring-modes.md` (SSOT, 101 lines) defines two modes — **kortform** (Pillar .30 / Audience .25 / Timing .20 / Angle .15 / Authority .10) and **long-form** (Pillar .30 / Depth .25 / Angle .20 / Authority .15 / Currency .10) — a weighted composite on a shared 0–10 scale, and a composite→band map (8.0–10 Immediate · 6.0–7.9 High · 4.0–5.9 Medium · 2.0–3.9 Low · 0–1.9 Skip), each band carrying a kortform + long-form action. `agents/trend-spotter.md:124-137` tells the agent to "score each dimension 1–10, take the weighted composite, rank highest-first, apply the bands" — **deterministic arithmetic with no test and no code.**
- **Ingress is ad hoc.** Nothing normalizes a source's raw output into one item envelope before it reaches `addTrend`; downstream would branch on source-type. B1 is the "one schema downstream never branches on" the concepts doc calls "the most load-bearing idea — build it first."

## 3. Scope — what is IN (RE-R1)

### B1 — canonical item schema + normalizer (`scripts/trends/src/item.ts`, NEW)
A `TrendItem` **ingress envelope** `{ source, title, url, publishedAt?, topics[], summary? }` — what a source/adapter emits — plus a pure `normalizeItem(raw): { ok: true, item } | { ok: false, errors }` and `normalizeItems(raw[]): { items, errors }` (batch partition). Deterministic: required-field validation (`source/title/url` present and non-empty → else a structured `{ok:false}` error naming the field, never a silent partial), whitespace normalization (reuse `normalizeField` from `store.ts`), topic normalize + dedupe, `publishedAt` ISO-8601-validate-**if-present** (the source's publish date — carried for forward-compat, **not persisted in R1**, and distinct from the store's `capturedAt`). **No `id` field:** the envelope carries none and the store already derives it via `addTrend`→`trendId`. Vocabulary kept as `topics` (≡ the concepts doc's `tags`; see §8). **The item→store bridge** (injecting `capturedAt`, persisting `publishedAt`) is **R2 orchestration — out of scope here**; R1 delivers the validated envelope + normalizer only.

### B2 — deterministic triage-scorer (`scripts/trends/src/score.ts`, NEW)
`ScoreMode = 'kortform' | 'long-form'`; per-mode weight constants **encoded from the SSOT** (with the SSOT's "ordering is the signal, not a measured coefficient" caveat as a comment). `DimensionScores` (mode-specific keys). `composite(scores, mode): number` — weighted sum, validates each dimension ∈ [1,10]. `band(composite): { priority, kortformAction, longformAction }` — the five-band map. `triage(scored, { mode, threshold }): { kept, dropped }` — kept = composite ≥ threshold, ranked composite-desc, each annotated with composite + band; dropped below. `threshold` is a single config parameter (default — see §8). The module owns **only** the arithmetic / bands / threshold; producing the five dimension scores stays model judgment (the deterministic/judgment split from the concepts doc §5 / SSOT analysis).

### CLI surface (`scripts/trends/src/cli.ts`, EDIT existing)
Two new subcommands over the new modules — both **read their JSON payload from stdin** (a raw item/batch for `normalize`; items-with-dimension-scores for `score`) and **emit JSON to stdout**, so they do **not** overload the existing `--json` *output* toggle (`cli.ts:79`). `score` takes `[--mode kortform|long-form] [--threshold N]` flags. Exit codes: **2** for a malformed invocation (missing required flag, unparseable stdin JSON) — matching `cli.ts`'s usage-error precedent (`cli.ts:54-63`); **0** for a well-formed call, even when the payload carries content-invalid items (reported as `{ok:false,...}` entries in the output, never via the exit code).

### Wiring + gate
- `agents/trend-spotter.md` (EDIT): replace the "do the composite + bands yourself" prose at L124-137 with a pointer to the scorer CLI as the owner of the composite/band/threshold step (the lift becomes real and grep-able; the agent still supplies the five judgment scores). Kept domain-general.
- `scripts/test-runner.sh` (EDIT): bump `TRENDS_TESTS_FLOOR` (recount live at land; it **stays inside the `if [ -x …/tsx ]` deps guard** — conditional, never hoisted out). Add a new **grep-only** Section **16g**, placed after Section 17 (de-niche) and **before** Section 18 (anti-erosion, which must run last so it sees every prior check), with deps-absent-safe checks (no `tsx`): (1) `grep` that `score.ts` encodes both mode weight-sets; (2) `grep -qF "src/cli.ts score"` that `trend-spotter.md` references the scorer CLI; (3) a non-vacuity self-test for those greps, per the house pattern (every sibling section 16c–17 has one). These are **unconditional** → bump `ASSERT_BASELINE_FLOOR` 84 → **live recount** (expected ~+3, not a pinned number). The arithmetic proof (`composite(all-tens)=10.0` ⇒ weights sum to 1.0) lives as a **unit test** in `score.test.ts` (conditional, counts toward the trends floor), since it needs the `tsx` runtime.

## 4. Non-goals — what is OUT (deferred)

- **The dated brief artifact / morning-brief file (B3)** — R2. *This is the rung that makes the stream visible to the operator; R1 is the foundation beneath it, not the stream itself.*
- **Session-start surfacing** of the brief (hull 4) — R2.
- **Store-schema migration / new persisted fields** (publishedAt, relevance, saturation, status — hull 3/5) — R2. R1 keeps `SCHEMA_VERSION = 1` untouched (no migration risk).
- **Freshness window / dedup-vs-seen-log / autonomous trigger (B4)** — R3.
- **Research-deepening (plan → isolated workers → synthesis, A1–A4)** — R3.
- **Adapter sub-agents / MCP fetch fan-out** — R2/R3.
- **Producing the five dimension scores in code** — stays model judgment, by design.

## 5. Boundaries / invariants (must hold)

- **TDD iron law:** the failing `item`/`score` tests **and** the failing gate self-test land BEFORE the implementation. RED proofs recorded (logic-RED, not import-RED).
- **No store-schema change:** `SCHEMA_VERSION` stays 1; `TrendRecord` untouched; no data migration.
- **Domain-general:** de-niche guard (Section 17) stays green; no vendor/sector tokens enter the edited `trend-spotter.md` prose.
- **SSOT discipline:** `trend-scoring-modes.md` remains the human source of truth; `score.ts` mirrors it; a test pins **the exact per-mode weights, the four band thresholds (8.0/6.0/4.0/2.0), and the ten band action strings** against the SSOT values, and asserts each mode sums to 1.0 — so silent drift in *any* of them (not just the weights) fails loudly. (A markdown-parsing cross-check of the SSOT table itself is deferred — out of scope.)
- **House conventions:** ESM + `node:test` + `tsx`; data-seam stays inline (no new shared util — the documented idiom); no new `.mjs` under `hooks/scripts/`; `.gitignore` already covers `scripts/trends/{node_modules,build}`.
- **No new `references/*.md`** (would trip the named-additions guard) and **no new agent/command** (counts stay 19/29/27). Brief+plan live under `docs/` (uncounted), TRACKED like `docs/second-brain/*` (general feature design).
- **Counts recounted live at land**, never pinned/guessed.

## 6. Success criteria (testable)

- **SC1** — `normalizeItem` returns a canonical item from a well-formed raw item; a missing/empty required field returns a structured `{ok:false}` error naming the field (not a silent partial); whitespace + topic dedupe applied; `publishedAt` validated-if-present, undefined-if-absent. `normalizeItems` partitions a batch into `{items, errors}`.
- **SC2** — `composite()` equals the SSOT formula exactly for both modes: all-tens → **10.0** (proves Σweights=1), and the **asymmetric golden vector `{10, 8, 6, 4, 2}` (in dimension order) → 7.0** for both modes (`10·.30 + 8·.25 + 6·.20 + 4·.15 + 2·.10`) — asymmetric so a weight↔dimension transposition is caught; a dimension outside [1,10] throws.
- **SC3** — `band()` returns the correct priority + **the exact mode-appropriate action string** (pinned against the SSOT) at every boundary (8.0, 6.0, 4.0, 2.0 edges).
- **SC4** — `triage()` keeps composite ≥ threshold, drops below, ranks kept composite-desc, annotates each kept item with composite + band.
- **SC5** — CLI `normalize`/`score` read JSON from stdin, emit valid JSON; exit **2** on a malformed invocation (missing flag / unparseable stdin), **0** on a well-formed call (content-invalid items reported in the payload, never via exit code).
- **SC6** — `(cd scripts/trends && npm test)` green at the bumped floor; gate's new unconditional section green; `ASSERT_BASELINE_FLOOR` bumped to the live recount; overall gate `FAIL=0`.
- **SC7** — `trend-spotter.md` references the scorer for the deterministic step (gate grep green); de-niche guard green; structure counts unchanged (19/29/27).

## 7. Verification

**Deterministic (gate):** `bash scripts/test-runner.sh` → `FAIL=0`; trends suite ≥ new floor; new self-test + wiring-grep pass; Section 17 de-niche green; ref/agent/command counts unchanged.
**Behavioural (manual):** run `npm run start -- normalize --json '<sample batch>'` and `... score --json '<scored sample>' --mode kortform --threshold 4.0`; eyeball that kept/dropped/bands match a hand-computed expectation on 3–4 items.

## 8. Open questions for the go-gate

1. **Default threshold.** Propose **4.0** — the SSOT's Medium-band floor and the agent's existing "score 4.0+" angle-mapping cutoff (`trend-spotter.md:179,274`). Confirm or set otherwise.
2. **Vocabulary `topics` vs `tags`.** The concepts doc envelope says `tags`; the codebase standardized on `topics`. Propose **keep `topics`** (consistency, no rename) and note `tags ≡ topics`. Confirm.
3. **CLI tests now?** Trends has no `cli.test.ts` today; siblings (brain, contract-gate) do. Propose **add light cli tests** for the two new subcommands (happy path + exit codes). Confirm vs defer.
4. **Wire `trend-spotter.md` now?** Propose **yes** — a prose pointer so the lift is real and gate-grep-able; full orchestration (fan-out, brief assembly) stays R2.

## 9. Light-Voyage review — folded

Three Opus reviewers ran on the drafts. **scope-guardian: ALIGNED** (every SC1–SC7 traces to a step; zero scope creep; NON-goals fully respected). **brief-reviewer: PROCEED_WITH_RISKS.** **plan-critic: REVISE (1 blocker, 6 majors).** All findings folded:

- **[BLOCKER, folded]** `TrendItem`→`TrendInput` does not map directly — live `TrendInput` requires `capturedAt` and has no `publishedAt`; the item has the opposite. → §3 B1 now scopes R1 to the validated envelope only; the item→store bridge (`capturedAt` injection, `publishedAt` persistence) is **R2**. A scope-tightening, not an addition.
- **[MAJOR, folded]** CLI `--json` is already an *output* toggle → §3 now reads payload from **stdin**, `--json` untouched.
- **[MAJOR, folded]** Gate-section placement + anti-erosion-last → §3 pins **Section 16g, before Section 18**; `TRENDS_TESTS_FLOOR` stays inside the deps guard.
- **[MAJOR, folded]** `ASSERT_BASELINE_FLOOR` was pinned to 86 against the recount-live rule, and the house self-test pattern makes it +3 → §3 now says **live recount (~+3), not pinned**.
- **[MAJOR, folded]** Only weights were drift-guarded → §5 SSOT discipline now pins **weights + band thresholds + the ten action strings**.
- **[MAJOR, folded]** Wiring grep literal unpinned → §3 pins `grep -qF "src/cli.ts score"`.
- **[testability, folded]** SC2 unnamed vector → **pinned `{10,8,6,4,2}`→7.0**; SC5 exit-code ambiguity → **explicit 2-vs-0 contract**; `id` hedge → **envelope carries no id**; `normalizeItems` batch shape → **SC1 extended**.
- **[minor, folded in plan]** RED-stub split per assertion type; explicit `node --import tsx` invocation (not `npm run start`); commit grouping fixed; STATE.md noted as land-bookkeeping; `config/trends-sources.template.md` added to the scope fence.

See `plan-re-r1.md` §Plan-critic — folded for the per-step resolution.