linkedin-studio/docs/research-engine/brief-re-r1.md

Brief — RE-R1: lift the research engine's item-schema + triage-scorer from agent prose to tested code

Slice: RE-R1 (research-engine rung-2, slice 1) — the first build slice of the daily research motor. Status: drafted, awaiting go-gate. Predecessors: "slice 2a" = the scoring SSOT references/trend-scoring-modes.md; "slice 2b" = the trend store scripts/trends/ (24/24). This slice adds the scoring/normalization code those two anticipate (trend-scoring-modes.md:96-100 names "any future research-engine pass that scores candidates before writing them to the trend store" as a consumer). Substrate: docs/research-engine-concepts.local.md §1, §3 (B1/B2), §5 ("anbefalt minste neste slice"). TDD-order: RED tests + RED gate self-test land before code.

1. Operator decision context (2026-06-23)

The operator lifted the research engine to Tier-1. Rationale, verbatim: "hele min arbeidsflyt hviler på at jeg får en jevn strøm av gode forslag til tema å skrive om." A/moat (lived-specifics · positioning · shareability · analytics→loop) remains the long-term nordstjerne but is built interactively in Maskinrommet from lived use, not as a TDD slice now.

This slice is exactly the one research-engine-concepts.local.md §5 named as the minimal next move if research is lifted: B1 (uniform item-schema, tested) + B2 (triage-scorer as code with threshold config). It moves "the only untestable layer" from agent prose into gated code, and is the foundation every later rung (brief artifact, surfacing, freshness, deepening) hangs on.

2. The gap — grounded in code

• Storage exists; scoring does not. scripts/trends/src/store.ts (183 LOC) stores/dedupes/queries TrendRecord {id,title,url,source,capturedAt,topics[],summary?} (types.ts:26-41). Its header states scoring "lives in the agent/command layer; this module only stores, dedupes, and serves." There is no scoring/routing/saturation/digest code anywhere in the repo.
• The arithmetic is prose the LLM runs in its head. references/trend-scoring-modes.md (SSOT, 101 lines) defines two modes — kortform (Pillar .30 / Audience .25 / Timing .20 / Angle .15 / Authority .10) and long-form (Pillar .30 / Depth .25 / Angle .20 / Authority .15 / Currency .10) — a weighted composite on a shared 0–10 scale, and a composite→band map (8.0–10 Immediate · 6.0–7.9 High · 4.0–5.9 Medium · 2.0–3.9 Low · 0–1.9 Skip), each band carrying a kortform + long-form action. agents/trend-spotter.md:124-137 tells the agent to "score each dimension 1–10, take the weighted composite, rank highest-first, apply the bands" — deterministic arithmetic with no test and no code.
• Ingress is ad hoc. Nothing normalizes a source's raw output into one item envelope before it reaches addTrend; downstream would branch on source-type. B1 is the "one schema downstream never branches on" the concepts doc calls "the most load-bearing idea — build it first."

3. Scope — what is IN (RE-R1)

B1 — canonical item schema + normalizer (scripts/trends/src/item.ts, NEW)

A TrendItem ingress envelope { source, title, url, publishedAt?, topics[], summary? } — what a source/adapter emits — plus a pure normalizeItem(raw): { ok: true, item } | { ok: false, errors } and normalizeItems(raw[]): { items, errors } (batch partition). Deterministic: required-field validation (source/title/url present and non-empty → else a structured {ok:false} error naming the field, never a silent partial), whitespace normalization (reuse normalizeField from store.ts), topic normalize + dedupe, publishedAt ISO-8601-validate-if-present (the source's publish date — carried for forward-compat, not persisted in R1, and distinct from the store's capturedAt). No id field: the envelope carries none and the store already derives it via addTrend→trendId. Vocabulary kept as topics (≡ the concepts doc's tags; see §8). The item→store bridge (injecting capturedAt, persisting publishedAt) is R2 orchestration — out of scope here; R1 delivers the validated envelope + normalizer only.

B2 — deterministic triage-scorer (scripts/trends/src/score.ts, NEW)

ScoreMode = 'kortform' | 'long-form'; per-mode weight constants encoded from the SSOT (with the SSOT's "ordering is the signal, not a measured coefficient" caveat as a comment). DimensionScores (mode-specific keys). composite(scores, mode): number — weighted sum, validates each dimension ∈ [1,10]. band(composite): { priority, kortformAction, longformAction } — the five-band map. triage(scored, { mode, threshold }): { kept, dropped } — kept = composite ≥ threshold, ranked composite-desc, each annotated with composite + band; dropped below. threshold is a single config parameter (default — see §8). The module owns only the arithmetic / bands / threshold; producing the five dimension scores stays model judgment (the deterministic/judgment split from the concepts doc §5 / SSOT analysis).

CLI surface (scripts/trends/src/cli.ts, EDIT existing)

Two new subcommands over the new modules — both read their JSON payload from stdin (a raw item/batch for normalize; items-with-dimension-scores for score) and emit JSON to stdout, so they do not overload the existing --json output toggle (cli.ts:79). score takes [--mode kortform|long-form] [--threshold N] flags. Exit codes: 2 for a malformed invocation (missing required flag, unparseable stdin JSON) — matching cli.ts's usage-error precedent (cli.ts:54-63); 0 for a well-formed call, even when the payload carries content-invalid items (reported as {ok:false,...} entries in the output, never via the exit code).

Wiring + gate

• agents/trend-spotter.md (EDIT): replace the "do the composite + bands yourself" prose at L124-137 with a pointer to the scorer CLI as the owner of the composite/band/threshold step (the lift becomes real and grep-able; the agent still supplies the five judgment scores). Kept domain-general.
• scripts/test-runner.sh (EDIT): bump TRENDS_TESTS_FLOOR (recount live at land; it stays inside the if [ -x …/tsx ] deps guard — conditional, never hoisted out). Add a new grep-only Section 16g, placed after Section 17 (de-niche) and before Section 18 (anti-erosion, which must run last so it sees every prior check), with deps-absent-safe checks (no tsx): (1) grep that score.ts encodes both mode weight-sets; (2) grep -qF "src/cli.ts score" that trend-spotter.md references the scorer CLI; (3) a non-vacuity self-test for those greps, per the house pattern (every sibling section 16c–17 has one). These are unconditional → bump ASSERT_BASELINE_FLOOR 84 → live recount (expected ~+3, not a pinned number). The arithmetic proof (composite(all-tens)=10.0 ⇒ weights sum to 1.0) lives as a unit test in score.test.ts (conditional, counts toward the trends floor), since it needs the tsx runtime.

4. Non-goals — what is OUT (deferred)

• The dated brief artifact / morning-brief file (B3) — R2. This is the rung that makes the stream visible to the operator; R1 is the foundation beneath it, not the stream itself.
• Session-start surfacing of the brief (hull 4) — R2.
• Store-schema migration / new persisted fields (publishedAt, relevance, saturation, status — hull 3/5) — R2. R1 keeps SCHEMA_VERSION = 1 untouched (no migration risk).
• Freshness window / dedup-vs-seen-log / autonomous trigger (B4) — R3.
• Research-deepening (plan → isolated workers → synthesis, A1–A4) — R3.
• Adapter sub-agents / MCP fetch fan-out — R2/R3.
• Producing the five dimension scores in code — stays model judgment, by design.

5. Boundaries / invariants (must hold)

• TDD iron law: the failing item/score tests and the failing gate self-test land BEFORE the implementation. RED proofs recorded (logic-RED, not import-RED).
• No store-schema change: SCHEMA_VERSION stays 1; TrendRecord untouched; no data migration.
• Domain-general: de-niche guard (Section 17) stays green; no vendor/sector tokens enter the edited trend-spotter.md prose.
• SSOT discipline: trend-scoring-modes.md remains the human source of truth; score.ts mirrors it; a test pins the exact per-mode weights, the four band thresholds (8.0/6.0/4.0/2.0), and the ten band action strings against the SSOT values, and asserts each mode sums to 1.0 — so silent drift in any of them (not just the weights) fails loudly. (A markdown-parsing cross-check of the SSOT table itself is deferred — out of scope.)
• House conventions: ESM + node:test + tsx; data-seam stays inline (no new shared util — the documented idiom); no new .mjs under hooks/scripts/; .gitignore already covers scripts/trends/{node_modules,build}.
• No new references/*.md (would trip the named-additions guard) and no new agent/command (counts stay 19/29/27). Brief+plan live under docs/ (uncounted), TRACKED like docs/second-brain/* (general feature design).
• Counts recounted live at land, never pinned/guessed.

6. Success criteria (testable)

• SC1 — normalizeItem returns a canonical item from a well-formed raw item; a missing/empty required field returns a structured {ok:false} error naming the field (not a silent partial); whitespace + topic dedupe applied; publishedAt validated-if-present, undefined-if-absent. normalizeItems partitions a batch into {items, errors}.
• SC2 — composite() equals the SSOT formula exactly for both modes: all-tens → 10.0 (proves Σweights=1), and the asymmetric golden vector {10, 8, 6, 4, 2} (in dimension order) → 7.0 for both modes (10·.30 + 8·.25 + 6·.20 + 4·.15 + 2·.10) — asymmetric so a weight↔dimension transposition is caught; a dimension outside [1,10] throws.
• SC3 — band() returns the correct priority + the exact mode-appropriate action string (pinned against the SSOT) at every boundary (8.0, 6.0, 4.0, 2.0 edges).
• SC4 — triage() keeps composite ≥ threshold, drops below, ranks kept composite-desc, annotates each kept item with composite + band.
• SC5 — CLI normalize/score read JSON from stdin, emit valid JSON; exit 2 on a malformed invocation (missing flag / unparseable stdin), 0 on a well-formed call (content-invalid items reported in the payload, never via exit code).
• SC6 — (cd scripts/trends && npm test) green at the bumped floor; gate's new unconditional section green; ASSERT_BASELINE_FLOOR bumped to the live recount; overall gate FAIL=0.
• SC7 — trend-spotter.md references the scorer for the deterministic step (gate grep green); de-niche guard green; structure counts unchanged (19/29/27).

7. Verification

Deterministic (gate): bash scripts/test-runner.sh → FAIL=0; trends suite ≥ new floor; new self-test + wiring-grep pass; Section 17 de-niche green; ref/agent/command counts unchanged. Behavioural (manual): run npm run start -- normalize --json '<sample batch>' and ... score --json '<scored sample>' --mode kortform --threshold 4.0; eyeball that kept/dropped/bands match a hand-computed expectation on 3–4 items.

8. Open questions for the go-gate

1. Default threshold. Propose 4.0 — the SSOT's Medium-band floor and the agent's existing "score 4.0+" angle-mapping cutoff (trend-spotter.md:179,274). Confirm or set otherwise.
2. Vocabulary topics vs tags. The concepts doc envelope says tags; the codebase standardized on topics. Propose keep topics (consistency, no rename) and note tags ≡ topics. Confirm.
3. CLI tests now? Trends has no cli.test.ts today; siblings (brain, contract-gate) do. Propose add light cli tests for the two new subcommands (happy path + exit codes). Confirm vs defer.
4. Wire trend-spotter.md now? Propose yes — a prose pointer so the lift is real and gate-grep-able; full orchestration (fan-out, brief assembly) stays R2.

9. Light-Voyage review — folded

Three Opus reviewers ran on the drafts. scope-guardian: ALIGNED (every SC1–SC7 traces to a step; zero scope creep; NON-goals fully respected). brief-reviewer: PROCEED_WITH_RISKS. plan-critic: REVISE (1 blocker, 6 majors). All findings folded:

• [BLOCKER, folded] TrendItem→TrendInput does not map directly — live TrendInput requires capturedAt and has no publishedAt; the item has the opposite. → §3 B1 now scopes R1 to the validated envelope only; the item→store bridge (capturedAt injection, publishedAt persistence) is R2. A scope-tightening, not an addition.
• [MAJOR, folded] CLI --json is already an output toggle → §3 now reads payload from stdin, --json untouched.
• [MAJOR, folded] Gate-section placement + anti-erosion-last → §3 pins Section 16g, before Section 18; TRENDS_TESTS_FLOOR stays inside the deps guard.
• [MAJOR, folded] ASSERT_BASELINE_FLOOR was pinned to 86 against the recount-live rule, and the house self-test pattern makes it +3 → §3 now says live recount (~+3), not pinned.
• [MAJOR, folded] Only weights were drift-guarded → §5 SSOT discipline now pins weights + band thresholds + the ten action strings.
• [MAJOR, folded] Wiring grep literal unpinned → §3 pins grep -qF "src/cli.ts score".
• [testability, folded] SC2 unnamed vector → pinned {10,8,6,4,2}→7.0; SC5 exit-code ambiguity → explicit 2-vs-0 contract; id hedge → envelope carries no id; normalizeItems batch shape → SC1 extended.
• [minor, folded in plan] RED-stub split per assertion type; explicit node --import tsx invocation (not npm run start); commit grouping fixed; STATE.md noted as land-bookkeeping; config/trends-sources.template.md added to the scope fence.

See plan-re-r1.md §Plan-critic — folded for the per-step resolution.