open/linkedin-studio
1
0
Fork 0
Code Issues Pull requests Releases Activity

feat(linkedin-studio): N17 — baseline-motor (median + variansbånd + minimum-N-refusal) [skip-docs]

Browse source 
Every reading now leads with "vs your own baseline", and no verdict is given
when N is too small to carry one.

- stats.ts: median + medianAbsoluteDeviation (robust pair; mean/stddev stay for
  the alert engine, which wants outlier sensitivity), rollingBaseline with a
  10-post positional window, median ± 1·MAD band floored at 0, and a typed
  insufficient-data refusal below MIN_BASELINE_N=5. readAgainstBaseline returns
  above/within/below-band, or no-verdict when the baseline was refused.
- baselineByGroup + buildBaselineBlock: per-format/per-pillar baselines, each
  judged on its own N; the reported period is excluded from its own baseline and
  compared on its median, not its mean.
- queue-join.ts (new): read-only date join supplying format/pillar from the post
  queue. Every ambiguity resolves to unlabelled, an entry labels at most one
  post, and a missing/broken queue degrades to no labels.
- weekly/monthly reports attach the block unconditionally (refusal included);
  optional in the types, so pre-N17 reports load unchanged.
- CLI: report output leads with the baseline; new `baseline [--by format|pillar]`
  verb with coverage reporting.
- report.md leads with baseline framing and prints the code's reading rather
  than judging the band by eye; WoW loses to the baseline on disagreement.
  analyze.md Step 2a tests whether the drop is real before diagnosing it.

TDD: 58 analytics tests written red first (144 -> 202). test-runner Section 16x,
23 unconditional checks + self-test (247 -> 270; anti-erosion floor 228 -> 251).
tsc clean. All suites green: trends 300, brain 134, editions 72,
specifics-bank 45, contract-gate 33, hooks 191, tests 35, render 60.

Also closes the OKF phase-4 scope follow-up in docs/okf-ingestion/plan.md §8
(coord round 2026-07-25): phase 4 tracks the contract, parse is in scope, and
our claim on read_concept/navigate_bundle is withdrawn as unnecessary.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxvWAjte7vPcF79QeSRvRJ
This commit is contained in:
Kjell Tore Guttormsen 2026-07-25 20:50:45 +02:00
commit e2ad190dda
15 changed files with 1723 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

386
scripts/analytics/tests/stats.test.ts
View file

@ -7,6 +7,15 @@ import {
percentChange,
deviationsFromMean,
weightedOutOfNetworkPct,
median,
medianAbsoluteDeviation,
rollingBaseline,
readAgainstBaseline,
baselineByGroup,
buildBaselineBlock,
BASELINE_WINDOW,
MIN_BASELINE_N,
BAND_K,
} from "../src/utils/stats.js";
describe("stats", () => {
@ -193,4 +202,381 @@ describe("stats", () => {
assert.equal(result, 41.7);
});
});
/**
* Baseline engine (N17). At ~2 posts/week every reading is small-N, so the
* engine exists to answer one question honestly: is this number different
* from the operator's own normal, or is it noise? Two disciplines are under
* test — ROBUSTNESS (median/MAD, not mean/stddev, because one viral post
* would otherwise define "normal") and REFUSAL (below the minimum N there is
* no verdict at all, rather than a confident reading of nothing).
*/
describe("median", () => {
test("should return the middle value for an odd count", () => {
assert.equal(median([30, 10, 20]), 20, "Must sort internally, not trust input order");
});
test("should average the two middle values for an even count", () => {
assert.equal(median([10, 20, 30, 40]), 25);
});
test("should return undefined for an empty list, never 0", () => {
// Same discipline as weightedOutOfNetworkPct: absent data stays absent.
assert.equal(median([]), undefined);
});
test("should be unmoved by a single extreme outlier", () => {
// The whole reason baseline framing uses median: mean([100,110,120,130,10000])
// is 2092 — a "baseline" no ordinary post could ever reach.
assert.equal(median([100, 110, 120, 130, 10000]), 120);
assert.equal(mean([100, 110, 120, 130, 10000]), 2092);
});
});
describe("medianAbsoluteDeviation", () => {
test("should return the median of absolute deviations from the median", () => {
// median = 20; deviations = [10, 0, 10] → MAD = 10
assert.equal(medianAbsoluteDeviation([10, 20, 30]), 10);
});
test("should return 0 for identical values", () => {
assert.equal(medianAbsoluteDeviation([50, 50, 50]), 0);
});
test("should return undefined for an empty list", () => {
assert.equal(medianAbsoluteDeviation([]), undefined);
});
test("should stay small when one outlier inflates the standard deviation", () => {
// stddev is dragged to ~3950 by the outlier; MAD stays at 10, which is
// what an honest "normal spread" band needs.
const values = [100, 110, 120, 130, 10000];
assert.equal(medianAbsoluteDeviation(values), 10);
assert.ok(standardDeviation(values) > 1000, "stddev is outlier-dominated by contrast");
});
});
describe("rollingBaseline", () => {
test("should refuse a verdict below the minimum N", () => {
const result = rollingBaseline([100, 200, 300]);
assert.equal(result.status, "insufficient-data", "3 posts cannot support a verdict");
if (result.status !== "insufficient-data") return;
assert.equal(result.n, 3);
assert.equal(result.required, MIN_BASELINE_N);
assert.ok(result.reason.length > 0, "Refusal must carry an operator-facing reason");
});
test("should refuse on an empty history rather than report a zero baseline", () => {
const result = rollingBaseline([]);
assert.equal(result.status, "insufficient-data");
if (result.status !== "insufficient-data") return;
assert.equal(result.n, 0);
});
test("should produce a median and a band once the minimum N is met", () => {
// median = 120, MAD = 10 → band = [110, 130] at k=1
const result = rollingBaseline([100, 110, 120, 130, 140]);
assert.equal(result.status, "ok");
if (result.status !== "ok") return;
assert.equal(result.n, 5);
assert.equal(result.median, 120);
assert.equal(result.mad, 10);
assert.deepEqual(result.band, { low: 110, high: 130 });
});
test("should keep only the trailing window when history exceeds it", () => {
// 12 values oldest→newest; the window is the last 10, so the two oldest
// (1 and 2) must not pull the median down.
const values = [1, 2, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190];
const result = rollingBaseline(values);
assert.equal(result.status, "ok");
if (result.status !== "ok") return;
assert.equal(result.n, BASELINE_WINDOW);
assert.equal(result.window, BASELINE_WINDOW);
assert.equal(result.median, 145, "Median of the last 10, not of all 12");
});
test("should treat the input as oldest-first and never reorder it", () => {
// Descending input: if the implementation sorted by value it would keep
// the wrong ten. The last 10 here are 100 down to 10 → median 55.
const values = [200, 190, 100, 90, 80, 70, 60, 50, 40, 30, 20, 10];
const result = rollingBaseline(values);
assert.equal(result.status, "ok");
if (result.status !== "ok") return;
assert.equal(result.median, 55, "Window is positional (recency), the median is value-sorted");
});
test("should floor the band at 0 for a naturally non-negative metric", () => {
// median 20, MAD 15 → arithmetic low is 5; with a wider spread it would
// go negative, which is meaningless for impressions or engagement rate.
const result = rollingBaseline([0, 5, 20, 35, 90]);
assert.equal(result.status, "ok");
if (result.status !== "ok") return;
assert.ok(result.band.low >= 0, "A negative lower bound is not a readable floor");
});
test("should accept a configurable minimum N and window", () => {
const result = rollingBaseline([10, 20, 30], { minN: 3, window: 3 });
assert.equal(result.status, "ok", "The threshold is a documented default, not a hard law");
if (result.status !== "ok") return;
assert.equal(result.median, 20);
});
test("should widen the band with a larger k", () => {
const narrow = rollingBaseline([100, 110, 120, 130, 140]);
const wide = rollingBaseline([100, 110, 120, 130, 140], { k: 2 });
assert.equal(narrow.status, "ok");
assert.equal(wide.status, "ok");
if (narrow.status !== "ok" || wide.status !== "ok") return;
assert.equal(BAND_K, 1, "Default k is documented as 1");
assert.equal(wide.band.high, 140);
assert.ok(wide.band.high > narrow.band.high);
});
test("should give a zero-width band for a perfectly steady history", () => {
// MAD 0 is legitimate: the operator's numbers really are that steady.
const result = rollingBaseline([50, 50, 50, 50, 50]);
assert.equal(result.status, "ok");
if (result.status !== "ok") return;
assert.deepEqual(result.band, { low: 50, high: 50 });
assert.equal(result.mad, 0);
});
});
describe("readAgainstBaseline", () => {
const baseline = rollingBaseline([100, 110, 120, 130, 140]); // median 120, band [110,130]
test("should call a value inside the band normal variation, not a trend", () => {
// 128 is 6.7% above the median — the kind of number that reads as "up"
// week-over-week while being entirely inside the operator's own spread.
assert.equal(readAgainstBaseline(128, baseline), "within-band");
});
test("should call a value above the band above baseline", () => {
assert.equal(readAgainstBaseline(400, baseline), "above-band");
});
test("should call a value below the band below baseline", () => {
assert.equal(readAgainstBaseline(40, baseline), "below-band");
});
test("should treat the band edges as inside the band", () => {
assert.equal(readAgainstBaseline(110, baseline), "within-band");
assert.equal(readAgainstBaseline(130, baseline), "within-band");
});
test("should return no-verdict when the baseline itself was refused", () => {
const refused = rollingBaseline([100, 200]);
assert.equal(
readAgainstBaseline(5000, refused),
"no-verdict",
"A refused baseline must not be readable as a verdict by the caller"
);
});
});
describe("baselineByGroup", () => {
// Format and pillar are not stored in analytics (they live in the post
// queue), so the engine takes a label resolver rather than assuming a
// field — the same join the newsletter calibration uses.
const p = (group: string, impressions: number) => ({ group, impressions });
const items = [
...Array.from({ length: 6 }, (_, i) => p("text", 100 + i * 10)),
...Array.from({ length: 2 }, (_, i) => p("carousel", 900 + i * 100)),
];
test("should compute one baseline per group", () => {
const result = baselineByGroup(
items,
(it) => it.group,
(it) => it.impressions
);
assert.equal(result.size, 2);
assert.equal(result.get("text")?.status, "ok");
});
test("should refuse per group, not globally", () => {
// 8 posts in total would clear the threshold; carousel alone has 2 and
// must be refused on its own count. This is the failure the plan names:
// a verdict about a format the operator has barely used.
const result = baselineByGroup(
items,
(it) => it.group,
(it) => it.impressions
);
const carousel = result.get("carousel");
assert.equal(carousel?.status, "insufficient-data");
if (carousel?.status !== "insufficient-data") return;
assert.equal(carousel.n, 2);
});
test("should skip items whose label is undefined rather than bucket them", () => {
// An unlabelled post (no queue entry) is not a group called "undefined".
const mixed = [...items, { group: undefined as unknown as string, impressions: 5000 }];
const result = baselineByGroup(
mixed,
(it) => it.group,
(it) => it.impressions
);
assert.equal(result.size, 2, "No phantom group for unlabelled posts");
});
test("should skip items whose value is not a finite number", () => {
const withGap = [...items, p("text", NaN)];
const result = baselineByGroup(
withGap,
(it) => it.group,
(it) => it.impressions
);
const text = result.get("text");
assert.equal(text?.status, "ok");
if (text?.status !== "ok") return;
assert.equal(text.n, 6, "The NaN row must not count toward N");
});
});
describe("buildBaselineBlock", () => {
const post = (publishedDate: string, impressions: number, engagementRate = 5) => ({
publishedDate,
metrics: { impressions, engagementRate },
});
const history = [
post("2026-01-05", 100, 4.0),
post("2026-01-07", 110, 4.5),
post("2026-01-12", 120, 5.0),
post("2026-01-14", 130, 5.5),
post("2026-01-19", 140, 6.0),
];
test("should carry a baseline for both headline metrics", () => {
const block = buildBaselineBlock(history, "2026-W05");
assert.equal(block.impressions.status, "ok");
assert.equal(block.engagementRate.status, "ok");
if (block.impressions.status !== "ok") return;
assert.equal(block.impressions.median, 120);
});
test("should label the period whose posts are excluded", () => {
const block = buildBaselineBlock(history, "2026-W05");
assert.equal(block.historyBefore, "2026-W05");
assert.equal(block.window, BASELINE_WINDOW);
assert.equal(block.required, MIN_BASELINE_N);
});
test("should refuse both metrics when history is too short", () => {
const block = buildBaselineBlock(history.slice(0, 2), "2026-W05");
assert.equal(block.impressions.status, "insufficient-data");
assert.equal(block.engagementRate.status, "insufficient-data");
});
test("should order history by publish date before taking the window", () => {
// Same five posts shuffled, plus six older ones so the window bites. If
// the block trusted file order, the window would hold the wrong posts.
const older = [
post("2025-12-01", 5),
post("2025-12-03", 6),
post("2025-12-05", 7),
post("2025-12-08", 8),
post("2025-12-10", 9),
post("2025-12-12", 10),
];
const shuffled = [history[4], older[2], history[0], older[5], history[2], older[0],
history[1], older[4], history[3], older[1], older[3]];
const block = buildBaselineBlock(shuffled, "2026-W05");
assert.equal(block.impressions.status, "ok");
if (block.impressions.status !== "ok") return;
// Window = last 10 by DATE: drops 2025-12-01 (5), keeps 6..10 and 100..140.
// Median of [6,7,8,9,10,100,110,120,130,140] = (10+100)/2 = 55.
assert.equal(block.impressions.median, 55, "Window must follow publish date, not array order");
});
test("should return a block for an empty history rather than throwing", () => {
const block = buildBaselineBlock([], "2026-W05");
assert.equal(block.impressions.status, "insufficient-data");
if (block.impressions.status !== "insufficient-data") return;
assert.equal(block.impressions.n, 0);
});
/**
* The reading is computed HERE, in code, not left to whoever renders the
* report. A renderer eyeballing whether 128 sits inside [110, 130] is the
* over-read the whole engine exists to prevent.
*/
describe("period reading", () => {
test("should read the period's own median against the band", () => {
// Baseline median 120, band [110, 130]. Period median 125 → inside.
const block = buildBaselineBlock(history, "2026-W05", [
post("2026-02-02", 120),
post("2026-02-04", 130),
]);
assert.equal(block.period.impressions.median, 125);
assert.equal(block.period.impressions.reading, "within-band");
});
test("should compare the period MEDIAN, not its mean", () => {
// Mean of [100, 10000] is 5050 — "above-band" on one viral post. The
// median is 5050's honest counterpart: 5050 vs median 5050… no: the
// median of two values is their midpoint, so use three posts where the
// mean and median genuinely disagree: [100, 110, 10000] → mean 3403,
// median 110, which is inside the band. The band must not fire.
const block = buildBaselineBlock(history, "2026-W05", [
post("2026-02-02", 100),
post("2026-02-04", 110),
post("2026-02-06", 10000),
]);
assert.equal(block.period.impressions.median, 110);
assert.equal(
block.period.impressions.reading,
"within-band",
"One viral post must not turn an ordinary week into a trend"
);
});
test("should read a genuinely high period as above the band", () => {
const block = buildBaselineBlock(history, "2026-W05", [
post("2026-02-02", 900),
post("2026-02-04", 1100),
]);
assert.equal(block.period.impressions.reading, "above-band");
});
test("should read a genuinely low period as below the band", () => {
const block = buildBaselineBlock(history, "2026-W05", [
post("2026-02-02", 10),
post("2026-02-04", 20),
]);
assert.equal(block.period.impressions.reading, "below-band");
});
test("should give no verdict when the baseline was refused", () => {
const block = buildBaselineBlock(history.slice(0, 2), "2026-W05", [
post("2026-02-02", 99999),
]);
assert.equal(
block.period.impressions.reading,
"no-verdict",
"A thin baseline cannot license a verdict about a big number"
);
});
test("should give no verdict and no median for a period with no posts", () => {
const block = buildBaselineBlock(history, "2026-W05", []);
assert.equal(block.period.impressions.median, undefined);
assert.equal(block.period.impressions.reading, "no-verdict");
});
test("should read engagement rate independently of impressions", () => {
// Baseline engagement median 5.0, MAD 0.5 → band [4.5, 5.5].
// Impressions ordinary, engagement clearly above: the two readings must
// not be collapsed into one verdict about "the week".
const block = buildBaselineBlock(history, "2026-W05", [
post("2026-02-02", 120, 9.0),
post("2026-02-04", 120, 9.4),
]);
assert.equal(block.period.impressions.reading, "within-band");
assert.equal(block.period.engagementRate.reading, "above-band");
});
});
});
});