feat(linkedin-studio): N17 — baseline-motor (median + variansbånd + minimum-N-refusal) [skip-docs]
Every reading now leads with "vs your own baseline", and no verdict is given when N is too small to carry one. - stats.ts: median + medianAbsoluteDeviation (robust pair; mean/stddev stay for the alert engine, which wants outlier sensitivity), rollingBaseline with a 10-post positional window, median ± 1·MAD band floored at 0, and a typed insufficient-data refusal below MIN_BASELINE_N=5. readAgainstBaseline returns above/within/below-band, or no-verdict when the baseline was refused. - baselineByGroup + buildBaselineBlock: per-format/per-pillar baselines, each judged on its own N; the reported period is excluded from its own baseline and compared on its median, not its mean. - queue-join.ts (new): read-only date join supplying format/pillar from the post queue. Every ambiguity resolves to unlabelled, an entry labels at most one post, and a missing/broken queue degrades to no labels. - weekly/monthly reports attach the block unconditionally (refusal included); optional in the types, so pre-N17 reports load unchanged. - CLI: report output leads with the baseline; new `baseline [--by format|pillar]` verb with coverage reporting. - report.md leads with baseline framing and prints the code's reading rather than judging the band by eye; WoW loses to the baseline on disagreement. analyze.md Step 2a tests whether the drop is real before diagnosing it. TDD: 58 analytics tests written red first (144 -> 202). test-runner Section 16x, 23 unconditional checks + self-test (247 -> 270; anti-erosion floor 228 -> 251). tsc clean. All suites green: trends 300, brain 134, editions 72, specifics-bank 45, contract-gate 33, hooks 191, tests 35, render 60. Also closes the OKF phase-4 scope follow-up in docs/okf-ingestion/plan.md §8 (coord round 2026-07-25): phase 4 tracks the contract, parse is in scope, and our claim on read_concept/navigate_bundle is withdrawn as unnecessary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxvWAjte7vPcF79QeSRvRJ
This commit is contained in:
parent
63506f7d5c
commit
e2ad190dda
15 changed files with 1723 additions and 14 deletions
11
CHANGELOG.md
11
CHANGELOG.md
|
|
@ -87,6 +87,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
- **Boundary map brought current (E#9)** — the README boundaries now state all three honestly: dwell **explicitly unmeasurable** (internal to LinkedIn, no count to transcribe, no API), saves partner-gated (`POST_SAVE` on `/memberCreatorPostAnalytics` from v202604), and reach native-but-not-exported with export status **unverified**.
|
||||
- analytics suite 119 → 144 (25 new tests: percent contract, either-half resolution, split cross-check, weighted roll-up, backward-compat); test-runner 232 → 247 (Section 16w: 15 unconditional greps + self-test; anti-erosion floor 213 → 228); `tsc` clean.
|
||||
|
||||
### Added — baseline engine: your own normal, and a refusal when N is too small (N17 / E#5)
|
||||
|
||||
- **Every reading leads with "vs your own baseline" (E#5)** — at a normal publishing cadence a week holds two or three posts, so an absolute total answers nothing on its own and a week-over-week percentage answers worse than nothing: it swings on sample composition and reads as a trend. The weekly and monthly reports now carry a `baseline` block built from the operator's own recent history, and `/linkedin:report` presents it **first**, absolutes second.
|
||||
- **Median and MAD, never mean and standard deviation** — one viral post would otherwise define "normal" as a number no ordinary post can reach, and every ordinary week after it would read as a decline. `median()` + `medianAbsoluteDeviation()` are the robust pair; `mean`/`standardDeviation` stay untouched for the alert engine, which *wants* outlier sensitivity. An empty history yields `undefined`, never a fabricated 0.
|
||||
- **A normal-range band instead of a bare number** — `median ± 1·MAD` (k configurable), floored at 0 for naturally non-negative metrics. `readAgainstBaseline()` returns `above-band` / `within-band` / `below-band`, and `within-band` is the load-bearing answer: a value 7 % above the median looks like momentum in any week-over-week percentage and is ordinary variation. The reading is computed **in code**, so a renderer never judges the band by eye.
|
||||
- **Minimum-N refusal — no verdict on noise** — below `MIN_BASELINE_N` (5, documented and overridable) the engine returns a typed `insufficient-data` refusal carrying `n`, `required` and an operator-facing reason, and `readAgainstBaseline` degrades to `no-verdict`. With fewer than five values a median moves further when one post enters than any real change would move it. The report prints the refusal verbatim, keeps showing the absolute numbers **without** a verdict, and may not substitute a week-over-week percentage for the missing one. An operator who has just started publishing gets this refusal for several weeks — that is the correct output.
|
||||
- **The baseline excludes the period it judges** — history is taken strictly *before* the reported week/month (`historyBefore`), so the comparison is never partly with itself, and the period is compared on its own **median**, not its mean. The window is counted in **posts** (last 10), not days: a calendar window silently shrinks to three posts in a slow month without the operator seeing it happen.
|
||||
- **Per-format and per-pillar baselines, each judged on its own N** — format and pillar are not analytics data (they live in the post queue), so a new read-only date join supplies them (`queue-join.ts`, ±2 days for a reschedule, hook-preview tiebreak). Every ambiguity resolves to **unlabelled** and an entry labels at most one post — a guessed label would poison the baseline it is measured against. New `baseline [--by format|pillar]` CLI verb prints the overall baseline plus per-group ones, reports its own coverage, and refuses per group: a format used twice gets no verdict even when the overall history is long.
|
||||
- **`/linkedin:analyze` tests the problem before diagnosing it** — a new Step 2a reads the band first. `within-band` says plainly that this looks like ordinary variation rather than a drop (and asks whether to continue) instead of manufacturing a diagnosis; `no-verdict` continues on self-report only, labelled provisional. Step 5's severity assessment now measures against the band rather than a remembered peak.
|
||||
- Additive and backward-compatible: `baseline` is optional in the report types, so pre-N17 reports load unchanged (and a consumer says "no baseline in this report" rather than assuming one). analytics suite 144 → 202 (58 new tests: median/MAD, window, band, refusal, grouping, period reading, queue join, report wiring); test-runner 247 → 270 (Section 16x: 23 unconditional greps + self-test; anti-erosion floor 228 → 251); `tsc` clean.
|
||||
|
||||
### Built feedback (awaiting consumer-side proof)
|
||||
|
||||
- **MR-F9** (demand-sweep «innenfra og ut») — built; the (a)/(b)/(c) evidence gate is a runtime demonstration, proven consumer-side (plugin agents don't resolve in the dev repo).
|
||||
|
|
|
|||
|
|
@ -38,7 +38,34 @@ Use AskUserQuestion to understand the situation:
|
|||
|
||||
## Step 2: Gather Data
|
||||
|
||||
If imported analytics data exists (`${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/analytics/`), delegate audience-pattern discovery to the `analytics-interpreter` agent (interpret mode) — invoke it via `Task` with `subagent_type: linkedin-studio:analytics-interpreter` (foreground, from this command layer) — to ground the diagnosis in what the data actually shows before relying on self-report.
|
||||
**Step 2a — first, establish whether there is a problem at all.** This command is
|
||||
invoked because something *feels* wrong, and the operator's own framing ("my reach
|
||||
dropped") is a hypothesis, not a measurement. Diagnosing a drop that never happened
|
||||
sends them chasing a phantom and teaches them to distrust their own numbers.
|
||||
|
||||
If imported analytics exist, generate the current report and read its `baseline`
|
||||
block (see `/linkedin:report` Step 4):
|
||||
|
||||
```bash
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/analytics/node_modules/.bin/tsx" "${CLAUDE_PLUGIN_ROOT}/scripts/analytics/src/cli.ts" report
|
||||
```
|
||||
|
||||
Read `baseline.period.<metric>.reading` and lead the diagnosis with it:
|
||||
|
||||
- **`within-band`** — the numbers are inside the operator's own normal range. Say so
|
||||
before going further: *"Your last posts are inside your normal range (median X,
|
||||
normal range Y–Z). This looks like ordinary variation rather than a drop."* Then ask
|
||||
whether they still want the diagnostic pass. Often the honest answer is that nothing
|
||||
is wrong and the fix is to keep publishing — do not manufacture a diagnosis to
|
||||
justify the command having been run.
|
||||
- **`below-band`** — the drop is real and measured. Proceed, and use the band figures
|
||||
as the size of the problem instead of a remembered percentage.
|
||||
- **`no-verdict`** — print `baseline.<metric>.reason` and say plainly that there is not
|
||||
enough history to tell a drop from variation. Continue on **self-report only**, and
|
||||
label the diagnosis as provisional. Never upgrade a refusal into a verdict because
|
||||
the operator sounds worried.
|
||||
|
||||
If imported analytics exist (`${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/analytics/`), delegate audience-pattern discovery to the `analytics-interpreter` agent (interpret mode) — invoke it via `Task` with `subagent_type: linkedin-studio:analytics-interpreter` (foreground, from this command layer) — to ground the diagnosis in what the data actually shows before relying on self-report.
|
||||
|
||||
Based on their answer, ask relevant follow-up questions:
|
||||
|
||||
|
|
@ -150,7 +177,11 @@ Run through this checklist:
|
|||
|
||||
## Step 5: Reach Drop Severity Assessment
|
||||
|
||||
Based on how much reach dropped:
|
||||
Based on how much reach dropped. **Measure the drop against the baseline band from
|
||||
Step 2a, not against the single best week the operator remembers** — a percentage
|
||||
computed from a remembered peak is a comparison with an outlier, and it will class
|
||||
ordinary variation as suppression. When the band says `within-band`, the severity is
|
||||
"normal fluctuation" regardless of what the percentage says.
|
||||
|
||||
### Down <25%
|
||||
**Diagnosis:** Normal fluctuation
|
||||
|
|
|
|||
|
|
@ -142,10 +142,41 @@ The report contains:
|
|||
- **week**: ISO week identifier
|
||||
- **generatedAt**: ISO timestamp when the report was generated
|
||||
- **summary**: Totals and averages — `totalPosts`, `totalImpressions`, `totalReactions`, `totalComments`, `totalShares`, `totalClicks`, optional `totalSaves` (manual entry only), optional `avgOutOfNetworkPct` (manual entry only — the impressions-weighted out-of-network share), `avgEngagementRate`, `avgImpressionsPerPost`
|
||||
- **baseline**: The operator's own normal, from posts published **before** this week —
|
||||
`window`, `required`, `historyBefore`, plus `impressions` and `engagementRate` (each
|
||||
either `{status: "ok", n, median, mad, band}` or
|
||||
`{status: "insufficient-data", n, required, reason}`), and `period` carrying this
|
||||
week's own median per metric with a `reading` of `above-band` / `within-band` /
|
||||
`below-band` / `no-verdict`. **This is what the report leads with (Step 6).**
|
||||
- **topPerformers** / **underperformers**: Best / weakest posts by engagement rate
|
||||
- **trends**: Week-over-week change — `impressionsTrend`, `engagementTrend`, `comparedTo`, `percentChange`
|
||||
- **alerts**: Anomalies and significant events (intra-week + week-over-week)
|
||||
|
||||
**Week-over-week vs baseline — which to trust.** `trends.percentChange` compares two
|
||||
adjacent weeks, so at two or three posts a week it swings wildly on sample composition
|
||||
alone; `baseline` compares against the operator's own spread. When the two disagree —
|
||||
a +38% week that reads `within-band` — **the baseline wins**, and say so explicitly:
|
||||
the percentage is real arithmetic on a sample too small to carry a conclusion.
|
||||
|
||||
## Step 4b: Per-format and per-pillar baselines (optional, when the queue has labels)
|
||||
|
||||
Format and pillar are not analytics data — they live in the post queue — so a separate
|
||||
surface joins them:
|
||||
|
||||
```bash
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/analytics/node_modules/.bin/tsx" "${CLAUDE_PLUGIN_ROOT}/scripts/analytics/src/cli.ts" baseline
|
||||
```
|
||||
|
||||
It prints an overall baseline plus one per format and per pillar, and reports its own
|
||||
coverage (how many posts it could not label). Use it to answer "is carousel actually
|
||||
working for me?" — but respect the per-group refusals: **each group is judged on its
|
||||
own N**, so a format used twice gets no verdict even when the overall history is long.
|
||||
That refusal is the honest answer to a question the operator will still want answered;
|
||||
do not fill the gap with the overall baseline as a proxy for the group.
|
||||
|
||||
Skip this step when the join reports no labels at all — say the grouped view needs
|
||||
queue entries rather than presenting an empty table.
|
||||
|
||||
## Step 5: Run Trend Analysis
|
||||
|
||||
Get additional context with trend analysis:
|
||||
|
|
@ -253,12 +284,55 @@ Your post on [topic] achieved 12,500 impressions — a personal best!
|
|||
|
||||
## Step 6: Present Formatted Report
|
||||
|
||||
**The report LEADS with the operator's own baseline, and the absolute numbers come
|
||||
second.** At a normal publishing cadence a single week holds two or three posts, so
|
||||
an absolute total answers no question on its own — "1,240 impressions" is only
|
||||
information once you know whether that is ordinary for this operator. Present the
|
||||
baseline block first, every time, and never reorder these two sections.
|
||||
|
||||
**Do not compute the verdict yourself.** The report JSON's `baseline` block already
|
||||
carries it, decided in code: `baseline.period.<metric>.reading` is one of
|
||||
`above-band` / `within-band` / `below-band` / `no-verdict`. Print what it says.
|
||||
Judging by eye whether a number sits inside the band is the over-read this block
|
||||
exists to prevent — a value 7% above the median looks like a trend and is not one.
|
||||
|
||||
Format the data into a readable report using this template:
|
||||
|
||||
```
|
||||
# LinkedIn Performance Report
|
||||
## Week {week}
|
||||
|
||||
### 📐 Vs Your Own Baseline
|
||||
|
||||
Baseline = your last {baseline.window} posts published **before** this {week/month}
|
||||
(n={baseline.impressions.n}). Median and normal range, not mean — one viral post
|
||||
must not redefine "normal".
|
||||
|
||||
| Metric | This week (median) | Your baseline | Normal range | Reads as |
|
||||
|--------|--------------------|---------------|--------------|----------|
|
||||
| Impressions/post | {baseline.period.impressions.median} | {baseline.impressions.median} | {band.low}–{band.high} | {reading} |
|
||||
| Engagement rate | {baseline.period.engagementRate.median}% | {baseline.engagementRate.median}% | {band.low}–{band.high}% | {reading} |
|
||||
|
||||
Render `reading` in words, not as the raw token:
|
||||
- `above-band` → **above your normal range** — a real move up, worth explaining
|
||||
- `below-band` → **below your normal range** — a real move down, worth diagnosing
|
||||
- `within-band` → **inside your normal range** — normal variation, not a trend.
|
||||
Say this plainly and do not dress it up as momentum in either direction.
|
||||
- `no-verdict` → see the refusal rule below
|
||||
|
||||
**When `baseline.<metric>.status` is `insufficient-data`: refuse the verdict.** Print
|
||||
`baseline.<metric>.reason` verbatim and state that this {week/month}'s numbers are
|
||||
reported **without** a verdict. Do not substitute a week-over-week percentage as a
|
||||
stand-in, do not call it a trend, and do not soften the refusal into a hedge like
|
||||
"early signs suggest" — with fewer than {baseline.required} prior posts there is no
|
||||
signal to be early about. The absolute numbers below are still shown; only the
|
||||
verdict is withheld. An operator who has just started publishing gets this refusal
|
||||
for several weeks, and that is the correct output, not a failure of the report.
|
||||
|
||||
**If the report JSON has no `baseline` key at all**, it was generated before
|
||||
baselines existed — say so, and offer to re-run Step 3 rather than reading the
|
||||
absolutes as if they had been framed.
|
||||
|
||||
### 📊 Key Metrics
|
||||
|
||||
| Metric | Total | Average per Post | vs. Last Week |
|
||||
|
|
|
|||
|
|
@ -261,10 +261,19 @@ the other's approval. The guard plan's own interop blocker (Python↔Node) is tr
|
|||
when it starts?~~ **Answered 2026-07-20 (trinn E, felles §6): phase 4 delivers the layer-1 primitives,
|
||||
not `materialize_bundle` parity** — no Node repo in the set runs door A (verified here: the only Python
|
||||
in this repo is `hooks/scripts/compile-hooks.py`, a build-time script never on a data path). Our register
|
||||
status therefore stays **`planned`**, not `blocked on phase 4 scope`. One follow-up outstanding: layer 1
|
||||
grew after the question was asked (`read_concept`, `navigate_bundle`, `routeLevel`), so we asked whether
|
||||
phase 4 tracks the set as it stands or as it evolves — the reader is decision-relevant for us, since our
|
||||
dependency is `serialize ∘ parse = identity`, not the writer alone.
|
||||
status therefore stays **`planned`**, not `blocked on phase 4 scope`. The follow-up (layer 1 grew after
|
||||
the question was asked — `read_concept`, `navigate_bundle`, `routeLevel`) is now **closed, 2026-07-25**:
|
||||
phase 4 tracks the **contract** (owned by the marketplace catalog), neither okr's evolving layer 1 nor a
|
||||
frozen snapshot of it, so a function appearing in layer 1 does not by itself enter scope. Our dependency
|
||||
is unaffected and turned out **narrower than the question implied**: `parse ∘ serialize = identity` is
|
||||
asserted per FILE (`scripts/brain/tests/ingest.test.ts:102`), and bundle navigation here is our own
|
||||
scaffold code, never a library call — so frontmatter emit/parse (their deliverable 4, which names parse
|
||||
alongside emit) is sufficient and we withdrew any claim on `read_concept` / `navigate_bundle`. Two
|
||||
requirements were filed instead, before their scope freeze: `write_index(mode="preserve-unowned")` must
|
||||
leave an existing index **byte-untouched** (ours never opens the file — `scripts/brain/src/scaffold.ts:133-141`),
|
||||
and an absent vocabulary must mean **"do not snap"**, not snap-to-`Dokument`. Recorded non-goal from their
|
||||
side: no phase-4 deliverable targets this repo beyond the check/index tools tolerating our `brain/` hub
|
||||
without complaining about `ingest/published/` — our `ingest/` grammar is never normalized (§5).
|
||||
|
||||
## 9. References
|
||||
|
||||
|
|
|
|||
|
|
@ -1,18 +1,28 @@
|
|||
import { parseLinkedInCSV } from "./parsers/csv-parser.js";
|
||||
import {
|
||||
getAnalyticsRoot,
|
||||
getDataRoot,
|
||||
ensureDirectories,
|
||||
saveBatch,
|
||||
loadAllPosts,
|
||||
} from "./utils/storage.js";
|
||||
import { detectAlerts } from "./utils/alerts.js";
|
||||
import { mean, standardDeviation, weightedOutOfNetworkPct } from "./utils/stats.js";
|
||||
import {
|
||||
mean,
|
||||
standardDeviation,
|
||||
weightedOutOfNetworkPct,
|
||||
rollingBaseline,
|
||||
baselineByGroup,
|
||||
BASELINE_WINDOW,
|
||||
MIN_BASELINE_N,
|
||||
} from "./utils/stats.js";
|
||||
import { loadQueueEntries, labelPosts } from "./utils/queue-join.js";
|
||||
import { generateWeeklyReport, getCurrentISOWeek } from "./reports/weekly.js";
|
||||
import { generateHeatmap } from "./reports/heatmap.js";
|
||||
import { generateMonthlyReport } from "./reports/monthly.js";
|
||||
import { join } from "node:path";
|
||||
import { existsSync } from "node:fs";
|
||||
import type { RankableMetric } from "./models/types.js";
|
||||
import type { RankableMetric, Baseline, BaselineBlock, BaselineReading } from "./models/types.js";
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
const command = args[0];
|
||||
|
|
@ -43,6 +53,61 @@ function round1(value: number): number {
|
|||
return Math.round(value * 10) / 10;
|
||||
}
|
||||
|
||||
/** Plain-language rendering of a band reading (N17). */
|
||||
const READING_TEXT: Record<BaselineReading, string> = {
|
||||
"above-band": "ABOVE your normal range",
|
||||
"within-band": "inside your normal range (normal variation, not a trend)",
|
||||
"below-band": "BELOW your normal range",
|
||||
"no-verdict": "no verdict",
|
||||
};
|
||||
|
||||
/**
|
||||
* Print the baseline block FIRST, before absolute totals (N17).
|
||||
*
|
||||
* Ordering is the point: the operator's own normal is the only frame in which a
|
||||
* weekly number means anything at ~2 posts/week, so it leads and the absolutes
|
||||
* follow. On a refusal this prints the refusal — the absolutes still get shown
|
||||
* below, but with no verdict attached to them.
|
||||
*/
|
||||
function printBaseline(baseline: BaselineBlock | undefined, unit: "week" | "month") {
|
||||
if (!baseline) return; // report generated before baselines existed
|
||||
|
||||
console.log(`Vs Your Own Baseline (last ${baseline.window} posts before this ${unit})`);
|
||||
console.log("─────────────────────────────────────");
|
||||
|
||||
const line = (
|
||||
label: string,
|
||||
base: BaselineBlock["impressions"],
|
||||
period: { median?: number; reading: BaselineReading },
|
||||
fmt: (value: number) => string
|
||||
) => {
|
||||
if (base.status !== "ok") {
|
||||
console.log(`${label} ${base.reason}`);
|
||||
return;
|
||||
}
|
||||
const own = period.median !== undefined ? fmt(period.median) : "no posts";
|
||||
console.log(
|
||||
`${label} this ${unit} ${own} vs baseline ${fmt(base.median)} ` +
|
||||
`(normal range ${fmt(base.band.low)}–${fmt(base.band.high)}, n=${base.n})`
|
||||
);
|
||||
console.log(`${" ".repeat(label.length)} → ${READING_TEXT[period.reading]}`);
|
||||
};
|
||||
|
||||
line(
|
||||
"Impressions/post:",
|
||||
baseline.impressions,
|
||||
baseline.period.impressions,
|
||||
(value) => Math.round(value).toLocaleString()
|
||||
);
|
||||
line(
|
||||
"Engagement rate: ",
|
||||
baseline.engagementRate,
|
||||
baseline.period.engagementRate,
|
||||
(value) => `${value.toFixed(2)}%`
|
||||
);
|
||||
console.log();
|
||||
}
|
||||
|
||||
function printUsage() {
|
||||
console.log(`
|
||||
LinkedIn Analytics CLI
|
||||
|
|
@ -53,9 +118,11 @@ Usage:
|
|||
node --import tsx src/cli.ts report --month YYYY-MM Generate monthly report with MoM comparison
|
||||
node --import tsx src/cli.ts trends [--period P] [--metric M] Show trends and alerts
|
||||
node --import tsx src/cli.ts heatmap Day-of-week performance matrix
|
||||
node --import tsx src/cli.ts baseline [--by format|pillar] Your own normal, per format and pillar
|
||||
|
||||
Options:
|
||||
--week W ISO week (e.g., 2026-W05), defaults to current week
|
||||
--by G Group the baseline by "format" or "pillar" (default: both)
|
||||
--period P Time period: "week" | "month" | "quarter" | "all" (default: "month")
|
||||
--metric M Metric to analyze: "impressions" | "reactions" | "comments" | "shares" | "clicks" | "engagementRate" (default: "impressions")
|
||||
|
||||
|
|
@ -155,6 +222,8 @@ async function handleReport(root: string, args: string[]) {
|
|||
console.log(`Generated at: ${new Date(report.generatedAt).toLocaleString()}`);
|
||||
console.log();
|
||||
|
||||
printBaseline(report.baseline, "week");
|
||||
|
||||
console.log("Summary");
|
||||
console.log("─────────────────────────────────────");
|
||||
console.log(`Total posts: ${report.summary.totalPosts}`);
|
||||
|
|
@ -351,6 +420,8 @@ async function handleMonthlyReport(root: string, month: string) {
|
|||
console.log(`Generated at: ${new Date(report.generatedAt).toLocaleString()}`);
|
||||
console.log();
|
||||
|
||||
printBaseline(report.baseline, "month");
|
||||
|
||||
console.log("Summary");
|
||||
console.log("─────────────────────────────────────");
|
||||
const s = report.summary;
|
||||
|
|
@ -470,6 +541,98 @@ async function handleHeatmap(root: string) {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-format and per-pillar baselines (N17).
|
||||
*
|
||||
* Format and pillar are not analytics data — they come from the post queue via
|
||||
* a date join (see queue-join.ts), so this surface states its own coverage: how
|
||||
* many posts it could label, and how many it could not. Each group is judged on
|
||||
* its own N, so a format the operator has used twice gets a refusal even when
|
||||
* the overall history is long.
|
||||
*/
|
||||
async function handleBaseline(root: string, args: string[]) {
|
||||
const by = parseOption(args, "--by");
|
||||
if (by !== undefined && by !== "format" && by !== "pillar") {
|
||||
console.error(`Unknown grouping: ${by}. Use "format" or "pillar".`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const posts = loadAllPosts(root);
|
||||
if (posts.length === 0) {
|
||||
console.log("No analytics data imported yet — nothing to build a baseline from.");
|
||||
return;
|
||||
}
|
||||
|
||||
const queueFile = join(getDataRoot("drafts"), "queue.json");
|
||||
const entries = loadQueueEntries(queueFile);
|
||||
const labelled = labelPosts(
|
||||
[...posts].sort((a, b) => a.publishedDate.localeCompare(b.publishedDate)),
|
||||
entries
|
||||
);
|
||||
|
||||
console.log("\nYour Own Baseline");
|
||||
console.log("═════════════════════════════════════");
|
||||
console.log(`Posts: ${posts.length}`);
|
||||
console.log(`Window: last ${BASELINE_WINDOW} posts per group, ${MIN_BASELINE_N} required for a verdict`);
|
||||
|
||||
if (entries.length === 0) {
|
||||
console.log(
|
||||
`\nNo post queue at ${queueFile} — format and pillar are unknown, so only the\n` +
|
||||
`overall baseline is available. Grouped baselines need queue entries written by\n` +
|
||||
`the create commands.`
|
||||
);
|
||||
}
|
||||
|
||||
const groups: Array<"format" | "pillar"> = by ? [by] : ["format", "pillar"];
|
||||
|
||||
// Overall first: it is the one baseline that needs no join and can never be
|
||||
// wrong about which bucket a post belongs to.
|
||||
const overall = rollingBaseline(labelled.map((post) => post.metrics.impressions));
|
||||
console.log("\nOverall (impressions/post)");
|
||||
console.log("─────────────────────────────────────");
|
||||
printGroupBaseline("all posts", overall);
|
||||
|
||||
for (const group of groups) {
|
||||
const unlabelled = labelled.filter((post) => post[group] === undefined).length;
|
||||
const byGroup = baselineByGroup(
|
||||
labelled,
|
||||
(post) => post[group],
|
||||
(post) => post.metrics.impressions
|
||||
);
|
||||
|
||||
console.log(`\nBy ${group} (impressions/post)`);
|
||||
console.log("─────────────────────────────────────");
|
||||
if (byGroup.size === 0) {
|
||||
console.log(`No post carries a ${group} label — nothing to group.`);
|
||||
} else {
|
||||
for (const [name, baseline] of [...byGroup.entries()].sort(([a], [b]) => a.localeCompare(b))) {
|
||||
printGroupBaseline(name, baseline);
|
||||
}
|
||||
}
|
||||
if (unlabelled > 0) {
|
||||
console.log(
|
||||
`(${unlabelled} of ${labelled.length} posts have no ${group} label and are ` +
|
||||
`excluded from the groups above — they are counted in Overall.)`
|
||||
);
|
||||
}
|
||||
}
|
||||
console.log();
|
||||
}
|
||||
|
||||
/** One group's line — a median and band, or the refusal, never both. */
|
||||
function printGroupBaseline(name: string, baseline: Baseline) {
|
||||
const label = `${name}:`.padEnd(22);
|
||||
if (baseline.status !== "ok") {
|
||||
console.log(`${label}no verdict — ${baseline.n} post(s), ${baseline.required} required`);
|
||||
return;
|
||||
}
|
||||
console.log(
|
||||
`${label}median ${Math.round(baseline.median).toLocaleString()} ` +
|
||||
`(normal range ${Math.round(baseline.band.low).toLocaleString()}–` +
|
||||
`${Math.round(baseline.band.high).toLocaleString()}, n=${baseline.n})`
|
||||
);
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const root = getAnalyticsRoot();
|
||||
ensureDirectories(root);
|
||||
|
|
@ -487,6 +650,9 @@ async function main() {
|
|||
case "heatmap":
|
||||
await handleHeatmap(root);
|
||||
break;
|
||||
case "baseline":
|
||||
await handleBaseline(root, args);
|
||||
break;
|
||||
default:
|
||||
printUsage();
|
||||
process.exit(command ? 1 : 0);
|
||||
|
|
|
|||
|
|
@ -73,6 +73,13 @@ export interface WeeklyReport {
|
|||
avgEngagementRate: number;
|
||||
avgImpressionsPerPost: number;
|
||||
};
|
||||
/**
|
||||
* The operator's own baseline, from posts published before this week (N17).
|
||||
* Optional in the type because reports generated before N17 do not carry it —
|
||||
* a consumer must say "no baseline in this report" rather than assume one.
|
||||
* Newly generated reports always carry it, refusal included.
|
||||
*/
|
||||
baseline?: BaselineBlock;
|
||||
topPerformers: PostAnalytics[];
|
||||
underperformers: PostAnalytics[];
|
||||
trends: {
|
||||
|
|
@ -89,6 +96,75 @@ export interface WeeklyReport {
|
|||
|
||||
export type TrendDirection = "up" | "down" | "stable";
|
||||
|
||||
/**
|
||||
* A usable baseline: the operator's own recent normal for one metric, plus the
|
||||
* spread around it. Robust by construction — median and MAD, never mean and
|
||||
* standard deviation, because a single viral post would otherwise define
|
||||
* "normal" as a number no ordinary post can reach.
|
||||
*/
|
||||
export interface BaselineVerdict {
|
||||
status: "ok";
|
||||
n: number; // posts actually in the window (never a padded or assumed count)
|
||||
window: number; // window size the baseline was computed with
|
||||
median: number;
|
||||
mad: number; // median absolute deviation — the spread the band is built from
|
||||
band: { low: number; high: number }; // median ± k·MAD, lower bound floored at 0
|
||||
}
|
||||
|
||||
/**
|
||||
* The refusal. Below the minimum N a median moves further when one post enters
|
||||
* than any real week-over-week change would move it, so there is no verdict to
|
||||
* give — and saying so is the feature. Carries the numbers the operator needs
|
||||
* to know how far off a verdict is, never a value that could be misread as one.
|
||||
*/
|
||||
export interface BaselineRefusal {
|
||||
status: "insufficient-data";
|
||||
n: number;
|
||||
required: number;
|
||||
reason: string; // operator-facing sentence, safe to print verbatim
|
||||
}
|
||||
|
||||
export type Baseline = BaselineVerdict | BaselineRefusal;
|
||||
|
||||
/**
|
||||
* How one number reads against a baseline. `within-band` is the load-bearing
|
||||
* value: it is the honest answer for a number that a week-over-week percentage
|
||||
* would dress up as a trend. `no-verdict` propagates a refusal so a caller
|
||||
* cannot accidentally read one.
|
||||
*/
|
||||
export type BaselineReading = "above-band" | "within-band" | "below-band" | "no-verdict";
|
||||
|
||||
/**
|
||||
* Baseline block attached to a period report. Computed from posts published
|
||||
* STRICTLY BEFORE the reported period, so "this week vs your baseline" is a
|
||||
* real comparison rather than partly a comparison with itself.
|
||||
*/
|
||||
export interface BaselineBlock {
|
||||
/** Window size used (posts, not days — see BASELINE_WINDOW). */
|
||||
window: number;
|
||||
/** Minimum N required for a verdict. */
|
||||
required: number;
|
||||
/**
|
||||
* The reported period itself, whose posts are EXCLUDED from the baseline.
|
||||
* The baseline is history strictly before it — otherwise "this week vs your
|
||||
* baseline" would be partly a comparison with itself.
|
||||
*/
|
||||
historyBefore: string;
|
||||
impressions: Baseline;
|
||||
engagementRate: Baseline;
|
||||
/**
|
||||
* The reported period read against the baseline above — computed in code, so
|
||||
* a renderer never has to decide for itself whether a number sits inside the
|
||||
* band. `median` is the period's own median (undefined when it has no posts);
|
||||
* `reading` is `no-verdict` whenever the baseline was refused or the period
|
||||
* is empty.
|
||||
*/
|
||||
period: {
|
||||
impressions: { median?: number; reading: BaselineReading };
|
||||
engagementRate: { median?: number; reading: BaselineReading };
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Metric keys that are always present and numeric — safe for trend/alert ranking
|
||||
* and `metrics[key]` index access. Excludes the optional, manually-entered
|
||||
|
|
@ -150,6 +226,8 @@ export interface MonthlyReport {
|
|||
avgEngagementRate: number;
|
||||
avgImpressionsPerPost: number;
|
||||
};
|
||||
/** Baseline from posts published before this month (N17) — see WeeklyReport. */
|
||||
baseline?: BaselineBlock;
|
||||
topPerformers: PostAnalytics[];
|
||||
byWeek: {
|
||||
week: string;
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import type { PostAnalytics, MonthlyReport } from "../models/types.js";
|
||||
import { loadAllPosts, loadMonthlyReport, saveMonthlyReport } from "../utils/storage.js";
|
||||
import { mean, weightedOutOfNetworkPct } from "../utils/stats.js";
|
||||
import { mean, weightedOutOfNetworkPct, buildBaselineBlock } from "../utils/stats.js";
|
||||
import { detectAlerts } from "../utils/alerts.js";
|
||||
import { getISOWeek } from "./weekly.js";
|
||||
|
||||
|
|
@ -115,6 +115,14 @@ export function generateMonthlyReport(root: string, month: string): MonthlyRepor
|
|||
avgEngagementRate,
|
||||
avgImpressionsPerPost,
|
||||
},
|
||||
// Baseline from earlier months only — same contract as the weekly report:
|
||||
// always present, refusal included. `startsWith(month)` above and the
|
||||
// `< month` prefix compare below partition the posts with no overlap.
|
||||
baseline: buildBaselineBlock(
|
||||
allPosts.filter(p => p.publishedDate.slice(0, 7) < month),
|
||||
month,
|
||||
monthPosts
|
||||
),
|
||||
topPerformers,
|
||||
byWeek,
|
||||
trends,
|
||||
|
|
|
|||
|
|
@ -1,5 +1,11 @@
|
|||
import type { PostAnalytics, WeeklyReport } from "../models/types.js";
|
||||
import { mean, trendDirection, percentChange, weightedOutOfNetworkPct } from "../utils/stats.js";
|
||||
import {
|
||||
mean,
|
||||
trendDirection,
|
||||
percentChange,
|
||||
weightedOutOfNetworkPct,
|
||||
buildBaselineBlock,
|
||||
} from "../utils/stats.js";
|
||||
import { detectAlerts, detectWeeklyAlerts } from "../utils/alerts.js";
|
||||
import { loadAllPosts, loadWeeklyReport, saveWeeklyReport } from "../utils/storage.js";
|
||||
|
||||
|
|
@ -120,6 +126,13 @@ export function generateWeeklyReport(analyticsRoot: string, week?: string): Week
|
|||
// Filter posts for target week
|
||||
const weekPosts = getPostsForWeek(allPosts, targetWeek);
|
||||
|
||||
// Baseline: the operator's own normal from EARLIER weeks only. The week
|
||||
// boundary is computed with the same expression as getPostsForWeek above, so
|
||||
// every post is either in-week or prior — never both, never neither.
|
||||
const priorPosts = allPosts.filter(
|
||||
post => getISOWeek(new Date(post.publishedDate)) < targetWeek
|
||||
);
|
||||
|
||||
// Initialize report structure
|
||||
const report: WeeklyReport = {
|
||||
week: targetWeek,
|
||||
|
|
@ -134,6 +147,9 @@ export function generateWeeklyReport(analyticsRoot: string, week?: string): Week
|
|||
avgEngagementRate: 0,
|
||||
avgImpressionsPerPost: 0,
|
||||
},
|
||||
// Always present on a generated report, refusal included: an absent block
|
||||
// cannot be told apart from a report made before baselines existed.
|
||||
baseline: buildBaselineBlock(priorPosts, targetWeek, weekPosts),
|
||||
topPerformers: [],
|
||||
underperformers: [],
|
||||
trends: {
|
||||
|
|
|
|||
143
scripts/analytics/src/utils/queue-join.ts
Normal file
143
scripts/analytics/src/utils/queue-join.ts
Normal file
|
|
@ -0,0 +1,143 @@
|
|||
import { existsSync, readFileSync } from "node:fs";
|
||||
|
||||
/**
|
||||
* The post queue's shape, as written by `hooks/scripts/queue-manager.mjs`
|
||||
* (`queueAdd`). Read-only here: analytics never writes the queue.
|
||||
*
|
||||
* Only the fields the join needs are typed. `pillar` and `format` are optional
|
||||
* because an entry can predate them or be written by hand.
|
||||
*/
|
||||
export interface QueueEntry {
|
||||
id: string;
|
||||
scheduled_date: string;
|
||||
pillar?: string;
|
||||
format?: string;
|
||||
hook_preview?: string;
|
||||
}
|
||||
|
||||
/** How far a publish date may drift from its planned date and still join. */
|
||||
const RESCHEDULE_TOLERANCE_DAYS = 2;
|
||||
|
||||
/**
|
||||
* Read the post queue.
|
||||
*
|
||||
* Returns [] for a missing, malformed, or non-array file. The join supplies
|
||||
* OPTIONAL grouping metadata, so a broken queue must degrade to "no labels" —
|
||||
* never take down a report the operator asked for. Rows without a
|
||||
* `scheduled_date` are dropped: they have nothing to join on.
|
||||
*/
|
||||
export function loadQueueEntries(queueFile: string): QueueEntry[] {
|
||||
if (!existsSync(queueFile)) return [];
|
||||
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(readFileSync(queueFile, "utf8"));
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
|
||||
if (!Array.isArray(parsed)) return [];
|
||||
|
||||
return parsed.filter(
|
||||
(row): row is QueueEntry =>
|
||||
typeof row === "object" &&
|
||||
row !== null &&
|
||||
typeof (row as QueueEntry).scheduled_date === "string"
|
||||
);
|
||||
}
|
||||
|
||||
/** Whole days between two YYYY-MM-DD dates, absolute. */
|
||||
function daysApart(a: string, b: string): number {
|
||||
const ms = Math.abs(Date.parse(`${a}T00:00:00Z`) - Date.parse(`${b}T00:00:00Z`));
|
||||
return Math.round(ms / 86_400_000);
|
||||
}
|
||||
|
||||
/**
|
||||
* Does this entry's hook look like the start of this post's title? The stored
|
||||
* title is a truncated prefix of the post and the hook preview is a truncated
|
||||
* prefix of the draft, so neither contains the other reliably — compare on the
|
||||
* shorter overlap, case-insensitively.
|
||||
*/
|
||||
function hookMatchesTitle(hook: string | undefined, title: string): boolean {
|
||||
if (!hook) return false;
|
||||
const a = hook.trim().toLowerCase();
|
||||
const b = title.trim().toLowerCase();
|
||||
if (a.length === 0) return false;
|
||||
const overlap = Math.min(a.length, b.length, 40);
|
||||
return a.slice(0, overlap) === b.slice(0, overlap);
|
||||
}
|
||||
|
||||
/**
|
||||
* Attach `pillar` and `format` to posts by joining the queue on publish date.
|
||||
*
|
||||
* The rules, in order. Every ambiguous case resolves to UNLABELLED — a guessed label is
|
||||
* worse than no label:
|
||||
* 1. Exact date match wins over a near one.
|
||||
* 2. Otherwise an entry within ±2 days joins (a plan the operator published a
|
||||
* day late is the same post).
|
||||
* 3. Two candidates at the same distance are disambiguated on the hook preview
|
||||
* against the post title; if that does not resolve to exactly one, the post
|
||||
* stays UNLABELLED.
|
||||
* 4. An entry labels at most one post. Reusing it would invent a second post of
|
||||
* that format, inflating the group's N with a post that never existed.
|
||||
*
|
||||
* Guessing is the failure mode to avoid: a post filed under the wrong format
|
||||
* corrupts the baseline it is then measured against. Unlabelled posts are
|
||||
* always kept in the returned list — they simply do not enter a group.
|
||||
*/
|
||||
export function labelPosts<T extends { publishedDate: string; title: string }>(
|
||||
posts: T[],
|
||||
entries: QueueEntry[]
|
||||
): Array<T & { pillar?: string; format?: string }> {
|
||||
const claimed = new Set<QueueEntry>();
|
||||
|
||||
/** Best entry for this post, or undefined when nothing resolves it. */
|
||||
const resolve = (post: T): QueueEntry | undefined => {
|
||||
const candidates = entries.filter(
|
||||
(candidate) =>
|
||||
!claimed.has(candidate) &&
|
||||
daysApart(candidate.scheduled_date, post.publishedDate) <= RESCHEDULE_TOLERANCE_DAYS
|
||||
);
|
||||
if (candidates.length === 0) return undefined;
|
||||
|
||||
// Closest first; an exact match is distance 0 and therefore always preferred.
|
||||
const closest = Math.min(
|
||||
...candidates.map((candidate) => daysApart(candidate.scheduled_date, post.publishedDate))
|
||||
);
|
||||
const tied = candidates.filter(
|
||||
(candidate) => daysApart(candidate.scheduled_date, post.publishedDate) === closest
|
||||
);
|
||||
if (tied.length === 1) return tied[0];
|
||||
|
||||
const byHook = tied.filter((candidate) => hookMatchesTitle(candidate.hook_preview, post.title));
|
||||
return byHook.length === 1 ? byHook[0] : undefined;
|
||||
};
|
||||
|
||||
// Exact matches are resolved first so a same-day post cannot lose its entry
|
||||
// to a neighbouring post that merely sits within tolerance.
|
||||
const order = [...posts.keys()].sort((a, b) => {
|
||||
const distance = (index: number) => {
|
||||
const dates = entries.map((entry) => daysApart(entry.scheduled_date, posts[index].publishedDate));
|
||||
return dates.length > 0 ? Math.min(...dates) : Number.POSITIVE_INFINITY;
|
||||
};
|
||||
return distance(a) - distance(b);
|
||||
});
|
||||
|
||||
const labels = new Map<number, QueueEntry>();
|
||||
for (const index of order) {
|
||||
const match = resolve(posts[index]);
|
||||
if (match) {
|
||||
claimed.add(match);
|
||||
labels.set(index, match);
|
||||
}
|
||||
}
|
||||
|
||||
return posts.map((post, index) => {
|
||||
const match = labels.get(index);
|
||||
return {
|
||||
...post,
|
||||
pillar: match?.pillar,
|
||||
format: match?.format,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
|
@ -1,4 +1,9 @@
|
|||
import type { TrendDirection } from "../models/types.js";
|
||||
import type {
|
||||
TrendDirection,
|
||||
Baseline,
|
||||
BaselineBlock,
|
||||
BaselineReading,
|
||||
} from "../models/types.js";
|
||||
|
||||
/**
|
||||
* Calculate arithmetic mean of values.
|
||||
|
|
@ -87,6 +92,267 @@ export function weightedOutOfNetworkPct(posts: ReachWeightable[]): number | unde
|
|||
return Math.round((weightedSum / totalWeight) * 10) / 10;
|
||||
}
|
||||
|
||||
/**
|
||||
* Baseline window, in POSTS — not days.
|
||||
*
|
||||
* A calendar window silently shrinks: at ~2 posts/week a 30-day window holds
|
||||
* about 8 posts, but one slow month turns it into 3 without the operator seeing
|
||||
* it happen. "Your last 10 posts" is a count they can verify, and at 2/week it
|
||||
* spans roughly five weeks — recent enough to reflect the current strategy,
|
||||
* long enough that one viral post does not own the median.
|
||||
*/
|
||||
export const BASELINE_WINDOW = 10;
|
||||
|
||||
/**
|
||||
* Minimum posts required before any verdict is given.
|
||||
*
|
||||
* With fewer than five values the median is the midpoint of at most two
|
||||
* observations, so one new post can shift it further than any genuine
|
||||
* week-over-week change — and the MAD around it is degenerate. Any "up" or
|
||||
* "down" read off such a baseline is a reading of noise. Five is the floor
|
||||
* where median and MAD both stop moving on single-post entry; it is a
|
||||
* documented default, overridable per call, not a law of statistics.
|
||||
*/
|
||||
export const MIN_BASELINE_N = 5;
|
||||
|
||||
/**
|
||||
* Band half-width in MADs. At k=1 the band covers the middle of the operator's
|
||||
* own spread, which is the question being asked ("is this number ordinary for
|
||||
* me?"). Widen it per call when a metric is genuinely noisier.
|
||||
*/
|
||||
export const BAND_K = 1;
|
||||
|
||||
/** Round to one decimal — the same reporting precision as the reach aggregate. */
|
||||
function round1(value: number): number {
|
||||
return Math.round(value * 10) / 10;
|
||||
}
|
||||
|
||||
/**
|
||||
* Median of the values. Returns undefined for an empty list — absent data stays
|
||||
* absent, and 0 would be a fabricated baseline.
|
||||
*
|
||||
* Sorts a copy numerically: callers pass recency-ordered data, and mutating or
|
||||
* trusting that order would silently corrupt both the caller's array and the
|
||||
* result.
|
||||
*/
|
||||
export function median(values: number[]): number | undefined {
|
||||
if (values.length === 0) return undefined;
|
||||
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const mid = Math.floor(sorted.length / 2);
|
||||
|
||||
return sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid];
|
||||
}
|
||||
|
||||
/**
|
||||
* Median absolute deviation — median(|x − median(x)|).
|
||||
*
|
||||
* The robust counterpart to standardDeviation above. Both exist on purpose:
|
||||
* the alert engine WANTS outlier sensitivity (a spike is the signal it hunts),
|
||||
* while baseline framing wants the opposite — a normal-range band that one
|
||||
* viral post cannot inflate. Returns undefined for an empty list; 0 for
|
||||
* identical values is a real answer, not a missing one.
|
||||
*/
|
||||
export function medianAbsoluteDeviation(values: number[]): number | undefined {
|
||||
const mid = median(values);
|
||||
if (mid === undefined) return undefined;
|
||||
|
||||
return median(values.map((value) => Math.abs(value - mid)));
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a rolling baseline from a metric's recent history.
|
||||
*
|
||||
* @param orderedValues - the metric, OLDEST FIRST. Order carries recency, so
|
||||
* the window is positional (the last `window` entries); the median inside it
|
||||
* is value-sorted. Non-finite entries are dropped before the window is taken,
|
||||
* so a gap never occupies a slot that a real post could have filled.
|
||||
* @param opts.window - override BASELINE_WINDOW
|
||||
* @param opts.minN - override MIN_BASELINE_N
|
||||
* @param opts.k - override BAND_K
|
||||
*
|
||||
* Returns either a verdict or an explicit refusal. There is no third state and
|
||||
* no "provisional" number: a caller that cannot get a verdict must say so
|
||||
* rather than print a median it was told not to trust.
|
||||
*/
|
||||
export function rollingBaseline(
|
||||
orderedValues: number[],
|
||||
opts: { window?: number; minN?: number; k?: number } = {}
|
||||
): Baseline {
|
||||
const window = opts.window ?? BASELINE_WINDOW;
|
||||
const minN = opts.minN ?? MIN_BASELINE_N;
|
||||
const k = opts.k ?? BAND_K;
|
||||
|
||||
const usable = orderedValues.filter((value) => Number.isFinite(value));
|
||||
const inWindow = usable.slice(-window);
|
||||
|
||||
if (inWindow.length < minN) {
|
||||
return {
|
||||
status: "insufficient-data",
|
||||
n: inWindow.length,
|
||||
required: minN,
|
||||
reason:
|
||||
`Too little data for a verdict: ${inWindow.length} post(s) in the baseline ` +
|
||||
`window, ${minN} required. A median built on fewer moves further when one ` +
|
||||
`post enters than any real change would move it.`,
|
||||
};
|
||||
}
|
||||
|
||||
const mid = median(inWindow) as number;
|
||||
const mad = medianAbsoluteDeviation(inWindow) as number;
|
||||
|
||||
return {
|
||||
status: "ok",
|
||||
n: inWindow.length,
|
||||
window,
|
||||
median: round1(mid),
|
||||
mad: round1(mad),
|
||||
band: {
|
||||
// Impressions and engagement rate are non-negative by nature; an
|
||||
// arithmetically negative lower bound is not a readable floor.
|
||||
low: round1(Math.max(0, mid - k * mad)),
|
||||
high: round1(mid + k * mad),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Read one value against a baseline.
|
||||
*
|
||||
* `within-band` is the answer that earns this function its place: a number
|
||||
* 7% above the median reads as "up" in any week-over-week percentage while
|
||||
* sitting entirely inside the operator's own ordinary spread. A refused
|
||||
* baseline yields `no-verdict`, so the refusal cannot be lost at the call site.
|
||||
*/
|
||||
export function readAgainstBaseline(value: number, baseline: Baseline): BaselineReading {
|
||||
if (baseline.status !== "ok") return "no-verdict";
|
||||
|
||||
// Edges count as inside: the band is the normal range, not an exclusive zone.
|
||||
if (value > baseline.band.high) return "above-band";
|
||||
if (value < baseline.band.low) return "below-band";
|
||||
return "within-band";
|
||||
}
|
||||
|
||||
/**
|
||||
* One baseline per group — the per-format and per-pillar view.
|
||||
*
|
||||
* Format and pillar are NOT stored in analytics (they live in the post queue,
|
||||
* `drafts/queue.json`), so the label arrives through a resolver rather than
|
||||
* being assumed as a field — the same join the newsletter's previous-edition
|
||||
* calibration performs. Items must be oldest-first; grouping preserves order.
|
||||
*
|
||||
* Each group is judged on ITS OWN count. Eight posts in total do not license a
|
||||
* verdict about the two that were carousels — that specific over-read is what
|
||||
* the refusal exists to prevent.
|
||||
*
|
||||
* Items whose label is empty or undefined are skipped and never collected into a phantom group:
|
||||
* an unlabelled post is missing metadata, not a format.
|
||||
*/
|
||||
export function baselineByGroup<T>(
|
||||
items: T[],
|
||||
keyOf: (item: T) => string | undefined,
|
||||
valueOf: (item: T) => number,
|
||||
opts: { window?: number; minN?: number; k?: number } = {}
|
||||
): Map<string, Baseline> {
|
||||
const grouped = new Map<string, number[]>();
|
||||
|
||||
for (const item of items) {
|
||||
const key = keyOf(item);
|
||||
if (key === undefined || key === null || key === "") continue;
|
||||
|
||||
const bucket = grouped.get(key);
|
||||
if (bucket) {
|
||||
bucket.push(valueOf(item));
|
||||
} else {
|
||||
grouped.set(key, [valueOf(item)]);
|
||||
}
|
||||
}
|
||||
|
||||
const result = new Map<string, Baseline>();
|
||||
for (const [key, values] of grouped) {
|
||||
result.set(key, rollingBaseline(values, opts));
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Minimal shape a post needs to feed a baseline block — keeps this usable from
|
||||
* both report builders without importing the full record (the ReachWeightable
|
||||
* pattern above).
|
||||
*/
|
||||
interface BaselineSourcePost {
|
||||
publishedDate: string;
|
||||
metrics: { impressions: number; engagementRate: number };
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the baseline block a period report carries.
|
||||
*
|
||||
* @param priorPosts - posts published STRICTLY BEFORE the reported period. The
|
||||
* caller filters, because "before this week" and "before this month" are its
|
||||
* own comparison to make; passing the period's own posts would make the
|
||||
* report partly a comparison with itself.
|
||||
* @param periodLabel - the excluded period, recorded so a reader can see what
|
||||
* the baseline is a baseline *for*.
|
||||
* @param periodPosts - the reported period's own posts, read against the
|
||||
* baseline. The comparison uses the period's MEDIAN, not its mean: comparing
|
||||
* a mean against a median band would let one viral post turn an ordinary week
|
||||
* into a "trend", which is the exact over-read this engine exists to stop.
|
||||
*
|
||||
* Sorts by publish date internally: storage order is file order, and the window
|
||||
* is positional, so an unsorted input would silently window the wrong posts.
|
||||
* Always returns a block — a refusal is a result, not an error, and an omitted
|
||||
* block would be indistinguishable from a report generated before this existed.
|
||||
*/
|
||||
export function buildBaselineBlock(
|
||||
priorPosts: BaselineSourcePost[],
|
||||
periodLabel: string,
|
||||
periodPosts: BaselineSourcePost[] = [],
|
||||
opts: { window?: number; minN?: number; k?: number } = {}
|
||||
): BaselineBlock {
|
||||
const ordered = [...priorPosts].sort((a, b) =>
|
||||
a.publishedDate.localeCompare(b.publishedDate)
|
||||
);
|
||||
|
||||
const impressions = rollingBaseline(
|
||||
ordered.map((post) => post.metrics.impressions),
|
||||
opts
|
||||
);
|
||||
const engagementRate = rollingBaseline(
|
||||
ordered.map((post) => post.metrics.engagementRate),
|
||||
opts
|
||||
);
|
||||
|
||||
/** Period median vs its baseline; no median means nothing to read. */
|
||||
const read = (
|
||||
values: number[],
|
||||
baseline: Baseline
|
||||
): { median?: number; reading: BaselineReading } => {
|
||||
const mid = median(values.filter((value) => Number.isFinite(value)));
|
||||
if (mid === undefined) return { reading: "no-verdict" };
|
||||
return { median: round1(mid), reading: readAgainstBaseline(mid, baseline) };
|
||||
};
|
||||
|
||||
return {
|
||||
window: opts.window ?? BASELINE_WINDOW,
|
||||
required: opts.minN ?? MIN_BASELINE_N,
|
||||
historyBefore: periodLabel,
|
||||
impressions,
|
||||
engagementRate,
|
||||
period: {
|
||||
impressions: read(
|
||||
periodPosts.map((post) => post.metrics.impressions),
|
||||
impressions
|
||||
),
|
||||
engagementRate: read(
|
||||
periodPosts.map((post) => post.metrics.engagementRate),
|
||||
engagementRate
|
||||
),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Calculate how many standard deviations a value is from the mean.
|
||||
* Returns 0 if standard deviation is 0.
|
||||
|
|
|
|||
|
|
@ -178,4 +178,38 @@ describe("generateMonthlyReport", () => {
|
|||
const report = generateMonthlyReport(root, "2026-03");
|
||||
assert.equal(report.month, "2026-03");
|
||||
});
|
||||
|
||||
/** Baseline framing (N17) — same contract as the weekly report. */
|
||||
test("attaches a baseline built only from earlier months", () => {
|
||||
const root = setupTestRoot([
|
||||
createPost("2026-01-05", 100, 4),
|
||||
createPost("2026-01-12", 110, 4),
|
||||
createPost("2026-02-02", 120, 4),
|
||||
createPost("2026-02-09", 130, 4),
|
||||
createPost("2026-02-16", 140, 4),
|
||||
// In-month outlier: must not feed the baseline it is compared against.
|
||||
createPost("2026-03-03", 99999, 9),
|
||||
]);
|
||||
|
||||
const report = generateMonthlyReport(root, "2026-03");
|
||||
|
||||
assert.ok(report.baseline, "A generated monthly report must carry a baseline block");
|
||||
assert.equal(report.baseline!.historyBefore, "2026-03");
|
||||
assert.equal(report.baseline!.impressions.status, "ok");
|
||||
if (report.baseline!.impressions.status !== "ok") return;
|
||||
assert.equal(report.baseline!.impressions.n, 5);
|
||||
assert.equal(report.baseline!.impressions.median, 120);
|
||||
});
|
||||
|
||||
test("refuses a monthly baseline verdict on thin history", () => {
|
||||
const root = setupTestRoot([
|
||||
createPost("2026-02-02", 120, 4),
|
||||
createPost("2026-03-03", 500, 9),
|
||||
]);
|
||||
|
||||
const report = generateMonthlyReport(root, "2026-03");
|
||||
|
||||
assert.ok(report.baseline);
|
||||
assert.equal(report.baseline!.impressions.status, "insufficient-data");
|
||||
});
|
||||
});
|
||||
|
|
|
|||
163
scripts/analytics/tests/queue-join.test.ts
Normal file
163
scripts/analytics/tests/queue-join.test.ts
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
import { describe, test, afterEach } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync, rmSync, writeFileSync, existsSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { loadQueueEntries, labelPosts } from "../src/utils/queue-join.js";
|
||||
import type { QueueEntry } from "../src/utils/queue-join.js";
|
||||
|
||||
/**
|
||||
* Format and pillar are NOT analytics data — they live in the post queue, which
|
||||
* the operator's create surfaces write. The per-format / per-pillar baseline
|
||||
* therefore rests on a JOIN, and a join that guesses is worse than no labels at
|
||||
* all: a post filed under the wrong format would poison the very baseline it is
|
||||
* being measured against. Every ambiguous case here resolves to UNLABELLED.
|
||||
*/
|
||||
describe("queue-join", () => {
|
||||
let tempDir: string;
|
||||
|
||||
afterEach(() => {
|
||||
if (tempDir && existsSync(tempDir)) {
|
||||
rmSync(tempDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
const post = (publishedDate: string, title = "Some post") => ({
|
||||
publishedDate,
|
||||
title,
|
||||
metrics: { impressions: 1000, engagementRate: 5 },
|
||||
});
|
||||
|
||||
const entry = (overrides: Partial<QueueEntry> & { scheduled_date: string }): QueueEntry => ({
|
||||
id: overrides.id ?? `q-${overrides.scheduled_date}`,
|
||||
scheduled_date: overrides.scheduled_date,
|
||||
pillar: overrides.pillar ?? "ai-agents",
|
||||
format: overrides.format ?? "text",
|
||||
hook_preview: overrides.hook_preview,
|
||||
});
|
||||
|
||||
describe("loadQueueEntries", () => {
|
||||
test("should return an empty list when there is no queue file", () => {
|
||||
tempDir = mkdtempSync(join(tmpdir(), "queue-join-"));
|
||||
const result = loadQueueEntries(join(tempDir, "queue.json"));
|
||||
assert.deepEqual(result, [], "A missing queue is unlabelled data, not an error");
|
||||
});
|
||||
|
||||
test("should return an empty list for malformed JSON rather than throwing", () => {
|
||||
tempDir = mkdtempSync(join(tmpdir(), "queue-join-"));
|
||||
const file = join(tempDir, "queue.json");
|
||||
writeFileSync(file, "{ not json", "utf8");
|
||||
assert.deepEqual(loadQueueEntries(file), []);
|
||||
});
|
||||
|
||||
test("should return an empty list when the file is not an array", () => {
|
||||
tempDir = mkdtempSync(join(tmpdir(), "queue-join-"));
|
||||
const file = join(tempDir, "queue.json");
|
||||
writeFileSync(file, JSON.stringify({ entries: [] }), "utf8");
|
||||
assert.deepEqual(loadQueueEntries(file), []);
|
||||
});
|
||||
|
||||
test("should read entries and keep those missing pillar or format", () => {
|
||||
tempDir = mkdtempSync(join(tmpdir(), "queue-join-"));
|
||||
const file = join(tempDir, "queue.json");
|
||||
writeFileSync(
|
||||
file,
|
||||
JSON.stringify([
|
||||
{ id: "a", scheduled_date: "2026-07-01", pillar: "ai", format: "text" },
|
||||
{ id: "b", scheduled_date: "2026-07-03" },
|
||||
]),
|
||||
"utf8"
|
||||
);
|
||||
|
||||
const result = loadQueueEntries(file);
|
||||
assert.equal(result.length, 2);
|
||||
assert.equal(result[1].pillar, undefined, "A partial entry is data; dropping it would hide posts");
|
||||
});
|
||||
|
||||
test("should skip rows with no scheduled_date, which cannot join on anything", () => {
|
||||
tempDir = mkdtempSync(join(tmpdir(), "queue-join-"));
|
||||
const file = join(tempDir, "queue.json");
|
||||
writeFileSync(
|
||||
file,
|
||||
JSON.stringify([{ id: "a", pillar: "ai", format: "text" }, { id: "b", scheduled_date: "2026-07-03" }]),
|
||||
"utf8"
|
||||
);
|
||||
assert.equal(loadQueueEntries(file).length, 1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("labelPosts", () => {
|
||||
test("should label a post whose publish date matches a queue entry exactly", () => {
|
||||
const result = labelPosts([post("2026-07-08")], [entry({ scheduled_date: "2026-07-08", format: "carousel" })]);
|
||||
assert.equal(result[0].format, "carousel");
|
||||
assert.equal(result[0].pillar, "ai-agents");
|
||||
});
|
||||
|
||||
test("should label through a small reschedule gap", () => {
|
||||
// A queue entry is a PLAN; the operator publishing a day late is normal.
|
||||
const result = labelPosts([post("2026-07-09")], [entry({ scheduled_date: "2026-07-08" })]);
|
||||
assert.equal(result[0].format, "text");
|
||||
});
|
||||
|
||||
test("should not label across a gap wider than two days", () => {
|
||||
const result = labelPosts([post("2026-07-15")], [entry({ scheduled_date: "2026-07-08" })]);
|
||||
assert.equal(result[0].format, undefined, "A week apart is a different post, not a reschedule");
|
||||
assert.equal(result[0].pillar, undefined);
|
||||
});
|
||||
|
||||
test("should prefer an exact date match over a near one", () => {
|
||||
const result = labelPosts(
|
||||
[post("2026-07-08")],
|
||||
[entry({ scheduled_date: "2026-07-07", format: "video" }), entry({ scheduled_date: "2026-07-08", format: "text" })]
|
||||
);
|
||||
assert.equal(result[0].format, "text");
|
||||
});
|
||||
|
||||
test("should leave a post unlabelled when two entries tie on the same date", () => {
|
||||
const result = labelPosts(
|
||||
[post("2026-07-08")],
|
||||
[
|
||||
entry({ id: "x", scheduled_date: "2026-07-08", format: "text" }),
|
||||
entry({ id: "y", scheduled_date: "2026-07-08", format: "carousel" }),
|
||||
]
|
||||
);
|
||||
assert.equal(result[0].format, undefined, "A coin-flip label would poison the format baseline");
|
||||
});
|
||||
|
||||
test("should disambiguate same-date entries on the hook preview", () => {
|
||||
const result = labelPosts(
|
||||
[post("2026-07-08", "The thing nobody tells you about agent evals")],
|
||||
[
|
||||
entry({ id: "x", scheduled_date: "2026-07-08", format: "text", hook_preview: "Three ways to lose a week" }),
|
||||
entry({
|
||||
id: "y",
|
||||
scheduled_date: "2026-07-08",
|
||||
format: "carousel",
|
||||
hook_preview: "The thing nobody tells you",
|
||||
}),
|
||||
]
|
||||
);
|
||||
assert.equal(result[0].format, "carousel", "The hook is the only honest tiebreaker available");
|
||||
});
|
||||
|
||||
test("should leave the post unlabelled when the queue is empty", () => {
|
||||
const result = labelPosts([post("2026-07-08")], []);
|
||||
assert.equal(result[0].format, undefined);
|
||||
});
|
||||
|
||||
test("should never consume one queue entry for two different posts", () => {
|
||||
// Two posts, one entry: labelling both would double-count the entry's
|
||||
// format and invent a second post of that format that never existed.
|
||||
const result = labelPosts([post("2026-07-08"), post("2026-07-09")], [entry({ scheduled_date: "2026-07-08" })]);
|
||||
const labelled = result.filter((r) => r.format !== undefined);
|
||||
assert.equal(labelled.length, 1, "An entry labels at most one post");
|
||||
assert.equal(labelled[0].publishedDate, "2026-07-08", "The exact match wins the entry");
|
||||
});
|
||||
|
||||
test("should preserve every post, labelled or not", () => {
|
||||
const posts = [post("2026-07-08"), post("2026-07-30")];
|
||||
const result = labelPosts(posts, [entry({ scheduled_date: "2026-07-08" })]);
|
||||
assert.equal(result.length, 2, "Unlabelled posts stay in the list; they are just not grouped");
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
@ -7,6 +7,15 @@ import {
|
|||
percentChange,
|
||||
deviationsFromMean,
|
||||
weightedOutOfNetworkPct,
|
||||
median,
|
||||
medianAbsoluteDeviation,
|
||||
rollingBaseline,
|
||||
readAgainstBaseline,
|
||||
baselineByGroup,
|
||||
buildBaselineBlock,
|
||||
BASELINE_WINDOW,
|
||||
MIN_BASELINE_N,
|
||||
BAND_K,
|
||||
} from "../src/utils/stats.js";
|
||||
|
||||
describe("stats", () => {
|
||||
|
|
@ -193,4 +202,381 @@ describe("stats", () => {
|
|||
assert.equal(result, 41.7);
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* Baseline engine (N17). At ~2 posts/week every reading is small-N, so the
|
||||
* engine exists to answer one question honestly: is this number different
|
||||
* from the operator's own normal, or is it noise? Two disciplines are under
|
||||
* test — ROBUSTNESS (median/MAD, not mean/stddev, because one viral post
|
||||
* would otherwise define "normal") and REFUSAL (below the minimum N there is
|
||||
* no verdict at all, rather than a confident reading of nothing).
|
||||
*/
|
||||
describe("median", () => {
|
||||
test("should return the middle value for an odd count", () => {
|
||||
assert.equal(median([30, 10, 20]), 20, "Must sort internally, not trust input order");
|
||||
});
|
||||
|
||||
test("should average the two middle values for an even count", () => {
|
||||
assert.equal(median([10, 20, 30, 40]), 25);
|
||||
});
|
||||
|
||||
test("should return undefined for an empty list, never 0", () => {
|
||||
// Same discipline as weightedOutOfNetworkPct: absent data stays absent.
|
||||
assert.equal(median([]), undefined);
|
||||
});
|
||||
|
||||
test("should be unmoved by a single extreme outlier", () => {
|
||||
// The whole reason baseline framing uses median: mean([100,110,120,130,10000])
|
||||
// is 2092 — a "baseline" no ordinary post could ever reach.
|
||||
assert.equal(median([100, 110, 120, 130, 10000]), 120);
|
||||
assert.equal(mean([100, 110, 120, 130, 10000]), 2092);
|
||||
});
|
||||
});
|
||||
|
||||
describe("medianAbsoluteDeviation", () => {
|
||||
test("should return the median of absolute deviations from the median", () => {
|
||||
// median = 20; deviations = [10, 0, 10] → MAD = 10
|
||||
assert.equal(medianAbsoluteDeviation([10, 20, 30]), 10);
|
||||
});
|
||||
|
||||
test("should return 0 for identical values", () => {
|
||||
assert.equal(medianAbsoluteDeviation([50, 50, 50]), 0);
|
||||
});
|
||||
|
||||
test("should return undefined for an empty list", () => {
|
||||
assert.equal(medianAbsoluteDeviation([]), undefined);
|
||||
});
|
||||
|
||||
test("should stay small when one outlier inflates the standard deviation", () => {
|
||||
// stddev is dragged to ~3950 by the outlier; MAD stays at 10, which is
|
||||
// what an honest "normal spread" band needs.
|
||||
const values = [100, 110, 120, 130, 10000];
|
||||
assert.equal(medianAbsoluteDeviation(values), 10);
|
||||
assert.ok(standardDeviation(values) > 1000, "stddev is outlier-dominated by contrast");
|
||||
});
|
||||
});
|
||||
|
||||
describe("rollingBaseline", () => {
|
||||
test("should refuse a verdict below the minimum N", () => {
|
||||
const result = rollingBaseline([100, 200, 300]);
|
||||
assert.equal(result.status, "insufficient-data", "3 posts cannot support a verdict");
|
||||
if (result.status !== "insufficient-data") return;
|
||||
assert.equal(result.n, 3);
|
||||
assert.equal(result.required, MIN_BASELINE_N);
|
||||
assert.ok(result.reason.length > 0, "Refusal must carry an operator-facing reason");
|
||||
});
|
||||
|
||||
test("should refuse on an empty history rather than report a zero baseline", () => {
|
||||
const result = rollingBaseline([]);
|
||||
assert.equal(result.status, "insufficient-data");
|
||||
if (result.status !== "insufficient-data") return;
|
||||
assert.equal(result.n, 0);
|
||||
});
|
||||
|
||||
test("should produce a median and a band once the minimum N is met", () => {
|
||||
// median = 120, MAD = 10 → band = [110, 130] at k=1
|
||||
const result = rollingBaseline([100, 110, 120, 130, 140]);
|
||||
assert.equal(result.status, "ok");
|
||||
if (result.status !== "ok") return;
|
||||
assert.equal(result.n, 5);
|
||||
assert.equal(result.median, 120);
|
||||
assert.equal(result.mad, 10);
|
||||
assert.deepEqual(result.band, { low: 110, high: 130 });
|
||||
});
|
||||
|
||||
test("should keep only the trailing window when history exceeds it", () => {
|
||||
// 12 values oldest→newest; the window is the last 10, so the two oldest
|
||||
// (1 and 2) must not pull the median down.
|
||||
const values = [1, 2, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190];
|
||||
const result = rollingBaseline(values);
|
||||
assert.equal(result.status, "ok");
|
||||
if (result.status !== "ok") return;
|
||||
assert.equal(result.n, BASELINE_WINDOW);
|
||||
assert.equal(result.window, BASELINE_WINDOW);
|
||||
assert.equal(result.median, 145, "Median of the last 10, not of all 12");
|
||||
});
|
||||
|
||||
test("should treat the input as oldest-first and never reorder it", () => {
|
||||
// Descending input: if the implementation sorted by value it would keep
|
||||
// the wrong ten. The last 10 here are 100 down to 10 → median 55.
|
||||
const values = [200, 190, 100, 90, 80, 70, 60, 50, 40, 30, 20, 10];
|
||||
const result = rollingBaseline(values);
|
||||
assert.equal(result.status, "ok");
|
||||
if (result.status !== "ok") return;
|
||||
assert.equal(result.median, 55, "Window is positional (recency), the median is value-sorted");
|
||||
});
|
||||
|
||||
test("should floor the band at 0 for a naturally non-negative metric", () => {
|
||||
// median 20, MAD 15 → arithmetic low is 5; with a wider spread it would
|
||||
// go negative, which is meaningless for impressions or engagement rate.
|
||||
const result = rollingBaseline([0, 5, 20, 35, 90]);
|
||||
assert.equal(result.status, "ok");
|
||||
if (result.status !== "ok") return;
|
||||
assert.ok(result.band.low >= 0, "A negative lower bound is not a readable floor");
|
||||
});
|
||||
|
||||
test("should accept a configurable minimum N and window", () => {
|
||||
const result = rollingBaseline([10, 20, 30], { minN: 3, window: 3 });
|
||||
assert.equal(result.status, "ok", "The threshold is a documented default, not a hard law");
|
||||
if (result.status !== "ok") return;
|
||||
assert.equal(result.median, 20);
|
||||
});
|
||||
|
||||
test("should widen the band with a larger k", () => {
|
||||
const narrow = rollingBaseline([100, 110, 120, 130, 140]);
|
||||
const wide = rollingBaseline([100, 110, 120, 130, 140], { k: 2 });
|
||||
assert.equal(narrow.status, "ok");
|
||||
assert.equal(wide.status, "ok");
|
||||
if (narrow.status !== "ok" || wide.status !== "ok") return;
|
||||
assert.equal(BAND_K, 1, "Default k is documented as 1");
|
||||
assert.equal(wide.band.high, 140);
|
||||
assert.ok(wide.band.high > narrow.band.high);
|
||||
});
|
||||
|
||||
test("should give a zero-width band for a perfectly steady history", () => {
|
||||
// MAD 0 is legitimate: the operator's numbers really are that steady.
|
||||
const result = rollingBaseline([50, 50, 50, 50, 50]);
|
||||
assert.equal(result.status, "ok");
|
||||
if (result.status !== "ok") return;
|
||||
assert.deepEqual(result.band, { low: 50, high: 50 });
|
||||
assert.equal(result.mad, 0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("readAgainstBaseline", () => {
|
||||
const baseline = rollingBaseline([100, 110, 120, 130, 140]); // median 120, band [110,130]
|
||||
|
||||
test("should call a value inside the band normal variation, not a trend", () => {
|
||||
// 128 is 6.7% above the median — the kind of number that reads as "up"
|
||||
// week-over-week while being entirely inside the operator's own spread.
|
||||
assert.equal(readAgainstBaseline(128, baseline), "within-band");
|
||||
});
|
||||
|
||||
test("should call a value above the band above baseline", () => {
|
||||
assert.equal(readAgainstBaseline(400, baseline), "above-band");
|
||||
});
|
||||
|
||||
test("should call a value below the band below baseline", () => {
|
||||
assert.equal(readAgainstBaseline(40, baseline), "below-band");
|
||||
});
|
||||
|
||||
test("should treat the band edges as inside the band", () => {
|
||||
assert.equal(readAgainstBaseline(110, baseline), "within-band");
|
||||
assert.equal(readAgainstBaseline(130, baseline), "within-band");
|
||||
});
|
||||
|
||||
test("should return no-verdict when the baseline itself was refused", () => {
|
||||
const refused = rollingBaseline([100, 200]);
|
||||
assert.equal(
|
||||
readAgainstBaseline(5000, refused),
|
||||
"no-verdict",
|
||||
"A refused baseline must not be readable as a verdict by the caller"
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("baselineByGroup", () => {
|
||||
// Format and pillar are not stored in analytics (they live in the post
|
||||
// queue), so the engine takes a label resolver rather than assuming a
|
||||
// field — the same join the newsletter calibration uses.
|
||||
const p = (group: string, impressions: number) => ({ group, impressions });
|
||||
const items = [
|
||||
...Array.from({ length: 6 }, (_, i) => p("text", 100 + i * 10)),
|
||||
...Array.from({ length: 2 }, (_, i) => p("carousel", 900 + i * 100)),
|
||||
];
|
||||
|
||||
test("should compute one baseline per group", () => {
|
||||
const result = baselineByGroup(
|
||||
items,
|
||||
(it) => it.group,
|
||||
(it) => it.impressions
|
||||
);
|
||||
assert.equal(result.size, 2);
|
||||
assert.equal(result.get("text")?.status, "ok");
|
||||
});
|
||||
|
||||
test("should refuse per group, not globally", () => {
|
||||
// 8 posts in total would clear the threshold; carousel alone has 2 and
|
||||
// must be refused on its own count. This is the failure the plan names:
|
||||
// a verdict about a format the operator has barely used.
|
||||
const result = baselineByGroup(
|
||||
items,
|
||||
(it) => it.group,
|
||||
(it) => it.impressions
|
||||
);
|
||||
const carousel = result.get("carousel");
|
||||
assert.equal(carousel?.status, "insufficient-data");
|
||||
if (carousel?.status !== "insufficient-data") return;
|
||||
assert.equal(carousel.n, 2);
|
||||
});
|
||||
|
||||
test("should skip items whose label is undefined rather than bucket them", () => {
|
||||
// An unlabelled post (no queue entry) is not a group called "undefined".
|
||||
const mixed = [...items, { group: undefined as unknown as string, impressions: 5000 }];
|
||||
const result = baselineByGroup(
|
||||
mixed,
|
||||
(it) => it.group,
|
||||
(it) => it.impressions
|
||||
);
|
||||
assert.equal(result.size, 2, "No phantom group for unlabelled posts");
|
||||
});
|
||||
|
||||
test("should skip items whose value is not a finite number", () => {
|
||||
const withGap = [...items, p("text", NaN)];
|
||||
const result = baselineByGroup(
|
||||
withGap,
|
||||
(it) => it.group,
|
||||
(it) => it.impressions
|
||||
);
|
||||
const text = result.get("text");
|
||||
assert.equal(text?.status, "ok");
|
||||
if (text?.status !== "ok") return;
|
||||
assert.equal(text.n, 6, "The NaN row must not count toward N");
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildBaselineBlock", () => {
|
||||
const post = (publishedDate: string, impressions: number, engagementRate = 5) => ({
|
||||
publishedDate,
|
||||
metrics: { impressions, engagementRate },
|
||||
});
|
||||
|
||||
const history = [
|
||||
post("2026-01-05", 100, 4.0),
|
||||
post("2026-01-07", 110, 4.5),
|
||||
post("2026-01-12", 120, 5.0),
|
||||
post("2026-01-14", 130, 5.5),
|
||||
post("2026-01-19", 140, 6.0),
|
||||
];
|
||||
|
||||
test("should carry a baseline for both headline metrics", () => {
|
||||
const block = buildBaselineBlock(history, "2026-W05");
|
||||
assert.equal(block.impressions.status, "ok");
|
||||
assert.equal(block.engagementRate.status, "ok");
|
||||
if (block.impressions.status !== "ok") return;
|
||||
assert.equal(block.impressions.median, 120);
|
||||
});
|
||||
|
||||
test("should label the period whose posts are excluded", () => {
|
||||
const block = buildBaselineBlock(history, "2026-W05");
|
||||
assert.equal(block.historyBefore, "2026-W05");
|
||||
assert.equal(block.window, BASELINE_WINDOW);
|
||||
assert.equal(block.required, MIN_BASELINE_N);
|
||||
});
|
||||
|
||||
test("should refuse both metrics when history is too short", () => {
|
||||
const block = buildBaselineBlock(history.slice(0, 2), "2026-W05");
|
||||
assert.equal(block.impressions.status, "insufficient-data");
|
||||
assert.equal(block.engagementRate.status, "insufficient-data");
|
||||
});
|
||||
|
||||
test("should order history by publish date before taking the window", () => {
|
||||
// Same five posts shuffled, plus six older ones so the window bites. If
|
||||
// the block trusted file order, the window would hold the wrong posts.
|
||||
const older = [
|
||||
post("2025-12-01", 5),
|
||||
post("2025-12-03", 6),
|
||||
post("2025-12-05", 7),
|
||||
post("2025-12-08", 8),
|
||||
post("2025-12-10", 9),
|
||||
post("2025-12-12", 10),
|
||||
];
|
||||
const shuffled = [history[4], older[2], history[0], older[5], history[2], older[0],
|
||||
history[1], older[4], history[3], older[1], older[3]];
|
||||
const block = buildBaselineBlock(shuffled, "2026-W05");
|
||||
assert.equal(block.impressions.status, "ok");
|
||||
if (block.impressions.status !== "ok") return;
|
||||
// Window = last 10 by DATE: drops 2025-12-01 (5), keeps 6..10 and 100..140.
|
||||
// Median of [6,7,8,9,10,100,110,120,130,140] = (10+100)/2 = 55.
|
||||
assert.equal(block.impressions.median, 55, "Window must follow publish date, not array order");
|
||||
});
|
||||
|
||||
test("should return a block for an empty history rather than throwing", () => {
|
||||
const block = buildBaselineBlock([], "2026-W05");
|
||||
assert.equal(block.impressions.status, "insufficient-data");
|
||||
if (block.impressions.status !== "insufficient-data") return;
|
||||
assert.equal(block.impressions.n, 0);
|
||||
});
|
||||
|
||||
/**
|
||||
* The reading is computed HERE, in code, not left to whoever renders the
|
||||
* report. A renderer eyeballing whether 128 sits inside [110, 130] is the
|
||||
* over-read the whole engine exists to prevent.
|
||||
*/
|
||||
describe("period reading", () => {
|
||||
test("should read the period's own median against the band", () => {
|
||||
// Baseline median 120, band [110, 130]. Period median 125 → inside.
|
||||
const block = buildBaselineBlock(history, "2026-W05", [
|
||||
post("2026-02-02", 120),
|
||||
post("2026-02-04", 130),
|
||||
]);
|
||||
assert.equal(block.period.impressions.median, 125);
|
||||
assert.equal(block.period.impressions.reading, "within-band");
|
||||
});
|
||||
|
||||
test("should compare the period MEDIAN, not its mean", () => {
|
||||
// Mean of [100, 10000] is 5050 — "above-band" on one viral post. The
|
||||
// median is 5050's honest counterpart: 5050 vs median 5050… no: the
|
||||
// median of two values is their midpoint, so use three posts where the
|
||||
// mean and median genuinely disagree: [100, 110, 10000] → mean 3403,
|
||||
// median 110, which is inside the band. The band must not fire.
|
||||
const block = buildBaselineBlock(history, "2026-W05", [
|
||||
post("2026-02-02", 100),
|
||||
post("2026-02-04", 110),
|
||||
post("2026-02-06", 10000),
|
||||
]);
|
||||
assert.equal(block.period.impressions.median, 110);
|
||||
assert.equal(
|
||||
block.period.impressions.reading,
|
||||
"within-band",
|
||||
"One viral post must not turn an ordinary week into a trend"
|
||||
);
|
||||
});
|
||||
|
||||
test("should read a genuinely high period as above the band", () => {
|
||||
const block = buildBaselineBlock(history, "2026-W05", [
|
||||
post("2026-02-02", 900),
|
||||
post("2026-02-04", 1100),
|
||||
]);
|
||||
assert.equal(block.period.impressions.reading, "above-band");
|
||||
});
|
||||
|
||||
test("should read a genuinely low period as below the band", () => {
|
||||
const block = buildBaselineBlock(history, "2026-W05", [
|
||||
post("2026-02-02", 10),
|
||||
post("2026-02-04", 20),
|
||||
]);
|
||||
assert.equal(block.period.impressions.reading, "below-band");
|
||||
});
|
||||
|
||||
test("should give no verdict when the baseline was refused", () => {
|
||||
const block = buildBaselineBlock(history.slice(0, 2), "2026-W05", [
|
||||
post("2026-02-02", 99999),
|
||||
]);
|
||||
assert.equal(
|
||||
block.period.impressions.reading,
|
||||
"no-verdict",
|
||||
"A thin baseline cannot license a verdict about a big number"
|
||||
);
|
||||
});
|
||||
|
||||
test("should give no verdict and no median for a period with no posts", () => {
|
||||
const block = buildBaselineBlock(history, "2026-W05", []);
|
||||
assert.equal(block.period.impressions.median, undefined);
|
||||
assert.equal(block.period.impressions.reading, "no-verdict");
|
||||
});
|
||||
|
||||
test("should read engagement rate independently of impressions", () => {
|
||||
// Baseline engagement median 5.0, MAD 0.5 → band [4.5, 5.5].
|
||||
// Impressions ordinary, engagement clearly above: the two readings must
|
||||
// not be collapsed into one verdict about "the week".
|
||||
const block = buildBaselineBlock(history, "2026-W05", [
|
||||
post("2026-02-02", 120, 9.0),
|
||||
post("2026-02-04", 120, 9.4),
|
||||
]);
|
||||
assert.equal(block.period.impressions.reading, "within-band");
|
||||
assert.equal(block.period.engagementRate.reading, "above-band");
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ import {
|
|||
getPostsForWeek,
|
||||
generateWeeklyReport,
|
||||
} from "../src/reports/weekly.js";
|
||||
import { saveBatch, saveWeeklyReport } from "../src/utils/storage.js";
|
||||
import { saveBatch, saveWeeklyReport, loadWeeklyReport } from "../src/utils/storage.js";
|
||||
import type { PostAnalytics, AnalyticsBatch } from "../src/models/types.js";
|
||||
|
||||
// Helper function to create test post data
|
||||
|
|
@ -662,4 +662,97 @@ describe("weekly", () => {
|
|||
assert.match(report.week, /^\d{4}-W\d{2}$/);
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* Baseline framing (N17). The report's job is to answer "is this different
|
||||
* from my own normal?" — which requires a baseline built from history BEFORE
|
||||
* the reported week, and an explicit refusal when that history is too thin.
|
||||
*/
|
||||
describe("generateWeeklyReport — baseline", () => {
|
||||
// 2026-W03 is Mon 2026-01-12 .. Sun 2026-01-18.
|
||||
function priorPost(day: string, impressions: number): PostAnalytics {
|
||||
return createTestPost({
|
||||
id: `prior-${day}`,
|
||||
publishedDate: day,
|
||||
metrics: { impressions, reactions: 5, comments: 1, shares: 0, clicks: 2, engagementRate: 4 },
|
||||
});
|
||||
}
|
||||
|
||||
test("should attach a baseline built only from earlier weeks", () => {
|
||||
tempDir = setupTempDir();
|
||||
const posts = [
|
||||
priorPost("2025-12-15", 100),
|
||||
priorPost("2025-12-17", 110),
|
||||
priorPost("2025-12-29", 120),
|
||||
priorPost("2025-12-31", 130),
|
||||
priorPost("2026-01-05", 140),
|
||||
// In-week post, deliberately extreme: if it leaked into the baseline the
|
||||
// median would move and the week would be compared against itself.
|
||||
createTestPost({
|
||||
id: "in-week",
|
||||
publishedDate: "2026-01-14",
|
||||
metrics: { impressions: 99999, reactions: 5, comments: 1, shares: 0, clicks: 2, engagementRate: 4 },
|
||||
}),
|
||||
];
|
||||
saveBatch(tempDir, createTestBatch({ posts }));
|
||||
|
||||
const report = generateWeeklyReport(tempDir, "2026-W03");
|
||||
|
||||
assert.ok(report.baseline, "A generated report must carry a baseline block");
|
||||
assert.equal(report.baseline!.historyBefore, "2026-W03");
|
||||
assert.equal(report.baseline!.impressions.status, "ok");
|
||||
if (report.baseline!.impressions.status !== "ok") return;
|
||||
assert.equal(report.baseline!.impressions.n, 5, "Only the five earlier posts feed it");
|
||||
assert.equal(report.baseline!.impressions.median, 120, "The in-week outlier must not leak in");
|
||||
});
|
||||
|
||||
test("should refuse a baseline verdict when prior history is too thin", () => {
|
||||
tempDir = setupTempDir();
|
||||
const posts = [
|
||||
priorPost("2025-12-15", 100),
|
||||
priorPost("2025-12-17", 110),
|
||||
createTestPost({ id: "in-week", publishedDate: "2026-01-14" }),
|
||||
];
|
||||
saveBatch(tempDir, createTestBatch({ posts }));
|
||||
|
||||
const report = generateWeeklyReport(tempDir, "2026-W03");
|
||||
|
||||
assert.ok(report.baseline);
|
||||
assert.equal(report.baseline!.impressions.status, "insufficient-data");
|
||||
if (report.baseline!.impressions.status !== "insufficient-data") return;
|
||||
assert.equal(report.baseline!.impressions.n, 2);
|
||||
assert.equal(report.baseline!.impressions.required, 5);
|
||||
});
|
||||
|
||||
test("should refuse rather than omit the block when there is no history at all", () => {
|
||||
tempDir = setupTempDir();
|
||||
saveBatch(tempDir, createTestBatch({
|
||||
posts: [createTestPost({ id: "in-week", publishedDate: "2026-01-14" })],
|
||||
}));
|
||||
|
||||
const report = generateWeeklyReport(tempDir, "2026-W03");
|
||||
|
||||
assert.ok(report.baseline, "An absent block is indistinguishable from a pre-N17 report");
|
||||
assert.equal(report.baseline!.impressions.status, "insufficient-data");
|
||||
});
|
||||
|
||||
test("should carry the baseline through to the saved report on disk", () => {
|
||||
tempDir = setupTempDir();
|
||||
const posts = [
|
||||
priorPost("2025-12-15", 100),
|
||||
priorPost("2025-12-17", 110),
|
||||
priorPost("2025-12-29", 120),
|
||||
priorPost("2025-12-31", 130),
|
||||
priorPost("2026-01-05", 140),
|
||||
createTestPost({ id: "in-week", publishedDate: "2026-01-14" }),
|
||||
];
|
||||
saveBatch(tempDir, createTestBatch({ posts }));
|
||||
|
||||
generateWeeklyReport(tempDir, "2026-W03");
|
||||
const reloaded = loadWeeklyReport(tempDir, "2026-W03");
|
||||
|
||||
assert.ok(reloaded?.baseline, "The block must survive the round-trip to disk");
|
||||
assert.equal(reloaded!.baseline!.impressions.status, "ok");
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -2709,6 +2709,227 @@ fi
|
|||
|
||||
echo ""
|
||||
|
||||
# --- Section 16x: Baseline Engine - Median, Band, Refusal (N17 / E#5) ---
|
||||
echo "--- Baseline Engine: median + band + minimum-N refusal (N17) ---"
|
||||
|
||||
# At a normal cadence a week holds two or three posts, so an absolute total answers
|
||||
# nothing on its own and a week-over-week percentage answers worse than nothing: it
|
||||
# swings on sample composition and reads as a trend. The engine's whole value is in
|
||||
# two disciplines, and BOTH fail silently if eroded:
|
||||
# (E#5 robust) median and MAD, never mean and stddev. One viral post would
|
||||
# otherwise define "normal" as a number no ordinary post reaches -
|
||||
# and the operator would read every ordinary week as a decline.
|
||||
# (E#5 refusal) below the minimum N there is NO verdict. This is the check most
|
||||
# likely to be "helpfully" removed later, because a refusal looks
|
||||
# like missing functionality rather than the feature it is.
|
||||
# (E#5 honesty) the baseline excludes the reported period (else the week is partly
|
||||
# compared with itself), the band is floored at 0, an unlabelled post
|
||||
# is never bucketed, and each group is judged on its OWN n.
|
||||
# (rendering) the report LEADS with the baseline and prints the code's reading
|
||||
# rather than eyeballing the band; the refusal is never softened
|
||||
# into a hedge.
|
||||
STATS_N17="scripts/analytics/src/utils/stats.ts"
|
||||
TYPES_N17="scripts/analytics/src/models/types.ts"
|
||||
QJOIN_N17="scripts/analytics/src/utils/queue-join.ts"
|
||||
WEEKLY_N17="scripts/analytics/src/reports/weekly.ts"
|
||||
MONTHLY_N17="scripts/analytics/src/reports/monthly.ts"
|
||||
RPT_N17="commands/report.md"
|
||||
ANA_N17="commands/analyze.md"
|
||||
|
||||
refusal_honest() { # $1 = text; honest iff it REFUSES a verdict, NAMES the threshold, and FORBIDS a substitute
|
||||
echo "$1" | grep -qF "insufficient-data" \
|
||||
&& echo "$1" | grep -qF "without" \
|
||||
&& echo "$1" | grep -qF "do not call it a trend"
|
||||
}
|
||||
|
||||
RF_SELFTEST_OK=1
|
||||
if ! refusal_honest "when status is insufficient-data, report the numbers without a verdict and do not call it a trend"; then
|
||||
RF_SELFTEST_OK=0; echo " non-vacuity FAIL: a fully-honest refusal probe was not detected"
|
||||
fi
|
||||
while IFS= read -r probe; do
|
||||
[ -z "$probe" ] && continue
|
||||
if refusal_honest "$probe"; then
|
||||
RF_SELFTEST_OK=0; echo " false-positive FAIL: under-specified refusal probe accepted -> $probe"
|
||||
fi
|
||||
done <<'NEGATIVE16X'
|
||||
when data is thin, report the numbers without a verdict and do not call it a trend
|
||||
status insufficient-data means show the week-over-week percentage without a verdict instead
|
||||
insufficient-data: say the trend is early but real, without a firm verdict
|
||||
NEGATIVE16X
|
||||
if [ "$RF_SELFTEST_OK" -eq 1 ]; then
|
||||
pass "refusal self-test: predicate needs the status token + no-verdict + no-trend-substitute (1 accepted, 3 under-specified rejected)"
|
||||
else
|
||||
fail "refusal self-test failed - the N17 refusal lint is vacuous or over-eager"
|
||||
fi
|
||||
|
||||
# (E#5 robust) the robust primitives exist and are documented as the outlier-proof pair
|
||||
if grep -qF "export function median" "$STATS_N17" 2>/dev/null \
|
||||
&& grep -qF "export function medianAbsoluteDeviation" "$STATS_N17" 2>/dev/null; then
|
||||
pass "stats exposes median + MAD, the outlier-robust pair the baseline is built on (E#5)"
|
||||
else
|
||||
fail "$STATS_N17 has no median/MAD - the baseline would rest on mean/stddev (E#5)"
|
||||
fi
|
||||
|
||||
# (E#5 robust) the reason mean/stddev were NOT used is written down, so it survives a refactor
|
||||
if grep -qF "one viral post" "$STATS_N17" 2>/dev/null; then
|
||||
pass "stats records WHY the baseline is median-based (one viral post must not define normal)"
|
||||
else
|
||||
fail "$STATS_N17 does not state why median/MAD replace mean/stddev - the choice will be undone (E#5)"
|
||||
fi
|
||||
|
||||
# (E#5 robust) absent data stays absent: an empty history is not a zero baseline
|
||||
if grep -qF "number | undefined" "$STATS_N17" 2>/dev/null \
|
||||
&& grep -qF "0 would be a fabricated baseline" "$STATS_N17" 2>/dev/null; then
|
||||
pass "median returns unknown for an empty history, never 0 (same discipline as the reach roll-up)"
|
||||
else
|
||||
fail "$STATS_N17 may return 0 for an empty history - a fabricated baseline (E#5)"
|
||||
fi
|
||||
|
||||
# (E#5 refusal) the threshold is a named constant with a stated rationale, and overridable
|
||||
if grep -qF "export const MIN_BASELINE_N" "$STATS_N17" 2>/dev/null \
|
||||
&& grep -qF "documented default, overridable per call" "$STATS_N17" 2>/dev/null; then
|
||||
pass "the minimum-N threshold is a named, justified, overridable constant (E#5)"
|
||||
else
|
||||
fail "$STATS_N17 has no justified MIN_BASELINE_N - the threshold is a magic number (E#5)"
|
||||
fi
|
||||
|
||||
# (E#5 refusal) the window is in POSTS, and the reason a calendar window was rejected is kept
|
||||
if grep -qF "export const BASELINE_WINDOW" "$STATS_N17" 2>/dev/null \
|
||||
&& grep -qF "silently shrinks" "$STATS_N17" 2>/dev/null; then
|
||||
pass "the window is counted in posts, with the calendar-window failure mode documented"
|
||||
else
|
||||
fail "$STATS_N17 window is unnamed or undocumented - a calendar window will creep back in (E#5)"
|
||||
fi
|
||||
|
||||
# (E#5 refusal) refusal is a first-class RESULT, not an exception or a null
|
||||
if grep -qF '"insufficient-data"' "$TYPES_N17" 2>/dev/null \
|
||||
&& grep -qF "BaselineRefusal" "$TYPES_N17" 2>/dev/null; then
|
||||
pass "the refusal is a typed result the caller must handle, not a null or a throw (E#5)"
|
||||
else
|
||||
fail "$TYPES_N17 has no BaselineRefusal type - a refusal can be lost at the call site (E#5)"
|
||||
fi
|
||||
|
||||
# (E#5 refusal) a refused baseline cannot be read as a verdict downstream
|
||||
if grep -qF '"no-verdict"' "$TYPES_N17" 2>/dev/null \
|
||||
&& grep -qF "readAgainstBaseline" "$STATS_N17" 2>/dev/null; then
|
||||
pass "a refused baseline propagates as no-verdict through the reading function (E#5)"
|
||||
else
|
||||
fail "$STATS_N17 reading path can produce a verdict from a refused baseline (E#5)"
|
||||
fi
|
||||
|
||||
# (E#5 honesty) the reported period is EXCLUDED from its own baseline
|
||||
if grep -qF "STRICTLY BEFORE" "$STATS_N17" 2>/dev/null \
|
||||
&& grep -qF "historyBefore" "$TYPES_N17" 2>/dev/null; then
|
||||
pass "the baseline excludes the reported period, so the comparison is not partly with itself"
|
||||
else
|
||||
fail "$STATS_N17 does not exclude the reported period from its own baseline (E#5)"
|
||||
fi
|
||||
|
||||
# (E#5 honesty) the period is compared on its MEDIAN, not its mean
|
||||
if grep -qF "period's MEDIAN, not its mean" "$STATS_N17" 2>/dev/null; then
|
||||
pass "the period is read against the band on its median (one viral post cannot fake a trend)"
|
||||
else
|
||||
fail "$STATS_N17 may compare a period mean against a median band (E#5)"
|
||||
fi
|
||||
|
||||
# (E#5 honesty) the band cannot go negative on a non-negative metric
|
||||
if grep -qF "Math.max(0, mid - k * mad)" "$STATS_N17" 2>/dev/null; then
|
||||
pass "the band's lower bound is floored at 0 for naturally non-negative metrics"
|
||||
else
|
||||
fail "$STATS_N17 band can present a negative lower bound (E#5)"
|
||||
fi
|
||||
|
||||
# (E#5 honesty) per-group refusal: each group judged on its own n, no phantom groups
|
||||
if grep -qF "export function baselineByGroup" "$STATS_N17" 2>/dev/null \
|
||||
&& grep -qF "judged on ITS OWN count" "$STATS_N17" 2>/dev/null; then
|
||||
pass "grouped baselines are judged per group, so a barely-used format gets a refusal (E#5)"
|
||||
else
|
||||
fail "$STATS_N17 grouped baseline can borrow the overall n for a thin group (E#5)"
|
||||
fi
|
||||
|
||||
# (E#5 honesty) an unlabelled post is skipped, never collected into a group called "undefined"
|
||||
if grep -qF "never collected into a phantom group" "$STATS_N17" 2>/dev/null; then
|
||||
pass "unlabelled posts are skipped rather than bucketed as a format that does not exist"
|
||||
else
|
||||
fail "$STATS_N17 may bucket unlabelled posts into a phantom group (E#5)"
|
||||
fi
|
||||
|
||||
# (join) the format/pillar join refuses to guess, and never reuses one entry twice
|
||||
if grep -qF "resolves to UNLABELLED" "$QJOIN_N17" 2>/dev/null \
|
||||
&& grep -qF "labels at most one post" "$QJOIN_N17" 2>/dev/null; then
|
||||
pass "the queue join leaves ambiguity unlabelled and never double-counts an entry (E#5)"
|
||||
else
|
||||
fail "$QJOIN_N17 join may guess a label or reuse an entry - it would poison the baseline (E#5)"
|
||||
fi
|
||||
|
||||
# (join) a missing or broken queue degrades to "no labels", never takes the report down
|
||||
if grep -qF "degrade to \"no labels\"" "$QJOIN_N17" 2>/dev/null; then
|
||||
pass "a missing/malformed queue yields no labels instead of failing the report"
|
||||
else
|
||||
fail "$QJOIN_N17 does not document the degrade-to-unlabelled contract (E#5)"
|
||||
fi
|
||||
|
||||
# (wiring) both period reports carry the block, refusal included - absence is ambiguous
|
||||
if grep -qF "buildBaselineBlock" "$WEEKLY_N17" 2>/dev/null \
|
||||
&& grep -qF "buildBaselineBlock" "$MONTHLY_N17" 2>/dev/null; then
|
||||
pass "weekly and monthly reports both attach a baseline block (E#5)"
|
||||
else
|
||||
fail "a period report does not attach a baseline block - the framing is unavailable (E#5)"
|
||||
fi
|
||||
|
||||
if grep -qF "refusal included" "$WEEKLY_N17" 2>/dev/null; then
|
||||
pass "the weekly block is always present, refusal included (absent != pre-N17 report)"
|
||||
else
|
||||
fail "$WEEKLY_N17 may omit the baseline block, which is indistinguishable from an old report"
|
||||
fi
|
||||
|
||||
# (rendering) the report LEADS with the baseline and does not compute the verdict itself
|
||||
if grep -qF "LEADS with the operator's own baseline" "$RPT_N17" 2>/dev/null \
|
||||
&& grep -qF "Do not compute the verdict yourself" "$RPT_N17" 2>/dev/null; then
|
||||
pass "/linkedin:report leads with the baseline and prints the code's reading (E#5)"
|
||||
else
|
||||
fail "$RPT_N17 does not lead with the baseline, or judges the band by eye (E#5)"
|
||||
fi
|
||||
|
||||
# (rendering) within-band is stated as normal variation, not dressed up as momentum
|
||||
if grep -qF "normal variation" "$RPT_N17" 2>/dev/null \
|
||||
&& grep -qF "not a trend" "$RPT_N17" 2>/dev/null; then
|
||||
pass "an in-band week is reported as normal variation rather than as momentum"
|
||||
else
|
||||
fail "$RPT_N17 may present in-band variation as a trend (E#5)"
|
||||
fi
|
||||
|
||||
# (rendering) the refusal is honest: named status, no verdict, no substitute metric
|
||||
if refusal_honest "$(cat "$RPT_N17" 2>/dev/null)"; then
|
||||
pass "/linkedin:report refuses the verdict on thin data without substituting a percentage (E#5)"
|
||||
else
|
||||
fail "report.md softens or skips the minimum-N refusal (E#5)"
|
||||
fi
|
||||
|
||||
# (rendering) week-over-week loses to the baseline when they disagree
|
||||
if grep -qF "the baseline wins" "$RPT_N17" 2>/dev/null; then
|
||||
pass "report.md resolves a WoW-vs-baseline disagreement in the baseline's favour (E#5)"
|
||||
else
|
||||
fail "$RPT_N17 does not say which frame wins when WoW and baseline disagree (E#5)"
|
||||
fi
|
||||
|
||||
# (rendering) the diagnostic surface verifies the problem is real BEFORE diagnosing it
|
||||
if grep -qF "establish whether there is a problem at all" "$ANA_N17" 2>/dev/null \
|
||||
&& grep -qF "within-band" "$ANA_N17" 2>/dev/null; then
|
||||
pass "/linkedin:analyze checks the drop against the band before diagnosing it (E#5)"
|
||||
else
|
||||
fail "$ANA_N17 diagnoses a reported drop without first testing it against the baseline (E#5)"
|
||||
fi
|
||||
|
||||
# (rendering) analyze refuses to upgrade a refusal into a diagnosis
|
||||
if grep -qF "Never upgrade a refusal into a verdict" "$ANA_N17" 2>/dev/null; then
|
||||
pass "analyze keeps a no-verdict provisional instead of diagnosing on self-report alone"
|
||||
else
|
||||
fail "$ANA_N17 may turn a no-verdict into a confident diagnosis (E#5)"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
# --- Section 18: Assertion-Count Anti-Erosion (SC6) ---
|
||||
# The lint self-modifies its own checks, so a green run could mask a silently dropped
|
||||
# assertion. Pin the total pass()+fail() invocations as a monotonic floor; the count
|
||||
|
|
@ -2779,12 +3000,22 @@ echo ""
|
|||
# weighted roll-up/zero-weight grep + import reach-column grep + report reach-split/
|
||||
# never-estimate compound grep + report reach do-next grep + A2-F11 gated-update
|
||||
# compound grep + shipped-template write-ban grep + boundary-map reach/unverified grep +
|
||||
# boundary-map dwell/saves grep + data-README entry-rules grep) = 228.
|
||||
# boundary-map dwell/saves grep + data-README entry-rules grep) = 228; +23 for N17's
|
||||
# twenty-three UNCONDITIONAL Section-16x checks (refusal self-test + median/MAD grep +
|
||||
# why-median rationale grep + empty-history-unknown grep + MIN_BASELINE_N justified-
|
||||
# constant grep + BASELINE_WINDOW posts-not-calendar grep + BaselineRefusal typed-result
|
||||
# grep + no-verdict propagation grep + excludes-reported-period grep + period-median-not-
|
||||
# mean grep + band-floored-at-0 grep + per-group own-n grep + no-phantom-group grep +
|
||||
# join-unlabelled/no-reuse grep + join-degrade-to-no-labels grep + weekly+monthly block
|
||||
# wiring grep + always-present/refusal-included grep + report leads-with-baseline/no-eyeball
|
||||
# compound grep + normal-variation-not-trend grep + report refusal-honest compound grep +
|
||||
# WoW-loses-to-baseline grep + analyze verify-before-diagnose grep + analyze never-upgrade-
|
||||
# refusal grep) = 251.
|
||||
# NB: the floor tracks the deps-absent MINIMUM (conditional TS suites warn-skip and drop
|
||||
# the count), so it is bumped only by UNCONDITIONAL new checks — NOT pinned to the
|
||||
# deps-present TOTAL_CHECKS (that would zero the warn-skip margin and false-fail a fresh
|
||||
# clone). Runs last so TOTAL_CHECKS sees every prior check.
|
||||
ASSERT_BASELINE_FLOOR=228
|
||||
ASSERT_BASELINE_FLOOR=251
|
||||
TOTAL_CHECKS=$((PASS + FAIL))
|
||||
if [ "$TOTAL_CHECKS" -ge "$ASSERT_BASELINE_FLOOR" ]; then
|
||||
pass "assertion-count anti-erosion: $TOTAL_CHECKS checks >= baseline floor $ASSERT_BASELINE_FLOOR"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue