feat(scripts): add STORM measurement harness with pre-registered thresholds
This commit is contained in:
parent
e9ff8ab023
commit
b194630842
3 changed files with 605 additions and 0 deletions
119
docs/storm-measurement.md
Normal file
119
docs/storm-measurement.md
Normal file
|
|
@ -0,0 +1,119 @@
|
||||||
|
# STORM adoption gate — pre-registered measurement protocol
|
||||||
|
|
||||||
|
**Status:** thresholds registered, **no measurement run yet.** This document is
|
||||||
|
committed *before* the first measurement by design: a threshold chosen after
|
||||||
|
seeing the numbers is not a threshold. **Harness:** `scripts/storm-measure.mjs`
|
||||||
|
(pinned by `tests/scripts/storm-measure.test.mjs`). **Decides:** whether the
|
||||||
|
bounded Phase 5 conversation loop (`commands/trekresearch.md` §Loop bound) is
|
||||||
|
worth turning on by default.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. What this gate measures — and what it does not
|
||||||
|
|
||||||
|
The gate measures **source and coverage breadth**:
|
||||||
|
|
||||||
|
| Metric | Definition | Arm |
|
||||||
|
|---|---|---|
|
||||||
|
| `unique_sources` | distinct external sources cited by the run | between-arm median gain |
|
||||||
|
| dimensions over baseline | `(dimensions − dimensions_baseline) / dimensions_baseline` | within-run median across the treatment arm |
|
||||||
|
|
||||||
|
It does **not** measure outline quality, answer correctness, synthesis
|
||||||
|
usefulness, or operator satisfaction. A breadth win is a *necessary* condition
|
||||||
|
for adoption, never a sufficient one. If the loop widens coverage by 40% and
|
||||||
|
the resulting briefs read worse, the correct action is still decline — the
|
||||||
|
number does not overrule a reading of the artifacts.
|
||||||
|
|
||||||
|
The second metric is a within-run delta by construction (`dimensions_baseline`
|
||||||
|
is the interview dimension count, `dimensions` the post-Phase-4.5 list), so it
|
||||||
|
needs no control arm; the control arm's value is 0 because the loop is inert
|
||||||
|
below `effort: high`.
|
||||||
|
|
||||||
|
## 2. Pre-registered thresholds
|
||||||
|
|
||||||
|
Both metrics must clear the bar. A strong result on one axis does not carry a
|
||||||
|
weak result on the other — the loop's claim is breadth on both.
|
||||||
|
|
||||||
|
| Median gain (BOTH metrics) | Verdict | Action |
|
||||||
|
|---|---|---|
|
||||||
|
| ≥ 30% | **adopt** | Flip the `VOYAGE_STORM_ENABLED` default (see §5) |
|
||||||
|
| < 15% | **decline** | Leave the mechanism default-off. This is a **no-op**: nothing is rolled back |
|
||||||
|
| 15% – 30% | **inconclusive** | Keep default-off, gather more runs, re-measure |
|
||||||
|
| either arm empty | **insufficient-data** | Not a decline — measure more |
|
||||||
|
|
||||||
|
`ADOPT_THRESHOLD = 0.30` and `DECLINE_THRESHOLD = 0.15` are exported constants
|
||||||
|
in `scripts/storm-measure.mjs` and pinned by the test suite. Changing them is a
|
||||||
|
deliberate, reviewable act, not a tuning knob to be nudged toward a result.
|
||||||
|
|
||||||
|
## 3. Excluded runs (the honest denominator)
|
||||||
|
|
||||||
|
Runs with `empty_turns > 0` are **excluded from the gain and reported as a
|
||||||
|
count**. An empty turn is one that spent budget and returned no findings, or
|
||||||
|
findings without citations. Including those runs decides adoption on a broken
|
||||||
|
denominator — the loop looks cheap because its failures are averaged into its
|
||||||
|
successes. The harness prints the excluded count on every invocation; if that
|
||||||
|
count is a large fraction of the treatment arm, the finding is about the loop's
|
||||||
|
reliability, and it should be read before the gain figure is read at all.
|
||||||
|
|
||||||
|
Rows predating the Step 9 measurement fields carry no `effort` and are dropped
|
||||||
|
as `legacy` with a count. A stats file where *no* row carries `effort` is a hard
|
||||||
|
error, not an empty treatment group: a schema gap must never present itself as
|
||||||
|
"no gain".
|
||||||
|
|
||||||
|
## 4. The measurement runs (operator-run, outside this plan)
|
||||||
|
|
||||||
|
The measurement itself is **not** part of the implementation plan that built
|
||||||
|
this harness. It is an operator-run gate between that plan and any adopt commit.
|
||||||
|
|
||||||
|
Protocol:
|
||||||
|
|
||||||
|
- **n ≥ 5 runs per arm.** Fewer, and the median is an anecdote.
|
||||||
|
- **The same question set in both arms.** Two briefs, run as `--project` runs.
|
||||||
|
- The arms differ in exactly one thing: whether the loop is enabled.
|
||||||
|
- Both arms append to the same `trekresearch-stats.jsonl`; `effort` is the
|
||||||
|
grouping key that separates them.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Control arm (loop inert) — n >= 5
|
||||||
|
claude -p "/trekresearch --project .claude/projects/<brief-standard-effort>"
|
||||||
|
|
||||||
|
# Treatment arm (loop live, effort: high in the brief's phase_signals) — n >= 5
|
||||||
|
VOYAGE_STORM_ENABLED=1 \
|
||||||
|
claude -p "/trekresearch --project .claude/projects/<brief-high-effort>"
|
||||||
|
|
||||||
|
# The gate
|
||||||
|
node scripts/storm-measure.mjs --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
|
||||||
|
|
||||||
|
# Machine-readable, for a decision record
|
||||||
|
node scripts/storm-measure.mjs --json --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
|
||||||
|
|
||||||
|
# SC activation check — did the latest high-effort run discover any dimension
|
||||||
|
# beyond its interview baseline? Exit 0 = yes.
|
||||||
|
node scripts/storm-measure.mjs --activation-check --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. What "adopt" concretely means
|
||||||
|
|
||||||
|
Adoption is **one constant**: `isStormEnabled()` in
|
||||||
|
`lib/util/research-loop-cap.mjs` currently requires `VOYAGE_STORM_ENABLED === '1'`.
|
||||||
|
Adopt = make the loop's budget non-zero without that opt-in, in a commit that
|
||||||
|
cites the measurement output.
|
||||||
|
|
||||||
|
This asymmetry is deliberate and was designed in before any code was written:
|
||||||
|
|
||||||
|
- **decline costs nothing** — the mechanism ships default-off, so declining is
|
||||||
|
doing nothing. No revert, no removal from a command file two other steps
|
||||||
|
already rewrote.
|
||||||
|
- **adopt costs one constant** — plus the enforcement hook
|
||||||
|
(`hooks/scripts/pre-agent-cap.mjs`) already in place to bound what gets turned
|
||||||
|
on, and the operator-visible cap-exhaustion message already required by
|
||||||
|
Phase 5's exit conditions.
|
||||||
|
|
||||||
|
## 6. Reading the result honestly
|
||||||
|
|
||||||
|
- The gate measures breadth. Say "breadth" in the decision record, not "quality".
|
||||||
|
- Report the excluded count alongside the gain, always. A 35% gain computed
|
||||||
|
after excluding 6 of 10 treatment runs is a finding about instability.
|
||||||
|
- `insufficient-data` is not a decline. Do not resolve it by lowering n.
|
||||||
|
- A verdict computed from a stats file mixing several question sets measures the
|
||||||
|
question sets, not the loop.
|
||||||
287
scripts/storm-measure.mjs
Normal file
287
scripts/storm-measure.mjs
Normal file
|
|
@ -0,0 +1,287 @@
|
||||||
|
#!/usr/bin/env node
|
||||||
|
// scripts/storm-measure.mjs
|
||||||
|
// Step 11 — the STORM adoption gate: deterministic Δ accounting over
|
||||||
|
// ${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl.
|
||||||
|
//
|
||||||
|
// What this decides: whether the bounded Phase 5 conversation loop buys enough
|
||||||
|
// extra source/coverage breadth to justify flipping VOYAGE_STORM_ENABLED on by
|
||||||
|
// default in lib/util/research-loop-cap.mjs. Nothing else. The thresholds live
|
||||||
|
// in docs/storm-measurement.md and were committed BEFORE any measurement run —
|
||||||
|
// that pre-registration is the whole point, so this script never invents them.
|
||||||
|
//
|
||||||
|
// What this does NOT measure: outline quality, answer correctness, or operator
|
||||||
|
// satisfaction. It measures breadth (distinct sources, dimensions covered).
|
||||||
|
// A breadth win is necessary for adoption, not sufficient on its own.
|
||||||
|
//
|
||||||
|
// Honesty properties, both load-bearing:
|
||||||
|
// - Runs with empty_turns > 0 are EXCLUDED from the gain and REPORTED. An
|
||||||
|
// empty turn means the loop spent budget and returned nothing; leaving those
|
||||||
|
// in decides adoption on a broken denominator.
|
||||||
|
// - A stats file carrying no `effort` field is a loud error. The silent
|
||||||
|
// failure this prevents is an empty treatment group reading as "no gain",
|
||||||
|
// which would decline the mechanism for a schema reason.
|
||||||
|
//
|
||||||
|
// Zero deps. Node stdlib only.
|
||||||
|
|
||||||
|
import { readFileSync, existsSync } from 'node:fs';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
|
||||||
|
// Pre-registered in docs/storm-measurement.md. Do not tune to fit a result.
|
||||||
|
export const ADOPT_THRESHOLD = 0.30;
|
||||||
|
export const DECLINE_THRESHOLD = 0.15;
|
||||||
|
|
||||||
|
const STATS_FILENAME = 'trekresearch-stats.jsonl';
|
||||||
|
|
||||||
|
// ---- pure core (unit-tested) -------------------------------------------------
|
||||||
|
|
||||||
|
/** @param {number[]} xs @returns {number|null} null for an empty list — 0 would read as a measurement. */
|
||||||
|
export function median(xs) {
|
||||||
|
if (!Array.isArray(xs) || xs.length === 0) return null;
|
||||||
|
const s = [...xs].sort((a, b) => a - b);
|
||||||
|
const mid = s.length >> 1;
|
||||||
|
return s.length % 2 ? s[mid] : (s[mid - 1] + s[mid]) / 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Parse a trekresearch-stats.jsonl body into effort-carrying records.
|
||||||
|
*
|
||||||
|
* Rows predating Step 9 have no `effort` field; they are dropped and counted as
|
||||||
|
* `legacy` rather than silently pooled into the control arm. A file where NO row
|
||||||
|
* carries `effort` throws — see the header note on silent failure.
|
||||||
|
*
|
||||||
|
* @param {string} text
|
||||||
|
* @returns {{records: object[], malformed: number, legacy: number}}
|
||||||
|
*/
|
||||||
|
export function parseStats(text) {
|
||||||
|
const lines = String(text ?? '').split('\n');
|
||||||
|
const records = [];
|
||||||
|
let malformed = 0;
|
||||||
|
let legacy = 0;
|
||||||
|
let parsed = 0;
|
||||||
|
|
||||||
|
for (const line of lines) {
|
||||||
|
if (!line.trim()) continue;
|
||||||
|
let rec;
|
||||||
|
try {
|
||||||
|
rec = JSON.parse(line);
|
||||||
|
} catch {
|
||||||
|
malformed++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
parsed++;
|
||||||
|
if (typeof rec?.effort !== 'string' || rec.effort.length === 0) {
|
||||||
|
legacy++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
records.push(rec);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (parsed === 0) {
|
||||||
|
throw new Error(`storm-measure: no records in stats file (${malformed} malformed line(s)).`);
|
||||||
|
}
|
||||||
|
if (records.length === 0) {
|
||||||
|
throw new Error(
|
||||||
|
`storm-measure: no records carry an \`effort\` field (${legacy} legacy row(s)). ` +
|
||||||
|
`The gate groups on \`effort\`; without it there is no treatment arm to measure. ` +
|
||||||
|
`Re-run the measurement set on a build that emits the Step 9 fields.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return { records, malformed, legacy };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Split off the runs that must not count toward a gain.
|
||||||
|
* @param {object[]} records
|
||||||
|
* @returns {{eligible: object[], excluded: number}}
|
||||||
|
*/
|
||||||
|
export function partitionEligible(records) {
|
||||||
|
const eligible = [];
|
||||||
|
let excluded = 0;
|
||||||
|
for (const r of records) {
|
||||||
|
const empty = Number(r.empty_turns ?? 0);
|
||||||
|
if (Number.isFinite(empty) && empty > 0) excluded++;
|
||||||
|
else eligible.push(r);
|
||||||
|
}
|
||||||
|
return { eligible, excluded };
|
||||||
|
}
|
||||||
|
|
||||||
|
function numbers(records, pick) {
|
||||||
|
return records.map(pick).filter((n) => Number.isFinite(n));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Relative gain (treatment − control)/control. null when control is absent or zero. */
|
||||||
|
function relGain(control, treatment) {
|
||||||
|
if (control === null || treatment === null || control === 0) return null;
|
||||||
|
return (treatment - control) / control;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Full measurement over a parsed record set.
|
||||||
|
*
|
||||||
|
* Treatment arm = `effort: high` (the only effort at which the loop runs).
|
||||||
|
* Control arm = every other effort.
|
||||||
|
*
|
||||||
|
* - sources: between-arm median gain in `unique_sources`.
|
||||||
|
* - dimensions: within-run median gain of (dimensions − dimensions_baseline)
|
||||||
|
* / dimensions_baseline across the treatment arm. It is a
|
||||||
|
* within-run delta by construction, so it needs no control arm —
|
||||||
|
* the control arm's is 0, the loop being inert there.
|
||||||
|
*
|
||||||
|
* @param {object[]} records
|
||||||
|
*/
|
||||||
|
export function measure(records) {
|
||||||
|
const { eligible, excluded } = partitionEligible(records);
|
||||||
|
const treatment = eligible.filter((r) => r.effort === 'high');
|
||||||
|
const control = eligible.filter((r) => r.effort !== 'high');
|
||||||
|
|
||||||
|
const srcControl = median(numbers(control, (r) => Number(r.unique_sources)));
|
||||||
|
const srcTreatment = median(numbers(treatment, (r) => Number(r.unique_sources)));
|
||||||
|
const sourcesGain = relGain(srcControl, srcTreatment);
|
||||||
|
|
||||||
|
const dimDeltas = treatment
|
||||||
|
.map((r) => ({ d: Number(r.dimensions), b: Number(r.dimensions_baseline) }))
|
||||||
|
.filter(({ d, b }) => Number.isFinite(d) && Number.isFinite(b) && b > 0)
|
||||||
|
.map(({ d, b }) => (d - b) / b);
|
||||||
|
const dimensionsGain = median(dimDeltas);
|
||||||
|
|
||||||
|
return {
|
||||||
|
control: { n: control.length, sources: srcControl },
|
||||||
|
treatment: { n: treatment.length, sources: srcTreatment },
|
||||||
|
sources: { control: srcControl, treatment: srcTreatment, gain: sourcesGain },
|
||||||
|
dimensions: { gain: dimensionsGain, n: dimDeltas.length },
|
||||||
|
excluded,
|
||||||
|
verdict: decideVerdict(sourcesGain, dimensionsGain),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pre-registered mapping. Adopt requires BOTH metrics to clear the bar: a
|
||||||
|
* breadth win on one axis alone is not the effect the loop claims.
|
||||||
|
*
|
||||||
|
* @param {number|null} sourcesGain
|
||||||
|
* @param {number|null} dimensionsGain
|
||||||
|
* @returns {'adopt'|'decline'|'inconclusive'|'insufficient-data'}
|
||||||
|
*/
|
||||||
|
export function decideVerdict(sourcesGain, dimensionsGain) {
|
||||||
|
if (sourcesGain === null || sourcesGain === undefined) return 'insufficient-data';
|
||||||
|
if (dimensionsGain === null || dimensionsGain === undefined) return 'insufficient-data';
|
||||||
|
if (sourcesGain >= ADOPT_THRESHOLD && dimensionsGain >= ADOPT_THRESHOLD) return 'adopt';
|
||||||
|
if (sourcesGain < DECLINE_THRESHOLD && dimensionsGain < DECLINE_THRESHOLD) return 'decline';
|
||||||
|
return 'inconclusive';
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* SC activation check: did the most recent `effort: high` run actually discover
|
||||||
|
* dimensions beyond its interview baseline?
|
||||||
|
* @param {object[]} records
|
||||||
|
*/
|
||||||
|
export function activationCheck(records) {
|
||||||
|
const high = records.filter((r) => r.effort === 'high');
|
||||||
|
if (high.length === 0) {
|
||||||
|
return { ok: false, reason: 'no `effort: high` run found in stats', discovered_dimensions: null };
|
||||||
|
}
|
||||||
|
const last = high[high.length - 1];
|
||||||
|
const d = Number(last.dimensions);
|
||||||
|
const b = Number(last.dimensions_baseline);
|
||||||
|
if (!Number.isFinite(d) || !Number.isFinite(b)) {
|
||||||
|
return { ok: false, reason: 'latest high run lacks dimensions/dimensions_baseline', discovered_dimensions: null };
|
||||||
|
}
|
||||||
|
const discovered = d - b;
|
||||||
|
return {
|
||||||
|
ok: discovered >= 1,
|
||||||
|
ts: last.ts ?? null,
|
||||||
|
dimensions: d,
|
||||||
|
dimensions_baseline: b,
|
||||||
|
discovered_dimensions: discovered,
|
||||||
|
conv_turns: Number(last.conv_turns ?? 0),
|
||||||
|
empty_turns: Number(last.empty_turns ?? 0),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- CLI shim ----------------------------------------------------------------
|
||||||
|
|
||||||
|
function pct(x) {
|
||||||
|
return x === null ? 'n/a' : `${(x * 100).toFixed(1)}%`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function defaultStatsPath(env = process.env) {
|
||||||
|
const dir = env.CLAUDE_PLUGIN_DATA;
|
||||||
|
return dir ? join(dir, STATS_FILENAME) : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseArgs(argv) {
|
||||||
|
const o = { stats: null, json: false, activation: false, help: false };
|
||||||
|
for (let i = 0; i < argv.length; i++) {
|
||||||
|
const a = argv[i];
|
||||||
|
if (a === '--stats') o.stats = argv[++i];
|
||||||
|
else if (a === '--json') o.json = true;
|
||||||
|
else if (a === '--activation-check') o.activation = true;
|
||||||
|
else if (a === '--help' || a === '-h') o.help = true;
|
||||||
|
else { process.stderr.write(`Unknown argument: ${a}\n`); process.exit(2); }
|
||||||
|
}
|
||||||
|
return o;
|
||||||
|
}
|
||||||
|
|
||||||
|
function mainCli() {
|
||||||
|
const o = parseArgs(process.argv.slice(2));
|
||||||
|
if (o.help) {
|
||||||
|
process.stdout.write(
|
||||||
|
'Usage: storm-measure.mjs [--stats FILE] [--activation-check] [--json]\n' +
|
||||||
|
' Default --stats: ${CLAUDE_PLUGIN_DATA}/' + STATS_FILENAME + '\n' +
|
||||||
|
' Thresholds (pre-registered, docs/storm-measurement.md): ' +
|
||||||
|
`adopt >= ${ADOPT_THRESHOLD * 100}%, decline < ${DECLINE_THRESHOLD * 100}%\n`,
|
||||||
|
);
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
const statsPath = o.stats || defaultStatsPath();
|
||||||
|
if (!statsPath) {
|
||||||
|
process.stderr.write('storm-measure: CLAUDE_PLUGIN_DATA is not set and no --stats FILE was given.\n');
|
||||||
|
process.exit(2);
|
||||||
|
}
|
||||||
|
if (!existsSync(statsPath)) {
|
||||||
|
process.stderr.write(`storm-measure: stats file not found: ${statsPath}\n`);
|
||||||
|
process.exit(2);
|
||||||
|
}
|
||||||
|
|
||||||
|
let parsed;
|
||||||
|
try {
|
||||||
|
parsed = parseStats(readFileSync(statsPath, 'utf-8'));
|
||||||
|
} catch (e) {
|
||||||
|
process.stderr.write(`${e.message}\n`);
|
||||||
|
process.exit(2);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (o.activation) {
|
||||||
|
const res = activationCheck(parsed.records);
|
||||||
|
process.stdout.write(JSON.stringify(res, null, 2) + '\n');
|
||||||
|
process.exit(res.ok ? 0 : 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
const m = measure(parsed.records);
|
||||||
|
|
||||||
|
if (o.json) {
|
||||||
|
process.stdout.write(JSON.stringify({ statsPath, ...m, malformed: parsed.malformed, legacy: parsed.legacy }, null, 2) + '\n');
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
const L = [];
|
||||||
|
L.push(`STORM adoption gate — ${statsPath}`);
|
||||||
|
L.push(` control (effort != high): n=${m.control.n} median unique_sources=${m.control.sources ?? 'n/a'}`);
|
||||||
|
L.push(` treatment (effort = high): n=${m.treatment.n} median unique_sources=${m.treatment.sources ?? 'n/a'}`);
|
||||||
|
L.push(` excluded (empty_turns > 0): ${m.excluded}`);
|
||||||
|
if (parsed.legacy) L.push(` legacy rows without \`effort\`: ${parsed.legacy}`);
|
||||||
|
if (parsed.malformed) L.push(` malformed lines: ${parsed.malformed}`);
|
||||||
|
L.push('');
|
||||||
|
L.push(` median gain, unique_sources: ${pct(m.sources.gain)}`);
|
||||||
|
L.push(` median gain, dimensions over baseline: ${pct(m.dimensions.gain)} (n=${m.dimensions.n})`);
|
||||||
|
L.push('');
|
||||||
|
L.push(` thresholds: adopt >= ${pct(ADOPT_THRESHOLD)} on BOTH · decline < ${pct(DECLINE_THRESHOLD)} on BOTH`);
|
||||||
|
L.push(` VERDICT: ${m.verdict}`);
|
||||||
|
process.stdout.write(L.join('\n') + '\n');
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||||
|
mainCli();
|
||||||
|
}
|
||||||
199
tests/scripts/storm-measure.test.mjs
Normal file
199
tests/scripts/storm-measure.test.mjs
Normal file
|
|
@ -0,0 +1,199 @@
|
||||||
|
// tests/scripts/storm-measure.test.mjs
|
||||||
|
// Step 11 — the STORM adoption gate's deterministic accounting core.
|
||||||
|
//
|
||||||
|
// The gate decides ONE thing: does the bounded Phase 5 loop buy enough extra
|
||||||
|
// source/coverage breadth to be worth flipping VOYAGE_STORM_ENABLED on by
|
||||||
|
// default. Thresholds are pre-registered in docs/storm-measurement.md BEFORE
|
||||||
|
// any measurement run, so this file pins the arithmetic that turns a
|
||||||
|
// trekresearch-stats.jsonl into a verdict — not the verdict itself.
|
||||||
|
//
|
||||||
|
// Two properties carry the gate's honesty:
|
||||||
|
// - runs with empty_turns > 0 are EXCLUDED from the gain and COUNTED, so
|
||||||
|
// adoption is never decided on a broken denominator, and
|
||||||
|
// - a stats file with no `effort` field is a loud error, never a silently
|
||||||
|
// empty group that reads as "no gain".
|
||||||
|
//
|
||||||
|
// Pattern: tests/scripts/synthesis-measure.test.mjs (pure core, no fixtures on disk).
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import {
|
||||||
|
median,
|
||||||
|
parseStats,
|
||||||
|
partitionEligible,
|
||||||
|
measure,
|
||||||
|
decideVerdict,
|
||||||
|
ADOPT_THRESHOLD,
|
||||||
|
DECLINE_THRESHOLD,
|
||||||
|
} from '../../scripts/storm-measure.mjs';
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// helpers — synthetic JSONL, one object per line, exactly as the orchestrator emits
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
function run({ effort, unique_sources, dimensions, dimensions_baseline, empty_turns = 0, conv_turns = 0 }) {
|
||||||
|
return JSON.stringify({
|
||||||
|
ts: '2026-08-12T00:00:00.000Z',
|
||||||
|
question: 'q',
|
||||||
|
mode: 'full',
|
||||||
|
scope: 'both',
|
||||||
|
engine: 'swarm',
|
||||||
|
effort,
|
||||||
|
unique_sources,
|
||||||
|
dimensions,
|
||||||
|
dimensions_baseline,
|
||||||
|
conv_turns,
|
||||||
|
empty_turns,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function jsonl(...lines) {
|
||||||
|
return lines.join('\n') + '\n';
|
||||||
|
}
|
||||||
|
|
||||||
|
// A control arm at 10 sources / 5 dimensions, and a treatment arm at 13
|
||||||
|
// sources / 8 dimensions: +30.0% sources, +60.0% dimensions.
|
||||||
|
const STANDARD = [
|
||||||
|
run({ effort: 'standard', unique_sources: 10, dimensions: 5, dimensions_baseline: 5 }),
|
||||||
|
run({ effort: 'standard', unique_sources: 10, dimensions: 5, dimensions_baseline: 5 }),
|
||||||
|
run({ effort: 'standard', unique_sources: 10, dimensions: 5, dimensions_baseline: 5 }),
|
||||||
|
];
|
||||||
|
const HIGH = [
|
||||||
|
run({ effort: 'high', unique_sources: 13, dimensions: 8, dimensions_baseline: 5, conv_turns: 3 }),
|
||||||
|
run({ effort: 'high', unique_sources: 13, dimensions: 8, dimensions_baseline: 5, conv_turns: 3 }),
|
||||||
|
run({ effort: 'high', unique_sources: 13, dimensions: 8, dimensions_baseline: 5, conv_turns: 3 }),
|
||||||
|
];
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// median
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
test('median: odd, even, single', () => {
|
||||||
|
assert.equal(median([3, 1, 2]), 2);
|
||||||
|
assert.equal(median([1, 2, 3, 4]), 2.5);
|
||||||
|
assert.equal(median([7]), 7);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('median: empty list is null, never 0 — 0 would read as a real measurement', () => {
|
||||||
|
assert.equal(median([]), null);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('median does not mutate its input', () => {
|
||||||
|
const xs = [3, 1, 2];
|
||||||
|
median(xs);
|
||||||
|
assert.deepEqual(xs, [3, 1, 2]);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// parseStats — the loud-error requirement
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
test('parseStats: a file with no effort field throws, it does not yield empty groups', () => {
|
||||||
|
const noEffort = jsonl(
|
||||||
|
JSON.stringify({ ts: 'x', unique_sources: 10, dimensions: 5, dimensions_baseline: 5 }),
|
||||||
|
JSON.stringify({ ts: 'y', unique_sources: 11, dimensions: 5, dimensions_baseline: 5 }),
|
||||||
|
);
|
||||||
|
assert.throws(() => parseStats(noEffort), /effort/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('parseStats: skips blank and malformed lines but keeps the good ones', () => {
|
||||||
|
const text = jsonl(STANDARD[0], '', 'not json', HIGH[0]);
|
||||||
|
const { records, malformed } = parseStats(text);
|
||||||
|
assert.equal(records.length, 2);
|
||||||
|
assert.equal(malformed, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('parseStats: an empty file throws rather than reporting a zero-gain verdict', () => {
|
||||||
|
assert.throws(() => parseStats(''), /no records/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// exclusion of broken runs
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
test('partitionEligible: runs with empty_turns > 0 are excluded and counted', () => {
|
||||||
|
const { records } = parseStats(jsonl(
|
||||||
|
...HIGH,
|
||||||
|
run({ effort: 'high', unique_sources: 99, dimensions: 8, dimensions_baseline: 5, empty_turns: 2 }),
|
||||||
|
));
|
||||||
|
const { eligible, excluded } = partitionEligible(records);
|
||||||
|
assert.equal(eligible.length, 3);
|
||||||
|
assert.equal(excluded, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('partitionEligible: empty_turns === 0 is eligible; a missing field counts as 0', () => {
|
||||||
|
const { records } = parseStats(jsonl(
|
||||||
|
STANDARD[0],
|
||||||
|
JSON.stringify({ ts: 'z', effort: 'standard', unique_sources: 10, dimensions: 5, dimensions_baseline: 5 }),
|
||||||
|
));
|
||||||
|
const { eligible, excluded } = partitionEligible(records);
|
||||||
|
assert.equal(eligible.length, 2);
|
||||||
|
assert.equal(excluded, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// measure — the known-answer test
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
test('measure: median gains on synthetic runs give the known answer', () => {
|
||||||
|
const m = measure(parseStats(jsonl(...STANDARD, ...HIGH)).records);
|
||||||
|
assert.equal(m.control.n, 3);
|
||||||
|
assert.equal(m.treatment.n, 3);
|
||||||
|
assert.equal(m.sources.control, 10);
|
||||||
|
assert.equal(m.sources.treatment, 13);
|
||||||
|
assert.ok(Math.abs(m.sources.gain - 0.30) < 1e-9, `sources gain ${m.sources.gain}`);
|
||||||
|
// (8 - 5) / 5 = 0.60 within each treatment run.
|
||||||
|
assert.ok(Math.abs(m.dimensions.gain - 0.60) < 1e-9, `dimensions gain ${m.dimensions.gain}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('measure: an excluded run cannot move the median', () => {
|
||||||
|
const withBroken = jsonl(
|
||||||
|
...STANDARD,
|
||||||
|
...HIGH,
|
||||||
|
run({ effort: 'high', unique_sources: 900, dimensions: 8, dimensions_baseline: 5, empty_turns: 1 }),
|
||||||
|
);
|
||||||
|
const m = measure(parseStats(withBroken).records);
|
||||||
|
assert.equal(m.excluded, 1);
|
||||||
|
assert.equal(m.sources.treatment, 13, 'the 900-source broken run must not reach the median');
|
||||||
|
assert.ok(Math.abs(m.sources.gain - 0.30) < 1e-9);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('measure: reports null gain (not 0) when an arm has no eligible runs', () => {
|
||||||
|
const m = measure(parseStats(jsonl(...HIGH)).records);
|
||||||
|
assert.equal(m.control.n, 0);
|
||||||
|
assert.equal(m.sources.gain, null);
|
||||||
|
assert.equal(m.verdict, 'insufficient-data');
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// verdict mapping — both sides of both thresholds
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
test('decideVerdict: at and above the adopt threshold', () => {
|
||||||
|
assert.equal(decideVerdict(ADOPT_THRESHOLD, ADOPT_THRESHOLD), 'adopt');
|
||||||
|
assert.equal(decideVerdict(0.55, 0.44), 'adopt');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('decideVerdict: just below the adopt threshold is inconclusive, not adopt', () => {
|
||||||
|
assert.equal(decideVerdict(ADOPT_THRESHOLD - 0.0001, 0.9), 'inconclusive');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('decideVerdict: below the decline threshold on both metrics declines', () => {
|
||||||
|
assert.equal(decideVerdict(0.14, 0.05), 'decline');
|
||||||
|
assert.equal(decideVerdict(DECLINE_THRESHOLD - 0.0001, 0), 'decline');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('decideVerdict: at the decline threshold is inconclusive, not decline', () => {
|
||||||
|
assert.equal(decideVerdict(DECLINE_THRESHOLD, DECLINE_THRESHOLD), 'inconclusive');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('decideVerdict: adopt needs BOTH metrics — one strong metric does not carry a weak one', () => {
|
||||||
|
assert.equal(decideVerdict(0.90, 0.10), 'inconclusive');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('decideVerdict: a null gain is insufficient data, never a decline', () => {
|
||||||
|
assert.equal(decideVerdict(null, 0.4), 'insufficient-data');
|
||||||
|
assert.equal(decideVerdict(0.4, null), 'insufficient-data');
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// pre-registration — the thresholds are the doc's, and the doc is committed first
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
test('thresholds are the pre-registered 30% / 15%', () => {
|
||||||
|
assert.equal(ADOPT_THRESHOLD, 0.30);
|
||||||
|
assert.equal(DECLINE_THRESHOLD, 0.15);
|
||||||
|
});
|
||||||
Loading…
Add table
Add a link
Reference in a new issue