feat(scripts): add STORM measurement harness with pre-registered thresholds

This commit is contained in:
Kjell Tore Guttormsen 2026-08-12 20:25:27 +02:00
commit b194630842
3 changed files with 605 additions and 0 deletions

119
docs/storm-measurement.md Normal file
View file

@ -0,0 +1,119 @@
# STORM adoption gate — pre-registered measurement protocol
**Status:** thresholds registered, **no measurement run yet.** This document is
committed *before* the first measurement by design: a threshold chosen after
seeing the numbers is not a threshold. **Harness:** `scripts/storm-measure.mjs`
(pinned by `tests/scripts/storm-measure.test.mjs`). **Decides:** whether the
bounded Phase 5 conversation loop (`commands/trekresearch.md` §Loop bound) is
worth turning on by default.
---
## 1. What this gate measures — and what it does not
The gate measures **source and coverage breadth**:
| Metric | Definition | Arm |
|---|---|---|
| `unique_sources` | distinct external sources cited by the run | between-arm median gain |
| dimensions over baseline | `(dimensions dimensions_baseline) / dimensions_baseline` | within-run median across the treatment arm |
It does **not** measure outline quality, answer correctness, synthesis
usefulness, or operator satisfaction. A breadth win is a *necessary* condition
for adoption, never a sufficient one. If the loop widens coverage by 40% and
the resulting briefs read worse, the correct action is still decline — the
number does not overrule a reading of the artifacts.
The second metric is a within-run delta by construction (`dimensions_baseline`
is the interview dimension count, `dimensions` the post-Phase-4.5 list), so it
needs no control arm; the control arm's value is 0 because the loop is inert
below `effort: high`.
## 2. Pre-registered thresholds
Both metrics must clear the bar. A strong result on one axis does not carry a
weak result on the other — the loop's claim is breadth on both.
| Median gain (BOTH metrics) | Verdict | Action |
|---|---|---|
| ≥ 30% | **adopt** | Flip the `VOYAGE_STORM_ENABLED` default (see §5) |
| < 15% | **decline** | Leave the mechanism default-off. This is a **no-op**: nothing is rolled back |
| 15% 30% | **inconclusive** | Keep default-off, gather more runs, re-measure |
| either arm empty | **insufficient-data** | Not a decline — measure more |
`ADOPT_THRESHOLD = 0.30` and `DECLINE_THRESHOLD = 0.15` are exported constants
in `scripts/storm-measure.mjs` and pinned by the test suite. Changing them is a
deliberate, reviewable act, not a tuning knob to be nudged toward a result.
## 3. Excluded runs (the honest denominator)
Runs with `empty_turns > 0` are **excluded from the gain and reported as a
count**. An empty turn is one that spent budget and returned no findings, or
findings without citations. Including those runs decides adoption on a broken
denominator — the loop looks cheap because its failures are averaged into its
successes. The harness prints the excluded count on every invocation; if that
count is a large fraction of the treatment arm, the finding is about the loop's
reliability, and it should be read before the gain figure is read at all.
Rows predating the Step 9 measurement fields carry no `effort` and are dropped
as `legacy` with a count. A stats file where *no* row carries `effort` is a hard
error, not an empty treatment group: a schema gap must never present itself as
"no gain".
## 4. The measurement runs (operator-run, outside this plan)
The measurement itself is **not** part of the implementation plan that built
this harness. It is an operator-run gate between that plan and any adopt commit.
Protocol:
- **n ≥ 5 runs per arm.** Fewer, and the median is an anecdote.
- **The same question set in both arms.** Two briefs, run as `--project` runs.
- The arms differ in exactly one thing: whether the loop is enabled.
- Both arms append to the same `trekresearch-stats.jsonl`; `effort` is the
grouping key that separates them.
```bash
# Control arm (loop inert) — n >= 5
claude -p "/trekresearch --project .claude/projects/<brief-standard-effort>"
# Treatment arm (loop live, effort: high in the brief's phase_signals) — n >= 5
VOYAGE_STORM_ENABLED=1 \
claude -p "/trekresearch --project .claude/projects/<brief-high-effort>"
# The gate
node scripts/storm-measure.mjs --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
# Machine-readable, for a decision record
node scripts/storm-measure.mjs --json --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
# SC activation check — did the latest high-effort run discover any dimension
# beyond its interview baseline? Exit 0 = yes.
node scripts/storm-measure.mjs --activation-check --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
```
## 5. What "adopt" concretely means
Adoption is **one constant**: `isStormEnabled()` in
`lib/util/research-loop-cap.mjs` currently requires `VOYAGE_STORM_ENABLED === '1'`.
Adopt = make the loop's budget non-zero without that opt-in, in a commit that
cites the measurement output.
This asymmetry is deliberate and was designed in before any code was written:
- **decline costs nothing** — the mechanism ships default-off, so declining is
doing nothing. No revert, no removal from a command file two other steps
already rewrote.
- **adopt costs one constant** — plus the enforcement hook
(`hooks/scripts/pre-agent-cap.mjs`) already in place to bound what gets turned
on, and the operator-visible cap-exhaustion message already required by
Phase 5's exit conditions.
## 6. Reading the result honestly
- The gate measures breadth. Say "breadth" in the decision record, not "quality".
- Report the excluded count alongside the gain, always. A 35% gain computed
after excluding 6 of 10 treatment runs is a finding about instability.
- `insufficient-data` is not a decline. Do not resolve it by lowering n.
- A verdict computed from a stats file mixing several question sets measures the
question sets, not the loop.

287
scripts/storm-measure.mjs Normal file
View file

@ -0,0 +1,287 @@
#!/usr/bin/env node
// scripts/storm-measure.mjs
// Step 11 — the STORM adoption gate: deterministic Δ accounting over
// ${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl.
//
// What this decides: whether the bounded Phase 5 conversation loop buys enough
// extra source/coverage breadth to justify flipping VOYAGE_STORM_ENABLED on by
// default in lib/util/research-loop-cap.mjs. Nothing else. The thresholds live
// in docs/storm-measurement.md and were committed BEFORE any measurement run —
// that pre-registration is the whole point, so this script never invents them.
//
// What this does NOT measure: outline quality, answer correctness, or operator
// satisfaction. It measures breadth (distinct sources, dimensions covered).
// A breadth win is necessary for adoption, not sufficient on its own.
//
// Honesty properties, both load-bearing:
// - Runs with empty_turns > 0 are EXCLUDED from the gain and REPORTED. An
// empty turn means the loop spent budget and returned nothing; leaving those
// in decides adoption on a broken denominator.
// - A stats file carrying no `effort` field is a loud error. The silent
// failure this prevents is an empty treatment group reading as "no gain",
// which would decline the mechanism for a schema reason.
//
// Zero deps. Node stdlib only.
import { readFileSync, existsSync } from 'node:fs';
import { join } from 'node:path';
// Pre-registered in docs/storm-measurement.md. Do not tune to fit a result.
export const ADOPT_THRESHOLD = 0.30;
export const DECLINE_THRESHOLD = 0.15;
const STATS_FILENAME = 'trekresearch-stats.jsonl';
// ---- pure core (unit-tested) -------------------------------------------------
/** @param {number[]} xs @returns {number|null} null for an empty list — 0 would read as a measurement. */
export function median(xs) {
if (!Array.isArray(xs) || xs.length === 0) return null;
const s = [...xs].sort((a, b) => a - b);
const mid = s.length >> 1;
return s.length % 2 ? s[mid] : (s[mid - 1] + s[mid]) / 2;
}
/**
* Parse a trekresearch-stats.jsonl body into effort-carrying records.
*
* Rows predating Step 9 have no `effort` field; they are dropped and counted as
* `legacy` rather than silently pooled into the control arm. A file where NO row
* carries `effort` throws see the header note on silent failure.
*
* @param {string} text
* @returns {{records: object[], malformed: number, legacy: number}}
*/
export function parseStats(text) {
const lines = String(text ?? '').split('\n');
const records = [];
let malformed = 0;
let legacy = 0;
let parsed = 0;
for (const line of lines) {
if (!line.trim()) continue;
let rec;
try {
rec = JSON.parse(line);
} catch {
malformed++;
continue;
}
parsed++;
if (typeof rec?.effort !== 'string' || rec.effort.length === 0) {
legacy++;
continue;
}
records.push(rec);
}
if (parsed === 0) {
throw new Error(`storm-measure: no records in stats file (${malformed} malformed line(s)).`);
}
if (records.length === 0) {
throw new Error(
`storm-measure: no records carry an \`effort\` field (${legacy} legacy row(s)). ` +
`The gate groups on \`effort\`; without it there is no treatment arm to measure. ` +
`Re-run the measurement set on a build that emits the Step 9 fields.`,
);
}
return { records, malformed, legacy };
}
/**
* Split off the runs that must not count toward a gain.
* @param {object[]} records
* @returns {{eligible: object[], excluded: number}}
*/
export function partitionEligible(records) {
const eligible = [];
let excluded = 0;
for (const r of records) {
const empty = Number(r.empty_turns ?? 0);
if (Number.isFinite(empty) && empty > 0) excluded++;
else eligible.push(r);
}
return { eligible, excluded };
}
function numbers(records, pick) {
return records.map(pick).filter((n) => Number.isFinite(n));
}
/** Relative gain (treatment control)/control. null when control is absent or zero. */
function relGain(control, treatment) {
if (control === null || treatment === null || control === 0) return null;
return (treatment - control) / control;
}
/**
* Full measurement over a parsed record set.
*
* Treatment arm = `effort: high` (the only effort at which the loop runs).
* Control arm = every other effort.
*
* - sources: between-arm median gain in `unique_sources`.
* - dimensions: within-run median gain of (dimensions dimensions_baseline)
* / dimensions_baseline across the treatment arm. It is a
* within-run delta by construction, so it needs no control arm
* the control arm's is 0, the loop being inert there.
*
* @param {object[]} records
*/
export function measure(records) {
const { eligible, excluded } = partitionEligible(records);
const treatment = eligible.filter((r) => r.effort === 'high');
const control = eligible.filter((r) => r.effort !== 'high');
const srcControl = median(numbers(control, (r) => Number(r.unique_sources)));
const srcTreatment = median(numbers(treatment, (r) => Number(r.unique_sources)));
const sourcesGain = relGain(srcControl, srcTreatment);
const dimDeltas = treatment
.map((r) => ({ d: Number(r.dimensions), b: Number(r.dimensions_baseline) }))
.filter(({ d, b }) => Number.isFinite(d) && Number.isFinite(b) && b > 0)
.map(({ d, b }) => (d - b) / b);
const dimensionsGain = median(dimDeltas);
return {
control: { n: control.length, sources: srcControl },
treatment: { n: treatment.length, sources: srcTreatment },
sources: { control: srcControl, treatment: srcTreatment, gain: sourcesGain },
dimensions: { gain: dimensionsGain, n: dimDeltas.length },
excluded,
verdict: decideVerdict(sourcesGain, dimensionsGain),
};
}
/**
* Pre-registered mapping. Adopt requires BOTH metrics to clear the bar: a
* breadth win on one axis alone is not the effect the loop claims.
*
* @param {number|null} sourcesGain
* @param {number|null} dimensionsGain
* @returns {'adopt'|'decline'|'inconclusive'|'insufficient-data'}
*/
export function decideVerdict(sourcesGain, dimensionsGain) {
if (sourcesGain === null || sourcesGain === undefined) return 'insufficient-data';
if (dimensionsGain === null || dimensionsGain === undefined) return 'insufficient-data';
if (sourcesGain >= ADOPT_THRESHOLD && dimensionsGain >= ADOPT_THRESHOLD) return 'adopt';
if (sourcesGain < DECLINE_THRESHOLD && dimensionsGain < DECLINE_THRESHOLD) return 'decline';
return 'inconclusive';
}
/**
* SC activation check: did the most recent `effort: high` run actually discover
* dimensions beyond its interview baseline?
* @param {object[]} records
*/
export function activationCheck(records) {
const high = records.filter((r) => r.effort === 'high');
if (high.length === 0) {
return { ok: false, reason: 'no `effort: high` run found in stats', discovered_dimensions: null };
}
const last = high[high.length - 1];
const d = Number(last.dimensions);
const b = Number(last.dimensions_baseline);
if (!Number.isFinite(d) || !Number.isFinite(b)) {
return { ok: false, reason: 'latest high run lacks dimensions/dimensions_baseline', discovered_dimensions: null };
}
const discovered = d - b;
return {
ok: discovered >= 1,
ts: last.ts ?? null,
dimensions: d,
dimensions_baseline: b,
discovered_dimensions: discovered,
conv_turns: Number(last.conv_turns ?? 0),
empty_turns: Number(last.empty_turns ?? 0),
};
}
// ---- CLI shim ----------------------------------------------------------------
function pct(x) {
return x === null ? 'n/a' : `${(x * 100).toFixed(1)}%`;
}
function defaultStatsPath(env = process.env) {
const dir = env.CLAUDE_PLUGIN_DATA;
return dir ? join(dir, STATS_FILENAME) : null;
}
function parseArgs(argv) {
const o = { stats: null, json: false, activation: false, help: false };
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === '--stats') o.stats = argv[++i];
else if (a === '--json') o.json = true;
else if (a === '--activation-check') o.activation = true;
else if (a === '--help' || a === '-h') o.help = true;
else { process.stderr.write(`Unknown argument: ${a}\n`); process.exit(2); }
}
return o;
}
function mainCli() {
const o = parseArgs(process.argv.slice(2));
if (o.help) {
process.stdout.write(
'Usage: storm-measure.mjs [--stats FILE] [--activation-check] [--json]\n' +
' Default --stats: ${CLAUDE_PLUGIN_DATA}/' + STATS_FILENAME + '\n' +
' Thresholds (pre-registered, docs/storm-measurement.md): ' +
`adopt >= ${ADOPT_THRESHOLD * 100}%, decline < ${DECLINE_THRESHOLD * 100}%\n`,
);
process.exit(0);
}
const statsPath = o.stats || defaultStatsPath();
if (!statsPath) {
process.stderr.write('storm-measure: CLAUDE_PLUGIN_DATA is not set and no --stats FILE was given.\n');
process.exit(2);
}
if (!existsSync(statsPath)) {
process.stderr.write(`storm-measure: stats file not found: ${statsPath}\n`);
process.exit(2);
}
let parsed;
try {
parsed = parseStats(readFileSync(statsPath, 'utf-8'));
} catch (e) {
process.stderr.write(`${e.message}\n`);
process.exit(2);
}
if (o.activation) {
const res = activationCheck(parsed.records);
process.stdout.write(JSON.stringify(res, null, 2) + '\n');
process.exit(res.ok ? 0 : 1);
}
const m = measure(parsed.records);
if (o.json) {
process.stdout.write(JSON.stringify({ statsPath, ...m, malformed: parsed.malformed, legacy: parsed.legacy }, null, 2) + '\n');
process.exit(0);
}
const L = [];
L.push(`STORM adoption gate — ${statsPath}`);
L.push(` control (effort != high): n=${m.control.n} median unique_sources=${m.control.sources ?? 'n/a'}`);
L.push(` treatment (effort = high): n=${m.treatment.n} median unique_sources=${m.treatment.sources ?? 'n/a'}`);
L.push(` excluded (empty_turns > 0): ${m.excluded}`);
if (parsed.legacy) L.push(` legacy rows without \`effort\`: ${parsed.legacy}`);
if (parsed.malformed) L.push(` malformed lines: ${parsed.malformed}`);
L.push('');
L.push(` median gain, unique_sources: ${pct(m.sources.gain)}`);
L.push(` median gain, dimensions over baseline: ${pct(m.dimensions.gain)} (n=${m.dimensions.n})`);
L.push('');
L.push(` thresholds: adopt >= ${pct(ADOPT_THRESHOLD)} on BOTH · decline < ${pct(DECLINE_THRESHOLD)} on BOTH`);
L.push(` VERDICT: ${m.verdict}`);
process.stdout.write(L.join('\n') + '\n');
process.exit(0);
}
if (import.meta.url === `file://${process.argv[1]}`) {
mainCli();
}

View file

@ -0,0 +1,199 @@
// tests/scripts/storm-measure.test.mjs
// Step 11 — the STORM adoption gate's deterministic accounting core.
//
// The gate decides ONE thing: does the bounded Phase 5 loop buy enough extra
// source/coverage breadth to be worth flipping VOYAGE_STORM_ENABLED on by
// default. Thresholds are pre-registered in docs/storm-measurement.md BEFORE
// any measurement run, so this file pins the arithmetic that turns a
// trekresearch-stats.jsonl into a verdict — not the verdict itself.
//
// Two properties carry the gate's honesty:
// - runs with empty_turns > 0 are EXCLUDED from the gain and COUNTED, so
// adoption is never decided on a broken denominator, and
// - a stats file with no `effort` field is a loud error, never a silently
// empty group that reads as "no gain".
//
// Pattern: tests/scripts/synthesis-measure.test.mjs (pure core, no fixtures on disk).
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import {
median,
parseStats,
partitionEligible,
measure,
decideVerdict,
ADOPT_THRESHOLD,
DECLINE_THRESHOLD,
} from '../../scripts/storm-measure.mjs';
// ---------------------------------------------------------------------------
// helpers — synthetic JSONL, one object per line, exactly as the orchestrator emits
// ---------------------------------------------------------------------------
function run({ effort, unique_sources, dimensions, dimensions_baseline, empty_turns = 0, conv_turns = 0 }) {
return JSON.stringify({
ts: '2026-08-12T00:00:00.000Z',
question: 'q',
mode: 'full',
scope: 'both',
engine: 'swarm',
effort,
unique_sources,
dimensions,
dimensions_baseline,
conv_turns,
empty_turns,
});
}
function jsonl(...lines) {
return lines.join('\n') + '\n';
}
// A control arm at 10 sources / 5 dimensions, and a treatment arm at 13
// sources / 8 dimensions: +30.0% sources, +60.0% dimensions.
const STANDARD = [
run({ effort: 'standard', unique_sources: 10, dimensions: 5, dimensions_baseline: 5 }),
run({ effort: 'standard', unique_sources: 10, dimensions: 5, dimensions_baseline: 5 }),
run({ effort: 'standard', unique_sources: 10, dimensions: 5, dimensions_baseline: 5 }),
];
const HIGH = [
run({ effort: 'high', unique_sources: 13, dimensions: 8, dimensions_baseline: 5, conv_turns: 3 }),
run({ effort: 'high', unique_sources: 13, dimensions: 8, dimensions_baseline: 5, conv_turns: 3 }),
run({ effort: 'high', unique_sources: 13, dimensions: 8, dimensions_baseline: 5, conv_turns: 3 }),
];
// ---------------------------------------------------------------------------
// median
// ---------------------------------------------------------------------------
test('median: odd, even, single', () => {
assert.equal(median([3, 1, 2]), 2);
assert.equal(median([1, 2, 3, 4]), 2.5);
assert.equal(median([7]), 7);
});
test('median: empty list is null, never 0 — 0 would read as a real measurement', () => {
assert.equal(median([]), null);
});
test('median does not mutate its input', () => {
const xs = [3, 1, 2];
median(xs);
assert.deepEqual(xs, [3, 1, 2]);
});
// ---------------------------------------------------------------------------
// parseStats — the loud-error requirement
// ---------------------------------------------------------------------------
test('parseStats: a file with no effort field throws, it does not yield empty groups', () => {
const noEffort = jsonl(
JSON.stringify({ ts: 'x', unique_sources: 10, dimensions: 5, dimensions_baseline: 5 }),
JSON.stringify({ ts: 'y', unique_sources: 11, dimensions: 5, dimensions_baseline: 5 }),
);
assert.throws(() => parseStats(noEffort), /effort/i);
});
test('parseStats: skips blank and malformed lines but keeps the good ones', () => {
const text = jsonl(STANDARD[0], '', 'not json', HIGH[0]);
const { records, malformed } = parseStats(text);
assert.equal(records.length, 2);
assert.equal(malformed, 1);
});
test('parseStats: an empty file throws rather than reporting a zero-gain verdict', () => {
assert.throws(() => parseStats(''), /no records/i);
});
// ---------------------------------------------------------------------------
// exclusion of broken runs
// ---------------------------------------------------------------------------
test('partitionEligible: runs with empty_turns > 0 are excluded and counted', () => {
const { records } = parseStats(jsonl(
...HIGH,
run({ effort: 'high', unique_sources: 99, dimensions: 8, dimensions_baseline: 5, empty_turns: 2 }),
));
const { eligible, excluded } = partitionEligible(records);
assert.equal(eligible.length, 3);
assert.equal(excluded, 1);
});
test('partitionEligible: empty_turns === 0 is eligible; a missing field counts as 0', () => {
const { records } = parseStats(jsonl(
STANDARD[0],
JSON.stringify({ ts: 'z', effort: 'standard', unique_sources: 10, dimensions: 5, dimensions_baseline: 5 }),
));
const { eligible, excluded } = partitionEligible(records);
assert.equal(eligible.length, 2);
assert.equal(excluded, 0);
});
// ---------------------------------------------------------------------------
// measure — the known-answer test
// ---------------------------------------------------------------------------
test('measure: median gains on synthetic runs give the known answer', () => {
const m = measure(parseStats(jsonl(...STANDARD, ...HIGH)).records);
assert.equal(m.control.n, 3);
assert.equal(m.treatment.n, 3);
assert.equal(m.sources.control, 10);
assert.equal(m.sources.treatment, 13);
assert.ok(Math.abs(m.sources.gain - 0.30) < 1e-9, `sources gain ${m.sources.gain}`);
// (8 - 5) / 5 = 0.60 within each treatment run.
assert.ok(Math.abs(m.dimensions.gain - 0.60) < 1e-9, `dimensions gain ${m.dimensions.gain}`);
});
test('measure: an excluded run cannot move the median', () => {
const withBroken = jsonl(
...STANDARD,
...HIGH,
run({ effort: 'high', unique_sources: 900, dimensions: 8, dimensions_baseline: 5, empty_turns: 1 }),
);
const m = measure(parseStats(withBroken).records);
assert.equal(m.excluded, 1);
assert.equal(m.sources.treatment, 13, 'the 900-source broken run must not reach the median');
assert.ok(Math.abs(m.sources.gain - 0.30) < 1e-9);
});
test('measure: reports null gain (not 0) when an arm has no eligible runs', () => {
const m = measure(parseStats(jsonl(...HIGH)).records);
assert.equal(m.control.n, 0);
assert.equal(m.sources.gain, null);
assert.equal(m.verdict, 'insufficient-data');
});
// ---------------------------------------------------------------------------
// verdict mapping — both sides of both thresholds
// ---------------------------------------------------------------------------
test('decideVerdict: at and above the adopt threshold', () => {
assert.equal(decideVerdict(ADOPT_THRESHOLD, ADOPT_THRESHOLD), 'adopt');
assert.equal(decideVerdict(0.55, 0.44), 'adopt');
});
test('decideVerdict: just below the adopt threshold is inconclusive, not adopt', () => {
assert.equal(decideVerdict(ADOPT_THRESHOLD - 0.0001, 0.9), 'inconclusive');
});
test('decideVerdict: below the decline threshold on both metrics declines', () => {
assert.equal(decideVerdict(0.14, 0.05), 'decline');
assert.equal(decideVerdict(DECLINE_THRESHOLD - 0.0001, 0), 'decline');
});
test('decideVerdict: at the decline threshold is inconclusive, not decline', () => {
assert.equal(decideVerdict(DECLINE_THRESHOLD, DECLINE_THRESHOLD), 'inconclusive');
});
test('decideVerdict: adopt needs BOTH metrics — one strong metric does not carry a weak one', () => {
assert.equal(decideVerdict(0.90, 0.10), 'inconclusive');
});
test('decideVerdict: a null gain is insufficient data, never a decline', () => {
assert.equal(decideVerdict(null, 0.4), 'insufficient-data');
assert.equal(decideVerdict(0.4, null), 'insufficient-data');
});
// ---------------------------------------------------------------------------
// pre-registration — the thresholds are the doc's, and the doc is committed first
// ---------------------------------------------------------------------------
test('thresholds are the pre-registered 30% / 15%', () => {
assert.equal(ADOPT_THRESHOLD, 0.30);
assert.equal(DECLINE_THRESHOLD, 0.15);
});