fix(storm-measure): exclude an unreadable empty_turns, and stop printing BOTH for an OR rule

Two defects in the adoption gate, both in the direction that flatters
adoption.

1. A non-numeric empty_turns counted as an eligible run.
   Number('many') is NaN, and the test was `Number.isFinite(empty) &&
   empty > 0`, so NaN fell through to the eligible branch. Measured
   before: 0 -> eligible, 2 -> excluded, undefined -> eligible, null ->
   eligible, 'many' -> ELIGIBLE, NaN -> ELIGIBLE. The exclusion is one of
   the two properties docs/storm-measurement.md names as carrying this
   gate's honesty, and the run whose bookkeeping broke is the run whose
   numbers deserve the least trust. Now excluded. Absent and null stay
   eligible via `?? 0` - a field never written is a genuine zero on a run
   where the loop never armed.

2. The printed threshold line said "adopt >= 30.0% on BOTH - decline <
   15.0% on BOTH" while decideVerdict evaluates OR on both sides. S82
   restored the pre-registered OR rule in the logic (c37bf50d) and left
   this line describing the stricter AND gate, one line above the verdict
   that OR produced. The summary is the only form of the rule most readers
   ever see, so it now states EITHER on both sides and that adopt wins
   ties. Found while fixing (1); not a review finding.

A test pins the printed line against the string "on BOTH" so the two
cannot drift apart again silently.

Review finding 24a76c21ffc694cd782cd449212c9502d31aeda6 (MINOR).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuGhWAbWyRFBFeemfhxoVv
This commit is contained in:
Kjell Tore Guttormsen 2026-08-12 22:59:08 +02:00
commit 7d5e4fcbcc
3 changed files with 61 additions and 3 deletions

View file

@ -17,6 +17,7 @@
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { readFileSync } from 'node:fs';
import {
median,
parseStats,
@ -127,6 +128,38 @@ test('partitionEligible: empty_turns === 0 is eligible; a missing field counts a
assert.equal(excluded, 0);
});
// A malformed empty_turns used to land in the ELIGIBLE arm: Number('many') is
// NaN, NaN fails the isFinite test, and the `else` branch pushed it in. The
// exclusion is one of the two properties carrying this gate's honesty, so a
// garbage value silently re-entering the denominator defeats it — and it does so
// in the direction that flatters adoption, since the run that broke is the run
// whose numbers are least trustworthy.
test('partitionEligible: a non-numeric empty_turns is EXCLUDED, never silently eligible', () => {
const { records } = parseStats(jsonl(
STANDARD[0],
JSON.stringify({
ts: 'm', effort: 'high', unique_sources: 999,
dimensions: 8, dimensions_baseline: 5, empty_turns: 'many',
}),
));
const { eligible, excluded } = partitionEligible(records);
assert.equal(excluded, 1, 'a value that cannot be read as a turn count is not evidence of zero empty turns');
assert.equal(eligible.length, 1);
});
test('partitionEligible: an unparsable empty_turns cannot move the median either', () => {
const m = measure(parseStats(jsonl(
...STANDARD,
...HIGH,
JSON.stringify({
ts: 'm', effort: 'high', unique_sources: 900,
dimensions: 8, dimensions_baseline: 5, empty_turns: {},
}),
)).records);
assert.equal(m.excluded, 1);
assert.equal(m.sources.treatment, 13, 'the 900-source malformed run must not reach the median');
});
// ---------------------------------------------------------------------------
// measure — the known-answer test
// ---------------------------------------------------------------------------
@ -209,3 +242,15 @@ test('thresholds are the pre-registered 30% / 15%', () => {
assert.equal(ADOPT_THRESHOLD, 0.30);
assert.equal(DECLINE_THRESHOLD, 0.15);
});
// The human-readable summary is the only form of the rule most readers will
// ever see. It said "on BOTH" on both sides while decideVerdict evaluated OR —
// so the report described a stricter gate than the one that produced the verdict
// printed one line below it.
test('the printed threshold line states the OR rule that decideVerdict actually applies', () => {
const src = readFileSync(new URL('../../scripts/storm-measure.mjs', import.meta.url), 'utf-8');
const line = src.split('\n').find((l) => l.includes('thresholds: adopt'));
assert.ok(line, 'the summary must still print its threshold rule');
assert.doesNotMatch(line, /on BOTH/, 'the rule is OR on both sides — printing BOTH misstates the gate');
assert.match(line, /EITHER/);
});