fix(storm-measure): restore the pre-registered OR decision rule

The brief pre-registers "median forbedring >= 30 % pa (a) eller (b) ->
adopt. < 15 % -> decline." decideVerdict() required BOTH metrics to clear
the adopt bar and BOTH to fall under the decline bar, a rule introduced at
execution time (plan Step 11 restated the brief's wording unchanged).

The divergence is behavioural, not cosmetic: sources +90% / dimensions +10%
is adopt under the brief and was inconclusive in code; sources +2% /
dimensions +20% is decline under the brief and was inconclusive in code.

Adopt is evaluated before decline, so the OR bars do not conflict where they
overlap. docs/storm-measurement.md §2 and the test pins follow the same rule.

The stricter AND rule may well be the better decision procedure — but
changing it after the fact is exactly what pre-registration exists to
prevent. Review finding c37bf50d.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011vPSXe88qp5aqWUqbDNWoF
This commit is contained in:
Kjell Tore Guttormsen 2026-08-12 22:15:07 +02:00
commit 4c4457f6e9
3 changed files with 34 additions and 14 deletions

View file

@ -31,14 +31,17 @@ below `effort: high`.
## 2. Pre-registered thresholds
Both metrics must clear the bar. A strong result on one axis does not carry a
weak result on the other — the loop's claim is breadth on both.
Either metric clearing the bar is enough. The brief pre-registers "median
forbedring ≥ 30 % på (a) eller (b) → adopt. < 15 % → decline" — a breadth win
on one axis counts, because either axis widening is the effect the loop claims.
Adopt is evaluated first, so a run that clears the adopt bar on one metric is
an adopt even when the other metric sits under the decline bar.
| Median gain (BOTH metrics) | Verdict | Action |
| Median gain | Verdict | Action |
|---|---|---|
| ≥ 30% | **adopt** | Flip the `VOYAGE_STORM_ENABLED` default (see §5) |
| < 15% | **decline** | Leave the mechanism default-off. This is a **no-op**: nothing is rolled back |
| 15% 30% | **inconclusive** | Keep default-off, gather more runs, re-measure |
| ≥ 30% on **either** metric | **adopt** | Flip the `VOYAGE_STORM_ENABLED` default (see §5) |
| < 15% on **either** metric (and no adopt) | **decline** | Leave the mechanism default-off. This is a **no-op**: nothing is rolled back |
| both metrics in 15% 30% | **inconclusive** | Keep default-off, gather more runs, re-measure |
| either arm empty | **insufficient-data** | Not a decline — measure more |
`ADOPT_THRESHOLD = 0.30` and `DECLINE_THRESHOLD = 0.15` are exported constants

View file

@ -155,8 +155,13 @@ export function measure(records) {
}
/**
* Pre-registered mapping. Adopt requires BOTH metrics to clear the bar: a
* breadth win on one axis alone is not the effect the loop claims.
* Pre-registered mapping, verbatim from the brief: "median forbedring >= 30 %
* (a) eller (b) adopt. < 15 % decline." OR on both sides, adopt
* evaluated first so a strong win on one axis is an adopt even when the
* other axis sits under the decline bar. A stricter AND rule may well be the
* better decision procedure, but changing it here is changing the
* pre-registration after the fact, which is the one thing the constraint
* exists to prevent.
*
* @param {number|null} sourcesGain
* @param {number|null} dimensionsGain
@ -165,8 +170,8 @@ export function measure(records) {
export function decideVerdict(sourcesGain, dimensionsGain) {
if (sourcesGain === null || sourcesGain === undefined) return 'insufficient-data';
if (dimensionsGain === null || dimensionsGain === undefined) return 'insufficient-data';
if (sourcesGain >= ADOPT_THRESHOLD && dimensionsGain >= ADOPT_THRESHOLD) return 'adopt';
if (sourcesGain < DECLINE_THRESHOLD && dimensionsGain < DECLINE_THRESHOLD) return 'decline';
if (sourcesGain >= ADOPT_THRESHOLD || dimensionsGain >= ADOPT_THRESHOLD) return 'adopt';
if (sourcesGain < DECLINE_THRESHOLD || dimensionsGain < DECLINE_THRESHOLD) return 'decline';
return 'inconclusive';
}

View file

@ -168,8 +168,8 @@ test('decideVerdict: at and above the adopt threshold', () => {
assert.equal(decideVerdict(0.55, 0.44), 'adopt');
});
test('decideVerdict: just below the adopt threshold is inconclusive, not adopt', () => {
assert.equal(decideVerdict(ADOPT_THRESHOLD - 0.0001, 0.9), 'inconclusive');
test('decideVerdict: both metrics between the bars is inconclusive, not adopt', () => {
assert.equal(decideVerdict(ADOPT_THRESHOLD - 0.0001, 0.2), 'inconclusive');
});
test('decideVerdict: below the decline threshold on both metrics declines', () => {
@ -181,8 +181,20 @@ test('decideVerdict: at the decline threshold is inconclusive, not decline', ()
assert.equal(decideVerdict(DECLINE_THRESHOLD, DECLINE_THRESHOLD), 'inconclusive');
});
test('decideVerdict: adopt needs BOTH metrics — one strong metric does not carry a weak one', () => {
assert.equal(decideVerdict(0.90, 0.10), 'inconclusive');
// The brief pre-registers "median forbedring >= 30 % på (a) eller (b) → adopt.
// < 15 % → decline." — OR on both sides, with adopt evaluated first.
test('decideVerdict: adopt needs EITHER metric — one strong metric carries a weak one', () => {
assert.equal(decideVerdict(0.90, 0.10), 'adopt');
assert.equal(decideVerdict(0.10, 0.90), 'adopt');
});
test('decideVerdict: either metric below the decline bar declines', () => {
assert.equal(decideVerdict(0.02, 0.20), 'decline');
assert.equal(decideVerdict(0.20, 0.02), 'decline');
});
test('decideVerdict: adopt outranks decline when one metric clears and the other is under the decline bar', () => {
assert.equal(decideVerdict(0.90, 0.10), 'adopt');
});
test('decideVerdict: a null gain is insufficient data, never a decline', () => {