fix(storm-measure): restore the pre-registered OR decision rule
The brief pre-registers "median forbedring >= 30 % pa (a) eller (b) -> adopt. < 15 % -> decline." decideVerdict() required BOTH metrics to clear the adopt bar and BOTH to fall under the decline bar, a rule introduced at execution time (plan Step 11 restated the brief's wording unchanged). The divergence is behavioural, not cosmetic: sources +90% / dimensions +10% is adopt under the brief and was inconclusive in code; sources +2% / dimensions +20% is decline under the brief and was inconclusive in code. Adopt is evaluated before decline, so the OR bars do not conflict where they overlap. docs/storm-measurement.md §2 and the test pins follow the same rule. The stricter AND rule may well be the better decision procedure — but changing it after the fact is exactly what pre-registration exists to prevent. Review finding c37bf50d. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011vPSXe88qp5aqWUqbDNWoF
This commit is contained in:
parent
21a96b9e31
commit
4c4457f6e9
3 changed files with 34 additions and 14 deletions
|
|
@ -31,14 +31,17 @@ below `effort: high`.
|
|||
|
||||
## 2. Pre-registered thresholds
|
||||
|
||||
Both metrics must clear the bar. A strong result on one axis does not carry a
|
||||
weak result on the other — the loop's claim is breadth on both.
|
||||
Either metric clearing the bar is enough. The brief pre-registers "median
|
||||
forbedring ≥ 30 % på (a) eller (b) → adopt. < 15 % → decline" — a breadth win
|
||||
on one axis counts, because either axis widening is the effect the loop claims.
|
||||
Adopt is evaluated first, so a run that clears the adopt bar on one metric is
|
||||
an adopt even when the other metric sits under the decline bar.
|
||||
|
||||
| Median gain (BOTH metrics) | Verdict | Action |
|
||||
| Median gain | Verdict | Action |
|
||||
|---|---|---|
|
||||
| ≥ 30% | **adopt** | Flip the `VOYAGE_STORM_ENABLED` default (see §5) |
|
||||
| < 15% | **decline** | Leave the mechanism default-off. This is a **no-op**: nothing is rolled back |
|
||||
| 15% – 30% | **inconclusive** | Keep default-off, gather more runs, re-measure |
|
||||
| ≥ 30% on **either** metric | **adopt** | Flip the `VOYAGE_STORM_ENABLED` default (see §5) |
|
||||
| < 15% on **either** metric (and no adopt) | **decline** | Leave the mechanism default-off. This is a **no-op**: nothing is rolled back |
|
||||
| both metrics in 15% – 30% | **inconclusive** | Keep default-off, gather more runs, re-measure |
|
||||
| either arm empty | **insufficient-data** | Not a decline — measure more |
|
||||
|
||||
`ADOPT_THRESHOLD = 0.30` and `DECLINE_THRESHOLD = 0.15` are exported constants
|
||||
|
|
|
|||
|
|
@ -155,8 +155,13 @@ export function measure(records) {
|
|||
}
|
||||
|
||||
/**
|
||||
* Pre-registered mapping. Adopt requires BOTH metrics to clear the bar: a
|
||||
* breadth win on one axis alone is not the effect the loop claims.
|
||||
* Pre-registered mapping, verbatim from the brief: "median forbedring >= 30 %
|
||||
* på (a) eller (b) → adopt. < 15 % → decline." OR on both sides, adopt
|
||||
* evaluated first — so a strong win on one axis is an adopt even when the
|
||||
* other axis sits under the decline bar. A stricter AND rule may well be the
|
||||
* better decision procedure, but changing it here is changing the
|
||||
* pre-registration after the fact, which is the one thing the constraint
|
||||
* exists to prevent.
|
||||
*
|
||||
* @param {number|null} sourcesGain
|
||||
* @param {number|null} dimensionsGain
|
||||
|
|
@ -165,8 +170,8 @@ export function measure(records) {
|
|||
export function decideVerdict(sourcesGain, dimensionsGain) {
|
||||
if (sourcesGain === null || sourcesGain === undefined) return 'insufficient-data';
|
||||
if (dimensionsGain === null || dimensionsGain === undefined) return 'insufficient-data';
|
||||
if (sourcesGain >= ADOPT_THRESHOLD && dimensionsGain >= ADOPT_THRESHOLD) return 'adopt';
|
||||
if (sourcesGain < DECLINE_THRESHOLD && dimensionsGain < DECLINE_THRESHOLD) return 'decline';
|
||||
if (sourcesGain >= ADOPT_THRESHOLD || dimensionsGain >= ADOPT_THRESHOLD) return 'adopt';
|
||||
if (sourcesGain < DECLINE_THRESHOLD || dimensionsGain < DECLINE_THRESHOLD) return 'decline';
|
||||
return 'inconclusive';
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -168,8 +168,8 @@ test('decideVerdict: at and above the adopt threshold', () => {
|
|||
assert.equal(decideVerdict(0.55, 0.44), 'adopt');
|
||||
});
|
||||
|
||||
test('decideVerdict: just below the adopt threshold is inconclusive, not adopt', () => {
|
||||
assert.equal(decideVerdict(ADOPT_THRESHOLD - 0.0001, 0.9), 'inconclusive');
|
||||
test('decideVerdict: both metrics between the bars is inconclusive, not adopt', () => {
|
||||
assert.equal(decideVerdict(ADOPT_THRESHOLD - 0.0001, 0.2), 'inconclusive');
|
||||
});
|
||||
|
||||
test('decideVerdict: below the decline threshold on both metrics declines', () => {
|
||||
|
|
@ -181,8 +181,20 @@ test('decideVerdict: at the decline threshold is inconclusive, not decline', ()
|
|||
assert.equal(decideVerdict(DECLINE_THRESHOLD, DECLINE_THRESHOLD), 'inconclusive');
|
||||
});
|
||||
|
||||
test('decideVerdict: adopt needs BOTH metrics — one strong metric does not carry a weak one', () => {
|
||||
assert.equal(decideVerdict(0.90, 0.10), 'inconclusive');
|
||||
// The brief pre-registers "median forbedring >= 30 % på (a) eller (b) → adopt.
|
||||
// < 15 % → decline." — OR on both sides, with adopt evaluated first.
|
||||
test('decideVerdict: adopt needs EITHER metric — one strong metric carries a weak one', () => {
|
||||
assert.equal(decideVerdict(0.90, 0.10), 'adopt');
|
||||
assert.equal(decideVerdict(0.10, 0.90), 'adopt');
|
||||
});
|
||||
|
||||
test('decideVerdict: either metric below the decline bar declines', () => {
|
||||
assert.equal(decideVerdict(0.02, 0.20), 'decline');
|
||||
assert.equal(decideVerdict(0.20, 0.02), 'decline');
|
||||
});
|
||||
|
||||
test('decideVerdict: adopt outranks decline when one metric clears and the other is under the decline bar', () => {
|
||||
assert.equal(decideVerdict(0.90, 0.10), 'adopt');
|
||||
});
|
||||
|
||||
test('decideVerdict: a null gain is insufficient data, never a decline', () => {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue