Wave 2 / Step 6 of the remediation plan.
Organic personal-post A/B tests gather a handful of posts per variant — far below
the volume a significance test needs — so the tool must not imply statistical
significance:
- Rename the results-table "Significant?" column to "Directional?" and define it
as "clears the ~20% minimum-meaningful-difference AND points the same way across
most posts" — a direction to test further, not a significant result.
- Reword the "20% significance rule" to a minimum-meaningful-difference effect-size
heuristic (explicitly NOT statistical significance).
- Replace the "3 = Medium, 5+ = High" confidence ladder with a directional-only
confidence section: treat every result as directional (not significant) given
realistic volume is well under ~50 conversions/variant; name a direction, not a
winner.
The 20% minimum-meaningful-difference threshold itself stays — it is a legitimate
effect-size heuristic; only the significance framing was the false claim.
Verify: no "Significant?"/"20% significance" remain; "directional" present;
structural lint 0 failed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kjell Tore Guttormsen
55bfe309eb