diff --git a/plugins/linkedin-studio/commands/ab-test.md b/plugins/linkedin-studio/commands/ab-test.md index 5d7790f..9c6dd44 100644 --- a/plugins/linkedin-studio/commands/ab-test.md +++ b/plugins/linkedin-studio/commands/ab-test.md @@ -274,7 +274,7 @@ Read each file and check if both variants have 3+ posts logged. Present only tes Read the test file. For each variant: - Calculate average for each metric (impressions, engagement rate, comments, reposts) - Calculate percentage difference: ((B_avg - A_avg) / A_avg) * 100 -- Apply the 20% significance rule from the framework +- Apply the framework's minimum-meaningful-difference threshold (default 20%). This is an effect-size heuristic for "is the gap worth acting on" — NOT a test of statistical significance (organic personal-post volume rarely reaches it) ### 2c.3: Cross-Reference Analytics Data @@ -298,13 +298,15 @@ Output the analysis in this format: **Posts per variant:** A: [X], B: [Y] ### Results Comparison -| Metric | Variant A (Avg) | Variant B (Avg) | Difference | Significant? | +| Metric | Variant A (Avg) | Variant B (Avg) | Difference | Directional? | |--------|----------------|----------------|------------|--------------| | Impressions | X | X | +X% | Yes/No | | Engagement Rate | X% | X% | +X% | Yes/No | | Comments | X | X | +X% | Yes/No | | Reposts | X | X | +X% | Yes/No | +_"Directional?" = the gap clears the ~20% minimum-meaningful-difference AND points the same way across most posts. It is a direction to test further, not a statistically significant result._ + ### Verdict [Clear recommendation based on the data:] - **Adopt B:** If B wins with >20% difference on primary metric @@ -312,11 +314,17 @@ Output the analysis in this format: - **Inconclusive:** If results are mixed or inconsistent across posts - **Extend test:** If sample size is borderline or results are close to 20% threshold -### Confidence Level -**[High/Medium/Low]** -- Based on sample size (3 = Medium, 5+ = High) -- Based on consistency across individual posts -- Based on alignment with secondary metrics +### Confidence Level (directional only) +**[Directional signal: weak / moderate / strong]** + +Organic personal-post volume rarely reaches statistical significance: with the +handful of posts per variant a creator realistically gathers (well under the +~50 conversions/variant a significance test would need), treat every result as +**directional, not significant**. Do not declare a statistically confident +"winner" — name a direction to test further. Judge the strength of that signal on: +- Consistency across individual posts (did B beat A on most posts, or one outlier?) +- Size of the gap relative to the ~20% minimum-meaningful-difference threshold +- Alignment with secondary metrics ### Key Insight [One sentence capturing the most important learning for their content strategy]