fix(linkedin-studio): S18 harden ab-test — soften "(Tue-Thu, 8-9 AM optimal)" precision over-claim to "(Tue-Thu mornings)"

This commit is contained in:
Kjell Tore Guttormsen 2026-06-19 13:55:41 +02:00
commit 2eb463f6cd
2 changed files with 60 additions and 1 deletions

View file

@ -125,7 +125,7 @@ Generate a posting schedule that alternates A/B on comparable days:
- Use the user's typical posting days (from state file or ask)
- Alternate variants across the same day slots each week
- Default: 3 posts per variant across 2 weeks (6 total)
- Suggest posting times based on algorithm-signals-reference.md (Tue-Thu, 8-9 AM optimal)
- Suggest posting times based on algorithm-signals-reference.md (Tue-Thu mornings)
### 2a.6: Set Success Criteria

View file

@ -1344,3 +1344,62 @@ weaker relevance match → less reach is defensible and "topic-relevance signal"
- Disposition: **FIXED** (2 edits, `commands/audit.md`) · 0 deferrals · axes a/c PASS, b + d PASS post-fix.
---
### /linkedin:ab-test — design/log/analyze/history/suggest A/B experiments → managed test files, challenger rewrite delegated to content-optimizer
**INTENT.** Stateful analytics command — the most composite class in the queue so far. CLI-backed (Bash
`ls`/`cat`/`mkdir -p` + `Write` of `[test-name].md` test files under the data-dir), agent-delegated
(`content-optimizer` via `Task subagent_type: linkedin-studio:content-optimizer`, `:119`), AND routing
(`/linkedin:post` `:448`, `/linkedin:report` `:459`, `/linkedin:analyze` `:467`). Measure-journey
(reachable via `/linkedin:measure`). Five lifecycle modes (2a2e) map 1:1 to the description: design ·
log · analyze · history · suggest. (STATE predicted `analytics-interpreter` delegation; actual is
`content-optimizer`.)
**SIMULATE.** ICP author → analyze an active hook experiment. Throwaway 2-experiment fixture in
`/private/tmp/claude-abtest-fixture/analytics/ab-tests/` (ACTIVE 6-post + COMPLETED), run via
`LINKEDIN_STUDIO_DATA=<fixture>` to exercise the real seam. Seam-discovery (`:37`,`:270`) found both
files; analyze arithmetic computed correct (eng-rate `:236` per-post 3.0%/5.0%; variant avg + %-diff
`:278279` → eng +66.7%, comments +200%, impressions/reposts +0%); directional test (`:311`) cleared
20% AND consistent across all 3 paired posts → "Adopt B (directional)"; significance honesty (`:320331`)
correctly flagged 3/variant << ~50 → directional, NOT significant. History-parse targets all present in
COMPLETED fixture. Empty `weekly-reports/` + `posts/` → graceful degradation (`:485488`). Fixture
discarded (`find -delete`).
**EVALUATE (8 predicates across 4 axes; class = stateful + analytics + agent-wiring + routing).**
- **P1 (axis-d wiring): PASS**`content-optimizer` resolves; `agents/content-optimizer.md` `name:`
matches the namespaced invocation form.
- **P2 (routing): PASS**`post`/`report`/`analyze`.md all resolve.
- **P3 (intent refs): PASS**`ab-testing-framework.md` + `algorithm-signals-reference.md` exist.
- **P4 (axis-b SSOT): 1 FUNN.** ② 20%-threshold **verbatim** from framework SSOT (`fw:173` = `ab:311`)
✅; ③ significance honesty under-claims (a real significance test needs more than ~50/variant → the
framing can only make the "directional not significant" verdict safer) ✅; **`:128` "(Tue-Thu, 8-9 AM
optimal)"** — sourced (cites the ref) but "optimal" over-claims precision against the SSOT's own `:23`
"trust the *ordering*, test the *number*", and Thu is 9 AM1 PM in SSOT (`alg:144`), not 8 AM; it is the
default slot held constant while testing another variable, so "optimal" is the wrong word.
- **P5 (data-seam): PASS** — all data via `${LINKEDIN_STUDIO_DATA:-…}/`; `:44` = state file (correct), not
a leak; refs via `${CLAUDE_PLUGIN_ROOT}/`.
- **P6 (graceful degradation): PASS** — empty weekly-reports/posts → proceed test-only (`:485488`); 4
error modes (`:474493`).
- **P7 (saves/dwell honesty): PASS** — no saves/dwell mention; eng-rate `:236` excludes saves (S16-consistent).
- **P8 (stateful artifact trace): PASS** — see SIMULATE; full analyze + history path sound against a
realistic file.
**HARDEN (1 surgical edit, `commands/ab-test.md`).**
1. [REWORK · axis-b] `:128` "(Tue-Thu, 8-9 AM optimal)" → "(Tue-Thu mornings)" — dropped the precision
over-claim ("optimal" + the inaccurate 8 AM window); the line stem already cites
`algorithm-signals-reference.md`, so source attribution is retained.
**Out-of-scope (logged, NOT touched in S18).** `:284` "baseline comparison" cross-ref vs `weekly-reports/`
— echoes the S15 "baselines/metadata phantom" vocabulary, but here it reads an existing artifact (the
report command's weekly reports), optional + graceful-degraded; legitimate soft context, not a phantom.
Left as-is (low, no-change).
**VERIFY.**
- Re-grepped final file: `grep "8-9 AM optimal"`**NONE**; `:128` = "...based on
algorithm-signals-reference.md (Tue-Thu mornings)".
- `git diff --stat` = 1 file, **+1 / 1**.
- `bash scripts/test-runner.sh``Passed: 81 · Failed: 0 · Warnings: 0`, **exit 0**; counts **29/19**
unchanged (.md-only edit).
- Disposition: **FIXED** (1 edit, `commands/ab-test.md`) · 0 deferrals · P1P3,P5P8 PASS, P4 PASS post-fix.
---