diff --git a/docs/devils-advocate-plan.md b/docs/devils-advocate-plan.md new file mode 100644 index 0000000..c32fe29 --- /dev/null +++ b/docs/devils-advocate-plan.md @@ -0,0 +1,66 @@ +# Devil's-advocate session — adversarial cold audit of Voyage (Dynamic Workflow) + +**Status:** Planned 2026-06-18 (post-S13). **Executes in the NEXT session** (after the operator's `/clear`). +**Operator mandate (2026-06-18, verbatim):** «Planlegg en "devils advocate" sesjon … hvor du nå oppdaterer STATE.md og forklarer at det skal brukes en Dynamic Workflow for analysen. … denne pluginen er veldig viktig for meg.» +**Opt-in:** The operator explicitly requested a **Dynamic Workflow** for the analysis — the next session is authorized to call the `Workflow` tool (multi-agent orchestration). This is a durable opt-in for that session. + +## Why this, why now +v5.5.0 just shipped. Today's reflection surfaced an uncomfortable fact: **Voyage was not used to build Voyage** — the whole upgrade arc was hand-orchestrated with bespoke planning docs. Before investing further, subject the plugin to a **cold, hostile audit that actively tries to prove it is over-engineered, wrong, or unmaintainable** — and see what survives. The operator cares about this plugin; the kindest thing is an honest adversary, not applause. + +## Objective (what the session must produce) +A severity-tagged, **adversarially-verified** critique written to **`docs/devils-advocate-results.md`**: where Voyage is genuinely weak, where its claims outrun its code, and what (if anything) should change. This is a steelman attack **plus** an honest rebuttal pass — not a feature wishlist, not vibes. Every finding cites `file:line`. + +## Why a Dynamic Workflow (not inline, not the Voyage pipeline) +- The audit is a **fan-out → verify → synthesize** shape across independent adversarial angles — exactly the Workflow tool's review/red-team pattern. Independent skeptics that can't see each other's reasoning catch more than one context reasoning sequentially, and the rebuttal pass kills plausible-but-unfair critiques. +- It is deliberately **not** run through Voyage's own `/trek*` pipeline: the point is to judge Voyage from the *outside*, cold — not to dogfood it (that is a separate exercise the prior session flagged). +- The operator asked for a Dynamic Workflow explicitly. + +## Adversarial dimensions — one finder each, build the STRONGEST case AGAINST +Each finder reads the actual code/docs (cite `file:line`); no claim without evidence. + +- **D1 — Ceremony vs. value (existential).** Steelman: "Voyage is a sophistication trap — Plan mode + `/code-review` + a TODO gets 90% of the outcome at 10% of the cognitive/token cost. 7 commands, 24 agents, 697 tests are overhead, not value." Exhibit A: today's non-dogfood. +- **D2 — Today's decisions were rationalizations, not conclusions.** Steelman each: CC-26/27 "narrow hybrid not wholesale" (genuine analysis or status-quo bias? the bake-offs were largely *designed-not-run* or deterministic estimates); NW2 "POSITIVE" at **+4.4% tokens** (is that positive? was fidelity real?); NW3 **Δ=0%** decline (the measurement was `chars/4` with *swept, not API-measured* BASE — is the decline robust or an unfalsifiable-estimate artifact?); v5.5.0 versioning (skipping 5.2–5.4 as "unreleased internal milestones" — coherent for consumers, or post-hoc tidy-up?). +- **D3 — Brief-as-PUBLIC-CONTRACT is fragile.** Steelman: the framing-enforcement invariant ("brief framing must match operator intent") is itself an admission that the pipeline *structurally polishes wrong premises*. Do the three 2.2 defensive layers actually prevent that failure, or just add friction + a checkbox? +- **D4 — Orchestration architecture rests on shifting harness behavior.** Steelman: the v2.4.0 "inline orchestration" migration was forced by a premise ("harness doesn't expose Agent to sub-agents") that turned out **false** (CC-01/S7). The architecture depends on harness behavior the plugin can't control; the dormant synthesis-agent + prose-vs-Workflow indecision are symptoms. +- **D5 — Maintainability / rot / solo-maintainer risk.** Steelman: 24 agents (one dormant), 7 commands, many docs, 697 tests — a large share pinning *prose* (doc-consistency). Is that testing value or testing wording? Is this past the complexity event horizon for one fork-and-own maintainer? +- **D6 — Claims vs. reality (honesty audit).** Steelman: README/CLAUDE.md make strong claims (context engineering, adversarial review, disciplined execution). Take the load-bearing ones and check whether the code delivers or whether it's aspirational prose. (Turn Voyage's own conformance discipline back on its self-description.) + +## Workflow shape (pipeline by default — refine the script on run) +```js +export const meta = { + name: 'voyage-devils-advocate', + description: "Adversarial cold audit of Voyage — steelman attacks + honest rebuttals", + phases: [ + { title: 'Attack', detail: 'one skeptic per dimension builds the strongest case against' }, + { title: 'Rebut', detail: 'steelman the defense; verdict STANDS / WEAKENED / REFUTED' }, + { title: 'Synthesize', detail: 'prioritize survivors + completeness critic' }, + ], +} +const DIMENSIONS = [ /* D1..D6, each {key, prompt} from the section above */ ] +const audited = await pipeline( + DIMENSIONS, + d => agent(`Devil's advocate on Voyage, dimension "${d.key}". ${d.prompt} Read the real code/docs, cite file:line, build the STRONGEST case AGAINST. No vibes.`, + { label:`attack:${d.key}`, phase:'Attack', schema: FINDING_SCHEMA, effort:'high' }), + (finding, d) => agent(`Steelman the DEFENSE against this critique, then judge each claim FATAL/MAJOR/MINOR and STANDS/WEAKENED/REFUTED under the best rebuttal: ${JSON.stringify(finding)}`, + { label:`rebut:${d.key}`, phase:'Rebut', schema: VERDICT_SCHEMA, effort:'high' }), +) +const synthesis = await agent(`Synthesize this adversarial audit into a prioritized verdict + a "what this audit might have missed" section. Findings+rebuttals: ${JSON.stringify(audited.filter(Boolean))}`, + { phase:'Synthesize', schema: SYNTHESIS_SCHEMA, effort:'high' }) +return synthesis +``` +- **FINDING_SCHEMA:** `{ dimension, findings: [{ claim, severity: FATAL|MAJOR|MINOR, evidence: [file:line], why_it_matters }] }` +- **VERDICT_SCHEMA:** `{ dimension, verdicts: [{ claim, strongest_rebuttal, verdict: STANDS|WEAKENED|REFUTED, residual_severity }] }` +- **SYNTHESIS_SCHEMA:** `{ survivors: [...ranked], refuted: [...], top_changes: [...], audit_blind_spots: [...] }` + +## Scale, models, cost +~6 attack + 6 rebut + 1 synthesis ≈ **13 agents**. `effort: high` on attack + synthesis (adversarial reasoning); Opus is the default (inherit session model) — pure-retrieval sub-steps may drop to `sonnet`. **Before launching, announce the plan + rough agent count** so the operator sees the scope (Workflows are token-heavy). + +## Success criteria (verifiseringsplikt) +1. `docs/devils-advocate-results.md` written, with **≥1 finding per dimension**, each carrying `file:line` evidence and an explicit **STANDS / WEAKENED / REFUTED** verdict after the rebuttal pass. +2. A synthesis that names which critiques are **REAL** (worth acting on, prioritized by severity × actionability) vs. **survived-as-fine**. +3. A **"what this audit might have missed"** section (completeness critic). +4. No finding asserted without evidence; rebuttals are genuine defenses, not strawmen. +5. The audit critiques *this very plan* too (the dimensions may be wrong/incomplete — say so). + +## Scope guard +This is an **audit that produces findings + recommendations**, NOT an implementation session. Do **not** start changing Voyage based on the results without a fresh operator go-ahead. Stop at the results doc + a short summary.