docs(voyage): plan S14 devil's-advocate audit via Dynamic Workflow
Cold adversarial audit of Voyage to run in the next session (post-/clear) as a Dynamic Workflow per operator request. 6 adversarial dimensions (ceremony-vs-value, today's-decisions-as-rationalizations, brief-contract fragility, orchestration on shifting harness behavior, maintainability/rot, claims-vs-reality) → rebuttal pass (STANDS/WEAKENED/REFUTED) → synthesis. Deliverable: docs/devils-advocate-results.md. Planning only — workflow not run this session. STATE.md (gitignored) points the cold-start session at docs/devils-advocate-plan.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
parent
056720e7f3
commit
a9927ef5af
1 changed files with 66 additions and 0 deletions
66
docs/devils-advocate-plan.md
Normal file
66
docs/devils-advocate-plan.md
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
# Devil's-advocate session — adversarial cold audit of Voyage (Dynamic Workflow)
|
||||
|
||||
**Status:** Planned 2026-06-18 (post-S13). **Executes in the NEXT session** (after the operator's `/clear`).
|
||||
**Operator mandate (2026-06-18, verbatim):** «Planlegg en "devils advocate" sesjon … hvor du nå oppdaterer STATE.md og forklarer at det skal brukes en Dynamic Workflow for analysen. … denne pluginen er veldig viktig for meg.»
|
||||
**Opt-in:** The operator explicitly requested a **Dynamic Workflow** for the analysis — the next session is authorized to call the `Workflow` tool (multi-agent orchestration). This is a durable opt-in for that session.
|
||||
|
||||
## Why this, why now
|
||||
v5.5.0 just shipped. Today's reflection surfaced an uncomfortable fact: **Voyage was not used to build Voyage** — the whole upgrade arc was hand-orchestrated with bespoke planning docs. Before investing further, subject the plugin to a **cold, hostile audit that actively tries to prove it is over-engineered, wrong, or unmaintainable** — and see what survives. The operator cares about this plugin; the kindest thing is an honest adversary, not applause.
|
||||
|
||||
## Objective (what the session must produce)
|
||||
A severity-tagged, **adversarially-verified** critique written to **`docs/devils-advocate-results.md`**: where Voyage is genuinely weak, where its claims outrun its code, and what (if anything) should change. This is a steelman attack **plus** an honest rebuttal pass — not a feature wishlist, not vibes. Every finding cites `file:line`.
|
||||
|
||||
## Why a Dynamic Workflow (not inline, not the Voyage pipeline)
|
||||
- The audit is a **fan-out → verify → synthesize** shape across independent adversarial angles — exactly the Workflow tool's review/red-team pattern. Independent skeptics that can't see each other's reasoning catch more than one context reasoning sequentially, and the rebuttal pass kills plausible-but-unfair critiques.
|
||||
- It is deliberately **not** run through Voyage's own `/trek*` pipeline: the point is to judge Voyage from the *outside*, cold — not to dogfood it (that is a separate exercise the prior session flagged).
|
||||
- The operator asked for a Dynamic Workflow explicitly.
|
||||
|
||||
## Adversarial dimensions — one finder each, build the STRONGEST case AGAINST
|
||||
Each finder reads the actual code/docs (cite `file:line`); no claim without evidence.
|
||||
|
||||
- **D1 — Ceremony vs. value (existential).** Steelman: "Voyage is a sophistication trap — Plan mode + `/code-review` + a TODO gets 90% of the outcome at 10% of the cognitive/token cost. 7 commands, 24 agents, 697 tests are overhead, not value." Exhibit A: today's non-dogfood.
|
||||
- **D2 — Today's decisions were rationalizations, not conclusions.** Steelman each: CC-26/27 "narrow hybrid not wholesale" (genuine analysis or status-quo bias? the bake-offs were largely *designed-not-run* or deterministic estimates); NW2 "POSITIVE" at **+4.4% tokens** (is that positive? was fidelity real?); NW3 **Δ=0%** decline (the measurement was `chars/4` with *swept, not API-measured* BASE — is the decline robust or an unfalsifiable-estimate artifact?); v5.5.0 versioning (skipping 5.2–5.4 as "unreleased internal milestones" — coherent for consumers, or post-hoc tidy-up?).
|
||||
- **D3 — Brief-as-PUBLIC-CONTRACT is fragile.** Steelman: the framing-enforcement invariant ("brief framing must match operator intent") is itself an admission that the pipeline *structurally polishes wrong premises*. Do the three 2.2 defensive layers actually prevent that failure, or just add friction + a checkbox?
|
||||
- **D4 — Orchestration architecture rests on shifting harness behavior.** Steelman: the v2.4.0 "inline orchestration" migration was forced by a premise ("harness doesn't expose Agent to sub-agents") that turned out **false** (CC-01/S7). The architecture depends on harness behavior the plugin can't control; the dormant synthesis-agent + prose-vs-Workflow indecision are symptoms.
|
||||
- **D5 — Maintainability / rot / solo-maintainer risk.** Steelman: 24 agents (one dormant), 7 commands, many docs, 697 tests — a large share pinning *prose* (doc-consistency). Is that testing value or testing wording? Is this past the complexity event horizon for one fork-and-own maintainer?
|
||||
- **D6 — Claims vs. reality (honesty audit).** Steelman: README/CLAUDE.md make strong claims (context engineering, adversarial review, disciplined execution). Take the load-bearing ones and check whether the code delivers or whether it's aspirational prose. (Turn Voyage's own conformance discipline back on its self-description.)
|
||||
|
||||
## Workflow shape (pipeline by default — refine the script on run)
|
||||
```js
|
||||
export const meta = {
|
||||
name: 'voyage-devils-advocate',
|
||||
description: "Adversarial cold audit of Voyage — steelman attacks + honest rebuttals",
|
||||
phases: [
|
||||
{ title: 'Attack', detail: 'one skeptic per dimension builds the strongest case against' },
|
||||
{ title: 'Rebut', detail: 'steelman the defense; verdict STANDS / WEAKENED / REFUTED' },
|
||||
{ title: 'Synthesize', detail: 'prioritize survivors + completeness critic' },
|
||||
],
|
||||
}
|
||||
const DIMENSIONS = [ /* D1..D6, each {key, prompt} from the section above */ ]
|
||||
const audited = await pipeline(
|
||||
DIMENSIONS,
|
||||
d => agent(`Devil's advocate on Voyage, dimension "${d.key}". ${d.prompt} Read the real code/docs, cite file:line, build the STRONGEST case AGAINST. No vibes.`,
|
||||
{ label:`attack:${d.key}`, phase:'Attack', schema: FINDING_SCHEMA, effort:'high' }),
|
||||
(finding, d) => agent(`Steelman the DEFENSE against this critique, then judge each claim FATAL/MAJOR/MINOR and STANDS/WEAKENED/REFUTED under the best rebuttal: ${JSON.stringify(finding)}`,
|
||||
{ label:`rebut:${d.key}`, phase:'Rebut', schema: VERDICT_SCHEMA, effort:'high' }),
|
||||
)
|
||||
const synthesis = await agent(`Synthesize this adversarial audit into a prioritized verdict + a "what this audit might have missed" section. Findings+rebuttals: ${JSON.stringify(audited.filter(Boolean))}`,
|
||||
{ phase:'Synthesize', schema: SYNTHESIS_SCHEMA, effort:'high' })
|
||||
return synthesis
|
||||
```
|
||||
- **FINDING_SCHEMA:** `{ dimension, findings: [{ claim, severity: FATAL|MAJOR|MINOR, evidence: [file:line], why_it_matters }] }`
|
||||
- **VERDICT_SCHEMA:** `{ dimension, verdicts: [{ claim, strongest_rebuttal, verdict: STANDS|WEAKENED|REFUTED, residual_severity }] }`
|
||||
- **SYNTHESIS_SCHEMA:** `{ survivors: [...ranked], refuted: [...], top_changes: [...], audit_blind_spots: [...] }`
|
||||
|
||||
## Scale, models, cost
|
||||
~6 attack + 6 rebut + 1 synthesis ≈ **13 agents**. `effort: high` on attack + synthesis (adversarial reasoning); Opus is the default (inherit session model) — pure-retrieval sub-steps may drop to `sonnet`. **Before launching, announce the plan + rough agent count** so the operator sees the scope (Workflows are token-heavy).
|
||||
|
||||
## Success criteria (verifiseringsplikt)
|
||||
1. `docs/devils-advocate-results.md` written, with **≥1 finding per dimension**, each carrying `file:line` evidence and an explicit **STANDS / WEAKENED / REFUTED** verdict after the rebuttal pass.
|
||||
2. A synthesis that names which critiques are **REAL** (worth acting on, prioritized by severity × actionability) vs. **survived-as-fine**.
|
||||
3. A **"what this audit might have missed"** section (completeness critic).
|
||||
4. No finding asserted without evidence; rebuttals are genuine defenses, not strawmen.
|
||||
5. The audit critiques *this very plan* too (the dimensions may be wrong/incomplete — say so).
|
||||
|
||||
## Scope guard
|
||||
This is an **audit that produces findings + recommendations**, NOT an implementation session. Do **not** start changing Voyage based on the results without a fresh operator go-ahead. Stop at the results doc + a short summary.
|
||||
Loading…
Add table
Add a link
Reference in a new issue