docs(voyage): S22 — happy-path dogfood results (blind spot #1/#4 measured)

Dogfooded /trekplan->/trekexecute on a real feature (voyage-doctor) against
a pre-registered scorecard. Q1: happy path produces a good, executable plan
but not self-sufficient (plan-critic C/71 vs self-score B+/88). Q4 DEMONSTRATED:
the adversarial review caught 3 real majors the planner+swarm missed, none in
the oracle — defects lived in plan->execute handoff fidelity. scope-guardian
ALIGNED. Caveats: n=1, oracle leaked into the swarm (pre-reg committed in the
explored repo), no cost measured.

Surfaced a MAJOR pipeline defect: /trekplan Phase 9 tells plan-critic +
scope-guardian to write JSON to /tmp for the dedup helper, but both agents
have only Read/Glob/Grep (no Write) -> the dedup step cannot run as documented.
Recorded as new backlog, not fixed (S22 scope was measurement).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB

docs/S22-happy-path-dogfood.md

@@ -73,4 +73,57 @@ Each risk is something a *flawed* plan could plausibly contain. Scoring records,
 
 ## RESULTS
 
-_(appended after the run — empty until `/trekplan` + `/trekexecute` complete)_
+**Run:** 2026-06-19, interactive main-context dogfood of `/trekplan --brief … → /trekexecute` on the `voyage-doctor` feature. Brief validated clean; plan written to `.claude/plans/trekplan-2026-06-19-voyage-doctor.md` (gitignored); execute ran in an isolated git worktree (`/private/tmp/claude-voyage-doctor-exec`) and was discarded — `main` HEAD unchanged at `aeee4c6`, working tree clean.
+
+**Pipeline actually exercised:** Phase 1 parse → 4b brief-reviewer (PROCEED) → Phase 5 swarm (7 agents: architecture-mapper, dependency-tracer, risk-assessor, task-finder, test-strategist, git-historian, convention-scanner; research-scout skipped — no external tech; ~345k subagent tokens) → Phase 7 synthesis → Phase 8 plan (passes `plan-validator --strict`) → Phase 9 plan-critic + scope-guardian → revise → execute (TDD) in worktree. **Not run** (bounded scope, recorded honestly, not silently skipped): the `effort: high` `gemini-bridge` 2nd-opinion pass; the full `/trekexecute` disciplined-executor harness (the plan was executed directly via TDD — the question is plan quality, not the executor's manifest ceremony).
+
+### ⚠️ Contamination caveat (load-bearing — read first)
+The pre-registration (`docs/S22-happy-path-dogfood.md`) was **committed into the repo before exploration**, so the Phase-5 swarm **read the answer key**: architecture-mapper, task-finder, and risk-assessor explicitly cite "the locked Success Criteria and real-risk oracle" and echo R1–R7. **The oracle leaked into the swarm.** This inflates any "plan handled the R-risks" claim. It does **not** weaken the Q4 finding — see below, the defects review caught were *outside* R1–R7. **Lesson:** when dogfooding a planning tool on its own repo, the scorecard must live *outside* the explored tree (or be committed after exploration). This is itself a finding about how to run this experiment.
+
+### Q1 — does the happy path produce a good plan? **Yes, but not a self-sufficient one.**
+- The plan was **executable and correct**: all 4 steps implementable; execute produced working code; **15/15** new tests pass; full suite **720 (718 pass / 2 skip / 0 fail)** = 705 baseline + 15, **zero regression**; CLI works (`project-doctor: PASS`, exit 0, valid `--json`).
+- The plan's code analysis was **unusually accurate** — plan-critic *independently verified* all 8 "load-bearing gotchas" against the source and rated every one TRUE (`combine()` first-parsed-only, `validateResearchDir` valid-on-empty, `discoverProject` un-try/catch'd, the trekreview exemption, etc.). The swarm surfaced real, code-grounded defects **beyond my oracle** (the `combine()` clobber, the crash vector, FM_MISSING guard) — a strong positive signal even net of contamination.
+- **But the plan was not executable as first written.** plan-critic scored it **C (71/100)** vs the planner's self-score **B+ (88)** — a ~17-pt self-inflation. The 3 majors (below) had to be fixed before a clean execute. **Verdict: the happy path produces a high-quality draft plan; the adversarial review is load-bearing, not decorative — without it the plan ships a real underspecification and a contradiction.**
+
+### Q4 — does the adversarial review catch real bugs? **Yes — decisively, and beyond the pre-registered target.**
+This is the audit's "never measured by anyone" question. Measured here: **plan-critic caught 3 real majors the planner (Opus 4.8 + a 7-agent swarm) genuinely missed**, none planted, none in the R1–R7 oracle:
+
+| Finding | Real? | In R1–R7 oracle? | Consequence if shipped |
+|---------|-------|------------------|------------------------|
+| **PC-1** `discoverProject` exposes `research` as a file-array, not a dir path; Step 1 said "validateResearchDir on the research/ dir" without deriving `join(dir,'research')` | ✅ verified | ❌ no | Executor blocked / invents the path — confirmed at execute (the path derivation was genuinely absent) |
+| **PC-2** the plan's own #1 *Critical* risk (`PROJECT_DIR_UNREADABLE` crash-isolation) was **untested** — a missing dir returns empty via `discoverProject`'s guard, never hitting the try/catch | ✅ verified | ❌ no (R7 covered "conditional branches", not this meta-gap) | The top risk ships unverified; a meta-catch the planner missed |
+| **PC-3** export-name contradiction: brief SC1 `doctorProject` vs plan/manifest `diagnoseProject` | ✅ verified | ❌ no | Executor following SC1 literally fails the manifest |
+
+plan-critic also did **not** trust the plan — it re-verified the code claims itself. scope-guardian returned **ALIGNED**: every SC covered, every Non-Goal (NG1–NG4) respected, every cited `file:line` confirmed exact; one borderline minor (progress/review beyond SC2). The two converged on that single minor.
+
+**Recall vs the pre-registered R1–R7 target:** the plan *handled* R1, R2, R4, R5, R6 correctly (composed not reimplemented; correct research truth-table incl. trekreview + topics>0 guard; presence-gated missing-vs-invalid; anchored slug strip; read-only). R3/R7 were *partially* tripped — the crash-isolation risk was handled **in code** but **under-tested**, and plan-critic **caught exactly that** (PC-2). So review recall on the one tripped pre-registered sub-risk = **1/1**. **The more important result:** the oracle *under-predicted where the defects would be.* The real majors lived in **plan→execute handoff fidelity** (an unspecified path, a name contradiction, an untested top-risk), not in the algorithmic risks I anticipated. Adversarial review earned its keep precisely on the class my foresight (and the swarm's) missed.
+
+### Execute outcome + an execute-phase finding
+Execute succeeded (15/15, suite green, CLI works). **One finding only execute could surface:** the revised plan's PC-2 fix prescribed `t.mock.method` to force `discoverProject` to throw — but it's an **ESM named import (read-only namespace binding)**, so `t.mock.method` can't redefine it. The executor had to substitute **dependency injection** (`opts.discover`). So even the *post-review* plan carried a residual gap that only contact with the runtime exposed — a reminder that plan review is not a substitute for execution.
+
+### Pipeline defects the dogfood surfaced (the bonus the S14 audit could not get — it never ran the pipeline)
+1. **`/trekplan` Phase 9 is broken as documented.** It instructs plan-critic + scope-guardian to "Write structured JSON output to `/tmp/…out.json`", then runs `plan-review-dedup.mjs` on those files. **Both agents' frontmatter grants only `Read/Glob/Grep` — no `Write`/`Bash`** — so the files are never created and the dedup step cannot run. Both agents fell back to returning JSON inline. **Severity: MAJOR** (a documented, wired step that cannot execute). Fix options: grant the reviewers `Write`, or have the orchestrator persist the returned JSON before calling the dedup helper.
+2. **Oracle-into-swarm contamination** (see caveat) — a real trap for dogfooding planning tools on their own repo.
+3. **`plan_version` not parsed.** The plan template emits `plan_version: 1.7` as prose in the "Generated by" line; `plan-validator` then warns `PLAN_NO_VERSION`. Minor template/validator mismatch — the validator looks for a frontmatter/parseable field the template doesn't emit.
+4. **Version skew.** The *installed* plugin (skill the operator invokes) is cached at **v5.1.1**; the repo under development is **v5.5.0**. The dogfood ran the v5.1.1 command text against v5.5.0 `lib/`. Harmless here (the phases are stable across the bump) but worth noting: operators dogfooding the installed plugin are not testing the dev tree.
+
+### Honest limitations
+- **Contamination** caps confidence in the "plan handled R1–R7" half of Q1 (the swarm saw R1–R7). The Q4 half is robust *because* the caught defects were outside the oracle.
+- **n = 1, one small feature, one domain** (an internal validator the planner knew well). Generalization to larger/unfamiliar features is unproven. A feature in an unfamiliar codebase would stress the swarm more and likely lower plan quality.
+- **No token/$ measurement** (audit Blind spot #3 remains open): ~345k Phase-5 + ~110k Phase-9 + ~26k brief-review subagent tokens observed, but not normalized to a per-run cost. Recorded, not analyzed.
+- **The `voyage-doctor` implementation worked** (15 passing tests, full suite green) and is genuinely useful, but was **discarded per operator scope** (deliverable = measurement). The plan persists locally at `.claude/plans/trekplan-2026-06-19-voyage-doctor.md`; productizing it is a future-session candidate, not part of S22.
+
+### Bottom line
+The happy path **works** and produces high-quality, executable plans — and the adversarial review is **load-bearing**: it caught 3 real majors the full planning swarm missed, the highest-value being defects in plan→execute handoff fidelity that neither the planner nor the pre-registered oracle anticipated. The flagship "context-engineering via specialized agents + adversarial review" claim is, for this one case, **demonstrated rather than asserted** — with the honest caveats that it is n=1, the oracle leaked into the swarm, no cost was measured, and the pipeline itself shipped a broken Phase-9 dedup step (defect #1) that this very run exposed.
+
+### Verification log (Verifiseringsplikt)
+| Claim | How verified |
+|-------|--------------|
+| Brief validates clean | `node lib/validators/brief-validator.mjs --json` → `{valid:true,errors:[],warnings:[]}` |
+| Plan passes schema | `node lib/validators/plan-validator.mjs --strict --json` → `valid:true, 4 steps, 0 errors` (1 soft `PLAN_NO_VERSION` warning = defect #3) |
+| 15 new tests pass; suite green | `node --test` in worktree → new file 15/15; full suite 720 (718/2/0) = 705 baseline +15 |
+| CLI works on live dir | `node lib/validators/project-doctor.mjs .claude/projects/2026-06-19-voyage-doctor` → `PASS`, exit 0; `--json` → valid JSON |
+| Execute isolated; main untouched | `git worktree remove` → `git status` clean, HEAD `aeee4c6`, `lib/validators/project-doctor.mjs` absent from main |
+| plan-critic's 3 majors are real | Each re-checked against source: PC-1 (`project-discovery.mjs` `research:string[]`, no dir field), PC-2 (`project-discovery.mjs:39` empty-guard precedes any throw), PC-3 (brief SC1 vs plan/manifest names) — all confirmed |
+| Reviewers lack Write (defect #1) | Agent registry: `voyage:plan-critic` + `voyage:scope-guardian` Tools = `Read, Glob, Grep`; both reported the write failure at runtime |
+| Contamination | architecture-mapper/task-finder/risk-assessor outputs explicitly cite `docs/S22-happy-path-dogfood.md` and R1–R7 |

docs/devils-advocate-results.md

@@ -159,3 +159,20 @@ Blind spot #2 ("no security / safety dimension") was the audit's self-named high
 **Code fix (finding #1).** `lib/exporters/endpoint-validator.mjs`: added `mappedV4(host)` — after IPv6 bracket-strip, an IPv4-mapped literal (`::ffff:a.b.c.d` dotted **or** `::ffff:HHHH:HHHH` hex-pair) is decoded to its embedded dotted-decimal IPv4, and every guard (loopback / RFC-1918 / link-local / `HARD_BLOCKED`) classifies on **that**. TDD: 4 new tests in `tests/hooks/otel-export-validators.test.mjs` (failing-first) — mapped loopback → `LOOPBACK_REJECTED`, mapped RFC-1918 → `RFC1918_REJECTED`, mapped metadata → `HARD_BLOCKED` (even with `VOYAGE_OTEL_ALLOW_PRIVATE=1`), and mapped **public** `::ffff:8.8.8.8` still valid over https (no over-block). Threat model remains narrow (endpoint is operator-set via `VOYAGE_OTEL_ENDPOINT`; export is opt-in via `VOYAGE_EXPORT_MODE`, default `off`; a brief cannot set env) — the fix closes a control under-delivering its documented contract, not a brief-reachable hole.
 
 **Forward-pointer (FLAGGED, NOT done — adjacent doc/code defect found while auditing the autonomy surface; this is S21b, not S21 security-core):** `operations.md:15` claims `autonomy-gate.mjs` runs the state machine `idle → approved → executing → merge-pending → main-merged`; the actual code states are `idle → gates_on/auto_running → paused_for_gate → completed`, and the `--gates {open|closed|adaptive}` table doesn't match the boolean (`--gates true|false`) the CLI shim and command docs use. Recommend a truth-pass + anti-false-claim pin in a follow-up session.
+
+---
+
+## S22 resolution — Blind spot #1 + #4 (happy-path plan quality + review efficacy) (2026-06-19, operator-gated)
+
+Blind spots #1 ("the audit never tested whether the happy path produces *good plans*") and #4 ("plan quality — does the adversarial review catch real bugs? — never measured by anyone") were the audit's self-named highest-value open questions. S22 measured them by **dogfooding `/trekplan → /trekexecute`** on a real, small Voyage feature (`voyage-doctor`, a project-coherence validator) against a **pre-registered ground-truth scorecard committed before the run**. Full record + verification log: **`docs/S22-happy-path-dogfood.md`**.
+
+**Verdicts.** Q1 (plan quality): the happy path produces a high-quality, executable plan — execute succeeded (15/15 new tests, full suite green, CLI works) — **but not a self-sufficient one**: plan-critic scored it C (71) vs the planner's self-score B+ (88), and 3 majors had to be fixed before a clean execute. Q4 (review efficacy): **demonstrated — the adversarial review caught 3 real majors the Opus-4.8 planner + 7-agent swarm genuinely missed** (an underspecified `research/`-dir path, an untested top-risk, a name contradiction), none planted, **none in the pre-registered oracle** — the defects lived in plan→execute handoff fidelity, which neither planner nor pre-registrar anticipated. scope-guardian returned ALIGNED (all SC/NG verified).
+
+**Honest caveats (do not over-claim):** (1) **n = 1**, one small feature in a domain the planner knew well; (2) the pre-registration was committed *into the explored repo*, so the Phase-5 swarm **read the oracle** (R1–R7 leaked) — robustly caveated, and the Q4 finding survives because the caught defects were *outside* R1–R7; (3) **no token/$ measurement** (Blind spot #3 still open); (4) the `voyage-doctor` code worked but was **discarded per scope** (deliverable = measurement).
+
+**Pipeline defects the dogfood surfaced (NEW — the S14 audit never ran the pipeline):**
+1. **MAJOR — `/trekplan` Phase 9 dedup is broken as documented.** plan-critic + scope-guardian are told to "write JSON to `/tmp/…out.json`" for `plan-review-dedup.mjs`, but both agents' frontmatter grants only `Read/Glob/Grep` — no `Write` — so the files are never created and the dedup step cannot run. Fix: grant the reviewers `Write`, or have the orchestrator persist their returned JSON before the dedup call.
+2. MINOR — plan template emits `plan_version` as prose; `plan-validator` warns `PLAN_NO_VERSION` (template/validator mismatch).
+3. NOTE — installed-plugin/repo version skew (cache v5.1.1 vs repo v5.5.0); dogfooding the installed plugin doesn't test the dev tree.
+
+These are **recorded, not fixed** (S22 scope was measurement; each is its own task under the one-task-per-session run-mode). Decision-matrix-style dispositions are **not** test-pinned.