open/voyage
1
0
Fork 0
Code Issues Pull requests Releases Activity

docs(voyage): S22 — happy-path dogfood results (blind spot #1/#4 measured)

Browse source 
Dogfooded /trekplan->/trekexecute on a real feature (voyage-doctor) against
a pre-registered scorecard. Q1: happy path produces a good, executable plan
but not self-sufficient (plan-critic C/71 vs self-score B+/88). Q4 DEMONSTRATED:
the adversarial review caught 3 real majors the planner+swarm missed, none in
the oracle — defects lived in plan->execute handoff fidelity. scope-guardian
ALIGNED. Caveats: n=1, oracle leaked into the swarm (pre-reg committed in the
explored repo), no cost measured.

Surfaced a MAJOR pipeline defect: /trekplan Phase 9 tells plan-critic +
scope-guardian to write JSON to /tmp for the dedup helper, but both agents
have only Read/Glob/Grep (no Write) -> the dedup step cannot run as documented.
Recorded as new backlog, not fixed (S22 scope was measurement).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
Kjell Tore Guttormsen 2026-06-19 20:53:21 +02:00
commit a366e332b7
2 changed files with 71 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

17
docs/devils-advocate-results.md
View file

@ -159,3 +159,20 @@ Blind spot #2 ("no security / safety dimension") was the audit's self-named high
**Code fix (finding #1).** `lib/exporters/endpoint-validator.mjs`: added `mappedV4(host)` — after IPv6 bracket-strip, an IPv4-mapped literal (`::ffff:a.b.c.d` dotted **or** `::ffff:HHHH:HHHH` hex-pair) is decoded to its embedded dotted-decimal IPv4, and every guard (loopback / RFC-1918 / link-local / `HARD_BLOCKED`) classifies on **that**. TDD: 4 new tests in `tests/hooks/otel-export-validators.test.mjs` (failing-first) — mapped loopback → `LOOPBACK_REJECTED`, mapped RFC-1918 → `RFC1918_REJECTED`, mapped metadata → `HARD_BLOCKED` (even with `VOYAGE_OTEL_ALLOW_PRIVATE=1`), and mapped **public** `::ffff:8.8.8.8` still valid over https (no over-block). Threat model remains narrow (endpoint is operator-set via `VOYAGE_OTEL_ENDPOINT`; export is opt-in via `VOYAGE_EXPORT_MODE`, default `off`; a brief cannot set env) — the fix closes a control under-delivering its documented contract, not a brief-reachable hole.
**Forward-pointer (FLAGGED, NOT done — adjacent doc/code defect found while auditing the autonomy surface; this is S21b, not S21 security-core):** `operations.md:15` claims `autonomy-gate.mjs` runs the state machine `idle → approved → executing → merge-pending → main-merged`; the actual code states are `idle → gates_on/auto_running → paused_for_gate → completed`, and the `--gates {open|closed|adaptive}` table doesn't match the boolean (`--gates true|false`) the CLI shim and command docs use. Recommend a truth-pass + anti-false-claim pin in a follow-up session.
---
## S22 resolution — Blind spot #1 + #4 (happy-path plan quality + review efficacy) (2026-06-19, operator-gated)
Blind spots #1 ("the audit never tested whether the happy path produces *good plans*") and #4 ("plan quality — does the adversarial review catch real bugs? — never measured by anyone") were the audit's self-named highest-value open questions. S22 measured them by **dogfooding `/trekplan → /trekexecute`** on a real, small Voyage feature (`voyage-doctor`, a project-coherence validator) against a **pre-registered ground-truth scorecard committed before the run**. Full record + verification log: **`docs/S22-happy-path-dogfood.md`**.
**Verdicts.** Q1 (plan quality): the happy path produces a high-quality, executable plan — execute succeeded (15/15 new tests, full suite green, CLI works) — **but not a self-sufficient one**: plan-critic scored it C (71) vs the planner's self-score B+ (88), and 3 majors had to be fixed before a clean execute. Q4 (review efficacy): **demonstrated — the adversarial review caught 3 real majors the Opus-4.8 planner + 7-agent swarm genuinely missed** (an underspecified `research/`-dir path, an untested top-risk, a name contradiction), none planted, **none in the pre-registered oracle** — the defects lived in plan→execute handoff fidelity, which neither planner nor pre-registrar anticipated. scope-guardian returned ALIGNED (all SC/NG verified).
**Honest caveats (do not over-claim):** (1) **n = 1**, one small feature in a domain the planner knew well; (2) the pre-registration was committed *into the explored repo*, so the Phase-5 swarm **read the oracle** (R1–R7 leaked) — robustly caveated, and the Q4 finding survives because the caught defects were *outside* R1–R7; (3) **no token/$ measurement** (Blind spot #3 still open); (4) the `voyage-doctor` code worked but was **discarded per scope** (deliverable = measurement).
**Pipeline defects the dogfood surfaced (NEW — the S14 audit never ran the pipeline):**
1. **MAJOR — `/trekplan` Phase 9 dedup is broken as documented.** plan-critic + scope-guardian are told to "write JSON to `/tmp/…out.json`" for `plan-review-dedup.mjs`, but both agents' frontmatter grants only `Read/Glob/Grep` — no `Write` — so the files are never created and the dedup step cannot run. Fix: grant the reviewers `Write`, or have the orchestrator persist their returned JSON before the dedup call.
2. MINOR — plan template emits `plan_version` as prose; `plan-validator` warns `PLAN_NO_VERSION` (template/validator mismatch).
3. NOTE — installed-plugin/repo version skew (cache v5.1.1 vs repo v5.5.0); dogfooding the installed plugin doesn't test the dev tree.
These are **recorded, not fixed** (S22 scope was measurement; each is its own task under the one-task-per-session run-mode). Decision-matrix-style dispositions are **not** test-pinned.