open/voyage
1
0
Fork 0
Code Issues Pull requests Releases Activity

fix(voyage): S17 — downgrade NW2 bake-off verdict + drop orchestrator Agent-grants

Browse source 
Survivor #3 (bake-off data un-archived): per-run JSON a1..b3.json was never
committed (git diff-filter=A empty), so the medians/jaccard ladder cannot be
re-derived or audited. Commit-path would require fabricated data (forbidden) →
downgraded "POSITIVE" to "opt-in-defensible (single un-archived run)" in
docs/T2-bakeoff-results.md (verdict line + §5 header + new §Reproducibility
caveat) and the v5.5.0 CHANGELOG entry. New doc-consistency pin fails if the
doc re-asserts bare POSITIVE or drops the un-archived disclosure.

Survivor #5 (latent dispatch risk): planning/research/review orchestrators are
inline reference docs no command invokes, yet shipped tools:["Agent",...]. The
harness cannot spawn sub-agents with Agent, so the grant was pure latent risk.
Dropped "Agent" from all three frontmatters; flipped the agent-frontmatter
invariant (its canonical home) from "must include Agent" to "must NOT include
Agent". Removed the duplicate orchestrator pin from doc-consistency to avoid a
double guard (S19 anti-bloat).

Tests 690 → 691 (689 pass / 2 skip / 0 fail). claude plugin validate passes
(1 accepted CLAUDE.md-at-root warning).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
Kjell Tore Guttormsen 2026-06-19 06:24:27 +02:00
commit eb5a7324a4
7 changed files with 48 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

19
docs/T2-bakeoff-results.md
View file

@ -2,8 +2,9 @@
**Status (S10):** Part A (build + smoke) **and** part B (full ≥3-runs/arm
measurement) **COMPLETE**. Operator GO 2026-06-18 (choice "a" — full ≥3× run).
**T2 §5 verdict: POSITIVE** — the Workflow substrate is fidelity-neutral; proceed to
the opt-in `--workflow` flag at S11. Full evidence in **§Full run (S10 part B)** at the
**T2 §5 verdict: opt-in-defensible** (POSITIVE on the measured axes, on a single
un-archived run — see **§Reproducibility caveat**) — the Workflow substrate is
fidelity-neutral on what was measured; proceed to the opt-in `--workflow` flag at S11. Full evidence in **§Full run (S10 part B)** at the
end of this doc. The smoke section below is retained for history.
Resolves: `docs/W1-narrow-wins-plan.md §S10` (build + de-risk **and** full measurement).
@ -160,7 +161,7 @@ Arm A median ~150s/run; Arm B ~230s ⇒ **+54%**. **Caveat:** not a controlled p
Arm B runs in the background; intermediate findings visible in the workflow transcript + `/workflows`; returns structured `{verdict, findings}`. Phase 7 rendering stays shared/prose; the operator-gate (review.md write) is unaffected. Arm B **frees the main context** during the review (the 3 workflows ran while main did other work) — Arm A occupies it end-to-end.
## VERDICT: **POSITIVE**
## VERDICT: **opt-in-defensible** (POSITIVE on measured axes; single un-archived run)
The Workflow substrate (Arm B) is **fidelity-equivalent** to the prose path (Arm A) on
the operator-meaningful axes — verdict 1.0, file-set 1.0, issue-coverage 100%,
@ -177,6 +178,18 @@ It is **not** a regression: the granularity ladder and within-vs-cross compariso
the divergence as intrinsic LLM nondeterminism, equal in both arms. Arm B's wall-time is
~+54% but non-blocking.
## Reproducibility caveat (Survivor #3, added S17)
The per-run arm outputs (`a1.json`..`b3.json`) were **never committed** — `git log
--diff-filter=A` finds no such file in any ref, and `scripts/bakeoff-fidelity.mjs`
requires them as input. The medians and the jaccard ladder above therefore **cannot be
regenerated, audited, or falsified** from the repo. This is why the verdict is labeled
**opt-in-defensible** rather than a clean POSITIVE: it justifies shipping `--workflow`
behind a flag (the measured axes did come out fidelity-neutral), but it rests on a
**single un-archived run** and is not a reproducible result. To upgrade it, a future
session must re-run the bake-off and commit the raw per-run JSON under
`tests/fixtures/bakeoff-rich/runs/` with a test that re-derives these numbers.
**→ S11:** proceed with the **opt-in `--workflow` flag** (prose stays default — preserves
the 2.1.154+ portability floor; see §Open decisions in the narrow-wins plan). The bake-off
supports making the Workflow path reachable; residuals are (a) the F4 `auto`/`bypass`