voyage/docs/T2-bakeoff-results.md

# T2 / NW2 — prose-vs-Workflow bake-off results

**Status (S10):** Part A (build + smoke) **and** part B (full ≥3-runs/arm
measurement) **COMPLETE**. Operator GO 2026-06-18 (choice "a" — full ≥3× run).
**T2 §5 verdict: opt-in-defensible** (POSITIVE on the measured axes, on a single
un-archived run — see **§Reproducibility caveat**) — the Workflow substrate is
fidelity-neutral on what was measured; proceed to the opt-in `--workflow` flag at S11. Full evidence in **§Full run (S10 part B)** at the
end of this doc. The smoke section below is retained for history.

Resolves: `docs/W1-narrow-wins-plan.md §S10` (build + de-risk **and** full measurement).

---

## Setup

| Item | Value |
|------|-------|
| Fixture | `tests/fixtures/bakeoff/` — real diff of commit `b149538` (NW1) + brief reconstructed from `docs/W1-narrow-wins-plan.md §S9` |
| Delivered diff | 3 files, +438/-3 (`commands/trekreview.md`, `lib/review/findings-schema.mjs`, `tests/lib/findings-schema.test.mjs`) |
| Triage | all 3 files `summary-only` (deterministic Phase-4 classifier) |
| Model | `opus` (reviewer + coordinator agent defaults) |
| Arm A (prose) | reviewers spawned via Agent tool, prose trailing-JSON contract (validated by NW1 `findings-schema`) |
| Arm B (Workflow) | `scripts/trekreview-armB.workflow.mjs` via Workflow tool: `parallel([conformance, correctness])` schema-forced → JS dedup-by-triplet → `agent(review-coordinator)` verdict schema |
| Fidelity metric | `lib/review/fidelity-diff.mjs` `fidelityDiffStructured` (verdict + jaccard over `(file,line,rule_key)`-IDs + severity/rule cross-check) |

## Smoke results (single run per arm)

| | Arm A (prose) | Arm B (Workflow) |
|---|---|---|
| Reviewers run | 2 | 2 |
| Raw findings | 0 | 1 |
| After coordinator | 0 (coordinator moot — 0 findings) | 0 (coordinator dropped the 1 finding) |
| **Verdict** | **ALLOW** | **ALLOW** |
| Agents | 2 | 3 (2 reviewers + coordinator) |
| Subagent tokens | ~72.3k (34.3k + 38.0k; no coordinator) | ~100.9k (incl. coordinator) |
| Wall-time | ~70 s (parallel reviewers) | ~157 s (full pipeline) |

### PRIMARY metric — output fidelity: **EQUIVALENT**

```
fidelityDiffStructured(ArmA, ArmB) =
  { verdictMatch: true, jaccard: 1, countA: 0, countB: 0,
    severityMismatches: [], ruleKeyMismatches: [], equivalent: true }
```

⚠ **Caveat — thin finding surface.** Both arms returned **0 final findings** on
this clean, TDD'd fixture, so fidelity is confirmed only at the **verdict** level
(ALLOW ≡ ALLOW); the finding-*set* fidelity is trivially equal at zero and was
**not stressed**. A reviewer-level divergence *did* appear (Arm B raised 1 raw
finding, its coordinator filtered it; Arm A raised 0) — masked at the verdict
level. Quantifying that divergence is exactly what the full run on a
richer-finding-surface fixture must do.

### Secondary metrics (smoke, single-run — not medians)

- **JSON-robustness (the F2 win):** Arm B's reviewers were **schema-forced**
  (StructuredOutput) — typed findings, zero `JSON.parse`; the 1 raw finding +
  the coordinator verdict both conformed with no re-ask. Arm A's trailing-JSON
  validated clean via NW1 `findings-schema`. Win demonstrated structurally; the
  parse-error/re-ask delta needs a fixture that actually provokes malformed JSON.
- **Classifier interference: 0.** Arm B's 2-agent fan-out + coordinator (3 agents)
  ran with no denied/missing spawns. Confirms S8 F4 for trekreview's small
  fan-out under the default mode. (`auto`/`bypass` still to be checked in the full run.)
- **Token cost:** preliminary and **not yet comparable** — Arm B ran a coordinator
  (on its 1 finding) that Arm A did not. Single run; no medians.
- **Control/visibility:** Arm B runs in the background; intermediate findings are
  visible in the workflow transcript + `/workflows`. Operator-gate (the review.md
  write) is unaffected — both arms return structured `{verdict, findings}` and
  Phase 7 rendering stays shared/prose.

## Smoke verdict

**SMOKE PASS — machinery validated.** Arm B (Workflow substrate) runs the full
Phase 5–6 pipeline end-to-end, fidelity-**equivalent** to Arm A at the verdict
level, with **zero classifier interference**. The build is sound: NW1 schema,
fidelity-diff, fixture, and the Arm B port all work together.

This is **not** the T2 §5 POSITIVE/NEGATIVE verdict — that needs the full
≥3-runs/arm measurement with a finding-rich fixture.

## Go / no-go recommendation (operator decides)

**Recommend: proceed to the full ≥3-runs/arm run (S10 part B)** with two changes:
1. **Use a richer-finding-surface fixture** (a larger real voyage commit, or seed
   the fixture with a few genuine issues) so finding-*set* fidelity is actually
   stressed — the smoke only proved verdict fidelity at 0 findings.
2. **Match the arms' coordinator path** (run Arm A's coordinator too, even at low
   finding counts) so the token/wall-time comparison is apples-to-apples, and
   add the `auto`/`bypass` classifier-interference check (F4).

If the operator prefers, S11 can instead record "port built + smoke-validated;
full measurement deferred" and integrate behind the opt-in `--workflow` flag on
the smoke evidence alone — weaker, but the substrate is demonstrably functional.

---

# Full run (S10 part B) — ≥3 runs/arm, rich-finding fixture

**This is the T2 §5 verdict.** Operator GO 2026-06-18 (choice "a"). Resolves the
measurement half of `docs/W1-narrow-wins-plan.md §S10`.

## Setup (vs smoke)

| Item | Value |
|------|-------|
| Fixture | `tests/fixtures/bakeoff-rich/` — JWT-auth brief + diff, **5 seeded blatant issues** (varied severity/rule_key, one dual-flaggable). Live reviewers surface **11–18 findings/run** — the smoke's 0-finding limitation is resolved. |
| Triage | all 3 files `deep-review` (auth/security surface) — pinned, passed to both arms |
| Both arms run the coordinator | yes (token now comparable; smoke ran A's coordinator only on its 1 finding) |
| Arm A (prose) | foreground reviewers (Agent tool, no `name`) → `validateReviewerOutput` (NW1) → `scripts/bakeoff-armA-merge.mjs` triplet-dedup → foreground `review-coordinator`. Runs ×3. |
| Arm B (Workflow) | `scripts/trekreview-armB.workflow.mjs` ×3 (StructuredOutput findings → JS triplet-dedup → coordinator verdict schema) |
| Analysis | `scripts/bakeoff-fidelity.mjs` — cross-arm + within-arm + granularity ladder |

## Raw results

| Run | Arm A (prose) | Arm B (Workflow) |
|-----|---------------|------------------|
| 1 | BLOCK · 13 findings · 86.3k tok · ~119s | BLOCK · 11 findings · 96.9k tok · ~230s |
| 2 | BLOCK · 12 findings · 92.8k tok · ~150s | BLOCK · 14 findings · 91.9k tok · ~201s |
| 3 | BLOCK · 16 findings · 95.8k tok · ~188s | BLOCK · 18 findings · 97.2k tok · ~249s |

All 6 runs → **BLOCK**. (Arm A tokens = 2 reviewers + coordinator subagent-tokens; wall = reviewer-phase max + coordinator.)

## Metrics vs T2 §5

### 1. PRIMARY — output fidelity

- **Verdict: EQUIVALENT.** Verdict-match rate **1.0** — all 9 cross-arm (Aᵢ×Bⱼ) pairs agree, all 6 runs BLOCK. This is the operator-meaningful gate decision.
- **Finding-set — granularity ladder (cross-arm median jaccard):**

  | Granularity | Cross-arm median [min,max] | Within-arm median (A / B) |
  |-------------|----------------------------|---------------------------|
  | `(file,line,rule_key)` triplet | 0.41 [0.29, 0.64] | 0.40 / 0.32 |
  | `(file,rule_key)` (ignore line) | **0.71** [0.58, 0.91] | 0.69 / 0.67 |
  | `rule_key` set | 0.86 [0.86, 1.00] | — |
  | `file` set | **1.00** | — |

- **Underlying-issue coverage: 5/5 core issues flagged in 6/6 runs (100%)** — alg-from-header, soft-fail-200, bcrypt-drift, missing-test, refresh error-handling.
- **rule_key mismatches: 0.** **severity mismatches: 6/9 pairs × 1** — traced to `MISSING_ERROR_HANDLING` rated MINOR (catalogue tier) in Arm A vs MAJOR (brief-Constraint framing) in Arm B. Does **not** change the verdict.
- **Read:** the substrate is **fidelity-neutral** — cross-arm divergence ≤ each arm's own run-to-run nondeterminism at *every* granularity (cross 0.71 ≥ within 0.67–0.69 at `(file,rule_key)`). The low triplet jaccard is **line-citation noise shared by both arms** (reviewers cite the same defect at line 24/25/26 or 44/46/50), not a substrate effect.

### 2. JSON-robustness (the F2 win)

- **Arm B:** StructuredOutput schema-forced — typed findings, zero `JSON.parse`, `rule_key` enum enforced **at the tool layer** (the agent literally cannot emit an out-of-catalogue key — stronger than NW1's post-hoc check).
- **Arm A:** 6/6 reviewer outputs valid via NW1 `validateReviewerOutput` (0 parse errors, 0 schema errors, **0 re-asks**); 3/3 coordinator trailing-json parsed clean.
- Both arms 0 re-asks this run ⇒ Arm B's win is **structural**, not a measured re-ask delta (the rich fixture did not provoke malformed JSON; a delta needs a JSON-hostile fixture).

### 3. Classifier interference: 0 under the session's active mode — auto/bypass UNTESTED (open residual)

Arm B ran **3 concurrent workflows = 9 concurrent agents**; Arm A ran **6 concurrent reviewers + 3 concurrent coordinators**. No denied/missing spawns in any arm under the session's active mode. Confirms S8 F4 at higher (9-agent) concurrency.
**Open residual (Survivor #18 / S21, 2026-06-19).** The W1 charter made a re-run under `auto`/`bypass` permission mode an *explicit* guard, because trekreview can run headless under exactly those modes. That re-run was **never performed** — the permission mode is operator-set, not settable from within an interactive session — and was originally footnoted rather than gating. Per the devil's-advocate audit (Survivor #18, recommendation #9) this result is therefore **not "metric satisfied"**: it holds only under the session's active mode. The mode that matters for **headless** trekreview (`auto`/`bypass`, ≥9-agent fan-out) remains **untested** and stays an **open residual** until measured from a genuinely headless run. trekreview's small fan-out showed 0 interference in S8 and here; the *large* fan-out (trekplan swarm) is explicitly out of narrow-wins scope.

### 4. Token cost (comparable — both ran coordinator)

Arm A median **92.8k** subagent-tokens/run; Arm B median **96.9k** ⇒ **+4.4%** (≤ +15% POSITIVE bar; far below +30% NEGATIVE). Arm A additionally burdens the **main context** with hand-orchestration (validate/dedup/prompt-build) that Arm B offloads to the workflow runtime — an uncounted Arm-A cost, i.e. a further point for Arm B.

### 5. Wall-time

Arm A median ~150s/run; Arm B ~230s ⇒ **+54%**. **Caveat:** not a controlled per-run comparison — Arm A's reviewers+coordinators were batch-parallelized across the 3 runs, Arm B ran 3 full pipelines concurrently. Arm B is **non-blocking** (background) and frees the main context for the duration.

### 6. Control / visibility

Arm B runs in the background; intermediate findings visible in the workflow transcript + `/workflows`; returns structured `{verdict, findings}`. Phase 7 rendering stays shared/prose; the operator-gate (review.md write) is unaffected. Arm B **frees the main context** during the review (the 3 workflows ran while main did other work) — Arm A occupies it end-to-end.

## VERDICT: **opt-in-defensible** (POSITIVE on measured axes; single un-archived run)

The Workflow substrate (Arm B) is **fidelity-equivalent** to the prose path (Arm A) on
the operator-meaningful axes — verdict 1.0, file-set 1.0, issue-coverage 100%,
`(file,rule_key)` jaccard ≥ within-arm — with **comparable tokens (+4.4%)**, **zero
classifier interference**, **structurally stronger JSON robustness**, and **better
control/visibility** (background + `/workflows` + frees main context). There is **no
substrate-attributable divergence**: cross-arm ≤ within-arm at every granularity.

**Caveat (surfaced per plan §Posture, not hidden):** the strict `fidelityDiffStructured`
`equivalent` flag (triplet-jaccard ≥ 0.7) is **0/9**. This is a **metric-calibration
artifact** — *both* arms score sub-0.7 against *themselves* at triplet granularity because
live reviewers vary line citations and rule_key choice per semantically-identical issue.
It is **not** a regression: the granularity ladder and within-vs-cross comparison isolate
the divergence as intrinsic LLM nondeterminism, equal in both arms. Arm B's wall-time is
~+54% but non-blocking.

## Reproducibility caveat (Survivor #3, added S17)

The per-run arm outputs (`a1.json`..`b3.json`) were **never committed** — `git log
--diff-filter=A` finds no such file in any ref, and `scripts/bakeoff-fidelity.mjs`
requires them as input. The medians and the jaccard ladder above therefore **cannot be
regenerated, audited, or falsified** from the repo. This is why the verdict is labeled
**opt-in-defensible** rather than a clean POSITIVE: it justifies shipping `--workflow`
behind a flag (the measured axes did come out fidelity-neutral), but it rests on a
**single un-archived run** and is not a reproducible result. To upgrade it, a future
session must re-run the bake-off and commit the raw per-run JSON under
`tests/fixtures/bakeoff-rich/runs/` with a test that re-derives these numbers.

**→ S11:** proceed with the **opt-in `--workflow` flag** (prose stays default — preserves
the 2.1.154+ portability floor; see §Open decisions in the narrow-wins plan). The bake-off
supports making the Workflow path reachable; residuals are (a) the F4 `auto`/`bypass`
explicit-mode check and (b) wiring the flag to reuse the NW1 schemas. A future fidelity
metric should score at `(file,rule_key)` granularity (line-noise-robust) rather than the
exact triplet.