voyage/docs/T2-bakeoff-results.md

T2 / NW2 — prose-vs-Workflow bake-off results

Status (S10): Part A (build + smoke) and part B (full ≥3-runs/arm measurement) COMPLETE. Operator GO 2026-06-18 (choice "a" — full ≥3× run). T2 §5 verdict: opt-in-defensible (POSITIVE on the measured axes, on a single un-archived run — see §Reproducibility caveat) — the Workflow substrate is fidelity-neutral on what was measured; proceed to the opt-in --workflow flag at S11. Full evidence in §Full run (S10 part B) at the end of this doc. The smoke section below is retained for history.

Resolves: docs/W1-narrow-wins-plan.md §S10 (build + de-risk and full measurement).

---

Setup

Item	Value
Fixture	tests/fixtures/bakeoff/ — real diff of commit b149538 (NW1) + brief reconstructed from docs/W1-narrow-wins-plan.md §S9
Delivered diff	3 files, +438/-3 (commands/trekreview.md, lib/review/findings-schema.mjs, tests/lib/findings-schema.test.mjs)
Triage	all 3 files summary-only (deterministic Phase-4 classifier)
Model	opus (reviewer + coordinator agent defaults)
Arm A (prose)	reviewers spawned via Agent tool, prose trailing-JSON contract (validated by NW1 findings-schema)
Arm B (Workflow)	scripts/trekreview-armB.workflow.mjs via Workflow tool: parallel([conformance, correctness]) schema-forced → JS dedup-by-triplet → agent(review-coordinator) verdict schema
Fidelity metric	lib/review/fidelity-diff.mjs fidelityDiffStructured (verdict + jaccard over (file,line,rule_key)-IDs + severity/rule cross-check)

Smoke results (single run per arm)

	Arm A (prose)	Arm B (Workflow)
Reviewers run	2	2
Raw findings	0	1
After coordinator	0 (coordinator moot — 0 findings)	0 (coordinator dropped the 1 finding)
Verdict	ALLOW	ALLOW
Agents	2	3 (2 reviewers + coordinator)
Subagent tokens	~72.3k (34.3k + 38.0k; no coordinator)	~100.9k (incl. coordinator)
Wall-time	~70 s (parallel reviewers)	~157 s (full pipeline)

PRIMARY metric — output fidelity: EQUIVALENT

fidelityDiffStructured(ArmA, ArmB) =
  { verdictMatch: true, jaccard: 1, countA: 0, countB: 0,
    severityMismatches: [], ruleKeyMismatches: [], equivalent: true }

⚠ Caveat — thin finding surface. Both arms returned 0 final findings on this clean, TDD'd fixture, so fidelity is confirmed only at the verdict level (ALLOW ≡ ALLOW); the finding-set fidelity is trivially equal at zero and was not stressed. A reviewer-level divergence did appear (Arm B raised 1 raw finding, its coordinator filtered it; Arm A raised 0) — masked at the verdict level. Quantifying that divergence is exactly what the full run on a richer-finding-surface fixture must do.

Secondary metrics (smoke, single-run — not medians)

• JSON-robustness (the F2 win): Arm B's reviewers were schema-forced (StructuredOutput) — typed findings, zero JSON.parse; the 1 raw finding + the coordinator verdict both conformed with no re-ask. Arm A's trailing-JSON validated clean via NW1 findings-schema. Win demonstrated structurally; the parse-error/re-ask delta needs a fixture that actually provokes malformed JSON.
• Classifier interference: 0. Arm B's 2-agent fan-out + coordinator (3 agents) ran with no denied/missing spawns. Confirms S8 F4 for trekreview's small fan-out under the default mode. (auto/bypass still to be checked in the full run.)
• Token cost: preliminary and not yet comparable — Arm B ran a coordinator (on its 1 finding) that Arm A did not. Single run; no medians.
• Control/visibility: Arm B runs in the background; intermediate findings are visible in the workflow transcript + /workflows. Operator-gate (the review.md write) is unaffected — both arms return structured {verdict, findings} and Phase 7 rendering stays shared/prose.

Smoke verdict

SMOKE PASS — machinery validated. Arm B (Workflow substrate) runs the full Phase 5–6 pipeline end-to-end, fidelity-equivalent to Arm A at the verdict level, with zero classifier interference. The build is sound: NW1 schema, fidelity-diff, fixture, and the Arm B port all work together.

This is not the T2 §5 POSITIVE/NEGATIVE verdict — that needs the full ≥3-runs/arm measurement with a finding-rich fixture.

Go / no-go recommendation (operator decides)

Recommend: proceed to the full ≥3-runs/arm run (S10 part B) with two changes:

1. Use a richer-finding-surface fixture (a larger real voyage commit, or seed the fixture with a few genuine issues) so finding-set fidelity is actually stressed — the smoke only proved verdict fidelity at 0 findings.
2. Match the arms' coordinator path (run Arm A's coordinator too, even at low finding counts) so the token/wall-time comparison is apples-to-apples, and add the auto/bypass classifier-interference check (F4).

If the operator prefers, S11 can instead record "port built + smoke-validated; full measurement deferred" and integrate behind the opt-in --workflow flag on the smoke evidence alone — weaker, but the substrate is demonstrably functional.

---

Full run (S10 part B) — ≥3 runs/arm, rich-finding fixture

This is the T2 §5 verdict. Operator GO 2026-06-18 (choice "a"). Resolves the measurement half of docs/W1-narrow-wins-plan.md §S10.

Setup (vs smoke)

Item	Value
Fixture	tests/fixtures/bakeoff-rich/ — JWT-auth brief + diff, 5 seeded blatant issues (varied severity/rule_key, one dual-flaggable). Live reviewers surface 11–18 findings/run — the smoke's 0-finding limitation is resolved.
Triage	all 3 files deep-review (auth/security surface) — pinned, passed to both arms
Both arms run the coordinator	yes (token now comparable; smoke ran A's coordinator only on its 1 finding)
Arm A (prose)	foreground reviewers (Agent tool, no name) → validateReviewerOutput (NW1) → scripts/bakeoff-armA-merge.mjs triplet-dedup → foreground review-coordinator. Runs ×3.
Arm B (Workflow)	scripts/trekreview-armB.workflow.mjs ×3 (StructuredOutput findings → JS triplet-dedup → coordinator verdict schema)
Analysis	scripts/bakeoff-fidelity.mjs — cross-arm + within-arm + granularity ladder

Raw results

Run	Arm A (prose)	Arm B (Workflow)
1	BLOCK · 13 findings · 86.3k tok · ~119s	BLOCK · 11 findings · 96.9k tok · ~230s
2	BLOCK · 12 findings · 92.8k tok · ~150s	BLOCK · 14 findings · 91.9k tok · ~201s
3	BLOCK · 16 findings · 95.8k tok · ~188s	BLOCK · 18 findings · 97.2k tok · ~249s

All 6 runs → BLOCK. (Arm A tokens = 2 reviewers + coordinator subagent-tokens; wall = reviewer-phase max + coordinator.)

Metrics vs T2 §5

1. PRIMARY — output fidelity

• Verdict: EQUIVALENT. Verdict-match rate 1.0 — all 9 cross-arm (Aᵢ×Bⱼ) pairs agree, all 6 runs BLOCK. This is the operator-meaningful gate decision.

• Finding-set — granularity ladder (cross-arm median jaccard):

  Granularity	Cross-arm median [min,max]	Within-arm median (A / B)
  (file,line,rule_key) triplet	0.41 [0.29, 0.64]	0.40 / 0.32
  (file,rule_key) (ignore line)	0.71 [0.58, 0.91]	0.69 / 0.67
  rule_key set	0.86 [0.86, 1.00]	—
  file set	1.00	—

• Underlying-issue coverage: 5/5 core issues flagged in 6/6 runs (100%) — alg-from-header, soft-fail-200, bcrypt-drift, missing-test, refresh error-handling.

• rule_key mismatches: 0. severity mismatches: 6/9 pairs × 1 — traced to MISSING_ERROR_HANDLING rated MINOR (catalogue tier) in Arm A vs MAJOR (brief-Constraint framing) in Arm B. Does not change the verdict.

• Read: the substrate is fidelity-neutral — cross-arm divergence ≤ each arm's own run-to-run nondeterminism at every granularity (cross 0.71 ≥ within 0.67–0.69 at (file,rule_key)). The low triplet jaccard is line-citation noise shared by both arms (reviewers cite the same defect at line 24/25/26 or 44/46/50), not a substrate effect.

2. JSON-robustness (the F2 win)

• Arm B: StructuredOutput schema-forced — typed findings, zero JSON.parse, rule_key enum enforced at the tool layer (the agent literally cannot emit an out-of-catalogue key — stronger than NW1's post-hoc check).
• Arm A: 6/6 reviewer outputs valid via NW1 validateReviewerOutput (0 parse errors, 0 schema errors, 0 re-asks); 3/3 coordinator trailing-json parsed clean.
• Both arms 0 re-asks this run ⇒ Arm B's win is structural, not a measured re-ask delta (the rich fixture did not provoke malformed JSON; a delta needs a JSON-hostile fixture).

3. Classifier interference: 0 under the session's active mode — auto/bypass UNTESTED (open residual)

Arm B ran 3 concurrent workflows = 9 concurrent agents; Arm A ran 6 concurrent reviewers + 3 concurrent coordinators. No denied/missing spawns in any arm under the session's active mode. Confirms S8 F4 at higher (9-agent) concurrency. Open residual (Survivor #18 / S21, 2026-06-19). The W1 charter made a re-run under auto/bypass permission mode an explicit guard, because trekreview can run headless under exactly those modes. That re-run was never performed — the permission mode is operator-set, not settable from within an interactive session — and was originally footnoted rather than gating. Per the devil's-advocate audit (Survivor #18, recommendation #9) this result is therefore not "metric satisfied": it holds only under the session's active mode. The mode that matters for headless trekreview (auto/bypass, ≥9-agent fan-out) remains untested and stays an open residual until measured from a genuinely headless run. trekreview's small fan-out showed 0 interference in S8 and here; the large fan-out (trekplan swarm) is explicitly out of narrow-wins scope.

4. Token cost (comparable — both ran coordinator)

Arm A median 92.8k subagent-tokens/run; Arm B median 96.9k ⇒ +4.4% (≤ +15% POSITIVE bar; far below +30% NEGATIVE). Arm A additionally burdens the main context with hand-orchestration (validate/dedup/prompt-build) that Arm B offloads to the workflow runtime — an uncounted Arm-A cost, i.e. a further point for Arm B.

5. Wall-time

Arm A median ~150s/run; Arm B ~230s ⇒ +54%. Caveat: not a controlled per-run comparison — Arm A's reviewers+coordinators were batch-parallelized across the 3 runs, Arm B ran 3 full pipelines concurrently. Arm B is non-blocking (background) and frees the main context for the duration.

6. Control / visibility

Arm B runs in the background; intermediate findings visible in the workflow transcript + /workflows; returns structured {verdict, findings}. Phase 7 rendering stays shared/prose; the operator-gate (review.md write) is unaffected. Arm B frees the main context during the review (the 3 workflows ran while main did other work) — Arm A occupies it end-to-end.

VERDICT: opt-in-defensible (POSITIVE on measured axes; single un-archived run)

The Workflow substrate (Arm B) is fidelity-equivalent to the prose path (Arm A) on the operator-meaningful axes — verdict 1.0, file-set 1.0, issue-coverage 100%, (file,rule_key) jaccard ≥ within-arm — with comparable tokens (+4.4%), zero classifier interference, structurally stronger JSON robustness, and better control/visibility (background + /workflows + frees main context). There is no substrate-attributable divergence: cross-arm ≤ within-arm at every granularity.

Caveat (surfaced per plan §Posture, not hidden): the strict fidelityDiffStructured equivalent flag (triplet-jaccard ≥ 0.7) is 0/9. This is a metric-calibration artifact — both arms score sub-0.7 against themselves at triplet granularity because live reviewers vary line citations and rule_key choice per semantically-identical issue. It is not a regression: the granularity ladder and within-vs-cross comparison isolate the divergence as intrinsic LLM nondeterminism, equal in both arms. Arm B's wall-time is ~+54% but non-blocking.

Reproducibility caveat (Survivor #3, added S17)

The per-run arm outputs (a1.json..b3.json) were never committed — git log --diff-filter=A finds no such file in any ref, and scripts/bakeoff-fidelity.mjs requires them as input. The medians and the jaccard ladder above therefore cannot be regenerated, audited, or falsified from the repo. This is why the verdict is labeled opt-in-defensible rather than a clean POSITIVE: it justifies shipping --workflow behind a flag (the measured axes did come out fidelity-neutral), but it rests on a single un-archived run and is not a reproducible result. To upgrade it, a future session must re-run the bake-off and commit the raw per-run JSON under tests/fixtures/bakeoff-rich/runs/ with a test that re-derives these numbers.

→ S11: proceed with the opt-in --workflow flag (prose stays default — preserves the 2.1.154+ portability floor; see §Open decisions in the narrow-wins plan). The bake-off supports making the Workflow path reachable; residuals are (a) the F4 auto/bypass explicit-mode check and (b) wiring the flag to reuse the NW1 schemas. A future fidelity metric should score at (file,rule_key) granularity (line-noise-robust) rather than the exact triplet.