voyage/docs/T2-bakeoff-results.md

T2 / NW2 — prose-vs-Workflow bake-off results

Status (S10): Build complete + smoke run (1 run/arm) done. Full ≥3-runs/arm measurement is pending operator go/no-go (operator posture: "build + smoke, then pause"). This document records the smoke; it is NOT the full T2 §5 verdict.

Resolves: the build + de-risk half of docs/W1-narrow-wins-plan.md §S10.

---

Setup

Item	Value
Fixture	tests/fixtures/bakeoff/ — real diff of commit b149538 (NW1) + brief reconstructed from docs/W1-narrow-wins-plan.md §S9
Delivered diff	3 files, +438/-3 (commands/trekreview.md, lib/review/findings-schema.mjs, tests/lib/findings-schema.test.mjs)
Triage	all 3 files summary-only (deterministic Phase-4 classifier)
Model	opus (reviewer + coordinator agent defaults)
Arm A (prose)	reviewers spawned via Agent tool, prose trailing-JSON contract (validated by NW1 findings-schema)
Arm B (Workflow)	scripts/trekreview-armB.workflow.mjs via Workflow tool: parallel([conformance, correctness]) schema-forced → JS dedup-by-triplet → agent(review-coordinator) verdict schema
Fidelity metric	lib/review/fidelity-diff.mjs fidelityDiffStructured (verdict + jaccard over (file,line,rule_key)-IDs + severity/rule cross-check)

Smoke results (single run per arm)

	Arm A (prose)	Arm B (Workflow)
Reviewers run	2	2
Raw findings	0	1
After coordinator	0 (coordinator moot — 0 findings)	0 (coordinator dropped the 1 finding)
Verdict	ALLOW	ALLOW
Agents	2	3 (2 reviewers + coordinator)
Subagent tokens	~72.3k (34.3k + 38.0k; no coordinator)	~100.9k (incl. coordinator)
Wall-time	~70 s (parallel reviewers)	~157 s (full pipeline)

PRIMARY metric — output fidelity: EQUIVALENT

fidelityDiffStructured(ArmA, ArmB) =
  { verdictMatch: true, jaccard: 1, countA: 0, countB: 0,
    severityMismatches: [], ruleKeyMismatches: [], equivalent: true }

⚠ Caveat — thin finding surface. Both arms returned 0 final findings on this clean, TDD'd fixture, so fidelity is confirmed only at the verdict level (ALLOW ≡ ALLOW); the finding-set fidelity is trivially equal at zero and was not stressed. A reviewer-level divergence did appear (Arm B raised 1 raw finding, its coordinator filtered it; Arm A raised 0) — masked at the verdict level. Quantifying that divergence is exactly what the full run on a richer-finding-surface fixture must do.

Secondary metrics (smoke, single-run — not medians)

• JSON-robustness (the F2 win): Arm B's reviewers were schema-forced (StructuredOutput) — typed findings, zero JSON.parse; the 1 raw finding + the coordinator verdict both conformed with no re-ask. Arm A's trailing-JSON validated clean via NW1 findings-schema. Win demonstrated structurally; the parse-error/re-ask delta needs a fixture that actually provokes malformed JSON.
• Classifier interference: 0. Arm B's 2-agent fan-out + coordinator (3 agents) ran with no denied/missing spawns. Confirms S8 F4 for trekreview's small fan-out under the default mode. (auto/bypass still to be checked in the full run.)
• Token cost: preliminary and not yet comparable — Arm B ran a coordinator (on its 1 finding) that Arm A did not. Single run; no medians.
• Control/visibility: Arm B runs in the background; intermediate findings are visible in the workflow transcript + /workflows. Operator-gate (the review.md write) is unaffected — both arms return structured {verdict, findings} and Phase 7 rendering stays shared/prose.

Smoke verdict

SMOKE PASS — machinery validated. Arm B (Workflow substrate) runs the full Phase 5–6 pipeline end-to-end, fidelity-equivalent to Arm A at the verdict level, with zero classifier interference. The build is sound: NW1 schema, fidelity-diff, fixture, and the Arm B port all work together.

This is not the T2 §5 POSITIVE/NEGATIVE verdict — that needs the full ≥3-runs/arm measurement with a finding-rich fixture.

Go / no-go recommendation (operator decides)

Recommend: proceed to the full ≥3-runs/arm run (S10 part B) with two changes:

1. Use a richer-finding-surface fixture (a larger real voyage commit, or seed the fixture with a few genuine issues) so finding-set fidelity is actually stressed — the smoke only proved verdict fidelity at 0 findings.
2. Match the arms' coordinator path (run Arm A's coordinator too, even at low finding counts) so the token/wall-time comparison is apples-to-apples, and add the auto/bypass classifier-interference check (F4).

If the operator prefers, S11 can instead record "port built + smoke-validated; full measurement deferred" and integrate behind the opt-in --workflow flag on the smoke evidence alone — weaker, but the substrate is demonstrably functional.