feat(voyage): S10 part B — NW2 full bake-off (rich fixture) → verdict POSITIVE
Run the full T2 §5 prose-vs-Workflow /trekreview bake-off (operator GO, choice "a"): 3 runs/arm on a rich-finding JWT-auth fixture, resolving the smoke's 0-finding limitation. Deliverables: - tests/fixtures/bakeoff-rich/ — JWT-auth brief + diff with 5 seeded blatant, brief-traceable issues (varied severity/rule_key, one dual-flaggable). - scripts/bakeoff-armA-merge.mjs — Arm A (prose) validate (NW1) + triplet-dedup, matching Arm B's dedup exactly. - scripts/bakeoff-fidelity.mjs — cross-arm + within-arm + granularity-ladder fidelity analysis over the structured arm outputs. - docs/T2-bakeoff-results.md §Full run — the T2 §5 verdict. Result (3 runs/arm, both arms ran the coordinator): - Verdict fidelity EQUIVALENT — all 6 runs BLOCK, cross-arm verdict-match 1.0. - Finding-set: substrate is fidelity-neutral. Cross-arm jaccard 0.41 (triplet) → 0.71 (file,rule_key) → 1.0 (file); cross-arm ≈ within-arm at every granularity. Issue coverage 5/5 in 6/6 runs. Low triplet jaccard is line-citation noise shared by both arms, not a substrate effect. - Token +4.4% (Arm B vs A; <=+15%). Classifier interference 0 at 9-agent concurrency. JSON-robustness: Arm B schema-forced; Arm A 6/6 valid via NW1. - VERDICT POSITIVE → S11 proceeds with opt-in --workflow flag. Caveat (per plan posture): strict triplet-jaccard>=0.7 flag is 0/9, a metric-calibration artifact (both arms sub-0.7 against themselves), not a regression. Residual: F4 auto/bypass explicit-mode check (mode not settable in-session). Suite green (662/660 pass, 2 skip); plugin validate clean (modulo the pre-existing root-CLAUDE.md warning). No production code changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
parent
869bf318d2
commit
f7c8aa45ab
6 changed files with 537 additions and 4 deletions
|
|
@ -1,10 +1,12 @@
|
|||
# T2 / NW2 — prose-vs-Workflow bake-off results
|
||||
|
||||
**Status (S10):** Build complete + **smoke run (1 run/arm) done**. Full ≥3-runs/arm
|
||||
measurement is **pending operator go/no-go** (operator posture: "build + smoke, then pause").
|
||||
**This document records the smoke; it is NOT the full T2 §5 verdict.**
|
||||
**Status (S10):** Part A (build + smoke) **and** part B (full ≥3-runs/arm
|
||||
measurement) **COMPLETE**. Operator GO 2026-06-18 (choice "a" — full ≥3× run).
|
||||
**T2 §5 verdict: POSITIVE** — the Workflow substrate is fidelity-neutral; proceed to
|
||||
the opt-in `--workflow` flag at S11. Full evidence in **§Full run (S10 part B)** at the
|
||||
end of this doc. The smoke section below is retained for history.
|
||||
|
||||
Resolves: the build + de-risk half of `docs/W1-narrow-wins-plan.md §S10`.
|
||||
Resolves: `docs/W1-narrow-wins-plan.md §S10` (build + de-risk **and** full measurement).
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -88,3 +90,96 @@ This is **not** the T2 §5 POSITIVE/NEGATIVE verdict — that needs the full
|
|||
If the operator prefers, S11 can instead record "port built + smoke-validated;
|
||||
full measurement deferred" and integrate behind the opt-in `--workflow` flag on
|
||||
the smoke evidence alone — weaker, but the substrate is demonstrably functional.
|
||||
|
||||
---
|
||||
|
||||
# Full run (S10 part B) — ≥3 runs/arm, rich-finding fixture
|
||||
|
||||
**This is the T2 §5 verdict.** Operator GO 2026-06-18 (choice "a"). Resolves the
|
||||
measurement half of `docs/W1-narrow-wins-plan.md §S10`.
|
||||
|
||||
## Setup (vs smoke)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Fixture | `tests/fixtures/bakeoff-rich/` — JWT-auth brief + diff, **5 seeded blatant issues** (varied severity/rule_key, one dual-flaggable). Live reviewers surface **11–18 findings/run** — the smoke's 0-finding limitation is resolved. |
|
||||
| Triage | all 3 files `deep-review` (auth/security surface) — pinned, passed to both arms |
|
||||
| Both arms run the coordinator | yes (token now comparable; smoke ran A's coordinator only on its 1 finding) |
|
||||
| Arm A (prose) | foreground reviewers (Agent tool, no `name`) → `validateReviewerOutput` (NW1) → `scripts/bakeoff-armA-merge.mjs` triplet-dedup → foreground `review-coordinator`. Runs ×3. |
|
||||
| Arm B (Workflow) | `scripts/trekreview-armB.workflow.mjs` ×3 (StructuredOutput findings → JS triplet-dedup → coordinator verdict schema) |
|
||||
| Analysis | `scripts/bakeoff-fidelity.mjs` — cross-arm + within-arm + granularity ladder |
|
||||
|
||||
## Raw results
|
||||
|
||||
| Run | Arm A (prose) | Arm B (Workflow) |
|
||||
|-----|---------------|------------------|
|
||||
| 1 | BLOCK · 13 findings · 86.3k tok · ~119s | BLOCK · 11 findings · 96.9k tok · ~230s |
|
||||
| 2 | BLOCK · 12 findings · 92.8k tok · ~150s | BLOCK · 14 findings · 91.9k tok · ~201s |
|
||||
| 3 | BLOCK · 16 findings · 95.8k tok · ~188s | BLOCK · 18 findings · 97.2k tok · ~249s |
|
||||
|
||||
All 6 runs → **BLOCK**. (Arm A tokens = 2 reviewers + coordinator subagent-tokens; wall = reviewer-phase max + coordinator.)
|
||||
|
||||
## Metrics vs T2 §5
|
||||
|
||||
### 1. PRIMARY — output fidelity
|
||||
|
||||
- **Verdict: EQUIVALENT.** Verdict-match rate **1.0** — all 9 cross-arm (Aᵢ×Bⱼ) pairs agree, all 6 runs BLOCK. This is the operator-meaningful gate decision.
|
||||
- **Finding-set — granularity ladder (cross-arm median jaccard):**
|
||||
|
||||
| Granularity | Cross-arm median [min,max] | Within-arm median (A / B) |
|
||||
|-------------|----------------------------|---------------------------|
|
||||
| `(file,line,rule_key)` triplet | 0.41 [0.29, 0.64] | 0.40 / 0.32 |
|
||||
| `(file,rule_key)` (ignore line) | **0.71** [0.58, 0.91] | 0.69 / 0.67 |
|
||||
| `rule_key` set | 0.86 [0.86, 1.00] | — |
|
||||
| `file` set | **1.00** | — |
|
||||
|
||||
- **Underlying-issue coverage: 5/5 core issues flagged in 6/6 runs (100%)** — alg-from-header, soft-fail-200, bcrypt-drift, missing-test, refresh error-handling.
|
||||
- **rule_key mismatches: 0.** **severity mismatches: 6/9 pairs × 1** — traced to `MISSING_ERROR_HANDLING` rated MINOR (catalogue tier) in Arm A vs MAJOR (brief-Constraint framing) in Arm B. Does **not** change the verdict.
|
||||
- **Read:** the substrate is **fidelity-neutral** — cross-arm divergence ≤ each arm's own run-to-run nondeterminism at *every* granularity (cross 0.71 ≥ within 0.67–0.69 at `(file,rule_key)`). The low triplet jaccard is **line-citation noise shared by both arms** (reviewers cite the same defect at line 24/25/26 or 44/46/50), not a substrate effect.
|
||||
|
||||
### 2. JSON-robustness (the F2 win)
|
||||
|
||||
- **Arm B:** StructuredOutput schema-forced — typed findings, zero `JSON.parse`, `rule_key` enum enforced **at the tool layer** (the agent literally cannot emit an out-of-catalogue key — stronger than NW1's post-hoc check).
|
||||
- **Arm A:** 6/6 reviewer outputs valid via NW1 `validateReviewerOutput` (0 parse errors, 0 schema errors, **0 re-asks**); 3/3 coordinator trailing-json parsed clean.
|
||||
- Both arms 0 re-asks this run ⇒ Arm B's win is **structural**, not a measured re-ask delta (the rich fixture did not provoke malformed JSON; a delta needs a JSON-hostile fixture).
|
||||
|
||||
### 3. Classifier interference: 0
|
||||
|
||||
Arm B ran **3 concurrent workflows = 9 concurrent agents**; Arm A ran **6 concurrent reviewers + 3 concurrent coordinators**. No denied/missing spawns in any arm under the session's active mode. Confirms S8 F4 at higher (9-agent) concurrency.
|
||||
**Residual:** an explicit `auto`/`bypass`-mode re-run was **not** performed — the permission mode is operator-set, not settable from within the session. trekreview's small fan-out showed 0 interference in S8 and here; the *large* fan-out (trekplan swarm) is explicitly out of narrow-wins scope.
|
||||
|
||||
### 4. Token cost (comparable — both ran coordinator)
|
||||
|
||||
Arm A median **92.8k** subagent-tokens/run; Arm B median **96.9k** ⇒ **+4.4%** (≤ +15% POSITIVE bar; far below +30% NEGATIVE). Arm A additionally burdens the **main context** with hand-orchestration (validate/dedup/prompt-build) that Arm B offloads to the workflow runtime — an uncounted Arm-A cost, i.e. a further point for Arm B.
|
||||
|
||||
### 5. Wall-time
|
||||
|
||||
Arm A median ~150s/run; Arm B ~230s ⇒ **+54%**. **Caveat:** not a controlled per-run comparison — Arm A's reviewers+coordinators were batch-parallelized across the 3 runs, Arm B ran 3 full pipelines concurrently. Arm B is **non-blocking** (background) and frees the main context for the duration.
|
||||
|
||||
### 6. Control / visibility
|
||||
|
||||
Arm B runs in the background; intermediate findings visible in the workflow transcript + `/workflows`; returns structured `{verdict, findings}`. Phase 7 rendering stays shared/prose; the operator-gate (review.md write) is unaffected. Arm B **frees the main context** during the review (the 3 workflows ran while main did other work) — Arm A occupies it end-to-end.
|
||||
|
||||
## VERDICT: **POSITIVE**
|
||||
|
||||
The Workflow substrate (Arm B) is **fidelity-equivalent** to the prose path (Arm A) on
|
||||
the operator-meaningful axes — verdict 1.0, file-set 1.0, issue-coverage 100%,
|
||||
`(file,rule_key)` jaccard ≥ within-arm — with **comparable tokens (+4.4%)**, **zero
|
||||
classifier interference**, **structurally stronger JSON robustness**, and **better
|
||||
control/visibility** (background + `/workflows` + frees main context). There is **no
|
||||
substrate-attributable divergence**: cross-arm ≤ within-arm at every granularity.
|
||||
|
||||
**Caveat (surfaced per plan §Posture, not hidden):** the strict `fidelityDiffStructured`
|
||||
`equivalent` flag (triplet-jaccard ≥ 0.7) is **0/9**. This is a **metric-calibration
|
||||
artifact** — *both* arms score sub-0.7 against *themselves* at triplet granularity because
|
||||
live reviewers vary line citations and rule_key choice per semantically-identical issue.
|
||||
It is **not** a regression: the granularity ladder and within-vs-cross comparison isolate
|
||||
the divergence as intrinsic LLM nondeterminism, equal in both arms. Arm B's wall-time is
|
||||
~+54% but non-blocking.
|
||||
|
||||
**→ S11:** proceed with the **opt-in `--workflow` flag** (prose stays default — preserves
|
||||
the 2.1.154+ portability floor; see §Open decisions in the narrow-wins plan). The bake-off
|
||||
supports making the Workflow path reachable; residuals are (a) the F4 `auto`/`bypass`
|
||||
explicit-mode check and (b) wiring the flag to reuse the NW1 schemas. A future fidelity
|
||||
metric should score at `(file,rule_key)` granularity (line-noise-robust) rather than the
|
||||
exact triplet.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue