Kjell Tore Guttormsen
869bf318d2
Build the prose-vs-Workflow bake-off machinery for /trekreview Phase 5-6 and
run a 1-run/arm smoke to de-risk before the full measurement (operator posture:
build + smoke, then pause for go/no-go on the full >=3-runs/arm run).
New:
- lib/review/fidelity-diff.mjs (+ tests) — the PRIMARY metric: parse two
review.md (or two structured arm outputs) and compare verdict + jaccard over
(file,line,rule_key)-IDs + per-finding severity/rule_key. Reuses jaccard +
frontmatter + NW1 findings-schema + finding-id. fidelityDiffStructured avoids
rendering review.md per run.
- scripts/trekreview-armB.workflow.mjs — Arm B: Phase 5-6 as a Workflow
(parallel([conformance, correctness]) schema-forced -> JS dedup-by-triplet ->
agent(review-coordinator) verdict schema). Path-based input via args (reviewers
carry Read). Inlines dedup + the 12-key rule_key enum (scripts have no imports).
- tests/fixtures/bakeoff/ — committable fixture: real diff of b149538 (NW1) +
brief reconstructed from plan S9. Both arms review the same pinned input.
- docs/T2-bakeoff-results.md — smoke results + verdict + go/no-go recommendation.
Smoke result: SMOKE PASS. Arm B runs the full pipeline (3 agents) with ZERO
classifier interference; fidelity EQUIVALENT to Arm A at the verdict level
(both ALLOW; jaccard 1.0). Caveat: the clean TDD'd fixture yielded ~0 findings,
so finding-SET fidelity was not stressed (only verdict fidelity proven). A
reviewer-level divergence appeared (Arm B raised 1 raw finding, coordinator
filtered it; Arm A raised 0) — to be quantified in the full run on a
richer-finding-surface fixture. NOT the T2 §5 POSITIVE/NEGATIVE verdict.
Suite 647 -> 662 (660 pass / 2 skip / 0 fail; +15 fidelity-diff). claude plugin
validate clean (known root-CLAUDE.md warning only). Plan: docs/W1-narrow-wins-plan.md S10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
90 lines
4.9 KiB
Markdown
90 lines
4.9 KiB
Markdown
# T2 / NW2 — prose-vs-Workflow bake-off results
|
||
|
||
**Status (S10):** Build complete + **smoke run (1 run/arm) done**. Full ≥3-runs/arm
|
||
measurement is **pending operator go/no-go** (operator posture: "build + smoke, then pause").
|
||
**This document records the smoke; it is NOT the full T2 §5 verdict.**
|
||
|
||
Resolves: the build + de-risk half of `docs/W1-narrow-wins-plan.md §S10`.
|
||
|
||
---
|
||
|
||
## Setup
|
||
|
||
| Item | Value |
|
||
|------|-------|
|
||
| Fixture | `tests/fixtures/bakeoff/` — real diff of commit `b149538` (NW1) + brief reconstructed from `docs/W1-narrow-wins-plan.md §S9` |
|
||
| Delivered diff | 3 files, +438/-3 (`commands/trekreview.md`, `lib/review/findings-schema.mjs`, `tests/lib/findings-schema.test.mjs`) |
|
||
| Triage | all 3 files `summary-only` (deterministic Phase-4 classifier) |
|
||
| Model | `opus` (reviewer + coordinator agent defaults) |
|
||
| Arm A (prose) | reviewers spawned via Agent tool, prose trailing-JSON contract (validated by NW1 `findings-schema`) |
|
||
| Arm B (Workflow) | `scripts/trekreview-armB.workflow.mjs` via Workflow tool: `parallel([conformance, correctness])` schema-forced → JS dedup-by-triplet → `agent(review-coordinator)` verdict schema |
|
||
| Fidelity metric | `lib/review/fidelity-diff.mjs` `fidelityDiffStructured` (verdict + jaccard over `(file,line,rule_key)`-IDs + severity/rule cross-check) |
|
||
|
||
## Smoke results (single run per arm)
|
||
|
||
| | Arm A (prose) | Arm B (Workflow) |
|
||
|---|---|---|
|
||
| Reviewers run | 2 | 2 |
|
||
| Raw findings | 0 | 1 |
|
||
| After coordinator | 0 (coordinator moot — 0 findings) | 0 (coordinator dropped the 1 finding) |
|
||
| **Verdict** | **ALLOW** | **ALLOW** |
|
||
| Agents | 2 | 3 (2 reviewers + coordinator) |
|
||
| Subagent tokens | ~72.3k (34.3k + 38.0k; no coordinator) | ~100.9k (incl. coordinator) |
|
||
| Wall-time | ~70 s (parallel reviewers) | ~157 s (full pipeline) |
|
||
|
||
### PRIMARY metric — output fidelity: **EQUIVALENT**
|
||
|
||
```
|
||
fidelityDiffStructured(ArmA, ArmB) =
|
||
{ verdictMatch: true, jaccard: 1, countA: 0, countB: 0,
|
||
severityMismatches: [], ruleKeyMismatches: [], equivalent: true }
|
||
```
|
||
|
||
⚠ **Caveat — thin finding surface.** Both arms returned **0 final findings** on
|
||
this clean, TDD'd fixture, so fidelity is confirmed only at the **verdict** level
|
||
(ALLOW ≡ ALLOW); the finding-*set* fidelity is trivially equal at zero and was
|
||
**not stressed**. A reviewer-level divergence *did* appear (Arm B raised 1 raw
|
||
finding, its coordinator filtered it; Arm A raised 0) — masked at the verdict
|
||
level. Quantifying that divergence is exactly what the full run on a
|
||
richer-finding-surface fixture must do.
|
||
|
||
### Secondary metrics (smoke, single-run — not medians)
|
||
|
||
- **JSON-robustness (the F2 win):** Arm B's reviewers were **schema-forced**
|
||
(StructuredOutput) — typed findings, zero `JSON.parse`; the 1 raw finding +
|
||
the coordinator verdict both conformed with no re-ask. Arm A's trailing-JSON
|
||
validated clean via NW1 `findings-schema`. Win demonstrated structurally; the
|
||
parse-error/re-ask delta needs a fixture that actually provokes malformed JSON.
|
||
- **Classifier interference: 0.** Arm B's 2-agent fan-out + coordinator (3 agents)
|
||
ran with no denied/missing spawns. Confirms S8 F4 for trekreview's small
|
||
fan-out under the default mode. (`auto`/`bypass` still to be checked in the full run.)
|
||
- **Token cost:** preliminary and **not yet comparable** — Arm B ran a coordinator
|
||
(on its 1 finding) that Arm A did not. Single run; no medians.
|
||
- **Control/visibility:** Arm B runs in the background; intermediate findings are
|
||
visible in the workflow transcript + `/workflows`. Operator-gate (the review.md
|
||
write) is unaffected — both arms return structured `{verdict, findings}` and
|
||
Phase 7 rendering stays shared/prose.
|
||
|
||
## Smoke verdict
|
||
|
||
**SMOKE PASS — machinery validated.** Arm B (Workflow substrate) runs the full
|
||
Phase 5–6 pipeline end-to-end, fidelity-**equivalent** to Arm A at the verdict
|
||
level, with **zero classifier interference**. The build is sound: NW1 schema,
|
||
fidelity-diff, fixture, and the Arm B port all work together.
|
||
|
||
This is **not** the T2 §5 POSITIVE/NEGATIVE verdict — that needs the full
|
||
≥3-runs/arm measurement with a finding-rich fixture.
|
||
|
||
## Go / no-go recommendation (operator decides)
|
||
|
||
**Recommend: proceed to the full ≥3-runs/arm run (S10 part B)** with two changes:
|
||
1. **Use a richer-finding-surface fixture** (a larger real voyage commit, or seed
|
||
the fixture with a few genuine issues) so finding-*set* fidelity is actually
|
||
stressed — the smoke only proved verdict fidelity at 0 findings.
|
||
2. **Match the arms' coordinator path** (run Arm A's coordinator too, even at low
|
||
finding counts) so the token/wall-time comparison is apples-to-apples, and
|
||
add the `auto`/`bypass` classifier-interference check (F4).
|
||
|
||
If the operator prefers, S11 can instead record "port built + smoke-validated;
|
||
full measurement deferred" and integrate behind the opt-in `--workflow` flag on
|
||
the smoke evidence alone — weaker, but the substrate is demonstrably functional.
|