Kjell Tore Guttormsen
869bf318d2
Build the prose-vs-Workflow bake-off machinery for /trekreview Phase 5-6 and
run a 1-run/arm smoke to de-risk before the full measurement (operator posture:
build + smoke, then pause for go/no-go on the full >=3-runs/arm run).
New:
- lib/review/fidelity-diff.mjs (+ tests) — the PRIMARY metric: parse two
review.md (or two structured arm outputs) and compare verdict + jaccard over
(file,line,rule_key)-IDs + per-finding severity/rule_key. Reuses jaccard +
frontmatter + NW1 findings-schema + finding-id. fidelityDiffStructured avoids
rendering review.md per run.
- scripts/trekreview-armB.workflow.mjs — Arm B: Phase 5-6 as a Workflow
(parallel([conformance, correctness]) schema-forced -> JS dedup-by-triplet ->
agent(review-coordinator) verdict schema). Path-based input via args (reviewers
carry Read). Inlines dedup + the 12-key rule_key enum (scripts have no imports).
- tests/fixtures/bakeoff/ — committable fixture: real diff of b149538 (NW1) +
brief reconstructed from plan S9. Both arms review the same pinned input.
- docs/T2-bakeoff-results.md — smoke results + verdict + go/no-go recommendation.
Smoke result: SMOKE PASS. Arm B runs the full pipeline (3 agents) with ZERO
classifier interference; fidelity EQUIVALENT to Arm A at the verdict level
(both ALLOW; jaccard 1.0). Caveat: the clean TDD'd fixture yielded ~0 findings,
so finding-SET fidelity was not stressed (only verdict fidelity proven). A
reviewer-level divergence appeared (Arm B raised 1 raw finding, coordinator
filtered it; Arm A raised 0) — to be quantified in the full run on a
richer-finding-surface fixture. NOT the T2 §5 POSITIVE/NEGATIVE verdict.
Suite 647 -> 662 (660 pass / 2 skip / 0 fail; +15 fidelity-diff). claude plugin
validate clean (known root-CLAUDE.md warning only). Plan: docs/W1-narrow-wins-plan.md S10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
50 lines
2 KiB
Markdown
50 lines
2 KiB
Markdown
# NW2 bake-off fixture (S10)
|
||
|
||
Fixed, committable input for the prose-vs-Workflow `/trekreview` bake-off
|
||
(T2 §5). Both arms review **the same delivered diff against the same brief**, so
|
||
any difference in `review.md` is attributable to the orchestration substrate
|
||
(prose Arm A vs Workflow Arm B), not to the input.
|
||
|
||
## What's here
|
||
|
||
- `brief.md` — the contract. Reconstructed faithfully from the **real** NW1
|
||
contract (`docs/W1-narrow-wins-plan.md §S9`): goal, SC1–SC4, NG1–NG3,
|
||
constraints, NFRs.
|
||
- `delivered.diff` — the **real** delivered diff of commit `b149538`
|
||
(`git diff b149538^ b149538`): the NW1 implementation
|
||
(`lib/review/findings-schema.mjs` + tests + `commands/trekreview.md` Phase 5).
|
||
3 files, +438/-3.
|
||
|
||
## Why this commit
|
||
|
||
`b149538` (NW1/S9) is a genuine delivered-diff + brief pair already in this
|
||
repo's history — real code, real contract, fully self-contained (no external
|
||
deps), and small/cheap (ideal for the de-risking smoke run). It is not
|
||
finding-free (a strict correctness reviewer has surface to flag), so the
|
||
fidelity comparison is informative. A larger commit can be substituted for the
|
||
full ≥3-runs/arm measurement if more finding surface is wanted.
|
||
|
||
## Triage map (deterministic, pinned)
|
||
|
||
Per the Phase-4 path-pattern classifier, none of the 3 files match `skip` or
|
||
`deep-review` patterns → all `summary-only`:
|
||
|
||
```
|
||
commands/trekreview.md → summary-only
|
||
lib/review/findings-schema.mjs → summary-only
|
||
tests/lib/findings-schema.test.mjs → summary-only
|
||
```
|
||
|
||
## How it's consumed
|
||
|
||
The bake-off pins Phases 1–4 (brief + diff + triage above) and passes them to
|
||
both arms:
|
||
|
||
- **Arm A (prose):** current `/trekreview` Phase 5–6 behaviour.
|
||
- **Arm B (Workflow):** `scripts/trekreview-armB.workflow.mjs` via the Workflow
|
||
tool, `args = { brief, diff, triage }`.
|
||
|
||
Each arm yields a structured `{verdict, findings}`; the PRIMARY metric is
|
||
`fidelityDiffStructured` (`lib/review/fidelity-diff.mjs`) — same verdict +
|
||
equivalent finding set (IDs / severities / rule_keys).
|
||
```
|