Kjell Tore Guttormsen
f7c8aa45ab
Run the full T2 §5 prose-vs-Workflow /trekreview bake-off (operator GO, choice "a"): 3 runs/arm on a rich-finding JWT-auth fixture, resolving the smoke's 0-finding limitation. Deliverables: - tests/fixtures/bakeoff-rich/ — JWT-auth brief + diff with 5 seeded blatant, brief-traceable issues (varied severity/rule_key, one dual-flaggable). - scripts/bakeoff-armA-merge.mjs — Arm A (prose) validate (NW1) + triplet-dedup, matching Arm B's dedup exactly. - scripts/bakeoff-fidelity.mjs — cross-arm + within-arm + granularity-ladder fidelity analysis over the structured arm outputs. - docs/T2-bakeoff-results.md §Full run — the T2 §5 verdict. Result (3 runs/arm, both arms ran the coordinator): - Verdict fidelity EQUIVALENT — all 6 runs BLOCK, cross-arm verdict-match 1.0. - Finding-set: substrate is fidelity-neutral. Cross-arm jaccard 0.41 (triplet) → 0.71 (file,rule_key) → 1.0 (file); cross-arm ≈ within-arm at every granularity. Issue coverage 5/5 in 6/6 runs. Low triplet jaccard is line-citation noise shared by both arms, not a substrate effect. - Token +4.4% (Arm B vs A; <=+15%). Classifier interference 0 at 9-agent concurrency. JSON-robustness: Arm B schema-forced; Arm A 6/6 valid via NW1. - VERDICT POSITIVE → S11 proceeds with opt-in --workflow flag. Caveat (per plan posture): strict triplet-jaccard>=0.7 flag is 0/9, a metric-calibration artifact (both arms sub-0.7 against themselves), not a regression. Residual: F4 auto/bypass explicit-mode check (mode not settable in-session). Suite green (662/660 pass, 2 skip); plugin validate clean (modulo the pre-existing root-CLAUDE.md warning). No production code changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
3.5 KiB
NW2 bake-off — rich-finding-surface fixture (S10 part B)
The smoke fixture (tests/fixtures/bakeoff/) reviewed a clean, TDD'd NW1 diff
and both arms returned 0 findings, so finding-set fidelity was never
stressed — only verdict fidelity at zero. This fixture fixes that: a realistic
JWT-auth diff + brief seeded with 5 blatant, brief-traceable issues spanning
varied severities and rule_keys, split across both reviewers. Both arms review
the same delivered.diff against the same brief.md, so any difference
in the {verdict, findings} is attributable to the orchestration substrate
(prose Arm A vs Workflow Arm B), not the input.
Modeled on the proven determinism scenario in
tests/fixtures/trekreview/review-run-A.md (same JWT-auth shape, same finding
families), but here it is a real diff + brief pair the live reviewers read —
not a synthetic pre-rendered review.
Seeded findings (expected ~5, varied)
| # | Issue | Where | Likely rule_key | Severity | Owner reviewer |
|---|---|---|---|---|---|
| 1 | /login returns 200 (not 401) on invalid credentials |
lib/handlers/login.mjs:17 |
UNIMPLEMENTED_CRITERION |
BLOCKER | conformance (SC2) |
| 2 | verifyToken reads the verify algorithm from a request header |
lib/auth/jwt.mjs:19–20 |
SECURITY_INJECTION and/or NON_GOAL_VIOLATED |
BLOCKER | correctness + conformance (NG1) |
| 3 | No test covers concurrent refresh (no test file in the diff) | lib/auth/refresh.mjs (whole) |
MISSING_TEST |
MAJOR | correctness (SC3) |
| 4 | Password check uses crypto.timingSafeEqual over plaintext, not bcrypt.compare per plan |
lib/handlers/login.mjs:13 |
PLAN_EXECUTE_DRIFT |
MAJOR | conformance/correctness (Plan Step 4) |
| 5 | refreshStore I/O (get/delete/set) is unwrapped — backend outage bubbles unhandled |
lib/auth/refresh.mjs:10,16,20 |
MISSING_ERROR_HANDLING |
MINOR | correctness (Constraint) |
Issue #2 is intentionally dual-flaggable (a security defect AND an explicit
Non-Goal violation) — it exercises the cross-reviewer overlap that the
(file,line,rule_key) triplet-dedup and the coordinator must handle. Real LLM
reviewers will vary exact line numbers and may surface extra latent issues (e.g.
the user.passwordHash NPE when the email is unknown); that variance is the
signal the bake-off measures, not noise to suppress.
Expected verdict (both arms): BLOCK (≥1 BLOCKER present).
Triage map (deterministic, pinned — passed to BOTH arms)
All three files are auth/security surface → deep-review:
lib/auth/jwt.mjs → deep-review
lib/handlers/login.mjs → deep-review
lib/auth/refresh.mjs → deep-review
How it's consumed
The bake-off pins Phases 1–4 (brief + diff + triage above) and passes them to
both arms via paths (reviewer agents carry Read):
- Arm A (prose): reviewers spawned FOREGROUND via the Agent tool with the
prose trailing-
json-block contract →validateReviewerOutput(NW1) → triplet-dedup →review-coordinator→{verdict, findings}. Harness:scripts/bakeoff-armA-merge.mjs. - Arm B (Workflow):
scripts/trekreview-armB.workflow.mjsvia the Workflow tool,args = { briefPath, diffPath, triage }(StructuredOutput-forced findings → JS triplet-dedup → coordinator verdict schema).
PRIMARY metric: fidelityDiffStructured (lib/review/fidelity-diff.mjs) — same
verdict + equivalent finding set (IDs / severities / rule_keys), jaccard
tolerance 0.7. Analysis harness: scripts/bakeoff-fidelity.mjs.