open/voyage
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity Actions
voyage/tests/fixtures/bakeoff-rich
History
Exact
Kjell Tore Guttormsen 440594f1b2 feat(eval): SKAL-1·4b offline gold-scored output eval 
Scores committed agent-run fixtures against the golden corpus at
(file, rule_key) granularity, building on the deterministic coordinator
contract (4a). Offline: committed reviewer payloads, no live agent spawn,
no LLM, no network (the LLM-in-the-loop grading is the separate 4c tier).

- lib/review/gold-scorer.mjs: scoreFindings (precision/recall/f1 at
  (file,rule_key) granularity, line+severity ignored) + scoreVerdict; pure,
  with documented vacuous-set conventions.
- tests/fixtures/bakeoff-rich/runs/run-perfect.json: committed run that
  reproduces all 5 seeded gold findings through runContract.
- tests/lib/gold-eval.test.mjs: the scoring RUN (precision/recall/f1 = 1.0,
  verdict == expected_verdict BLOCK, nothing suppressed/skipped).
- lib/util/test-census.mjs: third census category (goldEval) — a scoring run
  is neither behavior coverage nor a doc-pin; honest-count invariant now 3-way.
- docs/eval-corpus/README.md: 4b moved from Future hardening to implemented.

Suite 809 -> 822 (820/0/2). gold-scorer covers TP+FP+FN+degenerate paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJQYC5vpkJWxndS55vQQZ6
2026-06-30 09:00:33 +02:00
..
runs feat(eval): SKAL-1·4b offline gold-scored output eval 2026-06-30 09:00:33 +02:00
brief.md feat(voyage): S10 part B — NW2 full bake-off (rich fixture) → verdict POSITIVE 2026-06-18 15:44:24 +02:00
delivered.diff feat(voyage): S10 part B — NW2 full bake-off (rich fixture) → verdict POSITIVE 2026-06-18 15:44:24 +02:00
gold.json feat(eval): add gold.json golden corpus + loader/validator test 2026-06-26 11:51:46 +02:00
README.md feat(voyage): S10 part B — NW2 full bake-off (rich fixture) → verdict POSITIVE 2026-06-18 15:44:24 +02:00

README.md

NW2 bake-off — rich-finding-surface fixture (S10 part B)

The smoke fixture (tests/fixtures/bakeoff/) reviewed a clean, TDD'd NW1 diff and both arms returned 0 findings, so finding-set fidelity was never stressed — only verdict fidelity at zero. This fixture fixes that: a realistic JWT-auth diff + brief seeded with 5 blatant, brief-traceable issues spanning varied severities and rule_keys, split across both reviewers. Both arms review the same delivered.diff against the same brief.md, so any difference in the {verdict, findings} is attributable to the orchestration substrate (prose Arm A vs Workflow Arm B), not the input.

Modeled on the proven determinism scenario in tests/fixtures/trekreview/review-run-A.md (same JWT-auth shape, same finding families), but here it is a real diff + brief pair the live reviewers read — not a synthetic pre-rendered review.

Seeded findings (expected ~5, varied)

# Issue Where Likely rule_key Severity Owner reviewer
1 /login returns 200 (not 401) on invalid credentials lib/handlers/login.mjs:17 UNIMPLEMENTED_CRITERION BLOCKER conformance (SC2)
2 verifyToken reads the verify algorithm from a request header lib/auth/jwt.mjs:19–20 SECURITY_INJECTION and/or NON_GOAL_VIOLATED BLOCKER correctness + conformance (NG1)
3 No test covers concurrent refresh (no test file in the diff) lib/auth/refresh.mjs (whole) MISSING_TEST MAJOR correctness (SC3)
4 Password check uses crypto.timingSafeEqual over plaintext, not bcrypt.compare per plan lib/handlers/login.mjs:13 PLAN_EXECUTE_DRIFT MAJOR conformance/correctness (Plan Step 4)
5 refreshStore I/O (get/delete/set) is unwrapped — backend outage bubbles unhandled lib/auth/refresh.mjs:10,16,20 MISSING_ERROR_HANDLING MINOR correctness (Constraint)

Issue #2 is intentionally dual-flaggable (a security defect AND an explicit Non-Goal violation) — it exercises the cross-reviewer overlap that the (file,line,rule_key) triplet-dedup and the coordinator must handle. Real LLM reviewers will vary exact line numbers and may surface extra latent issues (e.g. the user.passwordHash NPE when the email is unknown); that variance is the signal the bake-off measures, not noise to suppress.

Expected verdict (both arms): BLOCK (≥1 BLOCKER present).

Triage map (deterministic, pinned — passed to BOTH arms)

All three files are auth/security surface → deep-review:

lib/auth/jwt.mjs → deep-review
lib/handlers/login.mjs → deep-review
lib/auth/refresh.mjs → deep-review

How it's consumed

The bake-off pins Phases 1–4 (brief + diff + triage above) and passes them to both arms via paths (reviewer agents carry Read):

  • Arm A (prose): reviewers spawned FOREGROUND via the Agent tool with the prose trailing-json-block contract → validateReviewerOutput (NW1) → triplet-dedup → review-coordinator → {verdict, findings}. Harness: scripts/bakeoff-armA-merge.mjs.
  • Arm B (Workflow): scripts/trekreview-armB.workflow.mjs via the Workflow tool, args = { briefPath, diffPath, triage } (StructuredOutput-forced findings → JS triplet-dedup → coordinator verdict schema).

PRIMARY metric: fidelityDiffStructured (lib/review/fidelity-diff.mjs) — same verdict + equivalent finding set (IDs / severities / rule_keys), jaccard tolerance 0.7. Analysis harness: scripts/bakeoff-fidelity.mjs.