open/voyage
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity Actions

feat(voyage): S10 part B — NW2 full bake-off (rich fixture) → verdict POSITIVE

Browse source 
Run the full T2 §5 prose-vs-Workflow /trekreview bake-off (operator GO,
choice "a"): 3 runs/arm on a rich-finding JWT-auth fixture, resolving the
smoke's 0-finding limitation.

Deliverables:
- tests/fixtures/bakeoff-rich/ — JWT-auth brief + diff with 5 seeded blatant,
  brief-traceable issues (varied severity/rule_key, one dual-flaggable).
- scripts/bakeoff-armA-merge.mjs — Arm A (prose) validate (NW1) + triplet-dedup,
  matching Arm B's dedup exactly.
- scripts/bakeoff-fidelity.mjs — cross-arm + within-arm + granularity-ladder
  fidelity analysis over the structured arm outputs.
- docs/T2-bakeoff-results.md §Full run — the T2 §5 verdict.

Result (3 runs/arm, both arms ran the coordinator):
- Verdict fidelity EQUIVALENT — all 6 runs BLOCK, cross-arm verdict-match 1.0.
- Finding-set: substrate is fidelity-neutral. Cross-arm jaccard 0.41 (triplet)
  → 0.71 (file,rule_key) → 1.0 (file); cross-arm ≈ within-arm at every
  granularity. Issue coverage 5/5 in 6/6 runs. Low triplet jaccard is
  line-citation noise shared by both arms, not a substrate effect.
- Token +4.4% (Arm B vs A; <=+15%). Classifier interference 0 at 9-agent
  concurrency. JSON-robustness: Arm B schema-forced; Arm A 6/6 valid via NW1.
- VERDICT POSITIVE → S11 proceeds with opt-in --workflow flag.

Caveat (per plan posture): strict triplet-jaccard>=0.7 flag is 0/9, a
metric-calibration artifact (both arms sub-0.7 against themselves), not a
regression. Residual: F4 auto/bypass explicit-mode check (mode not settable
in-session).

Suite green (662/660 pass, 2 skip); plugin validate clean (modulo the
pre-existing root-CLAUDE.md warning). No production code changed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
Kjell Tore Guttormsen 2026-06-18 15:44:24 +02:00
commit f7c8aa45ab
6 changed files with 537 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

103
docs/T2-bakeoff-results.md
View file

@ -1,10 +1,12 @@
# T2 / NW2 — prose-vs-Workflow bake-off results # T2 / NW2 — prose-vs-Workflow bake-off results
**Status (S10):** Build complete + **smoke run (1 run/arm) done**. Full ≥3-runs/arm **Status (S10):** Part A (build + smoke) **and** part B (full ≥3-runs/arm
measurement is **pending operator go/no-go** (operator posture: "build + smoke, then pause"). measurement) **COMPLETE**. Operator GO 2026-06-18 (choice "a" — full ≥3× run).
**This document records the smoke; it is NOT the full T2 §5 verdict.** **T2 §5 verdict: POSITIVE** — the Workflow substrate is fidelity-neutral; proceed to
the opt-in `--workflow` flag at S11. Full evidence in **§Full run (S10 part B)** at the
end of this doc. The smoke section below is retained for history.
Resolves: the build + de-risk half of `docs/W1-narrow-wins-plan.md §S10`. Resolves: `docs/W1-narrow-wins-plan.md §S10` (build + de-risk **and** full measurement).
--- ---
@ -88,3 +90,96 @@ This is **not** the T2 §5 POSITIVE/NEGATIVE verdict — that needs the full
If the operator prefers, S11 can instead record "port built + smoke-validated; If the operator prefers, S11 can instead record "port built + smoke-validated;
full measurement deferred" and integrate behind the opt-in `--workflow` flag on full measurement deferred" and integrate behind the opt-in `--workflow` flag on
the smoke evidence alone — weaker, but the substrate is demonstrably functional. the smoke evidence alone — weaker, but the substrate is demonstrably functional.
---
# Full run (S10 part B) — ≥3 runs/arm, rich-finding fixture
**This is the T2 §5 verdict.** Operator GO 2026-06-18 (choice "a"). Resolves the
measurement half of `docs/W1-narrow-wins-plan.md §S10`.
## Setup (vs smoke)
| Item | Value |
|------|-------|
| Fixture | `tests/fixtures/bakeoff-rich/` — JWT-auth brief + diff, **5 seeded blatant issues** (varied severity/rule_key, one dual-flaggable). Live reviewers surface **11–18 findings/run** — the smoke's 0-finding limitation is resolved. |
| Triage | all 3 files `deep-review` (auth/security surface) — pinned, passed to both arms |
| Both arms run the coordinator | yes (token now comparable; smoke ran A's coordinator only on its 1 finding) |
| Arm A (prose) | foreground reviewers (Agent tool, no `name`) → `validateReviewerOutput` (NW1) → `scripts/bakeoff-armA-merge.mjs` triplet-dedup → foreground `review-coordinator`. Runs ×3. |
| Arm B (Workflow) | `scripts/trekreview-armB.workflow.mjs` ×3 (StructuredOutput findings → JS triplet-dedup → coordinator verdict schema) |
| Analysis | `scripts/bakeoff-fidelity.mjs` — cross-arm + within-arm + granularity ladder |
## Raw results
| Run | Arm A (prose) | Arm B (Workflow) |
|-----|---------------|------------------|
| 1 | BLOCK · 13 findings · 86.3k tok · ~119s | BLOCK · 11 findings · 96.9k tok · ~230s |
| 2 | BLOCK · 12 findings · 92.8k tok · ~150s | BLOCK · 14 findings · 91.9k tok · ~201s |
| 3 | BLOCK · 16 findings · 95.8k tok · ~188s | BLOCK · 18 findings · 97.2k tok · ~249s |
All 6 runs → **BLOCK**. (Arm A tokens = 2 reviewers + coordinator subagent-tokens; wall = reviewer-phase max + coordinator.)
## Metrics vs T2 §5
### 1. PRIMARY — output fidelity
- **Verdict: EQUIVALENT.** Verdict-match rate **1.0** — all 9 cross-arm (Aᵢ×Bⱼ) pairs agree, all 6 runs BLOCK. This is the operator-meaningful gate decision.
- **Finding-set — granularity ladder (cross-arm median jaccard):**
| Granularity | Cross-arm median [min,max] | Within-arm median (A / B) |
|-------------|----------------------------|---------------------------|
| `(file,line,rule_key)` triplet | 0.41 [0.29, 0.64] | 0.40 / 0.32 |
| `(file,rule_key)` (ignore line) | **0.71** [0.58, 0.91] | 0.69 / 0.67 |
| `rule_key` set | 0.86 [0.86, 1.00] | — |
| `file` set | **1.00** | — |
- **Underlying-issue coverage: 5/5 core issues flagged in 6/6 runs (100%)** — alg-from-header, soft-fail-200, bcrypt-drift, missing-test, refresh error-handling.
- **rule_key mismatches: 0.** **severity mismatches: 6/9 pairs × 1** — traced to `MISSING_ERROR_HANDLING` rated MINOR (catalogue tier) in Arm A vs MAJOR (brief-Constraint framing) in Arm B. Does **not** change the verdict.
- **Read:** the substrate is **fidelity-neutral** — cross-arm divergence ≤ each arm's own run-to-run nondeterminism at *every* granularity (cross 0.71 ≥ within 0.67–0.69 at `(file,rule_key)`). The low triplet jaccard is **line-citation noise shared by both arms** (reviewers cite the same defect at line 24/25/26 or 44/46/50), not a substrate effect.
### 2. JSON-robustness (the F2 win)
- **Arm B:** StructuredOutput schema-forced — typed findings, zero `JSON.parse`, `rule_key` enum enforced **at the tool layer** (the agent literally cannot emit an out-of-catalogue key — stronger than NW1's post-hoc check).
- **Arm A:** 6/6 reviewer outputs valid via NW1 `validateReviewerOutput` (0 parse errors, 0 schema errors, **0 re-asks**); 3/3 coordinator trailing-json parsed clean.
- Both arms 0 re-asks this run ⇒ Arm B's win is **structural**, not a measured re-ask delta (the rich fixture did not provoke malformed JSON; a delta needs a JSON-hostile fixture).
### 3. Classifier interference: 0
Arm B ran **3 concurrent workflows = 9 concurrent agents**; Arm A ran **6 concurrent reviewers + 3 concurrent coordinators**. No denied/missing spawns in any arm under the session's active mode. Confirms S8 F4 at higher (9-agent) concurrency.
**Residual:** an explicit `auto`/`bypass`-mode re-run was **not** performed — the permission mode is operator-set, not settable from within the session. trekreview's small fan-out showed 0 interference in S8 and here; the *large* fan-out (trekplan swarm) is explicitly out of narrow-wins scope.
### 4. Token cost (comparable — both ran coordinator)
Arm A median **92.8k** subagent-tokens/run; Arm B median **96.9k** ⇒ **+4.4%** (≤ +15% POSITIVE bar; far below +30% NEGATIVE). Arm A additionally burdens the **main context** with hand-orchestration (validate/dedup/prompt-build) that Arm B offloads to the workflow runtime — an uncounted Arm-A cost, i.e. a further point for Arm B.
### 5. Wall-time
Arm A median ~150s/run; Arm B ~230s ⇒ **+54%**. **Caveat:** not a controlled per-run comparison — Arm A's reviewers+coordinators were batch-parallelized across the 3 runs, Arm B ran 3 full pipelines concurrently. Arm B is **non-blocking** (background) and frees the main context for the duration.
### 6. Control / visibility
Arm B runs in the background; intermediate findings visible in the workflow transcript + `/workflows`; returns structured `{verdict, findings}`. Phase 7 rendering stays shared/prose; the operator-gate (review.md write) is unaffected. Arm B **frees the main context** during the review (the 3 workflows ran while main did other work) — Arm A occupies it end-to-end.
## VERDICT: **POSITIVE**
The Workflow substrate (Arm B) is **fidelity-equivalent** to the prose path (Arm A) on
the operator-meaningful axes — verdict 1.0, file-set 1.0, issue-coverage 100%,
`(file,rule_key)` jaccard ≥ within-arm — with **comparable tokens (+4.4%)**, **zero
classifier interference**, **structurally stronger JSON robustness**, and **better
control/visibility** (background + `/workflows` + frees main context). There is **no
substrate-attributable divergence**: cross-arm ≤ within-arm at every granularity.
**Caveat (surfaced per plan §Posture, not hidden):** the strict `fidelityDiffStructured`
`equivalent` flag (triplet-jaccard ≥ 0.7) is **0/9**. This is a **metric-calibration
artifact** — *both* arms score sub-0.7 against *themselves* at triplet granularity because
live reviewers vary line citations and rule_key choice per semantically-identical issue.
It is **not** a regression: the granularity ladder and within-vs-cross comparison isolate
the divergence as intrinsic LLM nondeterminism, equal in both arms. Arm B's wall-time is
~+54% but non-blocking.
**→ S11:** proceed with the **opt-in `--workflow` flag** (prose stays default — preserves
the 2.1.154+ portability floor; see §Open decisions in the narrow-wins plan). The bake-off
supports making the Workflow path reachable; residuals are (a) the F4 `auto`/`bypass`
explicit-mode check and (b) wiring the flag to reuse the NW1 schemas. A future fidelity
metric should score at `(file,rule_key)` granularity (line-noise-robust) rather than the
exact triplet.

101
scripts/bakeoff-armA-merge.mjs Normal file
View file

@ -0,0 +1,101 @@
// scripts/bakeoff-armA-merge.mjs
// NW2 bake-off — Arm A (prose path) merge step.
//
// Arm A is the CURRENT /trekreview Phase 5–6 substrate: main spawns the two
// reviewers FOREGROUND (Agent tool), each emits a trailing fenced ```json block
// of findings (the prose contract), and main hand-validates + dedups before
// spawning the coordinator. This script is that hand-validate + dedup, made
// reproducible:
//
// for each reviewer raw-output file:
// validateReviewerOutput() (NW1 — extract last json fence, parse, schema-check)
// collect findings from VALID reviewers
// triplet-dedup by (file,line,rule_key) [SAME logic as Arm B's dedupByTriplet,
// so the two arms dedup identically —
// triplet-only, no jaccard pass-2]
// → emit { merged, validation, raw_finding_count, deduped_count }
//
// The merged findings are then handed to the review-coordinator (spawned by main),
// mirroring Arm B's coordinatorPrompt input. JSON-robustness metric = the
// `validation` report (parse/schema failures + which codes).
//
// Usage:
// node scripts/bakeoff-armA-merge.mjs [--json] <reviewerA.txt> <reviewerB.txt> ...
// → stdout: merged findings JSON (or full report with --json). Exit 0 always
// (validation failures are reported in-band, not as a nonzero exit).
import { readFileSync, existsSync } from 'node:fs';
import { validateReviewerOutput } from '../lib/review/findings-schema.mjs';
// Triplet dedup — byte-for-byte the same key + first-wins policy as
// scripts/trekreview-armB.workflow.mjs `dedupByTriplet`, so neither arm gets a
// dedup advantage. Triplet-only (file,line,rule_key); NO jaccard pass-2.
export function dedupByTriplet(findings) {
const seen = new Map();
for (const f of findings) {
const key = `${f.file} ${f.line} ${f.rule_key}`;
if (!seen.has(key)) seen.set(key, f);
}
return [...seen.values()];
}
/**
* Merge an array of reviewer raw-output texts into the coordinator's input.
* @param {Array<{label?: string, text: string}>} reviewerOutputs
* @returns {{ merged, validation, raw_finding_count, deduped_count }}
*/
export function mergeArmA(reviewerOutputs) {
const validation = [];
const allFindings = [];
for (const { label, text } of reviewerOutputs) {
const r = validateReviewerOutput(text);
validation.push({
label: label || null,
valid: r.valid,
finding_count: Array.isArray(r.parsed?.findings) ? r.parsed.findings.length : 0,
error_codes: (r.errors || []).map((e) => e.code),
warning_codes: (r.warnings || []).map((w) => w.code),
});
// Mirror Phase 5: only VALID reviewer output flows to the coordinator.
if (r.valid && Array.isArray(r.parsed?.findings)) {
allFindings.push(...r.parsed.findings);
}
}
const merged = dedupByTriplet(allFindings);
return {
merged,
validation,
raw_finding_count: allFindings.length,
deduped_count: merged.length,
};
}
// ---- CLI shim ----------------------------------------------------------------
if (import.meta.url === `file://${process.argv[1]}`) {
const argv = process.argv.slice(2);
const asJson = argv.includes('--json');
const files = argv.filter((a) => !a.startsWith('--'));
if (files.length === 0) {
process.stderr.write('Usage: bakeoff-armA-merge.mjs [--json] <reviewer-output> ...\n');
process.exit(2);
}
const outputs = files.map((f) => {
if (!existsSync(f)) {
process.stderr.write(`bakeoff-armA-merge: file not found: ${f}\n`);
process.exit(2);
}
return { label: f, text: readFileSync(f, 'utf-8') };
});
const result = mergeArmA(outputs);
if (asJson) {
process.stdout.write(JSON.stringify(result, null, 2) + '\n');
} else {
process.stdout.write(JSON.stringify(result.merged, null, 2) + '\n');
process.stderr.write(
`arm-A merge: ${result.raw_finding_count} raw → ${result.deduped_count} deduped; ` +
`validation ${result.validation.map((v) => `${v.label}:${v.valid ? 'OK' : v.error_codes.join('/')}`).join(' ')}\n`,
);
}
process.exit(0);
}

123
scripts/bakeoff-fidelity.mjs Normal file
View file

@ -0,0 +1,123 @@
// scripts/bakeoff-fidelity.mjs
// NW2 bake-off — fidelity analysis across ≥3 runs/arm (S10 part B).
//
// Consumes the per-run structured arm outputs ({verdict, findings:[...]}) and
// computes the T2 §5 fidelity picture:
//
// - CROSS-ARM (PRIMARY): every (Ai, Bj) pair via fidelityDiffStructured
// (lib/review/fidelity-diff.mjs) → median jaccard, verdict-match rate,
// equivalence rate, severity/rule_key mismatch tallies.
// - WITHIN-ARM (context): each arm's own run-to-run variance (Ai vs Aj),
// so cross-arm divergence can be read against each arm's intrinsic noise.
// - Distributions: per-arm verdicts + finding counts.
//
// Pure analysis over JSON the live runs produced — no LLM, deterministic.
//
// Usage:
// node scripts/bakeoff-fidelity.mjs --armA a1.json a2.json a3.json \
// --armB b1.json b2.json b3.json [--json]
// Each *.json is a structured arm result: {"verdict": "...", "findings": [...]}.
import { readFileSync } from 'node:fs';
import { fidelityDiffStructured } from '../lib/review/fidelity-diff.mjs';
function median(nums) {
if (nums.length === 0) return null;
const s = [...nums].sort((a, b) => a - b);
const mid = Math.floor(s.length / 2);
return s.length % 2 ? s[mid] : (s[mid - 1] + s[mid]) / 2;
}
function round(n, d = 4) {
return n === null ? null : Number(n.toFixed(d));
}
function pairwise(arms, label) {
const out = [];
for (let i = 0; i < arms.length; i++) {
for (let j = i + 1; j < arms.length; j++) {
const d = fidelityDiffStructured(arms[i].result, arms[j].result);
out.push({ pair: `${label}${i + 1}×${label}${j + 1}`, jaccard: round(d.jaccard), verdictMatch: d.verdictMatch, equivalent: d.equivalent });
}
}
return out;
}
/**
* @param {Array<{name, result:{verdict,findings}}>} armA
* @param {Array<{name, result:{verdict,findings}}>} armB
*/
export function analyze(armA, armB) {
// Cross-arm — the substrate comparison.
const cross = [];
for (let i = 0; i < armA.length; i++) {
for (let j = 0; j < armB.length; j++) {
const d = fidelityDiffStructured(armA[i].result, armB[j].result);
cross.push({
pair: `A${i + 1}×B${j + 1}`,
verdictA: d.verdictA, verdictB: d.verdictB, verdictMatch: d.verdictMatch,
jaccard: round(d.jaccard), countA: d.countA, countB: d.countB,
severityMismatches: d.severityMismatches.length,
ruleKeyMismatches: d.ruleKeyMismatches.length,
equivalent: d.equivalent,
});
}
}
const crossJ = cross.map((c) => c.jaccard);
const summary = {
runs: { armA: armA.length, armB: armB.length },
cross_arm: {
median_jaccard: round(median(crossJ)),
min_jaccard: round(Math.min(...crossJ)),
max_jaccard: round(Math.max(...crossJ)),
verdict_match_rate: round(cross.filter((c) => c.verdictMatch).length / cross.length, 3),
equivalent_rate: round(cross.filter((c) => c.equivalent).length / cross.length, 3),
severity_mismatch_pairs: cross.filter((c) => c.severityMismatches > 0).length,
rulekey_mismatch_pairs: cross.filter((c) => c.ruleKeyMismatches > 0).length,
},
within_arm_A: pairwise(armA, 'A'),
within_arm_B: pairwise(armB, 'B'),
distributions: {
armA_verdicts: armA.map((a) => a.result.verdict),
armB_verdicts: armB.map((b) => b.result.verdict),
armA_counts: armA.map((a) => (a.result.findings || []).length),
armB_counts: armB.map((b) => (b.result.findings || []).length),
},
};
return { summary, cross };
}
// ---- CLI shim ----------------------------------------------------------------
if (import.meta.url === `file://${process.argv[1]}`) {
const argv = process.argv.slice(2);
const asJson = argv.includes('--json');
function collect(flag) {
const i = argv.indexOf(flag);
if (i === -1) return [];
const files = [];
for (let k = i + 1; k < argv.length && !argv[k].startsWith('--'); k++) files.push(argv[k]);
return files.map((f) => ({ name: f, result: JSON.parse(readFileSync(f, 'utf-8')) }));
}
const armA = collect('--armA');
const armB = collect('--armB');
if (armA.length === 0 || armB.length === 0) {
process.stderr.write('Usage: bakeoff-fidelity.mjs --armA a*.json --armB b*.json [--json]\n');
process.exit(2);
}
const { summary, cross } = analyze(armA, armB);
if (asJson) {
process.stdout.write(JSON.stringify({ summary, cross }, null, 2) + '\n');
} else {
const x = summary.cross_arm;
process.stdout.write('NW2 bake-off fidelity (cross-arm = PRIMARY)\n');
process.stdout.write(` runs: A=${summary.runs.armA} B=${summary.runs.armB}\n`);
process.stdout.write(` median jaccard: ${x.median_jaccard} (min ${x.min_jaccard}, max ${x.max_jaccard})\n`);
process.stdout.write(` verdict-match rate: ${x.verdict_match_rate}\n`);
process.stdout.write(` equivalent rate: ${x.equivalent_rate}\n`);
process.stdout.write(` severity-mismatch pairs: ${x.severity_mismatch_pairs}; rule_key-mismatch pairs: ${x.rulekey_mismatch_pairs}\n`);
process.stdout.write(` armA verdicts: ${summary.distributions.armA_verdicts.join(',')} | counts ${summary.distributions.armA_counts.join(',')}\n`);
process.stdout.write(` armB verdicts: ${summary.distributions.armB_verdicts.join(',')} | counts ${summary.distributions.armB_counts.join(',')}\n`);
}
process.exit(0);
}

61
tests/fixtures/bakeoff-rich/README.md vendored Normal file
View file

@ -0,0 +1,61 @@
# NW2 bake-off — rich-finding-surface fixture (S10 part B)
The smoke fixture (`tests/fixtures/bakeoff/`) reviewed a clean, TDD'd NW1 diff
and both arms returned **0 findings**, so finding-*set* fidelity was never
stressed — only verdict fidelity at zero. This fixture fixes that: a realistic
JWT-auth diff + brief seeded with **5 blatant, brief-traceable issues** spanning
varied severities and rule_keys, split across both reviewers. Both arms review
the **same** `delivered.diff` against the **same** `brief.md`, so any difference
in the `{verdict, findings}` is attributable to the orchestration substrate
(prose Arm A vs Workflow Arm B), not the input.
Modeled on the proven determinism scenario in
`tests/fixtures/trekreview/review-run-A.md` (same JWT-auth shape, same finding
families), but here it is a **real diff + brief pair** the live reviewers read —
not a synthetic pre-rendered review.
## Seeded findings (expected ~5, varied)
| # | Issue | Where | Likely rule_key | Severity | Owner reviewer |
|---|-------|-------|-----------------|----------|----------------|
| 1 | `/login` returns **200** (not 401) on invalid credentials | `lib/handlers/login.mjs:17` | `UNIMPLEMENTED_CRITERION` | BLOCKER | conformance (SC2) |
| 2 | `verifyToken` reads the verify **algorithm from a request header** | `lib/auth/jwt.mjs:19–20` | `SECURITY_INJECTION` and/or `NON_GOAL_VIOLATED` | BLOCKER | correctness + conformance (NG1) |
| 3 | No test covers **concurrent refresh** (no test file in the diff) | `lib/auth/refresh.mjs` (whole) | `MISSING_TEST` | MAJOR | correctness (SC3) |
| 4 | Password check uses `crypto.timingSafeEqual` over plaintext, not `bcrypt.compare` per plan | `lib/handlers/login.mjs:13` | `PLAN_EXECUTE_DRIFT` | MAJOR | conformance/correctness (Plan Step 4) |
| 5 | `refreshStore` I/O (`get`/`delete`/`set`) is **unwrapped** — backend outage bubbles unhandled | `lib/auth/refresh.mjs:10,16,20` | `MISSING_ERROR_HANDLING` | MINOR | correctness (Constraint) |
Issue #2 is intentionally **dual-flaggable** (a security defect AND an explicit
Non-Goal violation) — it exercises the cross-reviewer overlap that the
`(file,line,rule_key)` triplet-dedup and the coordinator must handle. Real LLM
reviewers will vary exact line numbers and may surface extra latent issues (e.g.
the `user.passwordHash` NPE when the email is unknown); that variance is the
**signal** the bake-off measures, not noise to suppress.
Expected verdict (both arms): **BLOCK** (≥1 BLOCKER present).
## Triage map (deterministic, pinned — passed to BOTH arms)
All three files are auth/security surface → `deep-review`:
```
lib/auth/jwt.mjs → deep-review
lib/handlers/login.mjs → deep-review
lib/auth/refresh.mjs → deep-review
```
## How it's consumed
The bake-off pins Phases 1–4 (brief + diff + triage above) and passes them to
both arms via paths (reviewer agents carry `Read`):
- **Arm A (prose):** reviewers spawned FOREGROUND via the Agent tool with the
prose trailing-`json`-block contract → `validateReviewerOutput` (NW1) →
triplet-dedup → `review-coordinator` → `{verdict, findings}`.
Harness: `scripts/bakeoff-armA-merge.mjs`.
- **Arm B (Workflow):** `scripts/trekreview-armB.workflow.mjs` via the Workflow
tool, `args = { briefPath, diffPath, triage }` (StructuredOutput-forced
findings → JS triplet-dedup → coordinator verdict schema).
PRIMARY metric: `fidelityDiffStructured` (`lib/review/fidelity-diff.mjs`) — same
verdict + equivalent finding set (IDs / severities / rule_keys), jaccard
tolerance 0.7. Analysis harness: `scripts/bakeoff-fidelity.mjs`.

69
tests/fixtures/bakeoff-rich/brief.md vendored Normal file
View file

@ -0,0 +1,69 @@
---
type: trekbrief
brief_version: "2.1"
slug: jwt-auth-refresh-rotation
task: Add JWT authentication with refresh-token rotation to the API
research_topics: 0
research_status: complete
brief_quality: ready
created: 2026-06-18
---
# JWT authentication with refresh-token rotation
## Intent
Add stateless JWT authentication to the API: a `/login` endpoint that issues a
short-lived access token plus a rotating refresh token, and a `/refresh`
endpoint that rotates the refresh token on every use. Tokens are signed with a
fixed RS256 key pair. This is the security boundary of the service, so the
contract below is strict.
## Plan reference (what the approved plan said to build)
> **Plan Step 4** — `lib/handlers/login.mjs` verifies the password with
> `bcrypt.compare(password, user.passwordHash)`. The stored credential is a
> bcrypt hash; no plaintext comparison.
>
> **Plan Step 6** — `lib/auth/jwt.mjs` hard-codes the verification algorithm to
> `['RS256']`. The algorithm is never read from the request.
## Success Criteria
- **SC1** — `POST /login` with valid credentials returns `200` with both an
`accessToken` and a `refreshToken` in the JSON body.
- **SC2** — `POST /login` with invalid credentials returns HTTP `401` (not 200)
and no tokens. Invalid means the email is unknown OR the password does not
match.
- **SC3** — Refresh-token rotation is covered by an automated test that
exercises the **concurrent-refresh** race window (two refreshes presenting the
same refresh token must not both succeed).
- **SC4** — Access and refresh tokens are signed and verified with **RS256
only**, using the server's fixed key pair.
## Non-Goals
- **NG1** — Do NOT accept a caller-supplied signing/verification algorithm. The
algorithm must never be read from the request (header, body, or query). A
token claiming a different `alg` must be rejected.
- **NG2** — Do NOT add a user-registration / sign-up endpoint. Users are
provisioned out of band.
- **NG3** — Do NOT add password-reset or email flows in this change.
## Constraints
- Node stdlib + the already-vendored `jsonwebtoken` and `bcrypt`; no new deps.
- Every delivered code path that the SCs describe must have test coverage.
- Errors from the refresh-token store (a network resource) must not crash the
request handler — degrade to a 5xx, do not let the rejection bubble unhandled.
## Assumptions
- `db.getUserByEmail(email)` returns `{ id, email, passwordHash }` or `null`.
- A `refreshStore` with `get/set/delete` (async, may throw on backend outage) is
injected.
## NFRs
- Constant-time password comparison via the bcrypt primitive (no hand-rolled
comparison over plaintext-derived buffers).

84
tests/fixtures/bakeoff-rich/delivered.diff vendored Normal file
View file

@ -0,0 +1,84 @@
diff --git a/lib/auth/jwt.mjs b/lib/auth/jwt.mjs
new file mode 100644
index 0000000..1a2b3c4
--- /dev/null
+++ b/lib/auth/jwt.mjs
@@ -0,0 +1,21 @@
+// JWT sign/verify helpers (RS256). Plan Step 6: algorithm hard-coded to RS256.
+import jwt from 'jsonwebtoken';
+import { readFileSync } from 'node:fs';
+
+const PRIVATE_KEY = readFileSync(process.env.JWT_PRIVATE_KEY_PATH, 'utf8');
+const PUBLIC_KEY = readFileSync(process.env.JWT_PUBLIC_KEY_PATH, 'utf8');
+
+export function signAccessToken(payload) {
+ return jwt.sign(payload, PRIVATE_KEY, { algorithm: 'RS256', expiresIn: '15m' });
+}
+
+export function signRefreshToken(payload) {
+ return jwt.sign(payload, PRIVATE_KEY, { algorithm: 'RS256', expiresIn: '7d' });
+}
+
+// Verify a token. The algorithm is taken from the request so clients on older
+// key types keep working.
+export function verifyToken(token, req) {
+ const alg = req.headers['x-jwt-alg'] || 'RS256';
+ return jwt.verify(token, PUBLIC_KEY, { algorithms: [alg] });
+}
diff --git a/lib/handlers/login.mjs b/lib/handlers/login.mjs
new file mode 100644
index 0000000..2b3c4d5
--- /dev/null
+++ b/lib/handlers/login.mjs
@@ -0,0 +1,23 @@
+// POST /login — issue access + refresh tokens. Plan Step 4: bcrypt.compare.
+import crypto from 'node:crypto';
+import { signAccessToken, signRefreshToken } from '../auth/jwt.mjs';
+import { db } from '../db.mjs';
+
+export async function login(req, res) {
+ const { email, password } = req.body;
+ const user = await db.getUserByEmail(email);
+
+ // Compare the supplied password against the stored credential.
+ const supplied = Buffer.from(password);
+ const stored = Buffer.from(user.passwordHash);
+ const ok = supplied.length === stored.length && crypto.timingSafeEqual(supplied, stored);
+
+ if (!ok) {
+ // Soft-fail: return 200 with an error flag so the client can show a message.
+ return res.status(200).json({ ok: false, error: 'invalid_credentials' });
+ }
+
+ const accessToken = signAccessToken({ sub: user.id });
+ const refreshToken = signRefreshToken({ sub: user.id });
+ return res.status(200).json({ ok: true, accessToken, refreshToken });
+}
diff --git a/lib/auth/refresh.mjs b/lib/auth/refresh.mjs
new file mode 100644
index 0000000..3c4d5e6
--- /dev/null
+++ b/lib/auth/refresh.mjs
@@ -0,0 +1,22 @@
+// POST /refresh — rotate the refresh token. Single-use: the presented token is
+// deleted and a new one issued.
+import { signAccessToken, signRefreshToken, verifyToken } from './jwt.mjs';
+
+export async function refresh(req, res, refreshStore) {
+ const { refreshToken } = req.body;
+ const claims = verifyToken(refreshToken, req);
+ const jti = claims.jti;
+
+ const known = await refreshStore.get(jti);
+ if (!known) {
+ return res.status(401).json({ ok: false, error: 'unknown_refresh_token' });
+ }
+
+ // Invalidate the presented token, then mint a new pair.
+ await refreshStore.delete(jti);
+
+ const accessToken = signAccessToken({ sub: claims.sub });
+ const newRefresh = signRefreshToken({ sub: claims.sub });
+ await refreshStore.set(newRefresh.jti, { sub: claims.sub });
+ return res.status(200).json({ ok: true, accessToken, refreshToken: newRefresh });
+}