Kjell Tore Guttormsen 869bf318d2 feat(voyage): S10 — NW2 part A (Workflow port + fidelity harness + smoke)

Build the prose-vs-Workflow bake-off machinery for /trekreview Phase 5-6 and
run a 1-run/arm smoke to de-risk before the full measurement (operator posture:
build + smoke, then pause for go/no-go on the full >=3-runs/arm run).

New:
- lib/review/fidelity-diff.mjs (+ tests) — the PRIMARY metric: parse two
  review.md (or two structured arm outputs) and compare verdict + jaccard over
  (file,line,rule_key)-IDs + per-finding severity/rule_key. Reuses jaccard +
  frontmatter + NW1 findings-schema + finding-id. fidelityDiffStructured avoids
  rendering review.md per run.
- scripts/trekreview-armB.workflow.mjs — Arm B: Phase 5-6 as a Workflow
  (parallel([conformance, correctness]) schema-forced -> JS dedup-by-triplet ->
  agent(review-coordinator) verdict schema). Path-based input via args (reviewers
  carry Read). Inlines dedup + the 12-key rule_key enum (scripts have no imports).
- tests/fixtures/bakeoff/ — committable fixture: real diff of b149538 (NW1) +
  brief reconstructed from plan S9. Both arms review the same pinned input.
- docs/T2-bakeoff-results.md — smoke results + verdict + go/no-go recommendation.

Smoke result: SMOKE PASS. Arm B runs the full pipeline (3 agents) with ZERO
classifier interference; fidelity EQUIVALENT to Arm A at the verdict level
(both ALLOW; jaccard 1.0). Caveat: the clean TDD'd fixture yielded ~0 findings,
so finding-SET fidelity was not stressed (only verdict fidelity proven). A
reviewer-level divergence appeared (Arm B raised 1 raw finding, coordinator
filtered it; Arm A raised 0) — to be quantified in the full run on a
richer-finding-surface fixture. NOT the T2 §5 POSITIVE/NEGATIVE verdict.

Suite 647 -> 662 (660 pass / 2 skip / 0 fail; +15 fidelity-diff). claude plugin
validate clean (known root-CLAUDE.md warning only). Plan: docs/W1-narrow-wins-plan.md S10.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB

2026-06-18 14:40:19 +02:00

3.4 KiB

Raw Blame History

type	brief_version	slug	task	research_topics	research_status	brief_quality	created
trekbrief	2.1	nw1-reviewer-output-schema	Codify the /trekreview Phase 5 reviewer-output JSON as a validated schema contract	0	complete	ready	2026-06-18

NW1 — reviewer-output schema contract

Intent

/trekreview Phase 5 launches two reviewer agents (brief-conformance-reviewer, code-correctness-reviewer) that each emit a trailing fenced json block of findings. The main context used to "collect the last fenced json block and JSON.parse() it; on parse error, ask the agent to re-emit" — a fragile contract (trekreview.md:202–204) that catches parse failure but not schema failure: a reviewer can emit valid JSON with a missing rule_key, a bad severity, or a non-numeric line, and it flows unchecked into the coordinator's dedup-by-(file,line,rule_key) triplet.

Goal

Codify the reviewer-output JSON as a validated schema. Main validates each reviewer's output against the schema (not merely parses it) and re-asks on schema failure as well as parse failure, with bounded retries. The load-bearing fields the downstream dedup triplet + verdict depend on — file, rule_key, severity, line — are hard errors; descriptive fields and unknown keys are tolerated (forward-compatible). No Workflow dependency — the substrate stays prose (this is the ungated, ship-regardless win).

Success Criteria

SC1 — A findings-schema validator accepts a well-formed findings array and rejects each malformation (missing rule_key, unknown rule_key, bad severity enum, non-numeric line, missing file) with a stable error code.
SC2 — lib/review/findings-schema.mjs implements the schema + validator, reusing the lib/util/result.mjs issue/result shape and the lib/review/rule-catalogue.mjs RULE_KEYS / SEVERITY_VALUES.
SC3 — commands/trekreview.md Phase 5 prose replaces the JSON.parse-only contract with "validate each reviewer's JSON against the findings schema; on schema failure re-ask for conforming JSON (bounded retries N=2); never feed unvalidated findings to the coordinator."
SC4 — node --test tests/lib/findings-schema.test.mjs is green; the full node --test suite still passes; claude plugin validate is clean (modulo the pre-existing root-CLAUDE.md warning).

Non-Goals

NG1 — Do NOT introduce a Workflow-tool dependency. NW1 is the prose-only tier; the substrate stays prose.
NG2 — Do NOT change the review.md output schema or lib/validators/review-validator.mjs. This is the reviewer-output contract (Phase 5 input), not the review.md artifact contract (Phase 7 output).
NG3 — Do NOT touch NW2 (Workflow port) or NW3 (synthesis-agent) scope.

Constraints

Zero runtime dependencies — Node stdlib only (project invariant).
3-layer module pattern (Content → Raw-text → CLI shim) mirroring the other lib/ validators.
TDD (Iron Law): failing tests first, then minimal code to pass.

Assumptions

Reviewers continue to emit the documented JSON shape ({reviewer, findings:[…]}) with the per-finding fields shown in the agent specs.
The 12-key rule catalogue and 4-tier severity enum are the contract surface for rule_key / severity validation.

NFRs

The schema validator adds no measurable latency to Phase 5 (pure in-process validation).
Error codes are stable strings so a bounded re-ask can quote them.

3.4 KiB Raw Blame History Unescape Escape