Kjell Tore Guttormsen
869bf318d2
Build the prose-vs-Workflow bake-off machinery for /trekreview Phase 5-6 and
run a 1-run/arm smoke to de-risk before the full measurement (operator posture:
build + smoke, then pause for go/no-go on the full >=3-runs/arm run).
New:
- lib/review/fidelity-diff.mjs (+ tests) — the PRIMARY metric: parse two
review.md (or two structured arm outputs) and compare verdict + jaccard over
(file,line,rule_key)-IDs + per-finding severity/rule_key. Reuses jaccard +
frontmatter + NW1 findings-schema + finding-id. fidelityDiffStructured avoids
rendering review.md per run.
- scripts/trekreview-armB.workflow.mjs — Arm B: Phase 5-6 as a Workflow
(parallel([conformance, correctness]) schema-forced -> JS dedup-by-triplet ->
agent(review-coordinator) verdict schema). Path-based input via args (reviewers
carry Read). Inlines dedup + the 12-key rule_key enum (scripts have no imports).
- tests/fixtures/bakeoff/ — committable fixture: real diff of b149538 (NW1) +
brief reconstructed from plan S9. Both arms review the same pinned input.
- docs/T2-bakeoff-results.md — smoke results + verdict + go/no-go recommendation.
Smoke result: SMOKE PASS. Arm B runs the full pipeline (3 agents) with ZERO
classifier interference; fidelity EQUIVALENT to Arm A at the verdict level
(both ALLOW; jaccard 1.0). Caveat: the clean TDD'd fixture yielded ~0 findings,
so finding-SET fidelity was not stressed (only verdict fidelity proven). A
reviewer-level divergence appeared (Arm B raised 1 raw finding, coordinator
filtered it; Arm A raised 0) — to be quantified in the full run on a
richer-finding-surface fixture. NOT the T2 §5 POSITIVE/NEGATIVE verdict.
Suite 647 -> 662 (660 pass / 2 skip / 0 fail; +15 fidelity-diff). claude plugin
validate clean (known root-CLAUDE.md warning only). Plan: docs/W1-narrow-wins-plan.md S10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
78 lines
3.4 KiB
Markdown
78 lines
3.4 KiB
Markdown
---
|
||
type: trekbrief
|
||
brief_version: "2.1"
|
||
slug: nw1-reviewer-output-schema
|
||
task: Codify the /trekreview Phase 5 reviewer-output JSON as a validated schema contract
|
||
research_topics: 0
|
||
research_status: complete
|
||
brief_quality: ready
|
||
created: 2026-06-18
|
||
---
|
||
|
||
# NW1 — reviewer-output schema contract
|
||
|
||
## Intent
|
||
|
||
`/trekreview` Phase 5 launches two reviewer agents (`brief-conformance-reviewer`,
|
||
`code-correctness-reviewer`) that each emit a trailing fenced `json` block of
|
||
findings. The main context used to "collect the last fenced json block and
|
||
`JSON.parse()` it; on parse error, ask the agent to re-emit" — a fragile
|
||
contract (`trekreview.md:202–204`) that catches *parse* failure but not
|
||
*schema* failure: a reviewer can emit valid JSON with a missing `rule_key`, a
|
||
bad `severity`, or a non-numeric `line`, and it flows unchecked into the
|
||
coordinator's dedup-by-`(file,line,rule_key)` triplet.
|
||
|
||
## Goal
|
||
|
||
Codify the reviewer-output JSON as a validated schema. Main validates each
|
||
reviewer's output against the schema (not merely parses it) and re-asks on
|
||
schema failure as well as parse failure, with bounded retries. The load-bearing
|
||
fields the downstream dedup triplet + verdict depend on — `file`, `rule_key`,
|
||
`severity`, `line` — are hard errors; descriptive fields and unknown keys are
|
||
tolerated (forward-compatible). No Workflow dependency — the substrate stays
|
||
prose (this is the ungated, ship-regardless win).
|
||
|
||
## Success Criteria
|
||
|
||
- **SC1** — A findings-schema validator accepts a well-formed findings array and
|
||
rejects each malformation (missing `rule_key`, unknown `rule_key`, bad
|
||
`severity` enum, non-numeric `line`, missing `file`) with a stable error code.
|
||
- **SC2** — `lib/review/findings-schema.mjs` implements the schema + validator,
|
||
reusing the `lib/util/result.mjs` issue/result shape and the
|
||
`lib/review/rule-catalogue.mjs` `RULE_KEYS` / `SEVERITY_VALUES`.
|
||
- **SC3** — `commands/trekreview.md` Phase 5 prose replaces the
|
||
`JSON.parse`-only contract with "validate each reviewer's JSON against the
|
||
findings schema; on schema failure re-ask for conforming JSON (bounded
|
||
retries N=2); never feed unvalidated findings to the coordinator."
|
||
- **SC4** — `node --test tests/lib/findings-schema.test.mjs` is green; the full
|
||
`node --test` suite still passes; `claude plugin validate` is clean (modulo the
|
||
pre-existing root-CLAUDE.md warning).
|
||
|
||
## Non-Goals
|
||
|
||
- **NG1** — Do NOT introduce a Workflow-tool dependency. NW1 is the prose-only
|
||
tier; the substrate stays prose.
|
||
- **NG2** — Do NOT change the `review.md` output schema or
|
||
`lib/validators/review-validator.mjs`. This is the *reviewer-output* contract
|
||
(Phase 5 input), not the review.md artifact contract (Phase 7 output).
|
||
- **NG3** — Do NOT touch NW2 (Workflow port) or NW3 (synthesis-agent) scope.
|
||
|
||
## Constraints
|
||
|
||
- Zero runtime dependencies — Node stdlib only (project invariant).
|
||
- 3-layer module pattern (Content → Raw-text → CLI shim) mirroring the other
|
||
`lib/` validators.
|
||
- TDD (Iron Law): failing tests first, then minimal code to pass.
|
||
|
||
## Assumptions
|
||
|
||
- Reviewers continue to emit the documented JSON shape (`{reviewer, findings:[…]}`)
|
||
with the per-finding fields shown in the agent specs.
|
||
- The 12-key rule catalogue and 4-tier severity enum are the contract surface
|
||
for `rule_key` / `severity` validation.
|
||
|
||
## NFRs
|
||
|
||
- The schema validator adds no measurable latency to Phase 5 (pure in-process
|
||
validation).
|
||
- Error codes are stable strings so a bounded re-ask can quote them.
|