Kjell Tore Guttormsen
d88245c06e
Ordre 5 (wave 1, P11 + P5), review_version 1.1. - K7: `verdict: ALLOW | WARN | BLOCK` is required in review.md (strict error, soft warning for 1.0 reviews). New CLI lib/review/verdict-gate.mjs: 0 = ALLOW/WARN, 1 = BLOCK, 2 = could not decide (never green); a verdict greener than the file's own BLOCKERs or JSON block is exit 2. Realpath main guard (the old file:// guard exits 0 under a symlink - measured). - No finding is struck for length: succinctness trims only MINOR/SUGGESTION text (catalogue tier), keeps BLOCKER/MAJOR whole, counts trims; the coordinator lists what it declined to judge. - Claude Code's /code-review and /security-review run as the engine via the Skill tool (never --fix/--comment/--post/ultra); lib/review/engine-findings.mjs turns their findings into a reviewer payload; two catalogue keys (CODE_REVIEW_DEFECT, SECURITY_REVIEW_VULNERABILITY), catalogue 12 -> 14. An engine that did not run forbids ALLOW. - Every BLOCKER is written as a numbered `### Step R<n>` with its source finding (strict error REVIEW_MISSING_REMEDIATION_STEP). - P5: lib/review/recurrence.mjs - a finding class in two review.md files (current + git history, no new store) becomes one proposal, routed by the catalogue's memory_route; written only between Voyage markers in .claude/rules/voyage-review-lessons.md under a size budget, CLAUDE.md gets a pointer only; the operator approves every line (Phase 8.7). - Workflow substrate: catalogue enum in step; a missing reviewer or engine forbids ALLOW there too (it returned ALLOW on zero findings before). - Fixtures review-run-A/B said WARN over a BLOCKER; now BLOCK. Borrowed and credited in code: Superpowers (declined-to-judge), ai-native-sdlc (cap only nits; mistake/fix/check), Spec Kit (managed block; converge steps), Pocock (retro routing), Kiro (verdict file + gate; evidence per lesson), Cherny (engine -> brief control order; review -> memory; budget). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| plan-with-source-findings.md | ||
| README.md | ||
| review-run-A.md | ||
| review-run-B.md | ||
trekreview determinism fixtures
Synthetic fixtures for the Jaccard-similarity determinism test in
tests/lib/review-determinism.test.mjs.
What's here
review-run-A.md— synthetic review with 5 findings on a fictional JWT auth taskreview-run-B.md— same fictional task, "re-reviewed" — same 5 findings as A plus 1 extra (a placeholder TODO that A missed)
Construction
Run A's finding-IDs are a strict subset of Run B's (A ⊂ B), so:
- Intersection:
|A ∩ B| = 5 - Union:
|A ∪ B| = 6 - Jaccard:
5 / 6 = 0.833…(above the 0.70 SC4 threshold frombrief.md)
Each ID is a real 40-char SHA1 computed via lib/parsers/finding-id.mjs:
sha1(file:line:rule_key). Don't hand-edit the IDs — recompute via the helper if
you change the underlying (file, line, rule_key) triplet, or both fixtures will
fall out of sync.
Why synthetic for v1.0
Hand-curated for v1.0. Edit JSON IDs directly to test new Jaccard scenarios.
Real-LLM determinism measurement is deferred to v1.1 once /trekreview
has produced enough real outputs to capture as fixtures.
These fixtures prove the Jaccard PIPELINE works given a known input — they do NOT measure real LLM determinism. The brief's SC4 (Jaccard ≥ 0.70 across two runs) is verified at the pipeline level today; capturing real LLM runs to verify the model-level claim is open work for v1.1.
Adding a new scenario
- Pick
(file, line, rule_key)triplets —rule_keymust be one of the 12 keys inlib/review/rule-catalogue.mjs. - Compute IDs via:
node -e "import('./lib/parsers/finding-id.mjs').then(({computeFindingId}) => console.log(computeFindingId('lib/foo.mjs', 42, 'SECURITY_INJECTION')))" - Add the IDs to
findings:block-style YAML in frontmatter and to### <id>subsections in the body. - Run
node lib/validators/review-validator.mjs --json tests/fixtures/trekreview/review-run-X.mdto confirm the fixture validates. - Update
tests/lib/review-determinism.test.mjsif you want a new assertion.