open/voyage
1
0
Fork 0
Code Issues Releases 13 Activity
voyage/tests/fixtures/trekreview
History
Exact
Kjell Tore Guttormsen d88245c06e
 feat(review): the review is a gate - verdict in the file, CC engine, project memory 
Ordre 5 (wave 1, P11 + P5), review_version 1.1.

- K7: `verdict: ALLOW | WARN | BLOCK` is required in review.md (strict
  error, soft warning for 1.0 reviews). New CLI lib/review/verdict-gate.mjs:
  0 = ALLOW/WARN, 1 = BLOCK, 2 = could not decide (never green); a verdict
  greener than the file's own BLOCKERs or JSON block is exit 2. Realpath
  main guard (the old file:// guard exits 0 under a symlink - measured).
- No finding is struck for length: succinctness trims only MINOR/SUGGESTION
  text (catalogue tier), keeps BLOCKER/MAJOR whole, counts trims; the
  coordinator lists what it declined to judge.
- Claude Code's /code-review and /security-review run as the engine via the
  Skill tool (never --fix/--comment/--post/ultra); lib/review/engine-findings.mjs
  turns their findings into a reviewer payload; two catalogue keys
  (CODE_REVIEW_DEFECT, SECURITY_REVIEW_VULNERABILITY), catalogue 12 -> 14.
  An engine that did not run forbids ALLOW.
- Every BLOCKER is written as a numbered `### Step R<n>` with its source
  finding (strict error REVIEW_MISSING_REMEDIATION_STEP).
- P5: lib/review/recurrence.mjs - a finding class in two review.md files
  (current + git history, no new store) becomes one proposal, routed by the
  catalogue's memory_route; written only between Voyage markers in
  .claude/rules/voyage-review-lessons.md under a size budget, CLAUDE.md gets
  a pointer only; the operator approves every line (Phase 8.7).
- Workflow substrate: catalogue enum in step; a missing reviewer or engine
  forbids ALLOW there too (it returned ALLOW on zero findings before).
- Fixtures review-run-A/B said WARN over a BLOCKER; now BLOCK.

Borrowed and credited in code: Superpowers (declined-to-judge), ai-native-sdlc
(cap only nits; mistake/fix/check), Spec Kit (managed block; converge steps),
Pocock (retro routing), Kiro (verdict file + gate; evidence per lesson),
Cherny (engine -> brief control order; review -> memory; budget).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 19:45:52 +02:00
..
plan-with-source-findings.md feat(voyage)!: bulk content rewrite ultra -> voyage/trek prose [skip-docs] 2026-05-05 15:08:20 +02:00
README.md feat(voyage)!: bulk content rewrite ultra -> voyage/trek prose [skip-docs] 2026-05-05 15:08:20 +02:00
review-run-A.md feat(review): the review is a gate - verdict in the file, CC engine, project memory 2026-10-06 19:45:52 +02:00
review-run-B.md feat(review): the review is a gate - verdict in the file, CC engine, project memory 2026-10-06 19:45:52 +02:00

README.md

trekreview determinism fixtures

Synthetic fixtures for the Jaccard-similarity determinism test in tests/lib/review-determinism.test.mjs.

What's here

  • review-run-A.md — synthetic review with 5 findings on a fictional JWT auth task
  • review-run-B.md — same fictional task, "re-reviewed" — same 5 findings as A plus 1 extra (a placeholder TODO that A missed)

Construction

Run A's finding-IDs are a strict subset of Run B's (A ⊂ B), so:

  • Intersection: |A ∩ B| = 5
  • Union: |A ∪ B| = 6
  • Jaccard: 5 / 6 = 0.833… (above the 0.70 SC4 threshold from brief.md)

Each ID is a real 40-char SHA1 computed via lib/parsers/finding-id.mjs: sha1(file:line:rule_key). Don't hand-edit the IDs — recompute via the helper if you change the underlying (file, line, rule_key) triplet, or both fixtures will fall out of sync.

Why synthetic for v1.0

Hand-curated for v1.0. Edit JSON IDs directly to test new Jaccard scenarios. Real-LLM determinism measurement is deferred to v1.1 once /trekreview has produced enough real outputs to capture as fixtures.

These fixtures prove the Jaccard PIPELINE works given a known input — they do NOT measure real LLM determinism. The brief's SC4 (Jaccard ≥ 0.70 across two runs) is verified at the pipeline level today; capturing real LLM runs to verify the model-level claim is open work for v1.1.

Adding a new scenario

  1. Pick (file, line, rule_key) triplets — rule_key must be one of the 12 keys in lib/review/rule-catalogue.mjs.
  2. Compute IDs via: 
    node -e "import('./lib/parsers/finding-id.mjs').then(({computeFindingId}) => console.log(computeFindingId('lib/foo.mjs', 42, 'SECURITY_INJECTION')))"
  3. Add the IDs to findings: block-style YAML in frontmatter and to ### <id> subsections in the body.
  4. Run node lib/validators/review-validator.mjs --json tests/fixtures/trekreview/review-run-X.md to confirm the fixture validates.
  5. Update tests/lib/review-determinism.test.mjs if you want a new assertion.