Kjell Tore Guttormsen
c909466c99
Sed-pipeline (16 patterns, longest-match-first) sweeper residuelle ultra*-treff i prose, command-narrativ, agent-prompts, hook-kommentarer, doc-prosa. Pipeline-utvidelser fra V4-prompten: - BSD-syntax [[:<:]]ultra[[:>:]] istedenfor \bultra\b (BSD sed mangler \b) - 6 compound-patterns for ultraplan/ultraexecute/ultraresearch/ultrabrief/ ultrareview/ultracontinue uten -local-suffiks - ultra*-stats glob -> trek*-stats glob - Linje-eksklusjon redusert til ultra-cc-architect (Q8); session-state- eksklusjonen var over-protektiv - File-eksklusjon utvidet til settings.json, package.json, plugin.json, hele .claude/-treet (gitignored + V5-territorium) Q8-undantak holdt: architecture-discovery.mjs + project-discovery.mjs urort. Filnavn-konvensjon holdt: .session-state.local.json + *.local.* preservert. Manuell narrative-fix: tests/lib/agent-frontmatter.test.mjs linje 10 mangled "/ultra*-local" til "/voyage*-local" (ingen slik kommando finnes); korrigert til "/trek*". Residualer utenfor scope (V5 handterer): package.json + .claude-plugin/ plugin.json (Step 12-14 versjons-bump). .claude/* er gitignored spec-historikk med tilsiktet BEFORE/AFTER-narrativ. Part of voyage-rebrand session 3 (Wave 4 / Step 10). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
47 lines
2 KiB
Markdown
47 lines
2 KiB
Markdown
# trekreview determinism fixtures
|
||
|
||
Synthetic fixtures for the Jaccard-similarity determinism test in
|
||
`tests/lib/review-determinism.test.mjs`.
|
||
|
||
## What's here
|
||
|
||
- `review-run-A.md` — synthetic review with 5 findings on a fictional JWT auth task
|
||
- `review-run-B.md` — same fictional task, "re-reviewed" — same 5 findings as A plus 1 extra (a placeholder TODO that A missed)
|
||
|
||
## Construction
|
||
|
||
Run A's finding-IDs are a strict subset of Run B's (`A ⊂ B`), so:
|
||
|
||
- Intersection: `|A ∩ B| = 5`
|
||
- Union: `|A ∪ B| = 6`
|
||
- Jaccard: `5 / 6 = 0.833…` (above the 0.70 SC4 threshold from `brief.md`)
|
||
|
||
Each ID is a real 40-char SHA1 computed via `lib/parsers/finding-id.mjs`:
|
||
`sha1(file:line:rule_key)`. Don't hand-edit the IDs — recompute via the helper if
|
||
you change the underlying `(file, line, rule_key)` triplet, or both fixtures will
|
||
fall out of sync.
|
||
|
||
## Why synthetic for v1.0
|
||
|
||
Hand-curated for v1.0. Edit JSON IDs directly to test new Jaccard scenarios.
|
||
Real-LLM determinism measurement is deferred to v1.1 once `/trekreview`
|
||
has produced enough real outputs to capture as fixtures.
|
||
|
||
These fixtures prove the Jaccard PIPELINE works given a known input — they do
|
||
NOT measure real LLM determinism. The brief's SC4 (Jaccard ≥ 0.70 across two
|
||
runs) is verified at the pipeline level today; capturing real LLM runs to
|
||
verify the model-level claim is open work for v1.1.
|
||
|
||
## Adding a new scenario
|
||
|
||
1. Pick `(file, line, rule_key)` triplets — `rule_key` must be one of the 12
|
||
keys in `lib/review/rule-catalogue.mjs`.
|
||
2. Compute IDs via:
|
||
```bash
|
||
node -e "import('./lib/parsers/finding-id.mjs').then(({computeFindingId}) => console.log(computeFindingId('lib/foo.mjs', 42, 'SECURITY_INJECTION')))"
|
||
```
|
||
3. Add the IDs to `findings:` block-style YAML in frontmatter and to `### <id>`
|
||
subsections in the body.
|
||
4. Run `node lib/validators/review-validator.mjs --json tests/fixtures/trekreview/review-run-X.md`
|
||
to confirm the fixture validates.
|
||
5. Update `tests/lib/review-determinism.test.mjs` if you want a new assertion.
|