voyage/evals/review-validate-passes-clean-review
Kjell Tore Guttormsen 7f7161b2b2
fix(proevesett): a capitalised opposite verdict refuses the says-* graders
Both patterns now start at the beginning of the reply ((?<![\s\S]), which
holds with or without a multiline flag) and refuse the whole reply when it
carries the opposite verdict in capitals (FAIL/FAILED for says-pass,
PASS/PASSED for says-fail), unless negated. The opposite word in lower case
does not refuse: correct case-6 replies say "did not pass" and, once on
687710a, "The other checks passed".

Chosen: refuse only the capitalised opposite, because refusing it in any
case would fail correct replies again, the instability this change removes.

Measured on 22 case-8 and 14 case-6 recorded replies (PM re-measurement +
this session): old says-pass 19/22, new 22/22; old says-fail 13/14, new
14/14; each grader on the other case's replies 0.

Suite after git add: 1236 tests, 1234/0/2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 11:34:29 +02:00
..
graders fix(proevesett): a capitalised opposite verdict refuses the says-* graders 2026-09-23 11:34:29 +02:00
case.yaml test(evals): Voyage's fixed test set as a plugin eval suite, expectations written before the first run 2026-09-23 07:52:34 +02:00
prompt.md test(evals): Voyage's fixed test set as a plugin eval suite, expectations written before the first run 2026-09-23 07:52:34 +02:00
scaffold.sh test(evals): Voyage's fixed test set as a plugin eval suite, expectations written before the first run 2026-09-23 07:52:34 +02:00