voyage/evals/review-validate-passes-clean-review
Kjell Tore Guttormsen 687710a975
fix(proevesett): says-pass/says-fail read the verdict in any letter case
Case 8 failed 2 of 12 in the PM re-measurement of ea66a98 on correct replies
that said "passed" without PASS. The patterns now spell each letter as a
class ([Pp][Aa][Ss][Ss]...) instead of relying on a regex flag, since a grader
file carries only `pattern`. The verb forms pass/passes/passed and
fail/fails/failed are verdicts; a preceding "not " or "n't " still refuses.

Measured on the PM's recorded replies (distinct runs): old says-pass 10/12 on
case 8 (reproduces the finding), new 12/12; says-pass on case 6 replies 0/8
both, says-fail 8/8 on case 6 and 0/12 on case 8 both.

Suite after git add: 1235 tests, 1233/0/2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 11:29:39 +02:00
..
graders fix(proevesett): says-pass/says-fail read the verdict in any letter case 2026-09-23 11:29:39 +02:00
case.yaml test(evals): Voyage's fixed test set as a plugin eval suite, expectations written before the first run 2026-09-23 07:52:34 +02:00
prompt.md test(evals): Voyage's fixed test set as a plugin eval suite, expectations written before the first run 2026-09-23 07:52:34 +02:00
scaffold.sh test(evals): Voyage's fixed test set as a plugin eval suite, expectations written before the first run 2026-09-23 07:52:34 +02:00