open/voyage
1
0
Fork 0
Code Issues Releases 5 Activity

fix(proevesett): a capitalised opposite verdict refuses the says-* graders

Browse source 
Both patterns now start at the beginning of the reply ((?<![\s\S]), which
holds with or without a multiline flag) and refuse the whole reply when it
carries the opposite verdict in capitals (FAIL/FAILED for says-pass,
PASS/PASSED for says-fail), unless negated. The opposite word in lower case
does not refuse: correct case-6 replies say "did not pass" and, once on
687710a, "The other checks passed".

Chosen: refuse only the capitalised opposite, because refusing it in any
case would fail correct replies again, the instability this change removes.

Measured on 22 case-8 and 14 case-6 recorded replies (PM re-measurement +
this session): old says-pass 19/22, new 22/22; old says-fail 13/14, new
14/14; each grader on the other case's replies 0.

Suite after git add: 1236 tests, 1234/0/2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 11:34:29 +02:00
commit 7f7161b2b2
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
2 changed files with 2 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

2
evals/review-validate-flags-bad-finding-id/graders/says-fail.md
View file

@ -1,4 +1,4 @@
---
type: regex
pattern: '(?<![Nn][Oo][Tt]\s)(?<![Nn][\x27’][Tt]\s)\b[Ff][Aa][Ii][Ll](?:[Ss]|[Ee][Dd])?\b'
pattern: '(?<![\s\S])(?![\s\S]*(?<![Nn][Oo][Tt]\s)\bPASS(?:ED)?\b)[\s\S]*(?<![Nn][Oo][Tt]\s)(?<![Nn][\x27’][Tt]\s)\b[Ff][Aa][Ii][Ll](?:[Ss]|[Ee][Dd])?\b'
---

2
evals/review-validate-passes-clean-review/graders/says-pass.md
View file

@ -1,4 +1,4 @@
---
type: regex
pattern: '(?<![Nn][Oo][Tt]\s)(?<![Nn][\x27’][Tt]\s)\b[Pp][Aa][Ss][Ss](?:[Ee][SsDd])?\b'
pattern: '(?<![\s\S])(?![\s\S]*(?<![Nn][Oo][Tt]\s)\bFAIL(?:ED)?\b)[\s\S]*(?<![Nn][Oo][Tt]\s)(?<![Nn][\x27’][Tt]\s)\b[Pp][Aa][Ss][Ss](?:[Ee][SsDd])?\b'
---