open/voyage
1
0
Fork 0
Code Issues Releases 5 Activity

test(proevesett): says-pass/says-fail must take the verdict in any letter case (red)

Browse source 
The PM re-measurement of ea66a98 had case 8 pass 10 of 12. Both failures were
correct replies that gave the verdict only in lower case ("passed"), and
\bPASS(?:ED)?\b is case-sensitive. The test pins lower-case pass/passes/passed
and fail/fails/failed as verdicts, keeps "not passed", "doesn't pass", "failed"
refused by says-pass, and checks that each case's recorded reply does not
satisfy the other case's grader.

Red on ea66a98: 10 of 11 in tests/lib/proevesett.test.mjs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 11:27:16 +02:00
commit aee9b42dda
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
1 changed files with 29 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

29
tests/lib/proevesett.test.mjs
View file

@ -210,6 +210,35 @@ test('says-pass / says-fail are not raw substrings: a negation does not pass the
for (const s of ['It did not FAILED.', 'FAILEDX']) assert.doesNotMatch(s, fail);
});
test('says-pass / says-fail take the verdict in any letter case, and still refuse the wrong one', () => {
const pass = new RegExp(grader('evals/review-validate-passes-clean-review/graders/says-pass.md').pattern);
const fail = new RegExp(grader('evals/review-validate-flags-bad-finding-id/graders/says-fail.md').pattern);
// Case-8 replies that failed says-pass in the PM re-measurement of ea66a98
// (2 of 12): correct, validator exit 0, but the verdict only in lower case.
const lowerPass = [
'`proj/review.md` passed the strict schema check, with 0 errors and 0 warnings.',
'`proj/review.md` passed the strict schema validator: 0 errors and 0 warnings.',
'`proj/review.md` passes the strict schema check, with no errors and no warnings.',
'Validation passed.',
];
for (const s of lowerPass) assert.match(s, pass);
for (const s of ['review.md failed the strict schema check.', 'Validation fails: 1 error.', 'Result: failed']) assert.match(s, fail);
// The wrong verdict, in any case, is still refused.
for (const s of ['not passed', 'Not passed.', 'The review did not pass.', 'It does not pass the strict validator.',
'It doesn’t pass.', "It doesn't pass.", 'failed', 'Validation failed.', 'bypass', 'passage']) {
assert.doesNotMatch(s, pass, s);
}
for (const s of ['not failed', 'It did not fail.', "It didn't fail.", 'passed', 'failover', 'failure']) {
assert.doesNotMatch(s, fail, s);
}
// Each case's real reply (as recorded) must not satisfy the other case's
// grader: case 6 replies say "did not pass", case 8 replies never say fail.
const case6 = '**FAIL**: `proj/review.md` did not pass the strict schema check (the validator exited with code 1).';
const case6b = '**FAIL:** `proj/review.md` does not pass the strict validator. The validator exited with code 1.';
for (const s of [case6, case6b]) { assert.doesNotMatch(s, pass, s); assert.match(s, fail, s); }
for (const s of lowerPass) assert.doesNotMatch(s, fail, s);
});
test('every case with a no-write grader also has no-bash-write, and it sees a write through Bash', () => {
const withNoWrite = cases().filter((c) => graders(c).includes('no-write.md'));
assert.ok(withNoWrite.length >= 5, withNoWrite.join(','));