open/voyage
1
0
Fork 0
Code Issues Releases 5 Activity

test(proevesett): red — case 4 passes with the sequencing gate's code switched off

Browse source 
Known-positive for case 4, in a scratch copy (never this tree):
brief-validator.mjs with the BRIEF_V51_MISSING_SIGNALS check made
'if (false)'. The case still passed 3 of 3: the child read the rule in
the command prose, halted, named phase_signals, and even found the
planted 'if (false)'. So the halt alone does not show the validator
fired, and no grader checked what expected_outcome already promised
('brief-validator ... reports BRIEF_V51_MISSING_SIGNALS').

New test: case 4 needs a names-code grader reading the validator's JSON
code from the trace (same form as the intent case). Red: 1 of 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 10:49:48 +02:00
commit 63b74bdaa4
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
1 changed files with 15 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

15
tests/lib/proevesett.test.mjs
View file

@ -131,6 +131,21 @@ test('plan-halts-without-phase-signals: a halt at the intent gate fails the case
assert.doesNotMatch(COMMAND_PROSE, re, 'the command prose must not trip the grader'); assert.doesNotMatch(COMMAND_PROSE, re, 'the command prose must not trip the grader');
}); });
// Known-positive, measured 2026-09-23: with the check in brief-validator.mjs
// disabled, the child still halted on the prose rule and named phase_signals
// (3 of 3 passed). The halt alone does not show that the validator fired.
test('plan-halts-without-phase-signals: the validator\'s own code must be in the trace', () => {
assert.ok(graders(SEQ_CASE).includes('names-code.md'), graders(SEQ_CASE).join(','));
const g = grader(`evals/${SEQ_CASE}/graders/names-code.md`);
assert.deepEqual([g.type, g.target], ['regex', 'trace']);
const re = new RegExp(g.pattern);
assert.match(inTrace({ valid: false, errors: [{ code: 'BRIEF_V51_MISSING_SIGNALS' }] }, 2), re);
assert.doesNotMatch(inTrace({ valid: true, errors: [], warnings: [] }, 2), re);
assert.doesNotMatch(inTrace({ valid: false, errors: [{ code: 'BRIEF_INTENT_NOT_APPROVED' }] }, 2), re);
assert.doesNotMatch(COMMAND_PROSE, re, 'the pattern must not be satisfiable by quoting the command prose');
assert.doesNotMatch(read('lib/validators/brief-validator.mjs'), re, 'nor by reading the validator source');
});
// ---- case 5: the missing-flag error comes from code ---- // ---- case 5: the missing-flag error comes from code ----
test('review-requires-project: the arg parser, run as trekreview.md spells it, prints the error (exit 1)', () => { test('review-requires-project: the arg parser, run as trekreview.md spells it, prints the error (exit 1)', () => {