test(proevesett): a case for the intent gate; case 5's error comes from code; graders see what they claim
New case plan-halts-without-intent-approval (expectation committed before
its first run): a 2.1 brief WITH phase_signals and no intent marker.
Graders: intent-approval.mjs --check ran; the trace carries the gate's own
JSON code for BRIEF_INTENT_NOT_APPROVED; no Agent; no plan.md.
review-requires-project: the arg parser now has a CLI that checks the
required flag and prints 'Error: --project <dir> is required.' + usage
(exit 1, code ARG_REQUIRED_MISSING). trekreview.md runs it with
$ARGUMENTS (it passed "$@", which the Bash tool never has, so the parser
never ran — 0 Bash calls in the PM's case-5 traces) and relays its stderr
instead of composing the line. The grader project-required (reply regex,
unstable on backticks) is replaced by parser-ran + names-missing-project
(trace, the parser's JSON code), the same form as names-rule.
Graders: no-error-code is bound to the validator's JSON output form and
covers every REVIEW_* code (WRONG_TYPE and VERSION_FORMAT were missing);
says-pass/says-fail reject a preceding NOT and a longer word; every case
with no-write also gets no-bash-write (redirect, tee, touch, cp, mv, rm,
sed -i in the Bash command). Checked on the PM's 10 recorded traces: no
false positive; known-positive/negative still split.
Red 348aa95 6/7 → green 7/7. Suite 1231: 1229 pass / 0 fail / 2 skip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
348aa95639
commit
3892a835dd
18 changed files with 127 additions and 14 deletions
|
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
type: tool_used
|
||||
tool: Bash
|
||||
input_match: 'intent-approval\.mjs --check'
|
||||
---
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
type: regex
|
||||
pattern: 'code\W{1,12}BRIEF_INTENT_NOT_APPROVED'
|
||||
target: trace
|
||||
---
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
type: tool_used
|
||||
tool: Agent
|
||||
min: 0
|
||||
max: 0
|
||||
---
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
type: file_exists
|
||||
path: "**/plan.md"
|
||||
exists: false
|
||||
---
|
||||
11
evals/plan-halts-without-intent-approval/prompt.md
Normal file
11
evals/plan-halts-without-intent-approval/prompt.md
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
---
|
||||
description: "/trekplan on a brief that passes the sequencing gate but carries no intent approval halts at the intent gate (veikart steg 1)."
|
||||
expected_outcome: "intent-approval.mjs --check runs and reports BRIEF_INTENT_NOT_APPROVED; /trekplan halts and names /trekbrief --approve; no Agent call (no exploration swarm), no plan.md."
|
||||
tags: [proevesett]
|
||||
runs: 1
|
||||
max_turns: 15
|
||||
timeout_seconds: 300
|
||||
allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
|
||||
---
|
||||
|
||||
/voyage:trekplan --project proj
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
type: tool_used
|
||||
tool: Bash
|
||||
input_match: '"command":\s*"(?:[^"\\]|\\.)*?(?:[^0-9&>=\-]>>?\s*(?!&|/dev/null)[^\s&|]|\b(?:tee|touch|cp|mv|rm)\s|\bsed\s+-i)'
|
||||
min: 0
|
||||
max: 0
|
||||
---
|
||||
7
evals/plan-requires-brief/graders/no-bash-write.md
Normal file
7
evals/plan-requires-brief/graders/no-bash-write.md
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
type: tool_used
|
||||
tool: Bash
|
||||
input_match: '"command":\s*"(?:[^"\\]|\\.)*?(?:[^0-9&>=\-]>>?\s*(?!&|/dev/null)[^\s&|]|\b(?:tee|touch|cp|mv|rm)\s|\bsed\s+-i)'
|
||||
min: 0
|
||||
max: 0
|
||||
---
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
type: regex
|
||||
pattern: 'code\W{1,12}ARG_REQUIRED_MISSING'
|
||||
target: trace
|
||||
---
|
||||
7
evals/review-requires-project/graders/no-bash-write.md
Normal file
7
evals/review-requires-project/graders/no-bash-write.md
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
type: tool_used
|
||||
tool: Bash
|
||||
input_match: '"command":\s*"(?:[^"\\]|\\.)*?(?:[^0-9&>=\-]>>?\s*(?!&|/dev/null)[^\s&|]|\b(?:tee|touch|cp|mv|rm)\s|\bsed\s+-i)'
|
||||
min: 0
|
||||
max: 0
|
||||
---
|
||||
5
evals/review-requires-project/graders/parser-ran.md
Normal file
5
evals/review-requires-project/graders/parser-ran.md
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
type: tool_used
|
||||
tool: Bash
|
||||
input_match: 'arg-parser\.mjs --command trekreview'
|
||||
---
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
---
|
||||
type: regex
|
||||
pattern: "--project <dir> is required"
|
||||
---
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
type: tool_used
|
||||
tool: Bash
|
||||
input_match: '"command":\s*"(?:[^"\\]|\\.)*?(?:[^0-9&>=\-]>>?\s*(?!&|/dev/null)[^\s&|]|\b(?:tee|touch|cp|mv|rm)\s|\bsed\s+-i)'
|
||||
min: 0
|
||||
max: 0
|
||||
---
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
---
|
||||
type: regex
|
||||
pattern: "FAIL"
|
||||
pattern: '(?<![Nn][Oo][Tt]\s)\bFAIL\b'
|
||||
---
|
||||
|
|
|
|||
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
type: tool_used
|
||||
tool: Bash
|
||||
input_match: '"command":\s*"(?:[^"\\]|\\.)*?(?:[^0-9&>=\-]>>?\s*(?!&|/dev/null)[^\s&|]|\b(?:tee|touch|cp|mv|rm)\s|\bsed\s+-i)'
|
||||
min: 0
|
||||
max: 0
|
||||
---
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
type: regex
|
||||
pattern: "REVIEW_(MISSING_FIELD|BAD_FINDING_ID|BAD_FINDINGS_TYPE|MISSING_SECTION|NOT_FOUND|READ_ERROR)"
|
||||
pattern: 'code\W{1,12}REVIEW_[A-Z_]+'
|
||||
target: trace
|
||||
match: not_contains
|
||||
---
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
---
|
||||
type: regex
|
||||
pattern: "PASS"
|
||||
pattern: '(?<![Nn][Oo][Tt]\s)\bPASS\b'
|
||||
---
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue