Commit graph

1 commit

Author SHA1 Message Date
0eb18f39e6
docs(proevesett): the eval suite's first run is 6 of 7, and the known-positive falls on a planted fault
docs/proevesett.md records what the suite covers and what it does not, how to
run it, the verbatim pass/fail definition with its source, and the first run:
6 of 7 cases (graders 25 of 26), 7 claude -p children, 54 s.

The red case, review-requires-project, lost only its text grader: the child
paraphrased a guard that lives in prose. A diagnostic re-run passed, so it is
unstable, not broken, and it stays red - loosening the pattern after the run
would fit the grader to the result.

Known-positive shown in a scratch copy: with the finding-ID check disabled in
review-validator.mjs, review-validate-flags-bad-finding-id drops from 1.0 to 0.6.

Also recorded, measured by probe: ${CLAUDE_PLUGIN_ROOT} is substituted in the
expanded command text but empty in the child's Bash env, and the plugin
directory is readable from the child's sandbox.

Veikart step 4 uses the suite as its yardstick: a rule change that lowers n stops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 07:56:32 +02:00