open/voyage
1
0
Fork 0
Code Issues Releases 13 Activity

feat(routing): every eval case names its event; the P10 eval gate is exempted with its reason (A1)

Browse source 
P10's model-free part (wave 4, order A1, decision 63). The routing test now
checks every case in evals/ for from=, not only the skill-tagged ones (R1):
a case with no skill tag is not routed, only checked, and the report gives
both counts ("45 cases checked for from=, 37 routed"). The eight test-set
cases get from=proevesett-eval-suite (d311f3b) or
from=proevesett-intent-gate (3892a83), the commits that added them.

The release gate that reads the last test-set run is written into the
runbook as exempted with its reason (it waits for the fixed test set to run
again) and still says where it goes (scripts/release-check.mjs).

review-630's prototype-tag stub gets from=test: under R1 an unrouted case is
still checked for from=, and the stub tests that a prototype name is never
routed, which stands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-10-19 18:15:00 +02:00
commit cf08df9d55
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
17 changed files with 145 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

12
docs/release-runbook.md
View file

@ -225,14 +225,18 @@ changes introduced by commits that are not ancestors of the previously
reverted merge." Revert the revert commit instead, and release it as a new
version.
## Not built yet: the eval gate (P10)
## Exempted: the eval gate (P10)
The local gate before a release that reads the date and score of the last run
of the fixed test set and compares it with the last change in the
configuration paths (commands, agents, skills, hooks, and a changed model alias
counts as a change). It waits for the test set to run again; a gate that can
never fire is not built. When it comes, it is one more check in
`scripts/release-check.mjs`, before the door check, and one more step in § 1.
counts as a change). It is exempted from the release, with a reason: it waits
for the fixed test set to run again, and a gate that can never fire is not
built. P10's model-free part is in place: every case in `evals/` names the
event it came from (`from=`), and the routing test (`lib/skills/routing.mjs`,
run by `npm test`) fails a case without it. When the gate comes, it is one more
check in `scripts/release-check.mjs`, before the door check, and one more step
in § 1.
## Credits

4
docs/skills-authoring.md
View file

@ -72,6 +72,10 @@ tags: [skill-effect, skill=test-first, from=baseline-tfd-crash-red]
```
`from=` names the event the case came from: a measured failure, a review finding, an analysis.
It is required on every case in `evals/`, not only the skill cases: a case of the fixed test set
(`tags: [proevesett, from=proevesett-eval-suite]`) has no skill tag, so it is not routed, but the
routing test still fails it without `from=`. Its report gives both counts, cases checked for
`from=` and cases routed.
- A **skill-trigger** case has one grader: `type: tool_used`, `tool: Skill`, with an `input_match`
that accepts `voyage:<name>` and the bare name.