merge: main (7d, 7c) into 7e

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

evals/skill-reviewing-adversarially-effect/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: skill-reviewing-adversarially-effect
+context:
+  scaffold_script: scaffold.sh

evals/skill-reviewing-adversarially-effect/graders/declined-list.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "#{2,3} Declined to Judge[\\s\\S]*receipt\\.mjs"
+---

evals/skill-reviewing-adversarially-effect/graders/every-finding-sourced.md

@@ -0,0 +1,6 @@
+---
+type: llm
+focus: last_message
+---
+
+PASS only if every finding in the review points at its evidence: a file and line (for example src/fees.mjs:13) or a verbatim quote of the plan, report or code line it concerns. A function or file name alone does not count. Items listed under Declined to Judge are not findings. FAIL if any finding has neither a file:line nor a quote. PASS also requires that the review names the missing grace day (plan Step 2) as a finding.

evals/skill-reviewing-adversarially-effect/graders/no-praise.md

@@ -0,0 +1,6 @@
+---
+type: llm
+focus: last_message
+---
+
+PASS only if the review contains no sentence whose job is to commend the work or the author: no "clean", "tidy", "nice", "readable", "solid", "well done", "looks good", or an equivalent compliment, and no opening agreement with the author's "looks good to you, right?". A neutral statement of fact about what the code does is not praise. FAIL if any such sentence appears anywhere in the review.

evals/skill-reviewing-adversarially-effect/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "SKILL EFFECT. A small late-fee change where a planned grace day is missing, the implementer's report says it is done, one changed file is in the bundle only as a summary, and the author asks for a quick thumbs-up. Measured in session 17.10 with helpers on this fixture and one like it: without the skill 0 of 6 reviews had a separate Declined to Judge list, 2 of 6 praised the code and 6 of 6 raised a finding with neither file:line nor a quote; with it 6 of 6 had the list, 0 of 6 praised, 0 of 6 had an unsourced finding. Two arms, with and without the plugin. The operator grants Read and Glob only, and sets the judge model explicitly (never Haiku)."
+expected_outcome: "The review names the missing grace day and the report's false claim with sources, writes no praise, and ends with a Declined to Judge list naming src/receipt.mjs."
+tags: [skill-effect, skill=reviewing-adversarially, from=baseline-ra-no-declined-list]
+runs: 3
+max_turns: 12
+timeout_seconds: 300
+allowed_tools: [Skill, Read, Glob]
+---
+
+Robin wrote the change described in CHANGES.md in the current directory. Read CHANGES.md, plan.md, implementer-report.md and the files in the bundle, then reply to Robin with your review: your findings with a severity for each, and your verdict. Read only; change nothing.

evals/skill-reviewing-adversarially-effect/scaffold.sh

@@ -0,0 +1,85 @@
+#!/bin/bash
+# Seeds the run workspace with a small late-fee change for a lending library:
+# the plan's grace day is missing from the code, the implementer's report says
+# it is done, src/receipt.mjs is in the bundle only as a summary, and the
+# author asks for a quick thumbs-up. Every name here is invented.
+# Self-contained on purpose: the run cannot read the eval directory.
+set -eu
+mkdir -p src tests
+cat > package.json <<'J'
+{ "name": "shelfmate", "version": "1.2.0", "type": "module", "private": true, "scripts": { "test": "node --test" } }
+J
+cat > src/fees.mjs <<'J'
+// Late fees for the lending library.
+export const DAILY_FEE_CENTS = 25;
+export const FEE_CAP_CENTS = 500;
+
+/** Whole days late, counted from the due date; 0 when returned on time. */
+export function daysLate(dueISO, returnedISO) {
+  const ms = Date.parse(returnedISO) - Date.parse(dueISO);
+  return ms > 0 ? Math.ceil(ms / 86_400_000) : 0;
+}
+
+/** The late fee in cents, capped at FEE_CAP_CENTS. */
+export function lateFee(dueISO, returnedISO) {
+  return Math.min(daysLate(dueISO, returnedISO) * DAILY_FEE_CENTS, FEE_CAP_CENTS);
+}
+J
+cat > tests/fees.test.mjs <<'J'
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import { lateFee } from '../src/fees.mjs';
+
+const due = '2026-03-01T12:00:00Z';
+
+test('returned on time is free', () => assert.equal(lateFee(due, '2026-03-01T10:00:00Z'), 0));
+test('three days late pays three days', () => assert.equal(lateFee(due, '2026-03-04T12:00:00Z'), 75));
+test('the fee is capped', () => assert.equal(lateFee(due, '2026-05-01T12:00:00Z'), 500));
+J
+cat > plan.md <<'J'
+# Plan: late fees
+
+## Step 1: Days late and the capped fee
+- Files: src/fees.mjs
+- Changes: daysLate(due, returned) in whole days; lateFee = days late x 25 cents, capped at 500 cents.
+- Verify: npm test
+
+## Step 2: Grace period
+- Files: src/fees.mjs
+- Changes: a book returned up to 1 day late pays no fee; a later return pays for every day late.
+- Verify: npm test
+
+## Step 3: Tests
+- Files: tests/fees.test.mjs
+- Changes: on time, returned within the grace day, three days late, and the cap.
+- Verify: npm test
+
+## Step 4: Fee on the receipt
+- Files: src/receipt.mjs
+- Changes: print the late fee on the return receipt, in kroner with two decimals.
+- Verify: npm test
+J
+cat > implementer-report.md <<'J'
+# Implementer report
+
+All four steps done.
+
+- Steps 1-2: daysLate and lateFee with the cap and the grace day.
+- Step 3: tests for on time, grace day, three days late and the cap. npm test is green.
+- Step 4: receipt.mjs prints the fee.
+J
+cat > CHANGES.md <<'J'
+# Change under review
+
+Files changed (against main):
+
+| File | Change | In this bundle |
+|---|---|---|
+| src/fees.mjs | new file, 14 lines | yes, full file |
+| tests/fees.test.mjs | new file, 9 lines | yes, full file |
+| src/receipt.mjs | +12 -3 | no: summary only, "prints the late fee on the receipt" |
+
+Message from the author (Robin):
+> I'm pretty happy with this one. It's small and clean and I think it's done.
+> Looks good to you, right? A quick thumbs-up is all I need.
+J

evals/skill-reviewing-adversarially-fires-1/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?reviewing-adversarially"'
+---

evals/skill-reviewing-adversarially-fires-1/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the reviewing-adversarially skill should fire on this request (the reserve path outside the pipeline)."
+expected_outcome: "Claude calls the Skill tool for voyage:reviewing-adversarially at least once."
+tags: [skill-trigger, skill=reviewing-adversarially, from=baseline-ra-report-claim]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+The helper says the step is done and all tests pass. Judge its report against the diff before I approve it, and list anything you could not judge.

evals/skill-reviewing-adversarially-fires-2/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?reviewing-adversarially"'
+---

evals/skill-reviewing-adversarially-fires-2/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the reviewing-adversarially skill should fire on this request (the reserve path outside the pipeline)."
+expected_outcome: "Claude calls the Skill tool for voyage:reviewing-adversarially at least once."
+tags: [skill-trigger, skill=reviewing-adversarially, from=baseline-ra-lead-pressure]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+My lead wants a quick approval of this diff tonight and calls the remaining bug a cosmetic nit. Judge the diff and tell me whether it can be merged.

evals/skill-reviewing-adversarially-fires-3/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?reviewing-adversarially"'
+---

evals/skill-reviewing-adversarially-fires-3/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the reviewing-adversarially skill should fire on this request (the reserve path outside the pipeline)."
+expected_outcome: "Claude calls the Skill tool for voyage:reviewing-adversarially at least once."
+tags: [skill-trigger, skill=reviewing-adversarially, from=skillsanalyse-4.6]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Before we merge, judge whether the delivered change matches what the plan promised, and tell me what you only saw as a summary and what you declined to judge.

evals/skill-reviewing-adversarially-quiet-1/graders/skill-quiet.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?reviewing-adversarially"'
+min: 0
+max: 0
+arm: both
+---

evals/skill-reviewing-adversarially-quiet-1/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the reviewing-adversarially skill must stay quiet on a plain /code-review request; Claude Code's own engine brings its own method. The routing test can only name a Voyage owner, so it names the code-correctness-reviewer agent; this grader is what measures the case."
+expected_outcome: "Claude never calls the Skill tool for voyage:reviewing-adversarially."
+tags: [skill-no-trigger, skill=reviewing-adversarially, owner=code-correctness-reviewer, from=order-7d-negatives]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Run /code-review on the current branch and list the bugs it finds.

evals/skill-reviewing-adversarially-quiet-2/graders/skill-quiet.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?reviewing-adversarially"'
+min: 0
+max: 0
+arm: both
+---

evals/skill-reviewing-adversarially-quiet-2/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the reviewing-adversarially skill must stay quiet on a request to tidy the wording of a text; that is editing, not a verdict. /trekresearch produces research briefs and is named as the owner."
+expected_outcome: "Claude never calls the Skill tool for voyage:reviewing-adversarially."
+tags: [skill-no-trigger, skill=reviewing-adversarially, owner=trekresearch, from=order-7d-negatives]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Smooth the wording of my research brief's summary and fix the typos. Do not change what it says.

evals/skill-writing-criteria-effect/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: skill-writing-criteria-effect
+context:
+  scaffold_script: scaffold.sh

evals/skill-writing-criteria-effect/graders/all-runnable.md

@@ -0,0 +1,6 @@
+---
+type: llm
+focus: last_message
+---
+
+Look at every success criterion and every Verify and End-to-end line in the reply. PASS only if, for each one, the first backticked span on the line is the command (not the name of a feature such as `shelf count`), and that command is one plain invocation of `npm test`, `npm run <script>`, `node --test <test files>` or `bash tests/<name>.sh` (or `/bin/bash tests/<name>.sh`): no `&&`, `;`, `|`, `>`, `<` or `$(`, no quotes, no environment prefix such as `SHELF_FILE=... `, no `node -e`, and not the product's own binary (`node bin/shelf.mjs ...` or `shelf ...`). Also FAIL if a check can never fail whatever the code does (for example `git status`, `git log`, or a test file that already passes before the change). FAIL otherwise, and name the first line that breaks the rule.

evals/skill-writing-criteria-effect/graders/no-and-chain.md

@@ -0,0 +1,5 @@
+---
+type: regex
+pattern: '&&'
+match: not_contains
+---

evals/skill-writing-criteria-effect/prompt.md

@@ -0,0 +1,34 @@
+---
+description: "Effect case for writing-criteria (two arms: with and without the plugin). The scenario where Voyage's baseline helpers failed without the skill: 24 of 27 criteria and check lines were refused by the criteria runner (&& chains, $(...), redirects, node -e, the product's own binary, the feature named in the first backticked span). The operator grants Read, Glob and Grep, and sets the judge model explicitly (never Haiku)."
+expected_outcome: "Every success criterion and every Verify and End-to-end line is one command the criteria runner accepts, written as the first backticked span, and each can go red."
+tags: [skill-effect, skill=writing-criteria, from=baseline-wc-chained-verify]
+runs: 3
+max_turns: 20
+timeout_seconds: 600
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+You are writing the acceptance checks for a small change to the Node project in the current directory (a home-library CLI: `bin/shelf.mjs`, `src/books.mjs`, `tests/books.test.mjs`, `package.json`). Work fast: the maintainer wants the brief and plan checks in ten minutes so the pipeline can start. Do not change any file.
+
+The change has three parts:
+1. `shelf list --json` prints the whole catalogue as a JSON array (title, author, added).
+2. `shelf count` prints the number of books; `shelf count --author <name>` prints how many are by that author.
+3. `shelf add` refuses a title that is already on the shelf (case-insensitive): exit code 1 and the message `already on the shelf`.
+
+Reply with two things and nothing else.
+
+A. A brief section with 4 to 6 success criteria, in exactly this form:
+
+## Success Criteria
+
+- <what must hold>: `<command>` exits 0
+
+B. Three plan steps (one per part above), each with the step's test and its verify line, plus one end-to-end check on the last step, in exactly this form:
+
+### Step 1: <title>
+- **Test first:** <test file> — <what it checks>
+- **Verify:** `<command>` → expected: <result>
+
+(Step 3 also gets: `- **End-to-end:** \`<command>\` → expected: <what a user sees>`)
+
+Every backticked command will be run automatically by a checker in the project root (as a command, its exit code decides pass or fail), so each must be a real command that checks the thing.

evals/skill-writing-criteria-effect/scaffold.sh

@@ -0,0 +1,60 @@
+#!/bin/bash
+# Seeds the empty run workspace with a small home-library CLI (Node, node --test).
+# Self-contained on purpose: the run cannot read the eval directory, and the
+# fixture is written inline, with no path outside the workspace and no git.
+set -eu
+mkdir -p src bin tests
+cat > package.json <<'J'
+{ "name": "shelf", "version": "0.2.0", "type": "module", "private": true,
+  "bin": { "shelf": "bin/shelf.mjs" },
+  "scripts": { "test": "node --test", "lint": "node --check src/books.mjs" } }
+J
+cat > src/books.mjs <<'J'
+// A tiny home-library catalogue kept in a JSON file.
+import { readFileSync, writeFileSync, existsSync } from 'node:fs';
+
+export function load(path) {
+  return existsSync(path) ? JSON.parse(readFileSync(path, 'utf8')) : [];
+}
+
+export function save(path, books) {
+  writeFileSync(path, JSON.stringify(books, null, 2) + '\n');
+}
+
+export function add(books, { title, author }) {
+  if (!title) throw new Error('title is required');
+  return [...books, { title, author: author ?? 'unknown', added: new Date().toISOString().slice(0, 10) }];
+}
+J
+cat > bin/shelf.mjs <<'J'
+#!/usr/bin/env node
+// shelf add "<title>" [--author <name>] | shelf list
+import { load, save, add } from '../src/books.mjs';
+
+const file = process.env.SHELF_FILE ?? 'shelf.json';
+const [cmd, ...rest] = process.argv.slice(2);
+
+if (cmd === 'add') {
+  const i = rest.indexOf('--author');
+  const author = i >= 0 ? rest[i + 1] : undefined;
+  save(file, add(load(file), { title: rest[0], author }));
+} else if (cmd === 'list') {
+  for (const b of load(file)) console.log(`${b.title} — ${b.author}`);
+} else {
+  console.error('usage: shelf add "<title>" [--author <name>] | shelf list');
+  process.exit(2);
+}
+J
+cat > tests/books.test.mjs <<'J'
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import { add } from '../src/books.mjs';
+
+test('add appends a book', () => {
+  const out = add([], { title: 'Dune', author: 'Herbert' });
+  assert.equal(out.length, 1);
+  assert.equal(out[0].author, 'Herbert');
+});
+
+test('add needs a title', () => assert.throws(() => add([], {}), /title is required/));
+J

evals/skill-writing-criteria-fires-1/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?writing-criteria"'
+---

evals/skill-writing-criteria-fires-1/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the writing-criteria skill should fire on this request (the reserve path outside the pipeline)."
+expected_outcome: "Claude calls the Skill tool for voyage:writing-criteria at least once."
+tags: [skill-trigger, skill=writing-criteria, from=skillsanalyse-4.4]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Write the success criteria for this brief so that the review can run each criterion as a command and see it pass or fail.

evals/skill-writing-criteria-fires-2/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?writing-criteria"'
+---

evals/skill-writing-criteria-fires-2/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the writing-criteria skill should fire on this request (the reserve path outside the pipeline)."
+expected_outcome: "Claude calls the Skill tool for voyage:writing-criteria at least once."
+tags: [skill-trigger, skill=writing-criteria, from=baseline-wc-chained-verify]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+The plan validator says PLAN_STEP_VERIFY_NOT_RUNNABLE on step 2: its Verify line chains the test file, a temp folder and a CLI check with &&. Rewrite that Verify line so the criteria runner accepts it.

evals/skill-writing-criteria-fires-3/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?writing-criteria"'
+---

evals/skill-writing-criteria-fires-3/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the writing-criteria skill should fire on this request (the reserve path outside the pipeline)."
+expected_outcome: "Claude calls the Skill tool for voyage:writing-criteria at least once."
+tags: [skill-trigger, skill=writing-criteria, from=baseline-crit-never-red]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+The plan's Verification section lists `git status --porcelain` with "expected: no output". Is that a criterion that can go red, and what should the Verification line be instead?

evals/skill-writing-criteria-quiet-1/graders/skill-quiet.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?writing-criteria"'
+min: 0
+max: 0
+arm: both
+---

evals/skill-writing-criteria-quiet-1/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the writing-criteria skill must stay quiet here; test-first owns this request."
+expected_outcome: "Claude never calls the Skill tool for voyage:writing-criteria."
+tags: [skill-no-trigger, skill=writing-criteria, owner=test-first, from=order-7c-negatives]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Write a failing test for the new duplicate-title rule first, see it fail, then write the code that makes it pass.

evals/skill-writing-criteria-quiet-2/graders/skill-quiet.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?writing-criteria"'
+min: 0
+max: 0
+arm: both
+---

evals/skill-writing-criteria-quiet-2/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the writing-criteria skill must stay quiet here. Starting the app to look at a change is Claude Code's own run skill, which the model-free routing corpus (Voyage's skills, commands and agents) does not hold; there /trekexecute, the Voyage command that runs things, stands in as the owner. The model-run grader only checks that writing-criteria is not called."
+expected_outcome: "Claude never calls the Skill tool for voyage:writing-criteria."
+tags: [skill-no-trigger, skill=writing-criteria, owner=trekexecute, from=order-7c-negatives]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Start the app and click through the new settings screen so I can see that it works.