feat(skills): voyage:annotating, the operator-annotation loop as one skill (order B)
The three producing commands name voyage:annotating where they build the page; the description of the page and the freehand revision left them. The skill: the person writes every note, Fix/Change edit only the quoted text, a Question is answered and changes nothing, the approval an edit costs is said before the edit, the page is rebuilt. Research documents on request (decision 62). Red baseline 3 runs per scenario without and with the skill; six eval cases; 11 pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
1b613421eb
commit
843897ae33
25 changed files with 550 additions and 46 deletions
4
evals/skill-annotating-effect/case.yaml
Normal file
4
evals/skill-annotating-effect/case.yaml
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
schema_version: "1.1"
|
||||
name: skill-annotating-effect
|
||||
context:
|
||||
scaffold_script: scaffold.sh
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
---
|
||||
type: llm
|
||||
focus:
|
||||
source: file
|
||||
path: .claude/voyage/projects/2026-10-19-potluck-diet/plan.md
|
||||
---
|
||||
|
||||
The plan started with these Files lines: Step 1 `src/signups.mjs`; Step 2 `src/cli.mjs`; Step 3 `src/cli.mjs`; Step 4 `README.md`. Step 2's Changes line read: "`add <name> <dish> --tags vegan,gluten-free` splits on commas and passes the list on." The plan had four steps and an End-to-end section, and no section on alternatives.
|
||||
|
||||
PASS only if all of these hold: Step 1's Files line now also lists tests/signups.test.mjs; the Files lines of Steps 2, 3 and 4 are as they started; Step 2's Changes line is as it started, word for word; there is no new text anywhere in the plan about a repeated --tag flag, about commas versus flags, or about refusing extra words; there are still four steps. FAIL if any of these does not hold.
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
---
|
||||
|
||||
PASS only if the reply answers the person's question: it compares the comma form (--tags a,b) with a repeated flag (--tag a --tag b) and says which is easier for the organiser to type, and why. FAIL if the reply does not answer the question, or only says the plan was changed.
|
||||
24
evals/skill-annotating-effect/prompt.md
Normal file
24
evals/skill-annotating-effect/prompt.md
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
---
|
||||
description: "Effect case for annotating (two arms: with and without the plugin). The scenario where Voyage's baseline helpers failed without the skill (measured in session 19.10, Opus 5.5 high, on this project): the person pasted one Fix and one Question back from the annotation page, and 3 of 3 answered the Question but also changed the plan because of it (a new row under alternatives, a new rule in Step 2), and 3 of 3 applied the Fix to steps the note did not name. The operator grants Read, Glob, Grep, Edit and Bash, and sets the judge model explicitly (never Haiku)."
|
||||
expected_outcome: "Step 1 lists tests/signups.test.mjs under Files; nothing else in plan.md changes; the reply answers why commas and not a repeated flag."
|
||||
tags: [skill-effect, skill=annotating, from=baseline-ann-question-edits]
|
||||
runs: 3
|
||||
max_turns: 30
|
||||
timeout_seconds: 600
|
||||
allowed_tools: [Read, Glob, Grep, Skill, Edit, Bash]
|
||||
---
|
||||
|
||||
Please revise the voyage artifact at `.claude/voyage/projects/2026-10-19-potluck-diet/plan.md` with the operator annotations below.
|
||||
Each annotation has an intent — **Fix** (something is wrong / fix it), **Change** (change wording/content),
|
||||
or **Question** (operator question — clarify or answer). The quote shows what the operator anchored to.
|
||||
Treat the operator notes as authoritative direction.
|
||||
|
||||
## Annotations (2 total)
|
||||
|
||||
### 1. [Fix] Section: Step 1: Store tags with a sign-up
|
||||
Quote: «Files: `src/signups.mjs`»
|
||||
Comment: Step 1 also edits tests/signups.test.mjs, list it under Files.
|
||||
|
||||
### 2. [Question] Section: Step 2: Parse tags on the command line
|
||||
Quote: «splits on commas and passes the list on»
|
||||
Comment: Why commas and not a repeated --tag flag? Which one is easier for the organiser to type?
|
||||
88
evals/skill-annotating-effect/scaffold.sh
Executable file
88
evals/skill-annotating-effect/scaffold.sh
Executable file
|
|
@ -0,0 +1,88 @@
|
|||
#!/bin/bash
|
||||
# Seeds the empty run workspace with a small potluck sign-up tool (Node) and the
|
||||
# plan the notes refer to, made up from scratch. Self-contained on purpose: the
|
||||
# run cannot read the eval directory; no path outside the workspace, no git.
|
||||
# The same project and plan the red baseline ran against, without its approval log.
|
||||
set -eu
|
||||
P=.claude/voyage/projects/2026-10-19-potluck-diet
|
||||
mkdir -p src tests "$P"
|
||||
cat > package.json <<'J'
|
||||
{ "name": "potluck", "version": "0.3.0", "type": "module", "private": true,
|
||||
"scripts": { "test": "node --test" } }
|
||||
J
|
||||
cat > README.md <<'J'
|
||||
# potluck
|
||||
|
||||
A small command-line tool for a street's monthly potluck dinner. Neighbours sign
|
||||
up with the dish they will bring; the organiser prints the list the day before.
|
||||
|
||||
node src/cli.mjs add <name> <dish>
|
||||
node src/cli.mjs list
|
||||
J
|
||||
cat > src/signups.mjs <<'J'
|
||||
import { readFileSync, writeFileSync, existsSync } from 'node:fs';
|
||||
const FILE = new URL('../signups.json', import.meta.url);
|
||||
export const load = () => (existsSync(FILE) ? JSON.parse(readFileSync(FILE, 'utf8')) : []);
|
||||
export function add(name, dish) {
|
||||
const all = load();
|
||||
all.push({ name, dish });
|
||||
writeFileSync(FILE, JSON.stringify(all, null, 2) + '\n');
|
||||
}
|
||||
J
|
||||
cat > src/cli.mjs <<'J'
|
||||
import { add, load } from './signups.mjs';
|
||||
const [cmd, name, dish] = process.argv.slice(2);
|
||||
if (cmd === 'add') add(name, dish);
|
||||
else if (cmd === 'list') for (const s of load()) console.log(`${s.name}\t${s.dish}`);
|
||||
else { console.error('usage: add|list'); process.exit(2); }
|
||||
J
|
||||
cat > tests/signups.test.mjs <<'J'
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
test('signups module loads', async () => {
|
||||
const m = await import('../src/signups.mjs');
|
||||
assert.equal(typeof m.add, 'function');
|
||||
});
|
||||
J
|
||||
cat > "$P/plan.md" <<'J'
|
||||
---
|
||||
plan_version: "1.9"
|
||||
brief: brief.md
|
||||
---
|
||||
|
||||
# Plan: Dietary tags on potluck sign-ups
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1: Store tags with a sign-up
|
||||
- Files: `src/signups.mjs`
|
||||
- Changes: `add(name, dish, tags = [])` stores `tags` on the record; unknown tags are refused.
|
||||
- Test first:
|
||||
- File: `tests/signups.test.mjs`
|
||||
- Verify: `npm test`
|
||||
|
||||
### Step 2: Parse tags on the command line
|
||||
- Files: `src/cli.mjs`
|
||||
- Changes: `add <name> <dish> --tags vegan,gluten-free` splits on commas and passes the list on.
|
||||
- Test first:
|
||||
- File: `tests/cli.test.mjs`
|
||||
- Verify: `npm test`
|
||||
|
||||
### Step 3: Print tags in the list
|
||||
- Files: `src/cli.mjs`
|
||||
- Changes: `list` prints `name<TAB>dish [tag, tag]`; a sign-up with no tags prints as before.
|
||||
- Test first:
|
||||
- File: `tests/cli.test.mjs`
|
||||
- Verify: `npm test`
|
||||
|
||||
### Step 4: Document the flag
|
||||
- Files: `README.md`
|
||||
- Changes: add `--tags` and the four allowed tags to the usage block.
|
||||
- Test first:
|
||||
- File: `tests/readme.test.mjs`
|
||||
- Verify: `npm test`
|
||||
|
||||
## End-to-end
|
||||
|
||||
- End-to-end: `npm test`
|
||||
J
|
||||
Loading…
Add table
Add a link
Reference in a new issue