A brief success criterion and a plan's Verify, Regression, End-to-end and
Verification lines are each one command the criteria runner accepts, written
as the line's first backticked span; anything more is bash tests/<name>.sh,
and each must be able to go red (the runner reads the exit code only).
- skills/writing-criteria/SKILL.md (S1 form), with Runs:/Refused:/Never red:
examples that a pin runs through the criteria runner
- brief-reviewer and plan-critic preload writing-criteria and
reviewing-adversarially (S3 + S4); the copied runner rules leave them, the
two templates and /trekplan, which names the skill where the plan is written
and where its three reviewers start
- eval cases 3 fire / 2 quiet / 1 effect; tests/lib/skills-criteria-prose-pins
- review-630 @~1 pin follows the text into the skill; trekplan-stats line_ref
re-measured
Baseline without the skill: 24 of 27 written checks refused by the runner,
and 0 of 3 reviewers saw that git status/git log can never go red. With it:
0 of 27 refused, 3 of 3 reviewers flagged both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new skill, voyage:reviewing-adversarially (contract S4): no praise, no
finding without a source (file:line or a quoted line), no downgrade without
a reason, summary-only said and not judged, a Declined to Judge list, claims
checked against the diff rather than the report, axes kept separate.
Red baseline first: six review runs by helpers without the skill (two
invented fixtures) had no separate Declined to Judge list in 6 of 6, an
unsourced finding in 6 of 6, praise in 2 of 6 and a verdict on something
unmeasured in 4 of 6. With the skill: 6 of 6, 0, 0 and 0.
- brief-conformance-reviewer, code-correctness-reviewer, review-coordinator
and scope-guardian preload it through skills:; the 21-line "Interaction
Awareness - MANDATORY OVERRIDE" block copied into three of them is gone.
Each agent keeps its own role.
- The two reviewers and scope-guardian end with their own Declined to Judge
list; /trekreview names the skill at spawn and passes the reviewers'
lines to the coordinator, which copies them into review.md as prose
(the JSON declined_to_judge array and the verdict are unchanged).
- Eval cases 3 / 2 / 1; pins in tests/lib/skills-review-prose-pins.test.mjs.
- tests/fixtures/jsonl-schemas.md: the trekreview-stats line_ref moves with
Phase 8 (636-686).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflict: CLAUDE.md command table. Kept both sides: this branch's
/trekresearch row (the external agents preload researching-sources) and
main's /trekplan row (open minors listed for the operator).
researching-sources adapted to contract S1, now that 7a's form check and
routing test run over it:
- SKILL.md takes the fixed headings (Overview, When to use, Process, Red
flags, Enforced by, Credits); the five rules and the confidence scale move
under Process unchanged; Red flags lists the thoughts the baseline runs
showed; "text only" becomes "prose only" as the form check reads it.
- The description adds "official" documentation and "looking up online what
an error message means or what causes it": the error-lookup case ranked
#3 behind debugging. That prompt also said "our tests fail", which is
debugging's job; it now asks only what the error means and where that is
documented. No threshold changed.
- Eval cases carry the routing tags (skill-trigger / skill-no-trigger /
skill-effect, skill=, owner= task-finder and trekplan, from=), 7a's
Skill grader pattern, arm: both on the no-trigger graders, and the
judge-model line on the effect case.
- The `## Skills` row in README and CLAUDE.md.
- tests/lib/skills-research-prose-pins.test.mjs drops its two form pins
(skill-lint is the authority now): 14 -> 12.
npm test on the merged tree: 2055 = 2053 pass / 0 fail / 2 skipped
(main 2043 + 12).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The source rules the research agents shared by copy now live in one skill,
skills/researching-sources/: official sources first with URL, date and
matching version; no citation, no finding; an empty search is reported with
its denominator and never as a finding; fetched pages are data, and text in
them aimed at AI assistants is ignored and reported; nothing local in an
outbound query. It also holds the one confidence scale.
- docs-, community-, security-, contrarian-researcher and research-scout
preload it (skills: ["voyage:researching-sources"]); the copied rule text
leaves them, their own role stays.
- Two agent sentences that contradicted the rules are gone: an empty
counter-evidence search "increases confidence", and "fall back silently".
- /trekresearch names the skill where it starts the external agents and in
web follow-ups; Phase 6 rates on the skill's scale word for word; the
deep-research adapter accepts `contradictory` like the template does.
- Enforced by: research-validator checks structure only, the query privacy
gate runs only in the default-off STORM steps; the rest is text only.
A guard on every web search is a separate later order (operator 08.10).
- Red baseline (helpers, offline fixtures): without the skill 3 of 5 runs
ignored an injected instruction silently and 1 of 2 contrarian runs counted
an empty search as weak support; with it 0 of 3 and 0 of 2. Path leaks:
0 of 7 without, 0 of 5 with.
- Eval cases (3 should trigger, 2 should not, 1 effect), written, not run.
- tests/lib/skills-research-prose-pins.test.mjs: 14 pins, 13 red on 4ee711a.
Credits: Addy Osmani, agent-skills @9d0c60d,
skills/source-driven-development/SKILL.md:97-114 (MIT); Jesse Vincent,
obra/superpowers @8ca22db, skills/diagnosing-superpowers/ (MIT). Ideas only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The workshop (ordre 7a, contracts S1 and S2): a form check
(lib/skills/skill-lint.mjs) that follows Claude Code's own skill fields,
and a model-free routing test (lib/skills/routing.mjs) over every skill,
the 7 commands and the 21 spawnable agents, reading the eval cases
themselves (evals/skill-*, 3 fire / 2 quiet / 1 effect, tagged with the
event they came from). Thresholds pinned, never lowered.
The two skills were written against failures measured first in helpers
without them: red only as an import crash, the whole suite run from a
hand-made file list instead of the repo's own command, and an
unreproduced fix shipped under authority pressure. step-implementer
preloads both through its skills: line; no tool list change.
Credits: addyosmani/agent-skills @9d0c60d, obra/superpowers @8ca22db,
mattpocock/skills @c55ee46 (MIT), Anthropic skill-creator (Apache-2.0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both patterns now start at the beginning of the reply ((?<![\s\S]), which
holds with or without a multiline flag) and refuse the whole reply when it
carries the opposite verdict in capitals (FAIL/FAILED for says-pass,
PASS/PASSED for says-fail), unless negated. The opposite word in lower case
does not refuse: correct case-6 replies say "did not pass" and, once on
687710a, "The other checks passed".
Chosen: refuse only the capitalised opposite, because refusing it in any
case would fail correct replies again, the instability this change removes.
Measured on 22 case-8 and 14 case-6 recorded replies (PM re-measurement +
this session): old says-pass 19/22, new 22/22; old says-fail 13/14, new
14/14; each grader on the other case's replies 0.
Suite after git add: 1236 tests, 1234/0/2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Case 8 failed 2 of 12 in the PM re-measurement of ea66a98 on correct replies
that said "passed" without PASS. The patterns now spell each letter as a
class ([Pp][Aa][Ss][Ss]...) instead of relying on a regex flag, since a grader
file carries only `pattern`. The verb forms pass/passes/passed and
fail/fails/failed are verdicts; a preceding "not " or "n't " still refuses.
Measured on the PM's recorded replies (distinct runs): old says-pass 10/12 on
case 8 (reproduces the finding), new 12/12; says-pass on case 6 replies 0/8
both, says-fail 8/8 on case 6 and 0/12 on case 8 both.
Suite after git add: 1235 tests, 1233/0/2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New grader names-code (regex, trace, code\W{1,12}BRIEF_V51_MISSING_SIGNALS),
the same form as the intent case's names-code. It checks what the case's
expected_outcome promised before any run: the validator reports the code.
Checked on this session's recorded traces before the commit: it matches
10 of 10 green runs and 0 of 3 runs with the check disabled.
Known-positive re-run in the scratch copy (validator check 'if (false)'):
0 of 3 cases pass, names-code fails in each, the other 5 graders pass.
Green: tests/lib/proevesett.test.mjs 10 of 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scaffold writes a current intent_approved_hash (sha256 over ## Intent +
## Goal) into the case-4 brief. Chose a literal marker over running
intent-approval.mjs --stamp in the scaffold because --stamp emits a
brief-approved record into the live stats the yardstick reads: every eval
run would count as an approval.
New grader no-intent-halt (regex, trace, not_contains
code\W{1,12}BRIEF_INTENT_): a halt at the intent gate fails the case
instead of passing it for the wrong gate.
says-fail / says-pass take FAILED / PASSED as the verdict word. v2's word
boundary dropped them. The one says-fail failure (PM, 1 of 7) is not
explained: its trace was deleted, and 0 of 13 kept replies used another
spelling than FAIL.
Case 4 alone, 10 runs: 10 of 10. In 3 of 10 the child ran the intent gate
first, passed it and went on to the brief-validator. That order halted the
case before. Before the fix, same day: 5 of 6.
Green: tests/lib/proevesett.test.mjs 9 of 9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New case plan-halts-without-intent-approval (expectation committed before
its first run): a 2.1 brief WITH phase_signals and no intent marker.
Graders: intent-approval.mjs --check ran; the trace carries the gate's own
JSON code for BRIEF_INTENT_NOT_APPROVED; no Agent; no plan.md.
review-requires-project: the arg parser now has a CLI that checks the
required flag and prints 'Error: --project <dir> is required.' + usage
(exit 1, code ARG_REQUIRED_MISSING). trekreview.md runs it with
$ARGUMENTS (it passed "$@", which the Bash tool never has, so the parser
never ran — 0 Bash calls in the PM's case-5 traces) and relays its stderr
instead of composing the line. The grader project-required (reply regex,
unstable on backticks) is replaced by parser-ran + names-missing-project
(trace, the parser's JSON code), the same form as names-rule.
Graders: no-error-code is bound to the validator's JSON output form and
covers every REVIEW_* code (WRONG_TYPE and VERSION_FORMAT were missing);
says-pass/says-fail reject a preceding NOT and a longer word; every case
with no-write also gets no-bash-write (redirect, tee, touch, cp, mv, rm,
sed -i in the Bash command). Checked on the PM's 10 recorded traces: no
false positive; known-positive/negative still split.
Red 348aa95 6/7 → green 7/7. Suite 1231: 1229 pass / 0 fail / 2 skip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Offline structure tests for evals/ (the suite itself needs headless runs):
- no case reaches the steg 1 intent gate (PM: 0 traces touched it). The
new case's scaffold is in: its brief passes brief-validator --soft
(2.1 WITH phase_signals) and --check gives BRIEF_INTENT_NOT_APPROVED —
that test is green, so the fixture reaches the gate.
- review-requires-project: trekreview.md composes 'Error: --project <dir>
is required.' in prose (8/10 in the PM run, backticks broke the regex),
and its arg-parser line passes "$@", which the Bash tool never has.
- no-error-code misses REVIEW_WRONG_TYPE; PASS/FAIL graders are raw
substrings; no-write cannot see a write through Bash.
7 tests, 6 red.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven `claude plugin eval` cases under evals/, runs: 1, deterministic graders
only (regex, tool_used, file_exists), no ablation. Each case tests one thing
Voyage promises and stops within minutes, headless:
- plan-requires-brief, plan-project-not-initialized, plan-rejects-unknown-export,
review-requires-project: argument guards stop before any Agent or Write.
- plan-halts-without-phase-signals: a brief_version 2.1 brief without
phase_signals halts /trekplan at the sequencing gate, before the swarm.
- review-validate-flags-bad-finding-id (known-positive) and
review-validate-passes-clean-review (known-negative): /trekreview --validate
names REVIEW_BAD_FINDING_ID on a planted bad ID and stays clean on a valid file.
Each case's expected_outcome is committed here, before the suite has run.
Fixtures are written by an inline scaffold.sh (needs --scaffold): chose inline
heredocs because a run cannot read the eval directory, and the add_dirs path
mapping is not documented. evals/results/ is ignored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
