feat(run): the run step names Claude Code's run; owner=run is a built-in owner (A2)

The coupling to Claude Code's built-in run skill (wave 4, order A2,
decisions 54 and 64). The plan's "run the thing" step, the executor's
End to end part and the plan template each say in one sentence that the
app is seen and driven with run, and that the verdict is still the
End-to-end: command, because run gives no exit code. The template quotes
the name instead of backticking it: every backticked span on a template
step bullet is read as a runnable example.

R2: lib/skills/routing.mjs keeps a short fixed list of Claude Code's own
skills a quiet case may name as owner (BUILTIN_OWNERS, only run, seen in
2.1.303). For such an owner only "not taken" is checked, and the report
says so; ROUTING_NEGATIVE_WON stands, and an owner on no list is still
ROUTING_UNKNOWN_OWNER. skill-writing-criteria-quiet-2 gets owner=run.

The trekplan.md paragraph is reflowed to its old line count so the stats
line_ref there does not move (order B edits the same file); the
trekexecute Phase 9 line_ref fixture moves by two lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-10-19 18:26:10 +02:00
commit 1989656a9c
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
12 changed files with 141 additions and 22 deletions

View file

@ -9,9 +9,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
### Doors
- two-way: every case in `evals/` names its event, and the routing test checks every case (P10, R1) — a revert lets a case without `from=` through again; the eight tags added to the fixed test set stay harmless.
- two-way: the coupling to Claude Code's `run` and the built-in owner list (R2) — a revert removes the three sentences and makes `owner=run` an unknown owner again; the criteria runner and `End-to-end:` are unchanged.
### Changed
- **The "run the thing" step names Claude Code's built-in `run` skill.** The plan's run step (`/trekplan`), the executor's end-to-end part (`/trekexecute`) and the plan template each say in one sentence that the app is seen and driven with `run`, and that the verdict is still the `End-to-end:` command through the criteria runner, because `run` gives no exit code (decisions 54 and 64).
- **A quiet eval case may name Claude Code's own `run` as its owner** (R2). The routing test keeps a short fixed list of built-in owners (`BUILTIN_OWNERS`, only `run`); for such an owner, which is outside the corpus, only "the skill does not rank first" is checked, and the report says so. `evals/skill-writing-criteria-quiet-2` now has `owner=run` instead of `/trekexecute` standing in.
- **P10's model-free part is done; the eval gate (P10) is exempted until the fixed test set runs again.** Every case in `evals/` now names the event it came from (`from=`), the eight test-set cases included (`from=proevesett-eval-suite`, `from=proevesett-intent-gate`). The routing test (`lib/skills/routing.mjs`) fails any case without it, routed or not, and its report gives both counts: cases checked for `from=` and cases routed. The release gate that would read the last test-set run is written into `docs/release-runbook.md` as exempted with its reason, not built: a gate that can never fire is not built.
## v6.5.0 — 2026-10-18 — Three skills for brief, criteria and review, English labels, and three trial-5 fixes

View file

@ -33,7 +33,7 @@ Voyage — a contract-driven Claude Code pipeline: brief, research, plan, execut
| `/trekbrief` | Brief — one question round sized by class (`ceremony: small\|bounded\|architectural`, heaviest in doubt, said aloud, only moved up), run in the form of `voyage:interviewing` (asked for by name; criteria by `voyage:writing-criteria`, which `brief-reviewer` is also told to judge by, with `voyage:reviewing-adversarially`), produces a task brief with explicit research plan; ends with the operator explicitly approving `## Intent` + `## Goal` ("sounds good" is not approval), logged in the target repo's hash-chained approval log and committed (the brief too when `artifact_store: git`) (`--approve <dir>` does only the approval, then writes the trekbrief record); optionally orchestrates the pipeline | session |
| `/trekresearch` | Research — deep local + external research, produces structured research brief. Opt-in `--engine {swarm\|deep-research}` delegates the external phase to Claude Code's built-in `/deep-research` workflow (swarm default). The external agents preload the `researching-sources` skill and are told it by name at spawn | session |
| `/trekplan` | Plan — brief-reviewer, explore, plan, review. Requires `--brief` or `--project`; halts unless the brief's intent is approved (hash-bound, stale on edit). Auto-discovers `architecture/overview.md` if present. Plan 1.9: the file first on every line — every `Files:` entry and `Test first:` `File:` line opens with its backticked path(s). From 1.8: `Test first:` per step + one `End-to-end:` step; every check written by `voyage:writing-criteria` (named where the plan is written and where its three reviewers start); a design part for `ceremony: architectural` (absent = architectural); critic loop of max 3 rounds, recorded in the plan, rounds 2+ in diff mode (rule A, counted by the gate; rule B in the reviewers), and `plan-gate.mjs` (0 go / 1 stop / 2 could not run) decides; open minors after it are listed for the operator to pick | session |
| `/trekexecute` | Execute — disciplined plan/session-spec executor: refuses a plan whose logged approval does not cover its current sha256 (K2); one fresh `step-implementer` helper per plan step, one controller that reads the diff (spec + quality verdicts), proof of red / floor guard / step check (K9) / `Regression:` (K10) / done line as commands, fix rounds that go up (max 5: helper → plan → operator), architecture rules (K5) + plan-file sweep (K6) after the last step, then its own review (`/simplify` → `/trekreview` → verdict gate, K7; BLOCK keeps the merge into main closed), closed stop list + decision log in the final report | session |
| `/trekexecute` | Execute — disciplined plan/session-spec executor: refuses a plan whose logged approval does not cover its current sha256 (K2); one fresh `step-implementer` helper per plan step, one controller that reads the diff (spec + quality verdicts), proof of red / floor guard / step check (K9) / `Regression:` (K10) / done line as commands, fix rounds that go up (max 5: helper → plan → operator), architecture rules (K5) + plan-file sweep (K6) after the last step, the app seen with Claude Code's built-in `run` while the `End-to-end:` command stays the verdict (`run` gives no exit code), then its own review (`/simplify` → `/trekreview` → verdict gate, K7; BLOCK keeps the merge into main closed), closed stop list + decision log in the final report | session |
| `/trekreview` | Review — a gate: Claude Code's `/code-review` + `/security-review` as engine, then Voyage's brief control (which stops); writes `review.md` with a required verdict that `lib/review/verdict-gate.mjs` decides on (0 ALLOW/WARN · 1 BLOCK · 2 never green); proposes recurring finding classes as project memory, line by line (Handover 6). The reviewers and coordinator preload `voyage:reviewing-adversarially` and are told it by name at spawn; each reviewer's Declined to Judge lines reach `review.md` | session |
| `/trekcontinue` | Continue — resumes the next session of a multi-session voyage project. Reads `.session-state.local.json` (Handover 7) and immediately begins executing | session |
| `/trekendsession` | End-session — mark the current session complete and write session-state pointing at the next session. Helper for informal multi-session flows | session |
@ -140,7 +140,7 @@ Full flag reference for each command (modes, `--gates`, `--profile`, breaking ch
| `voyage:writing-criteria` | Each criterion / `Verify:` / `Regression:` / `End-to-end:` is one command the criteria runner accepts, as the line's first backticked span; anything more is `bash tests/<name>.sh`; it must be able to go red (runner deterministic, go-red prose only) | `brief-reviewer` + `plan-critic` `skills:` (preload); `/trekplan` names it |
> **Commands are the trunk** (operator, 06.10). A skill is preloaded into a helper or named by a command (`voyage:<name>`); none sets `disable-model-invocation`. Form = S1, checked by `lib/skills/skill-lint.mjs`; `lib/skills/routing.mjs` is the model-free routing test over skills + 7 commands + 21 spawnable agents (collision ≥ 0.75 fails; thresholds pinned, never lowered). Eval cases `evals/skill-*` (3 fire / 2 quiet / 1 effect) are written, not run.
> Every case in `evals/` (the test set too) carries `from=<event>`; the routing test fails one without it and reports both counts, checked for `from=` and routed
> Every case in `evals/` (the test set too) carries `from=<event>`; the routing test fails one without it and reports both counts, checked for `from=` and routed; a quiet case's owner may be Claude Code's own `run` (`BUILTIN_OWNERS`, R2), outside the corpus, so only "not taken" is checked
> (P10's model-free part, R1; the release eval gate is exempted until the test set runs again). Method (red baseline first) → `docs/skills-authoring.md`.
## Reference docs (read on demand)

View file

@ -328,7 +328,7 @@ Every implementation step includes:
- **Checkpoint:** — git commit after success
- **Manifest:** — the objective completion predicate (Hard Rule 17)
One step per plan is the "run the thing" step (`End-to-end:`): the product run the way a user runs it. Without it a result is "not verified end to end", never green.
One step per plan is the "run the thing" step (`End-to-end:`): the product run the way a user runs it. Without it a result is "not verified end to end", never green. The app is seen and driven with Claude Code's built-in `run` skill; the verdict is still the `End-to-end:` command, because `run` gives no exit code.
**Checks that run, and can fail.** Every `Verify:`, `Regression:`, `End-to-end:` and `## Verification` command, like every brief success criterion, is written by the `voyage:writing-criteria` skill: one command the criteria runner accepts (a known test runner, plain words, no pipe or `&&`), a script under `tests/` (`bash tests/<name>.sh`) for anything more, and a command that can go red. `/trekplan` asks for it where the plan is written and where it starts its reviewers; `brief-reviewer` and `plan-critic` have it preloaded and report a criterion the runner refuses, or one that exits 0 whatever the code does (`git status`, `git log`, a test file that already passes).
@ -346,7 +346,7 @@ Borrowed, with credit: per-step interfaces, Global Constraints and Review Focus
Reads a plan from `/trekplan` and implements it with strict discipline. No guessing, no improvising — follows the plan exactly.
Per step: record the start point → **one fresh helper** (`step-implementer`) writes the step's test first, proves it red, locks it, then applies Changes exactly as written → the executor proves it by command: **red at the start point, green at the result** (`red-proof.mjs`), Verify (exit code is truth), manifest audit, and the **floor guard** on the diff (`test-integrity.mjs`: removed asserts, skipped or deleted tests, muted checks, stubs, loosened thresholds) → **one controller** reads the diff — never the helper's summary — and gives two verdicts, *spec* and *quality* → a script writes the step's **done line** only when the test command exits 0 (`done-line.mjs`) → Checkpoint commit. After all steps: independent Phase 7.5 manifest audit from git log + filesystem (ignoring agent bookkeeping); drift triggers Phase 7.6 bounded recovery. A plan with no passing run-the-thing criterion ends **not verified end to end**, never green.
Per step: record the start point → **one fresh helper** (`step-implementer`) writes the step's test first, proves it red, locks it, then applies Changes exactly as written → the executor proves it by command: **red at the start point, green at the result** (`red-proof.mjs`), Verify (exit code is truth), manifest audit, and the **floor guard** on the diff (`test-integrity.mjs`: removed asserts, skipped or deleted tests, muted checks, stubs, loosened thresholds) → **one controller** reads the diff — never the helper's summary — and gives two verdicts, *spec* and *quality* → a script writes the step's **done line** only when the test command exits 0 (`done-line.mjs`) → Checkpoint commit. After all steps: independent Phase 7.5 manifest audit from git log + filesystem (ignoring agent bookkeeping); drift triggers Phase 7.6 bounded recovery. A plan with no passing run-the-thing criterion ends **not verified end to end**, never green. Claude Code's built-in `run` skill can show the app working, but it gives no exit code, so the run-the-thing verdict stays with the plan's `End-to-end:` command.
A run starts only on a plan you approved **as it is now**: the approval log holds the plan's sha256, so a plan edited after its approval is refused before step 1. At step end the repo's own tools check the step's diff (the step check) and the step's `Regression:` suite must stay green. After the last step come the architecture rules and the sweep for files the plan does not name, and then **the review starts by itself**: `/simplify` → `/trekreview` → the verdict gate. A BLOCK keeps the merge into main closed.
@ -675,7 +675,7 @@ and the model-free routing test `lib/skills/routing.mjs` (no two descriptions am
commands and agents may collide; 3 firing, 2 quiet and 1 effect case per skill). The eval cases sit
in `evals/skill-*` for the operator's own eval run. Every case in `evals/`, the fixed test set's
too, names the event it came from (`from=`); the routing test fails a case without it and reports
how many cases it checked for `from=` and how many it routed. How to write one, starting with a baseline the
how many cases it checked for `from=` and how many it routed. A quiet case's owner may be Claude Code's own `run` skill, which is outside the corpus: then only "the skill does not rank first" is checked. How to write one, starting with a baseline the
helper fails without the skill: [docs/skills-authoring.md](docs/skills-authoring.md).
Borrowed, with credit: the form check and routing test from addyosmani/agent-skills (MIT); the red

View file

@ -2087,7 +2087,9 @@ verification failure must not hide a manifest drift.
### End to end — "run the thing" (P9 (6))
Tests passing is not the product working. Copy `end_to_end.status` from the
Tests passing is not the product working. To see and drive the app, use
Claude Code's built-in `run` skill; the verdict is still the `End-to-end:`
command, because `run` gives no exit code. Copy `end_to_end.status` from the
runner's JSON into `end_to_end` — the runner decides it, not this prose:
- `"verified"` — the verification passed and at least one passed criterion

View file

@ -792,17 +792,17 @@ Write the plan following the template structure. The plan MUST include:
(`PLAN_STEP_TESTS_ONLY`).
**The "run the thing" step.** At least one step carries
`End-to-end: \`<command>\` → expected: <what the user sees>`: the product run
the way a user runs it (the CLI, the server and a request, the app; for iOS,
the simulator). Without it the result is "not verified end to end", never
green. The command passes the criteria runner's allowlist — the product's
own binary does not, so the run is wrapped in a script:
`End-to-end: \`<command>\` → expected: <what the user sees>`: the product run the way a
user runs it (the CLI, the server and a request, the app; for iOS, the simulator). Without
it the result is "not verified end to end", never green. To see and drive the app, use
Claude Code's built-in `run` skill; the verdict is still the `End-to-end:` command, because
`run` gives no exit code. The command passes the criteria runner's allowlist — the
product's own binary does not, so the run is wrapped in a script:
`End-to-end: \`bash tests/e2e.sh\``. The same command is also listed under
`## Verification`: Phase 7 of `/trekexecute` runs that section, and a passed
criterion that runs the step's `End-to-end:` command is what verifies the
run end to end. `End-to-end:` rides on the last step that builds behaviour,
with `tests/e2e.sh` in that step's `Files:`: a step of its own that only adds
the script is a step of only tests, and it can never be proven red.
`## Verification`: Phase 7 of `/trekexecute` runs that section, and a passed criterion that
runs the step's `End-to-end:` command is what verifies the run end to end. `End-to-end:`
rides on the last step that builds behaviour, with `tests/e2e.sh` in that step's `Files:`:
a step of its own that only adds the script is a step of only tests, and it can never be proven red.
10. **Execution Strategy** — For plans with > 5 steps: group steps into sessions
(3–5 steps each), organize sessions into waves (parallel where independent),
specify scope fences per session. Omit for plans with ≤ 5 steps.

View file

@ -51,7 +51,10 @@ the 7 commands and the 21 spawnable agents, with no model:
- Each skill has at least 3 prompts where it should fire, 2 where it should not, and 1 effect
case. A firing prompt must rank the skill in the top 3; every one currently ranks it first, and
that floor is held. A prompt where the skill should stay quiet names the command, agent or skill
that owns it, and that owner must rank higher.
that owns it, and that owner must rank higher. The owner may also be one of Claude Code's own
skills on a short fixed list in `routing.mjs` (`BUILTIN_OWNERS`, only `run` for now). Such an
owner is outside the corpus, so only "not taken" is checked: the skill must not rank first, and
the report says so for each such case.
- The thresholds are pinned in `tests/lib/skill-routing.test.mjs` and never lowered. A failing case
is fixed in a description, never in a number. You do not change another component's
description to make room for yours.

View file

@ -1,7 +1,7 @@
---
description: "Routing case: the writing-criteria skill must stay quiet here. Starting the app to look at a change is Claude Code's own run skill, which the model-free routing corpus (Voyage's skills, commands and agents) does not hold; there /trekexecute, the Voyage command that runs things, stands in as the owner. The model-run grader only checks that writing-criteria is not called."
description: "Routing case: the writing-criteria skill must stay quiet here. Starting the app to look at a change is Claude Code's own run skill. It is outside the model-free routing corpus (Voyage's skills, commands and agents), so the routing test only checks that writing-criteria does not rank first. The model-run grader only checks that writing-criteria is not called."
expected_outcome: "Claude never calls the Skill tool for voyage:writing-criteria."
tags: [skill-no-trigger, skill=writing-criteria, owner=trekexecute, from=order-7c-negatives]
tags: [skill-no-trigger, skill=writing-criteria, owner=run, from=order-7c-negatives]
runs: 3
max_turns: 4
allowed_tools: [Read, Glob, Grep, Skill]

View file

@ -20,6 +20,8 @@
// A case is tagged in its prompt.md frontmatter:
// tags: [skill-trigger | skill-no-trigger | skill-effect, skill=<name>,
// owner=<command|agent|skill name> (negatives), from=<event>]
// An owner may also be one of Claude Code's own skills on BUILTIN_OWNERS (R2);
// it stands outside the corpus, so only "the skill does not rank first" is checked.
// `from=` names the event the case came from (P10, R1): it is required on every
// case in evals/, and a case with no skill tag (the fixed test set) is not
// routed, only checked for `from=`.
@ -212,6 +214,16 @@ export function loadCases(root) {
// ---- the check ----
/**
* Claude Code's own skills a negative case may name as its owner (R2). They
* stand outside the corpus, so for such an owner only "not taken" is checked:
* the skill under test must not rank first. Kept short and fixed.
* run - the bundled "Launch and drive your app to see a change working"
* skill, seen in Claude Code 2.1.303 (code.claude.com/docs/en/skills,
* "Run and verify your app").
*/
export const BUILTIN_OWNERS = Object.freeze(['run']);
/**
* Run the whole routing check over a plugin root.
* @returns {{ errors: object[], warnings: object[], stats: object }}
@ -228,6 +240,7 @@ export function checkRouting(root, t = THRESHOLDS) {
const cases = all.filter((c) => c.kind);
const errors = [];
const warnings = [];
const notes = [];
let positives = 0; let rank1 = 0; let negatives = 0;
for (const c of collisions(corpus, t)) {
@ -265,6 +278,9 @@ export function checkRouting(root, t = THRESHOLDS) {
}
if (!c.owner) {
errors.push({ code: 'ROUTING_NEGATIVE_NO_OWNER', case: c.case, message: `${c.case}: a negative case names the owner that should take the prompt (owner=...)` });
} else if (!byName.has(c.owner) && BUILTIN_OWNERS.includes(c.owner)) {
notes.push({ code: 'ROUTING_OWNER_OUTSIDE_CORPUS', case: c.case,
message: `${c.case}: owner=${c.owner} is Claude Code's own; owner outside the corpus: only "not taken" checked` });
} else if (!byName.has(c.owner)) {
errors.push({ code: 'ROUTING_UNKNOWN_OWNER', case: c.case, message: `${c.case}: owner=${c.owner} is no skill, command or spawnable agent` });
} else {
@ -290,7 +306,7 @@ export function checkRouting(root, t = THRESHOLDS) {
if (positives && rate < t.MIN_RANK1_RATE) {
errors.push({ code: 'ROUTING_RANK1_FLOOR', message: `${rank1} of ${positives} positives rank their skill first (${(rate * 100).toFixed(0)}%); the floor is ${(t.MIN_RANK1_RATE * 100).toFixed(0)}%` });
}
return { errors, warnings, stats: { entries: entries.length, skills: skills.length, sourced: all.length, cases: cases.length, positives, negatives, rank1 } };
return { errors, warnings, notes, stats: { entries: entries.length, skills: skills.length, sourced: all.length, cases: cases.length, positives, negatives, rank1 } };
}
// ---- CLI ----
@ -308,6 +324,7 @@ if (isMain(import.meta.url)) {
else {
for (const e of r.errors) process.stdout.write(`FAIL ${e.code}: ${e.message}\n`);
for (const w of r.warnings) process.stdout.write(`warn ${w.code}: ${w.message}\n`);
for (const n of r.notes) process.stdout.write(`note ${n.code}: ${n.message}\n`);
process.stdout.write(`${r.stats.entries} descriptions, ${r.stats.sourced} cases checked for from=, ${r.stats.cases} routed, ${r.stats.rank1}/${r.stats.positives} positives first; ${r.errors.length} finding(s)\n`);
}
process.exit(r.errors.length ? 1 : 0);

View file

@ -145,7 +145,7 @@ Several test files share one `File:` line the same way:
- File: `path/to/test.ts` *(existing | new)*
- Verifies: {what the test checks}
- Pattern: `path/to/existing-test.ts`
- **End-to-end:** `{a script that runs the product the way a user does — the app, the CLI, the simulator; usually bash tests/e2e.sh}` → expected: {what the user sees} *(the same command also stands under ## Verification)*
- **End-to-end:** `{a script that runs the product the way a user does — the app, the CLI, the simulator; usually bash tests/e2e.sh}` → expected: {what the user sees} *(the same command also stands under ## Verification; to see and drive the app, use Claude Code's built-in "run" skill — the verdict is still the End-to-end: command, because "run" gives no exit code)*
- **Verify:** `{exact command}` → expected: `{output}`
- **On failure:** {revert | retry | skip | escalate} — {specific instructions}
- **Checkpoint:** `git add {the step's Files and its test files} && git commit -m "{conventional commit message}"`

View file

@ -22,7 +22,7 @@
| trekbrief-stats | ts, task, slug, mode, interview_turns, review_iterations, brief_quality, research_topics, auto_research, auto_result, intent_approved, project_dir, order_id, order_claimed_at, order_reason | commands/trekbrief.md (orchestrator-emit Phase 7; order_* added by lib/stats/stats-append.mjs) | trekbrief.md:1128-1152 | profile, phase_models, profile_source | none |
| trekresearch-stats | ts, question, mode, scope, engine, slug, project_dir, brief_path, dimensions, dimensions_baseline, dimensions_baseline_preserved, effort, conv_turns, empty_turns, unique_sources, agents_local, agents_external, gemini_used, confidence, contradictions, open_questions | lib/stats/stats-append.mjs (trekresearch.md Stats tracking) | trekresearch.md:782-808 | profile, phase_models, parallel_agents, external_research_enabled, profile_source | none |
| trekplan-stats | ts, task, mode, slug, brief_path, project_dir, codebase_size, codebase_files, agents_deployed, deep_dives, research_briefs_used, research_scout_used, critic_verdict, guardian_verdict, outcome | lib/stats/stats-append.mjs (trekplan.md Phase 12) | trekplan.md:1277-1305 | profile, phase_models, parallel_agents, profile_source | none |
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2907-2929 | profile, phase_models, profile_source | none |
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2909-2931 | profile, phase_models, profile_source | none |
| trekexecute-stats (autonomy events) | ts, event, known_event, payload | lib/stats/event-emit.mjs `emit()` | event-emit.mjs:78-99 | payload.profile, payload.phase_models, payload.profile_source | none |
| trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | lib/stats/stats-append.mjs (trekreview.md Phase 8) | trekreview.md:636-686 | profile, phase_models, profile_source | none |

View file

@ -0,0 +1,59 @@
// tests/lib/run-coupling-prose-pins.test.mjs
// The coupling to Claude Code's built-in `run` skill (wave 4, order A2,
// decisions 54 and 64). The plan's "run the thing" step, the executor's
// `### End to end` and the plan template's run step each say in one sentence
// that the app is seen and driven with `run`, and that the verdict is still
// the `End-to-end:` command, because `run` gives no exit code. Each part is
// read by its heading or marker, never by line number.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { readFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
const read = (rel) => readFileSync(join(ROOT, rel), 'utf8');
/** From the line holding `marker` to the first line where `stop` is true. */
function partAt(rel, marker, stop) {
const lines = read(rel).split('\n');
const start = lines.findIndex((l) => l.includes(marker));
assert.ok(start !== -1, `${rel} must hold "${marker}"`);
let end = lines.length;
for (let i = start + 1; i < lines.length; i++) if (stop(lines[i])) { end = i; break; }
return lines.slice(start, end).join(' ').replace(/\s+/g, ' ');
}
const PARTS = [
// An indented paragraph of a numbered list item: it ends at a blank or unindented line.
['commands/trekplan.md', '**The "run the thing" step.**', (l) => l.trim() === '' || /^\S/.test(l)],
['commands/trekexecute.md', '### End to end — "run the thing" (P9 (6))', (l) => l.startsWith('### ')],
['templates/plan-template.md', '### Step 2: {description — the "run the thing" step}', (l) => l.startsWith('### ')],
];
// The template quotes the names instead of backticking them: every backticked
// span on a template step bullet is read as a runnable example
// (tests/lib/skills-criteria-prose-pins.test.mjs), and `run` is no command.
const RUN = '(?:`run`|"run")';
for (const [rel, marker, stop] of PARTS) {
test(`run coupling: ${rel} names Claude Code's run and keeps the verdict on the End-to-end: command`, () => {
const p = partAt(rel, marker, stop);
assert.match(p, new RegExp(`Claude Code's built-in ${RUN} skill`), `${rel}: the run step must name Claude Code's built-in run`);
assert.match(p, /the verdict is still the `?End-to-end:`? command/, `${rel}: the verdict must stay the End-to-end: command`);
assert.match(p, new RegExp(`${RUN} gives no exit code`), `${rel}: the reason must be said`);
});
}
test('run coupling: the run step still says what it runs, through the criteria runner', () => {
const p = partAt(...PARTS[0]);
assert.match(p, /criteria runner's allowlist/);
assert.match(p, /End-to-end: \\`bash tests\/e2e\.sh\\`/);
});
test('run coupling: the writing-criteria quiet case is owned by run, with no stand-in left in its description', () => {
const md = read('evals/skill-writing-criteria-quiet-2/prompt.md');
assert.match(md, /^tags: \[[^\]]*\bowner=run\b[^\]]*\]$/m);
assert.doesNotMatch(md, /stands in/);
assert.doesNotMatch(md, /owner=trekexecute/);
});

View file

@ -174,6 +174,40 @@ test('routing: a negative must name an owner that exists and outranks the skill'
});
});
// ---- R2: an owner from Claude Code's own skills (wave 4, order A2) ----
const R2_SPEC = (prompt, owner = 'run') => ({
skills: { 'tidy-imports': 'Sorts the import lines of a source file. Use when imports are out of order.' },
commands: { 'rotate-logs': 'Compresses and deletes old log files on disk.' },
cases: { n1: { tags: ['skill-no-trigger', 'skill=tidy-imports', `owner=${owner}`, 'from=test'], prompt } },
});
test('routing (R2): owner=run, a built-in Claude Code skill, is no unknown owner, and the report says what was checked', () => {
withPlugin(R2_SPEC('start the app and click through the new screen'), (root) => {
const r = checkRouting(root);
assert.ok(!r.errors.some((e) => e.case === 'n1'), JSON.stringify(r.errors));
const note = (r.notes || []).find((n) => n.case === 'n1');
assert.ok(note, JSON.stringify(r));
assert.match(note.message, /owner outside the corpus: only "not taken" checked/);
const out = spawnSync('node', [CLI, '--root', root], { encoding: 'utf8' });
assert.match(out.stdout, /note ROUTING_OWNER_OUTSIDE_CORPUS: n1: owner=run [^\n]*owner outside the corpus: only "not taken" checked/, out.stdout);
});
});
test('routing (R2): a skill that ranks first is still red when the owner is run', () => {
withPlugin(R2_SPEC('sort the import lines of the source file'), (root) => {
const r = checkRouting(root);
assert.ok(r.errors.some((e) => e.code === 'ROUTING_NEGATIVE_WON' && e.case === 'n1'), JSON.stringify(r.errors));
});
});
test('routing (R2): an owner neither in the corpus nor on the built-in list is still unknown', () => {
withPlugin(R2_SPEC('start the app and click through the new screen', 'nosuchthing'), (root) => {
const r = checkRouting(root);
assert.ok(r.errors.some((e) => e.code === 'ROUTING_UNKNOWN_OWNER' && e.case === 'n1'), JSON.stringify(r.errors));
});
});
test('routing: every case names the event it came from (P10)', () => {
withPlugin({
skills: { 'tidy-imports': 'Sorts the import lines of a source file. Use when imports are out of order.' },