Compare commits
63 commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 8d39e1d4a5 | |||
| 451969083b | |||
| 0799d6e914 | |||
| 22058459f8 | |||
| cd1d5c8738 | |||
| db3b8f5491 | |||
| 77ccf6ba06 | |||
| dcc71d9577 | |||
| 5c37b95dfb | |||
| 84fbee2313 | |||
| 8b7a849a76 | |||
| 357e17b176 | |||
| 937482067d | |||
| 76818b2459 | |||
| 4ec979747b | |||
| 0e657de023 | |||
| a6bed277d0 | |||
| 581489a513 | |||
| 9d8e043959 | |||
| 60e9e7ae5c | |||
| 926b768543 | |||
| 0c634b6636 | |||
| cb5dba9542 | |||
| 440594f1b2 | |||
| da418e653d | |||
| 15d172521d | |||
| 2b8ca9a044 | |||
| b148b6f8b8 | |||
| 27a7573b06 | |||
| 816bf2a5fc | |||
| 4bda2621eb | |||
| 6dea478de2 | |||
| 6bd50a42dd | |||
| be183e4617 | |||
| 68c9bef38f | |||
| 7e78d076f9 | |||
| 46d51f8088 | |||
| 62ebc28e3f | |||
| a9c442c201 | |||
| 708ba04571 | |||
| 5144e53129 | |||
| 844492fbcf | |||
| 971604d870 | |||
| e374a7c0ff | |||
| 84fa055e9f | |||
| f8d9d7fef9 | |||
| fe54ec91b4 | |||
| 8112e4f45c | |||
| ef569c084e | |||
| fd0e5775eb | |||
| 65a51b2667 | |||
| e34082d79a | |||
| d2f9ccb690 | |||
| e986b10431 | |||
| 0d8a5e8c5e | |||
| 213cf388de | |||
| 2df0cbb372 | |||
| 2849157ba2 | |||
| 9dc50a2047 | |||
| 6c00df573b | |||
| 2ff771779a | |||
| 153adc7fa7 | |||
| 1fc06502c9 |
98 changed files with 4868 additions and 663 deletions
|
|
@ -1,12 +1,23 @@
|
||||||
{
|
{
|
||||||
"name": "voyage",
|
"name": "voyage",
|
||||||
"description": "Voyage — brief, research, plan, execute, review, continue. Contract-driven Claude Code pipeline. /trekbrief, /trekplan, and /trekreview each end by building a self-contained operator-annotation HTML (scripts/annotate.mjs, modelled on claude-code-100x): select text or click any element, pick intent (Fiks/Endre/Spørsmål), write comment, copy structured prompt, paste back, Claude revises the .md.",
|
"description": "Voyage — brief, research, plan, execute, review, continue. Contract-driven Claude Code pipeline. /trekbrief, /trekplan, and /trekreview each end by building a self-contained operator-annotation HTML (scripts/annotate.mjs, modelled on claude-code-100x): select text or click any element, pick intent (Fiks/Endre/Spørsmål), write comment, copy structured prompt, paste back, Claude revises the .md.",
|
||||||
"version": "5.5.0",
|
"version": "5.9.1",
|
||||||
"author": {
|
"author": {
|
||||||
"name": "Kjell Tore Guttormsen"
|
"name": "Kjell Tore Guttormsen"
|
||||||
},
|
},
|
||||||
"homepage": "https://git.fromaitochitta.com/open/ktg-plugin-marketplace/src/branch/main/plugins/voyage",
|
"homepage": "https://git.fromaitochitta.com/open/ktg-plugin-marketplace/src/branch/main/plugins/voyage",
|
||||||
"repository": "https://git.fromaitochitta.com/open/ktg-plugin-marketplace.git",
|
"repository": "https://git.fromaitochitta.com/open/ktg-plugin-marketplace.git",
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
"keywords": ["voyage", "trek", "planning", "implementation", "research", "context-engineering", "agents", "adversarial-review", "headless", "execution"]
|
"keywords": [
|
||||||
|
"voyage",
|
||||||
|
"trek",
|
||||||
|
"planning",
|
||||||
|
"implementation",
|
||||||
|
"research",
|
||||||
|
"context-engineering",
|
||||||
|
"agents",
|
||||||
|
"adversarial-review",
|
||||||
|
"headless",
|
||||||
|
"execution"
|
||||||
|
]
|
||||||
}
|
}
|
||||||
|
|
|
||||||
7
.gitignore
vendored
7
.gitignore
vendored
|
|
@ -19,7 +19,9 @@ blob-report/
|
||||||
# Local configuration / session files
|
# Local configuration / session files
|
||||||
*.local.*
|
*.local.*
|
||||||
|
|
||||||
# STATE.md — current state-of-play (overskrives ved sesjonsslutt, gitignored per ~/.claude/CLAUDE.md).
|
# STATE.md — current state-of-play. LOCAL-ONLY per ~/.claude/CLAUDE.md:
|
||||||
|
# origin is open/ (an OFFENTLIG/public mirror) → STATE must NEVER be pushed there.
|
||||||
|
# Kept local for continuity only. History scrubbed 2026-06-26 (was wrongly tracked 23 commits).
|
||||||
STATE.md
|
STATE.md
|
||||||
|
|
||||||
# Local planning docs (briefs, design notes, observations) — never committed.
|
# Local planning docs (briefs, design notes, observations) — never committed.
|
||||||
|
|
@ -30,8 +32,7 @@ docs/ultracontinue-design-notes.md
|
||||||
# Ultraplan project directories — briefs, research, plans, progress all local.
|
# Ultraplan project directories — briefs, research, plans, progress all local.
|
||||||
.claude/projects/
|
.claude/projects/
|
||||||
|
|
||||||
# --- session/local state (gitignored per ~/.claude polyrepo-konvensjon) ---
|
# --- session/local state (gitignored) — STATE.md is LOCAL-ONLY (open/ = public), se ~/.claude/CLAUDE.md ---
|
||||||
STATE.md
|
|
||||||
REMEMBER.md
|
REMEMBER.md
|
||||||
ROADMAP.md
|
ROADMAP.md
|
||||||
TODO.md
|
TODO.md
|
||||||
|
|
|
||||||
132
CHANGELOG.md
132
CHANGELOG.md
|
|
@ -4,6 +4,138 @@ All notable changes to this project will be documented in this file.
|
||||||
|
|
||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||||
|
|
||||||
|
## v5.9.1 — 2026-07-03 — Fix /trekendsession load-time crash (eager-exec placeholders)
|
||||||
|
|
||||||
|
Patch, no functional additions.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- `/trekendsession` was unusable in every invocation: two of its three `` !`...` `` eager-exec blocks (Phase 3 atomic-write, Phase 4 validator call) contained unresolved runtime placeholders (`<project-dir>` etc.). The harness executes eager-exec blocks at command LOAD time, so zsh parsed `<project-dir>` as input redirection and the command aborted before the model saw a single instruction. Both blocks are now plain runtime Bash fences with the `{curly}` placeholder convention (shell-inert), matching `trekplan.md`/`trekresearch.md`. The Phase 1 discovery block (self-contained) keeps its legitimate eager-exec prefix; `trekcontinue.md`'s discovery block was runtime-verified unaffected.
|
||||||
|
- Latent secondary bug in the same blocks: cwd-relative plugin paths (`lib/validators/...`, `./lib/util/atomic-write.mjs`) would have failed with `ERR_MODULE_NOT_FOUND` even after substitution, since the Bash cwd is the user's repo. Both now use absolute `${CLAUDE_PLUGIN_ROOT}` paths per the existing command convention (Node ESM accepts absolute-path import specifiers — verified on Node 18+).
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- Regression guard `tests/commands/trekendsession.test.mjs`: scans every `` !` ``-block in `commands/*.md` for unresolved `<angle>`/`{curly}` placeholders (this bug class is silent until first invocation), plus structure tests pinning Phase 3/4 as runtime Bash with `${CLAUDE_PLUGIN_ROOT}` paths and exactly one surviving eager block. Suite baseline 828 → 832 (830 pass / 0 fail / 2 skip).
|
||||||
|
|
||||||
|
## v5.9.0 — 2026-07-02 — Fable model tier + deep-research engine
|
||||||
|
|
||||||
|
Additive, plus one behavior alignment: profile `phase_models` now reach sub-agent spawn sites (previously documented but never wired), and the seven command orchestrators no longer pin `model: opus` — frontmatter omits `model:`, so the orchestrator follows the session model.
|
||||||
|
|
||||||
|
### Fable model tier
|
||||||
|
|
||||||
|
- `fable` (→ Claude Fable 5, Mythos-class, positioned above Opus) is an accepted model value throughout the validation chain: `BASE_ALLOWED_MODELS` widened to `['sonnet', 'opus', 'fable']` in `lib/validators/profile-validator.mjs` — the single source imported by brief-validator and phase-signal-resolver (two-layer gate preserved; accept-fable AND reject-unknown-model covered at both layers). No env gate — haiku's `VOYAGE_ALLOW_HAIKU` opt-in stays as-is.
|
||||||
|
- `/trekbrief` Phase 3.5 tier loop offers a 4th option: `fable → {effort: high, model: fable}`. AskUserQuestion's 4-option maximum is now fully used — a 5th tier requires a loop redesign. The fable tier reuses `effort: high` orchestration semantics; `EFFORT_LEVELS` is unchanged.
|
||||||
|
- New built-in profile `lib/profiles/fable.yaml` (all six phases on `fable`, modeled on premium; registered in `BUILTIN_NAMES` with a `loadProfile('fable')` canary test so a registry regression fails loudly instead of silently resolving premium). Premium stays the default.
|
||||||
|
- Reasoning effort is inherited from the session: Fable 5's default effort is `high`, NOT xhigh, and switching model resets effort — set xhigh at session level (`/effort xhigh`, the `effortLevel` setting, or `CLAUDE_CODE_EFFORT_LEVEL`). Documented canonically in `docs/profiles.md` §Model & effort axes.
|
||||||
|
- `claude-fable-5` added to the cost `PRICE_TABLE` ($10/MTok input, $50/MTok output; cache write 5m $12.50 / 1h $20; cache read $1 — verified 2026-07-02 against the official platform pricing docs). `PRICE_TABLE_VERSION` bumped to `2026-07-02`. Without the entry, every fable run would report `cost_usd: null` in the observability export.
|
||||||
|
- Profile tables + allowlist prose updated across README, `docs/profiles.md`, `docs/operations.md`, `docs/HANDOVER-CONTRACTS.md`, `docs/architecture.md`, `docs/command-modes.md`, templates, and CLAUDE.md; the S15 doc pins now machine-check the fable row cell-for-cell against `fable.yaml`. The `^(opus|sonnet)…` regex claim in two docs was corrected — validation is an exact string match against `BASE_ALLOWED_MODELS`; the regex never existed in code.
|
||||||
|
|
||||||
|
### Behavior alignment: profile `phase_models` now reach sub-agent spawns
|
||||||
|
|
||||||
|
- Pre-existing wiring gap (found in exploration): all four pipeline commands invoked only the brief-only `phase-signal-resolver.mjs`, so the `?? profile.phase_models[<phase>]` half of the documented composition rule never executed — `--profile <x>` never reached sub-agent spawns.
|
||||||
|
- Fixed with a single composed resolver: `resolver.mjs --resolve-phase-model` now returns `{effort, model, source}` (brief > profile > default, with effort passed through atomically) and is the one CLI the four pipeline commands invoke. A doc-consistency pin requires the composed invocation and forbids the brief-only CLI in command Bash blocks.
|
||||||
|
- **Behavior change (contract alignment):** `--profile economy/balanced` now genuinely reaches sub-agent spawn sites for the first time — behavior aligns with what the docs have long claimed. The premium default is unaffected in practice (premium resolves `opus`, which equals the frontmatter fallback).
|
||||||
|
- Command frontmatter: the `model: opus` line is DELETED from all seven commands — omission (not the disputed `inherit` literal) is the spelling both official surfaces document as session-inheritance, guarded by a frontmatter-absence doc pin. Accepted tradeoff: in a sonnet session the orchestrator runs on sonnet; re-add a frontmatter pin for deterministic orchestrator choice. The 24 `agents/*.md` `model: opus` pins are untouched (spawn-time injection wins; frontmatter is the fallback). `/trekcontinue`/`/trekendsession` spawn no exploration swarm and get no spawn-site injection; the continue phase is covered at resolver level and follows the session model.
|
||||||
|
|
||||||
|
### Bundled unreleased work (since v5.8.0)
|
||||||
|
|
||||||
|
- `/trekresearch --engine {swarm|deep-research}` (`581489a..9374820`): opt-in delegation of the external research phase to Claude Code's built-in `/deep-research` workflow, with in-context adapter + self-check, availability fallback to swarm (never hard-fails), and doc-consistency pins across surfaces.
|
||||||
|
- brief-validator CLI no-flag invocation fix (`926b768`).
|
||||||
|
- Deep-research engine research notes + docs (`60e9e7a`, `9d8e043`).
|
||||||
|
|
||||||
|
### Operator + consume-side notes
|
||||||
|
|
||||||
|
- The operator-global CLAUDE.md policy "Opus 4.8 default for all subagents" predates the fable tier; updating it is an operator action outside this repo.
|
||||||
|
- `/plugin update` compares against a stale local marketplace clone and can report "already at the latest version" after this release (Claude Code issues #35752 / #38271, both closed-not-planned). Reliable refresh: remove + re-add the marketplace, or `git pull --ff-only` in the marketplace clone. Cross-version skew consequence: a stale cached v5.8 brief-validator REJECTS fable-bearing briefs with `BRIEF_INVALID_MODEL` — enum widening is safe for new readers of old data, not old readers of new data.
|
||||||
|
- Org `availableModels` with `enforceAvailableModels: true` can make an inheriting orchestrator silently fall back to the first allowed model.
|
||||||
|
|
||||||
|
### Release hygiene
|
||||||
|
|
||||||
|
- Suite (measured with bare `npm test` at release): **828 (826 pass / 0 fail / 2 skipped)** — +17 over the pre-release ground-truth baseline of 811 measured 2026-07-02 (allowlist/gate coverage, fable profile pins, composed-resolver + frontmatter-absence doc pins, PRICE_TABLE case).
|
||||||
|
- Version sync: `plugin.json`, `package.json`, `package-lock.json`, README badge, CHANGELOG top entry all at `5.9.0`, guarded by `doc-consistency.test.mjs`.
|
||||||
|
|
||||||
|
## v5.8.0 — 2026-06-30 — offline gold-scored output eval (SKAL-1·4b)
|
||||||
|
|
||||||
|
Additive — no behavior change, no breaking change. Internal eval infrastructure only (`lib/` + `tests/` + docs); no command, agent, profile, or Handover contract touched.
|
||||||
|
|
||||||
|
### Gold-scored output eval (SKAL-1·4b)
|
||||||
|
|
||||||
|
- The review-coordinator self-eval gains its **scoring run**, building on the deterministic coordinator contract + golden corpus shipped as the 4a foundation in v5.7.0.
|
||||||
|
- `lib/review/gold-scorer.mjs`: `scoreFindings(runFindings, goldFindings)` matches at **`(file, rule_key)` granularity** (line + severity deliberately ignored) → `{ tp, fp, fn, precision, recall, f1, matched, missed, spurious }`; `scoreVerdict` checks exact verdict match. Pure — no I/O, no LLM, no network. Vacuous-set conventions (empty run → recall 0; empty gold → precision 0; f1 collapses to 0) are documented in the module header.
|
||||||
|
- `tests/fixtures/bakeoff-rich/runs/run-perfect.json`: the first **committed run** — reviewer payloads that, fed through `runContract`, reproduce all 5 seeded gold findings. The regression guard: any future contract change that silently suppresses or skips a seeded finding breaks the eval.
|
||||||
|
- `tests/lib/gold-eval.test.mjs`: the **scoring run**, wired into `node --test`. Asserts `run-perfect` scores precision/recall/f1 = 1.0 and `verdict == expected_verdict` (BLOCK). Offline: committed payloads, no live agent spawn (the LLM-in-the-loop grading is the separate 4c tier).
|
||||||
|
- **Third test-census category.** `lib/util/test-census.mjs` now reports a `goldEval` bucket (matched by `GOLD_EVAL_FILE_RE`) separately from `behavior` and `docPins` — a scoring run is neither behavior coverage nor a prose pin, so the honest-count invariant is a 3-way sum.
|
||||||
|
- `docs/eval-corpus/README.md`: SKAL-1·4b moved from "Future hardening" to an implemented section documenting the scorer, run format, and census category.
|
||||||
|
- Suite: **822 (820 pass / 0 fail / 2 skip)**, up from 809 (+10 scorer behavior tests, +3 eval scoring-run tests). The scorer test covers the discriminating paths (false negatives + false positives) and the degenerate empty-run / empty-gold cases, not merely the all-match case.
|
||||||
|
- Version sync: `plugin.json`, `package.json`, `package-lock.json`, README badge, CHANGELOG top entry all at `5.8.0`, guarded by `doc-consistency.test.mjs`.
|
||||||
|
|
||||||
|
## v5.7.1 — 2026-06-29 — relocate agent `<example>` blocks to body (always-loaded token trim)
|
||||||
|
|
||||||
|
Performance/packaging change — no behavior change, no breaking change. (M4)
|
||||||
|
|
||||||
|
### Always-loaded token trim
|
||||||
|
|
||||||
|
- The 17 example-bearing agents each carried two `<example>` blocks (34 total) inside their `description:` frontmatter. voyage launches its agents **by name** from the orchestrator commands, so those auto-selection examples were paying a per-turn token tax for a path the pipeline never uses. They are now relocated **verbatim** into each agent's body under a `## When to use — examples` section — preserved, not deleted.
|
||||||
|
- Frontmatter `description` chars across all 24 agents: **17,672 → 4,945** — roughly **3,180 fewer always-loaded tokens per turn** for every session with voyage enabled. The 7 zero-example agents are untouched; no system prompt, tools, model, or `name` changed.
|
||||||
|
- New invariant test in `tests/lib/agent-frontmatter.test.mjs`: no `<example>` in any frontmatter `description`, and ≥ 34 `<example>` retained across agent bodies (relocation moves, never deletes). Suite: **807 pass / 0 fail / 2 skip**.
|
||||||
|
- **The saving only reaches a machine after `/plugin marketplace update` + reload** — the delta is not instant on the machine that ships the release.
|
||||||
|
- Version sync: `plugin.json`, `package.json`, `package-lock.json`, README badge, CHANGELOG top entry all at `5.7.1`, guarded by `doc-consistency.test.mjs`.
|
||||||
|
|
||||||
|
## v5.7.0 — 2026-06-26 — opt-in token/cost metering (SKAL-2) + eval foundation (SKAL-1·4a)
|
||||||
|
|
||||||
|
Additive — no breaking change. Two unreleased work-streams land together.
|
||||||
|
|
||||||
|
### Opt-in token/cost metering (SKAL-2) — the headline
|
||||||
|
|
||||||
|
- Pure token-usage parser + cache-aware USD cost (`lib/stats/token-usage.mjs`): parses MAIN-CONTEXT usage from the transcript, dedups by requestId (last-wins, streaming-placeholder mitigation), excludes sidechain records, and REFUSES to estimate (`cost_usd:null, is_estimate:true`) for models absent from the frozen PRICE_TABLE.
|
||||||
|
- CWE-212 export boundary: token-usage schema allowlist in the OTLP exporter; `session_id`/`transcript_path`/`cwd` stripped at export, asserted both ways.
|
||||||
|
- Cross-session aggregation in `cache-analyzer` (total tokens + cost).
|
||||||
|
- Capture folded into the EXISTING `otel-export.mjs` Stop hook, gated behind the `VOYAGE_TOKEN_METER` env var (default off), fail-open. v1 scope = main-context only; sub-agent turns are a documented v2 follow-on.
|
||||||
|
|
||||||
|
### Eval foundation (SKAL-1·4a)
|
||||||
|
|
||||||
|
- `gold.json` golden corpus + loader/validator; review-coordinator contract reference impl + deterministic test; two-sided gate coverage for the `BRIEF_*` BLOCKERs; eval-corpus frozen-failure home.
|
||||||
|
|
||||||
|
### Release hygiene
|
||||||
|
|
||||||
|
- Version sync: `plugin.json`, `package.json`, `package-lock.json`, README badge, CHANGELOG top entry all at `5.7.0`, guarded by `doc-consistency.test.mjs`.
|
||||||
|
- Canonical `node --test`: **807** (805 pass / 0 fail / 2 skipped).
|
||||||
|
|
||||||
|
## v5.6.1 — 2026-06-24 — leaner always-loaded agent listing (reference/dormant agent descriptions trimmed)
|
||||||
|
|
||||||
|
Additive — no breaking change, **no runtime behavior change**. Trims the always-loaded token cost of the agent listing that Claude Code injects into every session.
|
||||||
|
|
||||||
|
### Reference/dormant agents now carry one-line descriptions
|
||||||
|
|
||||||
|
- **The 3 `*-orchestrator` reference docs** (`planning-/research-/review-orchestrator`) and the **dormant `synthesis-agent`** carried multi-paragraph `description:` frontmatter (full rationale + CC-2.1.172 history + a usage example) that Claude Code injects into every session — despite none of them being spawnable from the live `/trek*` pipeline. Their `description:` is now a single line; the full rationale already lives, and remains, in each file's body. **~700 tokens trimmed** from the always-loaded agent listing, no capability change.
|
||||||
|
- The three orchestrators retain the self-declaration **"reference document, not a spawnable capability"** (pinned by `doc-consistency.test.mjs`); `synthesis-agent` retains its **DORMANT / not-wired** flag and the `docs/T1-synthesis-poc-results.md` pointer.
|
||||||
|
- Surfaced via dogfooding with `config-audit`'s always-loaded token audit. Canonical `node --test`: **754 pass, 0 fail** (756 total, 2 skipped); the agent-inventory + frontmatter pins stay green.
|
||||||
|
|
||||||
|
### Version sync
|
||||||
|
|
||||||
|
- `plugin.json`, `package.json`, `package-lock.json`, the README badge, and the CHANGELOG top entry all at `5.6.1`, guarded by the version-consistency test in `tests/lib/doc-consistency.test.mjs`.
|
||||||
|
|
||||||
|
## v5.6.0 — 2026-06-20 — `/trekexecute` loop hardening: machine-verifiable completion + bounded recovery
|
||||||
|
|
||||||
|
Additive — no breaking change. Hardens the `/trekexecute` execution loop so termination is *machine-verifiable* and recovery is *bounded by an explicit budget*, closing the "runs forever" and "declares done without proof" failure modes. Five focused changes; canonical `node --test` 744 → **756** (0 fail).
|
||||||
|
|
||||||
|
### Machine-verifiable completion gate (the headline)
|
||||||
|
|
||||||
|
- **Stop-signal contract** — `/trekexecute` reports `completed` only when a machine-checkable contract holds: the Phase 7.5 manifest audit PASSes, the stop signal exits 0, and the success criteria are green. **Hard Rule 18** is promoted from prose guidance to an enforced gate, anchored in the Phase 7.5 audit rather than the model's self-assessment.
|
||||||
|
|
||||||
|
### Bounded recovery — cap hierarchy + global budget
|
||||||
|
|
||||||
|
- **`TREKEXECUTE_MAX_RECOVERY_ITERATIONS`** — a global recovery-iteration budget (default **25**, env-overridable) sits above the per-step caps as a three-axis cap hierarchy (recovery depth · per-step retry · global budget), codified as **Hard Rule 20**. Stops an unbounded recover→retry→recover spiral that the per-step caps alone could not bound.
|
||||||
|
- **`iterations_remaining` signal** — surfaced in the progress schema and the summary JSON, with a deterministic gate cross-check (`iterations_remaining == cap − (recovery_depth + Σ attempts-beyond-first)`) that catches a never-decremented counter. Shape-validated in `lib/.../progress-validator.mjs` (`PROGRESS_ITERATIONS_REMAINING_INVALID`: non-negative integer, additive-optional — never required at top level).
|
||||||
|
|
||||||
|
### Doc consistency
|
||||||
|
|
||||||
|
- **Fan-out hedge harmonized** — the `CLAUDE.md` design-principle line is aligned to the measured claim (parallel wall-clock + structured artifact handoffs as the load-bearing benefit; main-context relief stays explicitly *not demonstrated*), with a banned-phrase **forward-guard** in `tests/lib/doc-consistency.test.mjs` so the over-claim cannot reappear.
|
||||||
|
|
||||||
|
### Release hygiene
|
||||||
|
|
||||||
|
- **Version sync** — `plugin.json`, `package.json`, `package-lock.json`, the README badge, and the CHANGELOG top entry all at `5.6.0`, guarded by the version-consistency test in `tests/lib/doc-consistency.test.mjs`.
|
||||||
|
|
||||||
## v5.5.0 — 2026-06-18 — Coordinated release: brief framing enforcement (2.2) + W1–W3 narrow wins
|
## v5.5.0 — 2026-06-18 — Coordinated release: brief framing enforcement (2.2) + W1–W3 narrow wins
|
||||||
|
|
||||||
The coordinated release held since the framing work. Supersedes the unreleased internal milestones **v5.2–v5.4** (W2 model/effort alignment, W3 guardrail hardening, the Handover-1 public-contract formalization), landing them together with the `brief_version 2.2` schema badge and the W1 narrow wins. **Additive — no breaking change for existing consumers:** every `2.0`/`2.1` brief still validates; the new requirements gate only on briefs that *declare* `brief_version: "2.2"`.
|
The coordinated release held since the framing work. Supersedes the unreleased internal milestones **v5.2–v5.4** (W2 model/effort alignment, W3 guardrail hardening, the Handover-1 public-contract formalization), landing them together with the `brief_version 2.2` schema badge and the W1 narrow wins. **Additive — no breaking change for existing consumers:** every `2.0`/`2.1` brief still validates; the new requirements gate only on briefs that *declare* `brief_version: "2.2"`.
|
||||||
|
|
|
||||||
16
CLAUDE.md
16
CLAUDE.md
|
|
@ -2,20 +2,20 @@
|
||||||
|
|
||||||
Voyage — a contract-driven Claude Code pipeline: brief, research, plan, execute, review, continue. Deep implementation planning and research with specialized agent swarms, external research, adversarial review, session decomposition, disciplined execution, and headless support.
|
Voyage — a contract-driven Claude Code pipeline: brief, research, plan, execute, review, continue. Deep implementation planning and research with specialized agent swarms, external research, adversarial review, session decomposition, disciplined execution, and headless support.
|
||||||
|
|
||||||
**Design principle: Context Engineering** — build the right context by orchestrating specialized agents. Each step in the pipeline (brief → research → plan → execute) produces a structured artifact that the next step consumes. (The claim that fanning out to a swarm *relieves* the main context is asserted-by-design, not measured end-to-end: the one delegation actually measured — the Phase-7 synthesis PoC — found Δ main-context ≈ 0, see `docs/T1-synthesis-poc-results.md`. The structured-artifact handoffs are the load-bearing benefit; main-context relief is not yet demonstrated.)
|
**Design principle: Context Engineering** — build the right context by orchestrating specialized agents. Each step in the pipeline (brief → research → plan → execute) produces a structured artifact that the next step consumes. The load-bearing benefit is the parallel wall-clock + structured artifact handoffs; main-context relief is asserted-by-design, not measured (Δ ≈ 0 in the one PoC — see `docs/T1-synthesis-poc-results.md`).
|
||||||
|
|
||||||
> **v3.0.0 — architect step extracted from this plugin.** The plan command still auto-discovers `architecture/overview.md` if present, so any compatible producer (architect plugin no longer publicly distributed; the architecture/overview.md slot remains available for any compatible producer) plugs into the same slot. See [CHANGELOG.md](CHANGELOG.md) for migration history.
|
> **Architecture slot.** The plan command auto-discovers `architecture/overview.md` if present — any compatible producer plugs in (the architect plugin is no longer publicly distributed). Migration history (v3.0.0 extraction) → [CHANGELOG.md](CHANGELOG.md).
|
||||||
|
|
||||||
> **Trinity context (2026-05-13, informational).** Voyage is Tier 1 (per-task) of a three-tier architecture in active design under the author's private marketplace: Tier 2 `app-creator` (per-app — "what does the app need, what's the next brief?") produces briefs Voyage consumes; Tier 3 `app-factory` (per-portfolio — "which app needs me now?") aggregates state across multiple app-creator instances. Both are pre-implementation and will ship to Forgejo when ready. **Asymmetry is a hard invariant:** Voyage stays unaware of Tier 2/3. Handover 1 (brief format) is the only integration point — any compatible producer can feed Voyage, app-creator is not privileged. Brief-schema changes are therefore breaking changes for downstream consumers, formalized as a public contract in v5.5.0 — see `docs/HANDOVER-CONTRACTS.md` §Handover 1 (PUBLIC CONTRACT).
|
> **Trinity context (informational).** Voyage is Tier 1 (per-task) of a three-tier architecture. **Asymmetry is a hard invariant:** Voyage stays unaware of Tier 2/3; Handover 1 (brief format) is the only integration point, no producer is privileged, and brief-schema changes are breaking for downstream consumers (formalized as a public contract in v5.5.0). Tier 2/3 producer detail + the public contract → `docs/HANDOVER-CONTRACTS.md` §Handover 1 (PUBLIC CONTRACT).
|
||||||
|
|
||||||
> **Cross-cutting invariant: brief framing must match operator intent (2026-05-15).** Etablert etter residiv. Briefen er pipelinens source of truth; operatørens intent lever i hodet + i memory-filer (`feedback_*`, `project_*`); pipelinen tvinger ikke alignment. Høyere reasoning-kraft polerer feil premiss istedenfor å utfordre det. **Tre lag av forsvar (input-siden), implementert i S6 som `brief_version 2.2`-gate (v5.5), alle BLOCKER ved brudd for briefer som deklarerer ≥ 2.2:** (1) eksplisitt `framing: preserve|refine|replace|new-direction` i brief-frontmatter, `AskUserQuestion`-validert i `/trekbrief` Phase 2.5 før brief-prosa skrives (ikke-skippbar, også i `--quick`); enum-feil → `BRIEF_INVALID_FRAMING` (alle versjoner), fravær ved ≥ 2.2 → `BRIEF_MISSING_FRAMING`; (2) memory-alignment som dimensjon 6 i `brief-reviewer` — sammenlikner brief-prosa + `framing` mot relevante memory-filer, rapporterer kun eksplisitte motsigelser (degraderer til score 5 N/A uten memory-kontekst), wired til Phase 4e-gate (`memory_alignment.score ≥ 4`); (3) obligatorisk `## TL;DR`-seksjon (≤ 5 linjer, soft-cap → `BRIEF_TLDR_TOO_LONG`) øverst i `brief.md`. Eksisterende 2.0/2.1-briefer forblir gyldige (forward+backward-compat, speiler `phase_signals ≥ 2.1`-presedensen); `trekreview`-briefer er unntatt. **Schema-akse bumpet (2.1→2.2); plugin-versjon-badge + CHANGELOG bumpes ved den koordinerte releasen (S10).** Kontrakt-evolusjon dokumentert i `docs/HANDOVER-CONTRACTS.md` §Handover 1. Den gamle manuelle stopgap-sjekken er dermed retired for ≥ 2.2-briefer.
|
> **Cross-cutting invariant: brief framing must match operator intent.** The brief is the pipeline's source of truth; operator intent lives in memory files — the pipeline must not polish a wrong premise. Enforced as the `brief_version 2.2` gate (v5.5), all BLOCKER for briefs declaring ≥ 2.2: (1) explicit `framing: preserve|refine|replace|new-direction` frontmatter, `AskUserQuestion`-validated in `/trekbrief` Phase 2.5; (2) memory-alignment as `brief-reviewer` dimension 6; (3) mandatory `## TL;DR`. Existing 2.0/2.1 briefs stay valid; `trekreview` briefs are exempt. Full implementation + contract evolution → `docs/HANDOVER-CONTRACTS.md` §Handover 1 (PUBLIC CONTRACT).
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
| Command | Description | Model |
|
| Command | Description | Model |
|
||||||
|---------|-------------|-------|
|
|---------|-------------|-------|
|
||||||
| `/trekbrief` | Brief — interactive interview produces a task brief with explicit research plan; optionally orchestrates the pipeline | opus |
|
| `/trekbrief` | Brief — interactive interview produces a task brief with explicit research plan; optionally orchestrates the pipeline | opus |
|
||||||
| `/trekresearch` | Research — deep local + external research, produces structured research brief | opus |
|
| `/trekresearch` | Research — deep local + external research, produces structured research brief. Opt-in `--engine {swarm\|deep-research}` delegates the external phase to Claude Code's built-in `/deep-research` workflow (swarm default) | opus |
|
||||||
| `/trekplan` | Plan — brief-reviewer, explore, plan, review. Requires `--brief` or `--project`. Auto-discovers `architecture/overview.md` if present | opus |
|
| `/trekplan` | Plan — brief-reviewer, explore, plan, review. Requires `--brief` or `--project`. Auto-discovers `architecture/overview.md` if present | opus |
|
||||||
| `/trekexecute` | Execute — disciplined plan/session-spec executor with failure recovery | opus |
|
| `/trekexecute` | Execute — disciplined plan/session-spec executor with failure recovery | opus |
|
||||||
| `/trekreview` | Review — independent post-hoc review of delivered code against the brief. Produces `review.md` with severity-tagged findings (Handover 6) | opus |
|
| `/trekreview` | Review — independent post-hoc review of delivered code against the brief. Produces `review.md` with severity-tagged findings (Handover 6) | opus |
|
||||||
|
|
@ -53,13 +53,15 @@ Full flag reference for each command (modes, `--gates`, `--profile`, breaking ch
|
||||||
| contrarian-researcher | opus | Counter-evidence, overlooked alternatives |
|
| contrarian-researcher | opus | Counter-evidence, overlooked alternatives |
|
||||||
| gemini-bridge | opus | Gemini Deep Research second opinion (conditional) |
|
| gemini-bridge | opus | Gemini Deep Research second opinion (conditional) |
|
||||||
|
|
||||||
> **Model & effort (CC 2.1.154+).** `opus` resolves to **Opus 4.8** (default reasoning effort `high`); `sonnet` to Sonnet 4.6. Select agents carry native `effort:` — retrieval agents (`task-finder`, `git-historian`, `dependency-tracer`, `architecture-mapper`) at `medium`, adversarial-reasoning agents (`plan-critic`, `risk-assessor`, `contrarian-researcher`, `review-coordinator`) at `high`. This native per-spawn **reasoning** effort is a different axis from brief `phase_signals.effort` (orchestration shape — which agents/passes run). See `docs/profiles.md` §Model & effort axes.
|
> **Inventory (S33 reconcile).** 24 agent files = **21 spawnable** (one, `synthesis-agent`, ships **dormant** — Δ≈0, wired to nothing) **+ 3 orchestrator reference docs** (`planning-/research-/review-orchestrator` document the inline `/trek*` workflow, not spawnable capabilities). All 24 stay `model: opus` (operator pin `40d8742`); the glue/mechanical/retrieval/dormant roles were reconsidered for a sonnet downgrade and **kept opus** — decision record: `docs/voyage-vs-cc-balance-analysis.md` §10.
|
||||||
|
|
||||||
|
> **Model & effort.** `opus` = Opus 4.8 (default reasoning effort `high`); `sonnet` = Sonnet 4.6; `fable` = Fable 5 (Mythos-class, above Opus — reasoning effort inherits from the session; xhigh requires a session-level setting). Select agents carry native per-spawn `effort:` (retrieval → `medium`, adversarial-reasoning → `high`) — a different axis from brief `phase_signals.effort` (orchestration shape: which agents/passes run). Per-agent table + axes → `docs/profiles.md` §Model & effort axes.
|
||||||
|
|
||||||
## Reference docs (read on demand)
|
## Reference docs (read on demand)
|
||||||
|
|
||||||
- **Architecture, workflows, project-directory contract, state, terminology:** `docs/architecture.md`
|
- **Architecture, workflows, project-directory contract, state, terminology:** `docs/architecture.md`
|
||||||
- **Quality infrastructure (`lib/` validators, parsers, autonomy primitives, hooks):** `docs/architecture.md` §Quality infrastructure
|
- **Quality infrastructure (`lib/` validators, parsers, autonomy primitives, hooks):** `docs/architecture.md` §Quality infrastructure
|
||||||
- **Autonomy gates (`--gates`), Path A/B/C decision:** `docs/operations.md`
|
- **Autonomy gates (`--gates`), Path A/B/C decision:** `docs/operations.md`
|
||||||
- **Profile system (`--profile economy/balanced/premium`), lookup order, custom profiles:** `docs/operations.md`
|
- **Profile system (`--profile economy/balanced/premium/fable`), lookup order, custom profiles:** `docs/operations.md`
|
||||||
- **Observability (Stop hook, OTLP/textfile export, SSRF mitigation):** `docs/operations.md`
|
- **Observability (Stop hook, OTLP/textfile export, SSRF mitigation):** `docs/operations.md`
|
||||||
- **Handover contracts (the 7 pipeline handovers):** `docs/HANDOVER-CONTRACTS.md`
|
- **Handover contracts (the 7 pipeline handovers):** `docs/HANDOVER-CONTRACTS.md`
|
||||||
|
|
|
||||||
104
README.md
104
README.md
|
|
@ -1,6 +1,6 @@
|
||||||
# trekplan — Brief, Research, Plan, Execute, Review, Continue
|
# trekplan — Brief, Research, Plan, Execute, Review, Continue
|
||||||
|
|
||||||

|

|
||||||

|

|
||||||

|

|
||||||
|
|
||||||
|
|
@ -10,21 +10,7 @@
|
||||||
|
|
||||||
A [Claude Code](https://docs.anthropic.com/en/docs/claude-code) plugin for deep implementation planning, multi-source research, autonomous execution, independent post-hoc review, and zero-friction multi-session resumption. Six commands, one pipeline:
|
A [Claude Code](https://docs.anthropic.com/en/docs/claude-code) plugin for deep implementation planning, multi-source research, autonomous execution, independent post-hoc review, and zero-friction multi-session resumption. Six commands, one pipeline:
|
||||||
|
|
||||||
> **What's new in v5.5.0 — brief framing enforcement (`brief_version 2.2`) + narrow wins** — `/trekbrief` now opens with a **framing declaration** (Phase 2.5): every brief records `framing: preserve | refine | replace | new-direction` *before* any prose is written, so the plan can't quietly polish a wrong premise. Three version-gated layers ship at `brief_version ≥ 2.2`:
|
> **What's new — v5.8.0: offline gold-scored output eval (SKAL-1·4b).** The review-coordinator self-eval gains a scoring run: `lib/review/gold-scorer.mjs` grades a committed agent-run fixture against the golden corpus at `(file, rule_key)` granularity (precision/recall/f1 + verdict match), and the suite census gains a third category (`goldEval`) so a scoring run is counted apart from behavior coverage and doc-pins. Offline + deterministic — committed reviewer payloads, no live agent spawn (the LLM-in-the-loop tier is the separate 4c). Internal eval infrastructure; no command/agent/Handover change. **v5.7.1:** leaner always-loaded agent listing — `<example>` blocks relocated to agent bodies (~3,180 tok/turn, no behavior change). **v5.7.0:** opt-in per-session token/cost metering (SKAL-2) + eval foundation (SKAL-1·4a). **v5.6.1:** one-line `description:` for the four reference/dormant agents (~700 tok). **v5.5.0:** brief **framing** enforcement (`brief_version 2.2`) + a `/trekreview` reviewer-schema contract. Additive — no breaking changes. **Full version history → [CHANGELOG.md](CHANGELOG.md).**
|
||||||
> - **Framing field** — a required `framing` enum in frontmatter; enum-checked on any version, and a missing value at `≥ 2.2` is a blocker. Collected in Phase 2.5 before prose, non-skippable even in `--quick`.
|
|
||||||
> - **Memory alignment** — a new `brief-reviewer` dimension compares the brief's Intent/Goal + framing against operator memory and flags *explicit* contradictions (a no-op that scores N/A when no memory is supplied).
|
|
||||||
> - **Obligatory `## TL;DR`** — a ≤ 5-line, framing-anchored summary at the top of every brief, so a wrong premise is caught at a glance.
|
|
||||||
> Existing `2.0`/`2.1` briefs stay valid (forward + backward compatible), mirroring the `phase_signals ≥ 2.1` gate. This is a **schema-axis** change — the plugin version badge bumped to **v5.5.0** at this coordinated release; see [docs/HANDOVER-CONTRACTS.md](docs/HANDOVER-CONTRACTS.md) §Handover 1 for the contract evolution.
|
|
||||||
> **Also in v5.5.0:** a reviewer-output **schema contract** for `/trekreview` (validates each reviewer's JSON, re-asks on schema failure); an opt-in **`--workflow`** path for `/trekreview` Phase 5–6 (bake-off POSITIVE; default stays prose to preserve the 2.1.154+ portability floor); **Opus 4.8** baseline with native `effort:` on 8 agents; exec-form hooks + enforced `disallowed-tools` on `/trekexecute`. Full arc in [CHANGELOG.md](CHANGELOG.md).
|
|
||||||
>
|
|
||||||
> **What's new in v5.1.1** — Remediation patch closing 11 of 12 findings from the v5.1.0 review (SC8 dogfood gate scheduled for sesjon 8). Lukker:
|
|
||||||
> - **Bug fixes (load-bearing):** YAML-number bypass in `brief-validator` (#8) + doc-consistency pin lock-in (#11) so the gate fires for both quoted and unquoted `brief_version`.
|
|
||||||
> - **Wiring:** `phase-signal-resolver` helper wired into all 4 downstream commands (#9) with TDD pair `resolvePhaseModel` + profile-resolver non-interference test (#4 SC5); `brief-validator` gate required uniformly in `/trekresearch` + `/trekexecute` (#12).
|
|
||||||
> - **Test refactor:** runtime SC1 walk for trekbrief (#1) + per-tier resolver-output + missing-signals falsification for `/trekplan`/`/trekresearch`/`/trekreview`/`/trekexecute` (#2 #3 #6 #10) + dedicated SC5 test (#7).
|
|
||||||
> - **Documentation:** Dogfood-gate scheduling in REMEMBER (#5, sesjon 8 manual) + Decision B high-effort behavior per command + brief Non-Goal/SC1 amendments + coordinator high-effort normalization.
|
|
||||||
> v5.1.1 is additive — no breaking changes against v5.1.0.
|
|
||||||
>
|
|
||||||
> **What v5.1 introduced** — `/trekbrief` Phase 3.5 commits per-phase `phase_signals` (effort + optional model for `research`/`plan`/`execute`/`review`) to `brief.md` frontmatter. `brief_version: 2.1` activates a validator-side sequencing gate (`BRIEF_V51_MISSING_SIGNALS`) so downstream commands halt with a friendly hint when signals are missing. Composition rule per downstream command: brief signal wins per-phase, profile fills gaps. `effort == low` activates the existing `--quick`-equivalent code-path in each command (`/trekexecute` low-effort = `--gates open` + sequential). Additive — no breaking changes; pre-2.1 briefs still validate.
|
|
||||||
|
|
||||||
| Command | What it does |
|
| Command | What it does |
|
||||||
|---------|-------------|
|
|---------|-------------|
|
||||||
|
|
@ -165,7 +151,7 @@ Output: `.claude/projects/{YYYY-MM-DD}-{slug}/brief.md`
|
||||||
|------|-------|----------|
|
|------|-------|----------|
|
||||||
| **Default** | `/trekbrief <task>` | Dynamic interview until quality gates pass. No question cap. |
|
| **Default** | `/trekbrief <task>` | Dynamic interview until quality gates pass. No question cap. |
|
||||||
| **Quick** | `/trekbrief --quick <task>` | Starts compact (optional sections get at most one probe), still escalates on weak required sections or failed review gate. |
|
| **Quick** | `/trekbrief --quick <task>` | Starts compact (optional sections get at most one probe), still escalates on weak required sections or failed review gate. |
|
||||||
| **Profile** | `/trekbrief --profile <name> <task>` | (v4.1.0) Pin model profile for the brief phase: `economy` / `balanced` / `premium` / `<custom>`. See [Profile system](#profile-system-v410) below. |
|
| **Profile** | `/trekbrief --profile <name> <task>` | (v4.1.0) Pin model profile for the brief phase: `economy` / `balanced` / `premium` / `fable` / `<custom>`. See [Profile system](#profile-system-v410) below. |
|
||||||
|
|
||||||
`/trekbrief` is **always interactive**. There is no foreground/background mode — the interview requires user input.
|
`/trekbrief` is **always interactive**. There is no foreground/background mode — the interview requires user input.
|
||||||
|
|
||||||
|
|
@ -209,6 +195,7 @@ Output:
|
||||||
| **External** | `/trekresearch --external <question>` | Only external research agents (skip codebase analysis) |
|
| **External** | `/trekresearch --external <question>` | Only external research agents (skip codebase analysis) |
|
||||||
| **Foreground** | `/trekresearch --fg <question>` | No-op alias (foreground is default since v2.4.0) |
|
| **Foreground** | `/trekresearch --fg <question>` | No-op alias (foreground is default since v2.4.0) |
|
||||||
| **Profile** | `/trekresearch --profile <name> <question>` | (v4.1.0) Pin model profile for the research phase. See [Profile system](#profile-system-v410). |
|
| **Profile** | `/trekresearch --profile <name> <question>` | (v4.1.0) Pin model profile for the research phase. See [Profile system](#profile-system-v410). |
|
||||||
|
| **Engine** | `/trekresearch --external --engine deep-research <question>` | Delegate the external phase to Claude Code's built-in `/deep-research` workflow; falls back to `swarm` if unavailable. Default `swarm`. |
|
||||||
|
|
||||||
Flags combine: `--project <dir> --external`.
|
Flags combine: `--project <dir> --external`.
|
||||||
|
|
||||||
|
|
@ -236,7 +223,7 @@ Output:
|
||||||
| **Foreground** | `/trekplan --project <dir> --fg` | No-op alias (foreground is default since v2.4.0) |
|
| **Foreground** | `/trekplan --project <dir> --fg` | No-op alias (foreground is default since v2.4.0) |
|
||||||
| **Quick** | `/trekplan --project <dir> --quick` | No agent swarm, lightweight scan only |
|
| **Quick** | `/trekplan --project <dir> --quick` | No agent swarm, lightweight scan only |
|
||||||
| **Decompose** | `/trekplan --decompose plan.md` | Split plan into headless session specs |
|
| **Decompose** | `/trekplan --decompose plan.md` | Split plan into headless session specs |
|
||||||
| **Export** | `/trekplan --export pr plan.md` | PR description, issue comment, or clean markdown |
|
| **Export** | `/trekplan --export headless plan.md` | Legacy alias for `--decompose` (the only remaining export format) |
|
||||||
| **Profile** | `/trekplan --profile <name> --project <dir>` | (v4.1.0) Pin model profile; emitted as `profile:` in plan.md frontmatter. See [Profile system](#profile-system-v410). |
|
| **Profile** | `/trekplan --profile <name> --project <dir>` | (v4.1.0) Pin model profile; emitted as `profile:` in plan.md frontmatter. See [Profile system](#profile-system-v410). |
|
||||||
|
|
||||||
`--brief` or `--project` is **required**. `/trekplan` with no brief exits with an error and a pointer to `/trekbrief`.
|
`--brief` or `--project` is **required**. `/trekplan` with no brief exits with an error and a pointer to `/trekbrief`.
|
||||||
|
|
@ -599,24 +586,65 @@ Claude never pre-generates suggestions in this flow.
|
||||||
|
|
||||||
## The full pipeline
|
## The full pipeline
|
||||||
|
|
||||||
|
Six commands, one pipeline — each step hands a structured artifact to the next (the seven [handover contracts](docs/HANDOVER-CONTRACTS.md)):
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
B["/trekbrief"]
|
||||||
|
R["/trekresearch"]
|
||||||
|
P["/trekplan"]
|
||||||
|
X["/trekexecute"]
|
||||||
|
V["/trekreview"]
|
||||||
|
C["/trekcontinue"]
|
||||||
|
A["architect plugin<br/>(opt-in, not bundled)"]
|
||||||
|
|
||||||
|
B -->|"H1 · brief.md · PUBLIC"| R
|
||||||
|
R -->|"H2 · research/NN-*.md"| P
|
||||||
|
A -.->|"H3 · architecture/overview.md"| P
|
||||||
|
P -->|"H4 · plan.md"| X
|
||||||
|
X -.->|"H5 · progress.json (--resume)"| X
|
||||||
|
X -->|"H7 · .session-state.local.json"| C
|
||||||
|
C -->|"next session"| X
|
||||||
|
X -.->|"git diff"| V
|
||||||
|
V -->|"H6 · review.md (loop back)"| P
|
||||||
```
|
```
|
||||||
/trekbrief /trekresearch /trekplan /trekexecute
|
|
||||||
┌──────────────┐ ┌───────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
|
Solid arrows are the forward pipeline; dashed are conditional/internal edges (`--resume`, the git-diff that feeds review, the opt-in architect input). **H1 (`brief.md`) is the only PUBLIC contract** — the single integration point for any upstream brief producer.
|
||||||
│ Interview │ │ 5 local agents │ │ brief-reviewer │ │ Parse plan │
|
|
||||||
│ ↓ │ │ 4 external agents │ │ ↓ │ │ ↓ │
|
### Agents per phase
|
||||||
│ Intent/Goal │ │ + Gemini bridge │ │ 6-8 exploration │ │ Detect sessions │
|
|
||||||
│ ↓ │ │ ↓ │ │ agents (parallel) │ │ ↓ │
|
Every command orchestrates its agents inline from the main context (all sub-agents are `model: opus`-pinned). `/trekexecute` is the exception — it spawns **no** sub-agents.
|
||||||
│ Research │ │ Triangulation │ │ ↓ │ │ Execute steps │
|
|
||||||
│ topics │ │ ↓ │ │ Opus planning │ │ (verify + manifest │
|
```mermaid
|
||||||
│ ↓ │ → brief → → → → → → → → → → → ↓ │→ │ + checkpoint) │
|
flowchart TB
|
||||||
│ brief.md │ │ research/*.md │ │ plan-critic + │ │ ↓ │
|
subgraph BR["/trekbrief"]
|
||||||
└──────────────┘ └───────────────────┘ │ scope-guardian │ │ Phase 7.5 manifest │
|
BRG["brief-reviewer (gate · ≤3 iter)"]
|
||||||
│ ↓ │ │ audit + 7.6 recovery│
|
end
|
||||||
│ plan.md │ │ ↓ │
|
subgraph RES["/trekresearch · Phase 4 — parallel"]
|
||||||
└─────────────────────┘ │ progress.json + done│
|
RL["LOCAL: architecture-mapper · dependency-tracer<br/>task-finder · git-historian · convention-scanner*"]
|
||||||
└─────────────────────┘
|
RE["EXTERNAL: docs-researcher · community-researcher<br/>security-researcher* · contrarian-researcher* · gemini-bridge*"]
|
||||||
|
end
|
||||||
|
subgraph PL["/trekplan"]
|
||||||
|
PLG["Phase 4b · brief-reviewer (gate)"]
|
||||||
|
PLS["Phase 5 — parallel (6 + 2*): architecture-mapper · dependency-tracer<br/>risk-assessor · task-finder · test-strategist · git-historian<br/>convention-scanner* · research-scout*"]
|
||||||
|
PLC["Phase 9 — parallel · plan-critic · scope-guardian"]
|
||||||
|
PLG --> PLS --> PLC
|
||||||
|
end
|
||||||
|
subgraph EX["/trekexecute · no sub-agents"]
|
||||||
|
EXI["inline step-loop + claude -p worktrees<br/>+ Phase 7.5/7.6 manifest audit (deterministic)"]
|
||||||
|
end
|
||||||
|
subgraph RV["/trekreview"]
|
||||||
|
RVR["Phase 5 — parallel · code-correctness-reviewer · brief-conformance-reviewer*"]
|
||||||
|
RVJ["Phase 6 · review-coordinator (Judge)"]
|
||||||
|
RVR --> RVJ
|
||||||
|
end
|
||||||
|
BR --> RES --> PL --> EX --> RV
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`* = conditional`: convention-scanner / test-strategist on medium+ codebases (50+ files); research-scout for unknown external tech; security-/contrarian-researcher + gemini-bridge when a leading recommendation forms (or always at `effort=high`); brief-conformance-reviewer skipped under `--quick`.
|
||||||
|
|
||||||
|
> **Which Claude Code primitive each phase uses — and the alternatives considered (Workflow substrate, delegated orchestrator, dormant synthesis-agent)** → see [docs/architecture.md §Primitives per step](docs/architecture.md#primitives-per-step-decision-matrix).
|
||||||
|
|
||||||
All artifacts live under `.claude/projects/{YYYY-MM-DD}-{slug}/`.
|
All artifacts live under `.claude/projects/{YYYY-MM-DD}-{slug}/`.
|
||||||
|
|
||||||
An opt-in upstream architect plugin (not bundled) can insert a Claude-Code-specific architecture-matching step between research and plan — `/trekplan` auto-discovers its `architecture/overview.md` output as priors when present.
|
An opt-in upstream architect plugin (not bundled) can insert a Claude-Code-specific architecture-matching step between research and plan — `/trekplan` auto-discovers its `architecture/overview.md` output as priors when present.
|
||||||
|
|
@ -751,13 +779,14 @@ An optional architect step between research and plan was previously available vi
|
||||||
|
|
||||||
## Profile system (v4.1.0)
|
## Profile system (v4.1.0)
|
||||||
|
|
||||||
Three built-in model profiles plus operator-defined `<custom>.yaml` (drop in `lib/profiles/`). Each profile pins `phase_models` for the six pipeline phases. The active profile is recorded in plan.md frontmatter as `profile: <name>` and emitted to JSONL stats for cost-attribution.
|
Four built-in model profiles plus operator-defined `<custom>.yaml` (drop in `lib/profiles/`). Each profile pins `phase_models` for the six pipeline phases. The active profile is recorded in plan.md frontmatter as `profile: <name>` and emitted to JSONL stats for cost-attribution.
|
||||||
|
|
||||||
| Profile | Brief | Research | Plan | Execute | Review | Continue | Use case |
|
| Profile | Brief | Research | Plan | Execute | Review | Continue | Use case |
|
||||||
|---------|-------|----------|------|---------|--------|----------|----------|
|
|---------|-------|----------|------|---------|--------|----------|----------|
|
||||||
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | Lowest cost; high-confidence small-scope tasks (opt-in via `--profile economy`) |
|
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | ⚠ **Experimental** (uncalibrated Jaccard floor) — lowest cost; high-confidence small-scope tasks (opt-in via `--profile economy`) |
|
||||||
| `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (opt-in via `--profile balanced`) |
|
| `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (opt-in via `--profile balanced`) |
|
||||||
| `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase (default since the 2026-05-13 operator decision) |
|
| `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase (default since the 2026-05-13 operator decision) |
|
||||||
|
| `fable` | fable | fable | fable | fable | fable | fable | Max quality — Fable 5 (Mythos-class, above Opus) on every phase (opt-in via `--profile fable`); reasoning effort inherits from the session |
|
||||||
|
|
||||||
Lookup order:
|
Lookup order:
|
||||||
|
|
||||||
|
|
@ -782,9 +811,9 @@ Default JSONL stats stream (`${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl`) is unchan
|
||||||
|
|
||||||
## Cost profile
|
## Cost profile
|
||||||
|
|
||||||
The default `premium` profile runs **Opus on every phase** — the orchestrator (one per command), the exploration and review swarms (5–10 sub-agents per command, all `model: opus`-pinned in `agents/*.md`), and the executor (one per plan session). The model is **uniform per phase**: there is no "Opus orchestrates, Sonnet runs the swarms" split — a phase resolves to one model and both the orchestrator and its sub-agents use it. For cheaper runs, opt into `--profile balanced` (Sonnet on brief/research/execute/continue, Opus on plan + review) or `--profile economy` (Sonnet everywhere). Per-command cost is published in `${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl` if you want exact numbers.
|
The default `premium` profile runs **Opus on every phase** of the pipeline's agent work — the exploration and review swarms (5–10 sub-agents per command; spawn sites inject the composed brief > profile > frontmatter resolution, with `agents/*.md` `model: opus` pins as the fallback) and the executor (one per plan session). The command orchestrator itself is not profile-controlled: as of v5.9, command frontmatter omits `model:`, so the orchestrator follows the session model. For cheaper runs, opt into `--profile balanced` (Sonnet on brief/research/execute/continue, Opus on plan + review) or `--profile economy` (Sonnet everywhere); for maximum quality, `--profile fable` (Fable 5 on every phase). Per-command cost is published in `${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl` if you want exact numbers.
|
||||||
|
|
||||||
The `opus` alias resolves to **Opus 4.8** (default reasoning effort `high`) and `sonnet` to Sonnet 4.6. Note two distinct effort axes that share the word "effort": brief `phase_signals.effort` (low/standard/high) tunes *orchestration shape* — how many agents and passes run — while native `effort:` on selected agents (retrieval at `medium`, adversarial-reasoning at `high`) tunes the *per-spawn reasoning budget*. See [`docs/profiles.md`](docs/profiles.md) § Model & effort axes.
|
The `opus` alias resolves to **Opus 4.8** (default reasoning effort `high`), `sonnet` to Sonnet 4.6, and `fable` to **Fable 5** (Mythos-class, above Opus; default reasoning effort `high` — xhigh requires a session-level setting, see [`docs/profiles.md`](docs/profiles.md)). Note two distinct effort axes that share the word "effort": brief `phase_signals.effort` (low/standard/high) tunes *orchestration shape* — how many agents and passes run — while native `effort:` on selected agents (retrieval at `medium`, adversarial-reasoning at `high`) tunes the *per-spawn reasoning budget*. See [`docs/profiles.md`](docs/profiles.md) § Model & effort axes.
|
||||||
|
|
||||||
For per-profile cost estimates, see [`docs/profiles.md`](docs/profiles.md).
|
For per-profile cost estimates, see [`docs/profiles.md`](docs/profiles.md).
|
||||||
|
|
||||||
|
|
@ -802,6 +831,7 @@ Top-level layout:
|
||||||
```
|
```
|
||||||
trekplan/
|
trekplan/
|
||||||
├── agents/ 24 specialized agents (all `model: opus`-pinned; per-phase model set by `--profile`)
|
├── agents/ 24 specialized agents (all `model: opus`-pinned; per-phase model set by `--profile`)
|
||||||
|
│ └ 21 spawnable (1 dormant: synthesis-agent, Δ≈0) + 3 orchestrator reference docs (not spawned)
|
||||||
├── commands/ 6 slash commands (trekbrief, trekresearch, trekplan, trekexecute, trekreview, trekcontinue) + trekendsession helper
|
├── commands/ 6 slash commands (trekbrief, trekresearch, trekplan, trekexecute, trekreview, trekcontinue) + trekendsession helper
|
||||||
├── templates/ Frontmatter templates for brief, research, plan, session, launch
|
├── templates/ Frontmatter templates for brief, research, plan, session, launch
|
||||||
├── hooks/ 7 hooks (pre-bash, pre-write, session-title, post-bash-stats, pre-compact-flush, post-compact-flush, otel-export)
|
├── hooks/ 7 hooks (pre-bash, pre-write, session-title, post-bash-stats, pre-compact-flush, post-compact-flush, otel-export)
|
||||||
|
|
|
||||||
|
|
@ -3,24 +3,6 @@ name: architecture-mapper
|
||||||
description: |
|
description: |
|
||||||
Use this agent when you need deep architecture analysis of a codebase — structure,
|
Use this agent when you need deep architecture analysis of a codebase — structure,
|
||||||
tech stack, patterns, anti-patterns, and key abstractions.
|
tech stack, patterns, anti-patterns, and key abstractions.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: Voyage exploration phase needs architecture overview
|
|
||||||
user: "/trekplan Add authentication to the API"
|
|
||||||
assistant: "Launching architecture-mapper to analyze codebase structure and patterns."
|
|
||||||
<commentary>
|
|
||||||
Phase 5 of trekplan triggers this agent for every codebase size.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User wants to understand an unfamiliar codebase
|
|
||||||
user: "Map out the architecture of this project"
|
|
||||||
assistant: "I'll use the architecture-mapper agent to analyze the codebase structure."
|
|
||||||
<commentary>
|
|
||||||
Direct architecture analysis request triggers the agent.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
effort: medium
|
effort: medium
|
||||||
color: cyan
|
color: cyan
|
||||||
|
|
@ -104,3 +86,23 @@ Structure your report with clear sections matching the 7 areas above. Include:
|
||||||
- A brief "Architecture Summary" paragraph at the top (3-4 sentences)
|
- A brief "Architecture Summary" paragraph at the top (3-4 sentences)
|
||||||
|
|
||||||
Do NOT include raw file listings — synthesize and organize the information.
|
Do NOT include raw file listings — synthesize and organize the information.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: Voyage exploration phase needs architecture overview
|
||||||
|
user: "/trekplan Add authentication to the API"
|
||||||
|
assistant: "Launching architecture-mapper to analyze codebase structure and patterns."
|
||||||
|
<commentary>
|
||||||
|
Phase 5 of trekplan triggers this agent for every codebase size.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User wants to understand an unfamiliar codebase
|
||||||
|
user: "Map out the architecture of this project"
|
||||||
|
assistant: "I'll use the architecture-mapper agent to analyze the codebase structure."
|
||||||
|
<commentary>
|
||||||
|
Direct architecture analysis request triggers the agent.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -5,24 +5,6 @@ description: |
|
||||||
checks completeness, consistency, testability, scope clarity, and
|
checks completeness, consistency, testability, scope clarity, and
|
||||||
research-plan validity. Catches problems early to avoid wasting tokens on
|
research-plan validity. Catches problems early to avoid wasting tokens on
|
||||||
exploration with a flawed brief.
|
exploration with a flawed brief.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: Voyage runs brief review before exploration
|
|
||||||
user: "/trekplan --project .claude/projects/2026-04-18-notifications"
|
|
||||||
assistant: "Reviewing brief quality before launching exploration agents."
|
|
||||||
<commentary>
|
|
||||||
Orchestrator Phase 1b triggers this agent after the brief is available.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User wants to validate a brief before planning
|
|
||||||
user: "Review this brief for completeness"
|
|
||||||
assistant: "I'll use the brief-reviewer agent to check brief quality."
|
|
||||||
<commentary>
|
|
||||||
Brief review request triggers the agent.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
color: magenta
|
color: magenta
|
||||||
tools: ["Read", "Glob", "Grep"]
|
tools: ["Read", "Glob", "Grep"]
|
||||||
|
|
@ -302,3 +284,23 @@ information that would strengthen the brief. List only if actionable.}
|
||||||
files or technologies exist, but deep code analysis is not your job.
|
files or technologies exist, but deep code analysis is not your job.
|
||||||
- **Research-plan checks are load-bearing.** A brief with `research_status: pending`
|
- **Research-plan checks are load-bearing.** A brief with `research_status: pending`
|
||||||
and missing research files is a scope hazard — flag it as a major risk.
|
and missing research files is a scope hazard — flag it as a major risk.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: Voyage runs brief review before exploration
|
||||||
|
user: "/trekplan --project .claude/projects/2026-04-18-notifications"
|
||||||
|
assistant: "Reviewing brief quality before launching exploration agents."
|
||||||
|
<commentary>
|
||||||
|
Orchestrator Phase 1b triggers this agent after the brief is available.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User wants to validate a brief before planning
|
||||||
|
user: "Review this brief for completeness"
|
||||||
|
assistant: "I'll use the brief-reviewer agent to check brief quality."
|
||||||
|
<commentary>
|
||||||
|
Brief review request triggers the agent.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -4,26 +4,6 @@ description: |
|
||||||
Use this agent when the research task requires practical, real-world experience rather
|
Use this agent when the research task requires practical, real-world experience rather
|
||||||
than official documentation — community sentiment, production war stories, known gotchas,
|
than official documentation — community sentiment, production war stories, known gotchas,
|
||||||
and what developers actually encounter when using a technology.
|
and what developers actually encounter when using a technology.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: trekresearch needs real-world experience data on a database migration
|
|
||||||
user: "/trekresearch What's the real-world experience with migrating from MongoDB to PostgreSQL?"
|
|
||||||
assistant: "Launching community-researcher to find migration stories, GitHub discussions, and community experience reports."
|
|
||||||
<commentary>
|
|
||||||
Official docs won't cover migration regrets or production war stories. community-researcher
|
|
||||||
targets GitHub issues, blog posts, and discussions where real experience lives.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: trekresearch is building a technology comparison
|
|
||||||
user: "/trekresearch Research community sentiment around adopting SvelteKit vs Next.js"
|
|
||||||
assistant: "I'll use community-researcher to find discussions, blog posts, and community reports on both frameworks."
|
|
||||||
<commentary>
|
|
||||||
Framework comparisons live in community discourse, not official docs. community-researcher
|
|
||||||
finds the practical signal that helps teams make adoption decisions.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
color: green
|
color: green
|
||||||
tools: ["WebSearch", "WebFetch", "mcp__tavily__tavily_search", "mcp__tavily__tavily_research"]
|
tools: ["WebSearch", "WebFetch", "mcp__tavily__tavily_search", "mcp__tavily__tavily_research"]
|
||||||
|
|
@ -133,3 +113,25 @@ End with a summary table:
|
||||||
Do not pick a side — report the split.
|
Do not pick a side — report the split.
|
||||||
- **Flag if a "problem" has since been fixed.** Check if the issue/complaint references a
|
- **Flag if a "problem" has since been fixed.** Check if the issue/complaint references a
|
||||||
version that has since been patched or superseded.
|
version that has since been patched or superseded.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: trekresearch needs real-world experience data on a database migration
|
||||||
|
user: "/trekresearch What's the real-world experience with migrating from MongoDB to PostgreSQL?"
|
||||||
|
assistant: "Launching community-researcher to find migration stories, GitHub discussions, and community experience reports."
|
||||||
|
<commentary>
|
||||||
|
Official docs won't cover migration regrets or production war stories. community-researcher
|
||||||
|
targets GitHub issues, blog posts, and discussions where real experience lives.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: trekresearch is building a technology comparison
|
||||||
|
user: "/trekresearch Research community sentiment around adopting SvelteKit vs Next.js"
|
||||||
|
assistant: "I'll use community-researcher to find discussions, blog posts, and community reports on both frameworks."
|
||||||
|
<commentary>
|
||||||
|
Framework comparisons live in community discourse, not official docs. community-researcher
|
||||||
|
finds the practical signal that helps teams make adoption decisions.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -4,26 +4,6 @@ description: |
|
||||||
Use this agent when the research task has an emerging conclusion that needs adversarial
|
Use this agent when the research task has an emerging conclusion that needs adversarial
|
||||||
stress-testing — find counter-evidence, overlooked alternatives, and reasons the leading
|
stress-testing — find counter-evidence, overlooked alternatives, and reasons the leading
|
||||||
answer might be wrong.
|
answer might be wrong.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: trekresearch has found evidence favoring a technology and needs the other side
|
|
||||||
user: "/trekresearch We're leaning toward adopting Kafka for our event streaming needs"
|
|
||||||
assistant: "Launching contrarian-researcher to find the strongest arguments against Kafka and what alternatives might serve better."
|
|
||||||
<commentary>
|
|
||||||
The research equivalent of plan-critic. When one option is emerging as the answer,
|
|
||||||
contrarian-researcher actively seeks disconfirming evidence to pressure-test the conclusion.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: trekresearch is comparing options and needs the downsides of the leading candidate
|
|
||||||
user: "/trekresearch Compare Redis vs Memcached — initial research favors Redis"
|
|
||||||
assistant: "I'll use contrarian-researcher to find the strongest case against Redis and scenarios where Memcached wins."
|
|
||||||
<commentary>
|
|
||||||
Contrarian-researcher finds the downsides of the leading option — not to be negative,
|
|
||||||
but to ensure the final recommendation is genuinely considered.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
effort: high
|
effort: high
|
||||||
color: red
|
color: red
|
||||||
|
|
@ -152,3 +132,25 @@ Followed by a **Verdict** section:
|
||||||
apply to a read-heavy workload. Assess relevance before reporting.
|
apply to a read-heavy workload. Assess relevance before reporting.
|
||||||
- **Check recency.** A problem from 2019 that the project fixed in 2021 is not current
|
- **Check recency.** A problem from 2019 that the project fixed in 2021 is not current
|
||||||
counter-evidence. Flag whether issues are current or historical.
|
counter-evidence. Flag whether issues are current or historical.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: trekresearch has found evidence favoring a technology and needs the other side
|
||||||
|
user: "/trekresearch We're leaning toward adopting Kafka for our event streaming needs"
|
||||||
|
assistant: "Launching contrarian-researcher to find the strongest arguments against Kafka and what alternatives might serve better."
|
||||||
|
<commentary>
|
||||||
|
The research equivalent of plan-critic. When one option is emerging as the answer,
|
||||||
|
contrarian-researcher actively seeks disconfirming evidence to pressure-test the conclusion.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: trekresearch is comparing options and needs the downsides of the leading candidate
|
||||||
|
user: "/trekresearch Compare Redis vs Memcached — initial research favors Redis"
|
||||||
|
assistant: "I'll use contrarian-researcher to find the strongest case against Redis and scenarios where Memcached wins."
|
||||||
|
<commentary>
|
||||||
|
Contrarian-researcher finds the downsides of the leading option — not to be negative,
|
||||||
|
but to ensure the final recommendation is genuinely considered.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -5,24 +5,6 @@ description: |
|
||||||
Produces a structured conventions report covering naming, directory layout,
|
Produces a structured conventions report covering naming, directory layout,
|
||||||
import style, error handling, test patterns, git commit style, and
|
import style, error handling, test patterns, git commit style, and
|
||||||
documentation patterns. Uses concrete examples from the codebase.
|
documentation patterns. Uses concrete examples from the codebase.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: Voyage exploration phase for a medium+ codebase
|
|
||||||
user: "/trekplan Add authentication to the API"
|
|
||||||
assistant: "Launching convention-scanner to discover coding patterns."
|
|
||||||
<commentary>
|
|
||||||
Phase 5 of trekplan triggers this agent for medium+ codebases (50+ files).
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User wants to understand a project's conventions before contributing
|
|
||||||
user: "What are the coding conventions in this project?"
|
|
||||||
assistant: "I'll use the convention-scanner agent to analyze the codebase."
|
|
||||||
<commentary>
|
|
||||||
Direct convention discovery request triggers the agent.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
color: yellow
|
color: yellow
|
||||||
tools: ["Read", "Glob", "Grep", "Bash"]
|
tools: ["Read", "Glob", "Grep", "Bash"]
|
||||||
|
|
@ -159,3 +141,23 @@ Based on existing conventions, new code should:
|
||||||
than scanning everything.
|
than scanning everything.
|
||||||
- **Stay focused.** This is about conventions — not architecture, dependencies, or risks.
|
- **Stay focused.** This is about conventions — not architecture, dependencies, or risks.
|
||||||
Those are handled by other agents.
|
Those are handled by other agents.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: Voyage exploration phase for a medium+ codebase
|
||||||
|
user: "/trekplan Add authentication to the API"
|
||||||
|
assistant: "Launching convention-scanner to discover coding patterns."
|
||||||
|
<commentary>
|
||||||
|
Phase 5 of trekplan triggers this agent for medium+ codebases (50+ files).
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User wants to understand a project's conventions before contributing
|
||||||
|
user: "What are the coding conventions in this project?"
|
||||||
|
assistant: "I'll use the convention-scanner agent to analyze the codebase."
|
||||||
|
<commentary>
|
||||||
|
Direct convention discovery request triggers the agent.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -3,24 +3,6 @@ name: dependency-tracer
|
||||||
description: |
|
description: |
|
||||||
Use this agent when you need to trace import chains, map data flow, or understand
|
Use this agent when you need to trace import chains, map data flow, or understand
|
||||||
how modules connect and what side effects they produce.
|
how modules connect and what side effects they produce.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: Voyage needs to understand module relationships for a task
|
|
||||||
user: "/trekplan Refactor the payment processing pipeline"
|
|
||||||
assistant: "Launching dependency-tracer to map module connections and data flow."
|
|
||||||
<commentary>
|
|
||||||
Phase 5 of trekplan triggers this agent to trace dependencies relevant to the task.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User needs to understand impact of changing a module
|
|
||||||
user: "What would break if I change the User model?"
|
|
||||||
assistant: "I'll use the dependency-tracer agent to trace all dependents of the User model."
|
|
||||||
<commentary>
|
|
||||||
Impact analysis request triggers the agent.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
effort: medium
|
effort: medium
|
||||||
color: blue
|
color: blue
|
||||||
|
|
@ -93,3 +75,23 @@ Structure as:
|
||||||
6. **Risk Flags** — circular deps, tight coupling, hidden side effects
|
6. **Risk Flags** — circular deps, tight coupling, hidden side effects
|
||||||
|
|
||||||
Include file paths and line numbers for every finding.
|
Include file paths and line numbers for every finding.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: Voyage needs to understand module relationships for a task
|
||||||
|
user: "/trekplan Refactor the payment processing pipeline"
|
||||||
|
assistant: "Launching dependency-tracer to map module connections and data flow."
|
||||||
|
<commentary>
|
||||||
|
Phase 5 of trekplan triggers this agent to trace dependencies relevant to the task.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User needs to understand impact of changing a module
|
||||||
|
user: "What would break if I change the User model?"
|
||||||
|
assistant: "I'll use the dependency-tracer agent to trace all dependents of the User model."
|
||||||
|
<commentary>
|
||||||
|
Impact analysis request triggers the agent.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -3,26 +3,6 @@ name: docs-researcher
|
||||||
description: |
|
description: |
|
||||||
Use this agent when the research task requires authoritative information from official
|
Use this agent when the research task requires authoritative information from official
|
||||||
documentation, RFCs, vendor specifications, or Microsoft/Azure documentation.
|
documentation, RFCs, vendor specifications, or Microsoft/Azure documentation.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: trekresearch needs to ground an OAuth2 implementation in official specs
|
|
||||||
user: "/trekresearch Research OAuth2 PKCE flow for our SPA"
|
|
||||||
assistant: "Launching docs-researcher to find the official RFC and vendor documentation for OAuth2 PKCE."
|
|
||||||
<commentary>
|
|
||||||
docs-researcher targets authoritative sources — RFCs, specs, official vendor docs —
|
|
||||||
not community opinions. This is the right agent for protocol and standards questions.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: trekresearch encounters an Azure-specific technology
|
|
||||||
user: "/trekresearch How should we configure Azure Service Bus for our event pipeline?"
|
|
||||||
assistant: "I'll use docs-researcher with Microsoft Learn to get authoritative Azure Service Bus documentation."
|
|
||||||
<commentary>
|
|
||||||
Microsoft/Azure technologies have dedicated MCP tools (microsoft_docs_search,
|
|
||||||
microsoft_docs_fetch) that docs-researcher uses for higher-quality results.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
color: blue
|
color: blue
|
||||||
tools: ["WebSearch", "WebFetch", "Read", "mcp__tavily__tavily_search", "mcp__tavily__tavily_research", "mcp__microsoft-learn__microsoft_docs_search", "mcp__microsoft-learn__microsoft_docs_fetch"]
|
tools: ["WebSearch", "WebFetch", "Read", "mcp__tavily__tavily_search", "mcp__tavily__tavily_research", "mcp__microsoft-learn__microsoft_docs_search", "mcp__microsoft-learn__microsoft_docs_fetch"]
|
||||||
|
|
@ -119,3 +99,25 @@ End with a summary table:
|
||||||
- **Flag conflicts between official sources.** When vendor docs and the spec disagree, report both.
|
- **Flag conflicts between official sources.** When vendor docs and the spec disagree, report both.
|
||||||
- **Stay focused.** Research only what the research question asks. Do not explore tangentially.
|
- **Stay focused.** Research only what the research question asks. Do not explore tangentially.
|
||||||
- **Official sources only.** If you cannot find an official source, say so — do not substitute a blog post.
|
- **Official sources only.** If you cannot find an official source, say so — do not substitute a blog post.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: trekresearch needs to ground an OAuth2 implementation in official specs
|
||||||
|
user: "/trekresearch Research OAuth2 PKCE flow for our SPA"
|
||||||
|
assistant: "Launching docs-researcher to find the official RFC and vendor documentation for OAuth2 PKCE."
|
||||||
|
<commentary>
|
||||||
|
docs-researcher targets authoritative sources — RFCs, specs, official vendor docs —
|
||||||
|
not community opinions. This is the right agent for protocol and standards questions.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: trekresearch encounters an Azure-specific technology
|
||||||
|
user: "/trekresearch How should we configure Azure Service Bus for our event pipeline?"
|
||||||
|
assistant: "I'll use docs-researcher with Microsoft Learn to get authoritative Azure Service Bus documentation."
|
||||||
|
<commentary>
|
||||||
|
Microsoft/Azure technologies have dedicated MCP tools (microsoft_docs_search,
|
||||||
|
microsoft_docs_fetch) that docs-researcher uses for higher-quality results.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -5,25 +5,6 @@ description: |
|
||||||
needed on a technology choice, architectural question, or complex research topic.
|
needed on a technology choice, architectural question, or complex research topic.
|
||||||
Provides triangulation value by running a completely independent research path
|
Provides triangulation value by running a completely independent research path
|
||||||
that can confirm or challenge findings from other agents.
|
that can confirm or challenge findings from other agents.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: trekresearch launches gemini-bridge for an independent second opinion on a technology choice
|
|
||||||
user: "/trekplan Should we use Kafka or NATS for our event streaming layer?"
|
|
||||||
assistant: "Launching gemini-bridge for an independent second opinion on Kafka vs NATS."
|
|
||||||
<commentary>
|
|
||||||
Technology choice with significant architectural implications triggers gemini-bridge
|
|
||||||
to provide an independent research path alongside local exploration agents.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: user wants deep research via Gemini on a complex architectural question
|
|
||||||
user: "Get me a Gemini deep research on event sourcing patterns for distributed systems"
|
|
||||||
assistant: "I'll use the gemini-bridge agent to run a deep research on event sourcing patterns."
|
|
||||||
<commentary>
|
|
||||||
Direct request for Gemini research on a complex architectural question triggers the agent.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
color: magenta
|
color: magenta
|
||||||
tools: ["mcp__gemini-mcp__gemini_deep_research", "mcp__gemini-mcp__gemini_get_research_status", "mcp__gemini-mcp__gemini_get_research_result", "mcp__gemini-mcp__gemini_research_followup"]
|
tools: ["mcp__gemini-mcp__gemini_deep_research", "mcp__gemini-mcp__gemini_get_research_status", "mcp__gemini-mcp__gemini_get_research_result", "mcp__gemini-mcp__gemini_research_followup"]
|
||||||
|
|
@ -147,3 +128,24 @@ and other external agents:*
|
||||||
- **Graceful degradation at every step.** Unavailable tool, failed research, timeout —
|
- **Graceful degradation at every step.** Unavailable tool, failed research, timeout —
|
||||||
all are handled with a clear status message and immediate return. Never leave the
|
all are handled with a clear status message and immediate return. Never leave the
|
||||||
pipeline hanging.
|
pipeline hanging.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: trekresearch launches gemini-bridge for an independent second opinion on a technology choice
|
||||||
|
user: "/trekplan Should we use Kafka or NATS for our event streaming layer?"
|
||||||
|
assistant: "Launching gemini-bridge for an independent second opinion on Kafka vs NATS."
|
||||||
|
<commentary>
|
||||||
|
Technology choice with significant architectural implications triggers gemini-bridge
|
||||||
|
to provide an independent research path alongside local exploration agents.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: user wants deep research via Gemini on a complex architectural question
|
||||||
|
user: "Get me a Gemini deep research on event sourcing patterns for distributed systems"
|
||||||
|
assistant: "I'll use the gemini-bridge agent to run a deep research on event sourcing patterns."
|
||||||
|
<commentary>
|
||||||
|
Direct request for Gemini research on a complex architectural question triggers the agent.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -3,24 +3,6 @@ name: git-historian
|
||||||
description: |
|
description: |
|
||||||
Use this agent to analyze git history for planning context — recent changes,
|
Use this agent to analyze git history for planning context — recent changes,
|
||||||
code ownership, hot files, and active branches relevant to the task.
|
code ownership, hot files, and active branches relevant to the task.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: Voyage exploration phase needs git context
|
|
||||||
user: "/trekplan Refactor the database layer"
|
|
||||||
assistant: "Launching git-historian to check recent changes and ownership of DB code."
|
|
||||||
<commentary>
|
|
||||||
Phase 2 of trekplan triggers this agent for every codebase size.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User wants to understand change history before modifying code
|
|
||||||
user: "Who has been changing the auth module recently?"
|
|
||||||
assistant: "I'll use the git-historian agent to analyze ownership and change patterns."
|
|
||||||
<commentary>
|
|
||||||
Git history analysis request triggers the agent.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
effort: medium
|
effort: medium
|
||||||
color: yellow
|
color: yellow
|
||||||
|
|
@ -122,3 +104,23 @@ Run `git status --short` to check for:
|
||||||
are risks the planner needs to know about.
|
are risks the planner needs to know about.
|
||||||
- **Use relative time.** "2 days ago" is more useful than a raw timestamp.
|
- **Use relative time.** "2 days ago" is more useful than a raw timestamp.
|
||||||
- **Never expose email addresses.** Use author names only.
|
- **Never expose email addresses.** Use author names only.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: Voyage exploration phase needs git context
|
||||||
|
user: "/trekplan Refactor the database layer"
|
||||||
|
assistant: "Launching git-historian to check recent changes and ownership of DB code."
|
||||||
|
<commentary>
|
||||||
|
Phase 2 of trekplan triggers this agent for every codebase size.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User wants to understand change history before modifying code
|
||||||
|
user: "Who has been changing the auth module recently?"
|
||||||
|
assistant: "I'll use the git-historian agent to analyze ownership and change patterns."
|
||||||
|
<commentary>
|
||||||
|
Git history analysis request triggers the agent.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -3,24 +3,6 @@ name: plan-critic
|
||||||
description: |
|
description: |
|
||||||
Use this agent when an implementation plan needs adversarial review — it finds
|
Use this agent when an implementation plan needs adversarial review — it finds
|
||||||
problems, never praises.
|
problems, never praises.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: Voyage adversarial review phase
|
|
||||||
user: "/trekplan Implement WebSocket real-time updates"
|
|
||||||
assistant: "Launching plan-critic to stress-test the implementation plan."
|
|
||||||
<commentary>
|
|
||||||
Phase 9 of trekplan triggers this agent to review the generated plan.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User wants a plan reviewed before execution
|
|
||||||
user: "Review this plan and find problems"
|
|
||||||
assistant: "I'll use the plan-critic agent to perform adversarial review."
|
|
||||||
<commentary>
|
|
||||||
Plan review request triggers the agent.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
effort: high
|
effort: high
|
||||||
color: red
|
color: red
|
||||||
|
|
@ -297,3 +279,23 @@ short, stable id for the finding class (used for exact-match dedup). Emit
|
||||||
|
|
||||||
Be specific. Reference exact plan sections, step numbers, and file paths.
|
Be specific. Reference exact plan sections, step numbers, and file paths.
|
||||||
Never use "generally" or "usually" — cite the specific problem in this specific plan.
|
Never use "generally" or "usually" — cite the specific problem in this specific plan.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: Voyage adversarial review phase
|
||||||
|
user: "/trekplan Implement WebSocket real-time updates"
|
||||||
|
assistant: "Launching plan-critic to stress-test the implementation plan."
|
||||||
|
<commentary>
|
||||||
|
Phase 9 of trekplan triggers this agent to review the generated plan.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User wants a plan reviewed before execution
|
||||||
|
user: "Review this plan and find problems"
|
||||||
|
assistant: "I'll use the plan-critic agent to perform adversarial review."
|
||||||
|
<commentary>
|
||||||
|
Plan review request triggers the agent.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -1,16 +1,6 @@
|
||||||
---
|
---
|
||||||
name: planning-orchestrator
|
name: planning-orchestrator
|
||||||
description: |
|
description: Reference document, not a spawnable capability — documents the /trekplan planning workflow that runs inline in main context (full rationale, CC-2.1.172 history, and phase map in body).
|
||||||
Inline reference (v2.4.0) — documents the planning workflow that
|
|
||||||
/trekplan executes in main context. Historically not spawned as a
|
|
||||||
sub-agent: before Claude Code 2.1.172 the harness did not expose the
|
|
||||||
Agent tool to sub-agents, so a background orchestrator could not spawn
|
|
||||||
the exploration swarm (architecture-mapper, task-finder, plan-critic,
|
|
||||||
etc.). As of CC 2.1.172 sub-agents can spawn sub-agents (up to 5 levels
|
|
||||||
deep), so a delegated-orchestration redesign is under evaluation (see
|
|
||||||
docs/cc-upgrade-2.1.181-decision-matrix.md, W1/CC-26). Until then the
|
|
||||||
/trekplan command orchestrates the phases below directly in the main
|
|
||||||
session.
|
|
||||||
model: opus
|
model: opus
|
||||||
color: cyan
|
color: cyan
|
||||||
tools: ["Read", "Glob", "Grep", "Write", "Edit", "Bash", "TaskCreate", "TaskUpdate"]
|
tools: ["Read", "Glob", "Grep", "Write", "Edit", "Bash", "TaskCreate", "TaskUpdate"]
|
||||||
|
|
|
||||||
|
|
@ -1,15 +1,6 @@
|
||||||
---
|
---
|
||||||
name: research-orchestrator
|
name: research-orchestrator
|
||||||
description: |
|
description: Reference document, not a spawnable capability — documents the /trekresearch workflow that runs inline in main context (full rationale, CC-2.1.172 history, and phase map in body).
|
||||||
Inline reference (v2.4.0) — documents the research workflow that
|
|
||||||
/trekresearch executes in main context. Historically not spawned as
|
|
||||||
a sub-agent: before Claude Code 2.1.172 the harness did not expose the
|
|
||||||
Agent tool to sub-agents, so a background orchestrator could not spawn
|
|
||||||
the research swarm. As of CC 2.1.172 sub-agents can spawn sub-agents
|
|
||||||
(up to 5 levels deep), so a delegated-orchestration redesign is under
|
|
||||||
evaluation (see docs/cc-upgrade-2.1.181-decision-matrix.md, W1/CC-26).
|
|
||||||
Until then the /trekresearch command orchestrates the phases below
|
|
||||||
directly in the main session.
|
|
||||||
model: opus
|
model: opus
|
||||||
color: cyan
|
color: cyan
|
||||||
tools: ["Read", "Glob", "Grep", "Write", "Edit", "Bash"]
|
tools: ["Read", "Glob", "Grep", "Write", "Edit", "Bash"]
|
||||||
|
|
|
||||||
|
|
@ -3,24 +3,6 @@ name: research-scout
|
||||||
description: |
|
description: |
|
||||||
Use this agent when the implementation task involves unfamiliar technologies, external
|
Use this agent when the implementation task involves unfamiliar technologies, external
|
||||||
APIs, or libraries where official documentation and known issues should be checked.
|
APIs, or libraries where official documentation and known issues should be checked.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: Voyage detects external technology in the task
|
|
||||||
user: "/trekplan Integrate Stripe payment processing"
|
|
||||||
assistant: "Launching research-scout to find Stripe documentation and best practices."
|
|
||||||
<commentary>
|
|
||||||
Phase 5 of trekplan conditionally triggers this agent when external tech is detected.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User needs research before implementation
|
|
||||||
user: "Research the best approach for WebSocket scaling"
|
|
||||||
assistant: "I'll use the research-scout agent to find documentation and best practices."
|
|
||||||
<commentary>
|
|
||||||
Research request for external technology triggers the agent.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
color: blue
|
color: blue
|
||||||
tools: ["WebSearch", "WebFetch", "Read"]
|
tools: ["WebSearch", "WebFetch", "Read"]
|
||||||
|
|
@ -118,3 +100,23 @@ End with a summary table:
|
||||||
- **Date everything.** Documentation ages — the reader needs to judge freshness.
|
- **Date everything.** Documentation ages — the reader needs to judge freshness.
|
||||||
- **Flag conflicts.** If official docs and community advice disagree, report both.
|
- **Flag conflicts.** If official docs and community advice disagree, report both.
|
||||||
- **Stay focused.** Research only what the task needs. Do not explore tangentially.
|
- **Stay focused.** Research only what the task needs. Do not explore tangentially.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: Voyage detects external technology in the task
|
||||||
|
user: "/trekplan Integrate Stripe payment processing"
|
||||||
|
assistant: "Launching research-scout to find Stripe documentation and best practices."
|
||||||
|
<commentary>
|
||||||
|
Phase 5 of trekplan conditionally triggers this agent when external tech is detected.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User needs research before implementation
|
||||||
|
user: "Research the best approach for WebSocket scaling"
|
||||||
|
assistant: "I'll use the research-scout agent to find documentation and best practices."
|
||||||
|
<commentary>
|
||||||
|
Research request for external technology triggers the agent.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -1,17 +1,6 @@
|
||||||
---
|
---
|
||||||
name: review-orchestrator
|
name: review-orchestrator
|
||||||
description: |
|
description: Reference document, not a spawnable capability — documents the /trekreview workflow that runs inline in main context (full rationale, CC-2.1.172 history, and phase map in body).
|
||||||
Inline reference (v3.2.0) — documents the review workflow that
|
|
||||||
/trekreview executes in main context. Historically not spawned
|
|
||||||
as a sub-agent: before Claude Code 2.1.172 the harness did not expose
|
|
||||||
the Agent tool to sub-agents, so a background orchestrator could not
|
|
||||||
spawn the reviewer swarm (brief-conformance-reviewer,
|
|
||||||
code-correctness-reviewer, review-coordinator). As of CC 2.1.172
|
|
||||||
sub-agents can spawn sub-agents (up to 5 levels deep), so a
|
|
||||||
delegated-orchestration redesign is under evaluation (see
|
|
||||||
docs/cc-upgrade-2.1.181-decision-matrix.md, W1/CC-26). Until then the
|
|
||||||
/trekreview command orchestrates the phases below directly in the main
|
|
||||||
session.
|
|
||||||
model: opus
|
model: opus
|
||||||
color: red
|
color: red
|
||||||
tools: ["Read", "Glob", "Grep", "Write", "Edit", "Bash", "TaskCreate", "TaskUpdate"]
|
tools: ["Read", "Glob", "Grep", "Write", "Edit", "Bash", "TaskCreate", "TaskUpdate"]
|
||||||
|
|
|
||||||
|
|
@ -3,24 +3,6 @@ name: risk-assessor
|
||||||
description: |
|
description: |
|
||||||
Use this agent when you need to identify risks, edge cases, failure modes, and
|
Use this agent when you need to identify risks, edge cases, failure modes, and
|
||||||
technical debt that could affect an implementation task.
|
technical debt that could affect an implementation task.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: Voyage exploration phase identifies potential risks
|
|
||||||
user: "/trekplan Migrate database from PostgreSQL to MongoDB"
|
|
||||||
assistant: "Launching risk-assessor to identify failure modes and edge cases for this migration."
|
|
||||||
<commentary>
|
|
||||||
Phase 5 of trekplan triggers this agent to find risks before planning begins.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User wants to understand risks before a change
|
|
||||||
user: "What could go wrong with this refactor?"
|
|
||||||
assistant: "I'll use the risk-assessor agent to map risks and failure modes."
|
|
||||||
<commentary>
|
|
||||||
Risk analysis request triggers the agent.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
effort: high
|
effort: high
|
||||||
color: yellow
|
color: yellow
|
||||||
|
|
@ -106,3 +88,23 @@ Produce a prioritized risk list:
|
||||||
**Low** = minor concerns worth noting
|
**Low** = minor concerns worth noting
|
||||||
|
|
||||||
Follow with a narrative section expanding on each Critical and High risk.
|
Follow with a narrative section expanding on each Critical and High risk.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: Voyage exploration phase identifies potential risks
|
||||||
|
user: "/trekplan Migrate database from PostgreSQL to MongoDB"
|
||||||
|
assistant: "Launching risk-assessor to identify failure modes and edge cases for this migration."
|
||||||
|
<commentary>
|
||||||
|
Phase 5 of trekplan triggers this agent to find risks before planning begins.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User wants to understand risks before a change
|
||||||
|
user: "What could go wrong with this refactor?"
|
||||||
|
assistant: "I'll use the risk-assessor agent to map risks and failure modes."
|
||||||
|
<commentary>
|
||||||
|
Risk analysis request triggers the agent.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -3,24 +3,6 @@ name: scope-guardian
|
||||||
description: |
|
description: |
|
||||||
Use this agent when you need to verify that an implementation plan matches its
|
Use this agent when you need to verify that an implementation plan matches its
|
||||||
requirements — catches scope creep and scope gaps.
|
requirements — catches scope creep and scope gaps.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: Voyage adversarial review phase checks scope alignment
|
|
||||||
user: "/trekplan Add caching to the API layer"
|
|
||||||
assistant: "Launching scope-guardian to verify plan matches requirements."
|
|
||||||
<commentary>
|
|
||||||
Phase 9 of trekplan triggers this agent alongside plan-critic.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User wants to verify plan doesn't do too much or too little
|
|
||||||
user: "Does this plan match what I asked for?"
|
|
||||||
assistant: "I'll use the scope-guardian agent to check scope alignment."
|
|
||||||
<commentary>
|
|
||||||
Scope verification request triggers the agent.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
color: magenta
|
color: magenta
|
||||||
tools: ["Read", "Glob", "Grep"]
|
tools: ["Read", "Glob", "Grep"]
|
||||||
|
|
@ -144,3 +126,23 @@ One object per scope-creep / gap / dependency finding. `file`/`line` point at th
|
||||||
plan step (or the brief requirement) the finding concerns; `rule_key` is a short,
|
plan step (or the brief requirement) the finding concerns; `rule_key` is a short,
|
||||||
stable id for the finding class (used for exact-match dedup against plan-critic).
|
stable id for the finding class (used for exact-match dedup against plan-critic).
|
||||||
Emit `"findings": []` when the plan is fully aligned.
|
Emit `"findings": []` when the plan is fully aligned.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: Voyage adversarial review phase checks scope alignment
|
||||||
|
user: "/trekplan Add caching to the API layer"
|
||||||
|
assistant: "Launching scope-guardian to verify plan matches requirements."
|
||||||
|
<commentary>
|
||||||
|
Phase 9 of trekplan triggers this agent alongside plan-critic.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User wants to verify plan doesn't do too much or too little
|
||||||
|
user: "Does this plan match what I asked for?"
|
||||||
|
assistant: "I'll use the scope-guardian agent to check scope alignment."
|
||||||
|
<commentary>
|
||||||
|
Scope verification request triggers the agent.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -3,26 +3,6 @@ name: security-researcher
|
||||||
description: |
|
description: |
|
||||||
Use this agent when the research task requires security investigation of a technology,
|
Use this agent when the research task requires security investigation of a technology,
|
||||||
dependency, or library — CVEs, audit history, supply chain risks, and OWASP relevance.
|
dependency, or library — CVEs, audit history, supply chain risks, and OWASP relevance.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: trekresearch is evaluating whether a dependency is safe to adopt
|
|
||||||
user: "/trekresearch Research whether we should trust the `node-fetch` library"
|
|
||||||
assistant: "Launching security-researcher to check CVE history, supply chain risk, and audit reports for node-fetch."
|
|
||||||
<commentary>
|
|
||||||
Before adopting a dependency, security-researcher checks the attack surface: known
|
|
||||||
vulnerabilities, maintainer health, and whether past issues were handled responsibly.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: trekresearch is assessing the security posture of a technology choice
|
|
||||||
user: "/trekresearch Evaluate the security implications of using JWT for session management"
|
|
||||||
assistant: "I'll use security-researcher to check known JWT vulnerabilities, OWASP guidance, and community security reports."
|
|
||||||
<commentary>
|
|
||||||
Technology choices have security tradeoffs. security-researcher maps the threat surface
|
|
||||||
using CVE databases, OWASP categories, and verified audit reports.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
color: red
|
color: red
|
||||||
tools: ["WebSearch", "WebFetch", "mcp__tavily__tavily_search", "mcp__tavily__tavily_research"]
|
tools: ["WebSearch", "WebFetch", "mcp__tavily__tavily_search", "mcp__tavily__tavily_research"]
|
||||||
|
|
@ -140,3 +120,25 @@ End with an overall security summary table:
|
||||||
risks from incomplete information.
|
risks from incomplete information.
|
||||||
- **Severity matters.** A CVSS 9.8 is not equivalent to a CVSS 3.2 — report scores
|
- **Severity matters.** A CVSS 9.8 is not equivalent to a CVSS 3.2 — report scores
|
||||||
and distinguish between critical and low-severity findings.
|
and distinguish between critical and low-severity findings.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: trekresearch is evaluating whether a dependency is safe to adopt
|
||||||
|
user: "/trekresearch Research whether we should trust the `node-fetch` library"
|
||||||
|
assistant: "Launching security-researcher to check CVE history, supply chain risk, and audit reports for node-fetch."
|
||||||
|
<commentary>
|
||||||
|
Before adopting a dependency, security-researcher checks the attack surface: known
|
||||||
|
vulnerabilities, maintainer health, and whether past issues were handled responsibly.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: trekresearch is assessing the security posture of a technology choice
|
||||||
|
user: "/trekresearch Evaluate the security implications of using JWT for session management"
|
||||||
|
assistant: "I'll use security-researcher to check known JWT vulnerabilities, OWASP guidance, and community security reports."
|
||||||
|
<commentary>
|
||||||
|
Technology choices have security tradeoffs. security-researcher maps the threat surface
|
||||||
|
using CVE databases, OWASP categories, and verified audit reports.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -4,24 +4,6 @@ description: |
|
||||||
Use this agent to decompose an trekplan into self-contained headless sessions.
|
Use this agent to decompose an trekplan into self-contained headless sessions.
|
||||||
Reads a plan file, analyzes step dependencies, groups steps into sessions,
|
Reads a plan file, analyzes step dependencies, groups steps into sessions,
|
||||||
identifies parallelism, and generates session specs + dependency graph + launch script.
|
identifies parallelism, and generates session specs + dependency graph + launch script.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User wants to run a plan across multiple headless sessions
|
|
||||||
user: "/trekplan --decompose .claude/plans/trekplan-2026-04-06-auth-refactor.md"
|
|
||||||
assistant: "Launching session-decomposer to split the plan into headless sessions."
|
|
||||||
<commentary>
|
|
||||||
The --decompose flag triggers this agent to analyze and split the plan.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User has a large plan and wants parallel execution
|
|
||||||
user: "Split this plan into sessions I can run in parallel"
|
|
||||||
assistant: "I'll use the session-decomposer to identify parallel session groups."
|
|
||||||
<commentary>
|
|
||||||
Plan decomposition request for parallel headless execution.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
color: green
|
color: green
|
||||||
tools: ["Read", "Glob", "Grep", "Write"]
|
tools: ["Read", "Glob", "Grep", "Write"]
|
||||||
|
|
@ -310,3 +292,23 @@ After all sessions complete, run:
|
||||||
wrong sequentiality only costs time.
|
wrong sequentiality only costs time.
|
||||||
- **Verify file existence.** Use Glob to confirm that files referenced in the
|
- **Verify file existence.** Use Glob to confirm that files referenced in the
|
||||||
plan actually exist before assigning them to sessions.
|
plan actually exist before assigning them to sessions.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User wants to run a plan across multiple headless sessions
|
||||||
|
user: "/trekplan --decompose .claude/plans/trekplan-2026-04-06-auth-refactor.md"
|
||||||
|
assistant: "Launching session-decomposer to split the plan into headless sessions."
|
||||||
|
<commentary>
|
||||||
|
The --decompose flag triggers this agent to analyze and split the plan.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User has a large plan and wants parallel execution
|
||||||
|
user: "Split this plan into sessions I can run in parallel"
|
||||||
|
assistant: "I'll use the session-decomposer to identify parallel session groups."
|
||||||
|
<commentary>
|
||||||
|
Plan decomposition request for parallel headless execution.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -1,28 +1,6 @@
|
||||||
---
|
---
|
||||||
name: synthesis-agent
|
name: synthesis-agent
|
||||||
description: |
|
description: DORMANT PoC (NW3/S12), NOT wired into /trekplan — a schema-conformant agent that distills the Phase-5/7 exploration outputs into one findings digest; kept as a re-measurable building block (see docs/T1-synthesis-poc-results.md).
|
||||||
Synthesises the trekplan Phase-5/7 exploration outputs (architecture-mapper,
|
|
||||||
dependency-tracer, task-finder, risk-assessor, test-strategist, git-historian,
|
|
||||||
convention-scanner, research) into the structured findings DIGEST that Phase 7
|
|
||||||
currently builds inline. Reads the outputs from disk and returns a single
|
|
||||||
schema-conformant digest — it never spawns sub-agents and never writes files.
|
|
||||||
|
|
||||||
STATUS — DORMANT (NW3 / S12). Built and measured as the CC-26 §6 synthesis-agent
|
|
||||||
PoC, then DECLINED per measurement: delegating only Phase 7 yields Δ main-context
|
|
||||||
≈ 0 because the Phase-5 swarm runs foreground, so its outputs are already resident
|
|
||||||
in main before synthesis. This agent is therefore NOT wired into /trekplan. It is
|
|
||||||
kept as a documented, schema-conformant building block so a future Phase-5
|
|
||||||
redesign (swarm-writes-to-disk / nested orchestrator) can be re-measured cheaply.
|
|
||||||
See docs/T1-synthesis-poc-results.md and docs/T1-cc26-delegated-orchestration.md §6.
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: A future Phase-5-writes-to-disk redesign wants to re-test delegated synthesis.
|
|
||||||
user: "Synthesise the exploration outputs in .claude/projects/X/exploration/ into a digest"
|
|
||||||
assistant: "Launching synthesis-agent to read those outputs and return a findings digest."
|
|
||||||
<commentary>
|
|
||||||
Only reachable deliberately (PoC / future re-measurement) — not from the live pipeline.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
color: cyan
|
color: cyan
|
||||||
tools: ["Read", "Glob", "Grep"]
|
tools: ["Read", "Glob", "Grep"]
|
||||||
|
|
|
||||||
|
|
@ -4,24 +4,6 @@ description: |
|
||||||
Use this agent to find all files, functions, types, and interfaces directly
|
Use this agent to find all files, functions, types, and interfaces directly
|
||||||
related to the planning task. Replaces generic Explore agents with targeted,
|
related to the planning task. Replaces generic Explore agents with targeted,
|
||||||
structured code discovery.
|
structured code discovery.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: Voyage exploration phase needs task-relevant code
|
|
||||||
user: "/trekplan Add authentication to the API"
|
|
||||||
assistant: "Launching task-finder to locate auth-related code, endpoints, and models."
|
|
||||||
<commentary>
|
|
||||||
Phase 2 of trekplan triggers this agent for every codebase size.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User wants to find code related to a specific feature
|
|
||||||
user: "Find all code related to payment processing"
|
|
||||||
assistant: "I'll use the task-finder agent to locate payment-related code."
|
|
||||||
<commentary>
|
|
||||||
Direct code discovery request triggers the agent.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
effort: medium
|
effort: medium
|
||||||
color: green
|
color: green
|
||||||
|
|
@ -146,3 +128,23 @@ Structure your report using three tiers:
|
||||||
- **Stay focused on the task.** Do not inventory the entire codebase — only what
|
- **Stay focused on the task.** Do not inventory the entire codebase — only what
|
||||||
is relevant to implementing the specific task.
|
is relevant to implementing the specific task.
|
||||||
- **Never read file contents that look like secrets or credentials.**
|
- **Never read file contents that look like secrets or credentials.**
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: Voyage exploration phase needs task-relevant code
|
||||||
|
user: "/trekplan Add authentication to the API"
|
||||||
|
assistant: "Launching task-finder to locate auth-related code, endpoints, and models."
|
||||||
|
<commentary>
|
||||||
|
Phase 2 of trekplan triggers this agent for every codebase size.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User wants to find code related to a specific feature
|
||||||
|
user: "Find all code related to payment processing"
|
||||||
|
assistant: "I'll use the task-finder agent to locate payment-related code."
|
||||||
|
<commentary>
|
||||||
|
Direct code discovery request triggers the agent.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -3,24 +3,6 @@ name: test-strategist
|
||||||
description: |
|
description: |
|
||||||
Use this agent when you need to design a test strategy for an implementation task —
|
Use this agent when you need to design a test strategy for an implementation task —
|
||||||
discovers existing patterns, maps coverage gaps, and recommends what tests to write.
|
discovers existing patterns, maps coverage gaps, and recommends what tests to write.
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: Voyage exploration phase for medium+ codebase
|
|
||||||
user: "/trekplan Add rate limiting to the API"
|
|
||||||
assistant: "Launching test-strategist to analyze existing test patterns and design test coverage."
|
|
||||||
<commentary>
|
|
||||||
Phase 5 of trekplan triggers this agent for medium and large codebases.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
|
|
||||||
<example>
|
|
||||||
Context: User wants to know how to test a feature
|
|
||||||
user: "What tests should I write for this new feature?"
|
|
||||||
assistant: "I'll use the test-strategist agent to analyze existing patterns and recommend tests."
|
|
||||||
<commentary>
|
|
||||||
Test planning request triggers the agent.
|
|
||||||
</commentary>
|
|
||||||
</example>
|
|
||||||
model: opus
|
model: opus
|
||||||
color: green
|
color: green
|
||||||
tools: ["Read", "Glob", "Grep", "Bash"]
|
tools: ["Read", "Glob", "Grep", "Bash"]
|
||||||
|
|
@ -95,3 +77,23 @@ For each test, provide:
|
||||||
5. **Test Dependencies** — fixtures, mocks, or setup code to create first
|
5. **Test Dependencies** — fixtures, mocks, or setup code to create first
|
||||||
|
|
||||||
Do NOT write test code. Describe what each test should verify and which patterns to follow.
|
Do NOT write test code. Describe what each test should verify and which patterns to follow.
|
||||||
|
|
||||||
|
## When to use — examples
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: Voyage exploration phase for medium+ codebase
|
||||||
|
user: "/trekplan Add rate limiting to the API"
|
||||||
|
assistant: "Launching test-strategist to analyze existing test patterns and design test coverage."
|
||||||
|
<commentary>
|
||||||
|
Phase 5 of trekplan triggers this agent for medium and large codebases.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: User wants to know how to test a feature
|
||||||
|
user: "What tests should I write for this new feature?"
|
||||||
|
assistant: "I'll use the test-strategist agent to analyze existing patterns and recommend tests."
|
||||||
|
<commentary>
|
||||||
|
Test planning request triggers the agent.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,6 @@
|
||||||
name: trekbrief
|
name: trekbrief
|
||||||
description: Interactive interview that produces a task brief with explicit research plan. Feeds /trekresearch and /trekplan. Optionally orchestrates the full pipeline end-to-end.
|
description: Interactive interview that produces a task brief with explicit research plan. Feeds /trekresearch and /trekplan. Optionally orchestrates the full pipeline end-to-end.
|
||||||
argument-hint: "[--quick] <task description>"
|
argument-hint: "[--quick] <task description>"
|
||||||
model: opus
|
|
||||||
allowed-tools: Agent, Read, Glob, Grep, Write, Edit, Bash, AskUserQuestion
|
allowed-tools: Agent, Read, Glob, Grep, Write, Edit, Bash, AskUserQuestion
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -367,13 +366,14 @@ in the question body so the operator sees why it was picked.
|
||||||
### The loop — 4 tier-coupled AskUserQuestion calls
|
### The loop — 4 tier-coupled AskUserQuestion calls
|
||||||
|
|
||||||
Loop over `[research, plan, execute, review]` in order. For each phase,
|
Loop over `[research, plan, execute, review]` in order. For each phase,
|
||||||
issue one `AskUserQuestion` with 3 options:
|
issue one `AskUserQuestion` with 4 options:
|
||||||
|
|
||||||
| Option | Maps to phase_signals entry |
|
| Option | Maps to phase_signals entry |
|
||||||
|--------|----------------------------|
|
|--------|----------------------------|
|
||||||
| **Low effort** | `{phase: <name>, effort: low, model: sonnet}` |
|
| **Low effort** | `{phase: <name>, effort: low, model: sonnet}` |
|
||||||
| **Standard (default)** | `{phase: <name>, effort: standard}` *(model omitted — composition falls through to profile)* |
|
| **Standard (default)** | `{phase: <name>, effort: standard}` *(model omitted — composition falls through to profile)* |
|
||||||
| **High effort** | `{phase: <name>, effort: high, model: opus}` |
|
| **High effort** | `{phase: <name>, effort: high, model: opus}` |
|
||||||
|
| **Fable (max quality)** | `{phase: <name>, effort: high, model: fable}` |
|
||||||
|
|
||||||
The proposed tier per phase (from the default-derivation heuristic) MUST be
|
The proposed tier per phase (from the default-derivation heuristic) MUST be
|
||||||
labelled `(default)` in the option list so the operator can one-click
|
labelled `(default)` in the option list so the operator can one-click
|
||||||
|
|
@ -384,6 +384,15 @@ The mapping table is canonical:
|
||||||
- `low → {effort: low, model: sonnet}` (force sonnet for the low-cost path)
|
- `low → {effort: low, model: sonnet}` (force sonnet for the low-cost path)
|
||||||
- `standard → {effort: standard}` (model omitted; composition rule resolves via profile)
|
- `standard → {effort: standard}` (model omitted; composition rule resolves via profile)
|
||||||
- `high → {effort: high, model: opus}` (force opus for the high-confidence path)
|
- `high → {effort: high, model: opus}` (force opus for the high-confidence path)
|
||||||
|
- `fable → {effort: high, model: fable}` (force Fable 5 for the max-quality path)
|
||||||
|
|
||||||
|
The fable tier reuses `effort: high` semantics — full swarm, contrarian +
|
||||||
|
gemini always-on; `EFFORT_LEVELS` is unchanged (Voyage effort is orchestration
|
||||||
|
shape, not model reasoning effort). Model reasoning effort is inherited from
|
||||||
|
the session: Fable 5's default effort is `high`, NOT xhigh. To run xhigh, the
|
||||||
|
operator sets it at session level via `/effort xhigh`, the `effortLevel`
|
||||||
|
setting, or `CLAUDE_CODE_EFFORT_LEVEL` — switching model resets effort to the
|
||||||
|
model default, so it does not follow the model.
|
||||||
|
|
||||||
### Force-stop handling
|
### Force-stop handling
|
||||||
|
|
||||||
|
|
@ -891,7 +900,7 @@ Never let stats failures block the workflow.
|
||||||
## Profile (v4.1)
|
## Profile (v4.1)
|
||||||
|
|
||||||
Accepts `--profile <name>` where `<name>` is one of `economy`, `balanced`,
|
Accepts `--profile <name>` where `<name>` is one of `economy`, `balanced`,
|
||||||
`premium`, or a custom profile under `voyage-profiles/`. Default: `premium`.
|
`premium`, `fable`, or a custom profile under `voyage-profiles/`. Default: `premium`.
|
||||||
|
|
||||||
Resolution order (per `lib/profiles/resolver.mjs`):
|
Resolution order (per `lib/profiles/resolver.mjs`):
|
||||||
1. `--profile` flag (source: `flag`)
|
1. `--profile` flag (source: `flag`)
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,6 @@
|
||||||
name: trekcontinue
|
name: trekcontinue
|
||||||
description: Resume the next session in a multi-session trekplan project. Reads .session-state.local.json and immediately begins the next session.
|
description: Resume the next session in a multi-session trekplan project. Reads .session-state.local.json and immediately begins the next session.
|
||||||
argument-hint: "[<project-dir> | --help]"
|
argument-hint: "[<project-dir> | --help]"
|
||||||
model: opus
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Ultracontinue Local v1.0
|
# Ultracontinue Local v1.0
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,6 @@
|
||||||
name: trekendsession
|
name: trekendsession
|
||||||
description: Mark the current session as complete and write session-state pointing at the next session. Helper for informal multi-session flows.
|
description: Mark the current session as complete and write session-state pointing at the next session. Helper for informal multi-session flows.
|
||||||
argument-hint: "<next-brief-path> <next-label> | --help"
|
argument-hint: "<next-brief-path> <next-label> | --help"
|
||||||
model: opus
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Voyage End-Session Local v1.0
|
# Voyage End-Session Local v1.0
|
||||||
|
|
@ -91,16 +90,16 @@ want an interactive flow, use `/trekcontinue --help` to see the full pipeline.
|
||||||
|
|
||||||
## Phase 3 — Atomically write `.session-state.local.json` + sibling NEXT-SESSION-PROMPT.local.md
|
## Phase 3 — Atomically write `.session-state.local.json` + sibling NEXT-SESSION-PROMPT.local.md
|
||||||
|
|
||||||
Write `<project-dir>/.session-state.local.json` with the schema-v1 object:
|
Write `{project_dir}/.session-state.local.json` with the schema-v1 object:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"schema_version": 1,
|
"schema_version": 1,
|
||||||
"project": "<project-dir>",
|
"project": "{project_dir}",
|
||||||
"next_session_brief_path": "<arg 1>",
|
"next_session_brief_path": "{arg 1}",
|
||||||
"next_session_label": "<arg 2>",
|
"next_session_label": "{arg 2}",
|
||||||
"status": "in_progress",
|
"status": "in_progress",
|
||||||
"updated_at": "<now, ISO-8601>"
|
"updated_at": "{now, ISO-8601}"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -115,14 +114,22 @@ Under `node --input-type=module -e "<script>" arg1 arg2 arg3`, Node sets
|
||||||
|
|
||||||
This phase ALSO writes a sibling `NEXT-SESSION-PROMPT.local.md` in the
|
This phase ALSO writes a sibling `NEXT-SESSION-PROMPT.local.md` in the
|
||||||
project directory with YAML frontmatter (`produced_by: trekendsession`,
|
project directory with YAML frontmatter (`produced_by: trekendsession`,
|
||||||
`produced_at: <ISO-8601>`, `project: <project-dir>`). Both files are written
|
`produced_at: {ISO-8601}`, `project: {project_dir}`). Both files are written
|
||||||
in a single ESM block so the writes succeed or fail together:
|
in a single ESM block so the writes succeed or fail together.
|
||||||
|
|
||||||
|
Run the block below via the Bash tool at runtime, substituting the resolved
|
||||||
|
values for the `{curly}` placeholders (Phase 1 gives `{project_dir}`, Phase 2
|
||||||
|
gives `{next_brief_path}` and `{next_label}`). This is NOT an eager-exec
|
||||||
|
block — the values do not exist at command-load time. The import path must
|
||||||
|
stay absolute via `${CLAUDE_PLUGIN_ROOT}` — your Bash cwd is the user's
|
||||||
|
repo, not the plugin root, so a cwd-relative import throws
|
||||||
|
`ERR_MODULE_NOT_FOUND`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
!`node --input-type=module -e "
|
node --input-type=module -e "
|
||||||
import path from 'node:path';
|
import path from 'node:path';
|
||||||
import { writeFileSync } from 'node:fs';
|
import { writeFileSync } from 'node:fs';
|
||||||
import { atomicWriteJson } from './lib/util/atomic-write.mjs';
|
import { atomicWriteJson } from '${CLAUDE_PLUGIN_ROOT}/lib/util/atomic-write.mjs';
|
||||||
const [, dir, brief, label] = process.argv;
|
const [, dir, brief, label] = process.argv;
|
||||||
const now = new Date().toISOString();
|
const now = new Date().toISOString();
|
||||||
const stateObj = { schema_version: 1, project: dir, next_session_brief_path: brief, next_session_label: label, status: 'in_progress', updated_at: now };
|
const stateObj = { schema_version: 1, project: dir, next_session_brief_path: brief, next_session_label: label, status: 'in_progress', updated_at: now };
|
||||||
|
|
@ -133,26 +140,28 @@ const promptBody = '---\\nproduced_by: trekendsession\\nproduced_at: ' + now + '
|
||||||
writeFileSync(promptFile, promptBody);
|
writeFileSync(promptFile, promptBody);
|
||||||
console.log(stateFile);
|
console.log(stateFile);
|
||||||
console.log(promptFile);
|
console.log(promptFile);
|
||||||
" '<project-dir>' '<next-brief-path>' '<next-label>'`
|
" '{project_dir}' '{next_brief_path}' '{next_label}'
|
||||||
```
|
```
|
||||||
|
|
||||||
## Phase 4 — Validate + narrate
|
## Phase 4 — Validate + narrate
|
||||||
|
|
||||||
Validate the freshly-written state file:
|
Validate the freshly-written state file via the Bash tool at runtime,
|
||||||
|
substituting the resolved `{project_dir}` (NOT eager-exec — the file does
|
||||||
|
not exist at command-load time):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
!`node lib/validators/session-state-validator.mjs --json <project-dir>/.session-state.local.json`
|
node ${CLAUDE_PLUGIN_ROOT}/lib/validators/session-state-validator.mjs --json {project_dir}/.session-state.local.json
|
||||||
```
|
```
|
||||||
|
|
||||||
If `valid: true`, print the success block matching `/trekcontinue` Phase 3
|
If `valid: true`, print the success block matching `/trekcontinue` Phase 3
|
||||||
narration (SC-8 cross-project consistency — same template both sides):
|
narration (SC-8 cross-project consistency — same template both sides):
|
||||||
|
|
||||||
```
|
```
|
||||||
Session state written: <project-dir>/.session-state.local.json
|
Session state written: {project_dir}/.session-state.local.json
|
||||||
|
|
||||||
Project: <project-dir>
|
Project: {project_dir}
|
||||||
Next session: <next-label>
|
Next session: {next_label}
|
||||||
Brief: <next-brief-path>
|
Brief: {next_brief_path}
|
||||||
|
|
||||||
In a fresh Claude session, run /trekcontinue to resume.
|
In a fresh Claude session, run /trekcontinue to resume.
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,6 @@
|
||||||
name: trekexecute
|
name: trekexecute
|
||||||
description: Disciplined plan executor — single-session or multi-session with parallel orchestration, failure recovery, and headless support
|
description: Disciplined plan executor — single-session or multi-session with parallel orchestration, failure recovery, and headless support
|
||||||
argument-hint: "[--project <dir>] [--fg | --resume | --dry-run | --validate | --step N | --session N] [plan.md]"
|
argument-hint: "[--project <dir>] [--fg | --resume | --dry-run | --validate | --step N | --session N] [plan.md]"
|
||||||
model: opus
|
|
||||||
allowed-tools: Read, Write, Edit, Bash, Glob, Grep, AskUserQuestion
|
allowed-tools: Read, Write, Edit, Bash, Glob, Grep, AskUserQuestion
|
||||||
disallowed-tools: Agent, TeamCreate
|
disallowed-tools: Agent, TeamCreate
|
||||||
---
|
---
|
||||||
|
|
@ -850,12 +849,24 @@ progress file.
|
||||||
"steps": {
|
"steps": {
|
||||||
"1": { "status": "pending", "attempts": 0, "error": null, "completed_at": null, "commit": null }
|
"1": { "status": "pending", "attempts": 0, "error": null, "completed_at": null, "commit": null }
|
||||||
},
|
},
|
||||||
|
"iterations_remaining": 25,
|
||||||
"entry_condition_checked": false,
|
"entry_condition_checked": false,
|
||||||
"exit_condition_checked": false,
|
"exit_condition_checked": false,
|
||||||
"summary": null
|
"summary": null
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**`iterations_remaining` (v1.7+, additive-optional):** the remaining
|
||||||
|
recovery/retry budget against the global hard cap (unit = **iterations**, per the
|
||||||
|
brief — token/cost budgeting is out of scope). Seeded to
|
||||||
|
`TREKEXECUTE_MAX_RECOVERY_ITERATIONS` (default 25) at start. **Single writer:** the
|
||||||
|
executor decrements it by 1 on **every recovery/retry iteration** — each step
|
||||||
|
attempt beyond the first, and each Phase 7.6 recovery dispatch. **Backfill-on-absent:**
|
||||||
|
when resuming a legacy `progress.json` that lacks the field, seed it to the cap
|
||||||
|
rather than hard-failing (forward/backward-compatible; shape-validated by
|
||||||
|
`progress-validator.mjs` — non-negative integer, never required-top). The field is
|
||||||
|
present in the spec at minimum — there is **no `(if wired)` conditional**.
|
||||||
|
|
||||||
### Mode-specific behavior
|
### Mode-specific behavior
|
||||||
|
|
||||||
**mode = execute (fresh):**
|
**mode = execute (fresh):**
|
||||||
|
|
@ -1228,6 +1239,38 @@ Record in progress file:
|
||||||
- `manifest_audit.status = "pass" | "drift"`
|
- `manifest_audit.status = "pass" | "drift"`
|
||||||
- `manifest_audit.drift_details = [{check, expected, actual}, ...]`
|
- `manifest_audit.drift_details = [{check, expected, actual}, ...]`
|
||||||
|
|
||||||
|
### Machine-verifiable completion gate (stop-signal contract)
|
||||||
|
|
||||||
|
> **Sealed inline so the termination rule survives even when the executor
|
||||||
|
> model is the very thing it constrains.**
|
||||||
|
>
|
||||||
|
> This is the `machine-verifiable completion gate` (the `stop-signal
|
||||||
|
> contract`): the executor may emit `result: completed` ONLY when BOTH hold —
|
||||||
|
>
|
||||||
|
> 1. **Objective predicate** — the **Phase 7.5** manifest audit (above) PASSED,
|
||||||
|
> re-verified directly from the filesystem + git log, ignoring the executor's
|
||||||
|
> own per-step bookkeeping. "The audit wins"; a self-reported `completed` over
|
||||||
|
> a drifting audit is OVERRIDDEN to `partial` (Phase 7.5, `:1221`).
|
||||||
|
> 2. **Explicit stop-signal** — a signal *distinct from the executor's
|
||||||
|
> self-assessment* is satisfied: the plan's Verify command(s) **passed (exit
|
||||||
|
> 0)** / lint clean / an explicit **DONE token emitted AFTER the Phase 7.5
|
||||||
|
> audit ran** (never before, never in place of it). A transcript that merely
|
||||||
|
> "feels done" is NOT a stop-signal — victory-declaration bias is the exact
|
||||||
|
> failure mode this gate exists to defeat.
|
||||||
|
> 3. **Counter liveness (deterministic, defeats "never decremented")** — the gate
|
||||||
|
> reconciles the budget counter against the ground-truth counters the Phase 7.5
|
||||||
|
> audit already derives:
|
||||||
|
> `iterations_remaining == TREKEXECUTE_MAX_RECOVERY_ITERATIONS − (recovery_depth + Σ steps.*.attempts-beyond-first)`.
|
||||||
|
> A field left un-decremented (or tampered) fails this reconciliation and the
|
||||||
|
> result is OVERRIDDEN `completed → partial`, exactly as the audit-override
|
||||||
|
> already does (`:1221`). So the counter's liveness is auditable at the gate,
|
||||||
|
> not merely asserted in prose.
|
||||||
|
>
|
||||||
|
> Termination authority therefore rests on the deterministic audit + an
|
||||||
|
> objective stop-signal, never on the model's narrative. Native `/goal` (when
|
||||||
|
> capability-probed and wired) is an OPTIONAL accelerator only: transcript-only,
|
||||||
|
> it cannot run the real check, so it can never be the sole gate.
|
||||||
|
|
||||||
## Phase 7.6 — Recovery dispatch (multi-session parent context only)
|
## Phase 7.6 — Recovery dispatch (multi-session parent context only)
|
||||||
|
|
||||||
**Preconditions:**
|
**Preconditions:**
|
||||||
|
|
@ -1274,6 +1317,38 @@ Status: partial (recovery_depth=2, escalated to user)
|
||||||
|
|
||||||
Do NOT dispatch a third recovery. Report to the user.
|
Do NOT dispatch a third recovery. Report to the user.
|
||||||
|
|
||||||
|
### Iteration caps + global recovery/retry budget
|
||||||
|
|
||||||
|
Every recovery/retry loop in this executor has a documented numeric bound.
|
||||||
|
These are **three distinct axes** — not one nested counter — plus a separate
|
||||||
|
child turn cap:
|
||||||
|
|
||||||
|
- **Per-step retry cap** — `attempts ≤ 3` (initial + **maximum 2 retries**;
|
||||||
|
Hard Rule 5 / Sub-step E). Bounds the retries of a *single* step.
|
||||||
|
- **Per-session recovery cap** — `recovery_depth < 2` (Phase 7.6 hard cap).
|
||||||
|
Bounds how deep recovery *dispatch* nests.
|
||||||
|
- **Global recovery/retry budget** — a new env-overridable constant
|
||||||
|
`TREKEXECUTE_MAX_RECOVERY_ITERATIONS` (default **25**) — the aggregate ceiling
|
||||||
|
on the **total count of recovery + retry iterations across the whole
|
||||||
|
execution**. This is the case the two caps above do NOT bound: many steps each
|
||||||
|
retrying up to 3× can sum unbounded. It primarily backstops the **retry
|
||||||
|
aggregate** (recovery dispatch is already hard-capped at 2). Surfaced and
|
||||||
|
enforced via the `iterations_remaining` counter (Phase 3 schema; decremented
|
||||||
|
on every recovery/retry iteration; reconciled at the Phase 7.5 completion
|
||||||
|
gate).
|
||||||
|
- **Child turn cap** — `--max-turns` (`TREKEXECUTE_MAX_TURNS`, default 50)
|
||||||
|
bounds model turns inside a single `-p` child; a *different* axis from the
|
||||||
|
three above.
|
||||||
|
|
||||||
|
Default **25** is an independent value from the cap research (practitioner range
|
||||||
|
10–50; OWASP Agentic ASI08 financial-DoS), tuned down from the research ~50
|
||||||
|
starting point for a tighter backstop. It is **not** a reconciliation with
|
||||||
|
`--max-turns` (50) — the axes are incomparable. Operators may override:
|
||||||
|
`TREKEXECUTE_MAX_RECOVERY_ITERATIONS=10 /trekexecute --project ...`. The
|
||||||
|
`effort=='low'` single-foreground-loop path and the wave path are both subject to
|
||||||
|
this aggregate ceiling; it is an execution-spec ceiling (enforced via the
|
||||||
|
`iterations_remaining` counter), not a code-loop edit.
|
||||||
|
|
||||||
## Phase 8 — Final report
|
## Phase 8 — Final report
|
||||||
|
|
||||||
Always produce a final report.
|
Always produce a final report.
|
||||||
|
|
@ -1467,6 +1542,7 @@ To resume: /trekexecute --resume {path}
|
||||||
"drift_details": [],
|
"drift_details": [],
|
||||||
"recovery_dispatched": false,
|
"recovery_dispatched": false,
|
||||||
"recovery_depth": 0,
|
"recovery_depth": 0,
|
||||||
|
"iterations_remaining": 25,
|
||||||
"legacy_plan": false,
|
"legacy_plan": false,
|
||||||
"progress_file": "{path}"
|
"progress_file": "{path}"
|
||||||
}
|
}
|
||||||
|
|
@ -1500,7 +1576,7 @@ Never let stats failures block the workflow.
|
||||||
## Profile (v4.1)
|
## Profile (v4.1)
|
||||||
|
|
||||||
Accepts `--profile <name>` where `<name>` is `economy`, `balanced`, `premium`,
|
Accepts `--profile <name>` where `<name>` is `economy`, `balanced`, `premium`,
|
||||||
or a custom profile under `voyage-profiles/`. Default: `premium`.
|
`fable`, or a custom profile under `voyage-profiles/`. Default: `premium`.
|
||||||
|
|
||||||
Resolution order (per `lib/profiles/resolver.mjs`):
|
Resolution order (per `lib/profiles/resolver.mjs`):
|
||||||
1. `--profile` flag (source: `flag`)
|
1. `--profile` flag (source: `flag`)
|
||||||
|
|
@ -1532,12 +1608,25 @@ model_for_phase = brief.phase_signals[<phase>]?.model ?? profile.phase_models[
|
||||||
```
|
```
|
||||||
|
|
||||||
The brief signal wins per-phase when present; the profile fills any
|
The brief signal wins per-phase when present; the profile fills any
|
||||||
gaps. Composition is mechanically resolved via
|
gaps. Both fields are mechanically resolved by the single composed CLI,
|
||||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/phase-signal-resolver.mjs`
|
invoked in Phase 2.4 alongside the sequencing-gate brief-validator call:
|
||||||
invoked in Phase 2.4; the resolved JSON is captured as `phase_signal_result`
|
|
||||||
and consumed when picking the orchestration model + parallel-wave
|
```bash
|
||||||
strategy. The resolver controls only the orchestrator — sub-agents read
|
# v5.9 — composed phase-model resolution (brief > profile > default) for the
|
||||||
`model:` from their own `agents/*.md` frontmatter (still pinned to `opus`).
|
# execute phase. ONE call returns {effort, model, source}; captured as
|
||||||
|
# phase_signal_result. Append --profile {profile} when the operator passed
|
||||||
|
# --profile.
|
||||||
|
node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model --phase execute --brief-path "{dir}/brief.md" [--profile {profile}] --json
|
||||||
|
```
|
||||||
|
|
||||||
|
`phase_signal_result.effort` is consumed when picking the execution
|
||||||
|
strategy (gates auto-escalation, parallel-wave choice — see High-effort
|
||||||
|
behavior below). The resolver does NOT control the orchestrator's own
|
||||||
|
model — that is fixed at invocation time (command frontmatter omits
|
||||||
|
`model:`, so it follows the session model) and cannot be switched mid-turn.
|
||||||
|
`/trekexecute` spawns no sub-agent swarm (Hard Rule 10), so
|
||||||
|
`phase_signal_result.model` has no spawn site here; it is returned for
|
||||||
|
cross-command uniformity and stats.
|
||||||
|
|
||||||
For `/trekexecute` specifically: `effort == 'low'` activates `--gates open`
|
For `/trekexecute` specifically: `effort == 'low'` activates `--gates open`
|
||||||
+ sequential-only execution (no worktree-isolated parallel waves — runs
|
+ sequential-only execution (no worktree-isolated parallel waves — runs
|
||||||
|
|
@ -1657,12 +1746,16 @@ code-path.
|
||||||
v1.6 plans: synthesized manifests apply with the same force, but
|
v1.6 plans: synthesized manifests apply with the same force, but
|
||||||
`legacy_plan: true` is logged in progress.
|
`legacy_plan: true` is logged in progress.
|
||||||
|
|
||||||
18. **Last-activity rule.** The executor's final tool call before writing
|
18. **Last-activity rule + stop-signal.** The executor's final tool call before
|
||||||
Phase 8 must be a manifest check (Phase 7.5 audit), never an arbitrary
|
writing Phase 8 must be a manifest check (Phase 7.5 audit), never an arbitrary
|
||||||
file review. This prevents the "hallucinated completion" failure mode
|
file review. This prevents the "hallucinated completion" failure mode
|
||||||
where a transcript ends on an unrelated Read and the agent self-reports
|
where a transcript ends on an unrelated Read and the agent self-reports
|
||||||
`completed` without verifying. If Phase 7.5 has not run, the executor
|
`completed` without verifying. If Phase 7.5 has not run, the executor
|
||||||
may not emit `result: completed` under any circumstances.
|
may not emit `result: completed` under any circumstances. Beyond that:
|
||||||
|
`result: completed` requires the **machine-verifiable completion gate**
|
||||||
|
(the **stop-signal contract**, Phase 7.5) satisfied in full — Phase 7.5
|
||||||
|
audit PASS **and** an objective stop-signal (Verify exit 0 / DONE token
|
||||||
|
emitted AFTER the audit), never executor self-assessment alone.
|
||||||
|
|
||||||
19. **push-before-cleanup.** After successful `git merge --no-ff` of a wave
|
19. **push-before-cleanup.** After successful `git merge --no-ff` of a wave
|
||||||
branch, run `git push origin <branch>` BEFORE `git worktree remove` and
|
branch, run `git push origin <branch>` BEFORE `git worktree remove` and
|
||||||
|
|
@ -1670,3 +1763,10 @@ code-path.
|
||||||
cleanup proceeds regardless. Rationale: this converts an unrecoverable
|
cleanup proceeds regardless. Rationale: this converts an unrecoverable
|
||||||
failure (worktree removed, branch deleted, work lost) into a recoverable
|
failure (worktree removed, branch deleted, work lost) into a recoverable
|
||||||
one (push succeeded, branch preserved on remote). Source: research/02 R3.
|
one (push succeeded, branch preserved on remote). Source: research/02 R3.
|
||||||
|
|
||||||
|
20. **Three distinct iteration axes.** Per-step `attempts` (≤3), per-session
|
||||||
|
`recovery_depth` (<2), and the global `TREKEXECUTE_MAX_RECOVERY_ITERATIONS`
|
||||||
|
budget (default 25 — aggregate recovery+retry backstop) are independent
|
||||||
|
bounds, not one nested counter; child `--max-turns` (50) is a fourth,
|
||||||
|
separate axis. Every recovery/retry loop is bounded by at least one of them.
|
||||||
|
See "Iteration caps + global recovery/retry budget".
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,7 @@
|
||||||
---
|
---
|
||||||
name: trekplan
|
name: trekplan
|
||||||
description: Deep implementation planning from a task brief. Requires --brief or --project. Runs parallel specialized agents, optional external research, and adversarial review.
|
description: Deep implementation planning from a task brief. Requires --brief or --project. Runs parallel specialized agents, optional external research, and adversarial review.
|
||||||
argument-hint: "--brief <path> | --project <dir> [--fg | --quick | --research <brief> | --decompose <plan> | --export <fmt> <plan>]"
|
argument-hint: "--brief <path> | --project <dir> [--fg | --quick | --research <brief> | --decompose <plan> | --export headless <plan>]"
|
||||||
model: opus
|
|
||||||
allowed-tools: Agent, Read, Glob, Grep, Write, Edit, Bash, AskUserQuestion, TaskCreate, TaskUpdate, TeamCreate, TeamDelete
|
allowed-tools: Agent, Read, Glob, Grep, Write, Edit, Bash, AskUserQuestion, TaskCreate, TaskUpdate, TeamCreate, TeamDelete
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -30,13 +29,19 @@ Pipeline position:
|
||||||
|
|
||||||
Parse `$ARGUMENTS` for mode flags. Order of precedence:
|
Parse `$ARGUMENTS` for mode flags. Order of precedence:
|
||||||
|
|
||||||
1. **`--export <format> <plan-path>`** — extract `{format}` (first token after
|
1. **`--export headless <plan-path>`** — extract `{format}` (first token after
|
||||||
`--export`) and `{plan-path}` (remainder). Valid formats: `pr`, `issue`,
|
`--export`) and `{plan-path}` (remainder). `headless` is the only export
|
||||||
`markdown`, `headless`. Set **mode = export**.
|
format; it is a backwards-compatible **alias for `--decompose`**. Set
|
||||||
|
**mode = decompose** and run the decomposition pipeline (Phase 1.5).
|
||||||
|
|
||||||
If format is not in the valid set:
|
The PR / issue / markdown export variants were removed — Claude Code
|
||||||
|
reformats a plan into a PR body, issue comment, or stripped markdown
|
||||||
|
ad-hoc on request, so a dedicated export path added maintenance without
|
||||||
|
value. If `{format}` is anything other than `headless`:
|
||||||
```
|
```
|
||||||
Error: unknown export format '{format}'. Valid: pr, issue, markdown, headless
|
Error: export format '{format}' is not supported. The only export is
|
||||||
|
'headless' (an alias for --decompose). For PR / issue / markdown output,
|
||||||
|
ask Claude to reformat the plan directly.
|
||||||
```
|
```
|
||||||
If the plan file does not exist:
|
If the plan file does not exist:
|
||||||
```
|
```
|
||||||
|
|
@ -69,10 +74,11 @@ Parse `$ARGUMENTS` for mode flags. Order of precedence:
|
||||||
# older brief that sidesteps framing enforcement.
|
# older brief that sidesteps framing enforcement.
|
||||||
node ${CLAUDE_PLUGIN_ROOT}/lib/validators/brief-validator.mjs --soft --json [--min-version {min_brief_version}] "{dir}/brief.md"
|
node ${CLAUDE_PLUGIN_ROOT}/lib/validators/brief-validator.mjs --soft --json [--min-version {min_brief_version}] "{dir}/brief.md"
|
||||||
|
|
||||||
# v5.1.1 — resolve per-phase brief-signal for plan phase. Result is
|
# v5.9 — composed phase-model resolution (brief > profile > default) for
|
||||||
# captured as phase_signal_result and used at Agent-spawn sites below
|
# the plan phase. ONE call returns {effort, model, source}; captured as
|
||||||
# to override the orchestrator model when a signal is present.
|
# phase_signal_result and injected at Agent-spawn sites below.
|
||||||
node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/phase-signal-resolver.mjs --brief "{dir}/brief.md" --phase plan --json
|
# Append --profile {profile} when the operator passed --profile.
|
||||||
|
node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model --phase plan --brief-path "{dir}/brief.md" [--profile {profile}] --json
|
||||||
|
|
||||||
# Research briefs (if any) — drift-warn only, none of these block the run
|
# Research briefs (if any) — drift-warn only, none of these block the run
|
||||||
[ -d "{dir}/research" ] && \
|
[ -d "{dir}/research" ] && \
|
||||||
|
|
@ -140,7 +146,7 @@ Usage: /trekplan --brief <path-to-brief.md>
|
||||||
/trekplan --brief <path> --research <research-brief.md>
|
/trekplan --brief <path> --research <research-brief.md>
|
||||||
/trekplan --project <dir> --fg
|
/trekplan --project <dir> --fg
|
||||||
/trekplan --project <dir> --quick
|
/trekplan --project <dir> --quick
|
||||||
/trekplan --export <pr|issue|markdown|headless> <plan-path>
|
/trekplan --export headless <plan-path>
|
||||||
/trekplan --decompose <plan-path>
|
/trekplan --decompose <plan-path>
|
||||||
|
|
||||||
A brief is required. Produce one with /trekbrief first.
|
A brief is required. Produce one with /trekbrief first.
|
||||||
|
|
@ -152,7 +158,7 @@ Modes:
|
||||||
--fg No-op alias (foreground is the only mode as of v2.4.0)
|
--fg No-op alias (foreground is the only mode as of v2.4.0)
|
||||||
--quick Skip exploration agent swarm; plan directly
|
--quick Skip exploration agent swarm; plan directly
|
||||||
--min-brief-version <ver> Warn (never block) if the brief is older than <ver> (e.g. 2.2)
|
--min-brief-version <ver> Warn (never block) if the brief is older than <ver> (e.g. 2.2)
|
||||||
--export Generate shareable output from an existing plan (no new planning)
|
--export headless Legacy alias for --decompose (the only remaining export format)
|
||||||
--decompose Split an existing plan into self-contained headless sessions
|
--decompose Split an existing plan into self-contained headless sessions
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
|
|
@ -160,7 +166,7 @@ Examples:
|
||||||
/trekplan --brief .claude/projects/2026-04-18-jwt-auth/brief.md
|
/trekplan --brief .claude/projects/2026-04-18-jwt-auth/brief.md
|
||||||
/trekplan --project .claude/projects/2026-04-18-jwt-auth --research extra.md
|
/trekplan --project .claude/projects/2026-04-18-jwt-auth --research extra.md
|
||||||
/trekplan --project .claude/projects/2026-04-18-jwt-auth --fg
|
/trekplan --project .claude/projects/2026-04-18-jwt-auth --fg
|
||||||
/trekplan --export pr .claude/plans/trekplan-2026-04-06-rate-limiting.md
|
/trekplan --export headless .claude/plans/trekplan-2026-04-06-rate-limiting.md
|
||||||
/trekplan --decompose .claude/plans/trekplan-2026-04-06-rate-limiting.md
|
/trekplan --decompose .claude/plans/trekplan-2026-04-06-rate-limiting.md
|
||||||
|
|
||||||
Migrating from v1.x? See MIGRATION.md in this plugin. The old --spec flag
|
Migrating from v1.x? See MIGRATION.md in this plugin. The old --spec flag
|
||||||
|
|
@ -187,7 +193,7 @@ If `research_status == pending` and `research_topics > 0`:
|
||||||
|
|
||||||
Report the detected mode:
|
Report the detected mode:
|
||||||
```
|
```
|
||||||
Mode: {foreground | quick | export | decompose}
|
Mode: {foreground | quick | decompose}
|
||||||
Brief: {brief_path}
|
Brief: {brief_path}
|
||||||
Project: {project_dir or "-"}
|
Project: {project_dir or "-"}
|
||||||
Research: {N local briefs, M extra via --research}
|
Research: {N local briefs, M extra via --research}
|
||||||
|
|
@ -233,89 +239,10 @@ If `fm.type === 'trekreview'`:
|
||||||
`type: brief` input simply omit the field. No `plan_version` bump is
|
`type: brief` input simply omit the field. No `plan_version` bump is
|
||||||
required for this addition (backwards compatible).
|
required for this addition (backwards compatible).
|
||||||
|
|
||||||
## Phase 1.5 — Export (runs only when mode = export)
|
## Phase 1.5 — Decompose (runs only when mode = decompose)
|
||||||
|
|
||||||
**Skip this phase entirely unless mode = export.**
|
**Skip this phase entirely unless mode = decompose.** (`--export headless` is a
|
||||||
|
backwards-compatible alias that sets mode = decompose, so it lands here too.)
|
||||||
Read the plan file. Extract these sections from the plan content:
|
|
||||||
- Task description (from Context section)
|
|
||||||
- Implementation steps (from Implementation Plan section)
|
|
||||||
- Risks (from Risks and Mitigations section)
|
|
||||||
- Test strategy (from Test Strategy section, if present)
|
|
||||||
- Scope estimate (from Estimated Scope section)
|
|
||||||
|
|
||||||
### Format: `pr`
|
|
||||||
|
|
||||||
Output a markdown block formatted as a PR description:
|
|
||||||
|
|
||||||
```
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
{2–3 sentence summary of what this change does and why}
|
|
||||||
|
|
||||||
## Changes
|
|
||||||
|
|
||||||
{Bulleted list of implementation steps, one line each}
|
|
||||||
|
|
||||||
## Test plan
|
|
||||||
|
|
||||||
{Bulleted checklist from test strategy, formatted as - [ ] items}
|
|
||||||
|
|
||||||
## Risks
|
|
||||||
|
|
||||||
{Risks from plan, abbreviated to 1 line each}
|
|
||||||
|
|
||||||
---
|
|
||||||
*Generated by trekplan from {plan filename}*
|
|
||||||
```
|
|
||||||
|
|
||||||
### Format: `issue`
|
|
||||||
|
|
||||||
Output a markdown block formatted as an issue comment:
|
|
||||||
|
|
||||||
```
|
|
||||||
## Implementation plan summary
|
|
||||||
|
|
||||||
**Task:** {task description}
|
|
||||||
**Plan file:** {plan path}
|
|
||||||
**Scope:** {N files, complexity}
|
|
||||||
|
|
||||||
### Proposed approach
|
|
||||||
{3–5 bullet points from key implementation steps}
|
|
||||||
|
|
||||||
### Open questions / risks
|
|
||||||
{Top 2–3 risks from plan}
|
|
||||||
|
|
||||||
---
|
|
||||||
*Generated by trekplan*
|
|
||||||
```
|
|
||||||
|
|
||||||
### Format: `markdown`
|
|
||||||
|
|
||||||
Output the plan content with internal metadata stripped:
|
|
||||||
- Remove the "Revisions" section
|
|
||||||
- Remove plan-critic and scope-guardian scores/verdicts
|
|
||||||
- Remove `[ASSUMPTION]` markers (but keep the surrounding sentence)
|
|
||||||
- Keep everything else verbatim
|
|
||||||
|
|
||||||
### Format: `headless`
|
|
||||||
|
|
||||||
This is a shortcut for `--decompose`. It runs the full session decomposition
|
|
||||||
pipeline and is equivalent to `--decompose {plan-path}`. Proceed to
|
|
||||||
Phase 1.6 (Decompose) below.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
After outputting the formatted block (for pr/issue/markdown), say:
|
|
||||||
```
|
|
||||||
Export complete ({format}). Copy the block above.
|
|
||||||
```
|
|
||||||
|
|
||||||
Then **stop**. Do not continue to any subsequent phase.
|
|
||||||
|
|
||||||
## Phase 1.6 — Decompose (runs only when mode = decompose or export headless)
|
|
||||||
|
|
||||||
**Skip this phase entirely unless mode = decompose or export format = headless.**
|
|
||||||
|
|
||||||
Read the plan file. Verify it contains an Implementation Plan section with
|
Read the plan file. Verify it contains an Implementation Plan section with
|
||||||
numbered steps. If no steps are found, report and stop:
|
numbered steps. If no steps are found, report and stop:
|
||||||
|
|
@ -897,7 +824,7 @@ Never let tracking failures block the main workflow.
|
||||||
|
|
||||||
## Profile (v4.1)
|
## Profile (v4.1)
|
||||||
|
|
||||||
Accepts `--profile <name>` where `<name>` is `economy`, `balanced`, `premium`,
|
Accepts `--profile <name>` where `<name>` is `economy`, `balanced`, `premium`, `fable`,
|
||||||
or a custom profile under `voyage-profiles/`. Default: `premium`.
|
or a custom profile under `voyage-profiles/`. Default: `premium`.
|
||||||
|
|
||||||
Resolution order (per `lib/profiles/resolver.mjs`):
|
Resolution order (per `lib/profiles/resolver.mjs`):
|
||||||
|
|
@ -932,13 +859,15 @@ model_for_phase = brief.phase_signals[<phase>]?.model ?? profile.phase_models[
|
||||||
```
|
```
|
||||||
|
|
||||||
The brief signal wins per-phase when present; the profile fills any
|
The brief signal wins per-phase when present; the profile fills any
|
||||||
gaps. Composition is mechanically resolved via
|
gaps. Both fields are mechanically resolved by the single composed CLI
|
||||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/phase-signal-resolver.mjs`
|
`node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model`
|
||||||
invoked in Phase 1; the resolved JSON is captured as `phase_signal_result`
|
invoked in Phase 1; the resolved JSON `{effort, model, source}` is captured
|
||||||
and passed to `Agent` tool calls explicitly. The resolver controls only
|
as `phase_signal_result` and passed to `Agent` tool calls explicitly. The
|
||||||
the orchestrator and the model parameter at Agent-spawn sites — sub-agents
|
resolver controls the `model` parameter at Agent-spawn sites only — the
|
||||||
otherwise read `model:` from their own `agents/*.md` frontmatter (still
|
orchestrator's own model is fixed at invocation time (command frontmatter
|
||||||
pinned to `opus`).
|
omits `model:`, so it follows the session model) and cannot be switched
|
||||||
|
mid-turn. Sub-agents fall back to `model:` in their own `agents/*.md`
|
||||||
|
frontmatter when no spawn-site injection happens.
|
||||||
|
|
||||||
For `/trekplan` specifically: `effort == 'low'` activates the existing
|
For `/trekplan` specifically: `effort == 'low'` activates the existing
|
||||||
`--quick`-equivalent code-path (skip Phase 5 agent swarm — plan directly
|
`--quick`-equivalent code-path (skip Phase 5 agent swarm — plan directly
|
||||||
|
|
@ -983,10 +912,11 @@ Standard and low effort: do NOT run the additional pass.
|
||||||
inadequate, stop and ask the user to run `/trekbrief` again.
|
inadequate, stop and ask the user to run `/trekbrief` again.
|
||||||
- **Scope**: Only explore the current working directory and its subdirectories.
|
- **Scope**: Only explore the current working directory and its subdirectories.
|
||||||
Never read files outside the repo (no ~/.env, no credentials, no other repos).
|
Never read files outside the repo (no ~/.env, no credentials, no other repos).
|
||||||
- **Cost**: Sub-agents use their pinned `model:` frontmatter (currently `opus`).
|
- **Cost**: Model resolution at Agent-spawn sites is a three-layer fallback:
|
||||||
When `phase_signals[<phase>].model` is set, the orchestrator AND Agent-spawn
|
brief `phase_signals[<phase>].model` > `profile.phase_models[<phase>]` >
|
||||||
sites use the resolved model (`phase_signal_result.model`) for that phase.
|
agent frontmatter `model:`. The composed resolver returns the first two
|
||||||
Frontmatter is the default; brief signal is the per-phase override.
|
layers as `phase_signal_result.model`; spawn sites inject it, and agent
|
||||||
|
frontmatter is the fallback when no injection happens.
|
||||||
- **Privacy**: Never log, store, or repeat file contents that look like
|
- **Privacy**: Never log, store, or repeat file contents that look like
|
||||||
secrets, tokens, or credentials. Never log prompt text.
|
secrets, tokens, or credentials. Never log prompt text.
|
||||||
- **No premature execution**: Do not modify any project files until the user
|
- **No premature execution**: Do not modify any project files until the user
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,7 @@
|
||||||
---
|
---
|
||||||
name: trekresearch
|
name: trekresearch
|
||||||
description: Deep research combining local codebase analysis with external knowledge, producing structured research briefs with triangulation and confidence ratings
|
description: Deep research combining local codebase analysis with external knowledge, producing structured research briefs with triangulation and confidence ratings
|
||||||
argument-hint: "[--project <dir>] [--quick | --local | --external | --fg] <research question>"
|
argument-hint: "[--project <dir>] [--quick | --local | --external | --fg] [--engine swarm|deep-research] <research question>"
|
||||||
model: opus
|
|
||||||
allowed-tools: Agent, Read, Glob, Grep, Write, Edit, Bash, AskUserQuestion, WebSearch, WebFetch, mcp__tavily__tavily_search, mcp__tavily__tavily_research
|
allowed-tools: Agent, Read, Glob, Grep, Write, Edit, Bash, AskUserQuestion, WebSearch, WebFetch, mcp__tavily__tavily_search, mcp__tavily__tavily_research
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -55,15 +54,18 @@ Supported flags:
|
||||||
Create `{dir}/research/` if it does not already exist.
|
Create `{dir}/research/` if it does not already exist.
|
||||||
|
|
||||||
When `{dir}/brief.md` exists, ALWAYS run the brief-validator (soft mode)
|
When `{dir}/brief.md` exists, ALWAYS run the brief-validator (soft mode)
|
||||||
AND the phase-signal-resolver for this command's phase before continuing.
|
AND the composed phase-model resolver for this command's phase before
|
||||||
The resolver's JSON output is captured as `phase_signal_result` and used
|
continuing. The resolver's JSON output `{effort, model, source}`
|
||||||
at Agent-spawn sites in Phase 4 to inject the brief-resolved model:
|
(brief signal > profile > default) is captured as `phase_signal_result`
|
||||||
|
and used at Agent-spawn sites in Phase 4 to inject the resolved model:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# When --min-brief-version was passed, append --min-version {min_brief_version}
|
# When --min-brief-version was passed, append --min-version {min_brief_version}
|
||||||
# so an older brief raises BRIEF_VERSION_BELOW_MINIMUM (warn, never block).
|
# so an older brief raises BRIEF_VERSION_BELOW_MINIMUM (warn, never block).
|
||||||
node ${CLAUDE_PLUGIN_ROOT}/lib/validators/brief-validator.mjs --soft --json [--min-version {min_brief_version}] "{dir}/brief.md"
|
node ${CLAUDE_PLUGIN_ROOT}/lib/validators/brief-validator.mjs --soft --json [--min-version {min_brief_version}] "{dir}/brief.md"
|
||||||
node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/phase-signal-resolver.mjs --brief "{dir}/brief.md" --phase research --json
|
# v5.9 — composed resolver: ONE call returns {effort, model, source}.
|
||||||
|
# Append --profile {profile} when the operator passed --profile.
|
||||||
|
node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model --phase research --brief-path "{dir}/brief.md" [--profile {profile}] --json
|
||||||
```
|
```
|
||||||
|
|
||||||
6. `--gates` — autonomy control. When present, set `gates_mode = true`. The
|
6. `--gates` — autonomy control. When present, set `gates_mode = true`. The
|
||||||
|
|
@ -80,6 +82,16 @@ Supported flags:
|
||||||
enforcement only fires at `≥ 2.2`. Absent → no version check. See
|
enforcement only fires at `≥ 2.2`. Absent → no version check. See
|
||||||
`docs/HANDOVER-CONTRACTS.md` §Handover 1 for the pre-2.2 enforcement hole.
|
`docs/HANDOVER-CONTRACTS.md` §Handover 1 for the pre-2.2 enforcement hole.
|
||||||
|
|
||||||
|
8. `--engine <name>` — opt-in external-research engine. Accepts `--engine <name>`
|
||||||
|
where `<name>` is `swarm` or `deep-research`. **Default: `swarm`** (unchanged
|
||||||
|
behavior). `swarm` runs Voyage's own external-research agent swarm;
|
||||||
|
`deep-research` delegates the external phase to Claude Code's built-in
|
||||||
|
`/deep-research` dynamic workflow and adapts its report into the research-brief
|
||||||
|
schema (requires Claude Code 2.1.154+ and dynamic workflows enabled; falls back
|
||||||
|
to `swarm` and notes the fallback if unavailable — never hard-fails). Orthogonal
|
||||||
|
to `--profile`/`phase_signals`; only affects the external phase. Set
|
||||||
|
**engine = {swarm|deep-research}** (the *requested* engine).
|
||||||
|
|
||||||
Flags can be combined:
|
Flags can be combined:
|
||||||
- `--local` — local-only research
|
- `--local` — local-only research
|
||||||
- `--external --quick` — external-only, lightweight
|
- `--external --quick` — external-only, lightweight
|
||||||
|
|
@ -87,7 +99,7 @@ Flags can be combined:
|
||||||
- `--quick` alone implies both local and external (lightweight)
|
- `--quick` alone implies both local and external (lightweight)
|
||||||
|
|
||||||
Defaults: **scope = both**, **execution = foreground** (only mode as of
|
Defaults: **scope = both**, **execution = foreground** (only mode as of
|
||||||
v2.4.0), **project_dir = none**.
|
v2.4.0), **project_dir = none**, **engine = swarm**.
|
||||||
|
|
||||||
After stripping flags, the remaining text is the **research question**.
|
After stripping flags, the remaining text is the **research question**.
|
||||||
|
|
||||||
|
|
@ -108,6 +120,7 @@ Modes:
|
||||||
--external Only external research agents (skip codebase analysis)
|
--external Only external research agents (skip codebase analysis)
|
||||||
--fg No-op alias (foreground is the only mode as of v2.4.0)
|
--fg No-op alias (foreground is the only mode as of v2.4.0)
|
||||||
--project Write brief into an trekbrief project folder (auto-indexed)
|
--project Write brief into an trekbrief project folder (auto-indexed)
|
||||||
|
--engine Opt-in external-research engine: swarm (default) | deep-research
|
||||||
|
|
||||||
Flags can be combined: --local, --external --quick, --project <dir> --external
|
Flags can be combined: --local, --external --quick, --project <dir> --external
|
||||||
|
|
||||||
|
|
@ -118,6 +131,7 @@ Examples:
|
||||||
/trekresearch --external What are the security implications of using Redis for sessions?
|
/trekresearch --external What are the security implications of using Redis for sessions?
|
||||||
/trekresearch --fg --local What patterns does this codebase use for database access?
|
/trekresearch --fg --local What patterns does this codebase use for database access?
|
||||||
/trekresearch --project .claude/projects/2026-04-18-jwt-auth --external What JWT library is best for Node.js?
|
/trekresearch --project .claude/projects/2026-04-18-jwt-auth --external What JWT library is best for Node.js?
|
||||||
|
/trekresearch --project <dir> --external --engine deep-research <research question>
|
||||||
```
|
```
|
||||||
|
|
||||||
Do not continue past this step if no question was provided.
|
Do not continue past this step if no question was provided.
|
||||||
|
|
@ -126,6 +140,7 @@ Report the detected mode:
|
||||||
```
|
```
|
||||||
Mode: {default | quick}, Scope: {both | local | external}, Execution: foreground
|
Mode: {default | quick}, Scope: {both | local | external}, Execution: foreground
|
||||||
Project: {project_dir or "-"}
|
Project: {project_dir or "-"}
|
||||||
|
Engine (requested): {swarm | deep-research}
|
||||||
Question: {research question}
|
Question: {research question}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -292,6 +307,63 @@ For each local agent, prompt with the research question, NOT a task description:
|
||||||
- convention-scanner: "Discover coding conventions relevant to evaluating {question}.
|
- convention-scanner: "Discover coding conventions relevant to evaluating {question}.
|
||||||
What patterns would a solution need to follow?"
|
What patterns would a solution need to follow?"
|
||||||
|
|
||||||
|
### Engine selection (scope = both or external)
|
||||||
|
|
||||||
|
`--engine` affects ONLY the external portion of research. The local agents
|
||||||
|
(`### Local agents` above) and Phases 6–7 (triangulation, synthesis, brief
|
||||||
|
writing) are **engine-agnostic** — they run identically regardless of engine.
|
||||||
|
|
||||||
|
`--engine` is **moot** (treated as `swarm`) whenever the external phase does not
|
||||||
|
run at all: `--local`, `--quick`, `effort == 'low'`, or a profile with
|
||||||
|
`external_research_enabled == false` (the `economy`/`balanced` auto-disable — see
|
||||||
|
Profile below). The profile's on/off switch wins. Initialize
|
||||||
|
`effective_engine = {requested engine}`.
|
||||||
|
|
||||||
|
**engine = swarm (default):** run the `### External agents` + `### Bridge agent`
|
||||||
|
blocks below unchanged. This is byte-for-byte the current path, so `--engine swarm`
|
||||||
|
changes nothing (SC1). Keep the native-swarm anchors intact ("in parallel",
|
||||||
|
"single message", `model: "opus"`).
|
||||||
|
|
||||||
|
**engine = deep-research:**
|
||||||
|
|
||||||
|
1. **Coarse pre-gate (best-effort, NOT a trust signal).** `Bash: claude --version`;
|
||||||
|
parse the leading `X.Y.Z` (e.g. from `2.1.196 (Claude Code)`) and compare
|
||||||
|
numerically against `2.1.154` — split each on `.` and compare major, then minor,
|
||||||
|
then patch as integers (do NOT string-compare; lexical comparison mis-orders
|
||||||
|
multi-digit patch numbers). If the version is `< 2.1.154`, OR if
|
||||||
|
`disableWorkflows: true` / `CLAUDE_CODE_DISABLE_WORKFLOWS=1` is set, skip to the
|
||||||
|
fallback (step 4). **If `claude` is not on PATH inside the Bash tool (possible
|
||||||
|
under `claude -p`) or the version cannot be parsed, treat the pre-gate as
|
||||||
|
*indeterminate* and proceed to step 2 — do NOT hard-fail.** There is no positive
|
||||||
|
availability probe (research Dim 4), so a passing pre-gate does not guarantee the
|
||||||
|
workflow runs; the post-hoc check (step 3) is the authoritative guard.
|
||||||
|
|
||||||
|
2. **Run.** Instruct Claude (in prose, this turn) to run
|
||||||
|
`/deep-research <research question>` and request per-claim citations. Note:
|
||||||
|
interactive default/acceptEdits triggers a per-run approval prompt; `claude -p` /
|
||||||
|
SDK / bypass runs immediately.
|
||||||
|
|
||||||
|
3. **Post-hoc presence + provenance check (the real guard).** Verify a real, cited
|
||||||
|
`/deep-research` report actually landed in context — substantive findings with
|
||||||
|
citations, not an empty/denied/errored turn and not bare error text. This check
|
||||||
|
must be **robust to all failure manifestations** (workflow disabled, approval
|
||||||
|
denied, runtime error, empty output), because the disabled-headless behavior is
|
||||||
|
undocumented: no recognizable cited report in context → fall back, regardless of
|
||||||
|
how the failure surfaces.
|
||||||
|
|
||||||
|
4. **On no real report (fallback):** set `effective_engine = swarm`, run the swarm
|
||||||
|
blocks below, and **log the fallback at this decision point** — print
|
||||||
|
`Engine: deep-research → swarm (fallback: <reason>)` and carry the reason into the
|
||||||
|
Phase-8 Present summary and the brief's `## Executive Summary`. **NEVER fabricate
|
||||||
|
or synthesize a substitute report** — a structurally-valid-but-invented brief
|
||||||
|
passes the structure-only validator and silently poisons `/trekplan`; that is the
|
||||||
|
worst outcome of this feature.
|
||||||
|
|
||||||
|
5. **On a real report:** keep `effective_engine = deep-research`, log
|
||||||
|
`Engine: deep-research (active)`, and carry the report into Phase 6 triangulation
|
||||||
|
as the external-findings input (adapted in Phase 7 — see the Deep-research engine
|
||||||
|
adapter below).
|
||||||
|
|
||||||
### External agents (scope = both or external)
|
### External agents (scope = both or external)
|
||||||
|
|
||||||
Launch the new research-specialized agents:
|
Launch the new research-specialized agents:
|
||||||
|
|
@ -373,6 +445,45 @@ Write the brief to the `brief_destination` computed in Phase 1:
|
||||||
|
|
||||||
Create the parent directory if it does not exist.
|
Create the parent directory if it does not exist.
|
||||||
|
|
||||||
|
### Deep-research engine adapter (engine = deep-research only)
|
||||||
|
|
||||||
|
**Only when `effective_engine == deep-research`.** The swarm path skips this
|
||||||
|
entirely — its findings already flow through Phases 6–7 unchanged (SC1).
|
||||||
|
|
||||||
|
Transform the in-context `/deep-research` report INTO
|
||||||
|
`@${CLAUDE_PLUGIN_ROOT}/templates/research-brief-template.md` — do NOT paste the
|
||||||
|
raw report. Specifically:
|
||||||
|
|
||||||
|
- Reduce the report to ≥ 1 `### {Dimension} -- Confidence: {high|medium|low}`
|
||||||
|
entry, each carrying **External findings** bullets with per-claim source URLs.
|
||||||
|
Local findings still come from the local agents (Phase 4) and are merged in per
|
||||||
|
dimension as usual.
|
||||||
|
- Emit a numeric `confidence ∈ [0,1]` in frontmatter and a 3-sentence
|
||||||
|
`## Executive Summary` (answer, confidence, key caveat).
|
||||||
|
- Populate `## Sources` from the report's citations.
|
||||||
|
- **If the report lacks per-claim URLs, lower the confidence and note the gap in
|
||||||
|
`## Open Questions` — do NOT fabricate URLs.** Provenance you cannot cite is not
|
||||||
|
provenance.
|
||||||
|
- If the report is large, bound the transform to the top dimensions to avoid
|
||||||
|
context truncation.
|
||||||
|
|
||||||
|
### Output self-check (engine = deep-research only)
|
||||||
|
|
||||||
|
**Only when `effective_engine == deep-research`.** After writing to
|
||||||
|
`brief_destination`, run the output validator and repair-or-fall-back. This mirrors
|
||||||
|
the trekplan Phase-8 write→validate→repair self-check; the swarm path does NOT run
|
||||||
|
it, so swarm behavior is unchanged (SC1):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node ${CLAUDE_PLUGIN_ROOT}/lib/validators/research-validator.mjs --json "{brief_destination}"
|
||||||
|
```
|
||||||
|
|
||||||
|
On `valid: false`, repair the brief to satisfy the reported errors and re-run the
|
||||||
|
validator. If it cannot be made valid (e.g. the report was too thin to yield even
|
||||||
|
one dimension), set `effective_engine = swarm`, fall back to the swarm engine for
|
||||||
|
this run (and log the fallback per the Engine selection step), rather than emit an
|
||||||
|
invalid brief.
|
||||||
|
|
||||||
## Phase 8 — Present and track
|
## Phase 8 — Present and track
|
||||||
|
|
||||||
Present a summary to the user:
|
Present a summary to the user:
|
||||||
|
|
@ -384,6 +495,7 @@ Present a summary to the user:
|
||||||
**Mode:** {default | quick}, Scope: {both | local | external}
|
**Mode:** {default | quick}, Scope: {both | local | external}
|
||||||
**Brief:** {brief_destination}
|
**Brief:** {brief_destination}
|
||||||
**Project:** {project_dir or "-"}
|
**Project:** {project_dir or "-"}
|
||||||
|
**Engine (effective):** {swarm | deep-research}{, with fallback reason if it fell back}
|
||||||
**Confidence:** {overall confidence 0.0-1.0}
|
**Confidence:** {overall confidence 0.0-1.0}
|
||||||
**Dimensions:** {N} researched
|
**Dimensions:** {N} researched
|
||||||
**Agents:** {N} local + {N} external + {gemini: used | unavailable | skipped}
|
**Agents:** {N} local + {N} external + {gemini: used | unavailable | skipped}
|
||||||
|
|
@ -418,6 +530,7 @@ Record format (one JSON line):
|
||||||
"question": "{research question (first 100 chars)}",
|
"question": "{research question (first 100 chars)}",
|
||||||
"mode": "{default|quick}",
|
"mode": "{default|quick}",
|
||||||
"scope": "{both|local|external}",
|
"scope": "{both|local|external}",
|
||||||
|
"engine": "{effective engine: swarm|deep-research}",
|
||||||
"slug": "{brief slug}",
|
"slug": "{brief slug}",
|
||||||
"project_dir": "{project_dir or null}",
|
"project_dir": "{project_dir or null}",
|
||||||
"brief_path": "{brief_destination}",
|
"brief_path": "{brief_destination}",
|
||||||
|
|
@ -435,7 +548,7 @@ If `${CLAUDE_PLUGIN_DATA}` is not set or not writable, skip tracking silently.
|
||||||
|
|
||||||
## Profile (v4.1)
|
## Profile (v4.1)
|
||||||
|
|
||||||
Accepts `--profile <name>` where `<name>` is `economy`, `balanced`, `premium`,
|
Accepts `--profile <name>` where `<name>` is `economy`, `balanced`, `premium`, `fable`,
|
||||||
or a custom profile under `voyage-profiles/`. Default: `premium`.
|
or a custom profile under `voyage-profiles/`. Default: `premium`.
|
||||||
|
|
||||||
Resolution order (per `lib/profiles/resolver.mjs`):
|
Resolution order (per `lib/profiles/resolver.mjs`):
|
||||||
|
|
@ -455,8 +568,8 @@ VOYAGE_PROFILE=balanced /trekresearch
|
||||||
```
|
```
|
||||||
|
|
||||||
Stats records emit `profile`, `phase_models`, `parallel_agents`,
|
Stats records emit `profile`, `phase_models`, `parallel_agents`,
|
||||||
`external_research_enabled`, and `profile_source` so operators can audit
|
`external_research_enabled`, `profile_source`, and `engine` so operators can
|
||||||
which profile drove which session.
|
audit which profile and engine drove which session.
|
||||||
|
|
||||||
## Composition rule (v5.1)
|
## Composition rule (v5.1)
|
||||||
|
|
||||||
|
|
@ -470,13 +583,15 @@ model_for_phase = brief.phase_signals[<phase>]?.model ?? profile.phase_models[
|
||||||
```
|
```
|
||||||
|
|
||||||
The brief signal wins per-phase when present; the profile fills any
|
The brief signal wins per-phase when present; the profile fills any
|
||||||
gaps. Composition is mechanically resolved via
|
gaps. Both fields are mechanically resolved by the single composed CLI
|
||||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/phase-signal-resolver.mjs`
|
`node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model`
|
||||||
invoked in Phase 1; the resolved JSON is captured as `phase_signal_result`
|
invoked in Phase 1; the resolved JSON `{effort, model, source}` is captured
|
||||||
and passed to `Agent` tool calls explicitly. The resolver controls only
|
as `phase_signal_result` and passed to `Agent` tool calls explicitly. The
|
||||||
the orchestrator and the model parameter at Agent-spawn sites — sub-agents
|
resolver controls the `model` parameter at Agent-spawn sites only — the
|
||||||
otherwise read `model:` from their own `agents/*.md` frontmatter (still
|
orchestrator's own model is fixed at invocation time (command frontmatter
|
||||||
pinned to `opus`).
|
omits `model:`, so it follows the session model) and cannot be switched
|
||||||
|
mid-turn. Sub-agents fall back to `model:` in their own `agents/*.md`
|
||||||
|
frontmatter when no spawn-site injection happens.
|
||||||
|
|
||||||
For `/trekresearch` specifically: `effort == 'low'` activates the
|
For `/trekresearch` specifically: `effort == 'low'` activates the
|
||||||
existing `--quick`-equivalent code-path (inline research, no agent swarm).
|
existing `--quick`-equivalent code-path (inline research, no agent swarm).
|
||||||
|
|
@ -521,10 +636,11 @@ Low effort: inline research only, no agent swarm (existing
|
||||||
Triangulate AFTER independent research.
|
Triangulate AFTER independent research.
|
||||||
- **Graceful degradation:** If MCP tools are unavailable (Tavily, Gemini, MS Learn),
|
- **Graceful degradation:** If MCP tools are unavailable (Tavily, Gemini, MS Learn),
|
||||||
proceed with available tools and note limitations in brief metadata.
|
proceed with available tools and note limitations in brief metadata.
|
||||||
- **Cost:** Sub-agents use their pinned `model:` frontmatter (currently `opus`).
|
- **Cost:** Model resolution at Agent-spawn sites is a three-layer fallback:
|
||||||
When `phase_signals[<phase>].model` is set, the orchestrator AND Agent-spawn
|
brief `phase_signals[<phase>].model` > `profile.phase_models[<phase>]` >
|
||||||
sites use the resolved model (`phase_signal_result.model`) for that phase.
|
agent frontmatter `model:`. The composed resolver returns the first two
|
||||||
Frontmatter is the default; brief signal is the per-phase override.
|
layers as `phase_signal_result.model`; spawn sites inject it, and agent
|
||||||
|
frontmatter is the fallback when no injection happens.
|
||||||
- **Privacy:** Never log secrets, tokens, or credentials.
|
- **Privacy:** Never log secrets, tokens, or credentials.
|
||||||
- **Honesty:** If the question is trivially answerable, say so. Don't inflate research.
|
- **Honesty:** If the question is trivially answerable, say so. Don't inflate research.
|
||||||
- **Scope of codebase:** Only analyze the current working directory for local research.
|
- **Scope of codebase:** Only analyze the current working directory for local research.
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,6 @@ description: |
|
||||||
review.md with severity-tagged findings (BLOCKER/MAJOR/MINOR/SUGGESTION)
|
review.md with severity-tagged findings (BLOCKER/MAJOR/MINOR/SUGGESTION)
|
||||||
per Handover 6 (review → plan).
|
per Handover 6 (review → plan).
|
||||||
argument-hint: "--project <dir> [--since <ref>] [--quick] [--validate] [--dry-run]"
|
argument-hint: "--project <dir> [--since <ref>] [--quick] [--validate] [--dry-run]"
|
||||||
model: opus
|
|
||||||
allowed-tools: Agent, Read, Glob, Grep, Write, Edit, Bash, AskUserQuestion
|
allowed-tools: Agent, Read, Glob, Grep, Write, Edit, Bash, AskUserQuestion
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -92,10 +91,12 @@ as the file is parseable:
|
||||||
```bash
|
```bash
|
||||||
node ${CLAUDE_PLUGIN_ROOT}/lib/validators/brief-validator.mjs --soft --json "{brief_path}"
|
node ${CLAUDE_PLUGIN_ROOT}/lib/validators/brief-validator.mjs --soft --json "{brief_path}"
|
||||||
|
|
||||||
# v5.1.1 — resolve the review-phase brief signal. The JSON is captured as
|
# v5.9 — composed phase-model resolution (brief > profile > default) for the
|
||||||
|
# review phase. ONE call returns {effort, model, source}; captured as
|
||||||
# phase_signal_result and used in Phase 7 at the reviewer-launch site to
|
# phase_signal_result and used in Phase 7 at the reviewer-launch site to
|
||||||
# inject the brief-resolved model.
|
# inject the resolved model. Append --profile {profile} when the operator
|
||||||
node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/phase-signal-resolver.mjs --brief "{brief_path}" --phase review --json
|
# passed --profile.
|
||||||
|
node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model --phase review --brief-path "{brief_path}" [--profile {profile}] --json
|
||||||
```
|
```
|
||||||
|
|
||||||
Read the JSON output. If `valid: false` AND any error has code
|
Read the JSON output. If `valid: false` AND any error has code
|
||||||
|
|
@ -420,7 +421,7 @@ the contract for that handover (see `docs/HANDOVER-CONTRACTS.md`).
|
||||||
|
|
||||||
## Profile (v4.1)
|
## Profile (v4.1)
|
||||||
|
|
||||||
Accepts `--profile <name>` where `<name>` is `economy`, `balanced`, `premium`,
|
Accepts `--profile <name>` where `<name>` is `economy`, `balanced`, `premium`, `fable`,
|
||||||
or a custom profile under `voyage-profiles/`. Default: `premium`.
|
or a custom profile under `voyage-profiles/`. Default: `premium`.
|
||||||
|
|
||||||
Resolution order (per `lib/profiles/resolver.mjs`):
|
Resolution order (per `lib/profiles/resolver.mjs`):
|
||||||
|
|
@ -452,13 +453,15 @@ model_for_phase = brief.phase_signals[<phase>]?.model ?? profile.phase_models[
|
||||||
```
|
```
|
||||||
|
|
||||||
The brief signal wins per-phase when present; the profile fills any
|
The brief signal wins per-phase when present; the profile fills any
|
||||||
gaps. Composition is mechanically resolved via
|
gaps. Both fields are mechanically resolved by the single composed CLI
|
||||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/phase-signal-resolver.mjs`
|
`node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model`
|
||||||
invoked in Phase 2; the resolved JSON is captured as `phase_signal_result`
|
invoked in Phase 2; the resolved JSON `{effort, model, source}` is captured
|
||||||
and passed to `Agent` tool calls explicitly. The resolver controls only
|
as `phase_signal_result` and passed to `Agent` tool calls explicitly. The
|
||||||
the orchestrator and the model parameter at Agent-spawn sites — sub-agents
|
resolver controls the `model` parameter at Agent-spawn sites only — the
|
||||||
otherwise read `model:` from their own `agents/*.md` frontmatter (still
|
orchestrator's own model is fixed at invocation time (command frontmatter
|
||||||
pinned to `opus`).
|
omits `model:`, so it follows the session model) and cannot be switched
|
||||||
|
mid-turn. Sub-agents fall back to `model:` in their own `agents/*.md`
|
||||||
|
frontmatter when no spawn-site injection happens.
|
||||||
|
|
||||||
For `/trekreview` specifically: `effort == 'low'` activates the existing
|
For `/trekreview` specifically: `effort == 'low'` activates the existing
|
||||||
`--quick`-equivalent code-path (skip the brief-conformance reviewer; run
|
`--quick`-equivalent code-path (skip the brief-conformance reviewer; run
|
||||||
|
|
@ -515,10 +518,11 @@ Low effort: skip the brief-conformance reviewer entirely (existing
|
||||||
`findings:\n - a\n - b`.
|
`findings:\n - a\n - b`.
|
||||||
- **Refuse-with-suggestion above 100 files / 100K tokens.** Never run
|
- **Refuse-with-suggestion above 100 files / 100K tokens.** Never run
|
||||||
blind on a giant diff. Use AskUserQuestion to surface the gate.
|
blind on a giant diff. Use AskUserQuestion to surface the gate.
|
||||||
- **Cost.** Sub-agents use their pinned `model:` frontmatter (currently `opus`).
|
- **Cost.** Model resolution at Agent-spawn sites is a three-layer fallback:
|
||||||
When `phase_signals[<phase>].model` is set, the orchestrator AND Agent-spawn
|
brief `phase_signals[<phase>].model` > `profile.phase_models[<phase>]` >
|
||||||
sites use the resolved model (`phase_signal_result.model`) for that phase.
|
agent frontmatter `model:`. The composed resolver returns the first two
|
||||||
Frontmatter is the default; brief signal is the per-phase override.
|
layers as `phase_signal_result.model`; spawn sites inject it, and agent
|
||||||
|
frontmatter is the fallback when no injection happens.
|
||||||
- **Privacy.** Never log secrets, tokens, or credentials in review.md.
|
- **Privacy.** Never log secrets, tokens, or credentials in review.md.
|
||||||
Findings citing files with secret-like content must redact the secret
|
Findings citing files with secret-like content must redact the secret
|
||||||
in the `detail` field.
|
in the `detail` field.
|
||||||
|
|
|
||||||
|
|
@ -45,6 +45,8 @@ Every validator exposes a CLI: `node lib/validators/<name>.mjs --json <path>` re
|
||||||
|
|
||||||
**Stability tier: PUBLIC CONTRACT.** Handover 1 is the *only* public integration boundary of the pipeline: `brief.md` is what an upstream producer hands to Voyage, and Voyage consumes it without any knowledge of who produced it. This asymmetry is a hard invariant (see `CLAUDE.md` §Trinity context) — the interactive `/trekbrief` interview is just one producer; a `manual` brief, or an external per-app / per-portfolio producer, is equally valid as long as the artifact conforms to the schema below. No producer is privileged. The consequence: changing this schema — renaming or removing a field, narrowing an enum, or promoting an optional field to required — is a **breaking change for every downstream consumer** and MUST follow the [breaking-change protocol](#breaking-change-protocol) (version bump + N-1 compatibility window). Additive *optional* fields are non-breaking by design: the validator tolerates unknown frontmatter keys silently (forward-compat), so a newer brief still validates against an older consumer. The v5.5.0 contract formalization established `brief_version` **2.1** as the public-contract baseline and, in the same coordinated release, **evolved it to `2.2`** under the breaking-change protocol — adding the required `framing` field + `## TL;DR` section gated at ≥ 2.2 (existing 2.0/2.1 briefs stay valid). See the Versioning paragraph below for the 2.2 details.
|
**Stability tier: PUBLIC CONTRACT.** Handover 1 is the *only* public integration boundary of the pipeline: `brief.md` is what an upstream producer hands to Voyage, and Voyage consumes it without any knowledge of who produced it. This asymmetry is a hard invariant (see `CLAUDE.md` §Trinity context) — the interactive `/trekbrief` interview is just one producer; a `manual` brief, or an external per-app / per-portfolio producer, is equally valid as long as the artifact conforms to the schema below. No producer is privileged. The consequence: changing this schema — renaming or removing a field, narrowing an enum, or promoting an optional field to required — is a **breaking change for every downstream consumer** and MUST follow the [breaking-change protocol](#breaking-change-protocol) (version bump + N-1 compatibility window). Additive *optional* fields are non-breaking by design: the validator tolerates unknown frontmatter keys silently (forward-compat), so a newer brief still validates against an older consumer. The v5.5.0 contract formalization established `brief_version` **2.1** as the public-contract baseline and, in the same coordinated release, **evolved it to `2.2`** under the breaking-change protocol — adding the required `framing` field + `## TL;DR` section gated at ≥ 2.2 (existing 2.0/2.1 briefs stay valid). See the Versioning paragraph below for the 2.2 details.
|
||||||
|
|
||||||
|
**Trinity producer context (informational).** In the author's private three-tier design, Voyage is **Tier 1** (per-task). Its upstream producers are **Tier 2 `app-creator`** (per-app — "what does the app need, what's the next brief?") and **Tier 3 `app-factory`** (per-portfolio — "which app needs me now?"), both pre-implementation and bound for Forgejo when ready. None is privileged: any schema-conforming producer is equally valid, and Voyage stays unaware of Tier 2/3 — this Handover is the only coupling.
|
||||||
|
|
||||||
**Producer:** `/trekbrief` Phase 4g (after `brief-reviewer` stop-gate passes or iteration cap is hit).
|
**Producer:** `/trekbrief` Phase 4g (after `brief-reviewer` stop-gate passes or iteration cap is hit).
|
||||||
|
|
||||||
**Consumer:** `/trekresearch` Phase 1 (mode parse + brief validation).
|
**Consumer:** `/trekresearch` Phase 1 (mode parse + brief validation).
|
||||||
|
|
@ -113,9 +115,17 @@ Optional but standard sections: `## Non-Goals`, `## Constraints`, `## Preference
|
||||||
- `BRIEF_INVALID_PHASE_SIGNALS` → strict halt; phase_signals must be a list of `{phase, effort?, model?}` entries.
|
- `BRIEF_INVALID_PHASE_SIGNALS` → strict halt; phase_signals must be a list of `{phase, effort?, model?}` entries.
|
||||||
- `BRIEF_INVALID_PHASE_SIGNAL_PHASE` → strict halt; phase ∉ `[research, plan, execute, review]`.
|
- `BRIEF_INVALID_PHASE_SIGNAL_PHASE` → strict halt; phase ∉ `[research, plan, execute, review]`.
|
||||||
- `BRIEF_INVALID_EFFORT` → strict halt; effort ∉ `[low, standard, high]`.
|
- `BRIEF_INVALID_EFFORT` → strict halt; effort ∉ `[low, standard, high]`.
|
||||||
- `BRIEF_INVALID_MODEL` → strict halt; model ∉ `BASE_ALLOWED_MODELS` (currently `[sonnet, opus]`).
|
- `BRIEF_INVALID_MODEL` → strict halt; model ∉ `BASE_ALLOWED_MODELS` (currently `[sonnet, opus, fable]`).
|
||||||
- `BRIEF_SIGNALS_MUTUALLY_EXCLUSIVE` → strict halt; cannot set both `phase_signals` and `phase_signals_partial: true`.
|
- `BRIEF_SIGNALS_MUTUALLY_EXCLUSIVE` → strict halt; cannot set both `phase_signals` and `phase_signals_partial: true`.
|
||||||
|
|
||||||
|
**Compatibility direction of the v5.9 allowlist widening (`fable`):** enum
|
||||||
|
widening is safe for new readers of old data, not old readers of new data.
|
||||||
|
Existing sonnet/opus briefs stay valid under the v5.9+ validator (non-breaking,
|
||||||
|
no `brief_version` bump — value-space extension, not a schema change). The
|
||||||
|
reverse does NOT hold: a fable-bearing brief REQUIRES a v5.9+ validator — an
|
||||||
|
older cached brief-validator (e.g. a stale v5.8 marketplace clone) rejects it
|
||||||
|
with `BRIEF_INVALID_MODEL`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Handover 2 — research/*.md → plan
|
## Handover 2 — research/*.md → plan
|
||||||
|
|
|
||||||
153
docs/agent-description-token-trim-brief.md
Normal file
153
docs/agent-description-token-trim-brief.md
Normal file
|
|
@ -0,0 +1,153 @@
|
||||||
|
---
|
||||||
|
type: trekbrief
|
||||||
|
brief_version: "2.2"
|
||||||
|
created: 2026-06-26
|
||||||
|
task: "Relocate <example> blocks out of voyage agent description frontmatter to cut always-loaded tokens"
|
||||||
|
slug: agent-description-token-trim
|
||||||
|
project_dir: .claude/projects/2026-06-26-agent-description-token-trim/
|
||||||
|
research_topics: 0
|
||||||
|
research_status: skipped
|
||||||
|
auto_research: false
|
||||||
|
interview_turns: 0
|
||||||
|
source: manual
|
||||||
|
framing: refine
|
||||||
|
phase_signals:
|
||||||
|
- phase: research
|
||||||
|
effort: low
|
||||||
|
- phase: plan
|
||||||
|
effort: standard
|
||||||
|
- phase: execute
|
||||||
|
effort: standard
|
||||||
|
- phase: review
|
||||||
|
effort: standard
|
||||||
|
---
|
||||||
|
|
||||||
|
# Task: Trim voyage agent `description:` frontmatter — relocate `<example>` blocks into the agent body
|
||||||
|
|
||||||
|
> **Cross-session coordination brief.** Authored 2026-06-26 by the **config-audit machine-tuning session** (a parallel session that audits this machine's Claude Code always-loaded token footprint). It is dropped here per operator instruction so the **voyage session** owns the implementation — config-audit does **not** edit the voyage repo. The source measurement lives in config-audit's local worklist (`config-audit/docs/machine-tuning-worklist.local.md`, item **M4**, session m). This brief is the contract; `/trekplan` can consume it directly.
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
voyage's 24 agent `description:` fields cost **~4,418 always-loaded tokens every turn** (injected into the system prompt of every session that has voyage enabled). **17** of those agents carry **two `<example>` blocks each (34 total, ~3,147 tok)** in their frontmatter `description`. Those example blocks exist to drive *autonomous* agent-selection — but voyage agents are launched **by explicit name** from the orchestrator commands, so the examples are cost without function. Move them into each agent's body (preserve, don't delete). Expected saving: **~2,500–3,100 always-loaded tokens.** framing: **refine** — this continues the token-trim line started in M1a (v5.6.1).
|
||||||
|
|
||||||
|
## Intent
|
||||||
|
|
||||||
|
The machine's #1 tuning lever is *always-loaded* tokens — context injected on every turn, paid on every request. config-audit's Fase-1 measurement (2026-06-26) found the machine's global layer is already lean (CLAUDE.md trimmed, rules/agents empty, output style off, MCP deferred), and the **single largest remaining tunable source is voyage's agent listing (~4,418 tok)**. The bulk of that is `<example>` blocks in the spawnable agents' `description:` frontmatter. These blocks follow the documented agent-authoring pattern whose purpose is to help the *main loop autonomously decide* when to delegate to an agent. voyage doesn't rely on that path: `/trekplan`, `/trekresearch`, and `/trekreview` launch their agents **deterministically by name** ("Launch the **architecture-mapper** agent", explicit per-codebase-size launch tables, explicit `voyage:review-coordinator` references). So the examples are paying a per-turn token tax for an auto-selection behavior voyage never uses.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Each of the **17** example-bearing voyage agents has its frontmatter `description:` reduced to its **triggering lead sentence(s)** (the "Use this agent to/when…" summary — what any residual autonomous selection actually needs), with the `<example>` blocks **relocated verbatim into the agent's markdown body** under a clearly-marked section (e.g. `## When to use — examples`). No agent is deleted, no behavior changes, no example content is lost. The injected agent-listing cost drops by ~2,500–3,100 tokens, landing on every machine that reloads voyage after release.
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
- **Do NOT delete the examples** — relocate them into the body. They remain useful documentation and `/trekbrief`/onboarding reference.
|
||||||
|
- **Do NOT change any agent's system prompt, body logic, tools, model, or `name`** (`name` is the agent's identity).
|
||||||
|
- **Do NOT touch the 7 zero-example agents** (`planning-orchestrator`, `research-orchestrator`, `review-orchestrator`, `synthesis-agent`, `review-coordinator`, `code-correctness-reviewer`, `brief-conformance-reviewer`) — already lean (M1a handled the reference/dormant ones; the orchestrators keep their pinned "reference document, not a spawnable capability" phrasing).
|
||||||
|
- **Do NOT modify the orchestrator commands** (`/trekplan` etc.) or change which agents exist.
|
||||||
|
- This is **not** a behavioral or capability change — purely a token/packaging optimization.
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- Frontmatter must remain valid YAML. The retained `description:` must keep a meaningful **triggering lead sentence**, because a few agents (notably the `*-researcher` agents) *can* legitimately be selected autonomously when a user asks a research question directly — keep their first 1–2 sentences descriptive enough to still trigger.
|
||||||
|
- **The test suite must stay green.** `node --test 'tests/**/*.test.mjs'` is the gate. voyage has inventory/frontmatter-pin tests — M1a's note warned that "synthesis-schema + inventory tests may pin description content." If any test asserts on `<example>` presence/count or description length, update that test's expectation **deliberately** (it is asserting the old structure) and document why.
|
||||||
|
- Follow voyage's release discipline: version-sync (plugin.json + package.json + README badge/What's-new + CHANGELOG), annotated tag, and catalog `ref` bump with `check-versions` green. This is a perf/token change → a **patch** release (e.g. v5.6.2) or fold into the work already in progress on the branch.
|
||||||
|
- The saving only reaches a machine after the operator reloads voyage (`/plugin marketplace update` + update + `/exit`) — note this in the CHANGELOG so the delta isn't expected instantly.
|
||||||
|
|
||||||
|
## Preferences
|
||||||
|
|
||||||
|
- Relocate the two examples per agent into a single body section with a stable heading so they're easy to find and so any future inventory test can pin the body instead of the frontmatter.
|
||||||
|
- Keep the diff mechanical and uniform across the 17 agents (same heading, same ordering) for reviewability.
|
||||||
|
- If a lead sentence is currently entangled with the first `<example>`, lightly rewrite it into a clean 1–2 sentence trigger — minimal, not a rewrite.
|
||||||
|
|
||||||
|
## Non-Functional Requirements
|
||||||
|
|
||||||
|
- **Zero new dependencies.**
|
||||||
|
- **Always-loaded `description` budget:** total frontmatter `description` chars across all 24 agents drops from **~17,672** to **≤ ~6,000** (~2,500–3,100 tok saved).
|
||||||
|
- No regression in agent auto-selection for the agents that are legitimately autonomous (the researchers) — spot-check that a direct research-style prompt still routes sensibly.
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
- **Full suite green:** `node --test 'tests/**/*.test.mjs'` exits 0 (same pass count, or deliberately updated pins with rationale).
|
||||||
|
- **No `<example>` in any frontmatter `description`:** the frontmatter-only scan below prints `0`:
|
||||||
|
```bash
|
||||||
|
python3 - <<'PY'
|
||||||
|
import re,glob
|
||||||
|
n=0
|
||||||
|
for f in glob.glob('agents/*.md'):
|
||||||
|
t=open(f).read(); m=re.match(r'^---\n(.*?)\n---', t, re.S)
|
||||||
|
fm=m.group(1) if m else ''
|
||||||
|
dm=re.search(r'description:\s*(.*?)(?=\n[a-zA-Z_][a-zA-Z_-]*:\s|\Z)', fm, re.S)
|
||||||
|
n+=(dm.group(1).count('<example>') if dm else 0)
|
||||||
|
print(n) # must be 0
|
||||||
|
PY
|
||||||
|
```
|
||||||
|
- **Examples preserved in bodies:** `grep -l '<example>' agents/*.md` still lists the 17 agents (the blocks moved, not vanished).
|
||||||
|
- **Agent count unchanged:** 24 agents, each with non-empty `name` + `description`. Plugin validates (plugin-validator / inventory tests green).
|
||||||
|
- **Token budget met:** total frontmatter `description` chars ≤ ~6,000 (measure with the per-agent script in the worklist / the snippet above adapted to sum lengths).
|
||||||
|
- **Behavior unchanged:** `/trekplan` (and trekresearch/trekreview) still launch the same agents by name — diff touches only `agents/*.md`, no command files.
|
||||||
|
|
||||||
|
## Research Plan
|
||||||
|
|
||||||
|
No external research needed — the codebase and this brief contain sufficient context for planning. (research_topics = 0.)
|
||||||
|
|
||||||
|
## Open Questions / Assumptions
|
||||||
|
|
||||||
|
- **[ASSUMPTION]** The 17 example-bearing agents are invoked **by name** from orchestrator commands — verified by the config-audit session against `voyage/commands/*.md` (explicit "Launch the X agent" prose + per-size launch tables + explicit `voyage:<name>` references). The `<example>` auto-trigger blocks are therefore non-load-bearing for voyage's actual invocation path.
|
||||||
|
- **[Q]** Do any voyage tests assert on `<example>` presence/count or `description` length in `agents/*.md`? **Check first** (`grep -rn 'example\|description' tests/`), so a pin is updated intentionally, not discovered as a red test.
|
||||||
|
- **[ASSUMPTION]** The `*-researcher` agents (community/contrarian/docs/security) and `gemini-bridge` may also be autonomously selected outside the trek pipeline → keep their lead sentences trigger-worthy (don't reduce to a bare label).
|
||||||
|
|
||||||
|
## Prior Attempts
|
||||||
|
|
||||||
|
- **M1a — voyage v5.6.1 (2026-06-24):** trimmed the 4 non-spawnable reference/dormant agents' (`planning/research/review-orchestrator` + `synthesis-agent`) verbose `description:` fields to one-liners; orchestrators kept their pinned phrase, synthesis kept its DORMANT flag. Suite stayed green (756). **M4 extends that same always-loaded-token-trim to the 17 spawnable agents' `<example>` blocks** — the larger remaining chunk M1a deliberately left.
|
||||||
|
|
||||||
|
## Reference data (config-audit Fase-1 measurement, 2026-06-26)
|
||||||
|
|
||||||
|
24 agents, **17,672 `description` chars (~4,418 tok)**, **34 `<example>` blocks**, ~12,591 chars (~3,147 tok) inside `<example>…</example>`. The 17 example-bearing agents (descending `description` size):
|
||||||
|
|
||||||
|
| agent | desc chars | examples |
|
||||||
|
|---|---:|---:|
|
||||||
|
| community-researcher | 1280 | 2 |
|
||||||
|
| contrarian-researcher | 1270 | 2 |
|
||||||
|
| gemini-bridge | 1226 | 2 |
|
||||||
|
| security-researcher | 1207 | 2 |
|
||||||
|
| docs-researcher | 1138 | 2 |
|
||||||
|
| session-decomposer | 946 | 2 |
|
||||||
|
| convention-scanner | 939 | 2 |
|
||||||
|
| brief-reviewer | 870 | 2 |
|
||||||
|
| research-scout | 847 | 2 |
|
||||||
|
| test-strategist | 823 | 2 |
|
||||||
|
| dependency-tracer | 818 | 2 |
|
||||||
|
| task-finder | 817 | 2 |
|
||||||
|
| git-historian | 795 | 2 |
|
||||||
|
| risk-assessor | 794 | 2 |
|
||||||
|
| architecture-mapper | 789 | 2 |
|
||||||
|
| scope-guardian | 750 | 2 |
|
||||||
|
| plan-critic | 692 | 2 |
|
||||||
|
|
||||||
|
Zero-example (leave as-is): review-coordinator (345), code-correctness-reviewer (292), brief-conformance-reviewer (266), synthesis-agent (228), planning-orchestrator (184), research-orchestrator (179), review-orchestrator (177).
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
- **Created:** 2026-06-26
|
||||||
|
- **Interview turns:** 0
|
||||||
|
- **Auto-research opted in:** no
|
||||||
|
- **Source:** manual (cross-session coordination brief from the config-audit machine-tuning session)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How to continue
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# (optional) confirm no test pins <example> in descriptions first:
|
||||||
|
grep -rn 'example' tests/ | grep -i descr
|
||||||
|
|
||||||
|
# plan + execute via the voyage pipeline:
|
||||||
|
/trekplan --project .claude/projects/2026-06-26-agent-description-token-trim/
|
||||||
|
/trekexecute --project .claude/projects/2026-06-26-agent-description-token-trim/
|
||||||
|
|
||||||
|
# or implement directly (it's a uniform, mechanical 17-file edit) and release as a patch (v5.6.2):
|
||||||
|
# - move each agent's 2 <example> blocks from frontmatter description into a body
|
||||||
|
# "## When to use — examples" section
|
||||||
|
# - node --test 'tests/**/*.test.mjs' (green)
|
||||||
|
# - version-sync + CHANGELOG + tag + catalog ref-bump (check-versions green)
|
||||||
|
```
|
||||||
|
|
@ -37,7 +37,7 @@ Doc-consistency test at `tests/lib/doc-consistency.test.mjs` pins agent-table co
|
||||||
|
|
||||||
**Brief:** 7-phase workflow: Parse mode → Create project dir → Phase 3 completeness loop (section-driven, no question cap) → Phase 3.5 per-phase effort dialog (v5.1) → Phase 4 draft/review/revise with `brief-reviewer` as stop-gate (max 3 iterations; gate = all dimensions ≥ 4 and research plan = 5) → Finalize (`brief.md` on pass, or `brief_quality: partial` on cap/force-stop) → Manual/auto opt-in → Stats. Always interactive. Auto mode runs research + plan inline in the main context (v2.4.0).
|
**Brief:** 7-phase workflow: Parse mode → Create project dir → Phase 3 completeness loop (section-driven, no question cap) → Phase 3.5 per-phase effort dialog (v5.1) → Phase 4 draft/review/revise with `brief-reviewer` as stop-gate (max 3 iterations; gate = all dimensions ≥ 4 and research plan = 5) → Finalize (`brief.md` on pass, or `brief_quality: partial` on cap/force-stop) → Manual/auto opt-in → Stats. Always interactive. Auto mode runs research + plan inline in the main context (v2.4.0).
|
||||||
|
|
||||||
**Phase 3.5 (v5.1) — adaptive-depth signals:** Between Phase 3 completeness exit and Phase 4 draft, the operator commits an effort level (`low | standard | high`) and an optional `model` (`sonnet | opus`) per downstream phase (`research`, `plan`, `execute`, `review`) via 4 tier-coupled `AskUserQuestion` calls. The choices land in `brief.md` frontmatter as `phase_signals:` (a list of `{phase, effort?, model?}` entries) when committed, or `phase_signals_partial: true` when the operator force-stops. `brief_version: 2.1` activates the **sequencing gate**: validator emits `BRIEF_V51_MISSING_SIGNALS` if a 2.1-versioned brief lacks both fields. Downstream commands surface a friendly hint pointing back to `/trekbrief` — enforcement is validator-only. Composition is documented prose in each downstream command's `## Composition rule (v5.1)` section: `brief.phase_signals[phase] > profile.phase_models[phase]`. The brief signal wins per-phase when present; the profile fills gaps. `effort == low` activates each command's existing `--quick`-equivalent code-path (`/trekexecute` low-effort = `--gates open` + sequential-only). High-effort behavior is deferred to v5.1.1 per brief Non-Goal.
|
**Phase 3.5 (v5.1) — adaptive-depth signals:** Between Phase 3 completeness exit and Phase 4 draft, the operator commits an effort level (`low | standard | high`) and an optional `model` (`sonnet | opus | fable`) per downstream phase (`research`, `plan`, `execute`, `review`) via 4 tier-coupled `AskUserQuestion` calls. The choices land in `brief.md` frontmatter as `phase_signals:` (a list of `{phase, effort?, model?}` entries) when committed, or `phase_signals_partial: true` when the operator force-stops. `brief_version: 2.1` activates the **sequencing gate**: validator emits `BRIEF_V51_MISSING_SIGNALS` if a 2.1-versioned brief lacks both fields. Downstream commands surface a friendly hint pointing back to `/trekbrief` — enforcement is validator-only. Composition is documented prose in each downstream command's `## Composition rule (v5.1)` section: `brief.phase_signals[phase] > profile.phase_models[phase]`. The brief signal wins per-phase when present; the profile fills gaps. `effort == low` activates each command's existing `--quick`-equivalent code-path (`/trekexecute` low-effort = `--gates open` + sequential-only). High-effort behavior is deferred to v5.1.1 per brief Non-Goal.
|
||||||
|
|
||||||
**Research:** Foreground workflow (v2.4.0): Parse mode → Interview → Parallel research swarm (5 local + 4 external + 1 bridge, spawned from main context) → Follow-ups → Triangulation → Synthesis + brief → Stats. With `--project`, writes to `{dir}/research/NN-slug.md`.
|
**Research:** Foreground workflow (v2.4.0): Parse mode → Interview → Parallel research swarm (5 local + 4 external + 1 bridge, spawned from main context) → Follow-ups → Triangulation → Synthesis + brief → Stats. With `--project`, writes to `{dir}/research/NN-slug.md`.
|
||||||
|
|
||||||
|
|
@ -49,6 +49,8 @@ Doc-consistency test at `tests/lib/doc-consistency.test.mjs` pins agent-table co
|
||||||
|
|
||||||
**Continue:** `/trekcontinue` reads `{dir}/.session-state.local.json` (Handover 7), validates schema-v1 via `session-state-validator`, narrates a 3-line summary (project / next-session-label / brief-path), and immediately begins executing the next session. Auto-discovers active project state files under `.claude/projects/*/.session-state.local.json` if no explicit `<project-dir>` argument. Operator-invoked only — never auto-loaded via SessionStart. The `/trekendsession` helper is the informal-flow producer: writes the same state file for ad-hoc multi-session handovers that don't run through `/trekexecute`.
|
**Continue:** `/trekcontinue` reads `{dir}/.session-state.local.json` (Handover 7), validates schema-v1 via `session-state-validator`, narrates a 3-line summary (project / next-session-label / brief-path), and immediately begins executing the next session. Auto-discovers active project state files under `.claude/projects/*/.session-state.local.json` if no explicit `<project-dir>` argument. Operator-invoked only — never auto-loaded via SessionStart. The `/trekendsession` helper is the informal-flow producer: writes the same state file for ad-hoc multi-session handovers that don't run through `/trekexecute`.
|
||||||
|
|
||||||
|
**Delegate the engine, keep the policy (cross-cutting; audited S32, 2026-06-20).** Every swarm and interview in the pipeline rides a *native* Claude Code primitive and re-implements no engine of its own: parallel sub-agent fan-out is a single-message multi-`Agent`-call dispatch (research swarm, exploration swarm, reviewer swarm), and interview turn-taking is `AskUserQuestion` (brief Phase 3, the research interview, the trekplan research-status gate, the trekreview scope gate). Voyage's value is the **policy** layered on top — never the scheduler, concurrency loop, or menu loop underneath: typed agent roles + effort defaults + codebase-size scaling (exploration); 4-angle decomposition (docs/community/security/contrarian) + per-source schemas + triangulation (research); the 12-key rule catalogue + no-cross-feed isolation + deterministic dedup/verdict (review); the section-driven completeness loop + framing gate (brief). Concretely: the brief's Phase-3 "selection rule" picks *which* question to ask (policy); `AskUserQuestion` does the asking (engine). The V01/V07/V08/V11/V24 balance-analysis audit confirmed all five already delegate natively — no engine re-implementation found; the doc-consistency test (`tests/lib/doc-consistency.test.mjs` § S32) pins the native-delegation prose so a hand-rolled engine cannot creep back in.
|
||||||
|
|
||||||
**Operator-UX guarantee (since v5.0.2):** `/trekbrief`, `/trekplan`, and `/trekreview` MUST always emit (a) a plain `file://<abs path>` URL AND (b) a copy-pasteable `open file://<abs path>` command in the final report block. The file:// URL must use an ABSOLUTE path (not relative or `~/`-prefixed) so terminals with cmd+click support (Ghostty, iTerm2, modern Terminal.app) can resolve it without shell interpretation. This is a non-negotiable operator-UX contract — the doc-consistency test pins both forms in all three commands' final report blocks.
|
**Operator-UX guarantee (since v5.0.2):** `/trekbrief`, `/trekplan`, and `/trekreview` MUST always emit (a) a plain `file://<abs path>` URL AND (b) a copy-pasteable `open file://<abs path>` command in the final report block. The file:// URL must use an ABSOLUTE path (not relative or `~/`-prefixed) so terminals with cmd+click support (Ghostty, iTerm2, modern Terminal.app) can resolve it without shell interpretation. This is a non-negotiable operator-UX contract — the doc-consistency test pins both forms in all three commands' final report blocks.
|
||||||
|
|
||||||
**Operator-annotation HTML (v5.0.3):** the last step of `/trekbrief`, `/trekplan`, and `/trekreview` runs `scripts/annotate.mjs` against the just-written `.md` and prints the resulting `file://<abs path>` link. The HTML is self-contained (zero npm deps, zero external network, design-system-styled, light + dark + print) and modelled on `~/repos/claude-code-100x/claude-code-100x/build-site.js` (lines 1431–2255). The operator opens the file, the document renders as a proper article (headings / paragraphs / lists / tables / code / quotes — every element gets a stable `data-anchor-id`). In annotation mode (default ON, pencil-toggle in topbar), the operator can **select any text or click any element** → a form popover opens at the cursor with: section context auto-detected from nearest h1/h2, the anchored snippet (selection if any, else element text), **three intent buttons (Fiks / Endre / Spørsmål)**, comment textarea, Save/Cancel. The sidebar (Show annotations button) lists every annotation grouped by section with intent badge + snippet + comment + delete; clicking a card scrolls to and flashes the source element. **Copy Prompt** assembles a structured markdown (`### N. [Intent] Section: <…>` + `Quote: «…»` + `Comment: …`) and copies to clipboard. Persistence: `localStorage` keyed on absolute artifact path (`voyage-annotate:v2:<abs path>`). v5.0.0 removed the v4.2/v4.3 bespoke playground SPA + `/trekrevise` + Handover 8; v5.0.1 pointed at `/playground document-critique` (Claude-leads, wrong direction); v5.0.2 was operator-led but too thin (line-click + freeform note, no intents); v5.0.3 matches the claude-code-100x reference the operator first pointed at, with pencil-toggle / selection capture / intent categories / popover form / structured export. See [CHANGELOG.md](../CHANGELOG.md) § v5.0.3.
|
**Operator-annotation HTML (v5.0.3):** the last step of `/trekbrief`, `/trekplan`, and `/trekreview` runs `scripts/annotate.mjs` against the just-written `.md` and prints the resulting `file://<abs path>` link. The HTML is self-contained (zero npm deps, zero external network, design-system-styled, light + dark + print) and modelled on `~/repos/claude-code-100x/claude-code-100x/build-site.js` (lines 1431–2255). The operator opens the file, the document renders as a proper article (headings / paragraphs / lists / tables / code / quotes — every element gets a stable `data-anchor-id`). In annotation mode (default ON, pencil-toggle in topbar), the operator can **select any text or click any element** → a form popover opens at the cursor with: section context auto-detected from nearest h1/h2, the anchored snippet (selection if any, else element text), **three intent buttons (Fiks / Endre / Spørsmål)**, comment textarea, Save/Cancel. The sidebar (Show annotations button) lists every annotation grouped by section with intent badge + snippet + comment + delete; clicking a card scrolls to and flashes the source element. **Copy Prompt** assembles a structured markdown (`### N. [Intent] Section: <…>` + `Quote: «…»` + `Comment: …`) and copies to clipboard. Persistence: `localStorage` keyed on absolute artifact path (`voyage-annotate:v2:<abs path>`). v5.0.0 removed the v4.2/v4.3 bespoke playground SPA + `/trekrevise` + Handover 8; v5.0.1 pointed at `/playground document-critique` (Claude-leads, wrong direction); v5.0.2 was operator-led but too thin (line-click + freeform note, no intents); v5.0.3 matches the claude-code-100x reference the operator first pointed at, with pencil-toggle / selection capture / intent categories / popover form / structured export. See [CHANGELOG.md](../CHANGELOG.md) § v5.0.3.
|
||||||
|
|
@ -78,6 +80,24 @@ The `.html` files (`brief.html`, `plan.html`, `review.html`) are produced by `sc
|
||||||
|
|
||||||
No code-level dependency between plugins — the contract is filesystem-level only.
|
No code-level dependency between plugins — the contract is filesystem-level only.
|
||||||
|
|
||||||
|
### Primitives per step (decision matrix)
|
||||||
|
|
||||||
|
Which native Claude Code primitive each pipeline step runs on today, and the alternatives that were considered and where they landed. Companion to *Delegate the engine, keep the policy* above; the full decision records live in `docs/voyage-vs-cc-balance-analysis.md`, `docs/T1-cc26-delegated-orchestration.md`, and `docs/T2-cc27-workflow-substrate.md`.
|
||||||
|
|
||||||
|
| Step | Primitive today (production) | Alternatives considered (status) |
|
||||||
|
|---|---|---|
|
||||||
|
| **brief** | Inline interview loop → `brief.md` (framing gate) | `AskUserQuestion` as the Q&A engine (recommended, not yet wired); Skill/Workflow auto-orchestration (rejected — one-shot, can't span slash-command/handover boundaries) |
|
||||||
|
| **research** | Inline spawns a foreground swarm; MCP per agent¹ | Delegated `synthesis-agent` (PoC measured Δ main-context ≈ 0 → ships **dormant**, wired to nothing) |
|
||||||
|
| **plan** | Inline: parallel explorer swarm → synthesis → adversarial `plan-critic` | Delegated orchestrator via nested sub-agents (CC-26 → lean NO, not wired; `planning-orchestrator.md` is a reference doc, not a spawnable agent) |
|
||||||
|
| **execute** | Inline step loop; multi-session via `git worktree` + `claude -p` waves; deterministic manifest audit | CC `TaskCreate`/`TodoWrite` for progress/resume (insufficient — carries no step status / attempts / SHA / drift → own typed `progress.json` contract) |
|
||||||
|
| **review** | Inline parallel reviewers (no cross-feed) → `review-coordinator` Judge | **Workflow** substrate for Phase 5–6 (bake-off POSITIVE: +4.4 % tokens / +54 % wall-time → shipped **opt-in `--workflow`**, not default; wholesale substrate swap declined) |
|
||||||
|
| **continue** | Inline reads `.session-state.local.json` → zero-confirm resume | CC `--resume` (transcript replay, not typed work-state → insufficient) |
|
||||||
|
| **cross-cutting** | 7 hook scripts: `pre-bash` + `pre-write` guards, `post-bash` stats, `session-title`, `pre-`/`post-compact` flush, **`Stop`→OTEL** export | — |
|
||||||
|
|
||||||
|
¹ MCP per research agent: `docs-researcher` → Microsoft Learn + Tavily · `community-`/`security-`/`contrarian-researcher` → Tavily (+ WebSearch/WebFetch) · `gemini-bridge` → Gemini Deep Research MCP. Graceful degradation when an MCP server is absent.
|
||||||
|
|
||||||
|
**Legend:** *production* = wired and active · *dormant* = shipped but wired to nothing (`synthesis-agent`) · *opt-in* = behind a flag (`--workflow`) · *not wired* = considered, deferred (delegated orchestrator, the `AskUserQuestion` brief engine).
|
||||||
|
|
||||||
## State
|
## State
|
||||||
|
|
||||||
All artifacts in one project directory (default):
|
All artifacts in one project directory (default):
|
||||||
|
|
|
||||||
79
docs/balance-backlog-plan.md
Normal file
79
docs/balance-backlog-plan.md
Normal file
|
|
@ -0,0 +1,79 @@
|
||||||
|
# Balance Backlog — Implementation Plan (multi-session)
|
||||||
|
|
||||||
|
**Status:** Active plan, decisions RESOLVED 2026-06-20. Mandated by operator ("vi skal gjøre alle backlog items").
|
||||||
|
**Source:** the §6 backlog of `docs/voyage-vs-cc-balance-analysis.md`.
|
||||||
|
**Operating mode:** ONE session per row → TDD (failing test first) → commit + push → update STATE → STOP. Direct surgical TDD, **not** Voyage-pipeline dogfood.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Decision Register — RESOLVED (operator 2026-06-20)
|
||||||
|
|
||||||
|
The analysis deferred three forks to the operator. All three resolved to the conservative option (preserve capability / security code / the deliberate pin):
|
||||||
|
|
||||||
|
| # | Item | **Resolution** | Effect on plan |
|
||||||
|
|---|------|----------------|----------------|
|
||||||
|
| **D1** | V09 gemini-bridge | **Keep it as an agent** (count stays 24; model governed by D3) | No removal/rewiring. Model stays opus (D3). |
|
||||||
|
| **D2** | V32 observability export | **Keep `lib/exporters/*` + `otel-export.mjs`; document the direct-export rationale** | Doc-only, no deletion of S21 security code. |
|
||||||
|
| **D3** | 24-agents `model: opus` pin (`40d8742`) | **Keep the pin firm** | V09/V35/V08/V11/V16 model changes → **document-only** (record considered-and-kept). No guard-test change. |
|
||||||
|
|
||||||
|
**Consequence:** the model + observability work collapses from code-deletion to a documentation record. Real code remains in V15 (export trim) and V30 (economy calibration). The plan is now **4 sessions** (2 code, 2 doc/audit).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cross-cutting constraints (every session)
|
||||||
|
|
||||||
|
- **No Handover-1 change.** The brief schema is the public contract; nothing here touches it. (V05 memory-alignment, the lone `breaking` capability, is KEEP-as-is — out of scope.)
|
||||||
|
- **Doc-consistency is guard-tested** (`tests/lib/doc-consistency.test.mjs`, `tests/lib/agent-frontmatter.test.mjs`). Any inventory/count/model wording change must keep them green and stay coherent with README / CLAUDE.md + the "24 agents opus" references in `docs/voyage-vs-cc-balance-analysis.md` + STATE. Fix the SOURCE a pin guards, not the test.
|
||||||
|
- **Test baseline:** 729 (`727/2/0`, bar `node --test`). Each session re-baselines + states the new count. `claude plugin validate` stays green (1 accepted warning).
|
||||||
|
- **Line numbers rot** — re-grep the live file before editing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Session sequence (finalized — 4 sessions)
|
||||||
|
|
||||||
|
### S31 — V15: trim plan-export to its load-bearing variant · [CODE · non-breaking · no pin]
|
||||||
|
- **Goal:** drop the `pr` / `issue` / `markdown` export variants from `trekplan --export` (CC auto-mode reformats markdown ad-hoc equally well); keep `--export headless` and relabel it as the decomposition entry it actually is.
|
||||||
|
- **Files:** `commands/trekplan.md` (Phase 1.5 `--export`), `docs/command-modes.md`, export tests under `tests/`.
|
||||||
|
- **TDD:** failing test asserting `pr|issue|markdown` are gone and `--export headless`(decompose) still resolves → implement → green.
|
||||||
|
- **Verify:** `node --test` (new count); `grep -c "export pr" commands/trekplan.md` = 0; `--export headless` intact; plugin validate.
|
||||||
|
- **Depends-on:** none.
|
||||||
|
|
||||||
|
### S32 — V01 + V07/V08/V11/V24: delegate-to-native hygiene · [CODE + AUDIT · non-breaking]
|
||||||
|
- **Goal:** (V01) delegate the literal brief-interview Q&A turn-taking to `AskUserQuestion` rather than a hand-rolled selection loop; (V07/V08/V11/V24) audit that the research/exploration/reviewer swarms ride native parallel Agent spawn + `AskUserQuestion` and re-implement no engine. Fix any real re-implementation; otherwise document the "delegate the engine, keep the policy" principle.
|
||||||
|
- **Files:** `commands/trekbrief.md` (Phase 3), `commands/trekresearch.md`, `commands/trekplan.md`, `commands/trekreview.md`, `docs/architecture.md` (principle note).
|
||||||
|
- **TDD:** where an edit is made, a test pinning the delegated behavior; the audit portion yields a documented finding (no code change if already native).
|
||||||
|
- **Verify:** `node --test`; plugin validate; principle note present in `docs/architecture.md`.
|
||||||
|
- **Depends-on:** none.
|
||||||
|
|
||||||
|
### S33 — Documentation consolidation: record the considered-and-kept decisions · [DOC · non-breaking]
|
||||||
|
Bundles the three items that D1–D3 turned into documentation:
|
||||||
|
- **V35 (doc half):** relabel `planning- / research- / review-orchestrator` as **reference docs, not spawnable capabilities**; reconcile the "24 agents" framing (= 21 spawnable + 3 reference docs + 1 dormant `synthesis-agent`) across README / CLAUDE.md / docs.
|
||||||
|
- **V32 rationale (D2):** add to `docs/observability.md` why custom exporters + SSRF/path/field guards rather than a native collector (preserves S21 hardening; deliberate direct-export choice).
|
||||||
|
- **Kept-opus rationale (D3):** record that opus on V09 (glue), V35 (dormant), V08/V11/V16 (mechanical/retrieval) was reconsidered and kept (pin `40d8742` stands) — a short note in the analysis doc / CLAUDE.md, no frontmatter change.
|
||||||
|
- **Files:** `README.md`, `CLAUDE.md`, `agents/*-orchestrator.md` (header clarity), `docs/observability.md`, `docs/voyage-vs-cc-balance-analysis.md` (decision-record addendum), `tests/lib/doc-consistency.test.mjs`.
|
||||||
|
- **TDD:** doc-consistency pin updated to the reconciled inventory framing → green.
|
||||||
|
- **Verify:** `node --test`; counts/model claims coherent across all docs; `agent-frontmatter.test.mjs` unchanged + green (no model change).
|
||||||
|
- **Depends-on:** none (settles the inventory baseline).
|
||||||
|
|
||||||
|
### S34 — V30: economy-profile Jaccard calibration · [CODE · non-breaking · pin-adjacent] ✅ DONE (2026-06-20)
|
||||||
|
- **Goal:** the `economy`-profile Jaccard floor (0.55) is grounded in parked synthetic fixtures (Step-17 calibration deferred). Either run the calibration against real fixtures and replace the floor, **or** clearly relabel `economy` as experimental/uncalibrated in docs + the profile.
|
||||||
|
- **Resolution: Path B — label, not calibrate.** The empirical run is v4.2-budget-gated ($60–120, unauthorized), so the fork is forced. `lib/profiles/economy.yaml` now carries `experimental: true` (validator type-checks the optional boolean); README + `docs/operations.md` + `docs/profiles.md` flag the `economy` row; the calibration doc cross-references the marker. (`lib/parsers/profile-jaccard.mjs` was untouched — it was listed for the calibration path only.)
|
||||||
|
- **TDD:** 5 tests — economy declares `experimental: true`, premium/balanced do not, validator rejects non-boolean `experimental`, every profile-doc economy row flagged experimental, and the flag tracks the calibration's `parked-synthetic` status. Baseline 739 → **744 (742/2/0)**.
|
||||||
|
- **Verify:** ✅ `node --test` green; ✅ `claude plugin validate` (1 accepted warning); economy's status now unambiguous + machine-checked.
|
||||||
|
- **Depends-on:** none (lowest urgency; placed last).
|
||||||
|
|
||||||
|
**Backlog complete — 4/4 sessions shipped (S31–S34). All 8 §6 items disposed; 2 real code changes (V15 export-trim, V30 economy-label), the rest doc/audit.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verification (plan-level)
|
||||||
|
|
||||||
|
- Every §6 backlog row maps to exactly one session: V15→S31; V01+V07/V08/V11/V24→S32; V35+V32+opus-pin record→S33; V30→S34. (8 items → 4 sessions after the conservative decisions.)
|
||||||
|
- No session changes Handover 1; no operator pin overridden (D3 kept firm).
|
||||||
|
- Each session ends green on `node --test` + `claude plugin validate`, doc-consistency reconciled.
|
||||||
|
|
||||||
|
## Out of scope (explicitly NOT in this plan)
|
||||||
|
|
||||||
|
- No gemini-bridge removal (D1 keep); no exporter deletion (D2 keep); no model downgrades (D3 firm).
|
||||||
|
- No wholesale Workflow substrate swap (CC-27 DECLINED; V27 stays opt-in); no delegated-orchestration redesign (CC-26 lean-NO).
|
||||||
|
- No brief-schema (Handover-1) change; no reopening of premium-default / framing-gate pins.
|
||||||
40
docs/claudemd-token-trim-brief.md
Normal file
40
docs/claudemd-token-trim-brief.md
Normal file
|
|
@ -0,0 +1,40 @@
|
||||||
|
# Brief — trim voyage CLAUDE.md to invariants (always-loaded token reduction)
|
||||||
|
|
||||||
|
**Author:** config-audit machine-tuning loop (KTG). **Date:** 2026-06-29.
|
||||||
|
**Why config-audit didn't do this directly:** voyage is the operator's plugin — config-audit only writes briefs here, never code. Run this in a voyage session.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
`CLAUDE.md` is injected **every turn** while working in the voyage repo (the whole per-repo always-loaded delta). Measured with config-audit `manifest`: **2,261 always-loaded tokens**. The tables (commands, agents) are invariant; the cost sits in six design-history / context **block-quote notes** whose detail already lives in referenced docs. Moving them out (leaving a terse invariant + pointer) should cut **~900–1,200 tok (~40–50 %)** with zero capability loss.
|
||||||
|
|
||||||
|
## Hard constraints (do NOT break these)
|
||||||
|
|
||||||
|
- `tests/lib/doc-consistency.test.mjs` reads CLAUDE.md. Keep the **Commands table** and the **Agents table including the `Model` column** and any **count** the test cross-checks. After editing, `node --test 'tests/**/*.test.mjs'` and `bash verify.sh` must stay green.
|
||||||
|
- Keep every **invariant fact** — just relocate the verbose *rationale/history* to the doc that already owns it, and leave a one-line pointer. Don't delete facts; move them.
|
||||||
|
- Docs-only change → no version bump / no catalog ref unless the doc-consistency test demands a badge sync (it shouldn't for a CLAUDE.md prose trim).
|
||||||
|
|
||||||
|
## Trim targets (line numbers as of commit at brief time; match by content)
|
||||||
|
|
||||||
|
| Block | ~chars / ~tok | Action | Destination (already referenced) |
|
||||||
|
|-------|---------------|--------|----------------------------------|
|
||||||
|
| L5 design-principle parenthetical (synthesis-PoC Δ≈0 caveat) | 620 / ~155 | **Keep the honesty caveat in one short clause** ("main-context relief is asserted-by-design, not measured — see doc"), move the PoC explanation out | `docs/T1-synthesis-poc-results.md` |
|
||||||
|
| L7 v3.0.0 architect-extraction note | 376 / ~94 | Replace with a one-liner: the plan command auto-discovers `architecture/overview.md` if present (migration history → CHANGELOG) | `CHANGELOG.md` |
|
||||||
|
| L9 Trinity context (Tier 1/2/3) | 859 / ~215 | Keep the **asymmetry invariant** terse ("Voyage stays unaware of Tier 2/3; Handover 1 = the only integration point; brief-schema changes are breaking"), move the Tier 2/3 description out | `docs/HANDOVER-CONTRACTS.md` §Handover 1 |
|
||||||
|
| **L11 brief-framing 3-layer gate** | **1,585 / ~396** | Biggest. Keep one invariant line ("brief framing must match operator intent — enforced as the `brief_version 2.2` gate: `framing` frontmatter + memory-alignment dim 6 + `## TL;DR`; all BLOCKER for ≥2.2"), move the implementation detail out | `docs/HANDOVER-CONTRACTS.md` §Handover 1 (PUBLIC CONTRACT) |
|
||||||
|
| L56 S33 agent-inventory reconcile | 740 / ~185 | Keep the terse fact ("24 agent files = 21 spawnable + 3 orchestrator reference docs; `synthesis-agent` dormant; all `opus`"), move the sonnet-downgrade rationale out | `docs/voyage-vs-cc-balance-analysis.md` §10 |
|
||||||
|
| L58 Model & effort axes | 588 / ~147 | Keep one line ("`opus`=Opus 4.8 / `high`; select agents carry native `effort:`; distinct from brief `phase_signals.effort`"), move the per-agent effort table + axes explanation out | `docs/profiles.md` §Model & effort axes |
|
||||||
|
|
||||||
|
**Net:** ~2,261 → ~1,300 tok. (Verify the after-figure with `node /path/to/config-audit/scanners/manifest.mjs <voyage-path>` → the `project` `claude-md` source.)
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
1. For each block above, confirm the destination doc already contains the detail (it's referenced, so it should) — if not, move the prose there first.
|
||||||
|
2. Replace each block-quote with its terse invariant + pointer.
|
||||||
|
3. Leave Commands/Agents tables byte-exact (counts + model column).
|
||||||
|
4. `node --test 'tests/**/*.test.mjs'` + `bash verify.sh` green; confirm `doc-consistency` passes.
|
||||||
|
5. Re-measure; commit `docs(claude-md): trim CLAUDE.md to invariants` to voyage's Forgejo. Δ lands on the machine after operator `/exit` + reload.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- This mirrors the same trim already done directly in config-audit (−662 tok), linkedin-studio (−2,266), ms-ai-architect (−346), okr (−15), portfolio-optimiser (−93) — same principle: CLAUDE.md carries invariants for working on the plugin; history/rationale lives in CHANGELOG + docs/.
|
||||||
|
- The voyage honesty culture (no overclaiming) is *preserved*, not trimmed: the synthesis-PoC Δ≈0 caveat stays as a one-liner; only the long-form explanation moves to its doc.
|
||||||
|
|
@ -9,7 +9,7 @@ Per-command flag tables, imported from `CLAUDE.md` via pointer.
|
||||||
| _(default)_ | Dynamic interview until quality gates pass → brief.md with research plan |
|
| _(default)_ | Dynamic interview until quality gates pass → brief.md with research plan |
|
||||||
| `--quick` | Compact start; still escalates if required sections are weak or the brief-review gate fails → brief.md with research plan |
|
| `--quick` | Compact start; still escalates if required sections are weak or the brief-review gate fails → brief.md with research plan |
|
||||||
| `--gates {true\|false}` | (v3.4.0) Boolean autonomy-gate flag; present → gating on. Policy (`gates_mode`) detailed under `## Autonomy mode` in `docs/operations.md`. |
|
| `--gates {true\|false}` | (v3.4.0) Boolean autonomy-gate flag; present → gating on. Policy (`gates_mode`) detailed under `## Autonomy mode` in `docs/operations.md`. |
|
||||||
| `--profile <name>` | (v4.1.0) Model profile: `economy` / `balanced` / `premium` / `<custom>`. Sets `phase_models` for the brief phase. See `## Profile system` in `docs/operations.md`. |
|
| `--profile <name>` | (v4.1.0) Model profile: `economy` / `balanced` / `premium` / `fable` / `<custom>`. Sets `phase_models` for the brief phase. See `## Profile system` in `docs/operations.md`. |
|
||||||
|
|
||||||
Always interactive. Phase 3 is a section-driven completeness loop (no hard cap on question count); Phase 4 runs a `brief-reviewer` stop-gate with max 3 review iterations. After writing the brief, asks the user to choose manual (print commands) or auto (Claude runs research + plan in foreground).
|
Always interactive. Phase 3 is a section-driven completeness loop (no hard cap on question count); Phase 4 runs a `brief-reviewer` stop-gate with max 3 review iterations. After writing the brief, asks the user to choose manual (print commands) or auto (Claude runs research + plan in foreground).
|
||||||
|
|
||||||
|
|
@ -26,6 +26,7 @@ Always interactive. Phase 3 is a section-driven completeness loop (no hard cap o
|
||||||
| `--gates {true\|false}` | (v3.4.0) Boolean autonomy-gate flag; present → gating on. Policy (`gates_mode`) detailed under `## Autonomy mode` in `docs/operations.md`. |
|
| `--gates {true\|false}` | (v3.4.0) Boolean autonomy-gate flag; present → gating on. Policy (`gates_mode`) detailed under `## Autonomy mode` in `docs/operations.md`. |
|
||||||
| `--min-brief-version <ver>` | (S18) Warn — never block — if an attached `--project` brief declares a version below `<ver>` (e.g. `2.2`), i.e. sidesteps framing enforcement |
|
| `--min-brief-version <ver>` | (S18) Warn — never block — if an attached `--project` brief declares a version below `<ver>` (e.g. `2.2`), i.e. sidesteps framing enforcement |
|
||||||
| `--profile <name>` | (v4.1.0) Model profile for the research phase. |
|
| `--profile <name>` | (v4.1.0) Model profile for the research phase. |
|
||||||
|
| `--engine {swarm\|deep-research}` | (deep-research-engine) Opt-in external-research engine; `deep-research` delegates the external phase to Claude Code's built-in `/deep-research` workflow (CC 2.1.154+), falls back to `swarm`. Default `swarm`. |
|
||||||
|
|
||||||
Flags combine: `--project <dir> --local`, `--external --quick`.
|
Flags combine: `--project <dir> --local`, `--external --quick`.
|
||||||
|
|
||||||
|
|
@ -39,7 +40,7 @@ Flags combine: `--project <dir> --local`, `--external --quick`.
|
||||||
| `--fg` | No-op alias (foreground is default since v2.4.0) |
|
| `--fg` | No-op alias (foreground is default since v2.4.0) |
|
||||||
| `--quick` | Plan directly (no agent swarm) |
|
| `--quick` | Plan directly (no agent swarm) |
|
||||||
| `--min-brief-version <ver>` | (S18) Warn — never block — if the brief declares a version below `<ver>` (e.g. `2.2`), i.e. sidesteps framing enforcement |
|
| `--min-brief-version <ver>` | (S18) Warn — never block — if the brief declares a version below `<ver>` (e.g. `2.2`), i.e. sidesteps framing enforcement |
|
||||||
| `--export <pr\|issue\|markdown\|headless> <plan>` | Generate shareable output from existing plan |
|
| `--export headless <plan>` | Legacy alias for `--decompose` — the only remaining export format (the `pr` / `issue` / `markdown` variants were removed; Claude reformats a plan ad-hoc on request) |
|
||||||
| `--decompose <plan>` | Split plan into self-contained headless sessions |
|
| `--decompose <plan>` | Split plan into self-contained headless sessions |
|
||||||
| `--gates {true\|false}` | (v3.4.0) Boolean autonomy-gate flag; present → gating on. Policy (`gates_mode`) detailed under `## Autonomy mode` in `docs/operations.md`. |
|
| `--gates {true\|false}` | (v3.4.0) Boolean autonomy-gate flag; present → gating on. Policy (`gates_mode`) detailed under `## Autonomy mode` in `docs/operations.md`. |
|
||||||
| `--profile <name>` | (v4.1.0) Model profile for the plan phase (and others, since plan emits `profile:` to plan.md frontmatter). |
|
| `--profile <name>` | (v4.1.0) Model profile for the plan phase (and others, since plan emits `profile:` to plan.md frontmatter). |
|
||||||
|
|
|
||||||
62
docs/deep-research-engine-brief.md
Normal file
62
docs/deep-research-engine-brief.md
Normal file
|
|
@ -0,0 +1,62 @@
|
||||||
|
---
|
||||||
|
type: trekbrief
|
||||||
|
brief_version: "2.2"
|
||||||
|
task: "Opt-in /deep-research-motor for /trekresearch ekstern-fase"
|
||||||
|
slug: deep-research-engine
|
||||||
|
framing: refine
|
||||||
|
status: ready
|
||||||
|
brief_quality: complete
|
||||||
|
research_topics: 3
|
||||||
|
research_status: complete
|
||||||
|
phase_signals_partial: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# Brief — Opt-in `/deep-research`-motor for `/trekresearch`
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
- `--engine {swarm|deep-research}` på `/trekresearch` ekstern-fase; `swarm` default (uendret oppførsel).
|
||||||
|
- `deep-research` delegerer ekstern research til Claude Codes innebygde workflow og adapterer inn i research-brief-skjemaet.
|
||||||
|
- Lokal analyse, triangulering og H2-output (`research/NN-*.md`) er motor-uavhengig.
|
||||||
|
- Feature-detekteres med auto-fallback til `swarm` — aldri hard feil.
|
||||||
|
- Surface-only: `commands/` + `agents/`, ingen nye `lib/`-avhengigheter.
|
||||||
|
|
||||||
|
## Intent
|
||||||
|
`/trekresearch` eier i dag hele den eksterne research-fasen selv (Tavily / MS-Learn / Gemini-sverm). Det er portabelt, men du vedlikeholder fan-out, kryssjekk og syntese selv. Anthropic sin innebygde `/deep-research` vedlikeholder fan-out, adversariell påstandsverifisering, sitatfiltrering og websøk for deg. Lar vi operatøren *velge* `/deep-research` for den eksterne fasen, får brukeren en vedlikeholdsfri "turbo" når den er tilgjengelig — uten at Voyage mister sin egen lokal-analyse, triangulering eller H2-kontrakt.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
`/trekresearch` får et `--engine {swarm|deep-research}`-valg på den eksterne fasen. `swarm` er default (dagens oppførsel). `deep-research` delegerer den eksterne fasen til den innebygde workflowen og adapterer resultatet inn i research-brief-skjemaet. Lokal analyse, triangulering og H2-output (`research/NN-*.md`) er uendret uansett motor.
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
- Bygge om `/trekresearch` på et eget dynamic-workflow-substrat (vurdert og forkastet, Δ≈0).
|
||||||
|
- Multi-provider (Perplexity / Google) — hører til en senere egen `workflow`-motor, ikke denne.
|
||||||
|
- Endre den lokale svermen, handover-kontraktene eller validator-skjemaene.
|
||||||
|
- Gjøre `/deep-research` til default.
|
||||||
|
|
||||||
|
## Constraints / NFR
|
||||||
|
- `/deep-research` krever Claude Code v2.1.154+ og må være på (Pro: via `/config`). Må feature-detekteres med auto-fallback til `swarm` — aldri hard feil.
|
||||||
|
- Output må passere `research-validator.mjs` (confidence ∈ [0,1], dimensions ≥ 1, påkrevde body-seksjoner).
|
||||||
|
- Ingen nye `lib/`-avhengigheter; overflate-endring i `commands/` + `agents/`.
|
||||||
|
- Doc-consistency: oppdater command-tabellen i `CLAUDE.md` + `README.md`, kjør `npm test`.
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
1. `/trekresearch --external --engine swarm "<q>"` gir identisk oppførsel som før (regresjon grønn).
|
||||||
|
2. `/trekresearch --external --engine deep-research "<q>"` produserer en `research/NN-*.md` der `research-validator.mjs --json` rapporterer `valid: true`.
|
||||||
|
3. Med dynamic workflows avskrudd faller `--engine deep-research` automatisk tilbake til `swarm` og logger valget — ingen exception.
|
||||||
|
4. `npm test` (inkl. doc-consistency) er grønn.
|
||||||
|
|
||||||
|
## Research Plan
|
||||||
|
1. **Programmatisk trigging av `/deep-research`** — Kan en plugin-kommando starte den innebygde workflowen og fange artefakten, eller må `commands/trekresearch.md` instruere Claude til å kjøre den? Scope: external · Konfidens: høy · Kost: lav.
|
||||||
|
`/trekresearch --external "Kan en Claude Code plugin-kommando programmatisk starte /deep-research og hente rapporten?"`
|
||||||
|
2. **Output-adapter** — Hvordan mappe `/deep-research` sin siterte rapport til research-brief-skjemaet (dimensjoner, confidence, sitater)? Scope: both · Konfidens: høy · Kost: lav.
|
||||||
|
`/trekresearch --project <dir> --local "Hva krever research-validator.mjs av research/NN-*.md?"`
|
||||||
|
3. **Feature-deteksjon + fallback** — Hvordan oppdage om workflows er på, og hvor i fasen fallback-grenen bør sitte? Scope: local · Konfidens: middels · Kost: lav.
|
||||||
|
|
||||||
|
> **Research-status (2026-06-30, operatør-beslutning «option A»):** Topic 1 (eneste ekte eksterne ukjente) er undersøkt → `docs/deep-research-engine-research.md` (validator-grønn). Funn: `/deep-research` er en innebygd **dynamic workflow** (ikke skill) → trigging KUN via prosa-instruksjon, output **inline i kontekst** (ingen on-disk-artefakt) → motoren må være instruksjons-basert + in-context transform. **SC3 er korrekt som skrevet** (`/deep-research` ER en dynamic workflow). Topic 2 (validator-skjema: `type/created/question` + `## Executive Summary`/`## Dimensions`, `dimensions ≥ 1`) og topic 3 (fallback-plassering) er lokale kode-spørsmål reklassifisert til `/trekplan`-utforskning. `research_status: complete` reflekterer denne beslutningen.
|
||||||
|
|
||||||
|
## Open Questions / Assumptions
|
||||||
|
- Antar at `/deep-research`-rapporten kan reduseres til ≥ 1 dimensjon med per-påstand-sitater uten å bryte trianguleringen. Verifiseres i topic 2.
|
||||||
|
- Uavklart om delegering skjer via instruksjon (trygt) eller programmatisk API (raskere) — topic 1 avgjør.
|
||||||
|
|
||||||
|
## Prior Attempts
|
||||||
|
- Dvalende synthesis-agent bygget, målt (Δ≈0) og forkastet → samme lærdom gjelder substrat-bytte: mål før du adopterer.
|
||||||
|
- Workflow-substratet er allerede dokumentert som vurdert-og-valgt-bort i `docs/architecture.md` §Primitives per step.
|
||||||
210
docs/deep-research-engine-research.md
Normal file
210
docs/deep-research-engine-research.md
Normal file
|
|
@ -0,0 +1,210 @@
|
||||||
|
---
|
||||||
|
type: trekresearch-brief
|
||||||
|
created: 2026-06-30
|
||||||
|
question: "Can a Claude Code plugin command programmatically trigger the built-in /deep-research workflow and capture its report, or must it instruct Claude to run it?"
|
||||||
|
confidence: 0.85
|
||||||
|
dimensions: 4
|
||||||
|
mcp_servers_used: []
|
||||||
|
local_agents_used: [claude-code-guide]
|
||||||
|
external_agents_used: []
|
||||||
|
slug: deep-research-engine
|
||||||
|
feeds_brief: docs/deep-research-engine-brief.md
|
||||||
|
research_topic: 1
|
||||||
|
---
|
||||||
|
|
||||||
|
# Research — Programmatic trigging of `/deep-research` from a plugin command
|
||||||
|
|
||||||
|
> Targeted single-topic research for the `deep-research-engine` brief (Topic 1 of
|
||||||
|
> the brief's Research Plan). Topics 2 & 3 are local Voyage-code questions folded
|
||||||
|
> into `/trekplan` exploration; only Topic 1 was a genuine external unknown.
|
||||||
|
> Method: `claude-code-guide` agent (Anthropic docs + CHANGELOG, cited) +
|
||||||
|
> direct inspection of this machine (CC 2.1.196 binary, a real local `/deep-research`
|
||||||
|
> run). Not run through the `/trekresearch` swarm — see brief reconcile note.
|
||||||
|
|
||||||
|
## Research Question
|
||||||
|
|
||||||
|
Can a Claude Code **plugin slash-command** (markdown under `commands/`)
|
||||||
|
**programmatically** start the built-in `/deep-research` workflow and **capture its
|
||||||
|
report artifact** for adaptation into the research-brief schema — or must the command
|
||||||
|
instead **instruct** Claude (in prose) to run `/deep-research` and then transform the
|
||||||
|
in-context result?
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
Programmatic trigger + file-based capture is **not feasible**: `/deep-research` is a
|
||||||
|
built-in **dynamic workflow** (not a skill), deliberately outside the Skill-tool
|
||||||
|
allowlist, and it returns its report **inline into conversation context with no
|
||||||
|
documented on-disk report artifact**. The **only reliable path is instruction-based
|
||||||
|
delegation** — the command's prose tells Claude to run `/deep-research <q>`, then
|
||||||
|
transforms the in-context report in the same turn. Confidence **high** (Anthropic docs +
|
||||||
|
CHANGELOG + local run), with one residual gap: there is no positive "is it enabled?"
|
||||||
|
probe, so feature-detection must lean on the documented *disable* switches + version
|
||||||
|
floor + a `swarm` default.
|
||||||
|
|
||||||
|
## Dimensions
|
||||||
|
|
||||||
|
### 1. Origin + gating -- Confidence: high
|
||||||
|
|
||||||
|
**External findings:**
|
||||||
|
- `/deep-research` is a **built-in dynamic workflow**, not a command and not a bundled
|
||||||
|
skill. `commands.md` marks the `/deep-research <question>` row as **[Workflow]**;
|
||||||
|
`workflows.md`: "Claude Code includes `/deep-research` as a built-in workflow."
|
||||||
|
[VERIFIED — code.claude.com/docs/en/commands.md, code.claude.com/docs/en/workflows.md]
|
||||||
|
- Gating: available on **all paid plans** (pro/max/team/enterprise) + API/Bedrock/Vertex/
|
||||||
|
Foundry. **On Pro it must be turned on** in the *Dynamic workflows* row of `/config`.
|
||||||
|
Also requires the **WebSearch tool** to be available.
|
||||||
|
[VERIFIED — workflows.md, commands.md bundled-workflows row]
|
||||||
|
- Version floor: dynamic workflows were **introduced in CC 2.1.154**; "Dynamic workflows
|
||||||
|
require Claude Code v2.1.154 or later." The brief's `v2.1.154+` + `(Pro: via /config)`
|
||||||
|
constraints are **both correct**. The exact version that first shipped the *named*
|
||||||
|
`/deep-research` workflow is **[NOT DOCUMENTED]** (only a 2.1.196 bugfix mentions it by
|
||||||
|
name) — treat 2.1.154 as the substrate floor, not a proven introduction point.
|
||||||
|
[VERIFIED — workflows.md + CHANGELOG 2.1.154]
|
||||||
|
|
||||||
|
**Local findings:**
|
||||||
|
- This machine runs **CC 2.1.196** (`claude --version`) — substrate floor satisfied.
|
||||||
|
- The exact skill-description string lives **compiled into the binary**
|
||||||
|
(`/Users/ktg/.local/share/claude/versions/2.1.196`, Mach-O 235 MB); there is **no
|
||||||
|
`SKILL.md`** for it anywhere under `~/.claude` (system-wide `find`/`grep` — only hits
|
||||||
|
are this brief + unrelated harness notes). Confirms "Anthropic-bundled, not user skill."
|
||||||
|
|
||||||
|
### 2. Invocation mechanism (programmatic vs instruction) -- Confidence: high
|
||||||
|
|
||||||
|
**External findings:**
|
||||||
|
- **Not** via the `Skill` tool. The Skill-tool built-in allowlist is closed: only
|
||||||
|
`/init`, `/review`, `/security-review` are reachable; "Other built-in commands such as
|
||||||
|
`/compact` are not." `/deep-research` is a Workflow and is not on that list.
|
||||||
|
[VERIFIED — code.claude.com/docs/en/skills.md]
|
||||||
|
- **Instruction-based delegation is the documented mechanism.** Workflows launch when the
|
||||||
|
user types the command, or **when Claude is asked in natural language** ("use a
|
||||||
|
workflow" / "run a workflow") or via the `ultracode` keyword. A plugin command whose
|
||||||
|
markdown instructs Claude to run `/deep-research <q>` is therefore the supported path.
|
||||||
|
[VERIFIED — workflows.md "Have Claude write a workflow"]
|
||||||
|
- **Approval gate caveat:** launching a workflow triggers a per-run approval prompt —
|
||||||
|
*every run* in default/acceptEdits; *first launch only* in auto; **never in `claude -p`
|
||||||
|
/ Agent SDK / bypass-permissions** ("the run starts immediately"). Voyage's headless
|
||||||
|
surface (`claude -p`) thus delegates without an interactive gate; interactive sessions
|
||||||
|
hit a prompt. [VERIFIED — workflows.md "Behavior and limits"]
|
||||||
|
- No documented blanket "commands/skills cannot nest" prohibition beyond the Skill-tool
|
||||||
|
allowlist + workflow runtime limits (no mid-run user input; 16 concurrent agents;
|
||||||
|
1000 agents/run). [VERIFIED — workflows.md; NOT DOCUMENTED for a general nesting ban]
|
||||||
|
|
||||||
|
### 3. Output capture -- Confidence: high
|
||||||
|
|
||||||
|
**External findings:**
|
||||||
|
- "When the run finishes, **the report lands in your session**"; "Claude's context holds
|
||||||
|
only the final answer." The report is **in-context**, not a file.
|
||||||
|
[VERIFIED — workflows.md]
|
||||||
|
- What *is* written to disk is the orchestration **script**, not the report: "Every run
|
||||||
|
writes its script to a file under your session's directory in `~/.claude/projects/`."
|
||||||
|
[VERIFIED — workflows.md "How a workflow runs"]
|
||||||
|
|
||||||
|
**Local findings:**
|
||||||
|
- A real `/deep-research` run on this machine left exactly one file —
|
||||||
|
`~/.claude/projects/<session>/workflows/scripts/deep-research-wf_<id>.js` — and **no
|
||||||
|
`.md` report** beside it. [VERIFIED — local filesystem inspection by claude-code-guide]
|
||||||
|
- **Consequence:** a calling command cannot `grep` a results file off disk (none is
|
||||||
|
documented to exist). It can only consume the report **as it sits in conversation
|
||||||
|
context in the same turn** — Claude reads its own prior output and transforms it into
|
||||||
|
research-brief schema. [INFERENCE — file-based capture not feasible; in-context
|
||||||
|
transform is the only avenue]
|
||||||
|
|
||||||
|
### 4. Feature detection + fallback -- Confidence: medium
|
||||||
|
|
||||||
|
**External findings:**
|
||||||
|
- **No positive enumeration / "is-enabled" API or flag is documented.** `claude --help`
|
||||||
|
exposes `--disable-slash-commands` but no "list skills/workflows" or "is-feature-on"
|
||||||
|
flag. [VERIFIED — local `claude --help`; NOT DOCUMENTED for any positive probe]
|
||||||
|
- The documented signals are the **off-switches**, read defensively: `/config` Dynamic-
|
||||||
|
workflows off; `disableWorkflows: true` / `disableBundledSkills: true` in settings.json;
|
||||||
|
`CLAUDE_CODE_DISABLE_WORKFLOWS=1` / `CLAUDE_CODE_DISABLE_BUNDLED_SKILLS=1`.
|
||||||
|
[VERIFIED — workflows.md "Turn workflows off"; CHANGELOG 2.1.x]
|
||||||
|
- **The gap:** the Pro `/config` *on*-state is the very thing you most need to detect, and
|
||||||
|
only the *disable* keys are documented; the persisted key/value for the Pro enable-state
|
||||||
|
is **[NOT DOCUMENTED]**, so it cannot be reliably grepped. A failed/disabled
|
||||||
|
`/deep-research` is also **not documented** to raise a signal a sibling command can
|
||||||
|
catch. [VERIFIED gap]
|
||||||
|
|
||||||
|
**Local findings:**
|
||||||
|
- On this machine both disable keys are unset/absent — workflows are not disabled.
|
||||||
|
|
||||||
|
## External Knowledge
|
||||||
|
|
||||||
|
### Best Practice
|
||||||
|
The "Have Claude write a workflow" + "Behavior and limits" sections of `workflows.md`
|
||||||
|
establish that workflows are operator/Claude-launched, run isolated, and return one
|
||||||
|
in-context report. The Skill-tool allowlist (`skills.md`) is the authoritative statement
|
||||||
|
that only three built-ins are tool-invocable.
|
||||||
|
|
||||||
|
### Known Issues
|
||||||
|
Per-run approval prompts outside `-p`/SDK/bypass mean an interactive `/trekresearch
|
||||||
|
--engine deep-research` will pause for operator approval on each launch — acceptable, but
|
||||||
|
worth documenting in the command UX. The absence of a positive availability probe is the
|
||||||
|
single biggest design constraint (see Dimension 4).
|
||||||
|
|
||||||
|
## Synthesis
|
||||||
|
|
||||||
|
Three cross-cutting insights that only emerge from combining the docs with Voyage's brief:
|
||||||
|
|
||||||
|
1. **The brief's SC3 is correct as written — an earlier review note was wrong.** Because
|
||||||
|
`/deep-research` *is itself* a dynamic workflow, "med dynamic workflows avskrudd faller
|
||||||
|
`--engine deep-research` tilbake til swarm" is the right feature-detection axis. A
|
||||||
|
prior brief-review remark that SC3 "conflated `/deep-research` (skill) with dynamic
|
||||||
|
workflows" was based on a wrong premise (that `/deep-research` was a skill) and is
|
||||||
|
retracted. **No SC3 brief edit is needed.**
|
||||||
|
|
||||||
|
2. **The engine must be instruction-based + in-context, never file-based.** Topic 1's
|
||||||
|
open question ("instruction (safe) vs. programmatic API (faster) — topic 1 decides")
|
||||||
|
resolves decisively to **instruction-based**: there is no programmatic API and no
|
||||||
|
on-disk report. The `deep-research` engine path in `commands/trekresearch.md` must
|
||||||
|
(a) instruct Claude to run `/deep-research <q>`, then (b) transform the in-context
|
||||||
|
report into the `research/NN-*.md` schema in the same turn. This is surface-only
|
||||||
|
(`commands/` prose), matching the brief's "ingen nye `lib/`-avhengigheter."
|
||||||
|
|
||||||
|
3. **SC3's "ingen exception" cannot rest on runtime detection — pin it to a default.**
|
||||||
|
Since no positive availability probe exists, robust fallback = version-floor check
|
||||||
|
(≥ 2.1.154) + disable-key heuristic (`disableWorkflows` / env) + **`--engine` default
|
||||||
|
of `swarm`** (explicit opt-in). The fallback is "graceful degradation by design,"
|
||||||
|
not "catch an exception at runtime." This refines, but does not contradict, SC3.
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
- **Adapter fidelity (brief Topic 2, partly answered locally):** `research-validator.mjs`
|
||||||
|
requires `type: trekresearch-brief` + `created` + `question`; validates `confidence ∈
|
||||||
|
[0,1]` and `dimensions ≥ 1` if present; body must carry `## Executive Summary` +
|
||||||
|
`## Dimensions`. So the `/deep-research` report **can** be reduced to ≥ 1 dimension with
|
||||||
|
per-claim citations and a confidence number — the brief's assumption holds. The
|
||||||
|
remaining open part (how cleanly the in-context report maps to per-dimension
|
||||||
|
local/external splits) is a `/trekplan` exploration concern, not an external unknown.
|
||||||
|
- **Exact intro version of the *named* `/deep-research` workflow** — not documented; the
|
||||||
|
2.1.154 dynamic-workflows floor is the safe pin.
|
||||||
|
|
||||||
|
## Recommendation
|
||||||
|
|
||||||
|
Build the `deep-research` engine as **instruction-based delegation with in-context
|
||||||
|
adaptation**, not a programmatic trigger:
|
||||||
|
1. `--engine deep-research` makes `commands/trekresearch.md`'s external phase instruct
|
||||||
|
Claude to run `/deep-research <q>` and transform the returned in-context report into
|
||||||
|
`research/NN-*.md` (validator-conformant: `## Executive Summary` + `## Dimensions`,
|
||||||
|
`confidence`, `dimensions ≥ 1`).
|
||||||
|
2. Feature-detect by **graceful degradation**: version floor + disable-key heuristic +
|
||||||
|
`--engine` default `swarm`. Do not depend on a positive availability probe (none
|
||||||
|
exists). Log the chosen engine. This satisfies SC3 without a runtime exception.
|
||||||
|
3. Keep `swarm` the default (brief's Non-Goal: "ikke default-bytte"). Document the
|
||||||
|
per-run approval prompt for interactive (non-`-p`) sessions.
|
||||||
|
|
||||||
|
Confidence in the recommendation: **high** for the mechanism (instruction-based +
|
||||||
|
in-context), **medium** for the exact fallback-detection ergonomics (the one documented
|
||||||
|
gap). This is sufficient to green-light `/trekplan` with Topic 1 resolved.
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
| # | Source | Type | Quality | Used in |
|
||||||
|
|---|--------|------|---------|---------|
|
||||||
|
| 1 | code.claude.com/docs/en/workflows.md | official | high | Dim 1,2,3,4 + Synthesis |
|
||||||
|
| 2 | code.claude.com/docs/en/commands.md | official | high | Dim 1 (Workflow classification, WebSearch req) |
|
||||||
|
| 3 | code.claude.com/docs/en/skills.md | official | high | Dim 2 (Skill-tool allowlist) |
|
||||||
|
| 4 | github.com/anthropics/claude-code CHANGELOG (2.1.154, 2.1.x, 2.1.196) | official | high | Dim 1,4 (version floor, disable keys) |
|
||||||
|
| 5 | Local: CC 2.1.196 binary inspection (`find`/`grep`, `claude --version`) | codebase | high | Dim 1 (bundled, no SKILL.md) |
|
||||||
|
| 6 | Local: real `/deep-research` run — only `.js` script written, no `.md` report | codebase | high | Dim 3 (no on-disk artifact) |
|
||||||
|
| 7 | Local: `lib/validators/research-validator.mjs` schema | codebase | high | Open Questions (adapter feasibility / Topic 2) |
|
||||||
104
docs/eval-corpus/README.md
Normal file
104
docs/eval-corpus/README.md
Normal file
|
|
@ -0,0 +1,104 @@
|
||||||
|
# Eval corpus — frozen failures for Voyage's own agents
|
||||||
|
|
||||||
|
This directory is the home for the **golden / frozen-failure corpus** that
|
||||||
|
grounds Voyage's self-evaluation (SKAL-1·4a, the eval-foundation tier). It
|
||||||
|
follows the Anthropic "collect 20–50 real cases" practice: every time a Voyage
|
||||||
|
review/coordinator agent **misfires** on a real task, the case is distilled into
|
||||||
|
a machine-readable record and added here, so the failure can never silently
|
||||||
|
regress.
|
||||||
|
|
||||||
|
This corpus is the gold that the **offline gold-scored output eval (SKAL-1·4b)**
|
||||||
|
scores against — that eval is now implemented and wired into `node --test`
|
||||||
|
(see [§Gold-scored output eval](#gold-scored-output-eval-skal-14b) below). The
|
||||||
|
corpus records here are committed fixtures + a schema; the scoring run is the
|
||||||
|
separate piece that consumes them.
|
||||||
|
|
||||||
|
## Seed example
|
||||||
|
|
||||||
|
The first corpus entry is
|
||||||
|
[`tests/fixtures/bakeoff-rich/gold.json`](../../tests/fixtures/bakeoff-rich/gold.json)
|
||||||
|
— the 5 brief-traceable seeded findings of the bakeoff-rich JWT-auth fixture.
|
||||||
|
`gold.json` is the **canonical machine-readable form**; the prose table in
|
||||||
|
`tests/fixtures/bakeoff-rich/README.md` is illustrative. When they disagree,
|
||||||
|
`gold.json` wins.
|
||||||
|
|
||||||
|
## Record schema (`voyage-eval-gold/1`)
|
||||||
|
|
||||||
|
A corpus file is one JSON object:
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
{
|
||||||
|
"schema": "voyage-eval-gold/1",
|
||||||
|
"source": "<path to the prose/fixture this was distilled from>",
|
||||||
|
"description": "<one line>",
|
||||||
|
"expected_verdict": "BLOCK | WARN | ALLOW", // review-coordinator Pass-4 outcome
|
||||||
|
"findings": [
|
||||||
|
{
|
||||||
|
"file": "<repo-relative path in the reviewed diff>",
|
||||||
|
"line": 0, // integer >= 0; 0 = file-scoped
|
||||||
|
"rule_key": "<member of lib/review/rule-catalogue.mjs RULE_CATALOGUE>",
|
||||||
|
"severity": "BLOCKER | MAJOR | MINOR | SUGGESTION",
|
||||||
|
"owner_reviewer": "conformance | correctness"
|
||||||
|
// optional eval-extension fields are permitted, e.g.:
|
||||||
|
// "dual_flaggable": "<a second rule_key the same issue could carry>"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Hard constraints
|
||||||
|
|
||||||
|
- **`rule_key` must be a member of `RULE_CATALOGUE`** (`lib/review/rule-catalogue.mjs`).
|
||||||
|
The catalogue is the contract; an invented rule_key is a corpus bug.
|
||||||
|
- **`severity` must be one of `SEVERITY_VALUES`** (`BLOCKER`, `MAJOR`, `MINOR`, `SUGGESTION`).
|
||||||
|
- **`expected_verdict`** is the deterministic Pass-4 outcome of
|
||||||
|
`lib/review/coordinator-contract.mjs::computeVerdict`: `BLOCKER ≥ 1 → BLOCK`,
|
||||||
|
else `MAJOR ≥ 1 → WARN`, else `ALLOW`.
|
||||||
|
- Finding-level fields (`file`, `line`, `rule_key`, `severity`) mirror
|
||||||
|
`FINDING_REQUIRED_FIELDS` (`lib/review/findings-schema.mjs`); `owner_reviewer`
|
||||||
|
and any extension fields are eval-specific additions (superset).
|
||||||
|
|
||||||
|
## Adding a new frozen failure
|
||||||
|
|
||||||
|
1. Distil the misfire into a `voyage-eval-gold/1` record (one `.json` file here,
|
||||||
|
or a new entry in an existing corpus file).
|
||||||
|
2. Confirm every `rule_key` is in the catalogue and `expected_verdict` matches
|
||||||
|
the Pass-4 computation.
|
||||||
|
3. Add (or extend) a loader test in the `tests/lib/gold-corpus.test.mjs` shape so
|
||||||
|
the record's shape + catalogue membership are pinned under `node --test`.
|
||||||
|
|
||||||
|
## Gold-scored output eval (SKAL-1·4b)
|
||||||
|
|
||||||
|
The offline scoring run that grades a recorded agent run against this corpus.
|
||||||
|
**Offline** = committed reviewer payloads, no live agent spawn, no LLM, no
|
||||||
|
network (the LLM-in-the-loop grading is the separate 4c tier).
|
||||||
|
|
||||||
|
- **Committed runs** live under `tests/fixtures/bakeoff-rich/runs/`. Each file is
|
||||||
|
the JSON **reviewer payloads** (one object per reviewer, `{ reviewer, findings }`)
|
||||||
|
that a recorded run produced. `run-perfect.json` is the regression guard: fed
|
||||||
|
through the coordinator contract it must reproduce every seeded gold finding.
|
||||||
|
- **The contract** `lib/review/coordinator-contract.mjs::runContract(payloads)`
|
||||||
|
turns those payloads into the deterministic coordinator output (4a).
|
||||||
|
- **The scorer** `lib/review/gold-scorer.mjs`:
|
||||||
|
- `scoreFindings(runFindings, goldFindings)` matches at **`(file, rule_key)`
|
||||||
|
granularity** (line + severity ignored) → `{ tp, fp, fn, precision, recall,
|
||||||
|
f1, matched, missed, spurious }`. Vacuous-set conventions (empty run →
|
||||||
|
recall 0; empty gold → precision 0; f1 collapses to 0) are documented in the
|
||||||
|
module header.
|
||||||
|
- `scoreVerdict(runVerdict, goldVerdict)` → exact verdict match.
|
||||||
|
- **The scoring run** `tests/lib/gold-eval.test.mjs` asserts `run-perfect`
|
||||||
|
reproduces gold at precision/recall/f1 = 1.0 and `verdict === expected_verdict`.
|
||||||
|
- **Third test-census category.** `lib/util/test-census.mjs` now reports a
|
||||||
|
`goldEval` bucket (matched by `GOLD_EVAL_FILE_RE`) separately from `behavior`
|
||||||
|
and `docPins` — a scoring run is neither behavior coverage nor a prose pin, so
|
||||||
|
the honest-count invariant is now a 3-way sum.
|
||||||
|
|
||||||
|
## Future hardening (not in this tier)
|
||||||
|
|
||||||
|
- A prose↔JSON cross-assertion (parse the fixture README table, diff against
|
||||||
|
`gold.json`) to mechanically bound the two-source-of-truth drift.
|
||||||
|
- Degraded-run fixtures (a run that misses or invents findings) to exercise the
|
||||||
|
scorer's discriminating path end-to-end; today that path is covered by
|
||||||
|
`tests/lib/gold-scorer.test.mjs` with inline synthetic findings.
|
||||||
|
- SKAL-1·4c: the LLM-in-the-loop eval that grades live agent runs (needs a
|
||||||
|
filesystem + model judgement, deliberately excluded from this deterministic tier).
|
||||||
|
|
@ -60,6 +60,7 @@ operator-private data (paths, prompts, brief content).
|
||||||
| `VOYAGE_TEXTFILE_DIR` | `${CLAUDE_PLUGIN_DATA}` | Directory for `voyage.prom` (textfile mode) |
|
| `VOYAGE_TEXTFILE_DIR` | `${CLAUDE_PLUGIN_DATA}` | Directory for `voyage.prom` (textfile mode) |
|
||||||
| `VOYAGE_OTEL_ENDPOINT` | _(none)_ | HTTPS URL for OTLP/HTTP POST |
|
| `VOYAGE_OTEL_ENDPOINT` | _(none)_ | HTTPS URL for OTLP/HTTP POST |
|
||||||
| `VOYAGE_OTEL_ALLOW_PRIVATE` | _(unset)_ | Set to `1` to allow loopback / RFC1918 endpoints |
|
| `VOYAGE_OTEL_ALLOW_PRIVATE` | _(unset)_ | Set to `1` to allow loopback / RFC1918 endpoints |
|
||||||
|
| `VOYAGE_TOKEN_METER` | _(unset)_ | Set to a truthy value to capture per-session token/cost into `token-usage-stats.jsonl` on Stop (default off → zero added latency). See **Token/cost metering** below. |
|
||||||
|
|
||||||
## Docker Compose quickstart
|
## Docker Compose quickstart
|
||||||
|
|
||||||
|
|
@ -88,6 +89,47 @@ the allowlist explicitly. This is intentional: `${CLAUDE_PLUGIN_DATA}` is
|
||||||
trusted local storage; OTel endpoints are operator-controlled and may be
|
trusted local storage; OTel endpoints are operator-controlled and may be
|
||||||
external.
|
external.
|
||||||
|
|
||||||
|
## Token/cost metering
|
||||||
|
|
||||||
|
> **SKAL-2.** Opt-in capture of per-session token usage and a cache-aware USD
|
||||||
|
> cost estimate, folded into the existing Stop hook (`hooks/scripts/otel-export.mjs`).
|
||||||
|
> No new hook is added — capture rides the same Stop event, so there is no
|
||||||
|
> second-Stop-hook ordering race.
|
||||||
|
|
||||||
|
**Activation.** Set `VOYAGE_TOKEN_METER` to any truthy value. When unset (the
|
||||||
|
default) the capture path is skipped entirely — zero added Stop latency. When
|
||||||
|
set, the hook reads the Claude Code transcript (`transcript_path` from the Stop
|
||||||
|
payload), sums token usage, derives cost, and **upserts** one record per session
|
||||||
|
into `${CLAUDE_PLUGIN_DATA}/token-usage-stats.jsonl`. Capture is fail-open: any
|
||||||
|
error (malformed payload, unreadable transcript) is swallowed and never blocks
|
||||||
|
Stop. Once captured, the record is exported like any other stats file when
|
||||||
|
`VOYAGE_EXPORT_MODE` is `textfile` or `otlp`.
|
||||||
|
|
||||||
|
**Schema (`token-usage`).** Flat numeric record:
|
||||||
|
`ts`, `session_id`, `scope`, `model`, `tokens_input`, `tokens_output`,
|
||||||
|
`tokens_cache_creation`, `tokens_cache_read`, `cost_usd`, `is_estimate`,
|
||||||
|
`price_table_version`. The field allowlist (`lib/exporters/field-allowlist.mjs`,
|
||||||
|
`TOKEN_USAGE_ALLOWED`) admits only the numeric + low-cardinality-label fields and
|
||||||
|
**strips `session_id` at export** (CWE-212). The exporter auto-promotes each
|
||||||
|
numeric field to a metric: `voyage_token_usage_tokens_input` (Prometheus) /
|
||||||
|
`voyage.token-usage.tokens_input` (OTLP).
|
||||||
|
|
||||||
|
**Cost contract (honesty).** `cost_usd` is computed from a dated, in-source
|
||||||
|
`PRICE_TABLE` (per-MTok USD), cache-aware:
|
||||||
|
`input + output + cache_creation×(5m write rate) + cache_read×(read rate)`.
|
||||||
|
Each record carries `price_table_version` (the date the prices were resolved)
|
||||||
|
and `is_estimate`. When the transcript's model is **not** in the price table, the
|
||||||
|
meter **refuses to guess**: `cost_usd` is `null` and `is_estimate` is `true`.
|
||||||
|
Prices are volatile — re-resolve them against the `claude-api` reference and bump
|
||||||
|
`PRICE_TABLE_VERSION` when they change.
|
||||||
|
|
||||||
|
**v1 limitation — MAIN-CONTEXT only.** The meter reads the main-session
|
||||||
|
transcript, which contains only `isSidechain:false` records. Sub-agent (swarm)
|
||||||
|
turns live in separate `agent-*.jsonl` sibling files and are **not** counted, so
|
||||||
|
`cost_usd` is a lower bound on total Voyage cost, not the session total. Every
|
||||||
|
record is stamped `scope:'main-context'` to make this explicit. Per-subagent
|
||||||
|
attribution is a documented v2 follow-on (it was a Non-Goal for v1).
|
||||||
|
|
||||||
## Security
|
## Security
|
||||||
|
|
||||||
The exporter is hardened against three CWE classes:
|
The exporter is hardened against three CWE classes:
|
||||||
|
|
@ -114,6 +156,40 @@ The exporter is hardened against three CWE classes:
|
||||||
| `prom/node-exporter` | `1.10.2` | textfile collector path normalization |
|
| `prom/node-exporter` | `1.10.2` | textfile collector path normalization |
|
||||||
| `grafana/grafana` | `11.4.0` | datasource provisioning hardening |
|
| `grafana/grafana` | `11.4.0` | datasource provisioning hardening |
|
||||||
|
|
||||||
|
## Why direct export rather than a native collector
|
||||||
|
|
||||||
|
A balance review (`docs/voyage-vs-cc-balance-analysis.md` §4, V32) asked
|
||||||
|
whether the custom exporters should be dropped in favour of pointing the
|
||||||
|
standard `OTEL_*` environment variables at a co-located OTLP collector,
|
||||||
|
letting that collector own egress and field selection. The operator
|
||||||
|
decision (D2, 2026-06-20) is to **keep direct export**. The rationale is
|
||||||
|
the security boundary, not a preference for re-hosting a collector:
|
||||||
|
|
||||||
|
- **The three guards run in-process, before any byte leaves Voyage.**
|
||||||
|
`path-validator.mjs` (CWE-22), `endpoint-validator.mjs` (CWE-918 / SSRF),
|
||||||
|
and `field-allowlist.mjs` (CWE-212) are applied inside `otel-export.mjs`
|
||||||
|
and covered by `tests/hooks/otel-export-validators.test.mjs`. The
|
||||||
|
records carry operator-private data (paths, prompts, brief content);
|
||||||
|
the allowlist drops everything not explicitly named before export.
|
||||||
|
- **A native-collector design moves that boundary out of audited code.**
|
||||||
|
Handing raw JSONL to a sidecar collector means either re-expressing the
|
||||||
|
field allowlist in collector config (a second source of truth that can
|
||||||
|
drift) or shipping un-allowlisted private fields and trusting the
|
||||||
|
collector's egress rules. The S21 SSRF hardening — 169.254.169.254
|
||||||
|
permanently blocked, loopback/RFC1918 gated behind
|
||||||
|
`VOYAGE_OTEL_ALLOW_PRIVATE` — is a property of `endpoint-validator.mjs`
|
||||||
|
and would have to be re-created in collector configuration to be
|
||||||
|
preserved.
|
||||||
|
- **The collector path is still available, by design.** Operators who
|
||||||
|
want collector semantics (retry, persistence, relabelling) use
|
||||||
|
`textfile` mode and scrape `voyage.prom` with node-exporter / vector /
|
||||||
|
otel-collector. Direct export is the minimal default, not a rejection
|
||||||
|
of collectors — it keeps the data-sanitization boundary in Voyage's
|
||||||
|
own validated code for the common case.
|
||||||
|
|
||||||
|
This is a deliberate direct-export-over-collector choice; the custom
|
||||||
|
exporters and their guards are kept, not pruned.
|
||||||
|
|
||||||
## Limitations
|
## Limitations
|
||||||
|
|
||||||
- **Stop-hook is normal-exit only.** If Claude Code crashes or is killed
|
- **Stop-hook is normal-exit only.** If Claude Code crashes or is killed
|
||||||
|
|
|
||||||
|
|
@ -28,13 +28,14 @@ A revived Path C (post-v2.2.xxx) would require: (1) re-architecting tool-list to
|
||||||
|
|
||||||
## Profile system (`--profile`, v4.1.0)
|
## Profile system (`--profile`, v4.1.0)
|
||||||
|
|
||||||
Three built-in model profiles plus operator-defined `<custom>.yaml`. Each profile pins `phase_models` for the six pipeline phases (`brief`, `research`, `plan`, `execute`, `review`, `continue`). Profile is recorded in plan.md frontmatter as `profile: <name>` and emitted to `${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl` for cost-attribution.
|
Four built-in model profiles plus operator-defined `<custom>.yaml`. Each profile pins `phase_models` for the six pipeline phases (`brief`, `research`, `plan`, `execute`, `review`, `continue`). Profile is recorded in plan.md frontmatter as `profile: <name>` and emitted to `${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl` for cost-attribution.
|
||||||
|
|
||||||
| Profile | Brief | Research | Plan | Execute | Review | Continue | Use case |
|
| Profile | Brief | Research | Plan | Execute | Review | Continue | Use case |
|
||||||
|---------|-------|----------|------|---------|--------|----------|----------|
|
|---------|-------|----------|------|---------|--------|----------|----------|
|
||||||
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | Lowest cost; high-confidence small-scope tasks (operator-opt-in via `--profile economy`) |
|
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | ⚠ **Experimental** (uncalibrated Jaccard floor) — lowest cost; high-confidence small-scope tasks (operator-opt-in via `--profile economy`) |
|
||||||
| `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (operator-opt-in via `--profile balanced`) |
|
| `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (operator-opt-in via `--profile balanced`) |
|
||||||
| `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase. Default since 2026-05-13 operator request; also the hardcoded resolver default returned by `resolveProfile()` in `lib/profiles/resolver.mjs` |
|
| `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase. Default since 2026-05-13 operator request; also the hardcoded resolver default returned by `resolveProfile()` in `lib/profiles/resolver.mjs` |
|
||||||
|
| `fable` | fable | fable | fable | fable | fable | fable | Max quality — Fable 5 (Mythos-class, above Opus) on every phase (operator-opt-in via `--profile fable`); reasoning effort inherits from the session — see `docs/profiles.md` §Model & effort axes |
|
||||||
|
|
||||||
### Lookup order
|
### Lookup order
|
||||||
|
|
||||||
|
|
@ -45,7 +46,7 @@ Three built-in model profiles plus operator-defined `<custom>.yaml`. Each profil
|
||||||
|
|
||||||
### Custom profiles
|
### Custom profiles
|
||||||
|
|
||||||
Create `voyage-profiles/<custom>.yaml` in the repo root (or `~/.claude/voyage-profiles/<custom>.yaml`) to define a **new** tier — the name must not be a built-in. The validator (`lib/validators/profile-validator.mjs`) enforces: every `phase_models[].phase` must be a known phase enum; every `phase_models[].model` must match `^(opus|sonnet)(\b|-).*` or one of the canonical short names. `findProfilePath` (`lib/profiles/resolver.mjs`) resolves **built-in first** (`lib/profiles/<name>.yaml` for `economy`/`balanced`/`premium`), then repo-root `voyage-profiles/`, then `~/.claude/voyage-profiles/`. A custom file named after a built-in therefore **cannot** shadow it (custom profiles must use new names); for the same custom name, repo-root takes precedence over home.
|
Create `voyage-profiles/<custom>.yaml` in the repo root (or `~/.claude/voyage-profiles/<custom>.yaml`) to define a **new** tier — the name must not be a built-in. The validator (`lib/validators/profile-validator.mjs`) enforces: every `phase_models[].phase` must be a known phase enum; every `phase_models[].model` must exactly match an entry in `BASE_ALLOWED_MODELS` (`['sonnet', 'opus', 'fable']`; `haiku` only with `VOYAGE_ALLOW_HAIKU=1`). `findProfilePath` (`lib/profiles/resolver.mjs`) resolves **built-in first** (`lib/profiles/<name>.yaml` for `economy`/`balanced`/`premium`/`fable`), then repo-root `voyage-profiles/`, then `~/.claude/voyage-profiles/`. A custom file named after a built-in therefore **cannot** shadow it (custom profiles must use new names); for the same custom name, repo-root takes precedence over home.
|
||||||
|
|
||||||
Drift between plan-frontmatter `profile:` and step-manifest `profile_used:` emits a `MANIFEST_PROFILE_DRIFT` warning from `plan-validator --strict` (Step 20). Plan remains valid; the warning surfaces accidental tier-mismatch.
|
Drift between plan-frontmatter `profile:` and step-manifest `profile_used:` emits a `MANIFEST_PROFILE_DRIFT` warning from `plan-validator --strict` (Step 20). Plan remains valid; the warning surfaces accidental tier-mismatch.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -6,14 +6,15 @@ cost estimation (with disclaimer).
|
||||||
|
|
||||||
## Built-in profiles
|
## Built-in profiles
|
||||||
|
|
||||||
Three pre-defined tiers ship with v4.1, located at
|
Four pre-defined tiers ship with the plugin (fable added in v5.9), located at
|
||||||
`lib/profiles/{economy,balanced,premium}.yaml`.
|
`lib/profiles/{economy,balanced,premium,fable}.yaml`.
|
||||||
|
|
||||||
| Profile | Brief | Research | Plan | Execute | Review | Continue | Use case |
|
| Profile | Brief | Research | Plan | Execute | Review | Continue | Use case |
|
||||||
|---------|-------|----------|------|---------|--------|----------|----------|
|
|---------|-------|----------|------|---------|--------|----------|----------|
|
||||||
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | Lowest cost; small-scope tasks where you have high confidence the brief is right |
|
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | ⚠ **Experimental** (uncalibrated Jaccard floor) — lowest cost; small-scope tasks where you have high confidence the brief is right |
|
||||||
| `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (plan synthesis + adversarial review); opt-in via `--profile balanced` |
|
| `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (plan synthesis + adversarial review); opt-in via `--profile balanced` |
|
||||||
| `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase + external research on (default since the 2026-05-13 operator decision) |
|
| `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase + external research on (default since the 2026-05-13 operator decision) |
|
||||||
|
| `fable` | fable | fable | fable | fable | fable | fable | Max quality — Fable 5 (Mythos-class, above Opus) on every phase; opt-in via `--profile fable`; reasoning effort inherits from the session (see Model & effort axes) |
|
||||||
|
|
||||||
`premium` is the default tier — set by the 2026-05-13 operator decision and
|
`premium` is the default tier — set by the 2026-05-13 operator decision and
|
||||||
matched by the hardcoded resolver default in `lib/profiles/resolver.mjs`. It
|
matched by the hardcoded resolver default in `lib/profiles/resolver.mjs`. It
|
||||||
|
|
@ -22,20 +23,34 @@ roughly 5× the sub-agent cost of an all-sonnet run, accepted as a deliberate
|
||||||
trade-off. Drop to `--profile balanced` (opus only on the two phases where
|
trade-off. Drop to `--profile balanced` (opus only on the two phases where
|
||||||
quality matters most — Plan synthesis + Review — and sonnet everywhere else)
|
quality matters most — Plan synthesis + Review — and sonnet everywhere else)
|
||||||
or `--profile economy` (sonnet everywhere) when cost or latency matters more
|
or `--profile economy` (sonnet everywhere) when cost or latency matters more
|
||||||
than depth.
|
than depth. Step up to `--profile fable` (Fable 5 on every phase) when
|
||||||
|
maximum quality is wanted end-to-end and cost is not a constraint.
|
||||||
|
|
||||||
`economy` is *strictly experimental* in v4.1. The cross-tier Jaccard
|
`economy` is *strictly experimental* in v4.1, and says so in the profile
|
||||||
floor (0.55) is grounded in parked-synthetic fixtures, not empirical
|
data itself: `lib/profiles/economy.yaml` carries `experimental: true`. The
|
||||||
runs (Step 17 calibration was deferred — see
|
cross-tier Jaccard floor (0.55) is grounded in parked-synthetic fixtures, not
|
||||||
`tests/synthetic/profile-jaccard-calibration.md`). If you observe
|
empirical runs (Step 17 calibration was deferred — see
|
||||||
economy-plan quality regressions, fall back to `balanced`.
|
`tests/synthetic/profile-jaccard-calibration.md`). The flag is pinned: it must
|
||||||
|
stay `true` while the calibration status is `parked-synthetic`, and must be
|
||||||
|
dropped in the same change that lands empirical calibration
|
||||||
|
(`status: empirical`). If you observe economy-plan quality regressions, fall
|
||||||
|
back to `balanced`.
|
||||||
|
|
||||||
## Model & effort axes
|
## Model & effort axes
|
||||||
|
|
||||||
`opus` and `sonnet` are model **aliases**, not pinned ids. As of Claude Code
|
`opus`, `sonnet`, and `fable` are model **aliases**, not pinned ids. As of
|
||||||
2.1.154 the `opus` alias resolves to **Opus 4.8**, whose default reasoning
|
Claude Code 2.1.154 the `opus` alias resolves to **Opus 4.8**, whose default
|
||||||
effort is **`high`**; `sonnet` resolves to Sonnet 4.6. The profile table above
|
reasoning effort is **`high`**; `sonnet` resolves to Sonnet 4.6; `fable`
|
||||||
selects *which alias* runs each phase — it does not touch reasoning effort.
|
resolves to **Fable 5** (Mythos-class, positioned above Opus), whose default
|
||||||
|
reasoning effort is also `high`. The profile table above selects *which
|
||||||
|
alias* runs each phase — it does not touch reasoning effort.
|
||||||
|
|
||||||
|
**Reasoning effort inherits from the session.** Voyage effort (orchestration
|
||||||
|
shape — which agents/passes run) and model reasoning effort are different
|
||||||
|
axes. Fable 5's default reasoning effort is `high`, NOT xhigh, and switching
|
||||||
|
model resets effort to the model default — xhigh does not follow the model.
|
||||||
|
To run the fable tier at xhigh, set it at session level: `/effort xhigh`, the
|
||||||
|
`effortLevel` setting, or `CLAUDE_CODE_EFFORT_LEVEL`.
|
||||||
|
|
||||||
Two different things share the word "effort" in Voyage. They are **orthogonal
|
Two different things share the word "effort" in Voyage. They are **orthogonal
|
||||||
axes** — same name, different mechanism:
|
axes** — same name, different mechanism:
|
||||||
|
|
@ -50,8 +65,8 @@ axes** — same name, different mechanism:
|
||||||
|
|
||||||
The `phase-signal-resolver.mjs` helper only reads the **orchestration** axis
|
The `phase-signal-resolver.mjs` helper only reads the **orchestration** axis
|
||||||
(`phase_signals.effort`, gated against `low/standard/high`) plus the optional
|
(`phase_signals.effort`, gated against `low/standard/high`) plus the optional
|
||||||
per-phase `model` (gated against `['sonnet','opus']`). It never emits native
|
per-phase `model` (gated against `['sonnet','opus','fable']`). It never emits
|
||||||
`effort:`.
|
native `effort:`.
|
||||||
|
|
||||||
**Native `effort:` on agents.** Voyage sets the reasoning axis statically on
|
**Native `effort:` on agents.** Voyage sets the reasoning axis statically on
|
||||||
selected agents, additively over the Opus-4.8 default:
|
selected agents, additively over the Opus-4.8 default:
|
||||||
|
|
@ -116,11 +131,13 @@ The validator (`lib/validators/profile-validator.mjs`) enforces:
|
||||||
|
|
||||||
- Every `phase_models[].phase` must be a known phase enum:
|
- Every `phase_models[].phase` must be a known phase enum:
|
||||||
`brief` / `research` / `plan` / `execute` / `review` / `continue`
|
`brief` / `research` / `plan` / `execute` / `review` / `continue`
|
||||||
- Every `phase_models[].model` must match `^(opus|sonnet)(\b|-).*` or
|
- Every `phase_models[].model` must exactly match an entry in
|
||||||
one of the canonical short names
|
`BASE_ALLOWED_MODELS` (`['sonnet', 'opus', 'fable']` in
|
||||||
|
`lib/validators/profile-validator.mjs`; `haiku` only with
|
||||||
|
`VOYAGE_ALLOW_HAIKU=1`)
|
||||||
- All six phases must be present (no partial profiles)
|
- All six phases must be present (no partial profiles)
|
||||||
|
|
||||||
The three built-in names (`economy`, `balanced`, `premium`) resolve to their
|
The four built-in names (`economy`, `balanced`, `premium`, `fable`) resolve to their
|
||||||
bundled yaml first — `findProfilePath()` returns the built-in before consulting
|
bundled yaml first — `findProfilePath()` returns the built-in before consulting
|
||||||
`voyage-profiles/`, so a same-named custom file is ignored and cannot shadow a
|
`voyage-profiles/`, so a same-named custom file is ignored and cannot shadow a
|
||||||
built-in. To customize, give your profile a new name and reference it via
|
built-in. To customize, give your profile a new name and reference it via
|
||||||
|
|
|
||||||
237
docs/voyage-vs-cc-balance-analysis.md
Normal file
237
docs/voyage-vs-cc-balance-analysis.md
Normal file
|
|
@ -0,0 +1,237 @@
|
||||||
|
# Voyage-vs-CC Balance Analysis
|
||||||
|
|
||||||
|
**Status:** Analysis — recommendations only, operator-gated before any implementation.
|
||||||
|
**Date:** 2026-06-20. **Baseline:** Claude Code 2.1.183 (latest; 2.1.182 never shipped).
|
||||||
|
**Charter:** `docs/voyage-vs-cc-balance-charter.md`. **Method substrate:** Dynamic Workflow (`wf_41bb3936-e6d`, 96 agents, ~2.97M subagent tokens, 4.6 min) + inline scout + an over-keeping meta-critic.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
- **Voyage's existence is justified by ONE thing the audit confirms end-to-end:** typed, versioned, cross-stage/cross-session **structured-artifact handovers** (brief → research → plan → execute → review → continue) plus **multi-session discipline**. CC 2.1.183 has no native analog for a semver-stable public artifact contract or a typed work-state checkpoint. This matches the plugin's own CLAUDE.md self-admission and the operator's primary real use (large tasks across many sessions).
|
||||||
|
- **It is NOT justified by the swarm-relieves-context claim** (measured Δ ≈ 0, unchanged here) and **not by re-implementing CC engines.** Wherever CC ships the engine, Voyage's only defensible role is a *thin policy layer that lowers the expertise bar* — never a re-implementation.
|
||||||
|
- **Disposition tally (35 capabilities, post-adversarial):** KEEP 25 · THIN_WRAP 6 · SIMPLIFY 4 · DROP→NATIVE 0.
|
||||||
|
- **The zero-DROP result is over-keeping at the edges.** A meta-critic flags **4 genuine downgrade candidates** (V09 Gemini-bridge, V15 text-export variants, V32 observability export, V35 dormant synthesis-agent). Acting on all of them trims the edges; it does **not** move the headline.
|
||||||
|
- **Nothing recommended here breaks Handover 1** (the public brief contract). Every operator-pinned decision the analysis brushes is flagged, never silently overridden.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Method & evidence base (Phase 0–1)
|
||||||
|
|
||||||
|
### 1.1 Post-2.1.181 CC delta (Verification #4)
|
||||||
|
|
||||||
|
Latest CC = **2.1.183** (June 19 2026). **2.1.182 never existed** (skipped build). 2.1.183 = bugfixes + auto-mode git safety guards only. **No new features touch the Workflow tool, plan mode, Artifacts, subagent nesting, or session checkpointing.** Several 2.1.183 fixes actually *repair* Voyage swarm failure modes (subagent-spawn 400s, empty WebSearch-in-subagent, silent thinking-only completions, tmux teammate launch, background-task premature termination) — no Voyage code change needed. **Conclusion: the CC-native overlap baseline is unchanged from the 2.1.181 decision matrix.**
|
||||||
|
|
||||||
|
### 1.2 Anchor verdicts carried in (not re-derived)
|
||||||
|
|
||||||
|
| Prior finding | Verdict | Source |
|
||||||
|
|---|---|---|
|
||||||
|
| Sub-agents can nest ≤5 deep | true since CC 2.1.172 (old "harness hides Agent tool" premise is FALSE) | `cc-upgrade-2.1.181-decision-matrix.md` CC-01 |
|
||||||
|
| Delegate the orchestration loop to a sub-agent (CC-26) | **lean NO** — only upside is main-context relief, measured Δ=0; new proliferation-classifier risk | `T1-cc26-delegated-orchestration.md` |
|
||||||
|
| Workflow tool as substrate (CC-27) | **selective hybrid** — schema-contract win real; wholesale swap DECLINED | `T2-cc27-workflow-substrate.md` |
|
||||||
|
| Synthesis delegated to a sub-agent (T1 PoC) | **Δ main-context = 0.0%** (exploration already runs foreground; digest returns on top) → dormant | `T1-synthesis-poc-results.md` |
|
||||||
|
| Flagship "swarm relieves context" | **unproven**; structured-artifact handovers are the load-bearing benefit | root `CLAUDE.md` (verbatim self-admission) |
|
||||||
|
|
||||||
|
### 1.3 The two decision axes
|
||||||
|
|
||||||
|
- **Axis A — Duplication:** does CC 2.1.183 do this natively? does it do it *better*?
|
||||||
|
- **Axis B — Expertise-bar:** does Voyage's wrapper let a non-CC-expert get the value *without* knowing CC internals?
|
||||||
|
|
||||||
|
**Disposition vocabulary:** KEEP (Voyage-unique/clearly better) · THIN_WRAP (CC is the engine; keep a thin accessibility layer, delegate the engine) · DROP→NATIVE (CC better + no real expertise gap; remove + document native path) · SIMPLIFY (keep, shed complexity CC now handles).
|
||||||
|
|
||||||
|
**The tension the analysis was forced to resolve per capability:** a feature CC does "better" raw can still deserve THIN_WRAP if raw CC requires expertise — *accessibility can justify a wrapper, never a re-implementation.*
|
||||||
|
|
||||||
|
### 1.4 How the classification ran
|
||||||
|
|
||||||
|
35 canonical capabilities (consolidated from 42 command-level + ~30 infra inventory rows). Each ran a 3-stage pipeline: **(A)** CC-overlap map (sonnet) → **(B)** A×B classification → disposition (opus) → **(C)** adversarial verify of every KEEP/DROP→NATIVE (opus, high effort). Stage-B/C agents read the actual Voyage code (validators, hooks, command prose) and cite `file:line`. A final **over-keeping meta-critic** then attacked the zero-DROP outcome holistically (§4).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. The balance thesis
|
||||||
|
|
||||||
|
Voyage sits on a host (CC) that has absorbed most of what a 2024-era "agent orchestration" plugin differentiated on: parallel sub-agent spawn, nesting, native `effort:`, plan mode, MCP, hooks, the Workflow tool. The audit's structural finding is that **CC absorbed the *primitives*, not the *contracts*.** Across all 35 capabilities, every CC overlap is `cc_partial` or `cc_absent` — CC ships the spawn/ask/hook/MCP engine, but never:
|
||||||
|
|
||||||
|
- a **versioned, semver-stable, PUBLIC artifact schema** an unrelated upstream producer can target (Handover 1),
|
||||||
|
- a **typed cross-session work-state checkpoint** with auto-discovery and zero-confirm resume (Handover 7),
|
||||||
|
- a **standing per-handover validator library** that gates the pipeline on written-artifact frontmatter,
|
||||||
|
- or the **opinionated domain policy** (research angles, review taxonomy, manifest predicates) a user would otherwise hand-author every run.
|
||||||
|
|
||||||
|
**So the balance is:**
|
||||||
|
|
||||||
|
- **Where CC has no engine (most KEEPs)** — brief contract, plan-schema enforcement, manifest audit, progress/resume, session-state, triangulation, the handover contract itself — Voyage is genuinely additive. KEEP.
|
||||||
|
- **Where CC has the engine (the THIN_WRAPs)** — research/exploration/reviewer swarms, the research interview, the Gemini bridge, observability — Voyage's job is to **ride the native primitive and add only the thin policy**, never to re-host the engine. THIN_WRAP, with a standing obligation to delegate the engine to CC.
|
||||||
|
- **Where the wrapper's own contract layer is thin or empty (the over-keeping edge)** — a "do-not-editorialize" pass-through agent, text reformatting, a re-hosted telemetry collector, a dormant measured-dead agent — the accessibility defense is rhetorical, and these are the DROP/downgrade candidates.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Phase 3 disposition matrix (Verification #1)
|
||||||
|
|
||||||
|
Each capability appears exactly once. Inventory count (35) = matrix rows (35). Axis A: `nb`=cc_native_better, `eq`=cc_native_equal, `pt`=cc_partial, `ab`=cc_absent. Axis B gap: H/M/L/none. **Adv** = per-capability adversarial verdict (— = not in KEEP/DROP scope). **B** = acting on it would change a handover contract. **P** = touches an operator-pinned decision.
|
||||||
|
|
||||||
|
| ID | Capability | A | B | Disposition | Adv | B | P | Why (one line) + evidence-ref |
|
||||||
|
|----|-----------|---|---|-------------|-----|---|---|-------------------------------|
|
||||||
|
| V01 | Interactive brief interview | pt | M | **KEEP** | upheld | · | P | Engine = AskUserQuestion; value = the completeness loop that emits Handover-1 `brief.md`. `brief-validator.mjs` |
|
||||||
|
| V02 | Framing-intent gate | ab | H | **KEEP** | upheld | · | P | No native typed-enum non-skippable input gate; `BRIEF_*FRAMING` codes. `brief-validator.mjs:110-154` |
|
||||||
|
| V03 | Per-phase effort signals | ab | H | **KEEP** | upheld | · | P | Native `effort:` is per-spawn reasoning, a *different* axis; no pipeline-wide phase signal. `HANDOVER-CONTRACTS.md §1` |
|
||||||
|
| V04 | Brief quality review + revise loop | ab | H | **KEEP** | upheld | · | P | No native draft→review→revise rubric loop; spawn ≠ engine. `brief-reviewer.md` |
|
||||||
|
| V05 | Memory-alignment defense | ab | H | **KEEP** | upheld | **B** | P | MEMORY.md is passive context; no native brief-vs-memory contradiction gate. `brief-reviewer.md:160-186` |
|
||||||
|
| V06 | Brief→pipeline auto-orchestration | ab | M | **KEEP** | upheld | · | · | Skill tool is one-shot; Workflow can't span slash-command/handover boundaries. `T2` |
|
||||||
|
| V07 | Research interview | pt | M | **THIN_WRAP** | — | · | · | AskUserQuestion is the engine; keep only the 4-dim/2-4-Q interview policy. `trekresearch.md §2` |
|
||||||
|
| V08 | External research swarm | pt | H | **THIN_WRAP** | — | · | P | CC ships parallel-spawn+WebSearch+MCP; keep the 4-angle decomposition + schemas. `docs/community/security/contrarian` |
|
||||||
|
| V09 | Gemini second opinion | eq | M | **THIN_WRAP** ⚠ | — | · | P | Engine = third-party `gemini-mcp`; agent is pure glue ("do not editorialize"). **DROP candidate (§4).** |
|
||||||
|
| V10 | Triangulation + confidence | ab | H | **KEEP** | upheld | · | · | No native per-dimension confidence enum + weighted scalar. `trekresearch.md §6` |
|
||||||
|
| V11 | Local exploration swarm | pt | H | **THIN_WRAP** | — | · | P | CC ships parallel Agent spawn; keep 7 typed roles + effort defaults + scaling. `agents/*` |
|
||||||
|
| V12 | Plan synthesis + schema enforcement | ab | H | **KEEP** | upheld | · | · | No native plan schema/`--strict` validator; load-bearing handover. `plan-validator.mjs` |
|
||||||
|
| V13 | Adversarial plan review | pt | H | **KEEP** | upheld | · | P | Spawn ≠ a 10-dim critic + scope-guardian + dedup workflow. `plan-critic.md` |
|
||||||
|
| V14 | Architecture-note auto-discovery | ab | M | **KEEP** | upheld | · | · | No native canonical-path + fallback + drift-WARN discovery. `architecture-discovery.mjs` |
|
||||||
|
| V15 | Plan export | pt | L | **SIMPLIFY** ⚠ | — | · | · | pr/issue/markdown = text reflow CC does ad-hoc; only `headless`(=decompose) has value. **DROP text variants (§4).** |
|
||||||
|
| V16 | Session decomposition | ab | H | **KEEP** | upheld | · | P | `--resume`/TodoWrite carry no dependency parse/wave/spec/launch.sh. `session-decomposer.md` |
|
||||||
|
| V17 | Disciplined step-execution loop | pt | H | **KEEP** | upheld | · | · | Manifest predicate (git-diff completion gate) has no CC-native analog. `trekexecute.md §6` |
|
||||||
|
| V18 | Pre-exec + runtime safety guardrails | pt | H | **KEEP** | upheld | · | · | Native `permissions.deny` is a glob matcher; denylist is semantic regex. `pre-bash-executor.mjs` |
|
||||||
|
| V19 | Multi-session parallel orchestration | ab | H | **KEEP** | upheld | · | · | `git worktree`+`claude -p` are leaves; wave/merge-gate/lock-race orchestration is Voyage. `trekexecute.md §2.6` |
|
||||||
|
| V20 | Manifest audit + recovery dispatch | ab | H | **KEEP** | upheld | · | · | No native post-exec audit / self-report-distrust / depth-capped recovery. `trekexecute.md §7.5-7.6` |
|
||||||
|
| V21 | Progress/resume contract | ab | H | **KEEP** | upheld | · | · | CC resume = conversation-level; no step status/attempts/SHA/drift. `progress` + `pre-compact-flush.mjs` |
|
||||||
|
| V22 | Dry-run / validate modes | ab | M | **KEEP** | upheld | · | · | No native read-only preview of a multi-session execution strategy. `trekexecute.md §5/§2.3` |
|
||||||
|
| V23 | Deterministic review triage gate | ab | M | **KEEP** | upheld | · | · | Native `/code-review` has no path classifier / refuse-gate / Coverage. `trekreview.md §4` |
|
||||||
|
| V24 | Independent reviewer swarm | pt | H | **THIN_WRAP** | — | · | · | CC ships fan-out; keep rule-catalogue + schema + no-cross-feed + taxonomy. `lib/review/*` |
|
||||||
|
| V25 | Review coordinator / Judge | ab | H | **KEEP** | upheld | · | · | No native multi-reviewer dedup + filter + verdict layer. `review-coordinator.md` |
|
||||||
|
| V26 | Review→remediation handover | ab | H | **KEEP** | upheld | · | · | No native typed cross-stage handover (severity→goal, source_findings). `review-validator.mjs` |
|
||||||
|
| V27 | Review Workflow substrate option | pt | H | **SIMPLIFY** | — | · | · | Opt-in port; +4.4% tokens/+54% wall-time, CC≥2.1.154 floor. Keep as opt-in, don't expand. `T2` |
|
||||||
|
| V28 | Session-state + zero-friction resume | ab | H | **KEEP** | upheld | · | · | `--resume` replays a transcript, not a typed work-state. `session-state-validator.mjs` |
|
||||||
|
| V29 | Structured-artifact handover contract | ab | H | **KEEP** | upheld | · | · | **The load-bearing value.** Workflow types I/O within one session, not a public cross-stage contract. `HANDOVER-CONTRACTS.md` |
|
||||||
|
| V30 | Profile system + native effort axis | pt | M | **KEEP** | upheld | · | P | Native model key is single-spawn; profiles span the whole multi-session run. `resolver.mjs` |
|
||||||
|
| V31 | Autonomy gates | ab | H | **KEEP** | upheld | · | P | Plan mode gates one boundary; no mid-exec per-phase/wave pause policy. `autonomy-gate.mjs` |
|
||||||
|
| V32 | Observability export | pt | H | **THIN_WRAP** ⚠ | — | · | · | CC ships Stop-hook+OTEL passthrough; exporters/guards re-host collector logic. **DROP candidate (§4).** |
|
||||||
|
| V33 | Operator annotation HTML | ab | M | **SIMPLIFY** | revised (KEEP→SIMPLIFY) | · | · | No native line-anchored note UI, but it's a generic md tool, not a pipeline capability. `scripts/annotate.mjs` |
|
||||||
|
| V34 | Artifact schema validators | ab | M | **KEEP** | upheld | · | · | Workflow schemas type one tool boundary; these validate persistent artifacts. `lib/validators/*` |
|
||||||
|
| V35 | Internal architecture artifacts | ab | none | **SIMPLIFY** ⚠ | — | · | P | Dormant synthesis-agent (Δ=0) + non-spawned orchestrator docs. **DROP dormant half (§4).** |
|
||||||
|
|
||||||
|
⚠ = flagged by the over-keeping meta-critic as a downgrade candidate beyond its workflow disposition (§4).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. The zero-DROP finding & over-keeping meta-review (Verification #5)
|
||||||
|
|
||||||
|
**The honest gap:** the per-capability adversarial pass challenged all 26 KEEP classifications (25 upheld, 1 revised: V33 KEEP→SIMPLIFY) — but because the classification produced **zero DROP→NATIVE**, the charter's "challenge ≥1 DROP→NATIVE" criterion had nothing to operate on. A self-audit concluding "drop nothing" is the exact pattern a skeptic should distrust. So a dedicated **over-keeping meta-critic** attacked the outcome and the recurring "primitive vs typed-contract" defense.
|
||||||
|
|
||||||
|
**Meta-critic verdict:** the zero-DROP outcome is **over-keeping at the edges**. The "primitive vs typed-contract" distinction is *load-bearing* for V08/V11/V24 (real policy a user must re-author every run) but *rhetorical* where the contract layer is itself thin or empty. Four downgrade candidates:
|
||||||
|
|
||||||
|
| Cap | Workflow said | Meta-critic says | Native path / what's lost | Contested? |
|
||||||
|
|-----|---------------|------------------|---------------------------|-----------|
|
||||||
|
| **V09** Gemini bridge | THIN_WRAP | **DROP→NATIVE** | Call `gemini-mcp` tools inline; agent explicitly *doesn't* reason → opus-on-glue is pure waste. The classify pass *also* flagged opus wasted here. | Low — both passes agree |
|
||||||
|
| **V15** plan export (pr/issue/markdown) | SIMPLIFY | **DROP the 3 text variants** | Ask auto-mode to reformat ad-hoc; only `--export headless` (=decomposition) survives, and that's V16 wearing a flag. | Low |
|
||||||
|
| **V32** observability export | THIN_WRAP | **DROP→NATIVE candidate** | Point OTEL env vars at a standard OTLP collector (which owns allowlists/egress); exporters re-host the collector. | **High** — code-verified pass says SSRF/path/field guards are net-new; this is a genuine direct-export-vs-collector architecture choice. Operator decision. |
|
||||||
|
| **V35** dormant synthesis-agent | SIMPLIFY | **DROP the dormant agent** | It's already dead (Δ=0, wired to nothing); orchestrator docs are docs, not capabilities. | Low |
|
||||||
|
|
||||||
|
**Caveat for the operator:** the meta-critic reasoned from the disposition table, not from re-reading the code, whereas the per-capability passes cite `file:line`. Treat V32 as genuinely contested. V09/V15/V35 are low-contest because both layers of analysis converge.
|
||||||
|
|
||||||
|
**Bottom line (meta-critic, verbatim sense):** dropping V09 + V15-text + (probably) V32 and demoting V35's dead agent *trims the edges decisively but does not move the headline* — the typed-handover + multi-session core is untouched and remains the only thing that justifies the plugin's existence.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Target-form for Voyage
|
||||||
|
|
||||||
|
What Voyage should *be*, given modern CC:
|
||||||
|
|
||||||
|
1. **A contract layer, not an orchestration engine.** Lead with the 7 typed handovers + multi-session discipline (V28/V29 + the validators V34). Stop marketing swarm-context-relief; it's measured Δ≈0.
|
||||||
|
2. **A thin accessibility skin over native CC engines** for everything CC now does well: research/exploration/reviewer swarms should *visibly delegate* to native parallel Agent spawn + AskUserQuestion + Stop hooks, adding only the opinionated policy (angles, roles, taxonomy, schemas). Never re-host an engine.
|
||||||
|
3. **Opinionated defaults that encode best practice** so a non-CC-expert gets disciplined planning/execution/review without knowing the harness — this is the durable Axis-B justification, confirmed `high_gap` on the execution and review cores.
|
||||||
|
4. **Lean at the edges:** no pass-through agents paying opus to forward bytes (V09), no hand-written reformatters for what auto-mode does ad-hoc (V15-text), no re-hosted telemetry collector unless the direct-export design is a deliberate, defended choice (V32), no dormant agents padding the capability count (V35).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Prioritized change backlog (Verification #6)
|
||||||
|
|
||||||
|
All items are **operator-gated** (scope-guard: analysis only). Tagged **[non-breaking]** / **[breaking]** and **[pinned]** where they touch an operator-pinned decision.
|
||||||
|
|
||||||
|
### Tier 1 — over-keeping trims (highest value, mostly non-breaking)
|
||||||
|
|
||||||
|
1. **[non-breaking][pinned]** **V09** — drop `gemini-bridge` as an opus agent; call `gemini-mcp` tools inline (or, if kept, downgrade opus→sonnet, since it explicitly does not reason). Touches the 24-opus pin → flag.
|
||||||
|
2. **[non-breaking]** **V15** — drop the `pr`/`issue`/`markdown` export variants (auto-mode does the reflow ad-hoc); keep `--export headless` and relabel it as the decomposition entry it actually is.
|
||||||
|
3. **[non-breaking, contested]** **V32** — decide direct-export-vs-collector explicitly. If a standard OTLP collector is acceptable, drop the custom exporters + reimplemented guards to native OTEL env-var passthrough; if direct export is a deliberate requirement, keep and document *why* (this is the one genuinely contested call).
|
||||||
|
4. **[non-breaking][pinned]** **V35** — strip the dormant `synthesis-agent`'s `model:opus`/`effort:high` frontmatter (or collapse it to a docs note); reclassify the 3 orchestrator reference docs as docs, not capabilities. Touches the 24-opus pin → flag.
|
||||||
|
|
||||||
|
### Tier 2 — THIN_WRAP hygiene (ensure delegate-the-engine, non-breaking)
|
||||||
|
|
||||||
|
5. **[non-breaking]** Audit V07/V08/V11/V24 implementations to confirm they ride native parallel-spawn / AskUserQuestion rather than hand-rolling, keeping only the policy layer. (Mostly already true — this is a standing guard against engine re-implementation, not a known defect.)
|
||||||
|
6. **[non-breaking][pinned]** Reconsider opus-for-all on mechanical/retrieval roles: V11 retrieval agents (already `effort:medium`), V16 `session-decomposer` (dependency parsing is mechanical), V08 researchers. Each touches the 24-opus pin → flag, don't override.
|
||||||
|
|
||||||
|
### Tier 3 — surfaced risks (no action mandated)
|
||||||
|
|
||||||
|
7. **[non-breaking][pinned]** **V30** — the `economy` profile's Jaccard floor (0.55) is grounded in parked synthetic fixtures (Step 17 calibration deferred). It is self-declared experimental; either calibrate or label clearly. → **RESOLVED (S34): label, not calibrate.** Empirical calibration is v4.2-budget-gated ($60–120, unauthorized); instead `lib/profiles/economy.yaml` now carries `experimental: true` (validator type-checks it) and every profile doc flags the `economy` row. Pinned in `profile-validator.test.mjs` + `doc-consistency.test.mjs` — the flag must track the calibration's `parked-synthetic` status.
|
||||||
|
8. **[non-breaking]** **V01** — minor: delegate the literal Q&A turn-taking to AskUserQuestion rather than a hand-rolled selection loop (internal hygiene; does not change the framing-gate or reviewer contract).
|
||||||
|
|
||||||
|
### Explicitly NOT recommended
|
||||||
|
|
||||||
|
- **No change to Handover 1** (brief schema) — the only public contract; any change is breaking. V02/V03/V05 stay.
|
||||||
|
- **No wholesale Workflow substrate swap** (CC-27 DECLINED stands); keep V27 as opt-in only.
|
||||||
|
- **No delegated-orchestration redesign** (CC-26 lean-NO stands; Δ=0).
|
||||||
|
- **No reopening** of the premium-default-profile or framing-gate operator pins — flagged where touched, never overridden.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Verification checklist (charter §Verifisering)
|
||||||
|
|
||||||
|
| # | Criterion | Status |
|
||||||
|
|---|-----------|--------|
|
||||||
|
| 1 | Each capability appears exactly once with disposition + rationale + evidence-ref; inventory count == matrix rows | ✅ 35 == 35 (§3) |
|
||||||
|
| 2 | Each "CC duplicate / does better" cites a specific CC feature + version | ✅ `cc_citation` per row (e.g. AskUserQuestion 2.0+, parallel Agent 2.1.154+, nesting 2.1.172, Workflow 2.1.154+, `effort:` 2.1.154, PreToolUse ~2.1.97, PreCompact 2.1.105) |
|
||||||
|
| 3 | Each "lowers expertise bar" names the concrete native steps a non-expert would otherwise do | ✅ captured in `expertise_steps_if_native` / `concrete_native_steps_saved` per capability |
|
||||||
|
| 4 | Post-2.1.181 delta actually run; latest version stated + each new item triaged | ✅ §1.1 — CC 2.1.183, 17 items triaged, none balance-relevant |
|
||||||
|
| 5 | Adversarial pass challenged ≥1 KEEP and ≥1 DROP→NATIVE, with verdict | ⚠️ 26 KEEPs challenged (1 revised); **zero DROP→NATIVE were produced**, so that half was addressed by the over-keeping meta-critic (§4) which named 4 DROP/downgrade candidates with verdicts. Honest deviation from the literal criterion, documented. |
|
||||||
|
| 6 | Output ends with a prioritized backlog, each item tagged breaking/non-breaking + operator-gated | ✅ §6 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Hard constraints honored
|
||||||
|
|
||||||
|
- **Trinity asymmetry:** no recommendation changes the brief schema (Handover 1). Voyage stays unaware of Tier 2/3. The one capability marked `breaking` (V05, memory-alignment, bound to `brief_version 2.2`) is recommended **KEEP-as-is** — no breaking change proposed.
|
||||||
|
- **Operator-pinned decisions:** premium default profile, 24-opus agents, framing-gate, plan-critic=10-dim, brief-reviewer=6-dim — all preserved. Where a backlog item brushes a pin (V09/V35 model frontmatter; opus-on-mechanical-roles; economy calibration), it is **flagged `[pinned]` and operator-gated**, never overridden.
|
||||||
|
- **Analysis only:** every item in §6 requires an explicit operator gate before implementation.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Meta-note (dogfooding the Workflow tool)
|
||||||
|
|
||||||
|
Running this analysis *on* the CC Workflow tool produced first-hand evidence for the audit's own V27/V29 rows: the `pipeline()` shape gave clean coverage (35/35, no row dropped), schema-forced output eliminated JSON-parse fragility (the CC-27 F2 win, reconfirmed), and the conditional adversarial stage worked. Costs also reconfirmed: ~2.97M subagent tokens / 96 agents / 4.6 min for a 35×3 fan-out — Workflow is a good substrate for *bounded, verifiable, parallel* fan-out, exactly the ~20% core CC-27 identified, and a poor fit for the ~80% judgment-heavy glue (scout + synthesis, which ran inline in main context here). This is dogfooding of the **Workflow tool**, not of Voyage's pipeline (the latter was declined by the operator).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Decision record — resolved forks (operator, 2026-06-20)
|
||||||
|
|
||||||
|
§4/§6 deferred three forks to the operator. All three resolved to the
|
||||||
|
**conservative** option (preserve the capability / the security code / the
|
||||||
|
deliberate pin). The model and observability work therefore collapses from
|
||||||
|
code-deletion to a documentation record; the only real code changes in the
|
||||||
|
backlog are V15 (export trim, S31) and V30 (economy calibration, S34). These
|
||||||
|
decisions are now the implemented baseline of the multi-session backlog plan
|
||||||
|
(`docs/balance-backlog-plan.md`), shipped across S31–S34.
|
||||||
|
|
||||||
|
| Fork | Item | Resolution | What changed |
|
||||||
|
|------|------|------------|--------------|
|
||||||
|
| **D1** | V09 `gemini-bridge` | **Keep it as an agent.** | No removal / inline-rewiring. The capability count stays 24; its model is governed by D3 (stays opus). |
|
||||||
|
| **D2** | V32 observability export | **Keep `lib/exporters/*` + `otel-export.mjs`; document the direct-export rationale.** | Doc-only — no deletion of S21 security code. The rationale (preserve the in-process path / SSRF / field-allowlist guards rather than re-host a collector) is recorded in `docs/observability.md` §"Why direct export rather than a native collector". |
|
||||||
|
| **D3** | 24-agent `model: opus` pin (`40d8742`) | **Keep the pin firm.** | Document-only. opus on V09 (glue — does not reason), V35 (dormant `synthesis-agent`), V11 (retrieval agents, already `effort:medium`), V16 (`session-decomposer`, mechanical parsing), and V08 (researchers) was reconsidered for a sonnet downgrade and **kept opus**. **No agent frontmatter is changed** — `tests/lib/agent-frontmatter.test.mjs` remains the structural source-of-truth and is untouched. |
|
||||||
|
|
||||||
|
**Inventory framing (V35 doc half, reconciled).** The "24 agents" headline is
|
||||||
|
**21 spawnable + 3 orchestrator reference docs**. The three orchestrators
|
||||||
|
(`planning-/research-/review-orchestrator`) document the inline `/trek*`
|
||||||
|
workflow and are not spawned as sub-agents. Of the 21 spawnable, one —
|
||||||
|
`synthesis-agent` — ships **dormant** (Δ≈0, wired to nothing;
|
||||||
|
`docs/T1-synthesis-poc-results.md`). Reconciled across `README.md`,
|
||||||
|
`CLAUDE.md`, and this doc; pinned in `tests/lib/doc-consistency.test.mjs` (S33).
|
||||||
|
|
||||||
|
**V30 outcome (S34, the lone Tier-3 code item).** The economy-calibration fork
|
||||||
|
also resolved conservatively: **label, do not calibrate.** The empirical run is
|
||||||
|
v4.2-budget-gated ($60–120, unauthorized) and was never in scope here. Instead
|
||||||
|
`economy`'s experimental status — previously prose-only in `docs/profiles.md` —
|
||||||
|
now lives in the profile **data** (`lib/profiles/economy.yaml experimental: true`,
|
||||||
|
type-checked by the profile-validator) and on every profile-doc `economy` row,
|
||||||
|
pinned so the flag tracks the calibration's `parked-synthetic` status and is
|
||||||
|
dropped in the same change that lands real calibration. This closes the backlog:
|
||||||
|
all 8 §6 items disposed across S31–S34 (2 code: V15+V30; the rest doc/audit).
|
||||||
|
|
||||||
|
**Still explicitly NOT done** (out of scope, per the plan): no `gemini-bridge`
|
||||||
|
removal (D1), no exporter deletion (D2), no model downgrade (D3), no Handover-1
|
||||||
|
change, no Workflow substrate swap (CC-27 declined), no delegated-orchestration
|
||||||
|
redesign (CC-26 lean-NO), no reopening of the premium-default / framing-gate pins.
|
||||||
71
docs/voyage-vs-cc-balance-charter.md
Normal file
71
docs/voyage-vs-cc-balance-charter.md
Normal file
|
|
@ -0,0 +1,71 @@
|
||||||
|
# Voyage-vs-CC Balance Analysis — Charter
|
||||||
|
|
||||||
|
**Status:** Launch-spec. **MANDATERT start på neste sesjon** (operatør 2026-06-20). Dette er ANALYSE-scope — anbefalinger only, operatør-gated FØR enhver implementasjon.
|
||||||
|
|
||||||
|
## Spørsmålet
|
||||||
|
|
||||||
|
Moderne Claude Code (2.1.183) gjør nativt mye av det Voyage en gang differensierte på (sub-agent-nesting, Workflow-tool, native `effort:`, plan mode, resume/state, Artifacts). Finn **balansen** der:
|
||||||
|
|
||||||
|
- **(Tilgjengelighet)** en bruker som IKKE er CC-ekspert fortsatt får Voyages verdi — Voyage koder beste praksis så du slipper å kunne harnessen.
|
||||||
|
- **(Ikke-duplisering)** Voyage IKKE reimplementerer CC-features som CC gjør bedre nativt.
|
||||||
|
|
||||||
|
**Anker-funn fra S29 (ikke re-deriver):** CC-26/CC-27 konkluderte at orkestrerings-substratet i økende grad er nativt; CLAUDE.md innrømmer selv at «swarm relieves context»-påstanden er umålt (Δ ≈ 0), og at **strukturerte artefakt-handovers + multi-sesjon-disiplin er den bærende verdien**. Operatørens egen primærbruk: store oppgaver over flere sesjoner.
|
||||||
|
|
||||||
|
## De to beslutnings-aksene
|
||||||
|
|
||||||
|
For hver Voyage-capability:
|
||||||
|
|
||||||
|
- **Akse A — Duplikasjon:** Gjør dagens CC dette nativt? Gjør CC det *bedre*?
|
||||||
|
- **Akse B — Ekspertise-bar:** Lar Voyages innpakning en ikke-CC-ekspert få verdien uten å kunne CC-internals?
|
||||||
|
|
||||||
|
**Disposition-vokabular (per capability):**
|
||||||
|
|
||||||
|
| Disposition | Når |
|
||||||
|
|-------------|-----|
|
||||||
|
| **KEEP** | Voyage-unik verdi (eller klart bedre), ikke duplikat. |
|
||||||
|
| **THIN-WRAP** | CC gjør jobben; Voyages verdi er å senke ekspertise-baren / opinionated defaults. Behold et tynt lag, deleger motoren til CC. |
|
||||||
|
| **DROP→NATIVE** | CC gjør det bedre OG ingen reell ekspertise-gap. Fjern fra Voyage; dokumentér den native veien. |
|
||||||
|
| **SIMPLIFY** | Behold, men kast kompleksitet CC nå håndterer. |
|
||||||
|
|
||||||
|
**Spenningen som MÅ løses eksplisitt:** en feature CC gjør «bedre» rått kan likevel fortjene THIN-WRAP hvis rå CC krever ekspertise. Tilgjengelighet kan rettferdiggjøre en wrapper; den kan ikke rettferdiggjøre en re-implementasjon.
|
||||||
|
|
||||||
|
## Metode (faset — «alt av metoder» autorisert: agent-swarm / Workflow / web-research)
|
||||||
|
|
||||||
|
- **Phase 0 — Evidensbase (ikke re-deriver).** Les: CC-NN-matrisen (`docs/cc-upgrade-2.1.181-decision-matrix.md`), `docs/subagent-delegation-audit.md`, `docs/T1-*`/`docs/T2-*`-resultater, CLAUDE.md design-prinsipp-innrømmelsen, CC-26/27-dispositionene. **Kjør den utsatte post-2.1.181-delta-sjekken** (2.1.182→siste, via `claude-code-guide`/WebSearch) — matrisen stopper på 2.1.181.
|
||||||
|
- **Phase 1 — Voyage capability-inventar.** Enumerér hver distinkte capability: 7 commands, 24 agenter, de 7 handovers, session-dekomponering, state/resume (`/trekcontinue`/`/trekendsession`), review-gates (plan-critic/scope-guardian/brief-reviewer/reviewere), framing-gate, rendering+annoterings-HTML, profiler, observability, autonomy-gates (`--gates`), headless/`-p`, lib-validatorer. Én rad per capability. (Fan ut lesere over commands/agents/lib/docs.)
|
||||||
|
- **Phase 2 — CC-overlap-mapping.** Per capability: dagens CC-native ekvivalent (feature + versjon) + dom «gjør CC det bedre».
|
||||||
|
- **Phase 3 — Dobbel-akse-klassifisering + adversarisk pass.** Klassifisér hver på A×B → disposition. Et contrarian/critic-pass utfordrer HVER KEEP («virkelig ikke duplikat?») og HVER DROP→NATIVE («virkelig ingen ekspertise-gap — ville en ikke-ekspert mistet verdien?»). Registrér utfordringene + verdiktene.
|
||||||
|
- **Phase 4 — Syntese.** Mål-form for Voyage + en prioritert endrings-backlog (handling per capability), som **skiller ikke-breaking fra breaking** (Trinity Handover-1-kontrakten + operatør-pinnede beslutninger er harde constraints — flagg, ikke bryt i stillhet). Operatør-gated.
|
||||||
|
|
||||||
|
**Output-artefakt:** `docs/voyage-vs-cc-balance-analysis.md`.
|
||||||
|
|
||||||
|
## Eksekvering — Dynamic Workflow (operatør-autorisert 2026-06-20)
|
||||||
|
|
||||||
|
Substrat: CC **Workflow-tool** (Dynamic Workflow) — IKKE ad-hoc Agent-kall, og IKKE dogfood via Voyages egen pipeline (`/trekbrief`→`/trekplan`; avvist av operatør). Begrunnelse: analysen er en capability-for-capability audit med adversarisk utfordring + syntese = Workflow-toolets kanoniske mønster, som gir (i) **deknings-garanti** (pipeline over hele capability-lista → ingen rad faller ut, jf. Verifisering #1), (ii) **innebygd adversarisk-verify** (parallelle skeptikere per disposition, jf. #5), (iii) parallellitet + typede schemas.
|
||||||
|
|
||||||
|
**Hybrid-form** (per Workflow-toolets egen anbefaling «scout inline først, deretter pipeline over arbeidslista»):
|
||||||
|
|
||||||
|
1. **Scout inline (main-context):** Phase 0 evidens-lesing + post-2.1.181-delta (1 research-agent) + Phase 1 capability-inventar → produserer arbeidslista (capability-rader). Dømmekraft-tungt; fan-outes IKKE.
|
||||||
|
2. **Workflow `pipeline()` per capability:** stage A = Phase 2 CC-overlap-map (m/ schema) → stage B = Phase 3 dobbel-akse-klassifisering (disposition-schema) → stage C = adversarisk verify (`parallel()` skeptikere som utfordrer KEEP/DROP→NATIVE).
|
||||||
|
3. **Syntese inline (main-context):** Phase 4 — skriv `docs/voyage-vs-cc-balance-analysis.md` + backlog fra de verifiserte radene. Dømmekraft-tungt; fan-outes IKKE.
|
||||||
|
|
||||||
|
Avgrensning: Workflow dekker KUN den parallelliserbare, verifiserbare kjernen (Phase 2–3) — speiler CC-27-funnet «~80 % glue, ~20 % fan-out». **Meta-bonus:** å kjøre analysen på Workflow-toolet gir førstehånds-evidens til analysens egen CC-27-rad (er Workflow-toolet et godt substrat?) — dogfooding av Workflow-toolet, ikke av Voyage.
|
||||||
|
|
||||||
|
## Verifisering (testbare kriterier)
|
||||||
|
|
||||||
|
1. **Dekning:** hver capability fra Phase 1 opptrer nøyaktig én gang i Phase 3-matrisen med disposition + begrunnelse + evidens-ref. (Inventar-antall == matrise-rad-antall.)
|
||||||
|
2. Hver «CC duplikat / gjør bedre» siterer en spesifikk CC-feature + versjon.
|
||||||
|
3. Hver «senker ekspertise-bar» navngir de konkrete CC-native stegene en ikke-ekspert ellers måtte gjort.
|
||||||
|
4. Post-2.1.181-delta faktisk kjørt: siste CC-versjon oppgitt + hvert nytt item triagert.
|
||||||
|
5. Adversarisk pass utfordret ≥ 1 KEEP og ≥ 1 DROP→NATIVE, registrert med verdikt.
|
||||||
|
6. Output ender med en prioritert endrings-backlog, hvert item tagget breaking/ikke-breaking og operatør-gated.
|
||||||
|
|
||||||
|
## Harde constraints (ikke bryt)
|
||||||
|
|
||||||
|
- **Trinity-asymmetri:** Voyage forblir uvitende om Tier 2/3; Handover 1 (brief-schema) er eneste integrasjonspunkt. Ingen anbefaling kan bryte den uten å flagges som public-contract breaking change.
|
||||||
|
- **Operatør-pinnede beslutninger** (premium default-profil, 24 agenter opus-pinnet, etc.): analysen MÅ gjerne utfordre dem, men må merke dem operatør-pinnet, ikke overstyre i stillhet.
|
||||||
|
- **Kun analyse:** anbefalinger + backlog. Ingen implementasjon uten egen operatør-gate (scope-guard).
|
||||||
|
|
||||||
|
## Kickoff (neste sesjon)
|
||||||
|
|
||||||
|
På kald start: les dette charteret + STATE, bekreft tilnærmingen (eller la operatør justere aksene/scope), start så Phase 0 (inline scout). Eksekver Phase 2–3 via **Dynamic Workflow** per «Eksekvering»-seksjonen (operatør-autorisert 2026-06-20). Charteret er skrevet S29 — ennå ikke committet (på disk i `docs/`, lesbart uansett).
|
||||||
|
|
@ -17,6 +17,7 @@
|
||||||
// - All stderr prefixed with [voyage].
|
// - All stderr prefixed with [voyage].
|
||||||
// - EXDEV mitigation: tmp file in same dir as target (do NOT use atomicWriteJson).
|
// - EXDEV mitigation: tmp file in same dir as target (do NOT use atomicWriteJson).
|
||||||
|
|
||||||
|
import { stdin } from 'node:process';
|
||||||
import { readFileSync, existsSync, writeFileSync, renameSync } from 'node:fs';
|
import { readFileSync, existsSync, writeFileSync, renameSync } from 'node:fs';
|
||||||
import { join, dirname } from 'node:path';
|
import { join, dirname } from 'node:path';
|
||||||
import { transformToPrometheus } from '../../lib/exporters/textfile-format.mjs';
|
import { transformToPrometheus } from '../../lib/exporters/textfile-format.mjs';
|
||||||
|
|
@ -24,6 +25,13 @@ import { transformToOtlpJson } from '../../lib/exporters/otlp-format.mjs';
|
||||||
import { validateTextfilePath } from '../../lib/exporters/path-validator.mjs';
|
import { validateTextfilePath } from '../../lib/exporters/path-validator.mjs';
|
||||||
import { validateOtlpEndpoint } from '../../lib/exporters/endpoint-validator.mjs';
|
import { validateOtlpEndpoint } from '../../lib/exporters/endpoint-validator.mjs';
|
||||||
import { applyFieldAllowlist } from '../../lib/exporters/field-allowlist.mjs';
|
import { applyFieldAllowlist } from '../../lib/exporters/field-allowlist.mjs';
|
||||||
|
import { captureTokenUsage } from '../../lib/stats/token-usage.mjs';
|
||||||
|
|
||||||
|
async function readStdin() {
|
||||||
|
let data = '';
|
||||||
|
for await (const chunk of stdin) data += chunk;
|
||||||
|
return data;
|
||||||
|
}
|
||||||
|
|
||||||
const VALID_MODES = new Set(['off', 'textfile', 'otlp']);
|
const VALID_MODES = new Set(['off', 'textfile', 'otlp']);
|
||||||
const TEXTFILE_NAME = 'voyage.prom';
|
const TEXTFILE_NAME = 'voyage.prom';
|
||||||
|
|
@ -38,6 +46,7 @@ const STATS_FILES = [
|
||||||
{ file: 'trekexecute-stats.jsonl', schema: 'trekexecute' },
|
{ file: 'trekexecute-stats.jsonl', schema: 'trekexecute' },
|
||||||
{ file: 'trekreview-stats.jsonl', schema: 'trekreview' },
|
{ file: 'trekreview-stats.jsonl', schema: 'trekreview' },
|
||||||
{ file: 'trekcontinue-stats.jsonl', schema: 'trekcontinue' },
|
{ file: 'trekcontinue-stats.jsonl', schema: 'trekcontinue' },
|
||||||
|
{ file: 'token-usage-stats.jsonl', schema: 'token-usage' },
|
||||||
];
|
];
|
||||||
|
|
||||||
function loadAndAllowlist(dataDir) {
|
function loadAndAllowlist(dataDir) {
|
||||||
|
|
@ -134,6 +143,26 @@ async function exportOtlp(records, env) {
|
||||||
(async () => {
|
(async () => {
|
||||||
try {
|
try {
|
||||||
const env = process.env;
|
const env = process.env;
|
||||||
|
|
||||||
|
// SKAL-2: opt-in main-context token/cost capture (default off → zero added
|
||||||
|
// latency). Rides this existing Stop hook rather than a new hook script —
|
||||||
|
// avoids a second-Stop-hook race and keeps the README hook-count pin intact.
|
||||||
|
// Runs BEFORE the export-mode gate so capture is independent of export.
|
||||||
|
// Fail-open: any error is swallowed; capture must never block Stop.
|
||||||
|
if (env.VOYAGE_TOKEN_METER) {
|
||||||
|
try {
|
||||||
|
const raw = await readStdin();
|
||||||
|
if (raw.trim()) {
|
||||||
|
const payload = JSON.parse(raw);
|
||||||
|
captureTokenUsage({
|
||||||
|
transcriptPath: payload.transcript_path,
|
||||||
|
sessionId: payload.session_id,
|
||||||
|
dataDir: env.CLAUDE_PLUGIN_DATA,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
} catch { /* fail-open: never block Stop, never throw */ }
|
||||||
|
}
|
||||||
|
|
||||||
const mode = (env.VOYAGE_EXPORT_MODE || 'off').toLowerCase();
|
const mode = (env.VOYAGE_EXPORT_MODE || 'off').toLowerCase();
|
||||||
|
|
||||||
if (mode === 'off') return;
|
if (mode === 'off') return;
|
||||||
|
|
|
||||||
|
|
@ -76,6 +76,16 @@ const TREKCONTINUE_ALLOWED = Object.freeze(new Set([
|
||||||
'ts', 'next_session_label', 'status', 'profile', 'profile_source',
|
'ts', 'next_session_label', 'status', 'profile', 'profile_source',
|
||||||
]));
|
]));
|
||||||
|
|
||||||
|
// Source: tests/fixtures/jsonl-schemas.md row 9 (token-usage — SKAL-2)
|
||||||
|
// CWE-212: numeric + low-cardinality-label fields ONLY. DENY BY OMISSION:
|
||||||
|
// session_id (UUID), transcript_path (filesystem path), cwd (filesystem path)
|
||||||
|
// are written into the jsonl for upsert keying but MUST NOT reach the exporter.
|
||||||
|
const TOKEN_USAGE_ALLOWED = Object.freeze(new Set([
|
||||||
|
'ts', 'scope', 'model',
|
||||||
|
'tokens_input', 'tokens_output', 'tokens_cache_creation', 'tokens_cache_read',
|
||||||
|
'cost_usd', 'is_estimate', 'price_table_version',
|
||||||
|
]));
|
||||||
|
|
||||||
// Schema-id → allowlist set
|
// Schema-id → allowlist set
|
||||||
const SCHEMA_ALLOWLISTS = Object.freeze({
|
const SCHEMA_ALLOWLISTS = Object.freeze({
|
||||||
'trekbrief': TREKBRIEF_ALLOWED,
|
'trekbrief': TREKBRIEF_ALLOWED,
|
||||||
|
|
@ -87,6 +97,7 @@ const SCHEMA_ALLOWLISTS = Object.freeze({
|
||||||
'post_bash_stats': POST_BASH_STATS_ALLOWED, // common alt-spelling
|
'post_bash_stats': POST_BASH_STATS_ALLOWED, // common alt-spelling
|
||||||
'trekreview': TREKREVIEW_ALLOWED,
|
'trekreview': TREKREVIEW_ALLOWED,
|
||||||
'trekcontinue': TREKCONTINUE_ALLOWED,
|
'trekcontinue': TREKCONTINUE_ALLOWED,
|
||||||
|
'token-usage': TOKEN_USAGE_ALLOWED,
|
||||||
});
|
});
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -135,4 +146,5 @@ export {
|
||||||
TREKEXECUTE_ALLOWED,
|
TREKEXECUTE_ALLOWED,
|
||||||
TREKREVIEW_ALLOWED,
|
TREKREVIEW_ALLOWED,
|
||||||
TREKCONTINUE_ALLOWED,
|
TREKCONTINUE_ALLOWED,
|
||||||
|
TOKEN_USAGE_ALLOWED,
|
||||||
};
|
};
|
||||||
|
|
|
||||||
|
|
@ -32,7 +32,7 @@ const OPTIONAL_KEYS = [
|
||||||
const OPTIONAL_BOOLEAN_KEYS = new Set(OPTIONAL_KEYS);
|
const OPTIONAL_BOOLEAN_KEYS = new Set(OPTIONAL_KEYS);
|
||||||
|
|
||||||
// Optional string-typed manifest keys (v4.1 Step 3 — additive forward-compat).
|
// Optional string-typed manifest keys (v4.1 Step 3 — additive forward-compat).
|
||||||
// `profile_used`: name of the model profile (economy|balanced|premium|<custom>) the
|
// `profile_used`: name of the model profile (economy|balanced|premium|fable|<custom>) the
|
||||||
// step was executed under. Absence is fine (v4.0 manifests have no
|
// step was executed under. Absence is fine (v4.0 manifests have no
|
||||||
// profile concept); presence MUST be a string.
|
// profile concept); presence MUST be a string.
|
||||||
// Unlike OPTIONAL_BOOLEAN_KEYS, absence is NOT defaulted — the field is simply
|
// Unlike OPTIONAL_BOOLEAN_KEYS, absence is NOT defaulted — the field is simply
|
||||||
|
|
|
||||||
|
|
@ -18,4 +18,9 @@ parallel_agents_min: 2
|
||||||
parallel_agents_max: 3
|
parallel_agents_max: 3
|
||||||
external_research_enabled: false
|
external_research_enabled: false
|
||||||
brief_reviewer_iter_cap: 1
|
brief_reviewer_iter_cap: 1
|
||||||
|
# Experimental: the cross-tier Jaccard floor (0.55) rests on parked-synthetic
|
||||||
|
# fixtures — empirical Step-17 calibration is deferred to v4.2. Drop this flag
|
||||||
|
# in the SAME change that lands calibration (status: empirical). See
|
||||||
|
# tests/synthetic/profile-jaccard-calibration.md and docs/profiles.md.
|
||||||
|
experimental: true
|
||||||
---
|
---
|
||||||
|
|
|
||||||
21
lib/profiles/fable.yaml
Normal file
21
lib/profiles/fable.yaml
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
---
|
||||||
|
profile_version: "1.0"
|
||||||
|
name: fable
|
||||||
|
phase_models:
|
||||||
|
- phase: brief
|
||||||
|
model: fable
|
||||||
|
- phase: research
|
||||||
|
model: fable
|
||||||
|
- phase: plan
|
||||||
|
model: fable
|
||||||
|
- phase: execute
|
||||||
|
model: fable
|
||||||
|
- phase: review
|
||||||
|
model: fable
|
||||||
|
- phase: continue
|
||||||
|
model: fable
|
||||||
|
parallel_agents_min: 6
|
||||||
|
parallel_agents_max: 8
|
||||||
|
external_research_enabled: true
|
||||||
|
brief_reviewer_iter_cap: 3
|
||||||
|
---
|
||||||
|
|
@ -67,6 +67,11 @@ export function resolvePhaseSignalFromFile(briefPath, phase) {
|
||||||
}
|
}
|
||||||
|
|
||||||
// CLI shim — mirrors lib/validators/brief-validator.mjs:168 pattern.
|
// CLI shim — mirrors lib/validators/brief-validator.mjs:168 pattern.
|
||||||
|
// Footgun guard (v5.9): this shim's `model` output is brief-signal-only — it
|
||||||
|
// never consults the profile layer. For command wiring, the composed resolver
|
||||||
|
// CLI (`resolver.mjs --resolve-phase-model`, brief > profile > default) is the
|
||||||
|
// single resolution source for {effort, model}. Do not re-wire commands/*.md
|
||||||
|
// Bash blocks back to this shim.
|
||||||
if (import.meta.url === `file://${process.argv[1]}`) {
|
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||||
const args = process.argv.slice(2);
|
const args = process.argv.slice(2);
|
||||||
const getArg = (name) => {
|
const getArg = (name) => {
|
||||||
|
|
|
||||||
|
|
@ -45,7 +45,7 @@ import { resolvePhaseSignal } from './phase-signal-resolver.mjs';
|
||||||
|
|
||||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||||
const BUILTIN_PROFILES_DIR = __dirname; // lib/profiles/
|
const BUILTIN_PROFILES_DIR = __dirname; // lib/profiles/
|
||||||
const BUILTIN_NAMES = new Set(['economy', 'balanced', 'premium']);
|
const BUILTIN_NAMES = new Set(['economy', 'balanced', 'premium', 'fable']);
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Resolve the path to a profile file.
|
* Resolve the path to a profile file.
|
||||||
|
|
@ -221,7 +221,13 @@ export function validateProfileFile(path, opts = {}) {
|
||||||
* @param {string|null} briefPath Absolute or repo-relative path to brief.md, or null
|
* @param {string|null} briefPath Absolute or repo-relative path to brief.md, or null
|
||||||
* @param {string[]|object} argv Full process.argv array OR parsed flags object
|
* @param {string[]|object} argv Full process.argv array OR parsed flags object
|
||||||
* @param {object} [env] Environment-variable record (defaults to process.env)
|
* @param {object} [env] Environment-variable record (defaults to process.env)
|
||||||
* @returns {{model: string, source: 'brief-signal'|'flag'|'env'|'default'}}
|
* @returns {{effort?: string, model: string, source: 'brief-signal'|'flag'|'env'|'default'}}
|
||||||
|
*
|
||||||
|
* `effort` (v5.9 ADDITIVE) is the brief signal's effort passed through when
|
||||||
|
* present — so commands consume ONE coherent {effort, model, source} result
|
||||||
|
* instead of two split CLI calls. Absent when the brief carries no valid
|
||||||
|
* effort signal for the phase (commands default to 'standard' per the
|
||||||
|
* composition rule).
|
||||||
*
|
*
|
||||||
* Error handling contract:
|
* Error handling contract:
|
||||||
* - Never throws. Any failure (ENOENT on briefPath, malformed YAML, missing
|
* - Never throws. Any failure (ENOENT on briefPath, malformed YAML, missing
|
||||||
|
|
@ -234,7 +240,10 @@ export function validateProfileFile(path, opts = {}) {
|
||||||
* directly; commands must inject {resolved model} at Agent-tool spawn sites.
|
* directly; commands must inject {resolved model} at Agent-tool spawn sites.
|
||||||
*/
|
*/
|
||||||
export function resolvePhaseModel(phase, briefPath, argv, env = process.env) {
|
export function resolvePhaseModel(phase, briefPath, argv, env = process.env) {
|
||||||
// Step 1: brief-signal lookup
|
// Step 1: brief-signal lookup. `effort` is captured independently of `model`
|
||||||
|
// so a signal like {effort: high} (no model) still passes effort through
|
||||||
|
// while the model falls to the profile layer.
|
||||||
|
let effort;
|
||||||
if (typeof briefPath === 'string' && briefPath.length > 0 && existsSync(briefPath)) {
|
if (typeof briefPath === 'string' && briefPath.length > 0 && existsSync(briefPath)) {
|
||||||
let fm = null;
|
let fm = null;
|
||||||
try {
|
try {
|
||||||
|
|
@ -246,8 +255,11 @@ export function resolvePhaseModel(phase, briefPath, argv, env = process.env) {
|
||||||
}
|
}
|
||||||
if (fm) {
|
if (fm) {
|
||||||
const signal = resolvePhaseSignal(fm, phase);
|
const signal = resolvePhaseSignal(fm, phase);
|
||||||
|
if (signal && typeof signal.effort === 'string') effort = signal.effort;
|
||||||
if (signal && typeof signal.model === 'string' && signal.model.length > 0) {
|
if (signal && typeof signal.model === 'string' && signal.model.length > 0) {
|
||||||
return { model: signal.model, source: 'brief-signal' };
|
return effort !== undefined
|
||||||
|
? { effort, model: signal.model, source: 'brief-signal' }
|
||||||
|
: { model: signal.model, source: 'brief-signal' };
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -278,7 +290,9 @@ export function resolvePhaseModel(phase, briefPath, argv, env = process.env) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
const model = phaseModels[phase] || 'opus';
|
const model = phaseModels[phase] || 'opus';
|
||||||
return { model, source: profile_source };
|
return effort !== undefined
|
||||||
|
? { effort, model, source: profile_source }
|
||||||
|
: { model, source: profile_source };
|
||||||
}
|
}
|
||||||
|
|
||||||
// CLI shim — invoked by commands/trek*.md via Bash.
|
// CLI shim — invoked by commands/trek*.md via Bash.
|
||||||
|
|
@ -300,7 +314,8 @@ if (import.meta.url === `file://${process.argv[1]}`) {
|
||||||
if (args.includes('--json')) {
|
if (args.includes('--json')) {
|
||||||
process.stdout.write(JSON.stringify(r) + '\n');
|
process.stdout.write(JSON.stringify(r) + '\n');
|
||||||
} else {
|
} else {
|
||||||
process.stdout.write(`model=${r.model} source=${r.source}\n`);
|
const effort = 'effort' in r ? ` effort=${r.effort}` : '';
|
||||||
|
process.stdout.write(`model=${r.model} source=${r.source}${effort}\n`);
|
||||||
}
|
}
|
||||||
process.exit(0);
|
process.exit(0);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
234
lib/review/coordinator-contract.mjs
Normal file
234
lib/review/coordinator-contract.mjs
Normal file
|
|
@ -0,0 +1,234 @@
|
||||||
|
// lib/review/coordinator-contract.mjs
|
||||||
|
// SKAL-1·4a — deterministic reference implementation of the review-coordinator
|
||||||
|
// 4-pass contract (agents/review-coordinator.md §"Your 4-pass process").
|
||||||
|
//
|
||||||
|
// This is a DETERMINISTIC SUBSET, not a full mirror of the LLM coordinator.
|
||||||
|
// It implements the pure, hermetic passes and DELIBERATELY EXCLUDES the parts
|
||||||
|
// that need a live filesystem or LLM judgement (which belong to the 4c
|
||||||
|
// LLM-in-the-loop eval, not this all-agree foundation tier):
|
||||||
|
// - Pass 2 "Accuracy" file-existence / line-plausibility glob (fs I/O).
|
||||||
|
// - Pass 2 "Actionability" imperative-verb heuristic — the real coordinator
|
||||||
|
// uses LLM judgement here; a verb-list approximation would DIVERGE from the
|
||||||
|
// contract being mirrored, so only the deterministic "recommended_action is
|
||||||
|
// present-and-non-empty when supplied" half is kept.
|
||||||
|
// - The doc's 4-tuple `(file,line,rule_key,title)` id recompute — the shipped
|
||||||
|
// `computeFindingId` is 3-arg `(file,line,rule_key)`; this module follows
|
||||||
|
// the shipped code and flags the doc divergence (the 4-tuple id is not
|
||||||
|
// producible by the current helper).
|
||||||
|
//
|
||||||
|
// What IS implemented, purely: Pass 1 (triplet dedup → highest-severity-wins
|
||||||
|
// survivor + conformance tiebreak + detail concat + raised_by provenance),
|
||||||
|
// Pass 2 succinctness + actionability-presence, Pass 3 reasonableness
|
||||||
|
// (citation / unknown-rule_key drop, severity-mismatch correction), Pass 4
|
||||||
|
// verdict thresholds. No LLM, no network, no time, no randomness.
|
||||||
|
//
|
||||||
|
// Reuses: SEVERITY_VALUES / RULE_KEYS / getRule (rule-catalogue.mjs),
|
||||||
|
// computeFindingId (finding-id.mjs, triplet), validateFindings
|
||||||
|
// (findings-schema.mjs). Triplet key format mirrors
|
||||||
|
// scripts/bakeoff-armA-merge.mjs:33; raised_by provenance mirrors
|
||||||
|
// lib/review/plan-review-dedup.mjs.
|
||||||
|
|
||||||
|
import { SEVERITY_VALUES, RULE_KEYS, getRule } from './rule-catalogue.mjs';
|
||||||
|
import { computeFindingId } from '../parsers/finding-id.mjs';
|
||||||
|
import { validateFindings } from './findings-schema.mjs';
|
||||||
|
|
||||||
|
export const JUDGE_TITLE_MAX = 100;
|
||||||
|
export const JUDGE_DETAIL_MAX = 800;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Catalogue-tier rank of a severity: lower number = higher severity.
|
||||||
|
* BLOCKER=0 … SUGGESTION=3; an unknown severity ranks last.
|
||||||
|
* @param {string} severity
|
||||||
|
* @returns {number}
|
||||||
|
*/
|
||||||
|
export function severityRank(severity) {
|
||||||
|
const i = SEVERITY_VALUES.indexOf(severity);
|
||||||
|
return i === -1 ? SEVERITY_VALUES.length : i;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isConformance(reviewer) {
|
||||||
|
return typeof reviewer === 'string' && reviewer.toLowerCase().includes('conformance');
|
||||||
|
}
|
||||||
|
|
||||||
|
function tripletKey(f) {
|
||||||
|
return `${f.file} ${f.line} ${f.rule_key}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Validate each reviewer payload and collect findings from the VALID ones,
|
||||||
|
* tagging each finding with its source reviewer (mirrors mergeArmA — invalid
|
||||||
|
* payloads are skipped, not crashed-on).
|
||||||
|
* @param {Array<{reviewer?: string, findings: object[]}>} reviewerPayloads
|
||||||
|
* @returns {{ findings: object[], skipped: Array<{reviewer: string|null, error_codes: string[]}> }}
|
||||||
|
*/
|
||||||
|
export function ingest(reviewerPayloads) {
|
||||||
|
const findings = [];
|
||||||
|
const skipped = [];
|
||||||
|
for (const payload of reviewerPayloads) {
|
||||||
|
const r = validateFindings(payload);
|
||||||
|
if (!r.valid) {
|
||||||
|
skipped.push({ reviewer: payload?.reviewer ?? null, error_codes: r.errors.map((e) => e.code) });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
for (const f of payload.findings) {
|
||||||
|
findings.push({ ...f, reviewer: f.reviewer ?? payload.reviewer ?? f.owner_reviewer ?? null });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return { findings, skipped };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pass 1 — dedup by (file, line, rule_key) triplet. Survivor = highest
|
||||||
|
* catalogue severity; severity tie → prefer the conformance reviewer; carries
|
||||||
|
* raised_by provenance, concatenates other reviewers' attribution into detail,
|
||||||
|
* and recomputes the id over the triplet.
|
||||||
|
* @param {object[]} findings
|
||||||
|
* @returns {object[]}
|
||||||
|
*/
|
||||||
|
export function dedupByTriplet(findings) {
|
||||||
|
const groups = new Map();
|
||||||
|
for (const f of findings) {
|
||||||
|
const key = tripletKey(f);
|
||||||
|
if (!groups.has(key)) groups.set(key, []);
|
||||||
|
groups.get(key).push(f);
|
||||||
|
}
|
||||||
|
const out = [];
|
||||||
|
for (const group of groups.values()) {
|
||||||
|
let survivor = group[0];
|
||||||
|
for (const f of group.slice(1)) {
|
||||||
|
const higher = severityRank(f.severity) < severityRank(survivor.severity);
|
||||||
|
const tieToConformance =
|
||||||
|
severityRank(f.severity) === severityRank(survivor.severity) &&
|
||||||
|
isConformance(f.reviewer) && !isConformance(survivor.reviewer);
|
||||||
|
if (higher || tieToConformance) survivor = f;
|
||||||
|
}
|
||||||
|
const raised_by = [...new Set(group.map((f) => f.reviewer).filter(Boolean))];
|
||||||
|
const others = group.filter((f) => f !== survivor);
|
||||||
|
let detail = survivor.detail;
|
||||||
|
if (others.length > 0) {
|
||||||
|
detail = survivor.detail ?? '';
|
||||||
|
for (const o of others) {
|
||||||
|
detail += `\nAlso flagged by ${o.reviewer ?? 'unknown'}: ${o.title ?? o.rule_key}.`;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const id = computeFindingId(survivor.file, survivor.line, survivor.rule_key);
|
||||||
|
out.push({ ...survivor, id, ...(detail !== undefined ? { detail } : {}), raised_by });
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pass 2 — HubSpot Judge (deterministic subset): drop on succinctness
|
||||||
|
* (title > 100 or detail > 800 chars) and actionability (recommended_action,
|
||||||
|
* when present, must be a non-empty string). The imperative-verb test is
|
||||||
|
* excluded (LLM judgement).
|
||||||
|
* @param {object[]} findings
|
||||||
|
* @returns {{ kept: object[], dropped: object[] }}
|
||||||
|
*/
|
||||||
|
export function judgeFilter(findings) {
|
||||||
|
const kept = [];
|
||||||
|
const dropped = [];
|
||||||
|
for (const f of findings) {
|
||||||
|
const titleLen = (f.title ?? '').length;
|
||||||
|
const detailLen = (f.detail ?? '').length;
|
||||||
|
let reason = null;
|
||||||
|
if (titleLen > JUDGE_TITLE_MAX) reason = 'succinctness:title';
|
||||||
|
else if (detailLen > JUDGE_DETAIL_MAX) reason = 'succinctness:detail';
|
||||||
|
else if ('recommended_action' in f &&
|
||||||
|
(typeof f.recommended_action !== 'string' || f.recommended_action.trim().length === 0)) {
|
||||||
|
reason = 'actionability:empty';
|
||||||
|
}
|
||||||
|
if (reason) dropped.push({ ...f, suppressed_reason: reason });
|
||||||
|
else kept.push(f);
|
||||||
|
}
|
||||||
|
return { kept, dropped };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pass 3 — Cloudflare reasonableness (deterministic subset): drop findings
|
||||||
|
* with no citation (empty file / line < 0) or an unknown rule_key; CORRECT a
|
||||||
|
* severity that does not match the catalogue tier (a correction, not a drop).
|
||||||
|
* The fs file-existence glob is excluded (I/O).
|
||||||
|
* @param {object[]} findings
|
||||||
|
* @returns {{ kept: object[], dropped: object[] }}
|
||||||
|
*/
|
||||||
|
export function reasonablenessFilter(findings) {
|
||||||
|
const kept = [];
|
||||||
|
const dropped = [];
|
||||||
|
for (const f of findings) {
|
||||||
|
if (typeof f.file !== 'string' || f.file.length === 0 ||
|
||||||
|
(typeof f.line === 'number' && f.line < 0)) {
|
||||||
|
dropped.push({ ...f, suppressed_reason: 'no-citation' });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (!RULE_KEYS.has(f.rule_key)) {
|
||||||
|
dropped.push({ ...f, suppressed_reason: 'unknown-rule_key' });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const rule = getRule(f.rule_key);
|
||||||
|
if (rule && f.severity !== rule.severity) {
|
||||||
|
kept.push({ ...f, severity: rule.severity, original_severity: f.severity });
|
||||||
|
} else {
|
||||||
|
kept.push(f);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return { kept, dropped };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pass 4 — compute the verdict from severity counts (after dedup + filtering).
|
||||||
|
* BLOCKER ≥ 1 → BLOCK; else MAJOR ≥ 1 → WARN; else ALLOW.
|
||||||
|
* @param {object[]} findings
|
||||||
|
* @returns {{ verdict: 'BLOCK'|'WARN'|'ALLOW', counts: Record<string, number> }}
|
||||||
|
*/
|
||||||
|
export function computeVerdict(findings) {
|
||||||
|
const counts = { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 };
|
||||||
|
for (const f of findings) {
|
||||||
|
if (counts[f.severity] !== undefined) counts[f.severity] += 1;
|
||||||
|
}
|
||||||
|
let verdict;
|
||||||
|
if (counts.BLOCKER >= 1) verdict = 'BLOCK';
|
||||||
|
else if (counts.MAJOR >= 1) verdict = 'WARN';
|
||||||
|
else verdict = 'ALLOW';
|
||||||
|
return { verdict, counts };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Run the full deterministic contract: ingest → Pass 1 → Pass 2 → Pass 3 → Pass 4.
|
||||||
|
* @param {Array<{reviewer?: string, findings: object[]}>} reviewerPayloads
|
||||||
|
* @returns {{ verdict: string, counts: Record<string, number>, findings: object[], suppressed: object[], skipped: object[] }}
|
||||||
|
*/
|
||||||
|
export function runContract(reviewerPayloads) {
|
||||||
|
const { findings: ingested, skipped } = ingest(reviewerPayloads);
|
||||||
|
const deduped = dedupByTriplet(ingested);
|
||||||
|
const judged = judgeFilter(deduped);
|
||||||
|
const reasoned = reasonablenessFilter(judged.kept);
|
||||||
|
const { verdict, counts } = computeVerdict(reasoned.kept);
|
||||||
|
return {
|
||||||
|
verdict,
|
||||||
|
counts,
|
||||||
|
findings: reasoned.kept,
|
||||||
|
suppressed: [...judged.dropped, ...reasoned.dropped],
|
||||||
|
skipped,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- CLI shim ----------------------------------------------------------------
|
||||||
|
|
||||||
|
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||||
|
const args = process.argv.slice(2);
|
||||||
|
const filePath = args.find((a) => !a.startsWith('--'));
|
||||||
|
if (!filePath) {
|
||||||
|
process.stderr.write('Usage: coordinator-contract.mjs [--json] <reviewer-payloads.json>\n');
|
||||||
|
process.exit(2);
|
||||||
|
}
|
||||||
|
const { readFileSync } = await import('node:fs');
|
||||||
|
const payloads = JSON.parse(readFileSync(filePath, 'utf-8'));
|
||||||
|
const result = runContract(Array.isArray(payloads) ? payloads : [payloads]);
|
||||||
|
if (args.includes('--json')) {
|
||||||
|
process.stdout.write(JSON.stringify(result, null, 2) + '\n');
|
||||||
|
} else {
|
||||||
|
process.stdout.write(`coordinator-contract: ${result.verdict} (${result.findings.length} findings, ${result.suppressed.length} suppressed)\n`);
|
||||||
|
}
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
BIN
lib/review/gold-scorer.mjs
Normal file
BIN
lib/review/gold-scorer.mjs
Normal file
Binary file not shown.
|
|
@ -43,6 +43,16 @@ export function summarize(lines) {
|
||||||
unique_event_names: [],
|
unique_event_names: [],
|
||||||
oldest_event_iso: null,
|
oldest_event_iso: null,
|
||||||
newest_event_iso: null,
|
newest_event_iso: null,
|
||||||
|
// SKAL-2 token/cost aggregation (additive; zero when no token records).
|
||||||
|
// With upsert semantics (one record per session_id), summing across lines
|
||||||
|
// = correct cross-session aggregate. cost_usd is summed only when finite;
|
||||||
|
// a null (refuse-to-estimate) record still counts in sessions_with_tokens.
|
||||||
|
total_tokens_input: 0,
|
||||||
|
total_tokens_output: 0,
|
||||||
|
total_tokens_cache_creation: 0,
|
||||||
|
total_tokens_cache_read: 0,
|
||||||
|
total_cost_usd: 0,
|
||||||
|
sessions_with_tokens: 0,
|
||||||
};
|
};
|
||||||
|
|
||||||
const durations = [];
|
const durations = [];
|
||||||
|
|
@ -71,6 +81,20 @@ export function summarize(lines) {
|
||||||
if (newestMs === null || t > newestMs) newestMs = t;
|
if (newestMs === null || t > newestMs) newestMs = t;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SKAL-2: aggregate token/cost from token-bearing records (token-usage
|
||||||
|
// schema). Detect by presence of any numeric token field.
|
||||||
|
const tokenKeys = ['tokens_input', 'tokens_output', 'tokens_cache_creation', 'tokens_cache_read'];
|
||||||
|
const hasTokens = tokenKeys.some(k => typeof obj[k] === 'number' && Number.isFinite(obj[k]));
|
||||||
|
if (hasTokens) {
|
||||||
|
summary.sessions_with_tokens++;
|
||||||
|
if (Number.isFinite(obj.tokens_input)) summary.total_tokens_input += obj.tokens_input;
|
||||||
|
if (Number.isFinite(obj.tokens_output)) summary.total_tokens_output += obj.tokens_output;
|
||||||
|
if (Number.isFinite(obj.tokens_cache_creation)) summary.total_tokens_cache_creation += obj.tokens_cache_creation;
|
||||||
|
if (Number.isFinite(obj.tokens_cache_read)) summary.total_tokens_cache_read += obj.tokens_cache_read;
|
||||||
|
// cost_usd may be null (refuse-to-estimate) — sum finite values only.
|
||||||
|
if (Number.isFinite(obj.cost_usd)) summary.total_cost_usd += obj.cost_usd;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (durations.length > 0) {
|
if (durations.length > 0) {
|
||||||
|
|
|
||||||
234
lib/stats/token-usage.mjs
Normal file
234
lib/stats/token-usage.mjs
Normal file
|
|
@ -0,0 +1,234 @@
|
||||||
|
// lib/stats/token-usage.mjs
|
||||||
|
// SKAL-2 — pure token-usage parser + cache-aware cost derivation for Voyage
|
||||||
|
// observability. Captures MAIN-CONTEXT token/cost from the Claude Code
|
||||||
|
// transcript (transcript_path). Each assistant record carries message.usage
|
||||||
|
// with a per-REQUEST snapshot.
|
||||||
|
//
|
||||||
|
// Verified 2026-06-26 against a real local transcript (216 lines): 70
|
||||||
|
// assistant records collapse to 31 distinct requestIds — records duplicate
|
||||||
|
// per requestId (streamed snapshots), so we dedup by requestId keeping the
|
||||||
|
// LAST occurrence (GH #28197 streaming-placeholder mitigation), then sum
|
||||||
|
// across requests. The main transcript carried zero isSidechain:true records.
|
||||||
|
//
|
||||||
|
// v1 scope = MAIN-CONTEXT only. Sub-agent (swarm) turns live in separate
|
||||||
|
// agent-*.jsonl siblings and are a documented v2 follow-on — so the
|
||||||
|
// main-transcript sum UNDER-counts total Voyage cost. Every record is stamped
|
||||||
|
// scope:'main-context' so no reader mistakes cost_usd for the session total.
|
||||||
|
//
|
||||||
|
// The four core functions (parseTranscriptUsage, deriveCost, buildRecord,
|
||||||
|
// upsertSessionRecord) are PURE (no I/O). captureTokenUsage is the impure
|
||||||
|
// shell that wires them to the filesystem (read transcript → upsert jsonl).
|
||||||
|
//
|
||||||
|
// Zero npm dependencies. Node stdlib only.
|
||||||
|
|
||||||
|
import { readFileSync, existsSync, writeFileSync, renameSync, statSync } from 'node:fs';
|
||||||
|
import { join, dirname } from 'node:path';
|
||||||
|
|
||||||
|
// Per-Mtok USD prices, resolved 2026-06-26 via the claude-api skill reference:
|
||||||
|
// base input/output from the model table; cache rates from the prompt-caching
|
||||||
|
// doc multipliers (cache_read 0.1x, write_5m 1.25x, write_1h 2.0x of input).
|
||||||
|
// claude-fable-5 resolved 2026-07-02 from the official platform pricing docs.
|
||||||
|
export const PRICE_TABLE = Object.freeze({
|
||||||
|
'claude-opus-4-8': Object.freeze({
|
||||||
|
input: 5.0,
|
||||||
|
output: 25.0,
|
||||||
|
cache_read: 0.5,
|
||||||
|
cache_write_5m: 6.25,
|
||||||
|
cache_write_1h: 10.0,
|
||||||
|
}),
|
||||||
|
'claude-fable-5': Object.freeze({
|
||||||
|
input: 10.0,
|
||||||
|
output: 50.0,
|
||||||
|
cache_read: 1.0,
|
||||||
|
cache_write_5m: 12.5,
|
||||||
|
cache_write_1h: 20.0,
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
|
||||||
|
// Date the PRICE_TABLE values were resolved/verified. Bump when prices change.
|
||||||
|
export const PRICE_TABLE_VERSION = '2026-07-02';
|
||||||
|
|
||||||
|
function num(v) {
|
||||||
|
return typeof v === 'number' && Number.isFinite(v) ? v : 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Single pass over transcript JSONL text. Keeps only main-chain assistant
|
||||||
|
* records (type==='assistant', isSidechain!==true), dedups by requestId
|
||||||
|
* keeping the LAST occurrence (per-request usage snapshots duplicate across
|
||||||
|
* streamed records), and sums the four usage token fields. Malformed lines
|
||||||
|
* are skipped.
|
||||||
|
*
|
||||||
|
* @returns {{tokens_input:number, tokens_output:number,
|
||||||
|
* tokens_cache_creation:number, tokens_cache_read:number}}
|
||||||
|
*/
|
||||||
|
export function parseTranscriptUsage(text) {
|
||||||
|
const lines = (text || '').split('\n');
|
||||||
|
// requestId -> usage (last occurrence wins). Records without a requestId
|
||||||
|
// get a unique sentinel key so each is still counted exactly once.
|
||||||
|
const byRequest = new Map();
|
||||||
|
let anon = 0;
|
||||||
|
for (const line of lines) {
|
||||||
|
const trimmed = line.trim();
|
||||||
|
if (trimmed === '') continue;
|
||||||
|
let obj;
|
||||||
|
try { obj = JSON.parse(trimmed); }
|
||||||
|
catch { continue; }
|
||||||
|
if (!obj || obj.type !== 'assistant') continue;
|
||||||
|
if (obj.isSidechain === true) continue;
|
||||||
|
const usage = obj.message && obj.message.usage;
|
||||||
|
if (!usage || typeof usage !== 'object') continue;
|
||||||
|
const key = typeof obj.requestId === 'string' && obj.requestId
|
||||||
|
? obj.requestId
|
||||||
|
: `__anon_${anon++}`;
|
||||||
|
byRequest.set(key, usage); // last occurrence wins
|
||||||
|
}
|
||||||
|
const totals = {
|
||||||
|
tokens_input: 0,
|
||||||
|
tokens_output: 0,
|
||||||
|
tokens_cache_creation: 0,
|
||||||
|
tokens_cache_read: 0,
|
||||||
|
};
|
||||||
|
for (const usage of byRequest.values()) {
|
||||||
|
totals.tokens_input += num(usage.input_tokens);
|
||||||
|
totals.tokens_output += num(usage.output_tokens);
|
||||||
|
totals.tokens_cache_creation += num(usage.cache_creation_input_tokens);
|
||||||
|
totals.tokens_cache_read += num(usage.cache_read_input_tokens);
|
||||||
|
}
|
||||||
|
return totals;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Cache-aware USD cost. Lumped cache_creation is priced at the 5m write rate
|
||||||
|
* (Claude Code's default cache TTL; verified the dominant case on a real
|
||||||
|
* transcript — the 1h split is a documented v2 refinement). When the model is
|
||||||
|
* absent from the table we REFUSE to estimate: {cost_usd:null, is_estimate:true}.
|
||||||
|
*
|
||||||
|
* @returns {{cost_usd:number|null, is_estimate:boolean}}
|
||||||
|
*/
|
||||||
|
export function deriveCost(totals, model, priceTable = PRICE_TABLE) {
|
||||||
|
const price = priceTable && priceTable[model];
|
||||||
|
if (!price) return { cost_usd: null, is_estimate: true };
|
||||||
|
const t = totals || {};
|
||||||
|
const cost =
|
||||||
|
(num(t.tokens_input) * price.input +
|
||||||
|
num(t.tokens_output) * price.output +
|
||||||
|
num(t.tokens_cache_creation) * price.cache_write_5m +
|
||||||
|
num(t.tokens_cache_read) * price.cache_read) / 1_000_000;
|
||||||
|
return { cost_usd: cost, is_estimate: false };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build the flat numeric record. Stamps scope:'main-context' and
|
||||||
|
* price_table_version. session_id is written for upsert keying but MUST be
|
||||||
|
* stripped at export (CWE-212, enforced by the field allowlist in Step 2).
|
||||||
|
*
|
||||||
|
* @param {{sessionId:string, model:string, totals:object,
|
||||||
|
* priceTable?:object, now:string}} args
|
||||||
|
*/
|
||||||
|
export function buildRecord({ sessionId, model, totals, priceTable = PRICE_TABLE, now }) {
|
||||||
|
const t = totals || {};
|
||||||
|
const { cost_usd, is_estimate } = deriveCost(t, model, priceTable);
|
||||||
|
return {
|
||||||
|
ts: now,
|
||||||
|
session_id: sessionId,
|
||||||
|
scope: 'main-context',
|
||||||
|
model,
|
||||||
|
tokens_input: num(t.tokens_input),
|
||||||
|
tokens_output: num(t.tokens_output),
|
||||||
|
tokens_cache_creation: num(t.tokens_cache_creation),
|
||||||
|
tokens_cache_read: num(t.tokens_cache_read),
|
||||||
|
cost_usd,
|
||||||
|
is_estimate,
|
||||||
|
price_table_version: PRICE_TABLE_VERSION,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pure read-modify-write. Parses existing JSONL lines, REPLACES the line whose
|
||||||
|
* session_id matches the record (dropping any further same-session dupes),
|
||||||
|
* else APPENDS. Returns the new file text (one line per session, trailing
|
||||||
|
* newline). One-line-per-session is what keeps cross-session sums from
|
||||||
|
* double-counting (vs append-and-grow).
|
||||||
|
*/
|
||||||
|
export function upsertSessionRecord(existingText, record) {
|
||||||
|
const lines = (existingText || '').split('\n');
|
||||||
|
const out = [];
|
||||||
|
let replaced = false;
|
||||||
|
for (const line of lines) {
|
||||||
|
const trimmed = line.trim();
|
||||||
|
if (trimmed === '') continue;
|
||||||
|
let obj;
|
||||||
|
try { obj = JSON.parse(trimmed); }
|
||||||
|
catch { out.push(trimmed); continue; } // preserve unparseable lines verbatim
|
||||||
|
if (obj && obj.session_id === record.session_id) {
|
||||||
|
if (!replaced) { out.push(JSON.stringify(record)); replaced = true; }
|
||||||
|
// else: duplicate same-session line — drop it
|
||||||
|
} else {
|
||||||
|
out.push(trimmed);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!replaced) out.push(JSON.stringify(record));
|
||||||
|
return out.join('\n') + '\n';
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Last main-chain (non-sidechain) assistant model in the transcript. Used to
|
||||||
|
* pick the price-table key. Returns null when no model is found (→ deriveCost
|
||||||
|
* refuses to estimate). Pure.
|
||||||
|
*/
|
||||||
|
export function lastMainChainModel(text) {
|
||||||
|
const lines = (text || '').split('\n');
|
||||||
|
let model = null;
|
||||||
|
for (const line of lines) {
|
||||||
|
const trimmed = line.trim();
|
||||||
|
if (trimmed === '') continue;
|
||||||
|
let obj;
|
||||||
|
try { obj = JSON.parse(trimmed); }
|
||||||
|
catch { continue; }
|
||||||
|
if (!obj || obj.type !== 'assistant' || obj.isSidechain === true) continue;
|
||||||
|
const m = obj.message && obj.message.model;
|
||||||
|
if (typeof m === 'string' && m) model = m;
|
||||||
|
}
|
||||||
|
return model;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Impure capture shell (Step 4). Reads the transcript, derives main-context
|
||||||
|
* token totals + cost, and UPSERTS a one-line-per-session record into
|
||||||
|
* {dataDir}/token-usage-stats.jsonl via an atomic temp+rename write
|
||||||
|
* (EXDEV mitigation: tmp lives in the same dir as the target).
|
||||||
|
*
|
||||||
|
* Returns the written record, or null when skipped (no path / no dataDir /
|
||||||
|
* transcript not a readable regular file). Throws propagate to the caller —
|
||||||
|
* the Stop hook wraps this in try/catch so capture stays fail-open.
|
||||||
|
*
|
||||||
|
* @param {{transcriptPath:string, sessionId:string, dataDir:string,
|
||||||
|
* now?:string}} args
|
||||||
|
*/
|
||||||
|
export function captureTokenUsage({ transcriptPath, sessionId, dataDir, now }) {
|
||||||
|
if (!transcriptPath || !dataDir) return null;
|
||||||
|
let st;
|
||||||
|
try { st = statSync(transcriptPath); }
|
||||||
|
catch { return null; }
|
||||||
|
if (!st.isFile()) return null;
|
||||||
|
|
||||||
|
const text = readFileSync(transcriptPath, 'utf-8');
|
||||||
|
const totals = parseTranscriptUsage(text);
|
||||||
|
const model = lastMainChainModel(text);
|
||||||
|
const record = buildRecord({
|
||||||
|
sessionId,
|
||||||
|
model,
|
||||||
|
totals,
|
||||||
|
now: now || new Date().toISOString(),
|
||||||
|
});
|
||||||
|
|
||||||
|
const outPath = join(dataDir, 'token-usage-stats.jsonl');
|
||||||
|
const existing = existsSync(outPath) ? readFileSync(outPath, 'utf-8') : '';
|
||||||
|
const updated = upsertSessionRecord(existing, record);
|
||||||
|
|
||||||
|
const tmpPath = join(dirname(outPath), '.token-usage-stats.jsonl.tmp');
|
||||||
|
writeFileSync(tmpPath, updated);
|
||||||
|
renameSync(tmpPath, outPath);
|
||||||
|
return record;
|
||||||
|
}
|
||||||
|
|
@ -1,10 +1,16 @@
|
||||||
// lib/util/test-census.mjs
|
// lib/util/test-census.mjs
|
||||||
// Census of the test suite: split top-level test() declarations into
|
// Census of the test suite: split top-level test() declarations into three
|
||||||
// "behavior" tests vs "doc-consistency pins" (string/existence assertions that
|
// honest categories:
|
||||||
// pin documentation against a source-of-truth). Makes the cited test count
|
// - "behavior" — ordinary behavior coverage (the default bucket).
|
||||||
// honest — a prose-pin is not the same coverage as a behavior test, so a single
|
// - "docPins" — doc-consistency pins (string/existence assertions that pin
|
||||||
|
// documentation against a source-of-truth).
|
||||||
|
// - "goldEval" — the offline gold-scored output eval scoring RUN (SKAL-1·4b):
|
||||||
|
// neither behavior nor prose-pin, but a run that scores a
|
||||||
|
// committed agent-run fixture against the golden corpus.
|
||||||
|
// Makes the cited test count honest — a prose-pin is not the same coverage as a
|
||||||
|
// behavior test, and a scoring run is a third thing again, so a single
|
||||||
// conflated total oversells behavior coverage.
|
// conflated total oversells behavior coverage.
|
||||||
// Devil's-advocate audit §Top changes #8 (S19).
|
// Devil's-advocate audit §Top changes #8 (S19); third bucket added in SKAL-1·4b.
|
||||||
//
|
//
|
||||||
// Metric: top-level `test(` declarations (static, deterministic, in-process).
|
// Metric: top-level `test(` declarations (static, deterministic, in-process).
|
||||||
// This is distinct from node:test's runtime total, which additionally counts
|
// This is distinct from node:test's runtime total, which additionally counts
|
||||||
|
|
@ -18,6 +24,11 @@ import { join } from 'node:path';
|
||||||
// is bucketed correctly without editing this module.
|
// is bucketed correctly without editing this module.
|
||||||
export const PIN_FILE_RE = /(doc-consistency|prose-pins)\.test\.mjs$/;
|
export const PIN_FILE_RE = /(doc-consistency|prose-pins)\.test\.mjs$/;
|
||||||
|
|
||||||
|
// Files whose tests are the offline gold-scored output eval scoring run
|
||||||
|
// (SKAL-1·4b). Kept as a regex (not a single filename) so a future split-out
|
||||||
|
// is bucketed correctly without editing this module.
|
||||||
|
export const GOLD_EVAL_FILE_RE = /gold-eval\.test\.mjs$/;
|
||||||
|
|
||||||
function walk(dir) {
|
function walk(dir) {
|
||||||
const out = [];
|
const out = [];
|
||||||
for (const e of readdirSync(dir, { withFileTypes: true })) {
|
for (const e of readdirSync(dir, { withFileTypes: true })) {
|
||||||
|
|
@ -32,18 +43,20 @@ function countTests(file) {
|
||||||
return (readFileSync(file, 'utf-8').match(/^\s*test\(/gm) || []).length;
|
return (readFileSync(file, 'utf-8').match(/^\s*test\(/gm) || []).length;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Returns { behavior, docPins, total, byFile } for all *.test.mjs under
|
// Returns { behavior, docPins, goldEval, total, byFile } for all *.test.mjs
|
||||||
// testsRoot. behavior + docPins === total by construction.
|
// under testsRoot. behavior + docPins + goldEval === total by construction.
|
||||||
export function censusTests(testsRoot) {
|
export function censusTests(testsRoot) {
|
||||||
const files = walk(testsRoot).sort();
|
const files = walk(testsRoot).sort();
|
||||||
const byFile = {};
|
const byFile = {};
|
||||||
let behavior = 0;
|
let behavior = 0;
|
||||||
let docPins = 0;
|
let docPins = 0;
|
||||||
|
let goldEval = 0;
|
||||||
for (const f of files) {
|
for (const f of files) {
|
||||||
const n = countTests(f);
|
const n = countTests(f);
|
||||||
byFile[f] = n;
|
byFile[f] = n;
|
||||||
if (PIN_FILE_RE.test(f)) docPins += n;
|
if (PIN_FILE_RE.test(f)) docPins += n;
|
||||||
|
else if (GOLD_EVAL_FILE_RE.test(f)) goldEval += n;
|
||||||
else behavior += n;
|
else behavior += n;
|
||||||
}
|
}
|
||||||
return { behavior, docPins, total: behavior + docPins, byFile };
|
return { behavior, docPins, goldEval, total: behavior + docPins + goldEval, byFile };
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -252,7 +252,11 @@ if (import.meta.url === `file://${process.argv[1]}`) {
|
||||||
const minIdx = args.indexOf('--min-version');
|
const minIdx = args.indexOf('--min-version');
|
||||||
const minBriefVersion = minIdx >= 0 ? args[minIdx + 1] : undefined;
|
const minBriefVersion = minIdx >= 0 ? args[minIdx + 1] : undefined;
|
||||||
// filePath is the first positional, skipping the --min-version value token.
|
// filePath is the first positional, skipping the --min-version value token.
|
||||||
const filePath = args.find((a, i) => !a.startsWith('--') && i !== minIdx + 1);
|
// Guard: when --min-version is absent (minIdx === -1) the skip index must be -1,
|
||||||
|
// not 0 — otherwise the no-flag invocation `brief-validator.mjs <brief.md>` drops
|
||||||
|
// the file (which sits at index 0) and bails to Usage.
|
||||||
|
const skipIdx = minIdx >= 0 ? minIdx + 1 : -1;
|
||||||
|
const filePath = args.find((a, i) => !a.startsWith('--') && i !== skipIdx);
|
||||||
if (!filePath) {
|
if (!filePath) {
|
||||||
process.stderr.write('Usage: brief-validator.mjs [--soft] [--min-version <x.y>] <brief.md>\n');
|
process.stderr.write('Usage: brief-validator.mjs [--soft] [--min-version <x.y>] <brief.md>\n');
|
||||||
process.exit(2);
|
process.exit(2);
|
||||||
|
|
|
||||||
|
|
@ -9,6 +9,10 @@
|
||||||
// parallel_agents_max : number (≥ parallel_agents_min)
|
// parallel_agents_max : number (≥ parallel_agents_min)
|
||||||
// external_research_enabled : boolean
|
// external_research_enabled : boolean
|
||||||
// brief_reviewer_iter_cap : number (≥ 1)
|
// brief_reviewer_iter_cap : number (≥ 1)
|
||||||
|
// experimental : boolean (OPTIONAL) — true marks a tier whose
|
||||||
|
// gating constants are not yet empirically
|
||||||
|
// calibrated (e.g. economy's parked-synthetic
|
||||||
|
// cross-tier Jaccard floor). Absent ⇒ stable.
|
||||||
//
|
//
|
||||||
// Issue codes:
|
// Issue codes:
|
||||||
// PROFILE_MISSING_FIELD — required top-level frontmatter field absent
|
// PROFILE_MISSING_FIELD — required top-level frontmatter field absent
|
||||||
|
|
@ -17,7 +21,7 @@
|
||||||
// PROFILE_READ_ERROR — file unreadable or parse-error
|
// PROFILE_READ_ERROR — file unreadable or parse-error
|
||||||
// PROFILE_NOT_FOUND — file does not exist
|
// PROFILE_NOT_FOUND — file does not exist
|
||||||
//
|
//
|
||||||
// Allowed model values: ['sonnet', 'opus']. Haiku is allowed only when
|
// Allowed model values: ['sonnet', 'opus', 'fable']. Haiku is allowed only when
|
||||||
// VOYAGE_ALLOW_HAIKU=1 (per global CLAUDE.md modellvalg-prinsipp: Haiku skal
|
// VOYAGE_ALLOW_HAIKU=1 (per global CLAUDE.md modellvalg-prinsipp: Haiku skal
|
||||||
// ikke brukes som default; eksplisitt opt-in for spesielle bruksmønstre).
|
// ikke brukes som default; eksplisitt opt-in for spesielle bruksmønstre).
|
||||||
|
|
||||||
|
|
@ -38,7 +42,7 @@ export const PROFILE_REQUIRED_PHASES = Object.freeze([
|
||||||
'brief', 'research', 'plan', 'execute', 'review', 'continue',
|
'brief', 'research', 'plan', 'execute', 'review', 'continue',
|
||||||
]);
|
]);
|
||||||
|
|
||||||
export const BASE_ALLOWED_MODELS = Object.freeze(['sonnet', 'opus']);
|
export const BASE_ALLOWED_MODELS = Object.freeze(['sonnet', 'opus', 'fable']);
|
||||||
|
|
||||||
function getAllowedModels(env = process.env) {
|
function getAllowedModels(env = process.env) {
|
||||||
if (env.VOYAGE_ALLOW_HAIKU === '1') {
|
if (env.VOYAGE_ALLOW_HAIKU === '1') {
|
||||||
|
|
@ -99,6 +103,12 @@ export function validateProfileContent(text, opts = {}) {
|
||||||
`brief_reviewer_iter_cap must be number (got ${typeof fm.brief_reviewer_iter_cap})`));
|
`brief_reviewer_iter_cap must be number (got ${typeof fm.brief_reviewer_iter_cap})`));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Optional field: `experimental` (boolean). Absent ⇒ tier is stable.
|
||||||
|
if ('experimental' in fm && typeof fm.experimental !== 'boolean') {
|
||||||
|
errors.push(issue('PROFILE_INVALID_ENUM',
|
||||||
|
`experimental must be boolean (got ${typeof fm.experimental})`));
|
||||||
|
}
|
||||||
|
|
||||||
// phase_models validation
|
// phase_models validation
|
||||||
if ('phase_models' in fm) {
|
if ('phase_models' in fm) {
|
||||||
if (!Array.isArray(fm.phase_models)) {
|
if (!Array.isArray(fm.phase_models)) {
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,11 @@
|
||||||
// lib/validators/progress-validator.mjs
|
// lib/validators/progress-validator.mjs
|
||||||
// Validate progress.json shape + resume-readiness.
|
// Validate progress.json shape + resume-readiness.
|
||||||
|
// Forward-compat: unknown keys are tolerated silently, so additive-optional fields land
|
||||||
|
// WITHOUT a schema_version bump. `iterations_remaining` (trekexecute's recovery/retry budget
|
||||||
|
// signal) is shape-checked here only when present (non-negative integer); it is NOT in
|
||||||
|
// PROGRESS_REQUIRED_TOP, so legacy progress.json without it still validates. The field's
|
||||||
|
// liveness (never-decremented) is enforced deterministically by the Phase 7.5 completion-gate
|
||||||
|
// cross-check in trekexecute, not by this per-validation shape check.
|
||||||
|
|
||||||
import { readFileSync, existsSync } from 'node:fs';
|
import { readFileSync, existsSync } from 'node:fs';
|
||||||
import { issue, fail } from '../util/result.mjs';
|
import { issue, fail } from '../util/result.mjs';
|
||||||
|
|
@ -45,6 +51,16 @@ export function validateProgressObject(parsed, opts = {}) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Additive-optional: iterations_remaining is trekexecute's recovery/retry budget signal.
|
||||||
|
// When present it must be a non-negative integer; absence is valid (backward-compat).
|
||||||
|
if (parsed.iterations_remaining !== undefined &&
|
||||||
|
!(Number.isInteger(parsed.iterations_remaining) && parsed.iterations_remaining >= 0)) {
|
||||||
|
errors.push(issue(
|
||||||
|
'PROGRESS_ITERATIONS_REMAINING_INVALID',
|
||||||
|
`iterations_remaining=${parsed.iterations_remaining} must be a non-negative integer`,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
if (parsed.steps && typeof parsed.steps === 'object') {
|
if (parsed.steps && typeof parsed.steps === 'object') {
|
||||||
const stepKeys = Object.keys(parsed.steps);
|
const stepKeys = Object.keys(parsed.steps);
|
||||||
if (typeof parsed.total_steps === 'number' && stepKeys.length !== parsed.total_steps) {
|
if (typeof parsed.total_steps === 'number' && stepKeys.length !== parsed.total_steps) {
|
||||||
|
|
|
||||||
4
package-lock.json
generated
4
package-lock.json
generated
|
|
@ -1,12 +1,12 @@
|
||||||
{
|
{
|
||||||
"name": "voyage",
|
"name": "voyage",
|
||||||
"version": "5.5.0",
|
"version": "5.9.1",
|
||||||
"lockfileVersion": 3,
|
"lockfileVersion": 3,
|
||||||
"requires": true,
|
"requires": true,
|
||||||
"packages": {
|
"packages": {
|
||||||
"": {
|
"": {
|
||||||
"name": "voyage",
|
"name": "voyage",
|
||||||
"version": "5.5.0",
|
"version": "5.9.1",
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
"engines": {
|
"engines": {
|
||||||
"node": ">=18"
|
"node": ">=18"
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
{
|
{
|
||||||
"name": "voyage",
|
"name": "voyage",
|
||||||
"version": "5.5.0",
|
"version": "5.9.1",
|
||||||
"description": "Voyage — brief, research, plan, execute, review, continue. Contract-driven Claude Code pipeline. /trekbrief, /trekplan, and /trekreview each end by building a self-contained operator-annotation HTML (scripts/annotate.mjs, modelled on claude-code-100x): select text or click any heading/paragraph/list-item, pick intent (Fiks/Endre/Spørsmål), write comment, copy structured prompt, paste back, Claude revises the .md.",
|
"description": "Voyage — brief, research, plan, execute, review, continue. Contract-driven Claude Code pipeline. /trekbrief, /trekplan, and /trekreview each end by building a self-contained operator-annotation HTML (scripts/annotate.mjs, modelled on claude-code-100x): select text or click any heading/paragraph/list-item, pick intent (Fiks/Endre/Spørsmål), write comment, copy structured prompt, paste back, Claude revises the .md.",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"engines": {
|
"engines": {
|
||||||
|
|
|
||||||
|
|
@ -141,7 +141,7 @@ introduced. This section bridges sessions — it's the "baton" in a relay race.}
|
||||||
- **Master plan:** `{plan file path}`
|
- **Master plan:** `{plan file path}`
|
||||||
- **Steps from plan:** {step N}–{step M}
|
- **Steps from plan:** {step N}–{step M}
|
||||||
- **Estimated complexity:** {low | medium | high}
|
- **Estimated complexity:** {low | medium | high}
|
||||||
- **Model recommendation:** {opus | sonnet} — {rationale}
|
- **Model recommendation:** {opus | sonnet | fable} — {rationale}
|
||||||
|
|
||||||
## Recovery Metadata
|
## Recovery Metadata
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -17,8 +17,9 @@ source: {interview | manual}
|
||||||
# plan polishing a wrong premise after a rejected iteration).
|
# plan polishing a wrong premise after a rejected iteration).
|
||||||
framing: {preserve | refine | replace | new-direction}
|
framing: {preserve | refine | replace | new-direction}
|
||||||
# v5.1 — per-phase effort + model signal (Phase 3.5).
|
# v5.1 — per-phase effort + model signal (Phase 3.5).
|
||||||
# `effort` ∈ {low, standard, high}. Omit `model:` for `standard` so composition
|
# `effort` ∈ {low, standard, high}; `model` ∈ {sonnet, opus, fable} (v5.9).
|
||||||
# falls through to profile resolver. Force-stop alternative is the commented
|
# Omit `model:` for `standard` so composition falls through to profile
|
||||||
|
# resolver. Force-stop alternative is the commented
|
||||||
# `phase_signals_partial: true` below (mutually exclusive with `phase_signals`).
|
# `phase_signals_partial: true` below (mutually exclusive with `phase_signals`).
|
||||||
phase_signals:
|
phase_signals:
|
||||||
- phase: research
|
- phase: research
|
||||||
|
|
|
||||||
|
|
@ -16,6 +16,7 @@ import { dirname, join } from 'node:path';
|
||||||
import { fileURLToPath } from 'node:url';
|
import { fileURLToPath } from 'node:url';
|
||||||
import { resolvePhaseSignal } from '../../lib/profiles/phase-signal-resolver.mjs';
|
import { resolvePhaseSignal } from '../../lib/profiles/phase-signal-resolver.mjs';
|
||||||
import { validateBriefContent, PHASE_SIGNAL_PHASES, EFFORT_LEVELS } from '../../lib/validators/brief-validator.mjs';
|
import { validateBriefContent, PHASE_SIGNAL_PHASES, EFFORT_LEVELS } from '../../lib/validators/brief-validator.mjs';
|
||||||
|
import { BASE_ALLOWED_MODELS } from '../../lib/validators/profile-validator.mjs';
|
||||||
import { parseDocument } from '../../lib/util/frontmatter.mjs';
|
import { parseDocument } from '../../lib/util/frontmatter.mjs';
|
||||||
|
|
||||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
|
|
@ -84,8 +85,8 @@ test('trekbrief — SC1: each of 4 phases has both effort AND model on full-sign
|
||||||
assert.ok(EFFORT_LEVELS.includes(r.effort),
|
assert.ok(EFFORT_LEVELS.includes(r.effort),
|
||||||
`phase=${phase}: effort "${r.effort}" not in EFFORT_LEVELS`);
|
`phase=${phase}: effort "${r.effort}" not in EFFORT_LEVELS`);
|
||||||
if ('model' in r) {
|
if ('model' in r) {
|
||||||
assert.ok(['sonnet', 'opus'].includes(r.model),
|
assert.ok(BASE_ALLOWED_MODELS.includes(r.model),
|
||||||
`phase=${phase}: model "${r.model}" not in [sonnet, opus]`);
|
`phase=${phase}: model "${r.model}" not in [${BASE_ALLOWED_MODELS.join(', ')}]`);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
@ -99,6 +100,19 @@ test('trekbrief — SC1: missing phase_signals + brief_version 2.1 triggers BRIE
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// --- v5.9 — fable tier option in the Phase 3.5 loop ---
|
||||||
|
|
||||||
|
test('trekbrief — v5.9 Phase 3.5 canonical mapping contains the fable row and offers 4 options', () => {
|
||||||
|
const text = read();
|
||||||
|
const startIdx = text.indexOf('## Phase 3.5');
|
||||||
|
assert.ok(startIdx >= 0, 'Phase 3.5 not found');
|
||||||
|
const section = text.slice(startIdx, text.indexOf('## Phase 4', startIdx));
|
||||||
|
assert.ok(section.includes('fable → {effort: high, model: fable}'),
|
||||||
|
'Phase 3.5 canonical mapping must contain the fable tier row');
|
||||||
|
assert.ok(section.includes('with 4 options'),
|
||||||
|
'Phase 3.5 loop must offer 4 options (AskUserQuestion maxItems: 4)');
|
||||||
|
});
|
||||||
|
|
||||||
// --- v5.5 — framing enforcement + TL;DR + memory-alignment prose-pins ---
|
// --- v5.5 — framing enforcement + TL;DR + memory-alignment prose-pins ---
|
||||||
|
|
||||||
test('trekbrief — v5.5 Phase 2.5 framing declaration heading present', () => {
|
test('trekbrief — v5.5 Phase 2.5 framing declaration heading present', () => {
|
||||||
|
|
|
||||||
127
tests/commands/trekendsession.test.mjs
Normal file
127
tests/commands/trekendsession.test.mjs
Normal file
|
|
@ -0,0 +1,127 @@
|
||||||
|
// tests/commands/trekendsession.test.mjs
|
||||||
|
// Regression tests for /trekendsession (commands/trekendsession.md).
|
||||||
|
//
|
||||||
|
// Bug (2026-07-03): two of the three !`...` eager-exec blocks contained
|
||||||
|
// unresolved placeholders (<project-dir> etc.). The harness executes
|
||||||
|
// eager-exec blocks at command LOAD time, so zsh parsed <project-dir> as
|
||||||
|
// input redirection and the command aborted before the model saw a single
|
||||||
|
// instruction. Eager-exec is only valid for self-contained commands.
|
||||||
|
//
|
||||||
|
// Pattern D (markdown structure) — assertions against command prose.
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { readFileSync, readdirSync } from 'node:fs';
|
||||||
|
import { dirname, join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
|
||||||
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const ROOT = join(HERE, '..', '..');
|
||||||
|
const COMMANDS_DIR = join(ROOT, 'commands');
|
||||||
|
const COMMAND_FILE = join(COMMANDS_DIR, 'trekendsession.md');
|
||||||
|
|
||||||
|
function readCommand() {
|
||||||
|
return readFileSync(COMMAND_FILE, 'utf8');
|
||||||
|
}
|
||||||
|
|
||||||
|
function extractPhase(commandText, phaseHeader) {
|
||||||
|
const startIdx = commandText.indexOf(phaseHeader);
|
||||||
|
if (startIdx === -1) return '';
|
||||||
|
const rest = commandText.slice(startIdx);
|
||||||
|
const nextPhase = rest.search(/\n## (?:Phase |Hard )/);
|
||||||
|
if (nextPhase === -1) return rest;
|
||||||
|
return rest.slice(0, nextPhase);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract all eager-exec blocks (!`...`) from a command/skill file,
|
||||||
|
// including multi-line blocks. Returns [{ content, line }].
|
||||||
|
function extractEagerBlocks(text) {
|
||||||
|
const blocks = [];
|
||||||
|
const re = /!`([^`]+)`/g;
|
||||||
|
let m;
|
||||||
|
while ((m = re.exec(text)) !== null) {
|
||||||
|
const line = text.slice(0, m.index).split('\n').length;
|
||||||
|
blocks.push({ content: m[1], line });
|
||||||
|
}
|
||||||
|
return blocks;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------
|
||||||
|
// Marketplace-wide regression guard: eager-exec blocks must be
|
||||||
|
// self-contained. An unresolved placeholder (<angle> or {curly}) in an
|
||||||
|
// eager block is executed verbatim by the shell at load time — <x> is
|
||||||
|
// parsed as input redirection and aborts the whole command load.
|
||||||
|
// ---------------------------------------------------------------
|
||||||
|
|
||||||
|
test('eager-exec guard — no !`-block in commands/ contains an unresolved placeholder', () => {
|
||||||
|
const offenders = [];
|
||||||
|
for (const file of readdirSync(COMMANDS_DIR).filter((f) => f.endsWith('.md'))) {
|
||||||
|
const text = readFileSync(join(COMMANDS_DIR, file), 'utf8');
|
||||||
|
for (const { content, line } of extractEagerBlocks(text)) {
|
||||||
|
// Placeholder conventions: <angle-word> or {curly_word}. Curly must
|
||||||
|
// contain a separator (- or _) so JS destructuring like {join} in a
|
||||||
|
// legitimate self-contained script does not false-positive; angle
|
||||||
|
// placeholders are unambiguous (shell would parse them as redirects).
|
||||||
|
if (/<[a-z][a-z0-9_-]*>/.test(content) || /\{[a-z][a-z0-9]*([_-][a-z0-9]+)+\}/.test(content)) {
|
||||||
|
offenders.push(`${file}:${line}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert.deepEqual(
|
||||||
|
offenders,
|
||||||
|
[],
|
||||||
|
`eager-exec !\`-blocks run at command LOAD time and must be self-contained; ` +
|
||||||
|
`placeholder found in: ${offenders.join(', ')}`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------
|
||||||
|
// trekendsession-specific: exactly one eager block (Phase 1 project
|
||||||
|
// discovery — self-contained, legitimate); Phases 3 and 4 are runtime
|
||||||
|
// Bash-tool commands with model-substituted values, never eager.
|
||||||
|
// ---------------------------------------------------------------
|
||||||
|
|
||||||
|
test('trekendsession — exactly one eager-exec block remains (Phase 1 discovery)', () => {
|
||||||
|
const cmd = readCommand();
|
||||||
|
const blocks = extractEagerBlocks(cmd);
|
||||||
|
assert.equal(
|
||||||
|
blocks.length,
|
||||||
|
1,
|
||||||
|
`expected exactly 1 eager-exec block (Phase 1 discovery), got ${blocks.length} at line(s) ${blocks.map((b) => b.line).join(', ')}`,
|
||||||
|
);
|
||||||
|
assert.match(
|
||||||
|
blocks[0].content,
|
||||||
|
/readdirSync\(root\)/,
|
||||||
|
'the surviving eager block must be the self-contained Phase 1 discovery script',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('trekendsession Phase 3 — atomic-write block is runtime Bash (no eager prefix) with plugin-root import', () => {
|
||||||
|
const phase3 = extractPhase(readCommand(), '## Phase 3 ');
|
||||||
|
assert.doesNotMatch(phase3, /!`/, 'Phase 3 must not use eager-exec — values exist only at runtime');
|
||||||
|
assert.match(
|
||||||
|
phase3,
|
||||||
|
/\$\{CLAUDE_PLUGIN_ROOT\}\/lib\/util\/atomic-write\.mjs/,
|
||||||
|
'Phase 3 import must use the absolute ${CLAUDE_PLUGIN_ROOT} path — cwd is the user repo, not the plugin root',
|
||||||
|
);
|
||||||
|
assert.doesNotMatch(
|
||||||
|
phase3,
|
||||||
|
/['"]\.\/lib\/util\/atomic-write\.mjs['"]/,
|
||||||
|
'Phase 3 must not import atomic-write.mjs via a cwd-relative path',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('trekendsession Phase 4 — validator call is runtime Bash (no eager prefix) with plugin-root path', () => {
|
||||||
|
const phase4 = extractPhase(readCommand(), '## Phase 4 ');
|
||||||
|
assert.doesNotMatch(phase4, /!`/, 'Phase 4 must not use eager-exec — the state-file path exists only at runtime');
|
||||||
|
assert.match(
|
||||||
|
phase4,
|
||||||
|
/\$\{CLAUDE_PLUGIN_ROOT\}\/lib\/validators\/session-state-validator\.mjs/,
|
||||||
|
'Phase 4 validator path must use the absolute ${CLAUDE_PLUGIN_ROOT} convention',
|
||||||
|
);
|
||||||
|
assert.doesNotMatch(
|
||||||
|
phase4,
|
||||||
|
/<[a-z][a-z0-9_-]*>/,
|
||||||
|
'Phase 4 must not use <angle> placeholders in commands — zsh parses <x> as input redirection',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
@ -73,3 +73,78 @@ test('trekexecute — SC7: brief_version 2.1 + no phase_signals + no partial →
|
||||||
`sequencing gate must fire; errors=${JSON.stringify(r.errors)}`,
|
`sequencing gate must fire; errors=${JSON.stringify(r.errors)}`,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// --- S38 loop-discipline hardening: completion gate (Step 2) ---
|
||||||
|
|
||||||
|
test('trekexecute — machine-verifiable completion gate + stop-signal contract literals present (SC a)', () => {
|
||||||
|
const text = read();
|
||||||
|
assert.ok(text.includes('machine-verifiable completion gate'),
|
||||||
|
'completion-gate literal must be grep-able');
|
||||||
|
assert.ok(text.includes('stop-signal contract'),
|
||||||
|
'stop-signal contract literal must be grep-able');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('trekexecute — completion gate anchors result:completed to Phase 7.5 audit + DONE-after-check (SC a)', () => {
|
||||||
|
const text = read();
|
||||||
|
const gateIdx = text.indexOf('machine-verifiable completion gate');
|
||||||
|
assert.ok(gateIdx >= 0, 'gate literal missing');
|
||||||
|
const section = text.slice(gateIdx, gateIdx + 1400);
|
||||||
|
assert.match(section, /Phase 7\.5/, 'gate must name Phase 7.5 as the objective predicate');
|
||||||
|
assert.match(section, /DONE/, 'gate must reference the DONE stop-signal token');
|
||||||
|
assert.match(section, /emitted AFTER/, 'gate must require DONE emitted AFTER the audit ran');
|
||||||
|
});
|
||||||
|
|
||||||
|
// --- S38 loop-discipline hardening: caps + global recovery budget (Step 3) ---
|
||||||
|
|
||||||
|
test('trekexecute — global recovery/retry budget TREKEXECUTE_MAX_RECOVERY_ITERATIONS default 25 (SC b)', () => {
|
||||||
|
const text = read();
|
||||||
|
assert.ok(text.includes('TREKEXECUTE_MAX_RECOVERY_ITERATIONS'),
|
||||||
|
'global recovery/retry budget constant must be documented');
|
||||||
|
assert.match(
|
||||||
|
text,
|
||||||
|
/TREKEXECUTE_MAX_RECOVERY_ITERATIONS[^\n]{0,40}\b25\b|default[^\n]{0,20}\b25\b[^\n]{0,40}aggregate/i,
|
||||||
|
'global budget must document numeric default 25 adjacent to the constant',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('trekexecute — every recovery/retry loop bounded + 3-axis cap hierarchy (SC b)', () => {
|
||||||
|
const text = read();
|
||||||
|
assert.match(text, /maximum 2 retries|Retry cap = 3 attempts/, 'per-step retry cap must be explicit');
|
||||||
|
assert.match(text, /recovery_depth < 2/, 'recovery-dispatch cap must be explicit');
|
||||||
|
const capIdx = text.indexOf('Iteration caps');
|
||||||
|
assert.ok(capIdx >= 0, 'cap-hierarchy section ("Iteration caps") must exist');
|
||||||
|
const section = text.slice(capIdx, capIdx + 1500);
|
||||||
|
assert.match(section, /attempts/, 'hierarchy must name per-step attempts axis');
|
||||||
|
assert.match(section, /recovery_depth/, 'hierarchy must name per-session recovery_depth axis');
|
||||||
|
assert.match(section, /TREKEXECUTE_MAX_RECOVERY_ITERATIONS/, 'hierarchy must name the global budget axis');
|
||||||
|
});
|
||||||
|
|
||||||
|
// --- S38 loop-discipline hardening: iterations_remaining budget signal (Step 4) ---
|
||||||
|
|
||||||
|
test('trekexecute — iterations_remaining surfaced in progress schema + summary JSON, no (if wired) (SC c)', () => {
|
||||||
|
const text = read();
|
||||||
|
assert.ok(text.includes('iterations_remaining'), 'iterations_remaining must be present (SC c)');
|
||||||
|
assert.ok(!/iterations_remaining[^\n]*\(if wired\)/.test(text),
|
||||||
|
'no "(if wired)" conditional — field present in the spec at minimum');
|
||||||
|
const schemaIdx = text.indexOf('### Progress file schema');
|
||||||
|
assert.ok(schemaIdx >= 0, 'progress schema section missing');
|
||||||
|
assert.match(text.slice(schemaIdx, schemaIdx + 1400), /iterations_remaining/,
|
||||||
|
'progress schema must include iterations_remaining');
|
||||||
|
const sumIdx = text.indexOf('"trekexecute_summary": {');
|
||||||
|
assert.ok(sumIdx >= 0, 'summary JSON block missing');
|
||||||
|
assert.match(text.slice(sumIdx, sumIdx + 900), /iterations_remaining/,
|
||||||
|
'summary JSON must include iterations_remaining (observable)');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('trekexecute — iterations_remaining decrement + backfill + gate cross-check documented (SC c)', () => {
|
||||||
|
const text = read();
|
||||||
|
assert.match(text, /decrement[^.\n]*iteration|every recovery\/retry iteration/i,
|
||||||
|
'decrement-per-iteration rule must be documented');
|
||||||
|
assert.match(text, /backfill|seed (it )?to (the )?cap/i,
|
||||||
|
'backfill-on-absent (legacy resume) rule must be documented');
|
||||||
|
const gateIdx = text.indexOf('machine-verifiable completion gate');
|
||||||
|
const gate = text.slice(gateIdx, gateIdx + 2200);
|
||||||
|
assert.match(gate, /iterations_remaining/, 'gate must cross-check iterations_remaining');
|
||||||
|
assert.match(gate, /recovery_depth/, 'gate cross-check must reconcile against recovery_depth');
|
||||||
|
assert.match(gate, /attempts/, 'gate cross-check must reconcile against attempts');
|
||||||
|
});
|
||||||
|
|
|
||||||
|
|
@ -71,3 +71,22 @@ test('trekplan — SC7: brief_version 2.1 + no phase_signals + no partial → BR
|
||||||
`sequencing gate must fire; errors=${JSON.stringify(r.errors)}`,
|
`sequencing gate must fire; errors=${JSON.stringify(r.errors)}`,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// --- S31 / V15: export trim — drop pr|issue|markdown, keep headless as a --decompose alias ---
|
||||||
|
|
||||||
|
test('trekplan — V15: pr/issue/markdown export variants are removed', () => {
|
||||||
|
const text = read();
|
||||||
|
// No remaining `--export pr` usage/example anywhere in the command.
|
||||||
|
assert.equal(/--export\s+pr\b/.test(text), false, '`--export pr` must be gone');
|
||||||
|
// The per-format subsections of the old Export phase must be deleted.
|
||||||
|
assert.ok(!text.includes('Format: `pr`'), 'pr export-format section must be gone');
|
||||||
|
assert.ok(!text.includes('Format: `issue`'), 'issue export-format section must be gone');
|
||||||
|
assert.ok(!text.includes('Format: `markdown`'), 'markdown export-format section must be gone');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('trekplan — V15: --export headless survives as a --decompose alias', () => {
|
||||||
|
const text = read();
|
||||||
|
assert.match(text, /--export\s+headless/, '`--export headless` must remain documented');
|
||||||
|
assert.match(text, /alias for [`*]*--decompose/i,
|
||||||
|
'`--export headless` must be labeled an alias for --decompose');
|
||||||
|
});
|
||||||
|
|
|
||||||
35
tests/commands/trekresearch-engine.test.mjs
Normal file
35
tests/commands/trekresearch-engine.test.mjs
Normal file
|
|
@ -0,0 +1,35 @@
|
||||||
|
// tests/commands/trekresearch-engine.test.mjs
|
||||||
|
// Step 1 (deep-research-engine): pin the contract the `--engine deep-research`
|
||||||
|
// adapter must hit. The adapted in-context `/deep-research` report, reduced into
|
||||||
|
// the research-brief schema, must pass research-validator under the strict
|
||||||
|
// default; and a brief missing a required section must fail. This is the one
|
||||||
|
// genuinely automatable slice of SC2 (schema, not provenance).
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { readFileSync } from 'node:fs';
|
||||||
|
import { dirname, join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { validateResearchContent } from '../../lib/validators/research-validator.mjs';
|
||||||
|
|
||||||
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const ROOT = join(HERE, '..', '..');
|
||||||
|
const FIXTURE = join(ROOT, 'tests', 'fixtures', 'research-deep-research-adapted.md');
|
||||||
|
|
||||||
|
test('deep-research adapter output contract — valid brief passes, missing section fails', () => {
|
||||||
|
const text = readFileSync(FIXTURE, 'utf-8');
|
||||||
|
|
||||||
|
// (a) positive: the adapter's target output passes the validator (default = strict).
|
||||||
|
const okResult = validateResearchContent(text);
|
||||||
|
assert.equal(okResult.valid, true, JSON.stringify(okResult.errors));
|
||||||
|
|
||||||
|
// (b) negative: stripping a required section makes it fail with RESEARCH_MISSING_SECTION,
|
||||||
|
// giving the contract teeth (a fixture that always passes proves nothing).
|
||||||
|
const mutated = text.replace('## Dimensions', '## Removed');
|
||||||
|
const badResult = validateResearchContent(mutated);
|
||||||
|
assert.equal(badResult.valid, false);
|
||||||
|
assert.ok(
|
||||||
|
badResult.errors.find(e => e.code === 'RESEARCH_MISSING_SECTION'),
|
||||||
|
'expected RESEARCH_MISSING_SECTION; got ' + JSON.stringify(badResult.errors),
|
||||||
|
);
|
||||||
|
});
|
||||||
44
tests/fixtures/bakeoff-rich/gold.json
vendored
Normal file
44
tests/fixtures/bakeoff-rich/gold.json
vendored
Normal file
|
|
@ -0,0 +1,44 @@
|
||||||
|
{
|
||||||
|
"schema": "voyage-eval-gold/1",
|
||||||
|
"source": "tests/fixtures/bakeoff-rich/README.md",
|
||||||
|
"description": "Golden corpus for SKAL-1·4a — the 5 brief-traceable seeded findings of the bakeoff-rich JWT-auth fixture, machine-readable. expected_verdict is the review-coordinator Pass-4 outcome (BLOCKER >= 1 -> BLOCK).",
|
||||||
|
"expected_verdict": "BLOCK",
|
||||||
|
"findings": [
|
||||||
|
{
|
||||||
|
"file": "lib/handlers/login.mjs",
|
||||||
|
"line": 17,
|
||||||
|
"rule_key": "UNIMPLEMENTED_CRITERION",
|
||||||
|
"severity": "BLOCKER",
|
||||||
|
"owner_reviewer": "conformance"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "lib/auth/jwt.mjs",
|
||||||
|
"line": 19,
|
||||||
|
"rule_key": "SECURITY_INJECTION",
|
||||||
|
"severity": "BLOCKER",
|
||||||
|
"owner_reviewer": "correctness",
|
||||||
|
"dual_flaggable": "NON_GOAL_VIOLATED"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "lib/auth/refresh.mjs",
|
||||||
|
"line": 0,
|
||||||
|
"rule_key": "MISSING_TEST",
|
||||||
|
"severity": "MAJOR",
|
||||||
|
"owner_reviewer": "correctness"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "lib/handlers/login.mjs",
|
||||||
|
"line": 13,
|
||||||
|
"rule_key": "PLAN_EXECUTE_DRIFT",
|
||||||
|
"severity": "MAJOR",
|
||||||
|
"owner_reviewer": "conformance"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "lib/auth/refresh.mjs",
|
||||||
|
"line": 10,
|
||||||
|
"rule_key": "MISSING_ERROR_HANDLING",
|
||||||
|
"severity": "MINOR",
|
||||||
|
"owner_reviewer": "correctness"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
17
tests/fixtures/bakeoff-rich/runs/run-perfect.json
vendored
Normal file
17
tests/fixtures/bakeoff-rich/runs/run-perfect.json
vendored
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"reviewer": "brief-conformance-reviewer",
|
||||||
|
"findings": [
|
||||||
|
{ "severity": "BLOCKER", "rule_key": "UNIMPLEMENTED_CRITERION", "file": "lib/handlers/login.mjs", "line": 17 },
|
||||||
|
{ "severity": "MAJOR", "rule_key": "PLAN_EXECUTE_DRIFT", "file": "lib/handlers/login.mjs", "line": 13 }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"reviewer": "code-correctness-reviewer",
|
||||||
|
"findings": [
|
||||||
|
{ "severity": "BLOCKER", "rule_key": "SECURITY_INJECTION", "file": "lib/auth/jwt.mjs", "line": 19 },
|
||||||
|
{ "severity": "MAJOR", "rule_key": "MISSING_TEST", "file": "lib/auth/refresh.mjs", "line": 0 },
|
||||||
|
{ "severity": "MINOR", "rule_key": "MISSING_ERROR_HANDLING", "file": "lib/auth/refresh.mjs", "line": 10 }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
45
tests/fixtures/brief-effort-fable.md
vendored
Normal file
45
tests/fixtures/brief-effort-fable.md
vendored
Normal file
|
|
@ -0,0 +1,45 @@
|
||||||
|
---
|
||||||
|
type: trekbrief
|
||||||
|
brief_version: "2.1"
|
||||||
|
created: 2026-07-02
|
||||||
|
task: "Fixture: high-effort all phases on fable (v5.9 allowlist test)"
|
||||||
|
slug: brief-effort-fable
|
||||||
|
project_dir: .claude/projects/2026-07-02-brief-effort-fable/
|
||||||
|
research_topics: 0
|
||||||
|
research_status: complete
|
||||||
|
auto_research: false
|
||||||
|
interview_turns: 4
|
||||||
|
source: fixture
|
||||||
|
phase_signals:
|
||||||
|
- phase: research
|
||||||
|
effort: high
|
||||||
|
model: fable
|
||||||
|
- phase: plan
|
||||||
|
effort: high
|
||||||
|
model: fable
|
||||||
|
- phase: execute
|
||||||
|
effort: high
|
||||||
|
model: fable
|
||||||
|
- phase: review
|
||||||
|
effort: high
|
||||||
|
model: fable
|
||||||
|
---
|
||||||
|
|
||||||
|
# Task: High-effort fable fixture
|
||||||
|
|
||||||
|
## Intent
|
||||||
|
|
||||||
|
Test fixture for the v5.9 fable model tier — all 4 phases at the
|
||||||
|
high effort tier with explicit fable model overrides. Mirrors
|
||||||
|
brief-effort-high.md with `model: opus` replaced by `model: fable`.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Resolver returns `{effort: 'high', model: 'fable'}` for each of the 4
|
||||||
|
PHASE_SIGNAL_PHASES.
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
- Validator passes with no BRIEF_INVALID_MODEL.
|
||||||
|
- resolvePhaseSignal(fm, phase).effort === 'high' for all 4 phases.
|
||||||
|
- resolvePhaseSignal(fm, phase).model === 'fable' for all 4 phases.
|
||||||
5
tests/fixtures/jsonl-schemas.md
vendored
5
tests/fixtures/jsonl-schemas.md
vendored
|
|
@ -27,6 +27,7 @@
|
||||||
| trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
|
| trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
|
||||||
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | commands/trekreview.md (orchestrator-emit Phase 8) | trekreview.md:255 | profile, phase_models, profile_source | none |
|
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | commands/trekreview.md (orchestrator-emit Phase 8) | trekreview.md:255 | profile, phase_models, profile_source | none |
|
||||||
| trekcontinue-stats | ts, project, next_session_label, status | commands/trekcontinue.md (orchestrator-emit Phase 5) | trekcontinue.md:289 | profile, profile_source | none |
|
| trekcontinue-stats | ts, project, next_session_label, status | commands/trekcontinue.md (orchestrator-emit Phase 5) | trekcontinue.md:289 | profile, profile_source | none |
|
||||||
|
| token-usage | ts, session_id, scope, model, tokens_input, tokens_output, tokens_cache_creation, tokens_cache_read, cost_usd, is_estimate, price_table_version | lib/stats/token-usage.mjs `captureTokenUsage` (via hooks/scripts/otel-export.mjs Stop hook, opt-in `VOYAGE_TOKEN_METER`) | token-usage.mjs:buildRecord | n/a (SKAL-2 schema) | session_id (stripped at export) |
|
||||||
|
|
||||||
## Field-allowlist input for Step 11
|
## Field-allowlist input for Step 11
|
||||||
|
|
||||||
|
|
@ -46,7 +47,9 @@ critic_verdict, guardian_verdict, outcome, plan_type, result,
|
||||||
steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step,
|
steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step,
|
||||||
verdict, reviewed_files_count, duration_ms, status, next_session_label,
|
verdict, reviewed_files_count, duration_ms, status, next_session_label,
|
||||||
event, known_event, success, scope,
|
event, known_event, success, scope,
|
||||||
profile, profile_source, parallel_agents, external_research_enabled
|
profile, profile_source, parallel_agents, external_research_enabled,
|
||||||
|
model, tokens_input, tokens_output, tokens_cache_creation, tokens_cache_read,
|
||||||
|
cost_usd, is_estimate, price_table_version
|
||||||
```
|
```
|
||||||
|
|
||||||
**EXPORT_DENYLIST** (PII or high-cardinality, never export):
|
**EXPORT_DENYLIST** (PII or high-cardinality, never export):
|
||||||
|
|
|
||||||
49
tests/fixtures/research-deep-research-adapted.md
vendored
Normal file
49
tests/fixtures/research-deep-research-adapted.md
vendored
Normal file
|
|
@ -0,0 +1,49 @@
|
||||||
|
---
|
||||||
|
type: trekresearch-brief
|
||||||
|
created: 2026-06-30
|
||||||
|
question: "Should /trekresearch delegate its external phase to the built-in /deep-research workflow?"
|
||||||
|
confidence: 0.8
|
||||||
|
dimensions: 2
|
||||||
|
mcp_servers_used: []
|
||||||
|
local_agents_used: []
|
||||||
|
external_agents_used:
|
||||||
|
- deep-research
|
||||||
|
---
|
||||||
|
|
||||||
|
# Deep-research engine adapter output
|
||||||
|
|
||||||
|
> Fixture: a `/deep-research` in-context report reduced into the research-brief
|
||||||
|
> schema by the `--engine deep-research` adapter (Step 4). Models the target the
|
||||||
|
> adapter must hit; not real engine output.
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
Delegating the external phase to the built-in `/deep-research` workflow is a
|
||||||
|
viable opt-in engine that supplies fan-out and cited claim-verification for free.
|
||||||
|
Confidence is medium-high on the mechanism but lower on availability, because the
|
||||||
|
workflow exposes no positive "is-enabled" probe. The load-bearing caveat is
|
||||||
|
provenance: structural validity does not certify that the cited URLs are real, so
|
||||||
|
a swarm fallback plus a human URL spot-check stay mandatory.
|
||||||
|
|
||||||
|
## Dimensions
|
||||||
|
|
||||||
|
### Engine mechanism -- Confidence: high
|
||||||
|
|
||||||
|
**External findings:**
|
||||||
|
- `/deep-research` is a built-in dynamic workflow reachable only by prose instruction, with no programmatic API (https://code.claude.com/docs/workflows).
|
||||||
|
- Its report lands in-context with no on-disk artifact, so the adapter must transform what is already in the turn (https://code.claude.com/docs/commands).
|
||||||
|
|
||||||
|
### Fallback ergonomics -- Confidence: high
|
||||||
|
|
||||||
|
**External findings:**
|
||||||
|
- There is no positive availability probe; only `disableWorkflows` / `CLAUDE_CODE_DISABLE_WORKFLOWS` off-switches and a 2.1.154 version floor are documented (https://code.claude.com/docs/skills).
|
||||||
|
- Disabled-workflow behavior under `claude -p` is undocumented, so the post-hoc presence check must be robust to every failure manifestation (https://github.com/anthropics/claude-code/issues/52272).
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
| # | Source | Type | Quality | Used in |
|
||||||
|
|---|--------|------|---------|---------|
|
||||||
|
| 1 | https://code.claude.com/docs/workflows | official | high | Engine mechanism |
|
||||||
|
| 2 | https://code.claude.com/docs/commands | official | high | Engine mechanism |
|
||||||
|
| 3 | https://code.claude.com/docs/skills | official | high | Fallback ergonomics |
|
||||||
|
| 4 | https://github.com/anthropics/claude-code/issues/52272 | community | medium | Fallback ergonomics |
|
||||||
8
tests/fixtures/token-usage/sample-transcript.jsonl
vendored
Normal file
8
tests/fixtures/token-usage/sample-transcript.jsonl
vendored
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
{"type":"user","message":{"role":"user","content":"hi"}}
|
||||||
|
{"type":"assistant","isSidechain":false,"requestId":"req_A","message":{"model":"claude-opus-4-8","usage":{"input_tokens":1,"output_tokens":1,"cache_creation_input_tokens":0,"cache_read_input_tokens":0}}}
|
||||||
|
{"type":"assistant","isSidechain":false,"requestId":"req_A","message":{"model":"claude-opus-4-8","usage":{"input_tokens":1000,"output_tokens":200,"cache_creation_input_tokens":400,"cache_read_input_tokens":5000}}}
|
||||||
|
{"type":"assistant","isSidechain":false,"requestId":"req_B","message":{"model":"claude-opus-4-8","usage":{"input_tokens":2000,"output_tokens":300,"cache_creation_input_tokens":600,"cache_read_input_tokens":1000}}}
|
||||||
|
{"type":"assistant","isSidechain":true,"requestId":"req_SIDE","message":{"model":"claude-opus-4-8","usage":{"input_tokens":9999,"output_tokens":9999,"cache_creation_input_tokens":9999,"cache_read_input_tokens":9999}}}
|
||||||
|
{"type":"system","content":"some system note with no usage"}
|
||||||
|
this is not valid json and must be skipped
|
||||||
|
{"type":"assistant","isSidechain":false,"requestId":"req_B","message":{"model":"claude-opus-4-8","usage":{"input_tokens":2000,"output_tokens":300,"cache_creation_input_tokens":600,"cache_read_input_tokens":1000}}}
|
||||||
|
|
@ -11,6 +11,8 @@ import { readFileSync } from 'node:fs';
|
||||||
import { join, dirname } from 'node:path';
|
import { join, dirname } from 'node:path';
|
||||||
import { fileURLToPath } from 'node:url';
|
import { fileURLToPath } from 'node:url';
|
||||||
import { transformToPrometheus, normalizeMetricName } from '../../lib/exporters/textfile-format.mjs';
|
import { transformToPrometheus, normalizeMetricName } from '../../lib/exporters/textfile-format.mjs';
|
||||||
|
import { transformToOtlpJson } from '../../lib/exporters/otlp-format.mjs';
|
||||||
|
import { applyFieldAllowlist } from '../../lib/exporters/field-allowlist.mjs';
|
||||||
|
|
||||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||||
const FIXTURES = join(__dirname, '..', 'fixtures');
|
const FIXTURES = join(__dirname, '..', 'fixtures');
|
||||||
|
|
@ -74,6 +76,38 @@ test('normalizeMetricName: dots/dashes/spaces → underscore, lowercase, voyage_
|
||||||
assert.equal(normalizeMetricName('METRIC NAME'), 'voyage_metric_name');
|
assert.equal(normalizeMetricName('METRIC NAME'), 'voyage_metric_name');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('SC#6: allowlisted token-usage record → Prometheus + OTLP metrics; PII absent (end-to-end)', () => {
|
||||||
|
// Raw record as written to token-usage-stats.jsonl (carries PII for upsert keying).
|
||||||
|
const raw = {
|
||||||
|
session_id: 'x',
|
||||||
|
transcript_path: '/p',
|
||||||
|
cwd: '/c',
|
||||||
|
ts: '2026-06-26T12:00:00.000Z',
|
||||||
|
scope: 'main-context',
|
||||||
|
model: 'claude-opus-4-8',
|
||||||
|
tokens_input: 12345,
|
||||||
|
tokens_output: 6789,
|
||||||
|
cost_usd: 0.42,
|
||||||
|
is_estimate: false,
|
||||||
|
price_table_version: '2026-06-26',
|
||||||
|
};
|
||||||
|
const rec = applyFieldAllowlist(raw, 'token-usage');
|
||||||
|
|
||||||
|
// Prometheus: numeric fields auto-promote to voyage_token_usage_* metrics.
|
||||||
|
const prom = transformToPrometheus([rec]);
|
||||||
|
assert.match(prom, /voyage_token_usage_tokens_input\b/);
|
||||||
|
assert.match(prom, /voyage_token_usage_cost_usd\b/);
|
||||||
|
// CWE-212: PII must never appear in exporter output.
|
||||||
|
assert.ok(!prom.includes('session_id'), 'session_id leaked into Prometheus output');
|
||||||
|
assert.ok(!prom.includes('transcript_path'), 'transcript_path leaked into Prometheus output');
|
||||||
|
assert.ok(!prom.includes('/p'), 'transcript_path value leaked into Prometheus output');
|
||||||
|
|
||||||
|
// OTLP parity: dot-separated, dash-preserved naming (NOT the Prometheus underscore form).
|
||||||
|
const otlp = JSON.stringify(transformToOtlpJson([rec]));
|
||||||
|
assert.ok(otlp.includes('voyage.token-usage.tokens_input'), 'OTLP token metric missing');
|
||||||
|
assert.ok(!otlp.includes('transcript_path'), 'transcript_path leaked into OTLP output');
|
||||||
|
});
|
||||||
|
|
||||||
test('determinism: identical input produces identical output (sorted keys)', () => {
|
test('determinism: identical input produces identical output (sorted keys)', () => {
|
||||||
const records = loadJsonl('stats-sample.jsonl');
|
const records = loadJsonl('stats-sample.jsonl');
|
||||||
const out1 = transformToPrometheus(records);
|
const out1 = transformToPrometheus(records);
|
||||||
|
|
|
||||||
147
tests/hooks/otel-export-token-capture.test.mjs
Normal file
147
tests/hooks/otel-export-token-capture.test.mjs
Normal file
|
|
@ -0,0 +1,147 @@
|
||||||
|
// tests/hooks/otel-export-token-capture.test.mjs
|
||||||
|
// SKAL-2 Step 4 — opt-in main-context token/cost capture folded into the Stop
|
||||||
|
// hook (otel-export.mjs), gated by VOYAGE_TOKEN_METER.
|
||||||
|
//
|
||||||
|
// Fail-open contract: any error → exit 0, no token file, Stop never blocked.
|
||||||
|
// Pattern: tests/hooks/otel-export.test.mjs (setupDataDir + runHookWithEnv).
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { mkdtempSync, rmSync, existsSync, readFileSync, writeFileSync } from 'node:fs';
|
||||||
|
import { join, dirname } from 'node:path';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { runHookWithEnv } from '../helpers/hook-helper.mjs';
|
||||||
|
|
||||||
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const HOOK_PATH = join(HERE, '..', '..', 'hooks', 'scripts', 'otel-export.mjs');
|
||||||
|
const FIXTURE = join(HERE, '..', 'fixtures', 'token-usage', 'sample-transcript.jsonl');
|
||||||
|
const TOKEN_FILE = 'token-usage-stats.jsonl';
|
||||||
|
|
||||||
|
const mkDataDir = () => mkdtempSync(join(tmpdir(), 'voyage-token-capture-'));
|
||||||
|
const stdinFor = (extra = {}) => JSON.stringify({ transcript_path: FIXTURE, session_id: 'sess-1', ...extra });
|
||||||
|
const readToken = (dir) => readFileSync(join(dir, TOKEN_FILE), 'utf-8').trim().split('\n').filter(Boolean).map(JSON.parse);
|
||||||
|
|
||||||
|
test('VOYAGE_TOKEN_METER=1 → writes token-usage-stats.jsonl with expected totals + scope:main-context', async () => {
|
||||||
|
const dir = mkDataDir();
|
||||||
|
try {
|
||||||
|
const r = await runHookWithEnv(HOOK_PATH, stdinFor(), {
|
||||||
|
VOYAGE_TOKEN_METER: '1',
|
||||||
|
VOYAGE_EXPORT_MODE: 'off',
|
||||||
|
CLAUDE_PLUGIN_DATA: dir,
|
||||||
|
});
|
||||||
|
assert.equal(r.code, 0);
|
||||||
|
assert.equal(existsSync(join(dir, TOKEN_FILE)), true, 'token file must be written');
|
||||||
|
const recs = readToken(dir);
|
||||||
|
assert.equal(recs.length, 1);
|
||||||
|
const rec = recs[0];
|
||||||
|
// Fixture totals: dedup-by-requestId + sidechain-exclusion (verified in Step 1).
|
||||||
|
assert.equal(rec.tokens_input, 3000);
|
||||||
|
assert.equal(rec.tokens_output, 500);
|
||||||
|
assert.equal(rec.tokens_cache_creation, 1000);
|
||||||
|
assert.equal(rec.tokens_cache_read, 6000);
|
||||||
|
assert.equal(rec.scope, 'main-context');
|
||||||
|
assert.equal(rec.model, 'claude-opus-4-8');
|
||||||
|
assert.ok(Math.abs(rec.cost_usd - 0.03675) < 1e-9, `cost ${rec.cost_usd}`);
|
||||||
|
assert.equal(rec.session_id, 'sess-1');
|
||||||
|
} finally {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('UPSERT — running twice for the same session_id keeps ONE line (not two)', async () => {
|
||||||
|
const dir = mkDataDir();
|
||||||
|
try {
|
||||||
|
const env = { VOYAGE_TOKEN_METER: '1', VOYAGE_EXPORT_MODE: 'off', CLAUDE_PLUGIN_DATA: dir };
|
||||||
|
await runHookWithEnv(HOOK_PATH, stdinFor(), env);
|
||||||
|
const r2 = await runHookWithEnv(HOOK_PATH, stdinFor(), env);
|
||||||
|
assert.equal(r2.code, 0);
|
||||||
|
const recs = readToken(dir);
|
||||||
|
assert.equal(recs.length, 1, 'upsert must NOT append a second line for the same session');
|
||||||
|
assert.equal(recs[0].tokens_input, 3000);
|
||||||
|
} finally {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('UPSERT — distinct session_ids append (one line per session)', async () => {
|
||||||
|
const dir = mkDataDir();
|
||||||
|
try {
|
||||||
|
const env = { VOYAGE_TOKEN_METER: '1', VOYAGE_EXPORT_MODE: 'off', CLAUDE_PLUGIN_DATA: dir };
|
||||||
|
await runHookWithEnv(HOOK_PATH, stdinFor({ session_id: 'A' }), env);
|
||||||
|
await runHookWithEnv(HOOK_PATH, stdinFor({ session_id: 'B' }), env);
|
||||||
|
const recs = readToken(dir);
|
||||||
|
assert.equal(recs.length, 2);
|
||||||
|
assert.deepEqual(recs.map(r => r.session_id).sort(), ['A', 'B']);
|
||||||
|
} finally {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('OPT-IN — VOYAGE_TOKEN_METER unset → NO token file written', async () => {
|
||||||
|
const dir = mkDataDir();
|
||||||
|
try {
|
||||||
|
const r = await runHookWithEnv(HOOK_PATH, stdinFor(), {
|
||||||
|
VOYAGE_TOKEN_METER: '', // explicit empty mimics unset
|
||||||
|
VOYAGE_EXPORT_MODE: 'off',
|
||||||
|
CLAUDE_PLUGIN_DATA: dir,
|
||||||
|
});
|
||||||
|
assert.equal(r.code, 0);
|
||||||
|
assert.equal(existsSync(join(dir, TOKEN_FILE)), false, 'no token file without opt-in');
|
||||||
|
} finally {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('FAIL-OPEN — malformed stdin → exit 0, no token file, no throw', async () => {
|
||||||
|
const dir = mkDataDir();
|
||||||
|
try {
|
||||||
|
const r = await runHookWithEnv(HOOK_PATH, 'not valid json {{{', {
|
||||||
|
VOYAGE_TOKEN_METER: '1',
|
||||||
|
VOYAGE_EXPORT_MODE: 'off',
|
||||||
|
CLAUDE_PLUGIN_DATA: dir,
|
||||||
|
});
|
||||||
|
assert.equal(r.code, 0);
|
||||||
|
assert.equal(existsSync(join(dir, TOKEN_FILE)), false);
|
||||||
|
} finally {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('FAIL-OPEN — missing transcript file → exit 0, no token file', async () => {
|
||||||
|
const dir = mkDataDir();
|
||||||
|
try {
|
||||||
|
const r = await runHookWithEnv(
|
||||||
|
HOOK_PATH,
|
||||||
|
JSON.stringify({ transcript_path: '/no/such/transcript.jsonl', session_id: 'x' }),
|
||||||
|
{ VOYAGE_TOKEN_METER: '1', VOYAGE_EXPORT_MODE: 'off', CLAUDE_PLUGIN_DATA: dir },
|
||||||
|
);
|
||||||
|
assert.equal(r.code, 0);
|
||||||
|
assert.equal(existsSync(join(dir, TOKEN_FILE)), false);
|
||||||
|
} finally {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('capture coexists with export — VOYAGE_TOKEN_METER=1 + textfile mode writes BOTH', async () => {
|
||||||
|
const dir = mkDataDir();
|
||||||
|
try {
|
||||||
|
// Seed a stats file so the export path has something to write.
|
||||||
|
writeFileSync(join(dir, 'trekplan-stats.jsonl'),
|
||||||
|
JSON.stringify({ ts: '2026-06-26T08:00:00.000Z', slug: 't', mode: 'default', codebase_files: 10, profile: 'premium' }) + '\n');
|
||||||
|
const r = await runHookWithEnv(HOOK_PATH, stdinFor(), {
|
||||||
|
VOYAGE_TOKEN_METER: '1',
|
||||||
|
VOYAGE_EXPORT_MODE: 'textfile',
|
||||||
|
CLAUDE_PLUGIN_DATA: dir,
|
||||||
|
});
|
||||||
|
assert.equal(r.code, 0);
|
||||||
|
assert.equal(existsSync(join(dir, TOKEN_FILE)), true, 'token file written');
|
||||||
|
assert.equal(existsSync(join(dir, 'voyage.prom')), true, 'export still completed');
|
||||||
|
// The exported textfile should now also carry the token metric (STATS_FILES wiring).
|
||||||
|
const prom = readFileSync(join(dir, 'voyage.prom'), 'utf-8');
|
||||||
|
assert.match(prom, /voyage_token_usage_tokens_input/);
|
||||||
|
assert.ok(!prom.includes('sess-1'), 'session_id must not leak into export (CWE-212)');
|
||||||
|
} finally {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
@ -13,6 +13,7 @@ import {
|
||||||
applyFieldAllowlist,
|
applyFieldAllowlist,
|
||||||
POST_BASH_STATS_ALLOWED,
|
POST_BASH_STATS_ALLOWED,
|
||||||
EVENT_EMIT_PAYLOAD_ALLOWED,
|
EVENT_EMIT_PAYLOAD_ALLOWED,
|
||||||
|
TOKEN_USAGE_ALLOWED,
|
||||||
} from '../../lib/exporters/field-allowlist.mjs';
|
} from '../../lib/exporters/field-allowlist.mjs';
|
||||||
|
|
||||||
// ---- path-validator: CWE-22 mitigation -------------------------------------
|
// ---- path-validator: CWE-22 mitigation -------------------------------------
|
||||||
|
|
@ -241,6 +242,40 @@ test('field-allowlist: Object.freeze on allowlists (drift-pin)', () => {
|
||||||
assert.equal(Object.isFrozen(POST_BASH_STATS_ALLOWED), true,
|
assert.equal(Object.isFrozen(POST_BASH_STATS_ALLOWED), true,
|
||||||
'POST_BASH_STATS_ALLOWED must be frozen — runtime mutation prevention');
|
'POST_BASH_STATS_ALLOWED must be frozen — runtime mutation prevention');
|
||||||
assert.equal(Object.isFrozen(EVENT_EMIT_PAYLOAD_ALLOWED), true);
|
assert.equal(Object.isFrozen(EVENT_EMIT_PAYLOAD_ALLOWED), true);
|
||||||
|
assert.equal(Object.isFrozen(TOKEN_USAGE_ALLOWED), true,
|
||||||
|
'TOKEN_USAGE_ALLOWED must be frozen — runtime mutation prevention');
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- token-usage allowlist (SKAL-2, CWE-212) -------------------------------
|
||||||
|
|
||||||
|
test('field-allowlist: token-usage INCLUDES numeric/label fields, EXCLUDES session_id/transcript_path/cwd (two-sided)', () => {
|
||||||
|
const record = {
|
||||||
|
ts: '2026-06-26T12:00:00.000Z',
|
||||||
|
session_id: 'uuid-secret',
|
||||||
|
transcript_path: '/Users/ktg/.claude/projects/x/sesn.jsonl',
|
||||||
|
cwd: '/Users/ktg/secret/project',
|
||||||
|
scope: 'main-context',
|
||||||
|
model: 'claude-opus-4-8',
|
||||||
|
tokens_input: 12345,
|
||||||
|
tokens_output: 6789,
|
||||||
|
tokens_cache_creation: 400,
|
||||||
|
tokens_cache_read: 5000,
|
||||||
|
cost_usd: 0.42,
|
||||||
|
is_estimate: false,
|
||||||
|
price_table_version: '2026-06-26',
|
||||||
|
};
|
||||||
|
const out = applyFieldAllowlist(record, 'token-usage');
|
||||||
|
// INCLUDED (numeric + low-cardinality labels)
|
||||||
|
for (const k of ['tokens_input', 'tokens_output', 'tokens_cache_creation',
|
||||||
|
'tokens_cache_read', 'cost_usd', 'is_estimate', 'price_table_version', 'scope', 'model']) {
|
||||||
|
assert.equal(k in out, true, `${k} MUST be allowlisted`);
|
||||||
|
}
|
||||||
|
assert.equal(out.tokens_input, 12345);
|
||||||
|
assert.equal(out._schema_id, 'token-usage');
|
||||||
|
// EXCLUDED (CWE-212 boundary)
|
||||||
|
assert.equal('session_id' in out, false, 'session_id MUST be stripped (CWE-212)');
|
||||||
|
assert.equal('transcript_path' in out, false, 'transcript_path MUST be stripped (CWE-212)');
|
||||||
|
assert.equal('cwd' in out, false, 'cwd MUST be stripped (CWE-212)');
|
||||||
});
|
});
|
||||||
|
|
||||||
test('field-allowlist: null/undefined record handled safely', () => {
|
test('field-allowlist: null/undefined record handled safely', () => {
|
||||||
|
|
|
||||||
|
|
@ -127,3 +127,39 @@ test('non-orchestrator agents do NOT include the Agent tool (no recursive swarmi
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// M4 (v5.7.1): examples-relocation invariant. The 34 <example> blocks belong in
|
||||||
|
// agent BODIES, not in the always-loaded `description:` frontmatter (voyage
|
||||||
|
// launches its agents by name, so the auto-selection examples are cost without
|
||||||
|
// function there). This pins the migration: examples cannot regress into
|
||||||
|
// frontmatter, and cannot be silently lost during the move.
|
||||||
|
function bodyOf(text) {
|
||||||
|
// Slice the file content AFTER the closing frontmatter `---`.
|
||||||
|
// Do NOT split on /^---$/m — a horizontal rule in the body would mis-split.
|
||||||
|
const m = text.match(/^---\r?\n[\s\S]*?\r?\n---\r?\n?/);
|
||||||
|
return m ? text.slice(m[0].length) : text;
|
||||||
|
}
|
||||||
|
|
||||||
|
test('no agents/*.md frontmatter contains an <example> block (M4: examples live in the body)', () => {
|
||||||
|
for (const f of agentFiles) {
|
||||||
|
const fm = extractFrontmatter(read(`agents/${f}`));
|
||||||
|
assert.ok(fm !== null, `agents/${f}: missing frontmatter`);
|
||||||
|
assert.equal(
|
||||||
|
(fm.match(/<example>/g) || []).length,
|
||||||
|
0,
|
||||||
|
`agents/${f}: <example> blocks must live in the body, not the always-loaded description: frontmatter (M4)`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('agent bodies retain at least 34 <example> blocks (M4: relocation moves, never deletes)', () => {
|
||||||
|
let total = 0;
|
||||||
|
for (const f of agentFiles) {
|
||||||
|
total += (bodyOf(read(`agents/${f}`)).match(/<example>/g) || []).length;
|
||||||
|
}
|
||||||
|
assert.ok(
|
||||||
|
total >= 34,
|
||||||
|
`expected >= 34 <example> blocks across agent bodies (17 agents x 2), got ${total} ` +
|
||||||
|
`— examples may have been deleted instead of relocated (M4)`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
|
||||||
84
tests/lib/cache-analyzer.test.mjs
Normal file
84
tests/lib/cache-analyzer.test.mjs
Normal file
|
|
@ -0,0 +1,84 @@
|
||||||
|
// tests/lib/cache-analyzer.test.mjs
|
||||||
|
// SKAL-2 Step 3 — token/cost aggregation in cache-analyzer's summarize().
|
||||||
|
// Also the first test coverage for cache-analyzer (previously zero-coverage):
|
||||||
|
// includes a regression guard that the pre-existing duration/event-name fields
|
||||||
|
// still compute correctly on mixed input.
|
||||||
|
//
|
||||||
|
// Hermetic: no LLM, network, time, or randomness. Idiom:
|
||||||
|
// tests/lib/token-usage.test.mjs (pure-module node:test style).
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { summarize } from '../../lib/stats/cache-analyzer.mjs';
|
||||||
|
|
||||||
|
const jsonl = (objs) => objs.map(o => JSON.stringify(o));
|
||||||
|
|
||||||
|
// ---- token/cost aggregation ------------------------------------------------
|
||||||
|
|
||||||
|
test('summarize — two token-bearing records → summed totals + cost', () => {
|
||||||
|
const s = summarize(jsonl([
|
||||||
|
{ ts: '2026-06-26T12:00:00Z', session_id: 'S1', scope: 'main-context', tokens_input: 1000, tokens_output: 200, tokens_cache_creation: 400, tokens_cache_read: 5000, cost_usd: 0.03675 },
|
||||||
|
{ ts: '2026-06-26T13:00:00Z', session_id: 'S2', scope: 'main-context', tokens_input: 2000, tokens_output: 300, tokens_cache_creation: 600, tokens_cache_read: 1000, cost_usd: 0.05 },
|
||||||
|
]));
|
||||||
|
assert.equal(s.total_tokens_input, 3000);
|
||||||
|
assert.equal(s.total_tokens_output, 500);
|
||||||
|
assert.equal(s.total_tokens_cache_creation, 1000);
|
||||||
|
assert.equal(s.total_tokens_cache_read, 6000);
|
||||||
|
assert.equal(s.sessions_with_tokens, 2);
|
||||||
|
assert.ok(Math.abs(s.total_cost_usd - 0.08675) < 1e-9, `cost ${s.total_cost_usd}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('summarize — cost_usd:null counts the session but is excluded from total_cost_usd', () => {
|
||||||
|
const s = summarize(jsonl([
|
||||||
|
{ session_id: 'S1', tokens_input: 100, cost_usd: 0.01 },
|
||||||
|
{ session_id: 'S2', tokens_input: 200, cost_usd: null, is_estimate: true },
|
||||||
|
]));
|
||||||
|
assert.equal(s.sessions_with_tokens, 2, 'null-cost session must still be counted');
|
||||||
|
assert.equal(s.total_tokens_input, 300);
|
||||||
|
assert.ok(Math.abs(s.total_cost_usd - 0.01) < 1e-9, `cost ${s.total_cost_usd}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('summarize — no token records → token totals stay zero (additive default)', () => {
|
||||||
|
const s = summarize(jsonl([
|
||||||
|
{ event: 'main-merge-gate', duration_ms: 100 },
|
||||||
|
]));
|
||||||
|
assert.equal(s.total_tokens_input, 0);
|
||||||
|
assert.equal(s.total_cost_usd, 0);
|
||||||
|
assert.equal(s.sessions_with_tokens, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- regression guard: pre-existing fields still computed ------------------
|
||||||
|
|
||||||
|
test('summarize — existing duration/event-name fields still correct on mixed input', () => {
|
||||||
|
const s = summarize(jsonl([
|
||||||
|
{ event: 'main-merge-gate', duration_ms: 100, ts: '2026-06-26T10:00:00Z' },
|
||||||
|
{ event: 'main-merge-approved', duration_ms: 200, ts: '2026-06-26T11:00:00Z' },
|
||||||
|
{ session_id: 'S1', tokens_input: 1000, cost_usd: 0.04, ts: '2026-06-26T12:00:00Z' },
|
||||||
|
]));
|
||||||
|
// Pre-existing behavior (regression guard)
|
||||||
|
assert.equal(s.total_events, 3);
|
||||||
|
assert.equal(s.events_with_duration, 2);
|
||||||
|
// Percentiles: durations [100,200] → floor(2·0.5)=floor(2·0.9)=idx 1 → both 200.
|
||||||
|
assert.equal(s.wall_time_ms_p50, 200);
|
||||||
|
assert.equal(s.wall_time_ms_p90, 200);
|
||||||
|
assert.equal(s.wall_time_ms_max, 200);
|
||||||
|
assert.deepEqual(s.unique_event_names, ['main-merge-approved', 'main-merge-gate']);
|
||||||
|
// Time range spans all ts-bearing lines (incl. the token-only record at 12:00).
|
||||||
|
assert.equal(s.oldest_event_iso, '2026-06-26T10:00:00.000Z');
|
||||||
|
assert.equal(s.newest_event_iso, '2026-06-26T12:00:00.000Z');
|
||||||
|
// New behavior coexists
|
||||||
|
assert.equal(s.total_tokens_input, 1000);
|
||||||
|
assert.equal(s.sessions_with_tokens, 1);
|
||||||
|
assert.ok(Math.abs(s.total_cost_usd - 0.04) < 1e-9);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('summarize — malformed/empty lines tolerated alongside token records', () => {
|
||||||
|
const s = summarize([
|
||||||
|
'',
|
||||||
|
'not json',
|
||||||
|
JSON.stringify({ session_id: 'S1', tokens_input: 50, cost_usd: 0.001 }),
|
||||||
|
]);
|
||||||
|
assert.equal(s.total_events, 1);
|
||||||
|
assert.equal(s.sessions_with_tokens, 1);
|
||||||
|
assert.equal(s.total_tokens_input, 50);
|
||||||
|
});
|
||||||
163
tests/lib/coordinator-contract.test.mjs
Normal file
163
tests/lib/coordinator-contract.test.mjs
Normal file
|
|
@ -0,0 +1,163 @@
|
||||||
|
// tests/lib/coordinator-contract.test.mjs
|
||||||
|
// SKAL-1·4a — deterministic test of the review-coordinator contract subset.
|
||||||
|
//
|
||||||
|
// Inline reviewer-JSON fixtures (idiom: tests/validators/brief-validator.test.mjs).
|
||||||
|
// Hermetic: no LLM, network, time, or randomness.
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import {
|
||||||
|
severityRank,
|
||||||
|
ingest,
|
||||||
|
dedupByTriplet,
|
||||||
|
judgeFilter,
|
||||||
|
reasonablenessFilter,
|
||||||
|
computeVerdict,
|
||||||
|
runContract,
|
||||||
|
} from '../../lib/review/coordinator-contract.mjs';
|
||||||
|
|
||||||
|
// ---- Pass 1 — dedup --------------------------------------------------------
|
||||||
|
|
||||||
|
test('dedupByTriplet — genuine cross-reviewer collapse (identical triplet) → 1, raised_by both', () => {
|
||||||
|
// Both reviewers flag the SAME (file,line,rule_key) triplet — this is the
|
||||||
|
// real collapse the coordinator performs.
|
||||||
|
const findings = [
|
||||||
|
{ file: 'lib/auth/jwt.mjs', line: 19, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER', title: 'algo from header', reviewer: 'correctness' },
|
||||||
|
{ file: 'lib/auth/jwt.mjs', line: 19, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER', title: 'NG1 violated', reviewer: 'conformance' },
|
||||||
|
];
|
||||||
|
const out = dedupByTriplet(findings);
|
||||||
|
assert.equal(out.length, 1, 'identical triplets must collapse to one');
|
||||||
|
assert.deepEqual([...out[0].raised_by].sort(), ['conformance', 'correctness']);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('dedupByTriplet — README #2 two DIFFERENT rule_keys at same file:line do NOT collapse', () => {
|
||||||
|
// The bakeoff-rich README marks issue #2 dual-flaggable via SECURITY_INJECTION
|
||||||
|
// AND NON_GOAL_VIOLATED. Those are DIFFERENT triplets → distinct defects → kept.
|
||||||
|
const findings = [
|
||||||
|
{ file: 'lib/auth/jwt.mjs', line: 19, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER', reviewer: 'correctness' },
|
||||||
|
{ file: 'lib/auth/jwt.mjs', line: 19, rule_key: 'NON_GOAL_VIOLATED', severity: 'BLOCKER', reviewer: 'conformance' },
|
||||||
|
];
|
||||||
|
assert.equal(dedupByTriplet(findings).length, 2);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('dedupByTriplet — survivor is the highest-severity finding in the group', () => {
|
||||||
|
const findings = [
|
||||||
|
{ file: 'x.mjs', line: 1, rule_key: 'SECURITY_INJECTION', severity: 'MINOR', reviewer: 'correctness' },
|
||||||
|
{ file: 'x.mjs', line: 1, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER', reviewer: 'correctness' },
|
||||||
|
];
|
||||||
|
const out = dedupByTriplet(findings);
|
||||||
|
assert.equal(out.length, 1);
|
||||||
|
assert.equal(out[0].severity, 'BLOCKER');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('dedupByTriplet — severity tie breaks toward the conformance reviewer', () => {
|
||||||
|
const findings = [
|
||||||
|
{ file: 'x.mjs', line: 1, rule_key: 'NON_GOAL_VIOLATED', severity: 'BLOCKER', title: 'corr', reviewer: 'correctness' },
|
||||||
|
{ file: 'x.mjs', line: 1, rule_key: 'NON_GOAL_VIOLATED', severity: 'BLOCKER', title: 'conf', reviewer: 'conformance' },
|
||||||
|
];
|
||||||
|
const out = dedupByTriplet(findings);
|
||||||
|
assert.equal(out.length, 1);
|
||||||
|
assert.equal(out[0].reviewer, 'conformance');
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- severity ranking (4-tier, incl. synthetic SUGGESTION) -----------------
|
||||||
|
|
||||||
|
test('severityRank — orders the full 4-tier SEVERITY_VALUES incl. SUGGESTION', () => {
|
||||||
|
assert.ok(severityRank('BLOCKER') < severityRank('MAJOR'));
|
||||||
|
assert.ok(severityRank('MAJOR') < severityRank('MINOR'));
|
||||||
|
assert.ok(severityRank('MINOR') < severityRank('SUGGESTION'));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('dedupByTriplet — SUGGESTION vs MINOR (synthetic) keeps the MINOR survivor', () => {
|
||||||
|
// gold corpus has no SUGGESTION; exercise the 4th tier synthetically.
|
||||||
|
const findings = [
|
||||||
|
{ file: 'x.mjs', line: 2, rule_key: 'MISSING_ERROR_HANDLING', severity: 'SUGGESTION', reviewer: 'correctness' },
|
||||||
|
{ file: 'x.mjs', line: 2, rule_key: 'MISSING_ERROR_HANDLING', severity: 'MINOR', reviewer: 'correctness' },
|
||||||
|
];
|
||||||
|
const out = dedupByTriplet(findings);
|
||||||
|
assert.equal(out.length, 1);
|
||||||
|
assert.equal(out[0].severity, 'MINOR');
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- Pass 4 — verdict thresholds -------------------------------------------
|
||||||
|
|
||||||
|
test('computeVerdict — BLOCKER>=1 → BLOCK, MAJOR-only → WARN, else ALLOW', () => {
|
||||||
|
assert.equal(computeVerdict([{ severity: 'BLOCKER' }, { severity: 'MAJOR' }]).verdict, 'BLOCK');
|
||||||
|
assert.equal(computeVerdict([{ severity: 'MAJOR' }, { severity: 'MINOR' }]).verdict, 'WARN');
|
||||||
|
assert.equal(computeVerdict([{ severity: 'MINOR' }]).verdict, 'ALLOW');
|
||||||
|
assert.equal(computeVerdict([{ severity: 'SUGGESTION' }]).verdict, 'ALLOW');
|
||||||
|
assert.equal(computeVerdict([]).verdict, 'ALLOW');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('computeVerdict — counts each severity tier', () => {
|
||||||
|
const { counts } = computeVerdict([
|
||||||
|
{ severity: 'BLOCKER' }, { severity: 'BLOCKER' }, { severity: 'MAJOR' }, { severity: 'MINOR' },
|
||||||
|
]);
|
||||||
|
assert.deepEqual(counts, { BLOCKER: 2, MAJOR: 1, MINOR: 1, SUGGESTION: 0 });
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- Pass 3 — reasonableness -----------------------------------------------
|
||||||
|
|
||||||
|
test('reasonablenessFilter — drops unknown rule_key + citation-less, corrects severity mismatch', () => {
|
||||||
|
const r = reasonablenessFilter([
|
||||||
|
{ file: 'x.mjs', line: 1, rule_key: 'NOPE_KEY', severity: 'BLOCKER' }, // unknown → drop
|
||||||
|
{ file: '', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR' }, // no file → drop
|
||||||
|
{ file: 'x.mjs', line: -1, rule_key: 'MISSING_TEST', severity: 'MAJOR' }, // line < 0 → drop
|
||||||
|
{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MINOR' }, // catalogue is MAJOR → correct, keep
|
||||||
|
]);
|
||||||
|
assert.equal(r.kept.length, 1);
|
||||||
|
assert.equal(r.dropped.length, 3);
|
||||||
|
assert.equal(r.kept[0].severity, 'MAJOR');
|
||||||
|
assert.equal(r.kept[0].original_severity, 'MINOR');
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- Pass 2 — judge --------------------------------------------------------
|
||||||
|
|
||||||
|
test('judgeFilter — drops over-long title and empty recommended_action', () => {
|
||||||
|
const j = judgeFilter([
|
||||||
|
{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'x'.repeat(101) }, // too long → drop
|
||||||
|
{ file: 'x.mjs', line: 2, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'ok', recommended_action: ' ' }, // empty action → drop
|
||||||
|
{ file: 'x.mjs', line: 3, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'ok' }, // keep (no action field is fine)
|
||||||
|
]);
|
||||||
|
assert.equal(j.kept.length, 1);
|
||||||
|
assert.equal(j.dropped.length, 2);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- ingest ----------------------------------------------------------------
|
||||||
|
|
||||||
|
test('ingest — collects findings from valid payloads, skips invalid ones', () => {
|
||||||
|
const { findings, skipped } = ingest([
|
||||||
|
{ reviewer: 'correctness', findings: [{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR' }] },
|
||||||
|
{ reviewer: 'bad', findings: [{ line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR' }] }, // missing file → invalid → skipped
|
||||||
|
]);
|
||||||
|
assert.equal(findings.length, 1);
|
||||||
|
assert.equal(findings[0].reviewer, 'correctness');
|
||||||
|
assert.equal(skipped.length, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- end-to-end ------------------------------------------------------------
|
||||||
|
|
||||||
|
test('runContract — two reviewers, dual-flag triplet collapses, verdict BLOCK', () => {
|
||||||
|
const result = runContract([
|
||||||
|
{ reviewer: 'correctness', findings: [
|
||||||
|
{ file: 'lib/auth/jwt.mjs', line: 19, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER', title: 'algo from header' },
|
||||||
|
{ file: 'lib/auth/refresh.mjs', line: 0, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'no concurrent test' },
|
||||||
|
] },
|
||||||
|
{ reviewer: 'conformance', findings: [
|
||||||
|
{ file: 'lib/auth/jwt.mjs', line: 19, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER', title: 'NG1' }, // collapses with correctness's
|
||||||
|
{ file: 'lib/handlers/login.mjs', line: 17, rule_key: 'UNIMPLEMENTED_CRITERION', severity: 'BLOCKER', title: '200 not 401' },
|
||||||
|
] },
|
||||||
|
]);
|
||||||
|
assert.equal(result.verdict, 'BLOCK');
|
||||||
|
assert.equal(result.findings.length, 3, '4 raw findings, jwt:19 SECURITY_INJECTION collapses → 3');
|
||||||
|
const jwt = result.findings.find((f) => f.file === 'lib/auth/jwt.mjs');
|
||||||
|
assert.deepEqual([...jwt.raised_by].sort(), ['conformance', 'correctness']);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('runContract — deterministic: identical input yields identical output', () => {
|
||||||
|
const input = [
|
||||||
|
{ reviewer: 'correctness', findings: [{ file: 'a.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 't' }] },
|
||||||
|
{ reviewer: 'conformance', findings: [{ file: 'b.mjs', line: 2, rule_key: 'UNIMPLEMENTED_CRITERION', severity: 'BLOCKER', title: 'u' }] },
|
||||||
|
];
|
||||||
|
assert.deepEqual(runContract(input), runContract(input));
|
||||||
|
});
|
||||||
|
|
@ -918,7 +918,7 @@ test('S15: default-profile name is consistent across resolver + all profile docs
|
||||||
assert.equal(profile_source, 'default', 'resolveProfile({}, {}) must report source=default');
|
assert.equal(profile_source, 'default', 'resolveProfile({}, {}) must report source=default');
|
||||||
assert.equal(def, 'premium', 'resolver hardcoded default is premium (operator decision 2026-05-13, commit 40d8742)');
|
assert.equal(def, 'premium', 'resolver hardcoded default is premium (operator decision 2026-05-13, commit 40d8742)');
|
||||||
|
|
||||||
const OTHERS = ['economy', 'balanced', 'premium'].filter((p) => p !== def);
|
const OTHERS = ['economy', 'balanced', 'premium', 'fable'].filter((p) => p !== def);
|
||||||
for (const doc of PROFILE_DOCS) {
|
for (const doc of PROFILE_DOCS) {
|
||||||
const body = read(doc);
|
const body = read(doc);
|
||||||
assert.ok(
|
assert.ok(
|
||||||
|
|
@ -941,7 +941,7 @@ test('S15: default-profile name is consistent across resolver + all profile docs
|
||||||
test('S15: profile tables encode each built-in yaml phase_models exactly', () => {
|
test('S15: profile tables encode each built-in yaml phase_models exactly', () => {
|
||||||
// Column order in every profile table: Profile | Brief | Research | Plan | Execute | Review | Continue | Use case
|
// Column order in every profile table: Profile | Brief | Research | Plan | Execute | Review | Continue | Use case
|
||||||
const PHASES = ['brief', 'research', 'plan', 'execute', 'review', 'continue'];
|
const PHASES = ['brief', 'research', 'plan', 'execute', 'review', 'continue'];
|
||||||
for (const name of ['economy', 'balanced', 'premium']) {
|
for (const name of ['economy', 'balanced', 'premium', 'fable']) {
|
||||||
const pm = loadProfile(name).phase_models; // {brief:'opus', ...}
|
const pm = loadProfile(name).phase_models; // {brief:'opus', ...}
|
||||||
const expected = PHASES.map((ph) => pm[ph]);
|
const expected = PHASES.map((ph) => pm[ph]);
|
||||||
for (const doc of PROFILE_DOCS) {
|
for (const doc of PROFILE_DOCS) {
|
||||||
|
|
@ -961,6 +961,46 @@ test('S15: profile tables encode each built-in yaml phase_models exactly', () =>
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// --- S34 (V30) — economy is self-declared experimental until the cross-tier
|
||||||
|
// Jaccard floor (0.55) is empirically calibrated (Step-17 calibration deferred
|
||||||
|
// to v4.2). The status must be visible in BOTH the profile data
|
||||||
|
// (lib/profiles/economy.yaml `experimental: true`) and every profile doc's
|
||||||
|
// economy row. Fix the SOURCE doc/profile, not these pins.
|
||||||
|
test('S34: economy profile row is flagged experimental across all profile docs', () => {
|
||||||
|
for (const doc of PROFILE_DOCS) {
|
||||||
|
const body = read(doc);
|
||||||
|
const row = body.split('\n').find((l) => /^\|\s*`economy`/.test(l));
|
||||||
|
assert.ok(row, `${doc}: profile table missing an \`economy\` row`);
|
||||||
|
assert.match(
|
||||||
|
row,
|
||||||
|
/experimental/i,
|
||||||
|
`${doc}: the \`economy\` table row must flag it experimental ` +
|
||||||
|
`(uncalibrated Jaccard floor, S34/V30) — matches lib/profiles/economy.yaml experimental: true`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('S34: economy experimental flag agrees with parked-synthetic calibration status', () => {
|
||||||
|
// economy.yaml experimental:true is honest ONLY while calibration is still
|
||||||
|
// parked-synthetic. When empirical calibration lands (status: empirical),
|
||||||
|
// drop experimental:true in the SAME change — this pin enforces the linkage.
|
||||||
|
const eco = parseDocument(read('lib/profiles/economy.yaml')).parsed.frontmatter;
|
||||||
|
const cal = parseDocument(read('tests/synthetic/profile-jaccard-calibration.md')).parsed.frontmatter;
|
||||||
|
if (cal.status === 'parked-synthetic') {
|
||||||
|
assert.equal(
|
||||||
|
eco.experimental,
|
||||||
|
true,
|
||||||
|
'while calibration is parked-synthetic, economy.yaml must stay experimental: true',
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
assert.notEqual(
|
||||||
|
eco.experimental,
|
||||||
|
true,
|
||||||
|
'calibration is no longer parked-synthetic — drop economy experimental: true in the same change',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
test('S15: README cost prose does not resurrect the false Sonnet-swarm claim', () => {
|
test('S15: README cost prose does not resurrect the false Sonnet-swarm claim', () => {
|
||||||
// All 24 sub-agents are model: opus (operator-pinned, commit 40d8742) and the
|
// All 24 sub-agents are model: opus (operator-pinned, commit 40d8742) and the
|
||||||
// model is uniform per phase — there is no "Opus orchestrates / Sonnet runs
|
// model is uniform per phase — there is no "Opus orchestrates / Sonnet runs
|
||||||
|
|
@ -1025,6 +1065,32 @@ test('S18: --min-brief-version is documented at the trekplan + trekresearch boun
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('deep-research-engine: --engine is documented + consistent across surfaces', () => {
|
||||||
|
// Cross-doc pin mirroring S18 (:1057). The opt-in external-research engine flag
|
||||||
|
// must be discoverable wherever /trekresearch flags live: the command itself
|
||||||
|
// plus the three reference surfaces.
|
||||||
|
for (const f of ['commands/trekresearch.md', 'docs/command-modes.md', 'CLAUDE.md', 'README.md']) {
|
||||||
|
assert.ok(
|
||||||
|
read(f).includes('--engine'),
|
||||||
|
`${f} must document the --engine flag (deep-research-engine)`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// README documents it specifically as an **Engine** mode-table row.
|
||||||
|
assert.ok(
|
||||||
|
/\*\*Engine\*\*/.test(read('README.md')),
|
||||||
|
'README.md must document --engine as an **Engine** mode row',
|
||||||
|
);
|
||||||
|
// The command prose must name both engine values and the swarm fallback, so the
|
||||||
|
// opt-in + graceful-degradation contract is pinned — not merely the flag string.
|
||||||
|
const research = read('commands/trekresearch.md');
|
||||||
|
assert.ok(/\bswarm\b/.test(research), 'trekresearch.md must name the swarm engine value');
|
||||||
|
assert.ok(/\bdeep-research\b/.test(research), 'trekresearch.md must name the deep-research engine value');
|
||||||
|
assert.ok(
|
||||||
|
/fall back|falls back/.test(research),
|
||||||
|
'trekresearch.md must document the swarm fallback (graceful degradation)',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
test('S18: HANDOVER-CONTRACTS documents the pre-2.2 zero-framing-enforcement hole', () => {
|
test('S18: HANDOVER-CONTRACTS documents the pre-2.2 zero-framing-enforcement hole', () => {
|
||||||
// The framing defense is producer-elective: a brief declaring ≤ 2.1 sidesteps
|
// The framing defense is producer-elective: a brief declaring ≤ 2.1 sidesteps
|
||||||
// it entirely. Handover 1 (PUBLIC CONTRACT) must disclose this and name the remedy.
|
// it entirely. Handover 1 (PUBLIC CONTRACT) must disclose this and name the remedy.
|
||||||
|
|
@ -1038,3 +1104,233 @@ test('S18: HANDOVER-CONTRACTS documents the pre-2.2 zero-framing-enforcement hol
|
||||||
'HANDOVER-CONTRACTS.md must state that pre-2.2 briefs receive zero framing enforcement',
|
'HANDOVER-CONTRACTS.md must state that pre-2.2 briefs receive zero framing enforcement',
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ── S32 — "delegate the engine, keep the policy": native-delegation hygiene ──
|
||||||
|
//
|
||||||
|
// Balance-analysis V01 / V07 / V08 / V11 / V24 audit (2026-06-20): the brief
|
||||||
|
// interview and the research / exploration / reviewer swarms must ride NATIVE
|
||||||
|
// Claude Code primitives — parallel `Agent` spawn in a single message +
|
||||||
|
// `AskUserQuestion` for interviews — and re-implement no scheduler /
|
||||||
|
// concurrency-loop / menu engine of their own. The audit found all five ALREADY
|
||||||
|
// native (no code change), so these pins are the standing guard against an
|
||||||
|
// engine creeping back in, plus the documented principle in docs/architecture.md.
|
||||||
|
// Fix the SOURCE (command prose / the architecture.md note), not the test.
|
||||||
|
|
||||||
|
const SWARM_COMMANDS = ['commands/trekresearch.md', 'commands/trekplan.md', 'commands/trekreview.md'];
|
||||||
|
|
||||||
|
test('S32: docs/architecture.md records the "delegate the engine, keep the policy" principle', () => {
|
||||||
|
const t = read('docs/architecture.md');
|
||||||
|
assert.ok(
|
||||||
|
/delegate the engine, keep the policy/i.test(t),
|
||||||
|
'docs/architecture.md must name the "delegate the engine, keep the policy" principle (V01/V07/V08/V11/V24 audit, S32)',
|
||||||
|
);
|
||||||
|
// The note must name the native primitives it delegates to (so the principle is
|
||||||
|
// anchored to real CC tools, not abstract prose).
|
||||||
|
assert.ok(
|
||||||
|
/AskUserQuestion/.test(t) && /\bAgent\b/.test(t),
|
||||||
|
'the principle note must name the native primitives delegated to (Agent + AskUserQuestion)',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('S32: each swarm command lists the Agent tool and mandates native parallel single-message spawn', () => {
|
||||||
|
for (const f of SWARM_COMMANDS) {
|
||||||
|
const text = read(f);
|
||||||
|
assert.ok(
|
||||||
|
/^allowed-tools:.*\bAgent\b/m.test(text),
|
||||||
|
`${f}: allowed-tools frontmatter must list Agent (the native swarm-spawn engine)`,
|
||||||
|
);
|
||||||
|
assert.ok(
|
||||||
|
/in parallel/i.test(text),
|
||||||
|
`${f}: must instruct parallel swarm spawn (native delegation, not a hand-rolled scheduler)`,
|
||||||
|
);
|
||||||
|
assert.ok(
|
||||||
|
/single (assistant )?message|one message|multiple tool calls/i.test(text),
|
||||||
|
`${f}: must mandate single-message multi-tool-call dispatch (the native parallel-Agent pattern)`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('S32: trekbrief Phase 3 delegates Q&A turn-taking to AskUserQuestion (V01 — selection rule is policy, asking is native)', () => {
|
||||||
|
const text = read('commands/trekbrief.md');
|
||||||
|
assert.ok(
|
||||||
|
/^allowed-tools:.*AskUserQuestion/m.test(text),
|
||||||
|
'commands/trekbrief.md: allowed-tools must list AskUserQuestion (the native Q&A engine)',
|
||||||
|
);
|
||||||
|
const p3Start = text.indexOf('## Phase 3 — Completeness loop');
|
||||||
|
assert.ok(p3Start >= 0, 'trekbrief.md must have a "## Phase 3 — Completeness loop" section');
|
||||||
|
const p3End = text.indexOf('## Phase 3.5', p3Start);
|
||||||
|
const p3 = text.slice(p3Start, p3End > p3Start ? p3End : undefined);
|
||||||
|
assert.ok(
|
||||||
|
/Use `AskUserQuestion` for every question/.test(p3),
|
||||||
|
'Phase 3 must delegate Q&A turn-taking to AskUserQuestion — the selection rule picks WHICH ' +
|
||||||
|
'question to ask (policy); AskUserQuestion does the asking (engine). No hand-rolled menu loop.',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── S33 — documentation consolidation: record the considered-and-kept decisions ──
|
||||||
|
//
|
||||||
|
// The D1–D3 forks (balance-backlog-plan.md) resolved conservatively, collapsing
|
||||||
|
// the model + observability work into a documentation record. These pins make
|
||||||
|
// the three deliverables enforced doc-truth:
|
||||||
|
// • V35 (doc half) — the "24 agents" framing is reconciled to its honest split
|
||||||
|
// (21 spawnable + 3 reference orchestrators, of which synthesis-agent is the
|
||||||
|
// one dormant spawnable). Counts are DERIVED from agents/ so they survive
|
||||||
|
// reword and track new files.
|
||||||
|
// • D2 (V32 rationale) — observability.md documents why direct export rather
|
||||||
|
// than a native collector (preserves the S21 in-process guards).
|
||||||
|
// • D3 (kept-opus) — the analysis doc carries a decision record citing the pin
|
||||||
|
// commit; the model frontmatter is NOT changed (agent-frontmatter.test.mjs
|
||||||
|
// stays the structural source-of-truth and is untouched by S33).
|
||||||
|
// Fix the SOURCE a pin guards, not the test.
|
||||||
|
|
||||||
|
test('S33: README + CLAUDE.md reconcile the agent inventory (spawnable + reference orchestrators = file count)', () => {
|
||||||
|
const agentFiles = listMd('agents');
|
||||||
|
const total = agentFiles.length;
|
||||||
|
const orchestrators = agentFiles.filter((f) => /-orchestrator\.md$/.test(f));
|
||||||
|
const refCount = orchestrators.length;
|
||||||
|
const spawnable = total - refCount;
|
||||||
|
assert.equal(refCount, 3, `expected exactly 3 *-orchestrator reference docs, got ${refCount}`);
|
||||||
|
for (const [name, body] of [['README.md', read('README.md')], ['CLAUDE.md', read('CLAUDE.md')]]) {
|
||||||
|
assert.ok(
|
||||||
|
body.includes(`${spawnable} spawnable`),
|
||||||
|
`${name} must state "${spawnable} spawnable" agents (derived: ${total} files − ${refCount} reference orchestrators)`,
|
||||||
|
);
|
||||||
|
assert.ok(
|
||||||
|
body.includes(`${refCount} orchestrator reference doc`),
|
||||||
|
`${name} must label the ${refCount} orchestrators as reference docs (not spawnable capabilities)`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('S33: the inventory framing names synthesis-agent as the one dormant spawnable', () => {
|
||||||
|
// 1 of the spawnable agents ships dormant (Δ≈0, wired to nothing). The framing
|
||||||
|
// must say so, so the headline count is honest about live vs. dormant capability.
|
||||||
|
for (const [name, body] of [['README.md', read('README.md')], ['CLAUDE.md', read('CLAUDE.md')]]) {
|
||||||
|
assert.ok(
|
||||||
|
/synthesis-agent/.test(body) && /dormant/i.test(body),
|
||||||
|
`${name} inventory framing must name synthesis-agent as the dormant spawnable`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('S33: the 3 orchestrator files declare themselves reference docs, not spawnable capabilities (V35 relabel)', () => {
|
||||||
|
for (const f of listMd('agents').filter((x) => /-orchestrator\.md$/.test(x))) {
|
||||||
|
assert.ok(
|
||||||
|
/reference document, not a spawnable capability/i.test(read(`agents/${f}`)),
|
||||||
|
`agents/${f} must declare "reference document, not a spawnable capability" (V35 doc-half relabel)`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('S33: docs/observability.md documents the direct-export-vs-collector rationale (D2)', () => {
|
||||||
|
const t = read('docs/observability.md');
|
||||||
|
assert.ok(
|
||||||
|
/direct[- ]export/i.test(t) && /collector/i.test(t),
|
||||||
|
'observability.md must document why Voyage exports directly rather than via a native collector (D2)',
|
||||||
|
);
|
||||||
|
// The rationale must anchor on the in-process security boundary it preserves
|
||||||
|
// (the S21 hardening), not read as an unjustified re-host.
|
||||||
|
assert.ok(
|
||||||
|
/allowlist/i.test(t) && /(SSRF|CWE-918|CWE-212)/.test(t),
|
||||||
|
'the direct-export rationale must cite the in-process guards it preserves (field allowlist + SSRF/path)',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('S33: the analysis doc records the D1–D3 resolved forks (considered-and-kept, pin firm)', () => {
|
||||||
|
const t = read('docs/voyage-vs-cc-balance-analysis.md');
|
||||||
|
assert.ok(/Decision record/i.test(t), 'analysis doc must carry a "Decision record" addendum for the resolved forks');
|
||||||
|
for (const d of ['D1', 'D2', 'D3']) {
|
||||||
|
assert.ok(new RegExp(`\\b${d}\\b`).test(t), `analysis-doc decision record must name fork ${d}`);
|
||||||
|
}
|
||||||
|
// D3 kept the model pin firm — the record must cite the pin commit so a future
|
||||||
|
// reader sees the downgrade was reconsidered-and-declined, not overlooked.
|
||||||
|
assert.ok(/40d8742/.test(t), 'decision record must cite the model-pin commit 40d8742 (D3 kept firm, document-only)');
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── S38 — fan-out hedge harmonization + relief-assertion forward-guard (SC d) ──
|
||||||
|
//
|
||||||
|
// The brief's SC(d) premise — an unhedged "fan-out relieves main context" string
|
||||||
|
// present in ~23 files awaiting a removal diff — is factually false: plan-time
|
||||||
|
// enumeration shows the unhedged set is ∅. Every live relief mention is already
|
||||||
|
// hedged (the sole one is CLAUDE.md:5, hedged since S18), and the
|
||||||
|
// docs/T2-bakeoff-results.md "frees the main context" lines are a DIFFERENT, true
|
||||||
|
// background-execution claim (must NOT be touched). SC(d) therefore ships as:
|
||||||
|
// (1) one surgical harmonization at CLAUDE.md:5 making the replacement phrase
|
||||||
|
// "parallel wall-clock" grep-positive (the Δ≈0 / "not yet demonstrated"
|
||||||
|
// qualifier kept intact), and
|
||||||
|
// (2) this durable forward-guard — a PRECISE relief-assertion matcher (verb-of-
|
||||||
|
// relief + main-context, NOT a bare "main context" grep that would false-fail
|
||||||
|
// benign locatives like "runs inline in main context") scoped to product-
|
||||||
|
// facing docs, requiring an adjacent hedge token on any match.
|
||||||
|
// Fix the SOURCE prose (add the hedge / drop the claim), never this test.
|
||||||
|
|
||||||
|
test('S38: CLAUDE.md carries the harmonized fan-out phrase "parallel wall-clock" (SC d)', () => {
|
||||||
|
assert.ok(
|
||||||
|
read('CLAUDE.md').includes('parallel wall-clock'),
|
||||||
|
'CLAUDE.md:5 must state the replacement phrase "parallel wall-clock + structured artifact handoffs" (SC d harmonization)',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('S38: forward-guard — no product-facing doc asserts main-context relief unhedged (SC d)', () => {
|
||||||
|
// Banned form: a verb of relief adjacent to "main context" (or "main-context …
|
||||||
|
// relief"), NOT a plain locative ("runs in main context"). Decision/research
|
||||||
|
// records (docs/*) are append-only history and out of scope by design.
|
||||||
|
const RELIEF = /(relieve|relieves|relieving|free|frees|offload|offloads)\b[^.\n]{0,40}\bmain[-\s]context\b|\bmain[-\s]context\b[^.\n]{0,20}\brelief\b|swarm\s+relieves\s+context/i;
|
||||||
|
const HEDGE = /Δ|not measured|asserted-by-design|not yet demonstrated|not demonstrated/i;
|
||||||
|
const productFacing = ['README.md', 'CLAUDE.md', ...listMd('commands').map((f) => `commands/${f}`)];
|
||||||
|
const offenders = [];
|
||||||
|
for (const rel of productFacing) {
|
||||||
|
read(rel).split('\n').forEach((ln, i) => {
|
||||||
|
if (RELIEF.test(ln) && !HEDGE.test(ln)) offenders.push(`${rel}:${i + 1} ${ln.trim().slice(0, 100)}`);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
assert.equal(
|
||||||
|
offenders.length,
|
||||||
|
0,
|
||||||
|
'Unhedged main-context-relief assertion(s) in product-facing docs — add an adjacent hedge ' +
|
||||||
|
'(Δ / "not measured" / "asserted-by-design" / "not yet demonstrated") or drop the claim:\n' +
|
||||||
|
offenders.join('\n'),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── v5.9 (fable-tier step 6) — composed-resolver wiring pin ────────────────
|
||||||
|
// The four pipeline commands must resolve {effort, model} via the composed
|
||||||
|
// CLI (`resolver.mjs --resolve-phase-model`, brief > profile > default). A
|
||||||
|
// direct `phase-signal-resolver.mjs --brief` invocation in a command Bash
|
||||||
|
// block is the brief-only CLI: it silently drops the profile layer (the
|
||||||
|
// AP2-1 footgun — `--profile <x>` would never reach sub-agent spawns again).
|
||||||
|
// If this pin fails, re-wire the command to the composed CLI — do not relax
|
||||||
|
// the pin.
|
||||||
|
|
||||||
|
for (const cmd of ['trekresearch', 'trekplan', 'trekreview', 'trekexecute']) {
|
||||||
|
test(`v5.9: commands/${cmd}.md invokes the composed resolver, not the brief-only CLI`, () => {
|
||||||
|
const text = read(`commands/${cmd}.md`);
|
||||||
|
assert.ok(
|
||||||
|
text.includes('--resolve-phase-model'),
|
||||||
|
`commands/${cmd}.md must invoke resolver.mjs --resolve-phase-model (composed brief > profile > default)`,
|
||||||
|
);
|
||||||
|
assert.ok(
|
||||||
|
!/phase-signal-resolver\.mjs --brief/.test(text),
|
||||||
|
`commands/${cmd}.md must not invoke the brief-only phase-signal-resolver CLI — the profile layer would be silently dropped`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── v5.9 (fable-tier step 7) — orchestrator model-pin ABSENCE pin ───────────
|
||||||
|
// Command frontmatter deliberately omits `model:` — omission is the only
|
||||||
|
// session-inheritance spelling documented by BOTH official surfaces (AP2-2:
|
||||||
|
// the `inherit` literal is disputed between skills.md and the plugin-dev
|
||||||
|
// command-frontmatter reference). Re-adding a pin locks the orchestrator to a
|
||||||
|
// fixed model in every session; if deterministic pinning is ever wanted again,
|
||||||
|
// do it consciously and update this pin's rationale.
|
||||||
|
|
||||||
|
test('v5.9: no commands/*.md frontmatter carries a model: key (session inheritance by omission)', () => {
|
||||||
|
const offenders = [];
|
||||||
|
for (const f of listMd('commands')) {
|
||||||
|
const doc = parseDocument(read(`commands/${f}`));
|
||||||
|
const fm = doc.parsed && doc.parsed.frontmatter;
|
||||||
|
if (fm && 'model' in fm) offenders.push(f);
|
||||||
|
}
|
||||||
|
assert.deepEqual(offenders, [],
|
||||||
|
`command frontmatter must omit model: (orchestrator follows the session model); offenders: ${offenders.join(', ')}`);
|
||||||
|
});
|
||||||
|
|
|
||||||
68
tests/lib/gold-corpus.test.mjs
Normal file
68
tests/lib/gold-corpus.test.mjs
Normal file
|
|
@ -0,0 +1,68 @@
|
||||||
|
// tests/lib/gold-corpus.test.mjs
|
||||||
|
// SKAL-1·4a — golden corpus loader/validator.
|
||||||
|
//
|
||||||
|
// gold.json is the machine-readable form of the 5 seeded findings in
|
||||||
|
// tests/fixtures/bakeoff-rich/README.md. This test pins its shape and ties
|
||||||
|
// every rule_key / severity back to the canonical catalogue so the corpus
|
||||||
|
// can never drift to an invented rule_key or severity tier.
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { readFileSync } from 'node:fs';
|
||||||
|
import { join, dirname } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { RULE_KEYS, SEVERITY_VALUES } from '../../lib/review/rule-catalogue.mjs';
|
||||||
|
import { FINDING_REQUIRED_FIELDS } from '../../lib/review/findings-schema.mjs';
|
||||||
|
|
||||||
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const ROOT = join(HERE, '..', '..');
|
||||||
|
const GOLD = join(ROOT, 'tests/fixtures/bakeoff-rich/gold.json');
|
||||||
|
|
||||||
|
const gold = JSON.parse(readFileSync(GOLD, 'utf-8'));
|
||||||
|
|
||||||
|
test('gold.json — parses and has exactly 5 findings', () => {
|
||||||
|
assert.ok(Array.isArray(gold.findings), 'findings must be an array');
|
||||||
|
assert.equal(gold.findings.length, 5);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('gold.json — corpus-level expected_verdict is BLOCK', () => {
|
||||||
|
assert.equal(gold.expected_verdict, 'BLOCK');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('gold.json — every finding carries the 5 required fields', () => {
|
||||||
|
// FINDING_REQUIRED_FIELDS (severity, rule_key, file, line) + the eval addition owner_reviewer.
|
||||||
|
const required = [...FINDING_REQUIRED_FIELDS, 'owner_reviewer'];
|
||||||
|
for (const [i, f] of gold.findings.entries()) {
|
||||||
|
for (const field of required) {
|
||||||
|
assert.ok(field in f, `findings[${i}] missing required field "${field}"`);
|
||||||
|
}
|
||||||
|
assert.ok(typeof f.file === 'string' && f.file.length > 0, `findings[${i}].file empty`);
|
||||||
|
assert.ok(Number.isInteger(f.line) && f.line >= 0, `findings[${i}].line must be an int >= 0`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('gold.json — every rule_key is a member of the canonical RULE_CATALOGUE', () => {
|
||||||
|
for (const [i, f] of gold.findings.entries()) {
|
||||||
|
assert.ok(RULE_KEYS.has(f.rule_key), `findings[${i}].rule_key "${f.rule_key}" not in catalogue`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('gold.json — every severity is a valid catalogue tier', () => {
|
||||||
|
for (const [i, f] of gold.findings.entries()) {
|
||||||
|
assert.ok(SEVERITY_VALUES.includes(f.severity), `findings[${i}].severity "${f.severity}" invalid`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('gold.json — owner_reviewer is one of the two reviewer roles', () => {
|
||||||
|
for (const [i, f] of gold.findings.entries()) {
|
||||||
|
assert.ok(
|
||||||
|
f.owner_reviewer === 'conformance' || f.owner_reviewer === 'correctness',
|
||||||
|
`findings[${i}].owner_reviewer "${f.owner_reviewer}" must be conformance|correctness`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('gold.json — contains at least one BLOCKER (consistent with expected_verdict BLOCK)', () => {
|
||||||
|
const hasBlocker = gold.findings.some((f) => f.severity === 'BLOCKER');
|
||||||
|
assert.ok(hasBlocker, 'expected_verdict is BLOCK but no BLOCKER finding present');
|
||||||
|
});
|
||||||
49
tests/lib/gold-eval.test.mjs
Normal file
49
tests/lib/gold-eval.test.mjs
Normal file
|
|
@ -0,0 +1,49 @@
|
||||||
|
// tests/lib/gold-eval.test.mjs
|
||||||
|
// SKAL-1·4b — the offline gold-scored output eval (the scoring RUN).
|
||||||
|
//
|
||||||
|
// This is the third test-census category (see lib/util/test-census.mjs):
|
||||||
|
// neither a behavior unit test nor a doc-consistency pin, but a SCORING RUN —
|
||||||
|
// it feeds a committed agent-run fixture through the deterministic coordinator
|
||||||
|
// contract (4a) and scores the result against the golden corpus at
|
||||||
|
// (file, rule_key) granularity. Offline: committed reviewer payloads, no live
|
||||||
|
// agent spawn, no LLM, no network. The all-agree foundation; the
|
||||||
|
// LLM-in-the-loop eval is the separate 4c tier.
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { readFileSync } from 'node:fs';
|
||||||
|
import { join, dirname } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { runContract } from '../../lib/review/coordinator-contract.mjs';
|
||||||
|
import { scoreFindings, scoreVerdict } from '../../lib/review/gold-scorer.mjs';
|
||||||
|
|
||||||
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const ROOT = join(HERE, '..', '..');
|
||||||
|
const gold = JSON.parse(readFileSync(join(ROOT, 'tests/fixtures/bakeoff-rich/gold.json'), 'utf-8'));
|
||||||
|
const runPerfect = JSON.parse(
|
||||||
|
readFileSync(join(ROOT, 'tests/fixtures/bakeoff-rich/runs/run-perfect.json'), 'utf-8'),
|
||||||
|
);
|
||||||
|
|
||||||
|
test('gold-eval — run-perfect reproduces gold at (file,rule_key): precision/recall/f1 = 1.0', () => {
|
||||||
|
const result = runContract(runPerfect);
|
||||||
|
const score = scoreFindings(result.findings, gold.findings);
|
||||||
|
assert.equal(score.precision, 1, `precision ${score.precision}; spurious=${JSON.stringify(score.spurious)}`);
|
||||||
|
assert.equal(score.recall, 1, `recall ${score.recall}; missed=${JSON.stringify(score.missed)}`);
|
||||||
|
assert.equal(score.f1, 1);
|
||||||
|
assert.equal(score.tp, gold.findings.length);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('gold-eval — run-perfect coordinator verdict matches gold expected_verdict (BLOCK)', () => {
|
||||||
|
const result = runContract(runPerfect);
|
||||||
|
assert.equal(result.verdict, gold.expected_verdict);
|
||||||
|
assert.ok(scoreVerdict(result.verdict, gold.expected_verdict));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('gold-eval — run-perfect drops nothing (no suppressed, no schema-skipped payloads)', () => {
|
||||||
|
// A clean run is the regression guard: any future contract change that
|
||||||
|
// silently suppresses or skips one of the 5 seeded findings breaks here.
|
||||||
|
const result = runContract(runPerfect);
|
||||||
|
assert.equal(result.skipped.length, 0, `skipped payloads: ${JSON.stringify(result.skipped)}`);
|
||||||
|
assert.equal(result.suppressed.length, 0, `suppressed findings: ${result.suppressed.length}`);
|
||||||
|
assert.equal(result.findings.length, gold.findings.length);
|
||||||
|
});
|
||||||
126
tests/lib/gold-scorer.test.mjs
Normal file
126
tests/lib/gold-scorer.test.mjs
Normal file
|
|
@ -0,0 +1,126 @@
|
||||||
|
// tests/lib/gold-scorer.test.mjs
|
||||||
|
// SKAL-1·4b — behavior test for the gold scorer.
|
||||||
|
//
|
||||||
|
// scoreFindings compares a recorded agent-run's findings against a golden
|
||||||
|
// corpus record at (file, rule_key) granularity (line + severity deliberately
|
||||||
|
// ignored). This pins the precision/recall/f1 math AND the discriminating
|
||||||
|
// paths (false negatives + false positives), not merely the all-match case —
|
||||||
|
// a scorer that always returned 1.0 would pass an all-match-only test.
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { scoreFindings, scoreVerdict } from '../../lib/review/gold-scorer.mjs';
|
||||||
|
|
||||||
|
// A minimal gold set: two distinct (file, rule_key) pairs.
|
||||||
|
const GOLD = [
|
||||||
|
{ file: 'a.mjs', line: 10, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' },
|
||||||
|
{ file: 'b.mjs', line: 0, rule_key: 'MISSING_TEST', severity: 'MAJOR' },
|
||||||
|
];
|
||||||
|
|
||||||
|
test('scoreFindings — perfect match scores precision/recall/f1 = 1', () => {
|
||||||
|
// Same pairs, different line/severity (must be ignored at (file,rule_key) granularity).
|
||||||
|
const run = [
|
||||||
|
{ file: 'a.mjs', line: 99, rule_key: 'SECURITY_INJECTION', severity: 'MINOR' },
|
||||||
|
{ file: 'b.mjs', line: 7, rule_key: 'MISSING_TEST', severity: 'BLOCKER' },
|
||||||
|
];
|
||||||
|
const s = scoreFindings(run, GOLD);
|
||||||
|
assert.equal(s.tp, 2);
|
||||||
|
assert.equal(s.fp, 0);
|
||||||
|
assert.equal(s.fn, 0);
|
||||||
|
assert.equal(s.precision, 1);
|
||||||
|
assert.equal(s.recall, 1);
|
||||||
|
assert.equal(s.f1, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scoreFindings — a missed gold pair is a false negative (recall < 1)', () => {
|
||||||
|
const run = [{ file: 'a.mjs', line: 10, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' }];
|
||||||
|
const s = scoreFindings(run, GOLD);
|
||||||
|
assert.equal(s.tp, 1);
|
||||||
|
assert.equal(s.fp, 0);
|
||||||
|
assert.equal(s.fn, 1);
|
||||||
|
assert.equal(s.precision, 1);
|
||||||
|
assert.equal(s.recall, 0.5);
|
||||||
|
assert.deepEqual(s.missed, ['b.mjs MISSING_TEST']);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scoreFindings — a spurious pair is a false positive (precision < 1)', () => {
|
||||||
|
const run = [
|
||||||
|
{ file: 'a.mjs', line: 10, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' },
|
||||||
|
{ file: 'b.mjs', line: 0, rule_key: 'MISSING_TEST', severity: 'MAJOR' },
|
||||||
|
{ file: 'c.mjs', line: 3, rule_key: 'PLACEHOLDER_IN_CODE', severity: 'MAJOR' },
|
||||||
|
];
|
||||||
|
const s = scoreFindings(run, GOLD);
|
||||||
|
assert.equal(s.tp, 2);
|
||||||
|
assert.equal(s.fp, 1);
|
||||||
|
assert.equal(s.fn, 0);
|
||||||
|
assert.equal(s.recall, 1);
|
||||||
|
assert.equal(s.precision, 2 / 3);
|
||||||
|
assert.deepEqual(s.spurious, ['c.mjs PLACEHOLDER_IN_CODE']);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scoreFindings — mixed FN + FP', () => {
|
||||||
|
const run = [
|
||||||
|
{ file: 'a.mjs', line: 10, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' }, // match
|
||||||
|
{ file: 'c.mjs', line: 3, rule_key: 'PLACEHOLDER_IN_CODE', severity: 'MAJOR' }, // spurious
|
||||||
|
];
|
||||||
|
const s = scoreFindings(run, GOLD);
|
||||||
|
assert.equal(s.tp, 1);
|
||||||
|
assert.equal(s.fp, 1);
|
||||||
|
assert.equal(s.fn, 1);
|
||||||
|
assert.equal(s.precision, 0.5);
|
||||||
|
assert.equal(s.recall, 0.5);
|
||||||
|
assert.equal(s.f1, 0.5);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scoreFindings — duplicate (file,rule_key) pairs collapse (set semantics)', () => {
|
||||||
|
const run = [
|
||||||
|
{ file: 'a.mjs', line: 10, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' },
|
||||||
|
{ file: 'a.mjs', line: 22, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' }, // same pair
|
||||||
|
{ file: 'b.mjs', line: 0, rule_key: 'MISSING_TEST', severity: 'MAJOR' },
|
||||||
|
];
|
||||||
|
const s = scoreFindings(run, GOLD);
|
||||||
|
assert.equal(s.tp, 2);
|
||||||
|
assert.equal(s.fp, 0);
|
||||||
|
assert.equal(s.fn, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scoreFindings — empty run: recall 0 (nothing found), precision vacuously 1, f1 0', () => {
|
||||||
|
const s = scoreFindings([], GOLD);
|
||||||
|
assert.equal(s.tp, 0);
|
||||||
|
assert.equal(s.fp, 0);
|
||||||
|
assert.equal(s.fn, 2);
|
||||||
|
assert.equal(s.recall, 0);
|
||||||
|
assert.equal(s.precision, 1);
|
||||||
|
assert.equal(s.f1, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scoreFindings — empty gold: precision 0 (all spurious), recall vacuously 1, f1 0', () => {
|
||||||
|
const run = [{ file: 'a.mjs', line: 10, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' }];
|
||||||
|
const s = scoreFindings(run, []);
|
||||||
|
assert.equal(s.tp, 0);
|
||||||
|
assert.equal(s.fp, 1);
|
||||||
|
assert.equal(s.fn, 0);
|
||||||
|
assert.equal(s.precision, 0);
|
||||||
|
assert.equal(s.recall, 1);
|
||||||
|
assert.equal(s.f1, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scoreFindings — both empty: precision/recall/f1 = 1 (matched nothing perfectly)', () => {
|
||||||
|
const s = scoreFindings([], []);
|
||||||
|
assert.equal(s.precision, 1);
|
||||||
|
assert.equal(s.recall, 1);
|
||||||
|
assert.equal(s.f1, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scoreFindings — tolerates null/undefined finding arrays', () => {
|
||||||
|
const s = scoreFindings(null, undefined);
|
||||||
|
assert.equal(s.tp, 0);
|
||||||
|
assert.equal(s.fp, 0);
|
||||||
|
assert.equal(s.fn, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scoreVerdict — exact verdict match is true, mismatch is false', () => {
|
||||||
|
assert.equal(scoreVerdict('BLOCK', 'BLOCK'), true);
|
||||||
|
assert.equal(scoreVerdict('WARN', 'BLOCK'), false);
|
||||||
|
assert.equal(scoreVerdict('ALLOW', 'ALLOW'), true);
|
||||||
|
});
|
||||||
|
|
@ -50,7 +50,7 @@ test('resolvePhaseSignal — defensive: null/non-object input returns null', ()
|
||||||
|
|
||||||
test('resolvePhaseSignal — drops model not in BASE_ALLOWED_MODELS (defense-in-depth gate)', () => {
|
test('resolvePhaseSignal — drops model not in BASE_ALLOWED_MODELS (defense-in-depth gate)', () => {
|
||||||
// MAJOR fix (S4): line that copies `model` must gate against the same
|
// MAJOR fix (S4): line that copies `model` must gate against the same
|
||||||
// allowlist brief-validator uses (BASE_ALLOWED_MODELS = ['sonnet','opus']),
|
// allowlist brief-validator uses (BASE_ALLOWED_MODELS = ['sonnet','opus','fable']),
|
||||||
// mirroring how effort is gated against EFFORT_LEVELS. A brief that slipped
|
// mirroring how effort is gated against EFFORT_LEVELS. A brief that slipped
|
||||||
// validation (hand-edited, validation skipped) must not hand a junk model
|
// validation (hand-edited, validation skipped) must not hand a junk model
|
||||||
// string to a command that then spawns an agent with `model: <junk>`.
|
// string to a command that then spawns an agent with `model: <junk>`.
|
||||||
|
|
@ -70,15 +70,17 @@ test('resolvePhaseSignal — drops model not in BASE_ALLOWED_MODELS (defense-in-
|
||||||
assert.ok(!('model' in review), 'model key absent for haiku');
|
assert.ok(!('model' in review), 'model key absent for haiku');
|
||||||
});
|
});
|
||||||
|
|
||||||
test('resolvePhaseSignal — keeps valid models (sonnet, opus) after gating', () => {
|
test('resolvePhaseSignal — keeps valid models (sonnet, opus, fable) after gating', () => {
|
||||||
const fm = {
|
const fm = {
|
||||||
phase_signals: [
|
phase_signals: [
|
||||||
{ phase: 'research', effort: 'low', model: 'sonnet' },
|
{ phase: 'research', effort: 'low', model: 'sonnet' },
|
||||||
{ phase: 'execute', effort: 'high', model: 'opus' },
|
{ phase: 'execute', effort: 'high', model: 'opus' },
|
||||||
|
{ phase: 'review', effort: 'high', model: 'fable' },
|
||||||
],
|
],
|
||||||
};
|
};
|
||||||
assert.equal(resolvePhaseSignal(fm, 'research').model, 'sonnet');
|
assert.equal(resolvePhaseSignal(fm, 'research').model, 'sonnet');
|
||||||
assert.equal(resolvePhaseSignal(fm, 'execute').model, 'opus');
|
assert.equal(resolvePhaseSignal(fm, 'execute').model, 'opus');
|
||||||
|
assert.equal(resolvePhaseSignal(fm, 'review').model, 'fable');
|
||||||
});
|
});
|
||||||
|
|
||||||
test('resolvePhaseSignalFromFile + CLI shim — writes JSON to stdout, exit 0', () => {
|
test('resolvePhaseSignalFromFile + CLI shim — writes JSON to stdout, exit 0', () => {
|
||||||
|
|
|
||||||
|
|
@ -50,6 +50,20 @@ test('SC #5: loadProfile("premium") returns all-opus', () => {
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('SC #5: loadProfile("fable") returns all-fable (BUILTIN_NAMES canary)', () => {
|
||||||
|
// Throws PROFILE_NOT_FOUND if 'fable' regresses out of BUILTIN_NAMES —
|
||||||
|
// the silent-fallback-to-premium failure mode this pin exists to catch.
|
||||||
|
const p = loadProfile('fable');
|
||||||
|
assert.equal(p.name, 'fable');
|
||||||
|
for (const phase of ['brief', 'research', 'plan', 'execute', 'review', 'continue']) {
|
||||||
|
assert.equal(p.phase_models[phase], 'fable', `fable ${phase} should be fable`);
|
||||||
|
}
|
||||||
|
assert.equal(p.parallel_agents_min, 6);
|
||||||
|
assert.equal(p.parallel_agents_max, 8);
|
||||||
|
assert.equal(p.external_research_enabled, true);
|
||||||
|
assert.equal(p.brief_reviewer_iter_cap, 3);
|
||||||
|
});
|
||||||
|
|
||||||
test('SC #5: loadProfile throws PROFILE_NOT_FOUND for unknown profile', () => {
|
test('SC #5: loadProfile throws PROFILE_NOT_FOUND for unknown profile', () => {
|
||||||
try {
|
try {
|
||||||
loadProfile('does-not-exist-xyz');
|
loadProfile('does-not-exist-xyz');
|
||||||
|
|
|
||||||
|
|
@ -60,3 +60,30 @@ test('resolvePhaseModel — Case 6 (defensive): null briefPath falls through to
|
||||||
assert.equal(r.model, 'opus', 'premium.plan default = opus');
|
assert.equal(r.model, 'opus', 'premium.plan default = opus');
|
||||||
assert.equal(r.source, 'default');
|
assert.equal(r.source, 'default');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// v5.9 — fable profile composition (brief SC 4) + effort passthrough coherence
|
||||||
|
|
||||||
|
test('resolvePhaseModel — --profile fable: all six phases resolve model fable (no brief signal)', () => {
|
||||||
|
for (const phase of ['brief', 'research', 'plan', 'execute', 'review', 'continue']) {
|
||||||
|
const r = resolvePhaseModel(phase, null, ['--profile', 'fable'], {});
|
||||||
|
assert.equal(r.model, 'fable', `fable.${phase} should be fable; got ${JSON.stringify(r)}`);
|
||||||
|
assert.equal(r.source, 'flag');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('resolvePhaseModel — brief signal (opus) beats --profile fable (composition precedence pin)', () => {
|
||||||
|
// brief-effort-high.md pins execute to model: opus. Brief must beat the fable profile.
|
||||||
|
const r = resolvePhaseModel('execute', FIXTURE('brief-effort-high.md'), ['--profile', 'fable'], {});
|
||||||
|
assert.equal(r.model, 'opus', `brief signal should beat fable profile; got ${JSON.stringify(r)}`);
|
||||||
|
assert.equal(r.source, 'brief-signal');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('resolvePhaseModel — composed output carries {effort, model} atomically (v5.9 passthrough)', () => {
|
||||||
|
// brief-effort-high.md: execute → {effort: high, model: opus}. The composed
|
||||||
|
// resolver must return BOTH fields from one call — the split-CLI design this
|
||||||
|
// passthrough replaced could drift effort and model apart.
|
||||||
|
const r = resolvePhaseModel('execute', FIXTURE('brief-effort-high.md'), [], {});
|
||||||
|
assert.equal(r.effort, 'high', `effort must pass through; got ${JSON.stringify(r)}`);
|
||||||
|
assert.equal(r.model, 'opus');
|
||||||
|
assert.equal(r.source, 'brief-signal');
|
||||||
|
});
|
||||||
|
|
|
||||||
|
|
@ -15,20 +15,21 @@ import { censusTests } from '../../lib/util/test-census.mjs';
|
||||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
const TESTS_ROOT = join(HERE, '..');
|
const TESTS_ROOT = join(HERE, '..');
|
||||||
|
|
||||||
test('suite census splits behavior tests from doc-consistency pins (S19)', (t) => {
|
test('suite census splits behavior / doc-pins / gold-eval (S19 + SKAL-1·4b)', (t) => {
|
||||||
const c = censusTests(TESTS_ROOT);
|
const c = censusTests(TESTS_ROOT);
|
||||||
// Honest-count invariant: the two buckets must account for every top-level
|
// Honest-count invariant: the three buckets must account for every top-level
|
||||||
// test() declaration — no silent drift between behavior and pin counts.
|
// test() declaration — no silent drift between behavior, pin, and eval counts.
|
||||||
assert.equal(c.behavior + c.docPins, c.total,
|
assert.equal(c.behavior + c.docPins + c.goldEval, c.total,
|
||||||
'census buckets must sum to the total declaration count');
|
'census buckets must sum to the total declaration count');
|
||||||
assert.ok(c.docPins > 0, 'doc-consistency pin bucket must be non-empty (regex/glob sanity)');
|
assert.ok(c.docPins > 0, 'doc-consistency pin bucket must be non-empty (regex/glob sanity)');
|
||||||
assert.ok(c.behavior > 0, 'behavior bucket must be non-empty (regex/glob sanity)');
|
assert.ok(c.behavior > 0, 'behavior bucket must be non-empty (regex/glob sanity)');
|
||||||
|
assert.ok(c.goldEval > 0, 'gold-eval scoring-run bucket must be non-empty (SKAL-1·4b present)');
|
||||||
// Report the split so the cited count is honest (audit §Top changes #8).
|
// Report the split so the cited count is honest (audit §Top changes #8).
|
||||||
// Metric = top-level test() declarations; node:test's runtime total counts
|
// Metric = top-level test() declarations; node:test's runtime total counts
|
||||||
// subtests too and is therefore ≥ this number.
|
// subtests too and is therefore ≥ this number.
|
||||||
t.diagnostic(
|
t.diagnostic(
|
||||||
`behavior=${c.behavior} doc-consistency-pins=${c.docPins} ` +
|
`behavior=${c.behavior} doc-consistency-pins=${c.docPins} ` +
|
||||||
`total=${c.total} (top-level test() declarations)`,
|
`gold-eval=${c.goldEval} total=${c.total} (top-level test() declarations)`,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
|
||||||
188
tests/lib/token-usage.test.mjs
Normal file
188
tests/lib/token-usage.test.mjs
Normal file
|
|
@ -0,0 +1,188 @@
|
||||||
|
// tests/lib/token-usage.test.mjs
|
||||||
|
// SKAL-2 Step 1 — pure token-usage parser + cache-aware cost derivation.
|
||||||
|
//
|
||||||
|
// Hermetic: no LLM, network, time, or randomness. Fixture transcript exercises
|
||||||
|
// the dedup-by-requestId (streaming-placeholder) and isSidechain-exclusion
|
||||||
|
// paths that were verified against a real local transcript at execute time.
|
||||||
|
// Idiom: tests/lib/coordinator-contract.test.mjs (pure-module node:test style).
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { readFileSync } from 'node:fs';
|
||||||
|
import { join, dirname } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import {
|
||||||
|
parseTranscriptUsage,
|
||||||
|
deriveCost,
|
||||||
|
buildRecord,
|
||||||
|
upsertSessionRecord,
|
||||||
|
lastMainChainModel,
|
||||||
|
PRICE_TABLE,
|
||||||
|
PRICE_TABLE_VERSION,
|
||||||
|
} from '../../lib/stats/token-usage.mjs';
|
||||||
|
|
||||||
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const FIXTURE = join(HERE, '..', 'fixtures', 'token-usage', 'sample-transcript.jsonl');
|
||||||
|
|
||||||
|
// ---- parseTranscriptUsage --------------------------------------------------
|
||||||
|
|
||||||
|
test('parseTranscriptUsage — sums main-chain final-per-requestId; excludes sidechain + placeholder dup', () => {
|
||||||
|
const text = readFileSync(FIXTURE, 'utf-8');
|
||||||
|
const totals = parseTranscriptUsage(text);
|
||||||
|
// req_A: placeholder {1,1,0,0} then real {1000,200,400,5000} → LAST wins.
|
||||||
|
// req_B: appears twice identically → dedup → counted once {2000,300,600,1000}.
|
||||||
|
// req_SIDE (isSidechain:true, all 9999) → EXCLUDED. user/system/malformed → skipped.
|
||||||
|
assert.deepEqual(totals, {
|
||||||
|
tokens_input: 3000, // 1000 + 2000 (NOT 1 — placeholder dropped)
|
||||||
|
tokens_output: 500, // 200 + 300
|
||||||
|
tokens_cache_creation: 1000, // 400 + 600
|
||||||
|
tokens_cache_read: 6000, // 5000 + 1000
|
||||||
|
});
|
||||||
|
// Sidechain 9999s must not leak into any field.
|
||||||
|
assert.ok(totals.tokens_input < 9999, 'sidechain tokens leaked into total');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('parseTranscriptUsage — empty / nullish input → zeroed totals', () => {
|
||||||
|
const zero = { tokens_input: 0, tokens_output: 0, tokens_cache_creation: 0, tokens_cache_read: 0 };
|
||||||
|
assert.deepEqual(parseTranscriptUsage(''), zero);
|
||||||
|
assert.deepEqual(parseTranscriptUsage(null), zero);
|
||||||
|
assert.deepEqual(parseTranscriptUsage(undefined), zero);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('parseTranscriptUsage — single-pass O(n) shape on a 500-line transcript', () => {
|
||||||
|
// 500 distinct-requestId main-chain assistant records, each input_tokens:10.
|
||||||
|
const lines = [];
|
||||||
|
for (let i = 0; i < 500; i++) {
|
||||||
|
lines.push(JSON.stringify({
|
||||||
|
type: 'assistant',
|
||||||
|
isSidechain: false,
|
||||||
|
requestId: `req_${i}`,
|
||||||
|
message: { model: 'claude-opus-4-8', usage: { input_tokens: 10, output_tokens: 0, cache_creation_input_tokens: 0, cache_read_input_tokens: 0 } },
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
const totals = parseTranscriptUsage(lines.join('\n'));
|
||||||
|
assert.equal(totals.tokens_input, 5000); // 500 × 10
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- deriveCost ------------------------------------------------------------
|
||||||
|
|
||||||
|
test('deriveCost — hand-computed value for a known model (cache-aware)', () => {
|
||||||
|
const totals = { tokens_input: 3000, tokens_output: 500, tokens_cache_creation: 1000, tokens_cache_read: 6000 };
|
||||||
|
const { cost_usd, is_estimate } = deriveCost(totals, 'claude-opus-4-8');
|
||||||
|
// (3000×5 + 500×25 + 1000×6.25 + 6000×0.5) / 1e6
|
||||||
|
// = (15000 + 12500 + 6250 + 3000) / 1e6 = 36750 / 1e6 = 0.03675
|
||||||
|
assert.equal(cost_usd, 0.03675);
|
||||||
|
assert.equal(is_estimate, false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('deriveCost — hand-computed value for claude-fable-5 (v5.9 fable tier)', () => {
|
||||||
|
const totals = { tokens_input: 3000, tokens_output: 500, tokens_cache_creation: 1000, tokens_cache_read: 6000 };
|
||||||
|
const { cost_usd, is_estimate } = deriveCost(totals, 'claude-fable-5');
|
||||||
|
// (3000×10 + 500×50 + 1000×12.5 + 6000×1) / 1e6
|
||||||
|
// = (30000 + 25000 + 12500 + 6000) / 1e6 = 73500 / 1e6 = 0.0735
|
||||||
|
assert.equal(cost_usd, 0.0735);
|
||||||
|
assert.equal(is_estimate, false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('deriveCost — refuse-to-estimate for an unknown model', () => {
|
||||||
|
const totals = { tokens_input: 3000, tokens_output: 500, tokens_cache_creation: 1000, tokens_cache_read: 6000 };
|
||||||
|
const out = deriveCost(totals, 'claude-unknown-9');
|
||||||
|
assert.equal(out.cost_usd, null);
|
||||||
|
assert.equal(out.is_estimate, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('PRICE_TABLE — frozen, opus-4-8 carries all five rate fields', () => {
|
||||||
|
assert.ok(Object.isFrozen(PRICE_TABLE), 'PRICE_TABLE must be frozen (drift-pin)');
|
||||||
|
const p = PRICE_TABLE['claude-opus-4-8'];
|
||||||
|
for (const k of ['input', 'output', 'cache_read', 'cache_write_5m', 'cache_write_1h']) {
|
||||||
|
assert.equal(typeof p[k], 'number', `missing rate field ${k}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- buildRecord -----------------------------------------------------------
|
||||||
|
|
||||||
|
test('buildRecord — stamps scope:main-context + price_table_version + flat numerics', () => {
|
||||||
|
const totals = { tokens_input: 3000, tokens_output: 500, tokens_cache_creation: 1000, tokens_cache_read: 6000 };
|
||||||
|
const rec = buildRecord({ sessionId: 'S1', model: 'claude-opus-4-8', totals, now: '2026-06-26T12:00:00Z' });
|
||||||
|
assert.equal(rec.scope, 'main-context');
|
||||||
|
assert.equal(rec.price_table_version, PRICE_TABLE_VERSION);
|
||||||
|
assert.equal(rec.ts, '2026-06-26T12:00:00Z');
|
||||||
|
assert.equal(rec.session_id, 'S1');
|
||||||
|
assert.equal(rec.model, 'claude-opus-4-8');
|
||||||
|
assert.equal(rec.tokens_input, 3000);
|
||||||
|
assert.equal(rec.cost_usd, 0.03675);
|
||||||
|
assert.equal(rec.is_estimate, false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('buildRecord — unknown model → cost_usd null + is_estimate true (honesty contract)', () => {
|
||||||
|
const rec = buildRecord({ sessionId: 'S9', model: 'mystery', totals: { tokens_input: 1 }, now: 't' });
|
||||||
|
assert.equal(rec.cost_usd, null);
|
||||||
|
assert.equal(rec.is_estimate, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- upsertSessionRecord ---------------------------------------------------
|
||||||
|
|
||||||
|
test('upsertSessionRecord — REPLACES a same-session line (not append)', () => {
|
||||||
|
const existing =
|
||||||
|
JSON.stringify({ session_id: 'S1', tokens_input: 1, scope: 'main-context' }) + '\n' +
|
||||||
|
JSON.stringify({ session_id: 'S2', tokens_input: 2, scope: 'main-context' }) + '\n';
|
||||||
|
const rec = buildRecord({ sessionId: 'S1', model: 'claude-opus-4-8', totals: { tokens_input: 999 }, now: 't' });
|
||||||
|
const out = upsertSessionRecord(existing, rec);
|
||||||
|
const recs = out.trim().split('\n').map(JSON.parse);
|
||||||
|
assert.equal(recs.length, 2, 'replace must NOT add a line');
|
||||||
|
const s1 = recs.find(r => r.session_id === 'S1');
|
||||||
|
assert.equal(s1.tokens_input, 999, 'S1 must carry the new totals');
|
||||||
|
assert.ok(recs.some(r => r.session_id === 'S2'), 'S2 must be preserved');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('upsertSessionRecord — APPENDS a new-session line', () => {
|
||||||
|
const existing = JSON.stringify({ session_id: 'S1', tokens_input: 1 }) + '\n';
|
||||||
|
const rec = buildRecord({ sessionId: 'S3', model: 'claude-opus-4-8', totals: { tokens_input: 7 }, now: 't' });
|
||||||
|
const out = upsertSessionRecord(existing, rec);
|
||||||
|
const recs = out.trim().split('\n').map(JSON.parse);
|
||||||
|
assert.equal(recs.length, 2, 'new session must append');
|
||||||
|
assert.ok(recs.some(r => r.session_id === 'S3' && r.tokens_input === 7));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('upsertSessionRecord — empty file → single record, trailing newline', () => {
|
||||||
|
const rec = buildRecord({ sessionId: 'S1', model: 'claude-opus-4-8', totals: { tokens_input: 5 }, now: 't' });
|
||||||
|
const out = upsertSessionRecord('', rec);
|
||||||
|
assert.ok(out.endsWith('\n'));
|
||||||
|
assert.deepEqual(out.trim().split('\n').map(JSON.parse).length, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- lastMainChainModel ----------------------------------------------------
|
||||||
|
|
||||||
|
test('lastMainChainModel — last main-chain model wins; sidechain excluded even when last', () => {
|
||||||
|
// Two distinct main-chain models (opus then sonnet) → sonnet (last) wins.
|
||||||
|
// A sidechain record sits AFTER sonnet: if exclusion broke it would win;
|
||||||
|
// if last-wins broke, opus (first) would win. Correct answer pins both.
|
||||||
|
const text = [
|
||||||
|
JSON.stringify({ type: 'assistant', isSidechain: false, message: { model: 'claude-opus-4-8', usage: {} } }),
|
||||||
|
JSON.stringify({ type: 'assistant', isSidechain: false, message: { model: 'claude-sonnet-4-6', usage: {} } }),
|
||||||
|
JSON.stringify({ type: 'assistant', isSidechain: true, message: { model: 'claude-sidechain-ZZZ', usage: {} } }),
|
||||||
|
JSON.stringify({ type: 'user', message: { content: 'ignored' } }),
|
||||||
|
].join('\n');
|
||||||
|
assert.equal(lastMainChainModel(text), 'claude-sonnet-4-6');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('lastMainChainModel — no main-chain model → null → deriveCost refuses to estimate', () => {
|
||||||
|
// Only a sidechain model, a user line, and a malformed line: no main-chain model.
|
||||||
|
const text = [
|
||||||
|
JSON.stringify({ type: 'assistant', isSidechain: true, message: { model: 'claude-opus-4-8', usage: {} } }),
|
||||||
|
JSON.stringify({ type: 'user', message: { content: 'hi' } }),
|
||||||
|
'not json',
|
||||||
|
].join('\n');
|
||||||
|
const model = lastMainChainModel(text);
|
||||||
|
assert.equal(model, null);
|
||||||
|
// null model must propagate to refuse-to-estimate (honesty contract).
|
||||||
|
const { cost_usd, is_estimate } = deriveCost({ tokens_input: 1000 }, model);
|
||||||
|
assert.equal(cost_usd, null);
|
||||||
|
assert.equal(is_estimate, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('lastMainChainModel — empty / nullish input → null', () => {
|
||||||
|
assert.equal(lastMainChainModel(''), null);
|
||||||
|
assert.equal(lastMainChainModel(null), null);
|
||||||
|
assert.equal(lastMainChainModel(undefined), null);
|
||||||
|
});
|
||||||
|
|
@ -14,6 +14,14 @@ ramp_target: v4.2
|
||||||
|
|
||||||
## Status: PARKED-SYNTHETIC
|
## Status: PARKED-SYNTHETIC
|
||||||
|
|
||||||
|
> **Profile linkage (S34/V30).** Because this calibration is still
|
||||||
|
> parked-synthetic, `lib/profiles/economy.yaml` carries `experimental: true`
|
||||||
|
> and every profile doc flags the `economy` row experimental. When the
|
||||||
|
> empirical calibration in *How to replace* below lands (`status: empirical`),
|
||||||
|
> drop `experimental: true` in the **same change** — the linkage is guarded by
|
||||||
|
> `tests/lib/doc-consistency.test.mjs` ("economy experimental flag agrees with
|
||||||
|
> parked-synthetic calibration status").
|
||||||
|
|
||||||
Empirical Jaccard calibration was deferred from v4.1 because the four
|
Empirical Jaccard calibration was deferred from v4.1 because the four
|
||||||
required `/trekplan` invocations cost an estimated $60-120 of LLM-budget
|
required `/trekplan` invocations cost an estimated $60-120 of LLM-budget
|
||||||
that was not authorized for the v4.1-execute-4b session. Per Step 17
|
that was not authorized for the v4.1-execute-4b session. Per Step 17
|
||||||
|
|
|
||||||
244
tests/validators/brief-gate-coverage.test.mjs
Normal file
244
tests/validators/brief-gate-coverage.test.mjs
Normal file
|
|
@ -0,0 +1,244 @@
|
||||||
|
// tests/validators/brief-gate-coverage.test.mjs
|
||||||
|
// SKAL-1·4a — two-sided (class-balanced) coverage for every error-severity
|
||||||
|
// BRIEF_* BLOCKER the brief-validator can emit.
|
||||||
|
//
|
||||||
|
// brief-validator.test.mjs already tests many codes; several were FAILURE-only
|
||||||
|
// (only the raising case asserted) and three were untested. This meta-test pins
|
||||||
|
// a class-balanced pair { fail, pass } for EACH in-scope BLOCKER and a parity
|
||||||
|
// guard so a code can't sit in IN_SCOPE_CODES without a paired case. It removes
|
||||||
|
// the one-sided-optimization risk programmatically, not by inspection.
|
||||||
|
//
|
||||||
|
// In scope = ERROR-severity BRIEF_* codes the validator pushes to errors[]
|
||||||
|
// (BLOCKERs). EXCLUDED: warning-severity codes (BRIEF_TLDR_TOO_LONG,
|
||||||
|
// BRIEF_VERSION_FORMAT, BRIEF_VERSION_BELOW_MINIMUM, BRIEF_PARTIAL_SKIPPED) — a
|
||||||
|
// "passing" counterpart to a warning is not a BLOCKER concern — and the
|
||||||
|
// file-level BRIEF_NOT_FOUND / BRIEF_READ_ERROR which live in validateBrief()
|
||||||
|
// (filesystem), not validateBriefContent().
|
||||||
|
//
|
||||||
|
// LIMITATION: brief-validator.mjs does not export its code set (codes are string
|
||||||
|
// literals inside issue(...)), so IN_SCOPE_CODES is HAND-MAINTAINED. A NEW
|
||||||
|
// validator BLOCKER must be added here manually; the parity guard catches
|
||||||
|
// list/case drift, not validator drift. (Operator chose the hand-maintained list
|
||||||
|
// over exporting BRIEF_ERROR_CODES from the forbidden_paths-guarded validator.)
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { validateBriefContent } from '../../lib/validators/brief-validator.mjs';
|
||||||
|
|
||||||
|
// ---- valid bases (mirror tests/validators/brief-validator.test.mjs) ----------
|
||||||
|
|
||||||
|
const GOOD_BRIEF = `---
|
||||||
|
type: trekbrief
|
||||||
|
brief_version: "2.0"
|
||||||
|
created: 2026-04-30
|
||||||
|
task: "Add JWT auth to API"
|
||||||
|
slug: jwt-auth
|
||||||
|
project_dir: .claude/projects/2026-04-30-jwt-auth/
|
||||||
|
research_topics: 2
|
||||||
|
research_status: pending
|
||||||
|
auto_research: false
|
||||||
|
interview_turns: 5
|
||||||
|
source: interview
|
||||||
|
---
|
||||||
|
|
||||||
|
# Task: JWT auth
|
||||||
|
|
||||||
|
## Intent
|
||||||
|
|
||||||
|
Why this matters.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
What success looks like.
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
- All tests pass.
|
||||||
|
`;
|
||||||
|
|
||||||
|
const GOOD_BRIEF_22 = `---
|
||||||
|
type: trekbrief
|
||||||
|
brief_version: "2.2"
|
||||||
|
created: 2026-06-18
|
||||||
|
task: "Add JWT auth to API"
|
||||||
|
slug: jwt-auth
|
||||||
|
project_dir: .claude/projects/2026-06-18-jwt-auth/
|
||||||
|
research_topics: 0
|
||||||
|
research_status: complete
|
||||||
|
auto_research: false
|
||||||
|
interview_turns: 5
|
||||||
|
source: interview
|
||||||
|
framing: new-direction
|
||||||
|
phase_signals_partial: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# Task: JWT auth
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
Net-new JWT auth; no prior brief to anchor against.
|
||||||
|
|
||||||
|
## Intent
|
||||||
|
|
||||||
|
Why this matters.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
What success looks like.
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
- All tests pass.
|
||||||
|
`;
|
||||||
|
|
||||||
|
const REVIEW_AS_BRIEF = `---
|
||||||
|
type: trekreview
|
||||||
|
task: "Review delivered trekreview v1.0"
|
||||||
|
slug: trekreview
|
||||||
|
project_dir: .claude/projects/2026-05-01-trekreview/
|
||||||
|
findings:
|
||||||
|
- 0123456789abcdef0123456789abcdef01234567
|
||||||
|
- fedcba9876543210fedcba9876543210fedcba98
|
||||||
|
---
|
||||||
|
|
||||||
|
# Review brief
|
||||||
|
|
||||||
|
## Intent
|
||||||
|
|
||||||
|
Adversarial review of delivered trekreview v1.0.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Find what was missed.
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
- All BLOCKER findings get a fix-plan.
|
||||||
|
`;
|
||||||
|
|
||||||
|
// signals blocks
|
||||||
|
const GOOD_SIGNALS = `phase_signals:
|
||||||
|
- phase: plan
|
||||||
|
effort: high
|
||||||
|
model: opus
|
||||||
|
`;
|
||||||
|
const BAD_PHASE_SIGNALS = `phase_signals:
|
||||||
|
- phase: nonsense
|
||||||
|
effort: high
|
||||||
|
`;
|
||||||
|
const BAD_EFFORT_SIGNALS = `phase_signals:
|
||||||
|
- phase: plan
|
||||||
|
effort: turbo
|
||||||
|
`;
|
||||||
|
const BAD_MODEL_SIGNALS = `phase_signals:
|
||||||
|
- phase: plan
|
||||||
|
effort: high
|
||||||
|
model: gpt4
|
||||||
|
`;
|
||||||
|
|
||||||
|
// convenience builders
|
||||||
|
const v21 = (s) => s.replace('brief_version: "2.0"', 'brief_version: "2.1"');
|
||||||
|
const withLine = (s, line) => s.replace('source: interview\n', `source: interview\n${line}`);
|
||||||
|
|
||||||
|
// ---- the canonical in-scope set + a class-balanced pair for each -------------
|
||||||
|
|
||||||
|
const IN_SCOPE_CODES = [
|
||||||
|
'BRIEF_MISSING_FIELD',
|
||||||
|
'BRIEF_BAD_STATUS',
|
||||||
|
'BRIEF_STATE_INCOHERENT',
|
||||||
|
'BRIEF_WRONG_TYPE',
|
||||||
|
'BRIEF_BAD_FINDINGS_TYPE',
|
||||||
|
'BRIEF_MISSING_SECTION',
|
||||||
|
'BRIEF_INVALID_FRAMING',
|
||||||
|
'BRIEF_MISSING_FRAMING',
|
||||||
|
'BRIEF_V51_MISSING_SIGNALS',
|
||||||
|
'BRIEF_SIGNALS_MUTUALLY_EXCLUSIVE',
|
||||||
|
'BRIEF_INVALID_PHASE_SIGNALS',
|
||||||
|
'BRIEF_INVALID_PHASE_SIGNAL_PHASE',
|
||||||
|
'BRIEF_INVALID_EFFORT',
|
||||||
|
'BRIEF_INVALID_MODEL',
|
||||||
|
];
|
||||||
|
|
||||||
|
const CASES = {
|
||||||
|
BRIEF_MISSING_FIELD: {
|
||||||
|
fail: GOOD_BRIEF.replace(/^research_topics: 2\n/m, ''),
|
||||||
|
pass: GOOD_BRIEF,
|
||||||
|
},
|
||||||
|
BRIEF_BAD_STATUS: {
|
||||||
|
fail: GOOD_BRIEF.replace('research_status: pending', 'research_status: maybe'),
|
||||||
|
pass: GOOD_BRIEF,
|
||||||
|
},
|
||||||
|
BRIEF_STATE_INCOHERENT: {
|
||||||
|
fail: GOOD_BRIEF.replace('research_status: pending', 'research_status: skipped'),
|
||||||
|
pass: GOOD_BRIEF,
|
||||||
|
},
|
||||||
|
BRIEF_WRONG_TYPE: {
|
||||||
|
fail: GOOD_BRIEF.replace('type: trekbrief', 'type: notabrief'),
|
||||||
|
pass: GOOD_BRIEF,
|
||||||
|
},
|
||||||
|
BRIEF_BAD_FINDINGS_TYPE: {
|
||||||
|
fail: REVIEW_AS_BRIEF.replace(/findings:\n - 0123[\s\S]*?- fedcba[0-9a-f]+/, 'findings: not-an-array'),
|
||||||
|
pass: REVIEW_AS_BRIEF,
|
||||||
|
},
|
||||||
|
BRIEF_MISSING_SECTION: {
|
||||||
|
fail: GOOD_BRIEF.replace(/## Intent\n\nWhy this matters\.\n\n/, ''),
|
||||||
|
pass: GOOD_BRIEF,
|
||||||
|
},
|
||||||
|
BRIEF_INVALID_FRAMING: {
|
||||||
|
fail: withLine(GOOD_BRIEF, 'framing: sideways\n'),
|
||||||
|
pass: GOOD_BRIEF_22,
|
||||||
|
},
|
||||||
|
BRIEF_MISSING_FRAMING: {
|
||||||
|
fail: GOOD_BRIEF_22.replace('framing: new-direction\n', ''),
|
||||||
|
pass: GOOD_BRIEF_22,
|
||||||
|
},
|
||||||
|
BRIEF_V51_MISSING_SIGNALS: {
|
||||||
|
fail: v21(GOOD_BRIEF),
|
||||||
|
pass: GOOD_BRIEF, // 2.0 — gate inactive
|
||||||
|
},
|
||||||
|
BRIEF_SIGNALS_MUTUALLY_EXCLUSIVE: {
|
||||||
|
fail: withLine(v21(GOOD_BRIEF), `phase_signals_partial: true\n${GOOD_SIGNALS}`),
|
||||||
|
pass: withLine(v21(GOOD_BRIEF), 'phase_signals_partial: true\n'),
|
||||||
|
},
|
||||||
|
BRIEF_INVALID_PHASE_SIGNALS: {
|
||||||
|
fail: withLine(GOOD_BRIEF, 'phase_signals: notalist\n'),
|
||||||
|
pass: withLine(v21(GOOD_BRIEF), GOOD_SIGNALS),
|
||||||
|
},
|
||||||
|
BRIEF_INVALID_PHASE_SIGNAL_PHASE: {
|
||||||
|
fail: withLine(v21(GOOD_BRIEF), BAD_PHASE_SIGNALS),
|
||||||
|
pass: withLine(v21(GOOD_BRIEF), GOOD_SIGNALS),
|
||||||
|
},
|
||||||
|
BRIEF_INVALID_EFFORT: {
|
||||||
|
fail: withLine(v21(GOOD_BRIEF), BAD_EFFORT_SIGNALS),
|
||||||
|
pass: withLine(v21(GOOD_BRIEF), GOOD_SIGNALS),
|
||||||
|
},
|
||||||
|
BRIEF_INVALID_MODEL: {
|
||||||
|
fail: withLine(v21(GOOD_BRIEF), BAD_MODEL_SIGNALS),
|
||||||
|
pass: withLine(v21(GOOD_BRIEF), GOOD_SIGNALS),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
test('gate coverage — CASES keys exactly match IN_SCOPE_CODES (no silent gap)', () => {
|
||||||
|
assert.deepEqual(Object.keys(CASES).sort(), [...IN_SCOPE_CODES].sort());
|
||||||
|
});
|
||||||
|
|
||||||
|
test('gate coverage — every in-scope BRIEF_* BLOCKER raises on its fail case', () => {
|
||||||
|
for (const code of IN_SCOPE_CODES) {
|
||||||
|
const r = validateBriefContent(CASES[code].fail, { strict: true });
|
||||||
|
assert.ok(
|
||||||
|
r.errors.find((e) => e.code === code),
|
||||||
|
`${code}: fail case did NOT raise it; errors=${JSON.stringify(r.errors.map((e) => e.code))}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('gate coverage — every in-scope BRIEF_* BLOCKER is absent on its (valid) pass case', () => {
|
||||||
|
for (const code of IN_SCOPE_CODES) {
|
||||||
|
const r = validateBriefContent(CASES[code].pass, { strict: true });
|
||||||
|
assert.ok(
|
||||||
|
!r.errors.find((e) => e.code === code),
|
||||||
|
`${code}: pass case wrongly raised it; errors=${JSON.stringify(r.errors.map((e) => e.code))}`,
|
||||||
|
);
|
||||||
|
assert.equal(r.valid, true, `${code}: pass case should be a fully valid brief; errors=${JSON.stringify(r.errors)}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
@ -1,5 +1,9 @@
|
||||||
import { test } from 'node:test';
|
import { test } from 'node:test';
|
||||||
import { strict as assert } from 'node:assert';
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { execFileSync } from 'node:child_process';
|
||||||
|
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { join } from 'node:path';
|
||||||
import { validateBriefContent } from '../../lib/validators/brief-validator.mjs';
|
import { validateBriefContent } from '../../lib/validators/brief-validator.mjs';
|
||||||
|
|
||||||
const GOOD_BRIEF = `---
|
const GOOD_BRIEF = `---
|
||||||
|
|
@ -251,6 +255,25 @@ test('validateBrief — v5.1.1: UNQUOTED brief_version 2.1 WITH phase_signals is
|
||||||
assert.ok(!r.errors.find(e => e.code === 'BRIEF_V51_MISSING_SIGNALS'));
|
assert.ok(!r.errors.find(e => e.code === 'BRIEF_V51_MISSING_SIGNALS'));
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// --- v5.9 — fable model tier (BASE_ALLOWED_MODELS widened to three values) ---
|
||||||
|
|
||||||
|
test('validateBrief — v5.9: fable phase_signals fixture accepted (no BRIEF_INVALID_MODEL)', () => {
|
||||||
|
const t = readFileSync(new URL('../fixtures/brief-effort-fable.md', import.meta.url), 'utf-8');
|
||||||
|
const r = validateBriefContent(t, { strict: true });
|
||||||
|
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||||
|
assert.ok(!r.errors.find(e => e.code === 'BRIEF_INVALID_MODEL'));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('validateBrief — v5.9: unknown model gpt5 in phase_signals rejected with BRIEF_INVALID_MODEL', () => {
|
||||||
|
const t = GOOD_BRIEF
|
||||||
|
.replace('brief_version: "2.0"', 'brief_version: "2.1"')
|
||||||
|
.replace('source: interview\n', `source: interview\n${SIGNALS_BLOCK.replace('model: opus', 'model: gpt5')}`);
|
||||||
|
const r = validateBriefContent(t, { strict: true });
|
||||||
|
assert.equal(r.valid, false);
|
||||||
|
assert.ok(r.errors.find(e => e.code === 'BRIEF_INVALID_MODEL'),
|
||||||
|
`expected BRIEF_INVALID_MODEL for gpt5, got: ${JSON.stringify(r.errors)}`);
|
||||||
|
});
|
||||||
|
|
||||||
// --- v5.5 — framing enforcement + obligatory TL;DR (gated at brief_version ≥ 2.2) ---
|
// --- v5.5 — framing enforcement + obligatory TL;DR (gated at brief_version ≥ 2.2) ---
|
||||||
// Operator decision (S6, option A1): framing + TL;DR are hard BLOCKERs for briefs
|
// Operator decision (S6, option A1): framing + TL;DR are hard BLOCKERs for briefs
|
||||||
// declaring brief_version ≥ 2.2; existing 2.0/2.1 briefs stay valid (forward-compat,
|
// declaring brief_version ≥ 2.2; existing 2.0/2.1 briefs stay valid (forward-compat,
|
||||||
|
|
@ -393,3 +416,38 @@ test('validateBrief — S18 min-version: trekreview brief is exempt (no framing
|
||||||
const r = validateBriefContent(REVIEW_AS_BRIEF, { minBriefVersion: '2.2' });
|
const r = validateBriefContent(REVIEW_AS_BRIEF, { minBriefVersion: '2.2' });
|
||||||
assert.ok(!r.warnings.find(w => w.code === 'BRIEF_VERSION_BELOW_MINIMUM'));
|
assert.ok(!r.warnings.find(w => w.code === 'BRIEF_VERSION_BELOW_MINIMUM'));
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// S56 — CLI arg-parsing regression. The no-flag invocation `brief-validator.mjs <brief.md>`
|
||||||
|
// used to bail to Usage/exit 2 because the --min-version skip index was 0 when the flag
|
||||||
|
// was absent, dropping the file positional (which sits at argv index 0).
|
||||||
|
test('CLI — no-flag invocation reaches validation, does not bail to Usage (S56 regression)', () => {
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), 'brief-cli-'));
|
||||||
|
try {
|
||||||
|
const file = join(dir, 'brief.md');
|
||||||
|
writeFileSync(file, GOOD_BRIEF);
|
||||||
|
// execFileSync throws on non-zero exit; pre-fix this bailed to Usage (exit 2).
|
||||||
|
const out = execFileSync(process.execPath, [
|
||||||
|
'lib/validators/brief-validator.mjs',
|
||||||
|
file,
|
||||||
|
], { encoding: 'utf-8' });
|
||||||
|
assert.match(out, /PASS/);
|
||||||
|
} finally {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('CLI — --min-version still locates the file positional after the value token (S56)', () => {
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), 'brief-cli-'));
|
||||||
|
try {
|
||||||
|
const file = join(dir, 'brief.md');
|
||||||
|
writeFileSync(file, GOOD_BRIEF);
|
||||||
|
const out = execFileSync(process.execPath, [
|
||||||
|
'lib/validators/brief-validator.mjs',
|
||||||
|
'--min-version', '2.0',
|
||||||
|
file,
|
||||||
|
], { encoding: 'utf-8' });
|
||||||
|
assert.match(out, /PASS/);
|
||||||
|
} finally {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
// tests/validators/profile-validator.test.mjs
|
// tests/validators/profile-validator.test.mjs
|
||||||
// SC #1, #2, #3: profile-validator validates lib/profiles/{economy,balanced,premium}.yaml
|
// SC #1, #2, #3: profile-validator validates lib/profiles/{economy,balanced,premium,fable}.yaml
|
||||||
// (innebygde profiler) plus rejects invalid models and invalid enum types.
|
// (innebygde profiler) plus rejects invalid models and invalid enum types.
|
||||||
|
|
||||||
import { test } from 'node:test';
|
import { test } from 'node:test';
|
||||||
|
|
@ -12,14 +12,15 @@ import {
|
||||||
validateProfileContent,
|
validateProfileContent,
|
||||||
PROFILE_REQUIRED_FIELDS,
|
PROFILE_REQUIRED_FIELDS,
|
||||||
PROFILE_REQUIRED_PHASES,
|
PROFILE_REQUIRED_PHASES,
|
||||||
|
BASE_ALLOWED_MODELS,
|
||||||
} from '../../lib/validators/profile-validator.mjs';
|
} from '../../lib/validators/profile-validator.mjs';
|
||||||
|
|
||||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||||
const REPO_ROOT = join(__dirname, '..', '..');
|
const REPO_ROOT = join(__dirname, '..', '..');
|
||||||
|
|
||||||
// SC #1: alle 3 innebygde profiler grønne
|
// SC #1: alle 4 innebygde profiler grønne
|
||||||
|
|
||||||
for (const profileName of ['economy', 'balanced', 'premium']) {
|
for (const profileName of ['economy', 'balanced', 'premium', 'fable']) {
|
||||||
test(`SC #1: lib/profiles/${profileName}.yaml validates clean`, () => {
|
test(`SC #1: lib/profiles/${profileName}.yaml validates clean`, () => {
|
||||||
const r = validateProfile(join(REPO_ROOT, 'lib', 'profiles', `${profileName}.yaml`));
|
const r = validateProfile(join(REPO_ROOT, 'lib', 'profiles', `${profileName}.yaml`));
|
||||||
assert.equal(r.valid, true,
|
assert.equal(r.valid, true,
|
||||||
|
|
@ -93,6 +94,77 @@ brief_reviewer_iter_cap: 1
|
||||||
`expected valid with VOYAGE_ALLOW_HAIKU=1, got: ${JSON.stringify(allowed.errors)}`);
|
`expected valid with VOYAGE_ALLOW_HAIKU=1, got: ${JSON.stringify(allowed.errors)}`);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Fable tier (v5.9): fable accepted under DEFAULT env (no opt-in flag),
|
||||||
|
// unknown models still rejected — the allowlist gate must demonstrably fire.
|
||||||
|
|
||||||
|
test('fable accepted in phase_models under default env (no env flag)', () => {
|
||||||
|
const fableProfile = `---
|
||||||
|
profile_version: "1.0"
|
||||||
|
name: fable-inline
|
||||||
|
phase_models:
|
||||||
|
- phase: brief
|
||||||
|
model: fable
|
||||||
|
- phase: research
|
||||||
|
model: fable
|
||||||
|
- phase: plan
|
||||||
|
model: fable
|
||||||
|
- phase: execute
|
||||||
|
model: fable
|
||||||
|
- phase: review
|
||||||
|
model: fable
|
||||||
|
- phase: continue
|
||||||
|
model: fable
|
||||||
|
parallel_agents_min: 6
|
||||||
|
parallel_agents_max: 8
|
||||||
|
external_research_enabled: true
|
||||||
|
brief_reviewer_iter_cap: 3
|
||||||
|
---
|
||||||
|
`;
|
||||||
|
const r = validateProfileContent(fableProfile, { env: { /* default: no flags */ } });
|
||||||
|
assert.equal(r.valid, true,
|
||||||
|
`expected fable accepted under default env, got: ${JSON.stringify(r.errors)}`);
|
||||||
|
assert.equal(r.errors.length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('unknown model gpt5 rejected with PROFILE_INVALID_MODEL under default env', () => {
|
||||||
|
const gpt5Profile = `---
|
||||||
|
profile_version: "1.0"
|
||||||
|
name: gpt5-inline
|
||||||
|
phase_models:
|
||||||
|
- phase: brief
|
||||||
|
model: gpt5
|
||||||
|
- phase: research
|
||||||
|
model: sonnet
|
||||||
|
- phase: plan
|
||||||
|
model: opus
|
||||||
|
- phase: execute
|
||||||
|
model: sonnet
|
||||||
|
- phase: review
|
||||||
|
model: opus
|
||||||
|
- phase: continue
|
||||||
|
model: sonnet
|
||||||
|
parallel_agents_min: 2
|
||||||
|
parallel_agents_max: 4
|
||||||
|
external_research_enabled: false
|
||||||
|
brief_reviewer_iter_cap: 1
|
||||||
|
---
|
||||||
|
`;
|
||||||
|
const r = validateProfileContent(gpt5Profile, { env: { /* default: no flags */ } });
|
||||||
|
assert.equal(r.valid, false);
|
||||||
|
const found = r.errors.find(e => e.code === 'PROFILE_INVALID_MODEL' && /gpt5/.test(e.message));
|
||||||
|
assert.ok(found, `expected PROFILE_INVALID_MODEL for gpt5, got: ${JSON.stringify(r.errors)}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
// BASE_ALLOWED_MODELS allowlist drift-pin (mirrors the PROFILE_REQUIRED_FIELDS pin)
|
||||||
|
|
||||||
|
test('BASE_ALLOWED_MODELS export drift-pin', () => {
|
||||||
|
assert.deepEqual(
|
||||||
|
[...BASE_ALLOWED_MODELS],
|
||||||
|
['sonnet', 'opus', 'fable'],
|
||||||
|
'BASE_ALLOWED_MODELS contract drift — pin contract',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
// Required fields presence
|
// Required fields presence
|
||||||
|
|
||||||
test('PROFILE_MISSING_FIELD when name absent', () => {
|
test('PROFILE_MISSING_FIELD when name absent', () => {
|
||||||
|
|
@ -148,3 +220,57 @@ test('PROFILE_REQUIRED_PHASES export drift-pin', () => {
|
||||||
'PROFILE_REQUIRED_PHASES contract drift — pin contract',
|
'PROFILE_REQUIRED_PHASES contract drift — pin contract',
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// S34 (V30) — economy carries an explicit `experimental` marker.
|
||||||
|
// The cross-tier Jaccard floor (0.55) rests on parked-synthetic fixtures;
|
||||||
|
// empirical Step-17 calibration is deferred to v4.2 ($60-120 LLM budget,
|
||||||
|
// unauthorized). Until a real calibration lands, `economy` is self-declared
|
||||||
|
// experimental in the profile DATA — not only in prose — so the status
|
||||||
|
// survives a docs rewrite. See tests/synthetic/profile-jaccard-calibration.md.
|
||||||
|
|
||||||
|
test('S34: economy.yaml declares experimental: true (uncalibrated Jaccard floor)', () => {
|
||||||
|
const r = validateProfile(join(REPO_ROOT, 'lib', 'profiles', 'economy.yaml'));
|
||||||
|
assert.equal(r.valid, true, `economy.yaml should validate: ${JSON.stringify(r.errors)}`);
|
||||||
|
assert.equal(r.parsed.frontmatter.experimental, true,
|
||||||
|
'economy.yaml must carry experimental: true until the Jaccard floor is empirically calibrated (S34/V30)');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('S34: premium + balanced are NOT flagged experimental', () => {
|
||||||
|
for (const name of ['premium', 'balanced']) {
|
||||||
|
const r = validateProfile(join(REPO_ROOT, 'lib', 'profiles', `${name}.yaml`));
|
||||||
|
assert.equal(r.valid, true, `${name}.yaml should validate: ${JSON.stringify(r.errors)}`);
|
||||||
|
assert.notEqual(r.parsed.frontmatter.experimental, true,
|
||||||
|
`${name}.yaml must not be flagged experimental`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('S34: validator type-checks experimental as boolean (PROFILE_INVALID_ENUM)', () => {
|
||||||
|
const bad = `---
|
||||||
|
profile_version: "1.0"
|
||||||
|
name: bad-experimental
|
||||||
|
phase_models:
|
||||||
|
- phase: brief
|
||||||
|
model: sonnet
|
||||||
|
- phase: research
|
||||||
|
model: sonnet
|
||||||
|
- phase: plan
|
||||||
|
model: sonnet
|
||||||
|
- phase: execute
|
||||||
|
model: sonnet
|
||||||
|
- phase: review
|
||||||
|
model: sonnet
|
||||||
|
- phase: continue
|
||||||
|
model: sonnet
|
||||||
|
parallel_agents_min: 2
|
||||||
|
parallel_agents_max: 3
|
||||||
|
external_research_enabled: false
|
||||||
|
brief_reviewer_iter_cap: 1
|
||||||
|
experimental: "yes"
|
||||||
|
---
|
||||||
|
`;
|
||||||
|
const r = validateProfileContent(bad);
|
||||||
|
assert.equal(r.valid, false, 'experimental: "yes" (string) must be rejected');
|
||||||
|
const found = r.errors.find(e => e.code === 'PROFILE_INVALID_ENUM' && /experimental/.test(e.message));
|
||||||
|
assert.ok(found, `expected PROFILE_INVALID_ENUM for experimental, got: ${JSON.stringify(r.errors)}`);
|
||||||
|
assert.match(found.message, /boolean/);
|
||||||
|
});
|
||||||
|
|
|
||||||
|
|
@ -65,6 +65,36 @@ test('validateProgress — step count mismatch is warning', () => {
|
||||||
assert.ok(r.warnings.find(w => w.code === 'PROGRESS_STEP_COUNT_MISMATCH'));
|
assert.ok(r.warnings.find(w => w.code === 'PROGRESS_STEP_COUNT_MISMATCH'));
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('validateProgress — iterations_remaining non-negative integer is valid', () => {
|
||||||
|
const p = goodProgress();
|
||||||
|
p.iterations_remaining = 7;
|
||||||
|
const r = validateProgressObject(p);
|
||||||
|
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('validateProgress — iterations_remaining absent is valid (backward-compat)', () => {
|
||||||
|
const p = goodProgress();
|
||||||
|
delete p.iterations_remaining;
|
||||||
|
const r = validateProgressObject(p);
|
||||||
|
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('validateProgress — iterations_remaining negative is invalid', () => {
|
||||||
|
const p = goodProgress();
|
||||||
|
p.iterations_remaining = -1;
|
||||||
|
const r = validateProgressObject(p);
|
||||||
|
assert.equal(r.valid, false);
|
||||||
|
assert.ok(r.errors.find(e => e.code === 'PROGRESS_ITERATIONS_REMAINING_INVALID'));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('validateProgress — iterations_remaining non-integer is invalid', () => {
|
||||||
|
const p = goodProgress();
|
||||||
|
p.iterations_remaining = 3.5;
|
||||||
|
const r = validateProgressObject(p);
|
||||||
|
assert.equal(r.valid, false);
|
||||||
|
assert.ok(r.errors.find(e => e.code === 'PROGRESS_ITERATIONS_REMAINING_INVALID'));
|
||||||
|
});
|
||||||
|
|
||||||
test('checkResumeReadiness — completed run cannot resume', () => {
|
test('checkResumeReadiness — completed run cannot resume', () => {
|
||||||
const p = goodProgress();
|
const p = goodProgress();
|
||||||
p.status = 'completed';
|
p.status = 'completed';
|
||||||
|
|
|
||||||
|
|
@ -26,6 +26,9 @@ fail() { echo "[FAIL] SC$1 — $2"; FAIL=$((FAIL + 1)); exit 1; }
|
||||||
|
|
||||||
# Tracked-file exclusions (paths preserved verbatim from old name).
|
# Tracked-file exclusions (paths preserved verbatim from old name).
|
||||||
# - CHANGELOG/TRADEMARKS/MIGRATION legitimately reference the old name.
|
# - CHANGELOG/TRADEMARKS/MIGRATION legitimately reference the old name.
|
||||||
|
# - cc-upgrade-2.1.181-decision-matrix.md cites CC features: `ultracode`
|
||||||
|
# (a real CC keyword, 2.1.160) and the `ultra-cc-architect` plugin name —
|
||||||
|
# both legitimate, not rebrand leftovers.
|
||||||
# - architecture-discovery.mjs + project-discovery.mjs are Q8 exceptions
|
# - architecture-discovery.mjs + project-discovery.mjs are Q8 exceptions
|
||||||
# pointing at the upstream architect producer slot.
|
# pointing at the upstream architect producer slot.
|
||||||
# - verify.sh self-references the forbidden patterns to detect them.
|
# - verify.sh self-references the forbidden patterns to detect them.
|
||||||
|
|
@ -33,6 +36,7 @@ fail() { echo "[FAIL] SC$1 — $2"; FAIL=$((FAIL + 1)); exit 1; }
|
||||||
exclude_path() {
|
exclude_path() {
|
||||||
case "$1" in
|
case "$1" in
|
||||||
*CHANGELOG.md|*TRADEMARKS.md|*MIGRATION.md) return 0 ;;
|
*CHANGELOG.md|*TRADEMARKS.md|*MIGRATION.md) return 0 ;;
|
||||||
|
*cc-upgrade-2.1.181-decision-matrix.md) return 0 ;;
|
||||||
*lib/validators/architecture-discovery.mjs) return 0 ;;
|
*lib/validators/architecture-discovery.mjs) return 0 ;;
|
||||||
*lib/parsers/project-discovery.mjs) return 0 ;;
|
*lib/parsers/project-discovery.mjs) return 0 ;;
|
||||||
*verify.sh) return 0 ;;
|
*verify.sh) return 0 ;;
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue