voyage/docs/profiles.md
Kjell Tore Guttormsen abc3083c1e
feat(execute): one fresh helper per step, a controller on the diff, proof by command
/trekexecute no longer builds steps itself. Each plan step is built by one
fresh helper (new agent step-implementer: opus, effort high, no Agent tool)
and judged by one controller (a general helper on opus) that reads the
diff from a recorded start_sha and gives two verdicts, spec and quality.
Fix rounds go up, maximum 5: three fixes by a fresh helper, then back to
the plan, then to the operator. No cheaper-model ladder.

Proof is a command, never the model's word:
- lib/verification/red-proof.mjs: the step's test must fail at the start
  point and pass at the result (exit 0/1/2; 2 is never green). Reproduces
  the 02243c6 red-first measurement (6 of 55) through the shim fixture.
- lib/verification/test-integrity.mjs: the floor guard on the diff
  (removed asserts, skipped/deleted tests, muted checks, stubs, loosened
  thresholds). Over the last 150 commits: 149 clean, 1 real finding.
- lib/verification/done-line.mjs: DONE step N only when the test command
  exits 0.

Test first is required from plan_version 1.8. The test lock (K8) is used
through its CLI and reported as not-locked while it is missing. A run
stops only on a closed list of four classes; every other call goes into a
decision log with its cost if wrong. Without a passing run-the-thing
criterion the result is not verified end to end, never green.

Handover 5 gains optional per-step start_sha, report_path, red_proof,
test_lock, fix_rounds (0-5) and run-level decisions and end_to_end; old
progress files still validate and resume.

No isolation on the helper: measured, a worktree-isolated subagent
branches from the default branch (02f320f), not the executor's HEAD
(2c89a6b), so it would miss the earlier steps.

Borrowings credited in commands/trekexecute.md ## Credits (Superpowers,
ai-native-sdlc, Kiro, GitHub Spec Kit, agent-skills, Cherny).

Tests: 1411 -> 1457 (1455 pass, 0 fail, 2 skipped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 19:58:18 +02:00

11 KiB
Raw Blame History

Profile system — voyage v4.1

This document describes the model profile system: built-in tiers, lookup precedence, custom-profile authoring, drift detection, and cost estimation (with disclaimer).

Built-in profiles

Four pre-defined tiers ship with the plugin (fable added in v5.9), located at lib/profiles/{economy,balanced,premium,fable}.yaml.

Profile Brief Research Plan Execute Review Continue Use case
economy sonnet sonnet sonnet sonnet sonnet sonnet ⚠ Experimental (uncalibrated Jaccard floor) — lowest cost; small-scope tasks where you have high confidence the brief is right
balanced sonnet sonnet opus sonnet opus sonnet Mixed — opus where reasoning depth pays off (plan synthesis + adversarial review); opt-in via --profile balanced
premium (default) opus opus opus opus opus opus Maximum quality — Opus on every phase + external research on (default since the 2026-05-13 operator decision)
fable fable fable fable fable fable fable Max quality — Fable 5.1 (the most capable model in Claude Code) on every phase; opt-in via --profile fable; reasoning effort inherits from the session (see Model & effort axes)

premium is the default tier — set by the 2026-05-13 operator decision and matched by the hardcoded resolver default in lib/profiles/resolver.mjs. It runs opus on every phase and turns external research on: maximum quality, at roughly 5× the sub-agent cost of an all-sonnet run, accepted as a deliberate trade-off. Drop to --profile balanced (opus only on the two phases where quality matters most — Plan synthesis + Review — and sonnet everywhere else) or --profile economy (sonnet everywhere) when cost or latency matters more than depth. Step up to --profile fable (Fable 5.1 on every phase) when maximum quality is wanted end-to-end and cost is not a constraint.

economy is strictly experimental in v4.1, and says so in the profile data itself: lib/profiles/economy.yaml carries experimental: true. The cross-tier Jaccard floor (0.55) is grounded in parked-synthetic fixtures, not empirical runs (Step 17 calibration was deferred — see tests/synthetic/profile-jaccard-calibration.md). The flag is pinned: it must stay true while the calibration status is parked-synthetic, and must be dropped in the same change that lands empirical calibration (status: empirical). If you observe economy-plan quality regressions, fall back to balanced.

Model & effort axes

opus, sonnet, and fable are model aliases, not pinned ids. On the Anthropic API (verified 2026-10-02 against the Claude Code model-config docs) the opus alias resolves to Opus 5.5 and sonnet to Sonnet 5.5, both with default reasoning effort medium; fable resolves to Fable 5.1 (the most capable model in Claude Code), whose default reasoning effort is high. Other providers resolve opus and sonnet to other versions. The profile table above selects which alias runs each phase — it does not touch reasoning effort.

Reasoning effort inherits from the session. Voyage effort (orchestration shape — which agents/passes run) and model reasoning effort are different axes. Fable 5.1's default reasoning effort is high, NOT xhigh, and Claude Code saves effort per model, so a level saved for one model does not carry to another. To run the fable tier at xhigh, choose it explicitly: /effort xhigh, --effort xhigh, or CLAUDE_CODE_EFFORT_LEVEL.

Two different things share the word "effort" in Voyage. They are orthogonal axes — same name, different mechanism:

Orchestration effort Reasoning effort (native)
Where brief phase_signals.effort, consumed by command prose native Claude Code effort: in agent frontmatter
Values low / standard / high low / medium / high / xhigh / max
Controls which agents/passes/gates run (swarm cardinality, gate strictness, sequential-vs-parallel) per-spawn reasoning-token budget the harness gives one agent
low means "run fewer agents" (the --quick-equivalent code-path) "think with a smaller budget"
Applied by Voyage command logic the harness, at spawn time

The phase-signal-resolver.mjs helper only reads the orchestration axis (phase_signals.effort, gated against low/standard/high) plus the optional per-phase model (gated against ['sonnet','opus','fable']). It never emits native effort:.

Native effort: on agents. Voyage sets the reasoning axis statically on selected agents, overriding the session level:

  • Nine agents pin high: the retrieval agents task-finder, git-historian, dependency-tracer, architecture-mapper, the adversarial-reasoning agents plan-critic, risk-assessor, contrarian-researcher, review-coordinator, and /trekexecute's step helper step-implementer. No agent pins below high (operator rule, 2026-10-05: standard is Opus 5.5 at effort High).
  • All other agents: unset → inherits the session's effort level (a session that nothing sets runs at its model's default — medium on Opus 5.5).

Native-effort precedence (harness): env CLAUDE_CODE_EFFORT_LEVEL > frontmatter effort: > session setting > model default, and a maxEffortLevel setting or an organization effort cap still limits the result. MAX_THINKING_TOKENS=0 has no effect on Opus 5.5, Sonnet 5.5 or the Fable models: thinking cannot be turned off there. availableModels restricts which models can be selected — it does not bound effort.

Decision tree

Are you uncertain whether the brief is correctly framed?
  └── Yes → premium (opus on brief + plan + review)
  └── No  → continue
            ↓
Is the change small (≤ 5 steps in the plan)?
  └── Yes → economy (sonnet everywhere)
  └── No  → balanced (opus on plan + review)

Special cases:
  - Critical-infrastructure plan          → premium
  - Migration with rollback risk          → premium
  - Research-heavy task (≥ 4 dimensions)  → balanced (research-stage benefits)
  - Bug fix with clear reproducer         → economy
  - Documentation-only PR                 → economy

Lookup order

Voyage resolves the profile in this priority order:

  1. Explicit --profile <name> flag — passed to the command
  2. Plan-file frontmatter profile: — when resuming via /trekexecute --resume or /trekcontinue
  3. VOYAGE_PROFILE environment variable — useful for headless CI
  4. Default premium — final fallback

The resolved value is recorded in two places:

  • Plan-file frontmatter profile: <name> and phase_models: [...]
  • Stats stream ${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl — profile, profile_source, phase_models, model_used, phase_models_resolved fields

profile_source distinguishes how the profile was resolved (flag / plan_frontmatter / env / default), so dashboards can surface unexpected env-var inheritance in CI.

Custom profiles

Drop a YAML file at lib/profiles/<name>.yaml to define a new tier. The validator (lib/validators/profile-validator.mjs) enforces:

  • Every phase_models[].phase must be a known phase enum: brief / research / plan / execute / review / continue
  • Every phase_models[].model must exactly match an entry in BASE_ALLOWED_MODELS (['sonnet', 'opus', 'fable'] in lib/validators/profile-validator.mjs; haiku only with VOYAGE_ALLOW_HAIKU=1)
  • All six phases must be present (no partial profiles)

The four built-in names (economy, balanced, premium, fable) resolve to their bundled yaml first — findProfilePath() returns the built-in before consulting voyage-profiles/, so a same-named custom file is ignored and cannot shadow a built-in. To customize, give your profile a new name and reference it via --profile <new-name> or VOYAGE_PROFILE=<new-name>.

Example custom profile

# lib/profiles/critical.yaml — opus everywhere except continue
phase_models:
  - phase: brief
    model: opus
  - phase: research
    model: opus
  - phase: plan
    model: opus
  - phase: execute
    model: opus
  - phase: review
    model: opus
  - phase: continue
    model: sonnet

Validate with: node lib/validators/profile-validator.mjs --json lib/profiles/critical.yaml

Drift detection

In --strict mode, plan-validator.mjs emits a MANIFEST_PROFILE_DRIFT warning when the plan-level profile: differs from any step manifest's profile_used. The warning is a signal, not a failure — the plan remains valid: true. This catches:

  • Manual edits where an operator changed a single step's profile
  • Resume from a partial run where the previous session used a different tier
  • Copy-paste errors when stitching plan fragments

To suppress the warning intentionally (e.g. when a critical step genuinely needs a higher tier), document the override in the step's prose and re-run with --soft to validate without strict-mode warnings.

Cost estimation

Disclaimer: the table below is an anslag, not a contractual SLA. Real cost depends on context size, agent-swarm cardinality, tool-use density, and Claude Code billing schedule. Treat these as rough order-of-magnitude.

Profile Brief Research Plan Execute Review Total
economy $0.10–0.50 $0.50–2.00 $0.50–2.00 $1.00–5.00 $0.20–1.00 $2–10
balanced $0.10–0.50 $0.50–2.00 $1.00–4.00 $1.00–5.00 $0.50–2.00 $3–14
premium $0.50–2.00 $0.50–2.00 $1.00–4.00 $1.00–5.00 $0.50–2.00 $4–15

Numbers are per full pipeline run (brief + research + plan + execute + review) on a moderate-complexity task. Numbers scale roughly linearly with the size of the resulting plan (10 steps ≈ baseline; 30 steps ≈ 3× the execute column).

Per-profile actuals are emitted to JSONL stats — pipe them through the OTel export (docs/observability.md) to get real cost-attribution graphs in Grafana. Replace the table above with your own measured numbers after ≥ 3 runs of each profile.

Deferred to v4.2

  • balanced.external_research_enabled operator-override — v4.1 omits this per scope-guardian SG2. v4.2 may add an opt-in flag to enable external research agents in the balanced tier without forcing premium.
  • Empirical Jaccard re-calibration — parked-synthetic fixtures in v4.1 use a 0.55 conservative starting threshold. v4.2 plans an empirical re-run with $60-120 LLM budget to derive a calibrated threshold from real economy-vs-premium plan pairs.
  • ROUGE-L + char-4gram MinHash as primary/secondary cross-tier gates per research/02 Recommendation #7. Jaccard remains the gate in v4.1; v4.2 may layer ROUGE-L on top.

See also