voyage/docs/profiles.md
Kjell Tore Guttormsen fdd3ad80d7 feat(voyage): W2 impl (S4) — gate resolver model, adopt native effort:, doc-truth
S4 of the 2.1.181 upgrade — implementation, not a gate. TDD: failing test
written first for the resolver gate, then the fix; suite green throughout.

- Resolver MAJOR (FIX): lib/profiles/phase-signal-resolver.mjs now imports
  BASE_ALLOWED_MODELS from profile-validator and gates `model`
  (if 'model' in entry && BASE_ALLOWED_MODELS.includes(entry.model)),
  mirroring the EFFORT_LEVELS gate one line up. Out-of-allowlist models
  (gpt-4, haiku) are dropped instead of handed to an agent spawn —
  defense-in-depth behind brief-validator's validation-time check. No
  circular import (brief-validator already imports the same symbol).
  +2 tests (drops-invalid / keeps-valid).
- Native effort: (SHIP, static additive): effort: frontmatter on 8 agents —
  retrieval (task-finder, git-historian, dependency-tracer,
  architecture-mapper) = medium; adversarial-reasoning (plan-critic,
  risk-assessor, contrarian-researcher, review-coordinator) = high. The
  other 15 stay unset -> inherit Opus-4.8 default (high). This per-spawn
  REASONING effort is a different axis from brief phase_signals.effort
  (ORCHESTRATION shape) per the S3 decision.
- Doc-truth + axis distinction: new canonical docs/profiles.md
  §Model & effort axes (opus->Opus 4.8 default-high; orchestration vs
  reasoning effort table; native-effort precedence; per-agent levels).
  Short notes in CLAUDE.md (after Agents table) and README.md (Cost
  profile), both pointing to profiles.md.
- Open (non-blocking, unchanged): only STATIC effort shipped — the
  verified-safe minimum. Profile-driven DYNAMIC effort still needs
  verification of the per-spawn effort param or env-var injection.

Matrix: new "S4 resolutions" section. Tests 582 total / 580 pass / 0 fail /
2 skip (was 578 pass; +2). claude plugin validate passes (only pre-existing
root-CLAUDE.md warning).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
2026-06-18 12:27:07 +02:00

9.2 KiB
Raw Blame History

Profile system — voyage v4.1

This document describes the model profile system: built-in tiers, lookup precedence, custom-profile authoring, drift detection, and cost estimation (with disclaimer).

Built-in profiles

Three pre-defined tiers ship with v4.1, located at lib/profiles/{economy,balanced,premium}.yaml.

Profile Brief Research Plan Execute Review Continue Use case
economy sonnet sonnet sonnet sonnet sonnet sonnet Lowest cost; small-scope tasks where you have high confidence the brief is right
balanced (default) sonnet sonnet opus sonnet opus sonnet Default — opus where reasoning depth pays off (plan synthesis + adversarial review)
premium opus sonnet opus sonnet opus sonnet Critical-path planning + review when budget allows

balanced is the v4.1 default. It puts opus on the two phases where quality matters most (Plan synthesis + Review) and sonnet everywhere else. This lands the cost/quality trade-off that solo-developers and small teams actually want.

economy is strictly experimental in v4.1. The cross-tier Jaccard floor (0.55) is grounded in parked-synthetic fixtures, not empirical runs (Step 17 calibration was deferred — see tests/synthetic/profile-jaccard-calibration.md). If you observe economy-plan quality regressions, fall back to balanced.

Model & effort axes

opus and sonnet are model aliases, not pinned ids. As of Claude Code 2.1.154 the opus alias resolves to Opus 4.8, whose default reasoning effort is high; sonnet resolves to Sonnet 4.6. The profile table above selects which alias runs each phase — it does not touch reasoning effort.

Two different things share the word "effort" in Voyage. They are orthogonal axes — same name, different mechanism:

Orchestration effort Reasoning effort (native)
Where brief phase_signals.effort, consumed by command prose native Claude Code effort: in agent frontmatter
Values low / standard / high low / medium / high / xhigh / max
Controls which agents/passes/gates run (swarm cardinality, gate strictness, sequential-vs-parallel) per-spawn reasoning-token budget the harness gives one agent
low means "run fewer agents" (the --quick-equivalent code-path) "think with a smaller budget"
Applied by Voyage command logic the harness, at spawn time

The phase-signal-resolver.mjs helper only reads the orchestration axis (phase_signals.effort, gated against low/standard/high) plus the optional per-phase model (gated against ['sonnet','opus']). It never emits native effort:.

Native effort: on agents. Voyage sets the reasoning axis statically on selected agents, additively over the Opus-4.8 default:

  • Retrieval agents → medium: task-finder, git-historian, dependency-tracer, architecture-mapper (structured discovery, not deep multi-step reasoning).
  • Adversarial-reasoning agents → high: plan-critic, risk-assessor, contrarian-researcher, review-coordinator (synthesis and stress-testing where reasoning depth pays off).
  • All other agents: unset → inherit the model default (Opus 4.8 = high).

Native-effort precedence (harness): env CLAUDE_CODE_EFFORT_LEVEL > frontmatter effort: > session setting > model default. MAX_THINKING_TOKENS=0 (or --thinking disabled) overrides effort entirely. Enterprise availableModels constrains the model alias only — it does not bound effort.

Decision tree

Are you uncertain whether the brief is correctly framed?
  └── Yes → premium (opus on brief + plan + review)
  └── No  → continue
            ↓
Is the change small (≤ 5 steps in the plan)?
  └── Yes → economy (sonnet everywhere)
  └── No  → balanced (opus on plan + review)

Special cases:
  - Critical-infrastructure plan          → premium
  - Migration with rollback risk          → premium
  - Research-heavy task (≥ 4 dimensions)  → balanced (research-stage benefits)
  - Bug fix with clear reproducer         → economy
  - Documentation-only PR                 → economy

Lookup order

Voyage resolves the profile in this priority order:

  1. Explicit --profile <name> flag — passed to the command
  2. Plan-file frontmatter profile: — when resuming via /trekexecute --resume or /trekcontinue
  3. VOYAGE_PROFILE environment variable — useful for headless CI
  4. Default balanced — final fallback

The resolved value is recorded in two places:

  • Plan-file frontmatter profile: <name> and phase_models: [...]
  • Stats stream ${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonlprofile, profile_source, phase_models, model_used, phase_models_resolved fields

profile_source distinguishes how the profile was resolved (flag / plan_frontmatter / env / default), so dashboards can surface unexpected env-var inheritance in CI.

Custom profiles

Drop a YAML file at lib/profiles/<name>.yaml to define a new tier. The validator (lib/validators/profile-validator.mjs) enforces:

  • Every phase_models[].phase must be a known phase enum: brief / research / plan / execute / review / continue
  • Every phase_models[].model must match ^(opus|sonnet)(\b|-).* or one of the canonical short names
  • All six phases must be present (no partial profiles)

Custom profiles override built-ins of the same name (lookup is alphabetical with <custom> taking precedence). You may NOT redefine balanced (the default tier is locked to prevent accidental override of headless CI behaviour); use a different name and reference it via --profile <new-name> or VOYAGE_PROFILE=<new-name>.

Example custom profile

# lib/profiles/critical.yaml — opus everywhere except continue
phase_models:
  - phase: brief
    model: opus
  - phase: research
    model: opus
  - phase: plan
    model: opus
  - phase: execute
    model: opus
  - phase: review
    model: opus
  - phase: continue
    model: sonnet

Validate with: node lib/validators/profile-validator.mjs --json lib/profiles/critical.yaml

Drift detection

In --strict mode, plan-validator.mjs emits a MANIFEST_PROFILE_DRIFT warning when the plan-level profile: differs from any step manifest's profile_used. The warning is a signal, not a failure — the plan remains valid: true. This catches:

  • Manual edits where an operator changed a single step's profile
  • Resume from a partial run where the previous session used a different tier
  • Copy-paste errors when stitching plan fragments

To suppress the warning intentionally (e.g. when a critical step genuinely needs a higher tier), document the override in the step's prose and re-run with --soft to validate without strict-mode warnings.

Cost estimation

Disclaimer: the table below is an anslag, not a contractual SLA. Real cost depends on context size, agent-swarm cardinality, tool-use density, and Claude Code billing schedule. Treat these as rough order-of-magnitude.

Profile Brief Research Plan Execute Review Total
economy $0.100.50 $0.502.00 $0.502.00 $1.005.00 $0.201.00 $210
balanced $0.100.50 $0.502.00 $1.004.00 $1.005.00 $0.502.00 $314
premium $0.502.00 $0.502.00 $1.004.00 $1.005.00 $0.502.00 $415

Numbers are per full pipeline run (brief + research + plan + execute + review) on a moderate-complexity task. Numbers scale roughly linearly with the size of the resulting plan (10 steps ≈ baseline; 30 steps ≈ 3× the execute column).

Per-profile actuals are emitted to JSONL stats — pipe them through the OTel export (docs/observability.md) to get real cost-attribution graphs in Grafana. Replace the table above with your own measured numbers after ≥ 3 runs of each profile.

Deferred to v4.2

  • balanced.external_research_enabled operator-override — v4.1 omits this per scope-guardian SG2. v4.2 may add an opt-in flag to enable external research agents in the balanced tier without forcing premium.
  • Empirical Jaccard re-calibration — parked-synthetic fixtures in v4.1 use a 0.55 conservative starting threshold. v4.2 plans an empirical re-run with $60-120 LLM budget to derive a calibrated threshold from real economy-vs-premium plan pairs.
  • ROUGE-L + char-4gram MinHash as primary/secondary cross-tier gates per research/02 Recommendation #7. Jaccard remains the gate in v4.1; v4.2 may layer ROUGE-L on top.

See also