Kjell Tore Guttormsen
abc3083c1e
/trekexecute no longer builds steps itself. Each plan step is built by one fresh helper (new agent step-implementer: opus, effort high, no Agent tool) and judged by one controller (a general helper on opus) that reads the diff from a recorded start_sha and gives two verdicts, spec and quality. Fix rounds go up, maximum 5: three fixes by a fresh helper, then back to the plan, then to the operator. No cheaper-model ladder. Proof is a command, never the model's word: - lib/verification/red-proof.mjs: the step's test must fail at the start point and pass at the result (exit 0/1/2; 2 is never green). Reproduces the02243c6red-first measurement (6 of 55) through the shim fixture. - lib/verification/test-integrity.mjs: the floor guard on the diff (removed asserts, skipped/deleted tests, muted checks, stubs, loosened thresholds). Over the last 150 commits: 149 clean, 1 real finding. - lib/verification/done-line.mjs: DONE step N only when the test command exits 0. Test first is required from plan_version 1.8. The test lock (K8) is used through its CLI and reported as not-locked while it is missing. A run stops only on a closed list of four classes; every other call goes into a decision log with its cost if wrong. Without a passing run-the-thing criterion the result is not verified end to end, never green. Handover 5 gains optional per-step start_sha, report_path, red_proof, test_lock, fix_rounds (0-5) and run-level decisions and end_to_end; old progress files still validate and resume. No isolation on the helper: measured, a worktree-isolated subagent branches from the default branch (02f320f), not the executor's HEAD (2c89a6b), so it would miss the earlier steps. Borrowings credited in commands/trekexecute.md ## Credits (Superpowers, ai-native-sdlc, Kiro, GitHub Spec Kit, agent-skills, Cherny). Tests: 1411 -> 1457 (1455 pass, 0 fail, 2 skipped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
234 lines
11 KiB
Markdown
234 lines
11 KiB
Markdown
# Profile system — voyage v4.1
|
||
|
||
This document describes the model profile system: built-in tiers,
|
||
lookup precedence, custom-profile authoring, drift detection, and
|
||
cost estimation (with disclaimer).
|
||
|
||
## Built-in profiles
|
||
|
||
Four pre-defined tiers ship with the plugin (fable added in v5.9), located at
|
||
`lib/profiles/{economy,balanced,premium,fable}.yaml`.
|
||
|
||
| Profile | Brief | Research | Plan | Execute | Review | Continue | Use case |
|
||
|---------|-------|----------|------|---------|--------|----------|----------|
|
||
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | ⚠ **Experimental** (uncalibrated Jaccard floor) — lowest cost; small-scope tasks where you have high confidence the brief is right |
|
||
| `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (plan synthesis + adversarial review); opt-in via `--profile balanced` |
|
||
| `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase + external research on (default since the 2026-05-13 operator decision) |
|
||
| `fable` | fable | fable | fable | fable | fable | fable | Max quality — Fable 5.1 (the most capable model in Claude Code) on every phase; opt-in via `--profile fable`; reasoning effort inherits from the session (see Model & effort axes) |
|
||
|
||
`premium` is the default tier — set by the 2026-05-13 operator decision and
|
||
matched by the hardcoded resolver default in `lib/profiles/resolver.mjs`. It
|
||
runs opus on every phase and turns external research on: maximum quality, at
|
||
roughly 5× the sub-agent cost of an all-sonnet run, accepted as a deliberate
|
||
trade-off. Drop to `--profile balanced` (opus only on the two phases where
|
||
quality matters most — Plan synthesis + Review — and sonnet everywhere else)
|
||
or `--profile economy` (sonnet everywhere) when cost or latency matters more
|
||
than depth. Step up to `--profile fable` (Fable 5.1 on every phase) when
|
||
maximum quality is wanted end-to-end and cost is not a constraint.
|
||
|
||
`economy` is *strictly experimental* in v4.1, and says so in the profile
|
||
data itself: `lib/profiles/economy.yaml` carries `experimental: true`. The
|
||
cross-tier Jaccard floor (0.55) is grounded in parked-synthetic fixtures, not
|
||
empirical runs (Step 17 calibration was deferred — see
|
||
`tests/synthetic/profile-jaccard-calibration.md`). The flag is pinned: it must
|
||
stay `true` while the calibration status is `parked-synthetic`, and must be
|
||
dropped in the same change that lands empirical calibration
|
||
(`status: empirical`). If you observe economy-plan quality regressions, fall
|
||
back to `balanced`.
|
||
|
||
## Model & effort axes
|
||
|
||
`opus`, `sonnet`, and `fable` are model **aliases**, not pinned ids. On the
|
||
Anthropic API (verified 2026-10-02 against the Claude Code model-config docs)
|
||
the `opus` alias resolves to **Opus 5.5** and `sonnet` to **Sonnet 5.5**, both
|
||
with default reasoning effort **`medium`**; `fable` resolves to **Fable 5.1**
|
||
(the most capable model in Claude Code), whose default reasoning effort is
|
||
**`high`**. Other providers resolve `opus` and `sonnet` to other versions. The
|
||
profile table above selects *which alias* runs each phase — it does not touch
|
||
reasoning effort.
|
||
|
||
**Reasoning effort inherits from the session.** Voyage effort (orchestration
|
||
shape — which agents/passes run) and model reasoning effort are different
|
||
axes. Fable 5.1's default reasoning effort is `high`, NOT xhigh, and Claude
|
||
Code saves effort per model, so a level saved for one model does not carry to
|
||
another. To run the fable tier at xhigh, choose it explicitly: `/effort xhigh`,
|
||
`--effort xhigh`, or `CLAUDE_CODE_EFFORT_LEVEL`.
|
||
|
||
Two different things share the word "effort" in Voyage. They are **orthogonal
|
||
axes** — same name, different mechanism:
|
||
|
||
| | **Orchestration effort** | **Reasoning effort (native)** |
|
||
|---|---|---|
|
||
| Where | brief `phase_signals.effort`, consumed by command prose | native Claude Code `effort:` in agent frontmatter |
|
||
| Values | `low` / `standard` / `high` | `low` / `medium` / `high` / `xhigh` / `max` |
|
||
| Controls | *which agents/passes/gates run* (swarm cardinality, gate strictness, sequential-vs-parallel) | *per-spawn reasoning-token budget* the harness gives one agent |
|
||
| `low` means | "run fewer agents" (the `--quick`-equivalent code-path) | "think with a smaller budget" |
|
||
| Applied by | Voyage command logic | the harness, at spawn time |
|
||
|
||
The `phase-signal-resolver.mjs` helper only reads the **orchestration** axis
|
||
(`phase_signals.effort`, gated against `low/standard/high`) plus the optional
|
||
per-phase `model` (gated against `['sonnet','opus','fable']`). It never emits
|
||
native `effort:`.
|
||
|
||
**Native `effort:` on agents.** Voyage sets the reasoning axis statically on
|
||
selected agents, overriding the session level:
|
||
|
||
- **Nine agents pin `high`:** the retrieval agents `task-finder`,
|
||
`git-historian`, `dependency-tracer`, `architecture-mapper`, the
|
||
adversarial-reasoning agents `plan-critic`, `risk-assessor`,
|
||
`contrarian-researcher`, `review-coordinator`, and `/trekexecute`'s step
|
||
helper `step-implementer`. No agent pins below `high`
|
||
(operator rule, 2026-10-05: standard is Opus 5.5 at effort High).
|
||
- **All other agents:** unset → inherits the session's effort level (a session
|
||
that nothing sets runs at its model's default — `medium` on Opus 5.5).
|
||
|
||
Native-effort precedence (harness): env `CLAUDE_CODE_EFFORT_LEVEL` > frontmatter
|
||
`effort:` > session setting > model default, and a `maxEffortLevel` setting or
|
||
an organization effort cap still limits the result. `MAX_THINKING_TOKENS=0` has
|
||
no effect on Opus 5.5, Sonnet 5.5 or the Fable models: thinking cannot be turned
|
||
off there. `availableModels` restricts which models can be selected — it does
|
||
**not** bound effort.
|
||
|
||
## Decision tree
|
||
|
||
```
|
||
Are you uncertain whether the brief is correctly framed?
|
||
└── Yes → premium (opus on brief + plan + review)
|
||
└── No → continue
|
||
↓
|
||
Is the change small (≤ 5 steps in the plan)?
|
||
└── Yes → economy (sonnet everywhere)
|
||
└── No → balanced (opus on plan + review)
|
||
|
||
Special cases:
|
||
- Critical-infrastructure plan → premium
|
||
- Migration with rollback risk → premium
|
||
- Research-heavy task (≥ 4 dimensions) → balanced (research-stage benefits)
|
||
- Bug fix with clear reproducer → economy
|
||
- Documentation-only PR → economy
|
||
```
|
||
|
||
## Lookup order
|
||
|
||
Voyage resolves the profile in this priority order:
|
||
|
||
1. **Explicit `--profile <name>` flag** — passed to the command
|
||
2. **Plan-file frontmatter `profile:`** — when resuming via
|
||
`/trekexecute --resume` or `/trekcontinue`
|
||
3. **`VOYAGE_PROFILE` environment variable** — useful for headless CI
|
||
4. **Default `premium`** — final fallback
|
||
|
||
The resolved value is recorded in two places:
|
||
|
||
- Plan-file frontmatter `profile: <name>` and `phase_models: [...]`
|
||
- Stats stream `${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl` —
|
||
`profile`, `profile_source`, `phase_models`, `model_used`,
|
||
`phase_models_resolved` fields
|
||
|
||
`profile_source` distinguishes how the profile was resolved (`flag` /
|
||
`plan_frontmatter` / `env` / `default`), so dashboards can surface
|
||
unexpected env-var inheritance in CI.
|
||
|
||
## Custom profiles
|
||
|
||
Drop a YAML file at `lib/profiles/<name>.yaml` to define a new tier.
|
||
The validator (`lib/validators/profile-validator.mjs`) enforces:
|
||
|
||
- Every `phase_models[].phase` must be a known phase enum:
|
||
`brief` / `research` / `plan` / `execute` / `review` / `continue`
|
||
- Every `phase_models[].model` must exactly match an entry in
|
||
`BASE_ALLOWED_MODELS` (`['sonnet', 'opus', 'fable']` in
|
||
`lib/validators/profile-validator.mjs`; `haiku` only with
|
||
`VOYAGE_ALLOW_HAIKU=1`)
|
||
- All six phases must be present (no partial profiles)
|
||
|
||
The four built-in names (`economy`, `balanced`, `premium`, `fable`) resolve to their
|
||
bundled yaml first — `findProfilePath()` returns the built-in before consulting
|
||
`voyage-profiles/`, so a same-named custom file is ignored and cannot shadow a
|
||
built-in. To customize, give your profile a new name and reference it via
|
||
`--profile <new-name>` or `VOYAGE_PROFILE=<new-name>`.
|
||
|
||
### Example custom profile
|
||
|
||
```yaml
|
||
# lib/profiles/critical.yaml — opus everywhere except continue
|
||
phase_models:
|
||
- phase: brief
|
||
model: opus
|
||
- phase: research
|
||
model: opus
|
||
- phase: plan
|
||
model: opus
|
||
- phase: execute
|
||
model: opus
|
||
- phase: review
|
||
model: opus
|
||
- phase: continue
|
||
model: sonnet
|
||
```
|
||
|
||
Validate with: `node lib/validators/profile-validator.mjs --json lib/profiles/critical.yaml`
|
||
|
||
## Drift detection
|
||
|
||
In `--strict` mode, `plan-validator.mjs` emits a `MANIFEST_PROFILE_DRIFT`
|
||
warning when the plan-level `profile:` differs from any step manifest's
|
||
`profile_used`. The warning is a *signal*, not a failure — the plan
|
||
remains `valid: true`. This catches:
|
||
|
||
- Manual edits where an operator changed a single step's profile
|
||
- Resume from a partial run where the previous session used a different
|
||
tier
|
||
- Copy-paste errors when stitching plan fragments
|
||
|
||
To suppress the warning intentionally (e.g. when a critical step
|
||
genuinely needs a higher tier), document the override in the step's
|
||
prose and re-run with `--soft` to validate without strict-mode warnings.
|
||
|
||
## Cost estimation
|
||
|
||
> **Disclaimer:** the table below is an *anslag*, not a contractual
|
||
> SLA. Real cost depends on context size, agent-swarm cardinality,
|
||
> tool-use density, and Claude Code billing schedule. Treat these as
|
||
> rough order-of-magnitude.
|
||
|
||
| Profile | Brief | Research | Plan | Execute | Review | Total |
|
||
|---------|-------|----------|------|---------|--------|-------|
|
||
| `economy` | $0.10–0.50 | $0.50–2.00 | $0.50–2.00 | $1.00–5.00 | $0.20–1.00 | **$2–10** |
|
||
| `balanced` | $0.10–0.50 | $0.50–2.00 | $1.00–4.00 | $1.00–5.00 | $0.50–2.00 | **$3–14** |
|
||
| `premium` | $0.50–2.00 | $0.50–2.00 | $1.00–4.00 | $1.00–5.00 | $0.50–2.00 | **$4–15** |
|
||
|
||
Numbers are per *full pipeline run* (brief + research + plan +
|
||
execute + review) on a moderate-complexity task. Numbers scale roughly
|
||
linearly with the size of the resulting plan (10 steps ≈ baseline; 30
|
||
steps ≈ 3× the execute column).
|
||
|
||
Per-profile actuals are emitted to JSONL stats — pipe them through the
|
||
OTel export (`docs/observability.md`) to get real cost-attribution
|
||
graphs in Grafana. Replace the table above with your own measured
|
||
numbers after ≥ 3 runs of each profile.
|
||
|
||
## Deferred to v4.2
|
||
|
||
- **`balanced.external_research_enabled` operator-override** —
|
||
v4.1 omits this per scope-guardian SG2. v4.2 may add an opt-in
|
||
flag to enable external research agents in the balanced tier
|
||
without forcing premium.
|
||
- **Empirical Jaccard re-calibration** — parked-synthetic fixtures
|
||
in v4.1 use a 0.55 conservative starting threshold. v4.2 plans an
|
||
empirical re-run with $60-120 LLM budget to derive a calibrated
|
||
threshold from real economy-vs-premium plan pairs.
|
||
- **ROUGE-L + char-4gram MinHash** as primary/secondary cross-tier
|
||
gates per research/02 Recommendation #7. Jaccard remains the gate
|
||
in v4.1; v4.2 may layer ROUGE-L on top.
|
||
|
||
## See also
|
||
|
||
- [`README.md` § Profile system](../README.md) — top-level overview
|
||
- [`CLAUDE.md` § Profile system](../CLAUDE.md) — internal reference
|
||
- [`docs/observability.md`](observability.md) — JSONL → OTel pipeline
|
||
- [`tests/synthetic/profile-jaccard-calibration.md`](../tests/synthetic/profile-jaccard-calibration.md)
|
||
— calibration status and threshold rationale
|
||
- [`lib/profiles/`](../lib/profiles/) — built-in profile YAMLs
|
||
- [`lib/validators/profile-validator.mjs`](../lib/validators/profile-validator.mjs)
|
||
— schema validator with CLI shim
|