voyage/docs/architecture.md
Kjell Tore Guttormsen 6dafdf2a2a fix(research-loop-cap): resolve the data root in code so the loop can run
CLAUDE_PLUGIN_DATA is empty in the Bash tool's process env, and the Phase 5
bash snippet is the cap's only caller. resolveLedgerPath() returned null there
and allowTurn() failed closed, so the budget gate denied turn 1 of every real
run: the loop this delivery exists to bound could never spend a turn, and the
pre-registered measurement could not be run at all.

resolveDataRoot() is now the single root for everything the loop writes --
CLAUDE_PLUGIN_DATA when the harness sets it, ~/.claude/voyage when it does
not. Three consumers resolve through it, which is the point: the cap ledger,
the PreToolUse hook's scope-marker lookup, and the command's bash snippets.
A writer and a reader that resolved the root separately are what made the
enforcement hook allow unconditionally in every real run while CLAUDE.md and
docs/architecture.md called it enforcing.

Same root cause, same commit:
- Marker write and remove now share ONE absolute-path guard and one root; the
  write requires a non-empty CLAUDE_CODE_SESSION_ID before composing the path
  (unset, the marker was named `.json`, which no lookup matches and no TTL
  sweep cleans up).
- The per-turn gates resolve VOYAGE_ROOT with a plugin-cache fallback and
  reserve exit 2 for "gate could not run". Interpolating an empty
  ${CLAUDE_PLUGIN_ROOT} ran `node /lib/...` -> exit 1, which the contract read
  as "privacy gate says no" -- an unsatisfiable rewrite loop no query could
  clear.

Two now-unreachable deny branches are removed rather than left as dead safety
claims (allowTurn's no_plugin_data_dir; the hook's uncountable-ledger deny).
The fail-closed stance stays where it is still real: a ledger that cannot be
WRITTEN denies the turn.

Verified end-to-end through the real bash snippets and the real hook with both
variables stripped and HOME sandboxed: marker written under the fallback root,
8 turns spent, 9th denied, hook exits 2, and exits 0 again after removal.
Note: the fallback exit-2 branch fires against the installed v5.9.1 cache,
which predates lib/util/research-loop-cap.mjs -- correct behaviour, and it
clears when the plugin is reinstalled.

Review findings 2670c10a, fbd6d534, 93550dfb.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011vPSXe88qp5aqWUqbDNWoF
2026-08-12 22:26:46 +02:00

140 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Voyage — Architecture, project layout, state, terminology
Imported from `CLAUDE.md` via pointer.
## Quality infrastructure (v3.4.0)
`lib/` contains zero-dep validators, parsers, and autonomy primitives wired into the four commands:
- `lib/util/{frontmatter,result,atomic-write,autonomy-gate}.mjs` — shared YAML-frontmatter parser + Result helpers + `atomicWriteJson(path, obj)` for tmp+rename writes + autonomy-gate state machine (v3.4.0)
- `lib/parsers/{plan-schema,manifest-yaml,project-discovery,arg-parser,bash-normalize,jaccard,finding-id}.mjs` — pure parsers (no I/O), unit-tested. `manifest-yaml` extended in v3.4.0 with additive `skip_commit_check` + `memory_write` flags (forward-compat: unknown keys ignored)
- `lib/review/{rule-catalogue,plan-review-dedup}.mjs` — version-pinned rule catalogue (12 keys) + Phase 9 inline dedup helpers (v3.4.0)
- `lib/stats/event-emit.mjs` — single-source stats event emitter for autonomy-gate transitions and main-merge-gate (v3.4.0)
- `lib/validators/{brief,research,plan,progress,session-state}-validator.mjs` — schema validators with CLI shims (`node lib/validators/X.mjs --json <path>`)
- `lib/validators/architecture-discovery.mjs` — drift-WARN external-contract discovery for `architecture/overview.md`
- `lib/util/research-loop-cap.mjs` — stateful, **default-off** turn budget for the `/trekresearch` bounded conversation loop. `allowTurn()` derives the used-turn count from its own append-only JSONL ledger; it never asks the caller how many turns it has spent, because a cap that does is not a cap. Budget = `TREKRESEARCH_MAX_CONV_TURNS` (default `3`, invalid values fall back to `3`) × `maxDimensions` (8, `settings.json:16`). Grants 0 unless `VOYAGE_STORM_ENABLED=1`, which is also the second condition on Phase 4.5's skip-guard (that phase does not call this module): unset, **both** STORM phases are inert. `resolveDataRoot()` is the single root for everything the loop writes — `CLAUDE_PLUGIN_DATA` when the harness sets it, `~/.claude/voyage` when it does not (it is empty in the Bash tool's process env, which is where the loop actually runs); the cap hook resolves through the same function, so writer and reader cannot disagree. A ledger that cannot be **written** still denies the turn (fail-closed — the opposite of `event-emit.mjs`, which is telemetry and must never block). CLI shim: `node lib/util/research-loop-cap.mjs --run-id ID --dimension D --effort E`
- `lib/validators/query-privacy-gate.mjs` — gates **every** outbound research query before it leaves the machine; the hard-block tier (secret-shaped strings) is not operator-overridable, so a query that trips it must be reformulated rather than forced through. CLI shim: `node lib/validators/query-privacy-gate.mjs "<query>"`
Wiring points (replaces previous prose-grep instructions):
- `/trekbrief` Phase 4g → `brief-validator` (post-write sanity check)
- `/trekplan` Phase 1 → `brief-validator --soft`, `research-validator --dir`, `architecture-discovery`
- `planning-orchestrator` Phase 5.5 → `plan-validator --strict` (replaces 3 `grep -cE` calls)
- `/trekexecute --validate``plan-validator --strict` + `progress-validator`
Tests under `tests/**/*.test.mjs` (~290 tests, 0 deps). `npm test` is the fork-readiness gate. v3.4.0 adds: synthetic determinism fixtures (`tests/synthetic/plan-run-*.md` + `review-run-*.md` + companion `*-determinism.test.mjs` enforcing Jaccard ≥ 0.833 SC7 floor) and hook baseline regression pins (`tests/hooks/{path-guard,bash-guard}.test.mjs` exercising `pre-write-executor.mjs` + `pre-bash-executor.mjs` denylist BLOCK paths).
Doc-consistency test at `tests/lib/doc-consistency.test.mjs` pins agent-table count, command-table coverage, plan_version invariant, settings.json scope cleanliness, Handover 7 presence, and `session-state-validator` CLI shim.
`docs/HANDOVER-CONTRACTS.md` is the single source of truth for the 7 pipeline handovers (brief→research, research→plan, architecture→plan EXTERNAL, plan→execute, progress.json resume, review→plan, `.session-state.local.json`). Read it before changing any artifact format.
`hooks/scripts/pre-compact-flush.mjs` (PreCompact event, CC v2.1.105+) fixes the documented P0 in `docs/trekexecute-v2-observations-from-config-audit-v4.md`: keeps `progress.json` in sync with git history before context compaction so `--resume` works after long conversations. Atomic write, monotonic only, never blocks compaction.
`hooks/scripts/session-title.mjs` (UserPromptSubmit, CC v2.1.94+) sets `sessionTitle` to `voyage:<command>:<slug>` for voyage-command invocations. Helps multi-session headless runs identify themselves in process lists.
`hooks/scripts/post-bash-stats.mjs` (PostToolUse, CC v2.1.97+) appends `duration_ms` for each Bash call into `${CLAUDE_PLUGIN_DATA}/trekexecute-stats.jsonl`. Useful for finding long-running verify or checkpoint commands.
`hooks/scripts/pre-agent-cap.mjs` (PreToolUse on `WebSearch|WebFetch|Task`) enforces the `/trekresearch` Phase 5 loop bound in the harness, so the cap is not merely prose the model is asked to obey. It counts spent turns read-only from the append-only ledger `research-loop-cap.mjs` writes, and denies (exit 2) once the budget is gone. Scope key = `session_id` + a marker file only the loop writes (`<resolveDataRoot()>/trekresearch-loop-scope/<session_id>.json`, the same root the ledger uses); without a marker the hook allows unconditionally, which is what keeps a globally-wired `PreToolUse` hook from over-blocking unrelated sessions. Stale markers auto-reset on a TTL and `VOYAGE_DISABLE_CAP_HOOK=1` is the kill switch. Defence in depth only — `lib/util/research-loop-cap.mjs` must stay correct if the hook stops firing (see `docs/spike-pretooluse-subagent-reach.md`).
`hooks/scripts/post-compact-flush.mjs` (PostCompact event, v3.4.0) re-injects `.session-state.local.json` after context compaction so multi-session work survives a compaction boundary. Companion to `pre-compact-flush.mjs` (which writes the state file before compaction); together they form the rehydrate cycle that keeps `/trekcontinue` reliable across long-running multi-session work.
## Architecture
**Brief:** 7-phase workflow: Parse mode → Create project dir → Phase 3 completeness loop (section-driven, no question cap) → Phase 3.5 per-phase effort dialog (v5.1) → Phase 4 draft/review/revise with `brief-reviewer` as stop-gate (max 3 iterations; gate = all dimensions ≥ 4 and research plan = 5) → Finalize (`brief.md` on pass, or `brief_quality: partial` on cap/force-stop) → Manual/auto opt-in → Stats. Always interactive. Auto mode runs research + plan inline in the main context (v2.4.0).
**Phase 3.5 (v5.1) — adaptive-depth signals:** Between Phase 3 completeness exit and Phase 4 draft, the operator commits an effort level (`low | standard | high`) and an optional `model` (`sonnet | opus | fable`) per downstream phase (`research`, `plan`, `execute`, `review`) via 4 tier-coupled `AskUserQuestion` calls. The choices land in `brief.md` frontmatter as `phase_signals:` (a list of `{phase, effort?, model?}` entries) when committed, or `phase_signals_partial: true` when the operator force-stops. `brief_version: 2.1` activates the **sequencing gate**: validator emits `BRIEF_V51_MISSING_SIGNALS` if a 2.1-versioned brief lacks both fields. Downstream commands surface a friendly hint pointing back to `/trekbrief` — enforcement is validator-only. Composition is documented prose in each downstream command's `## Composition rule (v5.1)` section: `brief.phase_signals[phase] > profile.phase_models[phase]`. The brief signal wins per-phase when present; the profile fills gaps. `effort == low` activates each command's existing `--quick`-equivalent code-path (`/trekexecute` low-effort = `--gates open` + sequential-only). High-effort behavior is deferred to v5.1.1 per brief Non-Goal.
**Research:** Foreground workflow (v2.4.0): Parse mode → Interview → Parallel research swarm (5 local + 4 external + 1 bridge, spawned from main context) → Follow-ups → Triangulation → Synthesis + brief → Stats. With `--project`, writes to `{dir}/research/NN-slug.md`.
**Plan:** Foreground workflow (v2.4.0): Parse mode (validate brief input) → Codebase sizing → Brief review (`brief-reviewer`) → Parallel exploration (6-8 agents, spawned from main context) → Deep-dives → Synthesis (with architecture-note cross-reference if present) → Planning → Adversarial review (`plan-critic` + `scope-guardian`) → Present/refine → Handoff. With `--project`, writes to `{dir}/plan.md` and auto-detects `{dir}/architecture/overview.md` (produced by an opt-in upstream architect plugin if installed; not bundled).
**Decompose:** Parse plan → Analyze step dependencies → Group into sessions → Identify parallel waves → Generate session specs + dependency graph + launch script.
**Execute:** Parse plan → Security scan (Phase 2.4) → Detect Execution Strategy → Single-session (step loop) or multi-session (parallel waves via `claude -p` with scoped `--allowedTools`) → Phase 7.5 manifest audit → Phase 7.6 bounded recovery (if partial) → Phase 8 atomically writes `progress.json` + `.session-state.local.json` (Handover 7) → Report. With `--project`, reads `{dir}/plan.md`. Phase 2.55 (pre-flight stop) and Phase 4 (entry-condition stop) also write `.session-state.local.json` so `/trekcontinue` can surface the stop and prompt for next steps.
**Continue:** `/trekcontinue` reads `{dir}/.session-state.local.json` (Handover 7), validates schema-v1 via `session-state-validator`, narrates a 3-line summary (project / next-session-label / brief-path), and immediately begins executing the next session. Auto-discovers active project state files under `.claude/projects/*/.session-state.local.json` if no explicit `<project-dir>` argument. Operator-invoked only — never auto-loaded via SessionStart. The `/trekendsession` helper is the informal-flow producer: writes the same state file for ad-hoc multi-session handovers that don't run through `/trekexecute`.
**Delegate the engine, keep the policy (cross-cutting; audited S32, 2026-06-20).** Every swarm and interview in the pipeline rides a *native* Claude Code primitive and re-implements no engine of its own: parallel sub-agent fan-out is a single-message multi-`Agent`-call dispatch (research swarm, exploration swarm, reviewer swarm), and interview turn-taking is `AskUserQuestion` (brief Phase 3, the research interview, the trekplan research-status gate, the trekreview scope gate). Voyage's value is the **policy** layered on top — never the scheduler, concurrency loop, or menu loop underneath: typed agent roles + effort defaults + codebase-size scaling (exploration); 4-angle decomposition (docs/community/security/contrarian) + per-source schemas + triangulation (research); the 12-key rule catalogue + no-cross-feed isolation + deterministic dedup/verdict (review); the section-driven completeness loop + framing gate (brief). Concretely: the brief's Phase-3 "selection rule" picks *which* question to ask (policy); `AskUserQuestion` does the asking (engine). The V01/V07/V08/V11/V24 balance-analysis audit confirmed all five already delegate natively — no engine re-implementation found; the doc-consistency test (`tests/lib/doc-consistency.test.mjs` § S32) pins the native-delegation prose so a hand-rolled engine cannot creep back in.
**Operator-UX guarantee (since v5.0.2):** `/trekbrief`, `/trekplan`, and `/trekreview` MUST always emit (a) a plain `file://<abs path>` URL AND (b) a copy-pasteable `open file://<abs path>` command in the final report block. The file:// URL must use an ABSOLUTE path (not relative or `~/`-prefixed) so terminals with cmd+click support (Ghostty, iTerm2, modern Terminal.app) can resolve it without shell interpretation. This is a non-negotiable operator-UX contract — the doc-consistency test pins both forms in all three commands' final report blocks.
**Operator-annotation HTML (v5.0.3):** the last step of `/trekbrief`, `/trekplan`, and `/trekreview` runs `scripts/annotate.mjs` against the just-written `.md` and prints the resulting `file://<abs path>` link. The HTML is self-contained (zero npm deps, zero external network, design-system-styled, light + dark + print) and modelled on `~/repos/claude-code-100x/claude-code-100x/build-site.js` (lines 14312255). The operator opens the file, the document renders as a proper article (headings / paragraphs / lists / tables / code / quotes — every element gets a stable `data-anchor-id`). In annotation mode (default ON, pencil-toggle in topbar), the operator can **select any text or click any element** → a form popover opens at the cursor with: section context auto-detected from nearest h1/h2, the anchored snippet (selection if any, else element text), **three intent buttons (Fiks / Endre / Spørsmål)**, comment textarea, Save/Cancel. The sidebar (Show annotations button) lists every annotation grouped by section with intent badge + snippet + comment + delete; clicking a card scrolls to and flashes the source element. **Copy Prompt** assembles a structured markdown (`### N. [Intent] Section: <…>` + `Quote: «…»` + `Comment: …`) and copies to clipboard. Persistence: `localStorage` keyed on absolute artifact path (`voyage-annotate:v2:<abs path>`). v5.0.0 removed the v4.2/v4.3 bespoke playground SPA + `/trekrevise` + Handover 8; v5.0.1 pointed at `/playground document-critique` (Claude-leads, wrong direction); v5.0.2 was operator-led but too thin (line-click + freeform note, no intents); v5.0.3 matches the claude-code-100x reference the operator first pointed at, with pencil-toggle / selection capture / intent categories / popover form / structured export. See [CHANGELOG.md](../CHANGELOG.md) § v5.0.3.
**Security:** 4-layer defense-in-depth: plugin hooks (pre-bash-executor, pre-write-executor), prompt-level denylist (works in headless sessions), pre-execution plan scan (Phase 2.4), scoped `--allowedTools` replacing `--dangerously-skip-permissions`. Hard Rules 14-16 enforce verify command security, repo-boundary writes, and sensitive path protection.
**Pipeline:** `/trekbrief` produces the task brief. `/trekresearch --project <dir>` fills in `{dir}/research/`. `/trekplan --project <dir>` reads brief + research to produce `{dir}/plan.md` (and auto-discovers `{dir}/architecture/overview.md` if an opt-in upstream architect plugin produced one). `/trekexecute --project <dir>` executes and writes `{dir}/progress.json`. `/trekreview --project <dir>` produces `{dir}/review.md`. `/trekbrief`, `/trekplan`, and `/trekreview` each end by running `scripts/annotate.mjs` on the just-written artifact, producing `{dir}/{artifact}.html` — a self-contained operator-annotation surface — and printing the `file://` link. The operator opens it, clicks lines, writes their own notes, copies a structured prompt, pastes back, Claude revises the `.md`. All artifacts live in one project directory.
**Project-directory contract (v3.0.0):** trekplan owns the directory layout below. The `architecture/` subdirectory is opt-in and produced by an opt-in upstream architect plugin (not bundled) — the architect plugin is no longer publicly distributed, but the `architecture/overview.md` slot remains available for any compatible producer.
```
.claude/projects/{YYYY-MM-DD}-{slug}/
brief.md ← trekbrief writes; everyone reads
brief.html ← trekbrief annotates (operator-annotation HTML, gitignored, re-buildable from brief.md)
research/*.md ← trekresearch writes; plan + architect read
architecture/ ← OPT-IN, owned by an opt-in upstream architect plugin (not bundled)
overview.md
gaps.md
plan.md ← trekplan writes; trekexecute reads
plan.html ← trekplan annotates
progress.json ← trekexecute writes
review.md ← trekreview writes; trekplan reads (Handover 6)
review.html ← trekreview annotates
```
The `.html` files (`brief.html`, `plan.html`, `review.html`) are produced by `scripts/annotate.mjs` and live alongside their `.md` siblings in the project directory. They are re-buildable from the `.md` source at any time (deterministic, byte-identical output on re-run), so they are conventionally gitignored along with the rest of `.claude/projects/`. Operator annotations live in browser `localStorage` keyed on the absolute artifact path — they survive refresh and browser-close, but are local to the operator's machine.
No code-level dependency between plugins — the contract is filesystem-level only.
### Primitives per step (decision matrix)
Which native Claude Code primitive each pipeline step runs on today, and the alternatives that were considered and where they landed. Companion to *Delegate the engine, keep the policy* above; the full decision records live in `docs/voyage-vs-cc-balance-analysis.md`, `docs/T1-cc26-delegated-orchestration.md`, and `docs/T2-cc27-workflow-substrate.md`.
| Step | Primitive today (production) | Alternatives considered (status) |
|---|---|---|
| **brief** | Inline interview loop → `brief.md` (framing gate) | `AskUserQuestion` as the Q&A engine (recommended, not yet wired); Skill/Workflow auto-orchestration (rejected — one-shot, can't span slash-command/handover boundaries) |
| **research** | Inline spawns a foreground swarm; MCP per agent¹ | Delegated `synthesis-agent` (PoC measured Δ main-context ≈ 0 → ships **dormant**, wired to nothing) |
| **plan** | Inline: parallel explorer swarm → synthesis → adversarial `plan-critic` | Delegated orchestrator via nested sub-agents (CC-26 → lean NO, not wired; `planning-orchestrator.md` is a reference doc, not a spawnable agent) |
| **execute** | Inline step loop; multi-session via `git worktree` + `claude -p` waves; deterministic manifest audit | CC `TaskCreate`/`TodoWrite` for progress/resume (insufficient — carries no step status / attempts / SHA / drift → own typed `progress.json` contract) |
| **review** | Inline parallel reviewers (no cross-feed) → `review-coordinator` Judge | **Workflow** substrate for Phase 56 (bake-off POSITIVE: +4.4 % tokens / +54 % wall-time → shipped **opt-in `--workflow`**, not default; wholesale substrate swap declined) |
| **continue** | Inline reads `.session-state.local.json` → zero-confirm resume | CC `--resume` (transcript replay, not typed work-state → insufficient) |
| **cross-cutting** | 8 hook scripts: `pre-bash` + `pre-write` guards, `pre-agent-cap` loop-bound enforcement, `post-bash` stats, `session-title`, `pre-`/`post-compact` flush, **`Stop`→OTEL** export | — |
¹ MCP per research agent: `docs-researcher` → Microsoft Learn + Tavily · `community-`/`security-`/`contrarian-researcher` → Tavily (+ WebSearch/WebFetch) · `gemini-bridge` → Gemini Deep Research MCP. Graceful degradation when an MCP server is absent.
**Legend:** *production* = wired and active · *dormant* = shipped but wired to nothing (`synthesis-agent`) · *opt-in* = behind a flag (`--workflow`) · *not wired* = considered, deferred (delegated orchestrator, the `AskUserQuestion` brief engine).
## State
All artifacts in one project directory (default):
- Project root: `.claude/projects/{YYYY-MM-DD}-{slug}/`
- `brief.md` + `brief.html` (task brief from `/trekbrief`; `.html` is the operator-annotation surface from `scripts/annotate.mjs`)
- `research/{NN}-{slug}.md` (research briefs from `/trekresearch --project`)
- `architecture/overview.md` + `architecture/gaps.md` (opt-in, produced by an opt-in upstream architect plugin, not bundled)
- `plan.md` + `plan.html` (from `/trekplan --project`)
- `sessions/session-*.md` (from `--decompose`)
- `progress.json` (from `/trekexecute --project`)
- `review.md` + `review.html` (from `/trekreview --project`)
- `.session-state.local.json` (Handover 7 — gitignored via `*.local.json`; written by `/trekexecute` Phase 8/2.55/4 or `/trekendsession`; read by `/trekcontinue`)
Legacy paths (still work without `--project`):
- Research briefs: `.claude/research/trekresearch-{date}-{slug}.md`
- Plans: `.claude/plans/trekplan-{date}-{slug}.md`
- Sessions: `.claude/trekplan-sessions/{slug}/session-*.md`
- Launch scripts: `.claude/trekplan-sessions/{slug}/launch.sh`
- Progress: `{plan-dir}/.trekexecute-progress-{slug}.json`
Stats:
- Brief stats: `${CLAUDE_PLUGIN_DATA}/trekbrief-stats.jsonl`
- Plan stats: `${CLAUDE_PLUGIN_DATA}/trekplan-stats.jsonl`
- Exec stats: `${CLAUDE_PLUGIN_DATA}/trekexecute-stats.jsonl`
- Research stats: `${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl`
- Continue stats: `${CLAUDE_PLUGIN_DATA}/trekcontinue-stats.jsonl`
## Terminology
- **Task brief** — produced by `/trekbrief`. Declares intent, goal, and research plan. Drives planning.
- **Research brief** — produced by `/trekresearch`. Answers a specific research question. Feeds planning.
- **Architecture note** — opt-in, produced by an opt-in upstream architect plugin (not bundled; the architect plugin is no longer publicly distributed, but the `architecture/overview.md` filesystem slot remains available for any compatible producer). Proposes which Claude Code features fit the task with brief-anchored rationale + explicit gaps. When present, enriches planning.
- **Review** — produced by `/trekreview`. Independent post-hoc review of delivered code against the task brief. **Handover 6 (review → plan)** routes BLOCKER + MAJOR findings into `/trekplan --brief review.md` for a remediation plan. The plan's optional `source_findings:` frontmatter list is the audit trail back to the consumed findings. MINOR + SUGGESTION are skipped for v1.0 plan-input.
- **Session state** — `.session-state.local.json` per project. **Handover 7** — produced by any session-end mechanism (`/trekexecute` Phase 8/2.55/4, `/trekendsession` helper, future graceful-handoff v2.2). Consumed by `/trekcontinue` to resume the next session in a fresh chat. Schema v1 is forward-compat (unknown top-level keys ignored). Never committed (gitignored via `*.local.json`).
A project typically has 1 task brief, 0N research briefs, 0 or 1 architecture note, 0N reviews (one per review iteration), and 0 or 1 session-state file (overwritten on every session-end).